Method

Each of the 100 tasks describes a plant: its devices, the I/O map with exact addresses, the operating specification, the completion condition and the safety rules. Tasks come in two suites, motion control and material handling, at three levels of difficulty.

The agent works in a sandbox through its own harness, such as Claude Code or Codex. Its submission is one IEC 61131-3 Structured Text program at /app/program.st. No PLC compiler or plant simulator is provided, so the program has to be right before it ever runs.

A separate verifier receives that one file. It compiles the program and runs it scan by scan, every 10 ms of plant time, in closed loop with a simulated plant: inputs are sampled, the program executes, outputs move the devices, and the plant advances.

Every task runs under several conditions, 476 across the benchmark, with device parameters, timings and disturbances varied inside the published ranges. Task success and safety are judged separately. A trial earns reward 1 if every condition reaches task success with no safety violation; anything less earns 0.

Pass@1 is the mean reward over a model’s trials, usually one per task on Easy and Medium and three on Hard. Hard pass@3 counts a Hard task as solved if any of its trials passed.

Before ranking models we audited every task of PLCWorld 5.2.6 for ways a score could reflect something other than control skill: rules the grader enforces but the prompt never states, signals whose meaning is ambiguous, a runtime that divided integers as floating point, conditions a program ignoring its sensors could still pass, and solutions that pass by idling.

5.2.6-ef1.2 clarifies prompts, adjusts or adds conditions so that blind schedules fail, and fixes the shared evaluator. Results on this site come from evaluator 5.2.6-ef1.2; reference replays of the original programs use the upstream evaluator (plcworld-evaluation/5.2.6).

Track 1 is adapted from PLCWorld: PLCWorld: Benchmarking LLM-Generated PLC Programs in Closed-Loop Plant Simulation, by Yunji Kim, Yunseok Lee, Hyunwoo Seo, Jaerim Choi, Woojin Lee (arXiv 2610.02982). The tasks, plants and original evaluator are theirs. The Harbor port, the audit, 5.2.6-ef1.2 and this site are by Environment Foundry.

PLCBench (arXiv 2608.26882, Zhou et al.) connects real controllers to closed-loop simulated processes and asks a different question: whether autonomous agents with network access to a PLC can cause sustained physical impact.

PLCWorld itself reports single-turn results on the original package; the leaderboard lists them apart because the two protocols are not directly comparable.

The tasks are public in Harbor format at github.com/Koomook/autofactory-bench (tag v1.2), as three datasets. Hard is the headline one.

• autofactory-bench@1.2Hard

  Tasks

  30

  Conditions

  170

  Reference programs

  26

• autofactory-bench-medium@1.2Medium

  Tasks

  40

  Conditions

  198

  Reference programs

  40

• autofactory-bench-easy@1.2Easy

  Tasks

  30

  Conditions

  108

  Reference programs

  30

REG=https://raw.githubusercontent.com/Koomook/autofactory-bench/main/registry.json
# Hard, the headline dataset
harbor run -d autofactory-bench@1.2 --registry-url $REG -a claude-code -m anthropic/claude-opus-5-5
# Medium
harbor run -d autofactory-bench-medium@1.2 --registry-url $REG -a codex -m openai/gpt-6.1-sol
# Easy, with the reference programs
harbor run -d autofactory-bench-easy@1.2 --registry-url $REG -a oracle

Grading runs in a separate verifier container. /app/program.st is the one file that crosses over from the agent’s environment, which has no PLC compiler or plant simulator, and the verifier is downloaded from the v1.2 tag when the image is built. The repository is public, so treat any run that reads the tasks’ tests, solutions or the verifier as invalid.

94 of 100 tasks ship a passing program for the oracle agent: 80 saved PLCWorld paper submissions and, for 14 Hard tasks with no public passing program, passing agent programs from our runs. Six Hard tasks (MAT-H05, MOT-H02, MOT-H03, MOT-H04, MOT-H06, MOT-H13) still have none; the PLCWorld authors’ reference recordings show they are solvable. Run end to end from the public registry, the oracle agent scores 1 on MAT-E01 and MAT-H04 and an agent that does nothing scores 0.

Environment Foundry’s packaging is Apache-2.0. PLCWorld content is redistributed with the authors’ permission and is not covered by that license; see NOTICE.md in the repository.

Add a model or report an error: bong@team-attention.com