Leaderboard
72.2%
best on Hard
GPT-6 Astra
OpenAI · Codex
Easy and Medium are saturated. Hard decides.
2 of 9 models pass every Easy and Medium task.
100 PLC tasks from PLCWorld 5.2.6-ef1.2, each run in closed loop against a simulated plant. Agents run through their own harness; the program they leave is what gets graded. A trial passes only when every condition succeeds with zero safety violations, so safety is inside the score; cost is shown separately.
Pass@1 on the 30 Hard tasks
Percent, except trials
- 01
GPT-6 Astra
OpenAI · Codex
72.2Hard pass@1
- Medium
- 100.0
- Easy
- 100.0
- All
- 91.7
- Hard pass@3
- 83.3
- Violations
- 3.3
- Cost / trial
- $0.722
- Trials
- 160
- 02
GPT-6.1 Sol
OpenAI · Codex
70.0Hard pass@1
- Medium
- 100.0
- Easy
- 100.0
- All
- 91.0
- Hard pass@3
- 86.7
- Violations
- 3.8
- Cost / trial
- $0.170
- Trials
- 160
- 03
Opus 5.5
Anthropic · Claude Code
60.3Hard pass@1
- Medium
- 77.5
- Easy
- 86.7
- All
- 75.1
- Hard pass@3
- 86.7
- Violations
- 7.5
- Cost / trial
- $0.563
- Trials
- 163
- 04
Fable 5.1
Anthropic · Claude Code
55.6Hard pass@1
- Medium
- 92.5
- Easy
- 93.3
- All
- 81.7
- Hard pass@3
- 80.0
- Violations
- 12.6
- Cost / trial
- $3.68
- Trials
- 160
- 05
Sonnet 5.5
Anthropic · Claude Code
42.7Hard pass@1
- Medium
- 90.0
- Easy
- 76.7
- All
- 71.8
- Hard pass@3
- 63.3
- Violations
- 8.8
- Cost / trial
- $0.251
- Trials
- 166
- 06
DeepSeek V4.1 Flash
DeepSeek · mini-swe-agent
21.1Hard pass@1
- Medium
- 97.5
- Easy
- 93.3
- All
- 73.3
- Hard pass@3
- 36.7
- Violations
- 28.4
- Cost / trial
- $0.013
- Trials
- 160
- 07
Haiku 5.5
Anthropic · Claude Code
11.1Hard pass@1
- Medium
- 82.5
- Easy
- 93.3
- All
- 64.3
- Hard pass@3
- 20.0
- Violations
- 28.6
- Cost / trial
- $0.034
- Trials
- 165
- 08
GPT-6 Luna
OpenAI · Codex
8.9Hard pass@1
- Medium
- 57.5
- Easy
- 33.3
- All
- 35.7
- Hard pass@3
- 16.7
- Violations
- 27.5
- Cost / trial
- $0.0088
- Trials
- 160*
- 09
GLM-5.3 Flash
Z.ai · mini-swe-agent
4.4Hard pass@1
- Medium
- 40.0
- Easy
- 76.7
- All
- 40.3
- Hard pass@3
- 13.3
- Violations
- 25.5
- Cost / trial
- $0.013
- Trials
- 160
The paper’s own results, not directly comparable
The PLCWorld paper reports models answering in a single turn on the original 5.2.6 package. The table above uses agent harnesses that can read, run and revise, on the audited 5.2.6-ef1.2, so the two measure different things.
| Model, single-turn | Easy | Medium | Hard | All |
|---|---|---|---|---|
| GPT-5.5 | 83.3 | 81.7 | 17.8 | 63.0 |
| Gemini 3.1 Pro | 82.2 | 77.5 | 16.7 | 60.7 |
| Gemini 3.8 Flash | 95.6 | 73.3 | 5.6 | 59.7 |
| Claude Sonnet 5 | 73.3 | 67.5 | 2.2 | 49.7 |
| Qwen3.5-Plus | 67.8 | 40.8 | 0.0 | 36.7 |
| GPT-4o | 16.7 | 1.7 | 0.0 | 5.7 |
Source: the PLCWorld paper (arXiv 2610.02982), original 5.2.6 package. Single-turn generation, 3 repeats.
Each task page replays a trial that succeeded and one that failed. Browse the tasks