AFB

AutoFactory Index

One benchmark score per model and harness, in the style of the Artificial Analysis Coding Agent Index. Its components weigh the same, every task weighs the same inside a component, and a trial passes only with zero safety violations. Cost, the safety gap, the violation profile and consistency sit beside the index, not in it. 9 models; every interval from 2000 task-level bootstrap resamples. Index afi-1.0.

Top index
86.1
GPT-6 Astra
Codex
Fewest trials with a violation
2.8%
GPT-6 Astra
Codex
Fewest S3 trials (draft)
1.7%
GPT-6 Astra
Codex
Best Hard pass^3
60.0%
GPT-6 Astra
Codex
Cheapest safe success
$0.025
DeepSeek V4.1 Flash
mini-swe-agent

What the index shows

  1. 01The top 2 cannot be ranked against each other: GPT-6 Astra, GPT-6.1 Sol. Their paired differences include zero.
  2. 02Violations cost DeepSeek V4.1 Flash the most: 65.6 points if violations are ignored, 58.4 with the safety gate, a gap of 7.2.
  3. 03DeepSeek V4.1 Flash has the most trials with a physical-damage (S3, draft) violation: 19%.
  4. 04Opus 5.5 solves 87% of Hard tasks in at least one of three trials, but only 21% in all three.
  5. 05A full run costs up to 419× more on one model than another. On the cost frontier: GPT-6 Astra, GPT-6.1 Sol, DeepSeek V4.1 Flash, GLM-5.3 Flash, GPT-6 Luna.
  6. 06Harness and model are confounded: each model ran in one harness, and open models are priced at a discounted gateway rate. Read gaps between harnesses as system differences.

Every row is a model in a harness. The harness shows as marker shape, colour and name.

  • Codex
  • Claude Code
  • mini-swe-agent

Inside the index

Benchmark scores only. A trial passes when every test condition reaches its goal with zero safety violations, so safety is part of the pass, as a failing test is in a coding benchmark.

AutoFactory Index

The mean of the included components’ safe-success rates, 0–100. Whiskers are 95% intervals; models whose intervals overlap are not ranked against each other.

025507510001GPT-6 AstraCodex86.102GPT-6.1 SolCodex85.003Fable 5.1Claude Code74.204Opus 5.5Claude Code70.905Sonnet 5.5Claude Code63.506DeepSeek V4.1 Flashmini-swe-agent58.407Haiku 5.5Claude Code49.108GLM-5.3 Flashmini-swe-agent30.109GPT-6 LunaCodex28.0
Components: PLC Hard (30 tasks) and PLC Easy + Medium (70 tasks), equal weight. A task’s score is the mean over its attempts. Machine understanding (FactoryBench) is pending a grader repair and not included. Intervals from 2000 cluster-bootstrap resamples of tasks, stratified by component.
Show the data
AutoFactory Index
Model and harnessAutoFactory Index95% CIRank intervalTierPLC HardPLC Easy + Medium
GPT-6 Astra · Codex86.178.9–92.81–2172.2 [57.8–85.6]100.0 [100.0–100.0]
GPT-6.1 Sol · Codex85.078.3–91.71–2170.0 [56.6–83.3]100.0 [100.0–100.0]
Fable 5.1 · Claude Code74.266.5–81.83–4255.6 [41.1–68.9]92.9 [85.7–98.6]
Opus 5.5 · Claude Code70.963.6–77.83–5260.3 [49.2–70.8]81.4 [72.9–90.0]
Sonnet 5.5 · Claude Code63.555.5–71.74–6342.7 [29.3–57.1]84.3 [75.7–92.9]
DeepSeek V4.1 Flash · mini-swe-agent58.452.1–65.65–6321.1 [10.0–34.4]95.7 [90.0–100.0]
Haiku 5.5 · Claude Code49.143.5–55.67–7411.1 [3.3–21.1]87.1 [78.6–94.3]
GLM-5.3 Flash · mini-swe-agent30.124.0–36.28–954.4 [1.1–8.9]55.7 [44.3–67.1]
GPT-6 Luna · Codex28.021.1–34.98–958.9 [2.2–17.8]47.1 [35.7–58.6]

Components

Each component’s safe-success rate. Easy and Medium are near the ceiling for most models; Hard separates them.

Components
Model and harnessPLC HardPLC Easy + MediumMachine understandingpendingAutoFactory Index
GPT-6 AstraCodex72.2100.0pending86.1
GPT-6.1 SolCodex70.0100.0pending85.0
Fable 5.1Claude Code55.692.9pending74.2
Opus 5.5Claude Code60.381.4pending70.9
Sonnet 5.5Claude Code42.784.3pending63.5
DeepSeek V4.1 Flashmini-swe-agent21.195.7pending58.4
Haiku 5.5Claude Code11.187.1pending49.1
GLM-5.3 Flashmini-swe-agent4.455.7pending30.1
GPT-6 LunaCodex8.947.1pending28.0
The thin line inside each bar is the 95% interval. Hover a cell for its bounds. The data table splits Easy, Medium and Hard.
Show the data
Components
Model and harnessEasyMediumHard
GPT-6 Astra · Codex100.0 [100.0–100.0]100.0 [100.0–100.0]72.2 [57.8–85.6]
GPT-6.1 Sol · Codex100.0 [100.0–100.0]100.0 [100.0–100.0]70.0 [56.6–83.3]
Fable 5.1 · Claude Code93.3 [83.3–100.0]92.5 [82.9–100.0]55.6 [41.1–68.9]
Opus 5.5 · Claude Code86.7 [73.5–97.2]77.5 [64.3–90.0]60.3 [49.2–70.8]
Sonnet 5.5 · Claude Code76.7 [60.7–91.4]90.0 [80.0–97.7]42.7 [29.3–57.1]
DeepSeek V4.1 Flash · mini-swe-agent93.3 [83.3–100.0]97.5 [91.7–100.0]21.1 [10.0–34.4]
Haiku 5.5 · Claude Code93.3 [82.8–100.0]82.5 [70.3–93.3]11.1 [3.3–21.1]
GLM-5.3 Flash · mini-swe-agent76.7 [61.5–90.9]40.0 [25.0–55.6]4.4 [1.1–8.9]
GPT-6 Luna · Codex33.3 [17.1–50.0]57.5 [41.9–73.5]8.9 [2.2–17.8]

Outside the index

Separate axes, never mixed into the score: how much violations cost (the safety gap), what kind of violations (draft severity), whether a rerun still passes (consistency), and what a run costs.

Safety gap

The outline is the success rate if violations were ignored; the filled bar is safe success, the score inside the index. The number is the gap.

0255075100DeepSeek V4.1 Flashmini-swe-agent−7.2Haiku 5.5Claude Code−4.2Fable 5.1Claude Code−3.9Sonnet 5.5Claude Code−1.6Opus 5.5Claude Code−1.5GLM-5.3 Flashmini-swe-agent−1.4GPT-6 AstraCodex−1.1GPT-6.1 SolCodex0.0GPT-6 LunaCodex0.0
Ungated success: every condition reaches its goal, violations ignored. Safe success: the same with zero violations. Gap = ungated − safe, in index points or component percentage points.
Show the data
Safety gap
Model and harnessPLC Hard: ungatedPLC Hard: safePLC Hard: gapPLC Easy + Medium: ungatedPLC Easy + Medium: safePLC Easy + Medium: gapIndex: ungatedIndex: safeIndex: gap
GPT-6 Astra · Codex74.472.22.2 [0.0–6.7]100.0100.00.0 [0.0–0.0]87.286.11.1 [0.0–3.3]
GPT-6.1 Sol · Codex70.070.00.0 [0.0–0.0]100.0100.00.0 [0.0–0.0]85.085.00.0 [0.0–0.0]
Fable 5.1 · Claude Code63.355.67.8 [1.1–15.6]92.992.90.0 [0.0–0.0]78.174.23.9 [0.6–7.8]
Opus 5.5 · Claude Code63.360.33.1 [0.0–6.7]81.481.40.0 [0.0–0.0]72.470.91.5 [0.0–3.3]
Sonnet 5.5 · Claude Code44.442.71.8 [0.0–4.7]85.784.31.4 [0.0–4.3]65.163.51.6 [0.0–3.7]
DeepSeek V4.1 Flash · mini-swe-agent35.621.114.4 [6.7–23.3]95.795.70.0 [0.0–0.0]65.658.47.2 [3.3–11.7]
Haiku 5.5 · Claude Code19.611.18.4 [3.1–14.7]87.187.10.0 [0.0–0.0]53.349.14.2 [1.6–7.3]
GLM-5.3 Flash · mini-swe-agent4.44.40.0 [0.0–0.0]58.655.72.9 [0.0–7.1]31.530.11.4 [0.0–3.6]
GPT-6 Luna · Codex8.98.90.0 [0.0–0.0]47.147.10.0 [0.0–0.0]28.028.00.0 [0.0–0.0]

Violation profile

How often a model trips a safety rule, and how bad the worst one was. Classes are a draft, pending review by a PLC engineer.

  • S3 physical damage
  • S2 product or process
  • S1 procedure
  • Rule not reported
0%10%20%30%GPT-6 AstraCodex2.8%GPT-6.1 SolCodex3.3%Opus 5.5Claude Code6.5%Fable 5.1Claude Code10.6%Sonnet 5.5Claude Code11.0%GLM-5.3 Flashmini-swe-agent23.9%Haiku 5.5Claude Code25.6%DeepSeek V4.1 Flashmini-swe-agent27.9%GPT-6 LunaCodex28.3%
Share of trials (or test conditions) with at least one violation of a safety-scope rule, task-weighted like the index. Each takes its worst class. Draft classes: S3 physical damage (collision, tilt or drop, travel limit, unsafe extraction or vessel move, station moved during coupling, moving contact lost); S2 product or process damage (wrong gripper, fixture or jig, occupied receiver, overflow, misaligned coupling, missing insert, capacity conflict, unsupported release, regrasp or bundle); S1 procedure (invalid request, axis busy, other). Goal rules never count.
Show the data
Violation profile
Model and harnessTrials with a violationS3 physical damageS2 product or processS1 procedureRule not reported
GPT-6 Astra · Codex2.8% [0.0%–7.2%]1.7 [0.0–5.0]1.1 [0.0–3.3]0.0 [0.0–0.0]0.0 [0.0–0.0]
GPT-6.1 Sol · Codex3.3% [0.6%–7.2%]2.2 [0.0–6.1]0.6 [0.0–1.7]0.6 [0.0–1.7]0.0 [0.0–0.0]
Fable 5.1 · Claude Code10.6% [5.6%–16.1%]2.2 [0.0–5.0]3.9 [0.6–8.3]3.9 [1.1–7.2]0.6 [0.0–1.7]
Opus 5.5 · Claude Code6.5% [2.5%–11.2%]1.8 [0.0–4.8]1.5 [0.0–3.5]1.1 [0.0–2.8]2.0 [0.0–4.5]
Sonnet 5.5 · Claude Code11.0% [5.3%–17.0%]4.0 [0.7–8.1]2.8 [0.6–5.6]2.7 [0.3–5.9]1.4 [0.0–3.6]
DeepSeek V4.1 Flash · mini-swe-agent27.9% [20.7%–35.1%]19.1 [11.8–26.3]5.0 [1.7–9.4]2.8 [0.6–6.1]1.1 [0.0–3.3]
Haiku 5.5 · Claude Code25.6% [19.6%–31.4%]17.1 [10.5–23.6]5.0 [1.7–9.4]2.3 [0.6–4.7]1.1 [0.0–3.3]
GLM-5.3 Flash · mini-swe-agent23.9% [16.4%–31.8%]13.3 [7.1–19.4]6.4 [2.9–10.2]1.7 [0.0–4.4]2.5 [0.0–5.8]
GPT-6 Luna · Codex28.3% [21.1%–35.0%]18.3 [10.6–26.1]8.9 [3.9–15.0]1.1 [0.0–2.8]0.0 [0.0–0.0]

Safety rules seen, with their draft class (pending engineer review)

Safety rules seen, with their draft class (pending engineer review)
RuleDraft classViolating conditions
no_station_moved_during_couplingS3177
no_fork_travel_collisionS3103
no_travel_limitS376
no_support_tilt_exceededS360
no_unsafe_vessel_moveS348
no_head_collisionS326
no_table_access_collisionS326
no_index_with_pin_insertedS324
no_moving_contact_lostS324
no_lift_moved_during_dockS318
no_sway_handoff_unsafeS312
no_unsafe_extractionS312
no_formation_support_lostS310
no_shuttle_moved_with_fork_outS310
no_extended_boom_sweepS39
no_gantry_rackingS37
no_rotation_before_clearanceS37
no_separator_collisionS36
no_bearing_moment_exceededS34
no_distributed_support_skewS34
opposed_drivesS31
no_coupling_misalignedS291
no_station_capacity_conflictS254
no_unsupported_bundleS254
no_wrong_gripperS238
no_occupied_receiverS236
no_unsupported_regraspS235
no_unsafe_reel_changeS230
no_wrong_fixtureS227
no_unsupported_stackS224
no_unavailable_jigS218
no_uncut_releaseS218
no_unsupported_transferS218
no_insert_missingS217
no_tank_emptyS212
unsupported_releaseS212
no_buffer_depletedS210
no_connected_waste_rollS29
no_incompatible_holdersS29
no_overwrapS29
no_pallet_not_supportedS29
no_unsupported_spareS29
no_unsupported_toolS28
no_incomplete_bundleS27
no_occupied_resourceS26
no_partial_deck_loadS26
no_receiver_overflowS26
no_pump_without_pathS25
no_unaligned_station_entryS24
no_incompatible_station_routeS23
no_skeleton_overflowS21
no_unfinished_wrapS21
no_invalid_requestS1151
no_frame_changed_while_movingS121
no_axis_busyS118
no_face_occludedS118
no_other_tool_engagedS118
no_aperture_lostS19
no_fork_not_dockedS19
no_pin_approach_before_poseS18
no_command_out_of_rangeS17
no_cross_zone_released_earlyS16
no_shared_master_stoppedS16
no_gear_tracking_lostS11
no_pack_head_busyS11

Consistency on Hard

Three trials per Hard task. The hollow mark solves a task every time, the filled mark at least once; the longer the line, the more a rerun matters.

0%20%40%60%80%100%GPT-6 AstraCodexGPT-6.1 SolCodexFable 5.1Claude CodeSonnet 5.5Claude CodeOpus 5.5Claude CodeDeepSeek V4.1 Flashmini-swe-agentHaiku 5.5Claude CodeGPT-6 LunaCodexGLM-5.3 Flashmini-swe-agent
pass^3, all three trialspass@1pass@3, any of threeOnly Hard has repeats, so the intervals are wide. Hover for them.
Show the data
Consistency on Hard
Model and harnesspass^3pass@1pass@3
GPT-6 Astra · Codex60.0 [43.3–76.7]72.2 [57.8–85.6]83.3 [70.0–96.7]
GPT-6.1 Sol · Codex56.7 [40.0–73.3]70.0 [56.6–83.3]86.7 [73.3–96.7]
Fable 5.1 · Claude Code36.7 [20.0–53.3]55.6 [41.1–68.9]80.0 [63.3–93.3]
Opus 5.5 · Claude Code20.8 [7.5–36.7]60.3 [49.2–70.8]86.7 [73.3–96.7]
Sonnet 5.5 · Claude Code24.7 [11.3–40.0]42.7 [29.3–57.1]63.3 [46.7–80.0]
DeepSeek V4.1 Flash · mini-swe-agent13.3 [3.3–26.7]21.1 [10.0–34.4]36.7 [20.0–53.3]
Haiku 5.5 · Claude Code6.7 [0.0–16.7]11.1 [3.3–21.1]20.0 [6.7–33.3]
GLM-5.3 Flash · mini-swe-agent0.0 [0.0–0.0]4.4 [1.1–8.9]13.3 [3.3–26.7]
GPT-6 Luna · Codex3.3 [0.0–10.0]8.9 [2.2–17.8]16.7 [6.7–30.0]

Index vs cost to run

Up and to the left is better. The dashed staircase is the cost frontier: no model is both cheaper and higher on the index than one on it.

Most attractive20406080100$1$3$10$30$100$300$1000Cost to run the benchmark (USD)AutoFactory IndexGPT-6 AstraGPT-6.1 SolFable 5.1Opus 5.5Sonnet 5.5DeepSeek V4.1 FlashHaiku 5.5GLM-5.3 FlashGPT-6 Luna
Cost axis is logarithmic. The shaded corner is cheaper than the median model and scores above the median.
Show the data
Index vs cost to run
Model and harnessAutoFactory IndexCost to run the benchmark (USD)Cost per safe successOn frontier
GPT-6 Astra · Codex86.1 [78.9–92.8]$115 [$105–$125]$0.855 [$0.736–$0.989]●
GPT-6.1 Sol · Codex85.0 [78.3–91.7]$27.24 [$24.33–$30.01]$0.205 [$0.172–$0.242]●
Fable 5.1 · Claude Code74.2 [66.5–81.8]$589 [$498–$678]$5.12 [$4.06–$6.39]
Opus 5.5 · Claude Code70.9 [63.6–77.8]$90.86 [$80.11–$101]$0.817 [$0.671–$0.983]
Sonnet 5.5 · Claude Code63.5 [55.5–71.7]$40.38 [$34.75–$45.82]$0.415 [$0.317–$0.534]
DeepSeek V4.1 Flash · mini-swe-agent58.4 [52.1–65.6]$2.17 [$2.01–$2.32]$0.025 [$0.021–$0.030]●
Haiku 5.5 · Claude Code49.1 [43.5–55.6]$5.42 [$4.74–$6.14]$0.076 [$0.061–$0.093]
GLM-5.3 Flash · mini-swe-agent30.1 [24.0–36.2]$2.04 [$1.50–$2.66]$0.047 [$0.033–$0.067]●
GPT-6 Luna · Codex28.0 [21.1–34.9]$1.41 [$1.26–$1.55]$0.034 [$0.026–$0.048]●

Cost to run the benchmark

USD for one pass over all 100 tasks at the protocol’s trial counts (160 trials), from the cost each scored trial recorded. Log scale.

$1$3$10$30$100$300$1000Fable 5.1Claude Code$589GPT-6 AstraCodex$115Opus 5.5Claude Code$90.86Sonnet 5.5Claude Code$40.38GPT-6.1 SolCodex$27.24Haiku 5.5Claude Code$5.42DeepSeek V4.1 Flashmini-swe-agent$2.17GLM-5.3 Flashmini-swe-agent$2.04GPT-6 LunaCodex$1.41
Claude Code and Codex rows use the API list price the harness reports; mini-swe-agent rows use their tokens at the OpenGateway ultrafast price. Trials voided by rate limits are not included, so real spend is higher. The table adds agent time and tokens.
Show the data
Cost to run the benchmark
Model and harnessCost to run the benchmark (USD)Cost per trialCost per safe successTime per trialAgent hoursInput tokens / trialOutput tokens / trial
GPT-6 Astra · Codex$115 [$105–$125]$0.722$0.855 [$0.736–$0.989]243 s10.8176k7k
GPT-6.1 Sol · Codex$27.24 [$24.33–$30.01]$0.170$0.205 [$0.172–$0.242]296 s13.1184k11k
Fable 5.1 · Claude Code$589 [$498–$678]$3.68$5.12 [$4.06–$6.39]707 s31.4649k51k
Opus 5.5 · Claude Code$90.86 [$80.11–$101]$0.568$0.817 [$0.671–$0.983]217 s9.7168k19k
Sonnet 5.5 · Claude Code$40.38 [$34.75–$45.82]$0.252$0.415 [$0.317–$0.534]155 s6.9101k17k
DeepSeek V4.1 Flash · mini-swe-agent$2.17 [$2.01–$2.32]$0.013$0.025 [$0.021–$0.030]692 s30.84.1M80k
Haiku 5.5 · Claude Code$5.42 [$4.74–$6.14]$0.034$0.076 [$0.061–$0.093]198 s8.8131k40k
GLM-5.3 Flash · mini-swe-agent$2.04 [$1.50–$2.66]$0.013$0.047 [$0.033–$0.067]533 s23.7800k32k
GPT-6 Luna · Codex$1.41 [$1.26–$1.55]$0.0088$0.034 [$0.026–$0.048]120 s5.3148k11k

Read with care

  • Safety is the pass condition, not a weight: a program that crashes a robot is a wrong answer. The safety gap and violation profile show what the gate removed.
  • Severity classes are a draft assigned by rule name and are pending review by a PLC engineer. Some tasks flag a violation without naming the rule; those show as “rule not reported”.
  • Cost bases differ: Claude and GPT at API list price, open models at a discounted gateway price. At list price the open models would cost more.
  • Claude ran in Claude Code, GPT in Codex and the open models in mini-swe-agent. Harness effects cannot be separated from model effects here.
  • Easy and Medium ran once per task, so consistency is measured on Hard only.
  • Machine understanding (FactoryBench) joins the index after its grader is repaired; the version number will change when it does.

Index afi-1.0 · evaluator 5.2.6-ef1.2 · generated 2026-10-24

Back to the leaderboard