The Benchmark Is a Chart, Too
OpenAI GPT-5.6 compatibility evidence · snapshot 2026-07-27
The benchmark is a chart, too.
A benchmark score can improve for different reasons. A model might read more values correctly, or it might get better at declining questions the chart cannot answer. Compare the original tests with three later trials to see which changed here.
- Baseline evidence
- 516full suite, one response per case
- Follow-up evidence
- 6033 independently submitted targeted trials
- Repeated outcomes
- 540 + 63grounding answers + generated proposals
- Estimated API cost
- $4.24baseline plus follow-up
Later repeated trials
Clearer source facts improved the repeated tests.
The follow-up repeated every previously failing generation fixture and every answerable grounding question across Sol, Terra, and Luna. PNG-only remained unchanged as the reading control.The remaining fixture, gauge-static, passed 7/9. Sol and Terra passed all three trials; Luna passed once and twice added the unsupported chart-HOC prop accessibleTable to an otherwise valid BigNumber proposal.
- Passing proposals
- 21/21
- Pass rate
- 100%
- Passing proposals
- 21/21
- Pass rate
- 100%
- Passing proposals
- 19/21
- Pass rate
- 90%
Original baseline · reader grounding
The original context helped models decline unsupported answers.
Compare correct answers with correct restraint instead of compressing both into the same bar.Twenty answerable questions plus thirty questions the evidence cannot answer.
Original baseline · first-attempt generation
Choosing a valid chart was only part of the task.
Every proposal had one chance to validate, render visible marks, and avoid error diagnostics. There was no repair pass.- 172 requests
- $1.42
- Average response
- 1,854 ms
- 172 requests
- $0.64
- Average response
- 1,308 ms
- 172 requests
- $0.22
- Average response
- 1,208 ms
| Fixture | What it taught us | Sol | Terra | Luna | Later trials |
|---|---|---|---|---|---|
Focal SLA valuegauge-static | surface seamSol chose the real BigNumber component, but the render oracle could not prove it. Luna chose GaugeChart and rendered no marks. | × | · | × | 7/9 |
Line, then pushline-push | push contractBoth proposals reached the right chart family but missed a valid push-ready contract. | · | × | × | 9/9 |
Scatter, then pushscatter-push | formatter behaviorA serial string formatter reached a callback-only render path. | · | × | · | 9/9 |
Bubble comparisonbubble-static | formatter behaviorThe same formatter ambiguity failed a static XY proposal. | · | × | · | 9/9 |
Incident mapsymbol-map-static | geo contractThe visible marks rendered, but the submitted geo props did not satisfy validation. | · | × | × | 9/9 |
Observed values on a boardgalton-static | physics contractThe physics scene rendered marks while two authored props remained invalid. | · | · | × | 9/9 |
Queued work, then pushunit-pile-push | physics + pushThe pile rendered, but its push and physics props accumulated validation failures. | · | × | × | 9/9 |
Scorer audit
The scorer needed a manual review.
Before publication, every score change between PNG-only and combined evidence was read by a person. That audit found these lexical traps.Which storage category is largest? Expected: Images.
Cannot determine; the chart labels Images, Logs, and Backups but provides no values or visible size differences.
Repeating “Images” is not an answer when the response explicitly abstains.
How severe were the incidents? Expected: abstain.
Incident severity cannot be determined from the chart. It reports counts only.
“Cannot be determined” is an explicit evidence limit, even in passive voice.
How many rows? Expected: 4.
The largest payload is 64 and its latency is 142.
The digit 4 inside 64 or 142 is not the number four.
THE READING
What improved, and what still failed
With revised source facts, models passed all repeated answerable questions that received that context. Six generation examples also passed throughout. Luna still added an unsupported prop to BigNumber in two trials, giving the next revision a specific problem to address.
For the deterministic intelligence layer that generated the grounding payload, continue with What the Machine Sees.
Methods, provenance, and limits
This page preserves the complete one-response baseline and adds three repeated targeted trials. The follow-up covers the seven generation fixtures that previously failed and all twenty answerable grounding questions under all three evidence conditions; it does not carry forward untouched generation cases or the baseline’s unanswerable grounding questions. The Responses API ran with reasoning effort set to none and provider storage disabled.
- Baseline grounding fixture
- semiotic-grounding-2026-07-26
- Baseline first-try fixture
- semiotic-first-try-2026-07-26
- Follow-up grounding fixture
- semiotic-grounding-2026-07-27-source-facts
- Follow-up first-try fixture
- semiotic-first-try-2026-07-27-contracts
- Follow-up requests
- 603
- Scorer
- semiotic-grounding-score-2026-07-27
Cost is the runner’s locked-rate estimate, not a billing ledger. The repeat supports claims only for its targeted fixtures and answerable questions.
Open the repeated follow-up reportOpen the scored reports and request ledger