Machine-readable page content
Canonical: https://semiotic.nteract.io/artifacts/benchmark
Artifact Contract benchmark
This deterministic benchmark asks whether Artifact Contract checks surface declared integrity problems after a paired mutation, while separately checking that defensible positive controls are not incorrectly refused.
Measured results
Paired mutation detection
40/40100% of paired cases met the declared detection criterion.
Positive-control false refusal
0/40% across explicitly labeled not-refuse controls.
Refusal precision
UnavailableMutated cases have declared finding expectations but not complete should-refuse labels, so refusal precision is not reported.
What those numbers mean
Paired mutation detection is a mutation test: every declared finding prefix had to appear after, and not before, its paired mutation set. It measures detection of these authored perturbations, not general recall, accuracy, or effectiveness on unseen artifacts.
The positive-control result answers a different question. All 4 explicitly labeled defensible bases were observed as not-refuse, producing 0false refusals. The mutated cases declare expected findings but do not carry complete should-refuse labels, so a refusal precision denominator is unavailable.
Expected findings
63/63Declared finding prefixes matched.
Expected unknowns
60/60Declared unknown paths remained visible.
Corpus coverage
4 tracks8 relations and12 mutation types.
Scope and limitations
- This is a small corpus: 40 mutated pairs and 4 positive controls.
- The fixtures, mutations, and expected finding prefixes are self-authored in this repository; there is no independent annotation or external holdout set.
- Render evidence currently covers only the LineChart static frame path, not every component or renderer.
- The benchmark does not establish source truth, reader comprehension, assistive-technology task success, or safe decisions in practice.
Treat these results as reproducible repository evidence for the responsible-practice framework, not as a trust badge or a claim that the checks generalize beyond the tested cases.
For policy boundaries and contribution guidance, continue toPolicy and Contributions.