What the Machine Sees — Semiotic

Home · Getting Started · Charts · Features · Playground

This page requires JavaScript for interactive chart demos.

AI / Machine-readable docs: llms.txt · llms-full.txt · CLAUDE.md · schema.json · API Reference · Examples

View on GitHub · npx semiotic-ai for CLI access

Machine-readable page content

Canonical: https://semiotic.nteract.io/examples/what-the-machine-sees

What the Machine Sees

A moment ago, in The Scroll You’re Telling, a chart watched a person read. Turn it around. Hand Semiotic a real dataset and watch the machine read it — choose the form, justify the choice, narrate what it shows, audit whether you could even perceive it, and lay out a path through it for a screen reader. No model is called.Every panel below is a deterministic function of the data.

Ask the data a question

What it sees

South Korea's life expectancy at birth, every year from 1990 to 2023.

34Rows

2Fields

noTime axis

xyearylifeExpectancy

What it weighed

Built-in charts and registered recipes, scored against this shape and thetrend intent. Click any candidate to overrule the engine.

What it chose

engine's pick · LineChart · Smooth trend

Intent Mark · trend

time-series: trend. Strong fit for trend (5/5)

{ "ididVersion": "0.1", "chartId": "machine-sees-chart", "title": "How long do people live, over time?", "intent": { "primary": "trend", "communicativeAct": "This is a trend chart; read it for the trajectory of lifeExpectancy, which rises from 71.6 (1,990) to 83.4 (2,023)." }, "reception": { "channels": [ "visual", "interactive", "screen-reader", "agent" ], "strengths": [ "familiar", "deterministic", "agent-readable" ], "scaffolds": [ "description", "accessible table", "navigation tree" ] }, "designContract": { "chartFamily": "time-series", "whyThisForm": "Strong fit for trend (5/5)", "risks": [ "smoothing hides individual outliers" ] }, "accessibility": { "description": "A line chart of lifeExpectancy by year. lifeExpectancy ranges from 71.6 (1,990) to 83.5 (2,021), with a mean of 78.39 across 34 points. Overall lifeExpectancy rises from 71.6 (1,990) to 83.4 (2,023), after peaking at 83.5 (2,021). This is a trend chart; read it for the trajectory of lifeExpectancy, which rises from 71.6 (1,990) to 83.4 (2,023).", "navigation": true, "dataFallback": true, "manualChecks": [ "screen-reader behavior", "keyboard order quality" ] } }

South Korea's life expectancy at birth, every year from 1990 to 2023.

What it says

The Lundgard four-level description, generated from the config alone. L4 — the chart’s communicative act — comes from the same intent scores the ranking used.

L1EncodingA line chart of lifeExpectancy by year.

L2StatisticslifeExpectancy ranges from 71.6 (1,990) to 83.5 (2,021), with a mean of 78.39 across 34 points.

L3TrendOverall lifeExpectancy rises from 71.6 (1,990) to 83.4 (2,023), after peaking at 83.5 (2,021).

L4IntentThis is a trend chart; read it for the trajectory of lifeExpectancy, which rises from 71.6 (1,990) to 83.4 (2,023).

What it checked

The same chart, graded against Chartability. The description it just wrote — attached as the chart’s title and summary — is what clears the blocking failures. The warnings that remain are judgment calls a human still owns.

✓ No blocking failures11/14 critical heuristics pass

18pass2warn6verify0fail

understandable.axis-labels — Missing axis label: xLabel, yLabel. Ticks alone may not name the variable.
compromising.table-static — The data table is read-only — not downloadable, sortable, or filterable.
perceivable.low-contrast critical — Mark colors come from the theme/CSS variables — contrast can't be verified statically.
perceivable.seizure-risk critical — No flashing detected statically. Confirm no element flashes more than 3×/sec.
operable.interaction-cues critical — Chart is interactive and has descriptive text. Confirm that text explains how to interact (keyboard + pointer).

How it reads aloud

The structure a screen reader walks — chart → axes/series → values. Move through it with the arrow keys; the matching mark highlights on the chart, and hovering the chart moves this cursor.

▾ A line chart of lifeExpectancy by year. lifeExpectancy ranges from 71.6 (1,990) to 83.5 (2,021), with a mean of 78.39 across 34 points. Overall lifeExpectancy rises from 71.6 (1,990) to 83.4 (2,023), after peaking at 83.5 (2,021).
- X axis: year, 1,990 to 2,023.
- Value axis: lifeExpectancy, 71.6 to 83.5.
- 1,990: 71.6
- 1,991: 72
- 1,992: 72.5
- 1,993: 73
- 1,994: 73.4
- 1,995: 73.7
- 1,996: 74.2
- 1,997: 74.6
- 1,998: 75
- 1,999: 75.4
- 2,000: 75.9
- 2,001: 76.4
- 2,002: 76.8
- 2,003: 77.2
- 2,004: 77.7
- 2,005: 78.2
- 2,006: 78.7
- 2,007: 79.1
- 2,008: 79.5
- 2,009: 80
- 2,010: 80.1
- 2,011: 80.6
- 2,012: 80.8
- 2,013: 81.3
- 2,014: 81.7
- 2,015: 82
- 2,016: 82.3
- 2,017: 82.6
- 2,018: 82.6
- 2,019: 83.2
- 2,020: 83.4
- 2,021: 83.5
- 2,022: 82.7
- 2,023: 83.4

Custom recipe contract · v0

This is not just a custom chart. It has an inspectable intent contract.

The frame knows how to render rectangles. The recipe manifest states that they mean repeated units, that the analytical intent is part-to-whole, and that readers should navigate category summaries rather than one hundred undifferentiated cells.

Waffle chart

clean proof

High income12 cellsUpper middle income43 cellsLower middle income42 cellsLow income3 cells

Why leave the catalog?

A unit grid makes composition more concrete and memorable than an abstract arc or line.

A pie chart would show share but provide less unit-level evidence and less interaction surface.

Primary intent: part-to-whole
Semantic unit: category, not cell
Reception strength: memorable + explainable

Recipe B · high flavorRota Urinarum

A bar chart would erase the historical spectrum, the color-to-diagnosis relationships, and the situated act of reading the wheel.

semiotic.recipe.urine-wheel.v0

Intent Mark · part-to-whole

XYCustomChart: part-to-whole for general-technical. A unit grid makes composition concrete through repeated, inspectable units.

{ "ididVersion": "0.1", "chartId": "machine-sees-waffle", "title": "Waffle chart", "intent": { "primary": "part-to-whole", "secondary": [ "compare-categories", "explanation" ], "communicativeAct": "This is an apportioning and explanatory chart: repeated units make the composition tangible while category summaries preserve the meaningful reading unit." }, "audience": { "primary": "general-technical", "familiarityAssumptions": { "waffleChart": "medium", "partToWhole": "high" }, "literacyTargets": [ { "feature": "unit-based composition", "rationale": "Readers can reason about proportions through repeated visible units." } ] }, "reception": { "channels": [ "visual", "interactive", "screen-reader", "agent" ], "strengths": [ "memorable", "glanceable", "explainable", "teaching-friendly" ], "risks": [ "precise comparison may be harder than a bar chart", "individual cells can become navigation noise" ], "scaffolds": [ "legend", "category summary", "accessible table", "description" ], "memorableForm": true }, "designContract": { "chartFamily": "XYCustomChart", "whyThisForm": "A unit grid makes composition concrete through repeated, inspectable units.", "whyNotDefault": "A pie chart would show share but provide less unit-level evidence and less interaction surface.", "risks": [ "precise comparison may be harder than a bar chart", "individual cells can become navigation noise" ], "misuse": [ "too many categories", "too many units", "false precision", "treating each cell as a separate semantic datum" ] }, "accessibility": { "description": "A 100-cell waffle chart showing population composition by income. 4 categories total 4.9B; each cell represents approximately 1% of the whole. Upper middle income is the largest category at 43%, represented by 43 of 100 cells. This is an apportioning and explanatory chart: repeated units make the composition tangible while category summaries preserve the meaningful reading unit.", "navigation": true, "dataFallback": true, "manualChecks": [ "screen-reader behavior", "keyboard order quality", "metaphor comprehension" ] }, "provenance": { "dataSources": [ "World Bank, World Development Indicators" ], "reviewStatus": "docs demo", "generatedBy": "Semiotic recipe intelligence" } }

Recipe manifest

Meaning that cannot be recovered from rectangles alone.

{ "id": "semiotic.recipe.waffle.v0", "name": "Waffle chart", "frameFamily": "XYCustomChart", "portability": "portable", "dataRoles": [ { "role": "category", "field": "income", "accessor": "categoryAccessor", "required": true, "semanticType": "nominal", "description": "The group represented by a set of unit cells.", "source": "data" }, { "role": "value", "field": "population", "accessor": "valueAccessor", "required": true, "semanticType": "quantitative", "description": "The magnitude represented by the number of cells.", "source": "data" } ], "encodings": [ { "channel": "position", "role": "category", "meaning": "Contiguous cells are grouped spatially by category.", "redundantWith": [ "color", "category summary" ] }, { "channel": "count", "role": "value", "meaning": "Each visible unit represents one normalized portion of the total.", "redundantWith": [ "category share", "accessible table" ] }, { "channel": "color", "role": "category", "meaning": "Color distinguishes categories but is not the sole accessible cue.", "redundantWith": [ "position", "legend", "category label" ] } ], "intents": [ { "id": "part-to-whole", "strength": "primary", "score": 5, "rationale": "Repeated units directly expose how categories compose a total." }, { "id": "compare-categories", "strength": "secondary", "score": 3, "rationale": "Unit counts support broad comparisons, though bars remain more precise." }, { "id": "explanation", "strength": "secondary", "score": 5, "rationale": "Visible units make normalization and share easy to teach." } ], "reception": { "channels": [ "visual", "interactive", "screen-reader", "agent" ], "strengths": [ "memorable", "glanceable", "explainable", "teaching-friendly" ], "risks": [ "precise comparison may be harder than a bar chart", "individual cells can become navigation noise" ], "scaffolds": [ "legend", "category summary", "accessible table", "description" ], "memorableForm": true }, "designContract": { "whyCustom": "A unit grid makes composition more concrete and memorable than an abstract arc or line.", "whyThisForm": "A unit grid makes composition concrete through repeated, inspectable units.", "whyNotDefault": "A pie chart would show share but provide less unit-level evidence and less interaction surface.", "defaultAlternative": "BarChart", "tradeoff": "A bar chart supports more precise comparison; the waffle favors composition and memorability.", "misuse": [ "too many categories", "too many units", "false precision", "treating each cell as a separate semantic datum" ] }, "accessibility": { "keyboardNavigation": "required", "accessibleTable": "required", "description": "required", "navigationGranularity": "category", "dataBearingSceneNodes": "required", "fallbackTable": true, "requiresTitle": true, "requiresSummary": true, "requiresAccessibleTable": true, "minimumHitTarget": 24, "tableRoles": [ "category", "value" ], "tableFields": [ { "role": "category", "label": "Category" }, { "role": "value", "label": "Value", "format": "number" }, { "field": "share", "label": "Share", "format": "percent" } ], "redundantEncodings": [ "category position", "category summary", "legend" ], "requirements": [ "Announce category value, share, and allocated unit count.", "Aggregate navigation by category so repeated cells do not become meaningless noise." ] } }

Generated L1–L4 description

L1EncodingA 100-cell waffle chart showing population composition by income.

L2Statistics4 categories total 4.9B; each cell represents approximately 1% of the whole.

L3TrendUpper middle income is the largest category at 43%, represented by 43 of 100 cells.

L4IntentThis is an apportioning and explanatory chart: repeated units make the composition tangible while category summaries preserve the meaningful reading unit.

Navigation tree

Four category nodes summarize the 100-cell visual surface.

▾ A 100-cell waffle chart showing population composition by income. 4 categories total 4.9B; each cell represents approximately 1% of the whole. Upper middle income is the largest category at 43%, represented by 43 of 100 cells. This is an apportioning and explanatory chart: repeated units make the composition tangible while category summaries preserve the meaningful reading unit.
- High income: 601.8M, 12.3%, 12 of 100 cells.
- Upper middle income: 2.1B, 43%, 43 of 100 cells.
- Lower middle income: 2.1B, 42.1%, 42 of 100 cells.
- Low income: 128.7M, 2.6%, 3 of 100 cells.

Observed-scene audit

The actual Waffle layout is run at this viewport and checked against its manifest.

16 observed checks pass100 scene nodes inspected

Declared semantics

Roles: category, value · intents:part-to-whole, compare-categories, explanation · fallback:declared

Observed evidence

geometry.finiteAll emitted geometry is finite.
geometry.nonzero-areaAll bounded marks have positive area.
geometry.boundsAll bounded marks stay inside the plot.
interaction.target-size100 targets are smaller than 24px in at least one dimension.
identity.stable-idsEvery data-bearing mark exposes a stable id.
identity.unique-idsStable mark ids are unique.
coverage.input-dataEvery input datum is represented by the observed scene.
coverage.scene-dataEvery data-bearing scene mark maps back to input data.
interaction.hit-targetsEvery observed data-bearing scene node participates in hit testing.
coverage.annotation-anchorsNo pointId annotations require scene resolution.
accessibility.table-fieldsThe accessible table preserves all recipe-declared fields.
accessibility.color-onlyEvery declared color encoding names a redundant cue.
accessibility.titleA chart title is present.
accessibility.summaryA chart summary is present.
accessibility.descriptionA chart description is present.
accessibility.fallback-tableThe accessible fallback table is enabled.
accessibility.navigation-depthThe navigation tree exposes structure below the root.
visual.contrast43 marks have fill contrast below 3:1.

Manual AT and reception checks

manual-at.screen-readerVerify real screen-reader behavior with the target browser/AT combinations.
manual-at.cognitive-loadAssess cognitive load and whether the custom metaphor is understood.
manual-at.keyboard-orderVerify that keyboard order is meaningful, not merely mechanically available.
manual-at.animationVerify that animation is not distracting and reduced-motion behavior is sufficient.
manual-at.receptionTest whether the memorable form is received as intended by the target audience.
manual-at.overlay-occlusionInspect labels, overlays, annotations, and important marks for clipping or occlusion.
manual-at.observationExercise hover, focus, selection, and observation emission in the rendered chart.

Why determinism is the point

A chart that can explain itself is a chart you can trust an agent with

Generative tools are eager to draw charts and careless about whether the chart is right, readable, or honest. Semiotic moves that judgment into deterministic code that ships beside each component. The engine can be wrong — it sometimes ties, or makes a defensible pick you’d overrule — but it is never opaque. The reasons, the rubric, and the caveats are all inspectable, and identical inputs always yield identical output.

It knows its own limits

The caveats are the most important thing on the page

“Only 16 rows — may feel sparse.” “Bubble area is harder to compare than length.” “Smoothing hides individual outliers.” A system that volunteers what’s wrong with its own recommendation is one an LLM or an analyst can build on without re-deriving the rules of good visualization — or hallucinating them. This is the trust layer underneath generated dataviz, not a replacement for the human on top.

Core implementation

Profile, suggest, describe, audit, navigate

Five pure functions from semiotic/ai, composed. The engine reads capability descriptors that ship next to every chart; no network, no API key, no model.

JSX

import { useChartSuggestions, describeChart, auditAccessibility, buildNavigationTree, AccessibleNavTree, } from "semiotic/ai" // 1 · The engine profiles the data and ranks every chart's capability. const { suggestions, profile } = useChartSuggestions(data, { intent }) const top = suggestions[0] const Chart = COMPONENT_MAP[top.component] // top.props spreads straight in // 2 · It describes what the chart shows (Lundgard L1–L4). const description = describeChart(top.component, top.props, { capability: { family: top.family, intentScores: top.intentScores }, }) // 3 · Its own description, attached, is what clears the accessibility audit. const enriched = { ...top.props, title, description: description.text } const audit = auditAccessibility(top.component, enriched, { describe: true }) // 4 · And it lays out a path a screen reader can walk. const tree = buildNavigationTree(top.component, top.props) <Chart {...enriched} /> <AccessibleNavTree tree={tree} /> // Not one model call. Every line is a deterministic function of the data.

Real data: World Bank, World Development Indicators, retrieved 2026-06-30 — 16 countries spanning every World Bank region and income group, 544 country-year observations, 1990–2023. The chart recommendations, descriptions, accessibility audit, and navigation tree are computed entirely in your browser with no language model.