Modern LLMs are interfaces, not just text generators. When an assistant answers a question about data, the answer can — and increasingly should — include visual artifacts: highlights on the chart the user is looking at, regions of interest, threshold lines, sub-selections, even a different chart entirely. We've been optimizing chat interfaces for text output for two years. Charts give us a parallel output channel that's underused.
Text is half the answer
The dominant LLM response pattern is a wall of prose. Even when the question is"where's the peak in this chart?" the answer comes back as a paragraph: "The peak appears to be around month 12 at approximately $2,240, which represents a notable increase from..." — and the reader's eye has to leave the chart, parse the paragraph, find the relevant month, look back at the chart, and locate the point.
Every step of that loop is friction. The peak is in the chart. The model has access to the chart's data. It can answer "where's the peak?" by drawing a circle around the peak, with the prose as supporting detail.
Demo: ask, see the chart respond
This is a canned version of the round trip. Each question button below pretends to ask a small local LLM; the model's response is a { text, annotations }object. The text goes into the transcript; the annotations land on the chart. You can stack questions to see how multiple annotations compose.
Click any question above. The text answer appears here; the visual answer appears on the chart simultaneously. Stack multiple questions to see the annotations compose.
Click Were there any unusual months? first — that's the canonical version of the example. The text names May and October as outliers; the chart simultaneously gets callouts on those two points. Reading the text confirms what the chart already showed. Reading the chart confirms what the text says. The two channels reinforce instead of duplicating.
Why this works, and why it doesn't break the chat metaphor
The chat surface stays familiar — there's a question and an answer in a transcript. What's new is that the answer has two faces:
- Text in the transcript, for the parts that need words: nuance, comparison, context, the "why" behind a value.
- Annotations on the chart, for the parts that need pixels:where the peak is, which months are below average, which twoobservations the question is about.
The split is not arbitrary. Some claims compress better as text ("revenue doubled"); others compress better as space ("here's the threshold and here are the six months below it"). When the model gets to choose, the answer fits the question's natural shape.
The contract
Concretely, this is what an LLM-backed answer looks like with a chart library that can render annotations:
async function onQuery(question, context) { const response = await callYourLLM({ question, chartSummary: context.summary, // min/max/mean/median per field chartData: context.data, // raw rows intent: inferIntent(question)?.intent, }) return { answer: response.text, annotations: response.highlights, // [{type: "callout", month: 5, revenue: 2200, label: "..."}, ...] } }The LLM is asked for two things and returns two things. The text is rendered in the chat transcript like any other LLM response; the annotations are passed through to the chart's annotations prop. No extra plumbing — both already exist as first-class chart concepts (callouts, thresholds, bands, trend lines, region highlights).
A small annotation vocabulary the model can use
The chart library defines the vocabulary; the LLM picks from it. A useful starting set:
callout — point a label at a specific observation. Use for"this is the peak", "this is the outlier".y-threshold / x-threshold — a horizontal or vertical reference line. Use for "the average is here", "before this date".band — a shaded region between two values. Use for "below target","within tolerance".trend / envelope — a statistical overlay. Use for"if we remove these outliers, the trend is...".enclose / rect-enclose — wrap a set of observations in a hull or rectangle. Use for "these three points form a cluster".
Each is JSON-serializable. The LLM doesn't draw pixels — it emits structured intent and the chart library handles the geometry. That's the right division of labor: language models are good at saying which observations matter and why; chart runtimes are good at converting that into pixels.
Beyond annotations — the broader pattern
Annotations are the entry point. Once you accept that LLM responses can have a visual face, the pattern extends:
- Selection responses. "Show me only the Q3 data" — the model returns a filter the chart applies. Same brushing surface used by humans.
- Chart-type swaps. "This isn't the right chart for that question" — the model returns a new
{ component, props } spec the runtime mounts in place of the current chart. The Semiotic capability layer can power this: the model consults suggestCharts and picks the best alternative. - Linked follow-ups. "What about by region?" — the model returns a companion chart that gets rendered alongside the current one, with its hover state linked to the first.
- Audience-calibrated responses. The same question to the same data could return a BoxPlot for a data-science audience and a BarChart for an executive — the model reads the audience profile and adjusts. (SeeChart Suggestions for how that calibration works.)
All of these are extensions of the same idea: the chart library is an output channel. LLMs that ignore it are leaving the most expressive part of the surface dark.
What to watch out for
Multimodal output isn't free of failure modes. Three to watch:
- Hallucinated annotations. A model that's wrong about the peak's location is now visibly wrong, with a callout pointing at the wrong dot. The fix is upstream: give the model the data summary and statistical context, not just the chart props, so its claims are grounded.
- Annotation clutter. A model that surfaces an annotation for every question accumulates noise. Either give the model a reset signal or accept that the chart needs a "clear annotations" affordance in the chat UI.
- Mode confusion. Users will eventually ask follow-up questions about the annotations themselves ("why did you highlight October?"). The chat history needs to include the annotations alongside the text so the next turn has full context.
- Interrogation — the
useChartInterrogation hook that ships this pattern as a first-class surface. The annotation-return contract is exactly what powers the demo above. - Annotations — the chart-library side of the vocabulary: every annotation type the LLM can emit.
- Chart Suggestions — what powers the chart-type-swap response mode mentioned above.