The most common AI-on-a-chart pattern today is "ask the chart". Type a question, get a paragraph back. It works, but it's lossy in both directions: the user has to verbalize which point they care about, and the AI has to verbalize where the answer applies. Both steps lose the spatial information that's already on screen and treat all questions as vague in a way no professional analyst would find realistic. There's a better shape: let the userpoint at a data point, and let the AI annotate the answer back onto it. In other words: a two-way anchored conversation.
The loop in three frames
- User hovers or clicks a data point. The chart fires an observation event; we capture which datum the user is looking at and pass it into the chat as the "focus."
- User asks a question. The LLM receives both the question AND the focused datum. The prompt is "answer this question about this specific row" and not "about the chart in general."
- AI responds in two channels. Text in the transcript, anchored note back on the chart at the same point. Future hover over that point shows the AI's rationale; future questions in the same conversation can reference earlier comments.
Try it
Click any month's data point to focus on it. The dashed ring marks the focus; the chat shows what's currently selected. Ask a question and the AI's answer arrives as a text bubble AND a small turquoise dot on the chart. Click the dot to see the AI's anchored note. Stack questions to build up a multi-point conversation.
Anchored conversationThis is no focus, please click a point
Click a chart point to focus on it, then ask a question. The answer comes back both as text here AND as a clickable AI note anchored to that point.
The interesting moves: click May and ask "why is this so high?" and the AI cites the spring promotion. Click October and ask "why is this so low?"and it cites the outage. Both rationales then live on the chart as clickable notes that survive the rest of the conversation. The chart accumulates institutional knowledge about itself. These are canned responses, of course. In the wild you'd need to wire this up to an LLM but in this case there's more than enough context that you would not need a frontier model.
Why anchoring matters
Three things change once the conversation has a spatial anchor:
- Pronouns work. "Why is this one higher?" becomes a well-formed question instead of a guessing game. The LLM doesn't have to triangulate from prose what point you meant.
- Comparisons get cheap. Click two points in succession and ask "what changed between these?" The AI compares them directly because both are explicit in the context.
- Answers persist where they're useful. The AI's rationale lives next to the point it explains. When someone else looks at this chart next week, they hover over October's dip and the explanation is right there. No re-asking, no re-discovering which means no rework. None of us like rework. Charts become accumulating notebooks of why-the-data-looks-this-way and that's a layer that can itself be mined for insights later.
Building it
Semiotic ships the two primitives this needs: useChartInterrogation for the conversation, useChartFocus for the point-of-interest signal. Wiring them together is one component:
import { LineChart, ObservationProvider } from "semiotic" import { useChartFocus, useChartInterrogation } from "semiotic/ai" function AnchoredChart({ data }) { // useChartFocus subscribes to the chart's observation store and returns // the latest hover/click as { datum, x, y, source }. Returns null when // the user has moved away or hasn't engaged yet. const focus = useChartFocus({ chartId: "sales" }) const { ask, history, annotations } = useChartInterrogation({ data, focus, // ← context.focus inside onQuery onQuery: async (question, ctx) => { // ctx.focus.datum is the row the user is asking about const response = await yourLLMCall({ question, focus: ctx.focus, summary: ctx.summary, }) return { answer: response.text, // Return annotations with a `note` field your overlay renders // them as clickable AI-anchored comments on the chart. annotations: response.highlights, } }, }) return ( <ObservationProvider> <LineChart data={data} chartId="sales" xAccessor="month" yAccessor="revenue" annotations={annotations} onObservation={() => {}} // any handler enables the store /> <YourChatUI history={history} onAsk={ask} focus={focus} /> </ObservationProvider> ) }The useChartFocus hook is opinionated about what counts as focus: hover, click, and selection by default; hover-end and click-end clear it. For a sticky-focus UI where hover doesn't count, pass{ types: ["click", "click-end"] } and only clicks update the AI's reference point.
The interrogation hook already returns annotations to the chart's standardannotations prop. The new piece is what those annotations can carry which is not just a label, but a note. An annotation like:
{ type: "callout", month: 5, revenue: 2200, label: "Promo-driven spike", note: "Spring 2024 product launch + 15% sitewide discount. Not repeatable; treat as one-off in forecasts." }The chart renders the callout natively. A small overlay (~30 lines, copyable from this page) finds annotations with a note field and renders a clickable marker that reveals the note on demand. The rationale lives on the chart; the rationale doesn't crowd the chart unless someone asks for it.
This is exactly the patternAdvanced Annotations demonstrates with the human-authored comment threads with the same UI shape, but populated by an LLM instead of typed by a teammate. The chart doesn't care where the comments came from.
Where to use this
- Operations dashboards. An on-call engineer hovers over an anomaly spike, asks "what happened here?" and the AI consults a runbook + incident history + deploy log and leaves an anchored note. Next time someone sees the spike, the note is already there. Not only is the insight captured in more than just the mind and slack channel of the operators but it avoids every other person who sees this chart wasting 15 minutes answering the same question.
- Financial models. An analyst clicks a forecast point that looks surprising, asks "why does the model show this?" and the AI walks through which inputs drove this value most, leaves a note explaining the dominant terms.
- Scientific exploration. A researcher clicks an outlier observation, asks "is this an artifact?" and the AI references the run log, the calibration history, similar past observations, and leaves a note classifying it. Anomalies graduate to insights or, if false, can be automatically fed back into the model to improve its accuracy.
- Customer support / sales review. A rep hovers over a usage dip for a specific account, asks "what's going on with this customer?" and the AI consults the CRM history and leaves an anchored explanation that the next rep also sees.
The pattern across all four:the chart is the primary surface, the AI is a teammate annotating it. Not a chat window that happens to talk about charts; a chart that accumulates explanations.
Failure modes worth thinking about
- Stale notes. Yesterday's AI explanation may be wrong today. Treat annotated notes as ephemeral by default and easy to dismiss. A note that hasn't been refreshed in 30 days probably shouldn't be surfaced with full confidence.
- Anchoring drift. If the dataset gets re-aggregated (weekly to monthly), the annotation's coordinates may no longer match anything meaningful. Tie notes to a stable identity (datum.id, a deterministic hash of the row), not pixel coordinates. That way the chart re-positions them on data shape changes.
- Authority confusion. Human comments and AI comments need visual differentiation. The convention this post uses--turquoise for AI, default for human--is one option; an
author field on each annotation is the more rigorous one. The audience needs to know which voice they're reading.