Here is the uncomfortable property of almost every front-end streaming chart: it getsslower as the stream gets faster. The standard pattern is to buffer raw records and redraw them, so the work per frame is tied to how many records arrived. Point the chart at a 50,000-messages-per-second Kafka topic and it falls over. That's not because drawing 50,000 points is useful (you can't see them), but because the renderer never got the memo that volume and resolution are different things.
Semiotic now ships a streaming-aggregation layer that fixes the axis. Instead of buffering raw points, it aggregates on ingest into bounded, event-time windows, so per-frame cost follows the resolution of the view—the number of windows you can actually see—not the arrival rate. It is, almost exactly, the Kafka Streams windowed-aggregation model expressed client-side. The whole thing is opt-in props on the realtime charts you already have.
Why this works
The insight isn't mine. It's the transferable core ofAGAMI: Scalable Visual Analytics over Multidimensional Data Streams (Lu, Wong, …, Joshi, Malensek — University of San Francisco, IEEE/ACM BDCAT 2020), a distributed-backend system whose central data-structure idea drops cleanly onto a client renderer:
Don't buffer raw records and evict the oldest. Aggregate on ingest into a compact, bounded summary built from online running statistics. Memory and per-frame work then scale with the resolution of the view, not the volume of the stream.
That allows us to treat bounded memory,downsampling / level-of-detail, backpressure, andwindowing semantics with a single mechanism. If your chart only ever holds a bounded number of window aggregates, memory is bounded by construction, the marks are an aggregate rather than dropped points, and a firehose renders the same as a trickle because the render work doesn't depend on the input rate. It's true in data analysis but especially in data visualization: we do not need to pretend that every datapoint needs to be seen to create effective visualization.
The firehose
The chart below pushes 15 events every 80 ms — a deliberate firehose. It never draws more than a couple of dozen marks. Each window carries a full online statistics object (count, mean, variance, min, max via Welford's method), so you can flip the rendered statistic and the band without touching the data path:
statband
0 events ingested → 0 window marks drawn
Watch the readout: tens of thousands of events, a few dozen marks. Theretain option caps the live window count, so the chart's memory is flat no matter how long it runs. The ±σ band isn't a second pass over a buffer, it just falls straight out of the per-window variance the accumulator already maintains.
<RealtimeLineChart ref={chartRef} timeAccessor="t" valueAccessor="v" aggregate={{ window: "tumbling", // or "hopping" (overlapping) | "session" (gap-bounded) size: "2s", // ms or a duration string ("10s", "1m30s") stat: "mean", // mean | sum | min | max | count band: "stddev", // "stddev" | "minmax" | "none" → envelope retain: 40, // bounded memory: keep the 40 newest windows }} />One naming caveat: this is referred to in Semiotic as theaggregation window. It is the interval events are reduced over. It isnot the same as windowMode ("sliding" | "growing"), which is the ring buffer's eviction policy. Two different concepts that the streaming literature unhelpfully gives the same word; Semiotic keeps them distinct.
Event-time, not arrival-time
The other half of the Kafka model is that windows key on the timestampin the record, not when it showed up. Real streams arrive jittery and merged from multiple sources, and a chart that appends in arrival order draws a zigzag that means nothing. Turn on eventTime and Semiotic buffers events for a boundedlateness / grace window and releases them in event-time order. A record older than watermark − lateness is late: dropped or kept per policy, and counted and surfaced through onObservation so lateness is a signal and not a silent bug. Toggle it off below to bring the zigzag back:
event-time
reordered within a 2s grace window · 0 late records dropped & counted
<RealtimeLineChart eventTime={{ lateness: "2s", latePolicy: "drop" }} // "drop" | "keep" onObservation={(o) => { if (o.type === "late-data") console.warn("late", o.eventTime, "watermark", o.watermark, "total", o.lateCount) }} />The tradeoff is explicit: a lateness window delays display by that much in exchange for correct ordering. This is a fact of all watermark strategies. That's the same deal Kafka Streams offers with its grace period, because you can't both show data instantly and guarantee it's in order.
What sets it apart from the usual front-end streaming kit
Most charting libraries treat "realtime" as "call setData faster." That gives you three failure modes the aggregation model sidesteps:
- Cost tied to volume. Re-rendering the raw buffer means a faster stream is a slower chart. Aggregation makes render cost O(visible windows) independent of input rate.
- Unbounded memory. "Keep the last N points" is a band-aid that still scales N with rate to preserve a time span. A retained window set is bounded in the dimension you actually care about: time on screen.
- Arrival-order lies. Plotting points as they land conflates network jitter with signal. Event-time windowing and a grace buffer separate the two, and report the lateness instead of hiding it.
The chart props are a helpful abstraction over pure modules, so you can drop to the primitives when you need them. RunningStats (Welford online stats with an O(1) parallel merge), WindowAccumulator (the windowing engine), andReorderBuffer (the lateness buffer) all import fromsemiotic/realtime and run headless:
import { WindowAccumulator } from "semiotic/realtime" const acc = new WindowAccumulator({ window: "tumbling", size: 60_000 }) acc.push(eventTimeMs, value) // O(1) amortized, any arrival order acc.emit() // [{ start, end, count, mean, stddev, min, max, partial }]When to use it
Aggregate when the stream is faster than the eye: metrics firehoses, high-frequency sensors, request traces, anything where the signal is the distribution over an interval, not the individual event. Use event-time mode whenever records can arrive out of order as in merged sources, mobile clients, replays.
Don't use it when every individual event matters and the rate is low. A handful of discrete events a second belong on a plainRealtimeLineChart orRealtimeSwarmChart with the raw push API, where you keep per-point identity, hover, and update(id, …). Aggregation is a reduction; if you need the rows, don't reduce them.
Where this shows up
The example here is a synthetic metric, but the shape is everywhere a bounded screen meets an unbounded stream:
- Observability / APM — p50/p95 latency over rolling windows, request rate per service, error counts by minute.
- Kafka / event pipelines — throughput and lag per partition, exactly the windowed-aggregation model these charts mirror.
- IoT sensor fleets — thousands of devices reporting on their own clocks, arriving late and out of order over flaky links.
- Financial tick data — OHLC bars are tumbling windows; a ±σ band is a volatility envelope.
- Log analytics — events-per-interval by level, where the interesting thing is the rate, not the line.
Related