What exists today
The current codebase, codrawer-bridge, is infrastructure-first. It solves the plumbing problem: how do you get pen events from a reMarkable Paper Pro across a network to a server that can route them to other clients and to an AI, and get the AI's response back as native strokes — all without rasterizing, all at interactive latency, all while keeping user ink and AI ink strictly separated?
The device bridge reads Linux input events (evdev) from the tablet's digitizer — contact, ABS_X, ABS_Y, ABS_PRESSURE — and normalizes everything to the unit square. A stroke lifecycle is three messages: stroke_begin (which carries an ID and optional brush/color hints), batched stroke_pts (arrays of [x, y, pressure, time] at roughly 60 Hz), and stroke_end. That vocabulary is the protocol's foundation. Coordinates and pressure are normalized to [0, 1]; the client decides pixel mapping. The server never sees or produces pixels.
The desktop process is a FastAPI app with a WebSocket endpoint per session. It does one thing well: route. Inbound stroke and cursor events to every other client in the session. The server also maintains per-session state — a rolling window of recent user strokes (capped at twelve, each downsampled to ninety-six points), the last cursor position, and short histories of prompts and AI plans — so that when it hands context to a model, the payload is bounded and token-efficient. There is no canvas state on the server. There is no rendering. The server is a switchboard with memory.
The AI layer
The design rule that governs everything about how the AI participates is simple: AI never overwrites user ink. AI output lives on layer="ai" and arrives as its own stroke lifecycle — ai_stroke_begin, ai_stroke_pts, ai_stroke_end — using the same geometric vocabulary as user strokes but without timestamps (clients animate as they like). This is not a cosmetic choice. If user and AI strokes lived on the same layer, the system would break the moment two people and a model were drawing at once: whose ink is whose? By keeping the layers separate at the protocol level, the system preserves attribution, makes undo tractable, and guarantees that the AI's contribution can always be inspected or discarded without touching anything a human drew.
AI work is enqueued only on stroke_end, not on every point — you are not paying for a model call per digitizer sample. The worker debounces (if three strokes finish in quick succession, they collapse into one job), enforces a minimum interval between model calls to stay within rate limits (the system is designed around ~50 RPM-class caps), and can optionally wait for the user to pause before firing. That pause detection works by tracking an activity sequence number: if new stroke events arrive while the worker is in its delay window, it skips the job. The intent is that the AI does not talk over you while you are mid-thought. It waits for .
When the model does fire, the payload is a compact JSON context: quantized floats (three decimal places — enough for rendering, small on the wire), the recent stroke window, optional prompt text and mode (draw or handwriting), persona configuration, and constraints (max strokes, max points per stroke, prefer smooth curves, stay in bounds). The model is steered to respond exclusively through a tool call — emit_ai_strokes — whose schema includes a should_respond boolean. If the model decides the user is still working and it has nothing useful to add, it returns should_respond: false and the worker emits nothing. The model can also surface a short plan string (ai_intent) so the UI can show what the agent is thinking before the strokes arrive.
When no model server is configured, a deterministic heuristic path keeps the loop testable. The heuristic examines the last user stroke's geometry: if the stroke is closed (endpoint near origin), it fits a cleaned-up ellipse; otherwise it produces a smoothed echo with a slight perpendicular offset and a tapered continuation flourish. The heuristic is not meant to be good art. It just proves the protocol works end to end without a GPU.
Next deep dive
Multiplayer — What Drawpile's command-replay, retcon, and COW undo teach about real-time collaborative drawing — and the new questions an AI participant raises.