You draw a circle on a tablet. Not a perfect circle — your circle, with your at the top and the slight overshoot where you closed it. The system does not flatten that into a bitmap and ask a vision model what it sees. It kept every point: the position, the pressure, the timing. It knows the stroke was closed because the endpoint returned to the origin within a threshold. It knows you paused. And because it has that structure, it can do something a pixel-diff system cannot: it can respond in the same language. The AI puts a second stroke on a separate layer — a cleaned-up ellipse, or a small annotation, or the start of something you did not ask for but that fits — and that stroke arrives as geometry you can inspect, edit, or build on. Your ink and the AI's ink coexist but never merge. That separation is not a feature; it is the core design constraint.
This is Codrawer. The broader ambition is what I call a Sketch Computing Interface: an environment where pen strokes are not just drawings but a general-purpose input language for invoking computation. Draw a character, and the system can rig it, it, . Sketch a world and it can populate the empty corners. Draw a graph and it runs graph algorithms on it; draw a circuit and a simulator lights up. Annotate anything with a symbol your collaborator recognizes, and the system learns that symbol as part of your shared vocabulary. The sketch is the program — and the program might be a story, a creature, a machine, or a proof — and the AI is an extension of your symbolic thinking, not a separate tool you prompt with text.
Thoughtful computing
I have been drawing since before I could read — which means drawing is how I learned to think, years before I trusted words to do the same job. The back of every notebook, the margin of every handout, the blank side of every worksheet a teacher ever handed me: that was the real workspace. Characters, and the they got into trouble in. Machines I wanted to exist badly enough to draw them into being. The systems humming under games I was designing. 2D and 3D animations, storyboarded frame by frame. Interfaces, , and philosophical pictures I had no words for yet and drew anyway. Art and reasoning were never two different things for me — a character design is a theory of a person, a machine sketch is an argument about how the world could work, a game map is a system you can walk around inside. So every time I sat down to code, I was translating out of that , generous medium into a much narrower one. The sketch was the thought. The code was only ever the transcription — and you always lose a little in the translation. And transcription has a cost. Every time you leave the sketchworld to type a function name or click through a menu, you break the spatial, gestural thread you were following. Cognitive science has a name for it: switch cost. Your working memory flushes, the spatial context you were maintaining collapses, and you have to rebuild it on the other side. The more different the target modality is from the one you were thinking in, the higher the cost.
Most computational interfaces demand that switch. You think spatially, then express sequentially. You think in rough proportion and adjacency, then express in exact coordinates or symbolic syntax. You think in gestures — a circling motion means “this whole region,” a crossing-out means “delete,” an arrow means “this goes there” — and then you reach for a mouse to select, a keyboard to type, a dialog to confirm. Every one of those transitions is a place where the thought could have stayed richer and more fluid if the interface had met you in the medium you were already using.
I think of this as part of thoughtful computing: the idea that the interface between a human mind and a computational system should respect the pace, texture, and ambiguity of actual thought. Chat is too serial. GUIs are too committed. The pen occupies a different space — fast enough to keep up with ideation, expressive enough to carry spatial relationships and visual shorthand, and imprecise enough that the system has to meet you partway. That requirement — the system must interpret, not just execute — is where collaboration starts. A Sketch Computing Interface treats ink as a first-class data type with temporal, spatial, and pressure dimensions, routes it through a protocol that preserves all of those dimensions, and makes them available to computation on the other side. The goal is that you never have to leave the sketchworld to invoke the computation. The computation comes to you, in your medium, at your pace.
Deep dives
- Lineage — Sutherland, Bret Victor, MathPaper, SketchREAD, Code Shaping, Visual Sketchpad, tldraw Make Real — where the idea comes from and what each system teaches.
- Infrastructure — The codrawer-bridge codebase: stroke-native protocol, FastAPI session router, AI worker loop, model integration, and heuristic fallback.
- Multiplayer — What Drawpile's command-replay, retcon, and COW undo teach about real-time collaborative drawing — and the new questions an AI participant raises.
- Vision — Gestural delegation, mark vocabularies, sketch-to-CAD, and staying in the sketchworld — the forward-looking argument for sketch computing.
Stack
- Device bridge: Python,
evdev, WebSocket client — reads tablet digitizer, normalizes, emits protocol messages - Desktop server: Python, FastAPI, WebSocket — session routing, stroke buffering, AI job queue, debounce and rate limiting
- Model gateway: Node.js (TypeScript), Vercel AI SDK — OpenAI-compatible
/v1/chat/completionswith Cerebras or other providers; structured tool-calling output - Protocol: JSON over WebSocket; stroke-native messages (
stroke_begin/pts/end,ai_stroke_*,cursor,prompt); normalized coordinates; layer separation - Dev viewer: Built-in HTML/Canvas client at
/viewer/<session_id>— user strokes in green, AI in red, virtual hand tip with heading interpolation - Record/replay: JSONL harness for simulating sessions without hardware
Status and roadmap
This is early-stage work. The infrastructure exists; the vision above is where it is headed. Below: what is done, what is next, and what is further out.
Done
- Stroke-native WebSocket protocol —
stroke_begin/pts/endwith normalized coordinates, pressure, and timing - FastAPI session router — multi-client broadcast, cursor tracking, per-session stroke buffers
- AI worker loop — debounced, rate-limited, pause-aware; triggers on
stroke_endand explicitprompt - Model integration — OpenAI-compatible gateway with structured tool-calling (
emit_ai_strokes/should_respond); optional multimodal context patch - Deterministic heuristic fallback — echo, closure detection, flourish; full loop testable without a model
- reMarkable Paper Pro bridge —
evdevsidecar that reads the digitizer and emits protocol messages - Dev viewer and record/replay harness
Next
- Proper client. A real drawing client beyond the dev viewer — canvas with undo, layer visibility toggles, brush selection. Probably web-based (Canvas/WebGL) so it runs everywhere.
- Stroke recognition layer. Classify closed vs. open strokes, detect common primitives (lines, arcs, rectangles, arrows), and expose recognized structure to the AI context alongside raw points. The bridge already detects closure in the heuristic path; this generalizes that into a recognition service.
- Gestural mark vocabulary. A small set of initial delegation marks — circle-to-scope, cross-to-delete, arrow-to-connect, custom glyph-to-invoke — with a learning loop so the system adapts to the user's personal shorthand over sessions.
- Multi-user sessions. The protocol supports multiple clients today; what is missing is presence indicators (who is drawing where), per-user undo (Drawpile-style UndoPoint markers), and latency hiding (local fork + retcon on conflict).
- Persistent session state. Save and resume sessions — stroke history to disk or object store, so a sketch-in-progress survives a disconnect.
Further out
- Computation backends. Plug-in architecture for domain solvers — graph algorithms, circuit simulation, FEA, CAS — invoked by gestural marks on the canvas. Each backend receives recognized stroke structure and returns results as AI-layer geometry or overlay data.
- Sketch-to-CAD pipeline. Free2CAD-style stroke-to-parametric-operation parsing integrated into the session, so a drawn profile becomes an extrudable solid on the AI layer without leaving the canvas.
- Evolving symbol vocabulary. The system learns which gestural marks a user (or group) uses consistently and proposes bindings — “you draw this glyph before running simulations; want me to treat it as a simulate command?” Shared vocabularies across multiplayer sessions become a kind of emergent notation.
- Cross-device. iPad, Android tablets, Wacom — any digitizer that reports position and pressure. The protocol is device-agnostic; the bridge layer per device is the only thing that changes.
- Thoughtful computing integrations. Connect Codrawer sessions to other computational environments — notebooks, terminals, version control — so that a sketch can reference a running process and a running process can annotate a sketch. The canvas becomes a workspace, not an island.