The first person to treat a drawing as a program was Ivan Sutherland. Sketchpad (1963) let you draw geometric shapes with a light pen on the TX-2 and then apply constraints — fix this length, make these parallel, instance this shape — so the drawing was the specification, maintained by a constraint solver. Sutherland did not have a network, or a collaborator, or an AI. But the foundational insight is his: if the system keeps the structure of what you drew, rather than a picture of what you drew, then the drawing can participate in computation.
Direct manipulation as computation
Bret Victor's Drawing Dynamic Visualizations (2013) pushed the same idea in a different direction. Victor built a tool where you draw a visualization by direct manipulation — placing marks, snapping to data — and the system infers the program behind the picture so it can re-render for different datasets. The key design move is that every drawing action is unambiguous: you do not draw freely and hope the system guesses what you meant; you draw from snap points that force you to express intent precisely. The result is computation expressed through geometric construction rather than algebraic expressions. Victor solved for determinism; Codrawer solves for expressiveness and ambiguity, because the AI is supposed to meet the human partway.
Pen-based recognition and invocation
In pen-based math, MathPaper lets you handwrite an expression, draw a ⇒ symbol, and the system evaluates it — the gesture is the invocation. SketchREAD (Alvarado & Davis, UIST 2004) builds Bayesian networks on the fly to recognize freely drawn diagrams — circuits, family trees — without training data, using context to resolve ambiguity. These systems proved that sketch recognition can be domain-general and that freehand input is rich enough to drive computation. They are single-user and offline, but the recognition problem they solved is the same one Codrawer will need to solve over a network with multiple participants.
The contemporary landscape
Recent work makes the landscape sharper. Code Shaping (Yen et al., CHI 2025) lets programmers draw free-form sketch annotations directly onto code and console output; the AI interprets the sketch as an edit intent and modifies the source. The finding that matters: programmers preferred spontaneous, unconstrained sketching over predefined brush types — they want to invent their own notation on the fly, and they expect the system to keep up. Visual Sketchpad (Hu et al., NeurIPS 2024) gives multimodal LMs the ability to draw intermediate sketches as part of chain-of-thought reasoning, improving math and spatial tasks by 8–13%. That work shows that sketching is useful for the model, not just the human. tldraw's Make Real goes by screenshotting a whiteboard sketch and asking GPT-4V to generate working UI code — impressive, but it the sketch and discards everything structural about it. The pipeline is image-in, code-out; there is no persistent geometric representation the model and the human share. Sketch-n-Sketch bridges direct manipulation and code for SVG, and AI Drawing Partner (2025) models co-creation through a cognitive-science framework that tracks interaction dynamics between human and AI during shared drawing sessions.
Where Codrawer sits
Codrawer sits at an intersection none of these systems fully occupy. It is real-time and networked like Drawpile (the networking angle is on the multiplayer page), stroke-native like Sketchpad and MathPaper, AI-collaborative like Code Shaping and AI Drawing Partner, and aimed at general-purpose computation like Victor's tool — but it keeps all of those properties simultaneously. The reason for building new infrastructure rather than extending an existing canvas is that the combination imposes constraints no single prior system was designed for: the protocol must preserve stroke structure end-to-end (ruling out raster pipelines like Make Real), route to multiple participants including AI agents (ruling out single-user recognition systems), and separate attribution by layer at the protocol level (ruling out conventional shared-canvas models where all ink is equal).
Next deep dive
Infrastructure — The codrawer-bridge codebase: stroke-native protocol, FastAPI session router, AI worker loop, model integration, and heuristic fallback.
Or explore: Multiplayer, Vision.