Strokes as invocation
The bridge as it stands is the transport layer. The vision is that strokes become a general-purpose invocation mechanism — not just input to a drawing program, but commands in a computational environment where the pen never yields to the keyboard. You draw a shape, and the system recognizes it not as a bitmap but as a structured gesture with semantic weight: a symbol in a language you and the system develop together over time. A circle around a subregion means “scope.” A zigzag annotation means “reject.” A quick arrow from one element to another means “connect.” A carefully drawn graph with labeled nodes is not a picture of a graph; it is a graph, and the system can traverse it, optimize it, simulate it. You stay in the sketchworld the entire time. The computation comes to the canvas, not the other way around.
Gestural delegation
Gestural delegation is the mechanism I want to build on top of this. You draw a mark — a personal glyph, a shorthand you have used since your first notebook — and that mark means “run this,” or “hand this off to the AI,” or “simulate the thing I just outlined.” Think of it as the pen equivalent of a function call, except you define the calling convention by drawing it, and the system learns your vocabulary over time. MathPaper demonstrated a narrow version of this: draw ⇒ next to an expression and the CAS evaluates it. The general case is a mark vocabulary where any sufficiently consistent gesture can bind to any computation — graph layout, stress analysis, code generation, constraint solving — and where the AI can propose new marks when it notices you doing the same thing repeatedly. The marks are an evolving symbolic language between you and the system, not predefined menu items disguised as gestures — legible to both because both operate on the same stroke-native representation.
The evolving symbol language
A mark does not arrive with a meaning. It earns one. You draw a small flag in the corner of a region; the system guesses you mean “watch this” and acts on it; you either let the guess stand or scribble it out, and the next time the guess is a little sharper. Meaning accretes through repetition and correction — a handshake repeated until it no longer needs to be negotiated. That loop is closer to how a child fixes a word through grounded use than to how a settings panel binds a shortcut. The mark comes to mean whatever it has reliably done.
The vocabulary is layered. At the bottom is your idiolect: the personal glyphs you have used since your first notebook, the ones nobody taught you and you could not fully explain. Above it is the shared dialect you build with a collaborator, or with the AI itself — marks you both agree to read the same way. And a mark is allowed to be polysemous: a small star means “this is the hero” on a character sheet and “emphasize this” over a diagram, disambiguated by where you drew it, the way natural language leans on context to keep a small inventory of words doing a large amount of work. A symbol language that could not do that would collapse into a menu with extra steps.
The hard parts are social and temporal. Languages drift, and yours will: a mark you used for “animate this” slowly grows into “animate this and loop it,” and the system has to let that meaning move without silently breaking what the mark used to do — which is why marks need lineage, a history you can inspect and roll back. Stranger and more interesting is the collision: when a symbol you invented gets used by someone whose private vocabulary gave it a different job, the system cannot simply pick a winner. It has to surface the conflict and let the two of you settle it, the way speakers of neighboring dialects negotiate a word in real time. The failure mode to avoid is a private language only you can read — fast, fluent, and opaque to everyone else, including future you. The discipline is that every mark stay teachable: legible to a collaborator, explainable to the AI, and reversible when it turns out to mean less than you thought.
CAD without leaving the page
CAD is the most obvious domain where this should work and where the current interface is most hostile to the way designers actually think. Every mechanical engineer I know sketches the part on paper before opening SolidWorks. The sketch captures the topology, the proportions, the functional intent — which surfaces mate, where the loads go, what moves relative to what. Then they throw the sketch away and re-enter everything through dialog boxes, dimension fields, and feature trees. The sketch was the design; the CAD model is the transcription. That transcription is where hours go and where intent gets lost.
The research community has been chipping away at this for decades. Teddy (Igarashi et al., SIGGRAPH 1999) showed that freehand 2D strokes can be inflated into 3D polygonal surfaces in real time — draw a silhouette, and the system makes wide parts fat and narrow parts thin. Users mastered it in ten minutes and made interesting models in minutes more. It proved that sketch-to-3D is not an AI problem; it is an interaction design problem. Free2CAD (Li et al., SIGGRAPH 2022) goes further: it takes ordered pen strokes as input and parses them into a sequence of parametric CAD operations — extrude, revolve, fillet — using a sequence-to-sequence transformer. The user draws as they would on paper; the system produces a real B-rep solid with editable feature history. DAVINCI (2024) does the inverse: given a raster sketch image, it jointly infers parametric primitives and geometric constraints (parallelism, coincidence, tangency) in a single pass.
What none of these systems do is let you stay on the page while the computation runs. Free2CAD produces a CAD file; you then open it in a separate program to simulate, check tolerances, or iterate. The sketch was input; the solid is output; the two live in different worlds. In a Sketch Computing Interface, the solid would appear on the AI layer of the same canvas, as geometry you can annotate, constrain, or send back for revision — without switching applications, without breaking spatial context, without losing the gestural thread. You draw a profile. You draw an arrow indicating extrusion direction. You draw a dimension mark. You draw your personal glyph for “simulate this.” The system extrudes, meshes, runs FEA, and overlays the stress field on the canvas in the same coordinate space as your sketch. You circle a high-stress region and scribble a fillet indication. The system re-runs. You never left the page.
The goal is not to replace CAD but to keep the designer in the cognitive mode where they do their best thinking — spatial, gestural, iterative — bringing the computation to them instead of demanding they leave it. The prerequisite is a protocol that preserves stroke structure end-to-end, a layer model that keeps human and machine contributions separate and independently revisable, and an AI that can interpret gestural marks as delegation rather than decoration. That is what Codrawer is for.
Next deep dive
Lineage — Sutherland, Bret Victor, MathPaper, SketchREAD, Code Shaping, Visual Sketchpad, tldraw Make Real — where the idea comes from and what each system teaches.
Or explore: Infrastructure, Multiplayer.