You walk up to a Looking Glass display at the MIT Museum. Cortana appears in the volume and asks you to put on a pair of glasses — BCI glasses, with EEG sensors. Once they're on, she speaks a line about the king and the pawn going back into the same box: the system is ready. Then she asks, Are you ready to start? You don't touch anything. The exhibit does not try to read arbitrary words out of your thoughts. The audio tells you how to answer: do mental math to signal YES and do not do mental math (stay closer to a neutral baseline) to signal NO. The BCI measures your EEG during those windows, extracts features that differ between “doing mental arithmetic” and “not doing it,” and a classifier maps that to YES or NO. If the system classifies YES, she welcomes you home. If NO (or timeout), she half-disappears and offers to try again. The whole loop is hands-free after the glasses go on: presence from your hands, choice from task-dependent brain activity the pipeline can measure.
The experience
The installation was exhibited at the MIT Museum. The flow is deliberately staged. First, the display and an Ultraleap depth sensor detect that someone is there — both hands in view. Cortana shows up and instructs you to put on the BCI glasses (a 3D model of the glasses appears on her; the asset was provided by Nataliya Kos'myna for the exhibit). The system waits until the Attentiv-U EEG is connected and streaming. Once the backend has the BCI link, Cortana delivers the chess metaphor and moves into the question: Are you ready to start?
You have two ten-second windows to respond. YES and NO are not telepathy: they are whatever label the BCI assigns when it decides your EEG during the window looks more like the doing mental math state versus the not doing mental math state you were instructed to use for each answer. That decision is sent over a WebSocket to a relay server, which forwards it to the Unity client driving the Looking Glass. On YES, Cortana plays a welcome-home animation and asks if you want to replay. On NO (or timeout), she partially and asks if you want to try again. The state machine in Unity handles all of this — hand visibility, BCI connection, timer, and branching — so the narrative stays coherent without any controllers or buttons.
How it works
The is rendered by Unity and displayed on a Looking Glass (with the Looking Glass Bridge running). An Ultraleap depth sensor (Leap Motion) tracks your hands: when both are in view, Cortana is parented to your left palm in the scene so she moves with you; when they leave the frame, she returns to a default pose and the UI can prompt you to bring your hands back.
The brain side is separate. The Attentiv-U EEG glasses stream to a backend (e.g. ws://backend.cortanahologram.com:3000). A small Node.js relay server runs two WebSocket servers: one for the BCI client (port 3000), one for Unity (port 3001). When the BCI sends a parsed YES or NO, the server broadcasts it to every connected Unity client. Unity never talks to the EEG directly — it only receives high-level answers from the relay. That keeps the pipeline modular: swap or upgrade the BCI hardware or classifier without changing the Unity experience logic.
Inside Unity, GameManager holds the state machine (e.g. hands in scene → glasses on → BCI ready → awaiting answer → welcome or try again). ConnectionWebSocket maintains the link to the relay and fires an AnswerEvent when a yes/no string arrives. Cortana (the component) drives the character: dissolve/reappear materials, animation triggers for each branch, and the audio clips for each line. The result is a single, continuous experience from “put on the glasses” to “welcome home” or “try again,” driven by presence and mental commands.
How the EEG works
Attentiv-U is a research wearable out of the MIT Media Lab — Nataliya Kos'myna, Pattie Maes, and colleagues — a normal-looking pair of glasses that hides both EEG and EOG (eye-movement) sensing in the frame, built to read attention and cognitive load in real time. There are no gel electrodes: selected parts of the frame are machined from pure silver. The temple tips behind your ears are the two EEG electrodes (sites TP9 and TP10 in the 10–20 system); the nose pads are the EOG electrodes; a silver plate at the nose bridge is the reference. It samples at 1000 Hz, amplifies and filters the biopotentials on the frame itself (roughly 0.5–50 Hz for EEG), and streams over Bluetooth, with bone-conduction audio for feedback — no cap, no gel, no wires, no second box on the desk.
Those silver EEG electrodes at the temple tips pick up tiny electrical potentials produced by cortical activity — the summed postsynaptic potentials of large populations of neurons, filtered through skull and skin. That raw signal is (microvolts) and mixed with (eye blinks, muscle, line power), so the first step on the device or in the BCI client is amplification and filtering: bandpass to the bands that carry task-relevant information (e.g. delta, theta, alpha, beta), and often artifact rejection or correction for ocular and motion noise.
The filtered signal is then digitized and windowed in time. For a binary YES/NO choice, the system needs a way to tell “user intended YES” from “user intended NO” (or “no clear response”). In this installation, the cues point at a cognitive-load style split: the audio instructs you to do mental math for YES and refrain from that for NO, so the classifier is really discriminating EEG patterns consistent with engaged mental arithmetic versus a more neutral baseline — not decoding a silent sentence. This also plays to exactly what AttentivU was built for: it is an attention-and-cognitive-load sensor, and doing hard math on cue is one of the most reliable ways to spike cognitive load deliberately — your blink rate even drops while you concentrate, which the nose-pad EOG can register. So “think hard math for yes” isn't a trick bolted onto the glasses; it rides the very signal they were designed to read. Other BCIs use different contrasts: P300 (attend to a flashing “YES” vs “NO” cue), or motor imagery (imagine one movement for YES and another for NO); the idea is always to pair each answer with a brain state that produces repeatable, separable signals.
In this pipeline, the classifier runs either on the BCI device itself or in a client that receives the EEG stream and connects to the relay. It outputs a single decision per query window: yes or no. That result is sent as JSON (e.g. {"data": {"answer": "yes"}}) to the bridge on port 3000. Unity, meanwhile, opens the ten-second window and sends START_COMMAND so the BCI side knows when to listen. If a YES or NO arrives before the window ends, the relay forwards it to Unity and the state machine branches; if not, the experience treats it as a timeout (NO). The two ten-second windows give enough time for the brain to produce a stable, classifiable pattern and for the pipeline to run amplification, feature extraction, and classification without feeling rushed.
So end-to-end: you perform one of two instructed cognitive tasks (mental math versus not); those tasks change your EEG in ways the classifier can separate; the output is a discrete YES or NO symbol; the relay delivers it to the hologram; Cortana responds. The exhibit at the MIT Museum made that loop tangible — no keyboard, no controller, just put on the glasses and follow the mental-math cue for yes or withhold it for no.
Built with
- Display: Looking Glass (Looking Glass Bridge + Unity plugin)
- Presence & placement: Ultraleap depth sensor (hand tracking)
- BCI: Attentiv-U EEG glasses
- Relay: Node.js, Express,
ws(WebSockets) - Experience: Unity, C#, TextMeshPro; Cortana FBX + exhibit animations, dissolve shaders. This page's live 3D preview uses additional Mixamo clips in the web viewer.
The AttentivU paper
The glasses come from this paper out of the MIT Media Lab. Citation and abstract below; the PDF is mirrored here for convenience and remains the copyright of its authors / IEEE.
Nataliya Kos'myna, Caitlin Morris, Utkarsh Sarawgi, Thanh Nguyen, and Pattie Maes. “AttentivU: a Wearable Pair of EEG and EOG Glasses for Real-Time Physiological Processing.” MIT Media Lab, 2019 (IEEE). [PDF · Media Lab]
Recently several research projects have explored using physiological sensors such as electroencephalography (EEG) or electrooculography (EOG) electrodes to measure the engagement of a user in different contexts and augment learning activities. However, these systems still suffer from limitations such as an absence of a socially acceptable design, or use of impractical gel-based electrodes. We present AttentivU, a device using both EEG and EOG for real-time monitoring of physiological data. The device is designed as a socially acceptable pair of glasses and employs silver electrodes as an alternative to the commonly used silver/silver chloride (Ag/AgCl) “wet” electrodes. A detailed description of the hardware design and proof of concept prototype is provided, as well as a side by side comparison of conventional wet electrodes.
Cortana and related characters are the property of Microsoft Corporation / 343 Industries (Halo). This project is an independent research and educational implementation and is not affiliated with or endorsed by Microsoft.