A neurosurgeon in Boston stands in the MIT.nano Immersion Lab. More than 3,000 miles away, his virtual avatar stands next to a resident in São Paulo as they practice delicate surgery on a model of a baby's brain. The resident wears a headset and watches the avatar demonstrate the procedure, then replicates it while asking the expert questions — in real time, or from a recorded session with an AI that answers from the surgeon's knowledge bank. That transcontinental link is holoportation: live volumetric capture and streaming so the expert is functionally in two places at once. This project made that pipeline real for surgery education, and I worked on strong narrative design and high-quality autonomous generation of assets for virtual sims in this space.
The project
MIT News covered it in February 2024: the MIT.nano Immersion Lab worked with EDUCSIM, an AR/VR medical simulator company and 2023 START.nano cohort member, to create a surgeon avatar for transcontinental training. Dr. Benjamin Warf — director of neonatal and congenital neurosurgery at Boston Children's Hospital, pioneer of the ETV/CPC procedure for infant hydrocephalus, and MacArthur Fellow — was digitally captured at MIT.nano using high-fidelity motion capture, volumetric video, and sensor-equipped gloves and clothing. He performed the procedure on a physical baby-brain simulator; that capture became his avatar.
Residents like Matheus Vasconcelos at Santa Casa de São Paulo School of Medical Sciences could then train with Warf's avatar in two modes: synchronous (Warf drives his avatar in real time from the US, walks around the virtual room, talks to the resident, guides the procedure) and asynchronous (residents watch the avatar's demonstrations and ask questions; answers are delivered in multiple languages by AI drawing on Warf's research and a curated Q&A bank). A December demo went further: volumetric live video between the Immersion Lab and Brazil and was spatialized in real time through AR headsets — a two-way holoportation link.
Timeline
- March 2023 — Giselle Coelho (EDUCSIM) begins working with Immersion Lab technical staff (e.g. Talis Reks) to create Warf's avatar; EDUCSIM joins START.nano.
- November 2023 — Avatar is training residents in Brazil (e.g. Matheus Vasconcelos, Santa Casa de São Paulo); first sessions with the digital twin.
- December 2023 — Two-way volumetric live demo: capture streams between Immersion Lab and Brazil, spatialized in AR headsets in real time.
- February 2024 — MIT News covers the project.
Outcomes
After training with Warf's avatar, neurosurgery resident Matheus Vasconcelos said: “It was my first training using this model, and it had excellent performance. As a resident, I now feel more confident and comfortable applying the technique in a real patient under the guidance of a professor.” Warf described watching his avatar with residents as an “almost experience” — like having an identical twin. The pipeline is now a proof of concept for extending expert reach to remote and underserved areas (e.g. Amazon region) without the cost and time of travel; Coelho's point stands: “This is a way to give them the same level of education that they would get in other places, and the same opportunity to be in touch with Dr. Warf.”
Partners and impact
EDUCSIM was founded by Giselle Coelho, a pediatric neurosurgeon at Santa Casa de São Paulo and Sabará Children's Hospital. She built realistic physical simulators (e.g. baby brain models that bleed and allow full procedure practice from incision to closure) and combined them with VR to shorten the learning curve. The company focuses on pediatric neurosurgery training — hydrocephalus, craniosynostosis, neuroendoscopy — and on bringing expert training to remote and underserved regions. Warf has a long history of global training (e.g. CURE Children's Hospital Uganda, NeuroKids); the avatar extends that reach without travel. As Coelho put it in the MIT piece: one baby in the Amazon traveled 30 hours by boat for hydrocephalus surgery — training more surgeons with tools like this “can change reality for this baby and can change the future.”
Holoportation stack
Microsoft Research open-sourced the underlying holoportation system as 3D Telecommunications (GitHub), used in their 3D Telemedicine project. The repo provides real-time volumetric capture from multiple RGB+D streams (e.g. Azure Kinect), Fusion (watertight mesh from Fusion4D-style algorithm), Render and Viewer (Unity), and orchestration via a control panel. In this project we had early access to that codebase, so we could integrate holoportation into the Immersion Lab–EDUCSIM pipeline and push real-time surgery instruction from the US to Brazil.
Stack at a glance:
- Capture & streaming: Azure Kinect, Fusion4D-style Fusion, Render/Viewer (Unity), control panel
- MR apps: Unity 2022.3, MRTK, OpenXR; EDUCSIM_HoloNet (Photon, Azure Spatial Anchors, OptiTrack), HoloAIT
- AI Q&A: Whisper (speech→text), GPT-3.5 (question→answer index), 26-pair Portuguese/English bank, pre-recorded Professor Wharf audio
- Avatar: MetaHuman, USD/FBX (Professor Wharf), synced animation and audio; facial pipeline: ElevenLabs audio → NVIDIA Audio2Face → facial blendshapes; custom model for Audio2Face; transfer of blendshapes onto the original mesh
What we built
Our 2023 Immersion Lab workspace was an XR telehealth / surgical education system for pediatric neuroendoscopy and hydrocephalus — the same EDUCSIM domain. In the codebase, “holoportation” is the concept: the remote expert as a 3D teacher in mixed reality, rather than a fully live volumetric stream in every path. We shipped two main pieces.
EDUCSIM_HoloNet (Unity, MRTK, Azure Kinect, Photon, OptiTrack): a mixed-reality app with a menu that branches to Surgery One and AI. Surgery One plays a single, authored animation of Professor Wharf — the 3D avatar (MetaHuman, USD/FBX) of Dr. Warf — explaining the procedure, with synced audio and avatar states (e.g. Idle-Talking / Idle-Non-Talking). The AI path leads to the Q&A experience. So the resident either watches the professor's surgery demo or asks questions and gets spoken answers.
HoloAIT implements the AI-driven Q&A in MR. The user speaks; Whisper turns speech to text. That question is sent to GPT-3.5 with a system prompt containing a fixed question–answer bank (26 pairs in Portuguese, plus English) on pediatric neuroendoscopy, hydrocephalus, ETV, shunts, complications, and related topics. The model returns a single integer (1–26, or 27 for no match). The app then plays the corresponding pre-recorded audio — Professor Wharf's voice in Portuguese (or English) — and drives the avatar to a for the duration of the clip. So the “AI” is semantic matching of the user's question to a curated answer, then playback of high-quality, pre-recorded explanation. No RAG; the full bank lives in the prompt so behavior is predictable and safe for education.
The Question Bank and supporting docs (AI DATA BANK ANSWERS, EDUCSIM AI Component, synthesized voices, papers) define the pedagogy: what residents can ask and what the professor “says.” The avatar assets — prof_warf_fala01–03, prof_warf_hidro_cirurgia, and related animations/videos — are the on-screen teacher. For the face, we used ElevenLabs for high-quality speech, then NVIDIA Audio2Face to generate facial blendshapes from that audio; a custom model was created for Audio2Face and used to drive the character. Those blendshapes were then transferred onto the original mesh for the authored demos. A separate MetaHuman version of Professor Wharf was built for an interactive, real-time version of the avatar. Together, the MR app + AI Q&A + Professor Wharf assets form the immersive, asynchronous side of the training pipeline that the MIT News piece describes; the synchronous, live-volumetric side relied on the capture and streaming stack we had early access to.
My role
My focus was on narrative design and autonomous asset generation — making the virtual sims feel purposeful rather than tech demos. I designed the flow from the menu into Surgery One versus the AI path, thinking about pacing and pedagogy: how long should the avatar demonstrate before the resident tries? When they ask a question, how quickly should the answer come? The Q&A bank structure, prompt design, and mapping from a user's spoken question to the right pre-recorded answer were all part of getting that experience to feel like a conversation with an expert, not a search engine.
On the avatar side, I built the facial animation pipeline: ElevenLabs for high-quality speech synthesis, then NVIDIA Audio2Face to generate facial blendshapes from that audio. We created a custom model for Audio2Face tuned to this character, and I handled transferring those blendshapes onto the original mesh for the authored surgery demonstrations. I also worked on a separate MetaHuman version of Professor Wharf for the interactive, real-time avatar — the one where he responds live rather than from recordings.
The through-line was that narrative and asset quality both matter for education. A surgeon avatar that moves stiffly or speaks unconvincingly breaks the training experience, no matter how good the underlying knowledge is. Getting the details right — lip sync, pacing, the feel of a real teacher standing next to you — was the point.
Demos: MetaHuman interactive Professor Wharf · Audio2Face → blendshape transfer onto original mesh
References and further reading: Brain surgery training from an avatar (MIT News, Feb 2024) · MIT.nano Immersion Lab · Talis Reks · EDUCSIM – pediatric neurosurgery training · Dr. Benjamin Warf (Boston Children's) · Microsoft 3D Telecommunications (GitHub)