TARS · Human–robot interaction research

Where it started

This page is about human–robot interaction research at TARS — and where it started for me. When I started as a freshman in 2019, the PhD student Robert Miller introduced me to VR; I began doing research with an HTC Vive and lighthouses. I was shocked by Google Earth VR's realism and its ability to work like a time machine; the philosophical, psychological, and cognitively fascinating sides of XR became impossible to ignore. I got to use the Quest for the first time, and as soon as I was on campus I was always hunting for the coolest possible things to do — virtual reality and human–computer interaction felt like the right place to be.

Robert and I later worked on a dataset comparing virtual and real object manipulation (pick up, move, place) in VR vs the real world — Unity scene, 3D-scanned objects, Azure Kinect capture — which we never finished as a paper (there's a dedicated page for it).

Sean was my professor in CS 141 Introduction to Computer Science I—my first CS class—where he first introduced object-oriented programming and how computers represent bits in a way that revealed everything about computation very clearly — strings, data structures, and the rest. He showed me around the TARS lab and walked me through what the group does. He encouraged me to explore ambitious ideas and to develop a modular drone platform; he also got me into linear algebra as soon as the Spring of the next semester, so I gained an intuition for its applications across 3D environments, transforms, computer graphics, and the like.

I wanted to build something like the Drones from Iain M. Banks's Culture novels: sentient, expressive machines with personality — the kind that communicate with aura fields ( that shift with mood: red for pleasure, green for friendliness, blue formality, grey frustration). In the Culture they're full citizens — sentient, with rights equal to the Minds — and they range from plain, functional forms to wildly ornate. I'd read over thirty books in the summer of 2020, and the inspiration for everything that followed was seeded there: creating science fiction with my own efforts and imagination.

Back then, language models and voice models weren't nearly as good as they are now. There are finally research and development tools high-quality enough to put real effort into building that kind of expressive flying robot — and the work we did at TARS on modular, repairable platforms and additive manufacturing is part of that thread. Below is the lab context, why we went additive, what we built, and where it points: personoids, embodied avatars, and HRI.

About TARS

The Terascale All-Sensing Research Studio (TARS) is led by Sean Banerjee and Natasha Banerjee (Associate Professors, Computer Science) and Ashutosh Shivakumar (Assistant Professor). When I was an undergraduate researcher there it was at Clarkson University; TARS is now at Wright State University, Dayton, OH. The lab does research in human-driven AI, dense multi-person interaction capture, and HRI; full publications, awards, and people are on the lab site.

Earlier emblem

Current logo

TARS has used both an earlier emblem (five figures in an octagonal sensor frame) and the current lab logo.

Why additive at TARS

What drew me to the additive side of the lab wasn't just “print a bracket” — it was the idea that if a robot is built from parts you can scan, model, and reprint, then maintenance and iteration become a matter of geometry and material, not bespoke supply chains. Parametric design, replaceability, and eventually self-repair and recomposition. That mindset fit the modular drone work perfectly: we wanted units that could be repaired in the field, swapped, and stacked, so 3D printing wasn't a one-off prototype step but part of the system's design language.

The basis of the lab: Panoptic Studio

To understand TARS you have to understand the Panoptic Studio at Carnegie Mellon, which Sean Banerjee co-authored. The core thesis: many views yield stronger 3D understanding than a few sophisticated sensors. The system was a geodesic dome (~5.5 m) with 521 synchronized sensors (480 VGA, 31 HD, 10 Kinect v2), producing markerless, multi-person 3D motion capture in social settings. Sensor fusion — cross-view, cross-modal, and cross-time — combined 2D pose estimates and depth patches into robust 3D skeletons. That same fusion mindset carries through everything TARS does: shape repair, handover, VR biometrics, and the drone work I was part of. For the full technical story, see the Panoptic Studio paper (ResearchGate) and the Panoptic dataset.

What we did

The modular drones we built used 3D-printed frames and off-the-shelf electronics so any part could be replaced without the whole system being special. v1 was the pancake — so called because you stack units on top of each other to form a pancake stack; it's conic as well. It was a collaborative effort: I drove bi-weekly sprints with Spencer Carr, AJ Lombardi, and Patrick O'Mahoney, coordinated the grad students and PhDs — including Xinchao Song, who I did a lot of the work with — and coordinated the efforts of the other undergrads. v2 was the big, heavy box with ball-and-socket joints at the corners. v3 was the small box with joints that can have magnets on the ends to connect the drones. I was involved in design and fabrication of 3D-printed components across those iterations: arm mounts, mating surfaces, ducted shrouds, and jigs for assembly. FDM (filament) printing was tuned for stiffness, weight, and crash resilience; holding a part you'd designed and iterated on made the “repair by reprinting” thesis concrete. CAD was in SolidWorks with parametric models so arm length, motor spacing, or connector geometry could change without redrawing from scratch.

After I left, the lab brought Crazyflie into the pipeline — a small, research-oriented quadcopter (Crazyflie 2.1 Plus) with strong SDK and swarm support, indoor-safe and easier to interface with. That let the team focus on stacking logic, coordination, and VR teleoperation while the heavier custom builds remained the platform for full MosaicDrone-style stacking. TARS has continued in that direction: automatic assembly with manipulator robots and repair of objects through learning their precise geometries. The work I was part of — design and fabrication of 3D-printed components, iteration on pancake (v1), big box (v2), and small box (v3) — sits inside that broader lab focus. That line leads to MosaicDrone (MOSAIC), where modular, stackable units and standardized interfaces are central, and additive and recyclofacturing are part of the pipeline.

TARS contributions to the field

My direct work was in additive manufacturing and modular drones, but the lab's broader output gives context for why that work matters. Here's the landscape from their publication list (full list on the lab site):

Shape repair and fabrication: DeepJoin, DeepMend, MendNet, Fantastic Breaks; automated 3D-printed restorations and repair-by-reprinting (ACM SCF, SIGGRAPH Asia, ECCV, CVPR, SGP).
Human–robot handover: HOH dataset, HI-Grasp, pre-handover timing and mimicry (NeurIPS Datasets, RO-MAN, ICRA, ARSO).
VR biometrics: Behavior-based authentication in VR, cross-system auth, motion forecasting (IEEE VR, AIxVR, AIVR, MMM — multiple Best Paper awards).
VR for learning and annotation: ClassesInVR, VRcabulary, VRmonic, VR-Hand-in-Hand (AIxVR, Best Demo).
Other: Connected health (CHASE), structure from motion (SIGMAP), thermal/depth sensing, workshops (AIVR4Exergame, HUBEDA, Secure Immersive Worlds).

TARS tutorials and resources

TARS publishes free online tutorials for integrating physiological sensing into virtual reality (VR Sensor Integration, supported by the De Luca Foundation). On this site, each module has its own page with my own explanations and subtutorials on top of the official material. Start at the tutorials index or jump to any module below.

Tutorials index · VR Sensor Integration →

Introduction

Tutorial modules

Events and additional resources

Troubleshooting resources

What stuck with me

I also took away a habit of thinking in interfaces — mechanical (mating surfaces, connectors), software (APIs, telemetry), and human (gestures, VR, voice). Getting the interfaces right meant the rest could iterate. That's stayed with me in every project since: whether it's AI avatars, holoportation pipelines, or tooling for collective creation, the question of how humans and systems meet at the boundary is where the interesting design lives.

Modular Aerial Robotics Project (MARP) — Design draft, v1 pancake / v2–v3 box prototypes, footage, and the VR teleoperation stack that embodied the drone.
MosaicDrone (MOSAIC) — The evolution into programmable aerial voxels: omnidirectional flight, mid-air docking, and swarm orchestration, with additive and recyclofacturing in the roadmap.
Drone Swarm Omniview (concept) — Mobile multi-modal sensor lab: swarm ring, LTC sync, VR/GUI control, SLURM-style data fusion (concept details in speculative section below).
Holoportation XR Telehealth — Real-time surgeon avatars and volumetric presence; the “avatar others can talk to” idea in a medical training context.

References: TARS Lab · TARS Publications · TARS at Clarkson (when I was there)

Speculative thoughts and reflections

Speculative: Personoids and virtual avatars in embodied robots

During my time at TARS I started thinking more concretely about personoids — not just characters on a screen, but virtual avatars embedded in embodied robots: a presence that has a body in the world (drones, manipulators, swarms) and a face, voice, and mind that feel like a person. The drones we built were a step toward that. In the VR teleoperation setup, you could embody the drone — see from its camera, fly with your hands and body — so the “pilot” and the “craft” were tightly coupled. The next step, for me, was the inverse: the drone (or robot) having an avatar that others can talk to and collaborate with, not just a pilot in a headset. So the body is real (it flies, it docks, it carries payload), but the “who” you interact with is a persistent identity — a personoid — that can show up in different embodiments or in mixed reality alongside physical motion.

That connects directly to the holoportation and AI-avatar work I do now: real-time volumetric capture, surgeon avatars for telemedicine, and AI that can answer from a knowledge bank in the voice of an expert. The through-line is: presence and identity in distributed, embodied systems. At TARS it was “you are the drone”; later it became “the drone (or the surgeon, or the tutor) can be present as an avatar that others perceive and work with.” Personoids are the generalization: agents that have personality, history, and appearance, and that can inhabit physical and virtual bodies. Additive and modular hardware give those bodies a way to be built, repaired, and recomposed — so the avatar isn't locked to one shell.

Speculative: Other reflections from that time

TARS was where I got serious about human–robot interaction, VR as a portal into another way of perceiving, and the idea that future robots might share resources, self-repair, and still have something like personality or whimsy. I'd read Iain M. Banks's Culture novels — the “Independent Drones” that communicate fluidly with humans and each other, build sand sculptures, and argue about ethics — and that image stayed with me: robots as participants in a social and material world, not just tools. The modular drone project was a concrete slice of that: we were building systems that could dock, share power, and eventually (in the MosaicDrone vision) form reconfigurable structures. The additive angle added another dimension: these systems could be repaired and extended by making new parts, so the boundary between “manufactured product” and “evolving artifact” blurred.

Speculative: Drone Swarm Omniview (preliminary concept)

During my time at TARS I worked on conceptual plans that extended the lab's omniview, multi-modal capture philosophy beyond fixed domes. The NSF-funded Gazebo 2.0 lab (Award 1730183) envisions a temporally synchronized, spatially calibrated sensor system (force plates, microphones, RGB and IR cameras, Kinects) in a 20×20×12 ft space, with LTC sync and computer vision to reconstruct 3D spatiotemporal data for human interaction — the same “many weak views fused into strong 3D” idea as the Panoptic Studio. The proposal I drafted, Drone Swarm Omniview, asked: what if that sensor array were mobile?

A of drones acting as a cooperative multi-sensor ring could capture omniview data in unstructured environments — sports, crowds, concerts, runners, human–robot interaction — where ground-based labs cannot go. The concept called for lightweight, modular drones (e.g. Raspberry Pi 4 + autonomous flight controllers, carbon-fiber 3D-printed frames), each hosting multiple sensors, with data streamed to a central station. Control would range from leader-based swarm behavior to “one object” modes ( over or track a subject); interfaces would include a GUI on a laptop or VR, where you could swap between first-person views from any drone or a third-person view built from the collective point cloud. Data fusion and task distribution would use something like SLURM for managing heterogeneous sensor streams; onboard compute would be supplemented by offloading to a workstation, with the swarm itself treated as a computer cluster for load-balancing.

The Panoptic Studio paper (ResearchGate) directly informed this proposal: the same thesis — integrate perceptual analyses over many viewpoints — applied to a mobile, reconfigurable sensor system. This remains a preliminary conceptual document; many details (hardware integration, sync, safety, regulatory) would need to be fleshed out for a full proposal or implementation. It sits alongside MARP and MosaicDrone as part of the thread from fixed omniview capture (Panoptic) to modular, mobile sensing.

Alif Jakir