Robert introduced me to VR when I was a freshman at Clarkson, and this project came out of that. We wanted to build the first dataset that paired VR object-manipulation tasks with corresponding real-world recordings — same objects, same task (pick up, move, place), same spatial configurations — so we could study how people actually behave differently across the two. The premise was simple: if VR is going to be used as a low-risk training tool for high-risk real-world tasks, you need to know where the gap is. We built the pipeline, ran captures, started the analysis — and never finished the paper. Robert was graduating in 2021 and I was deep in classes for my two degrees; life got in the way. I still think about remaking the Unity scene and picking this back up.
What we captured
The task was straightforward: pick up an object and move it to a target spot. We had six physical objects — a vase, a water bottle, a box of mac and cheese, an applesauce cup, an ice cube tray, and a bottle of isopropyl alcohol — each 3D-scanned so the virtual versions were one-to-one with the real ones. Eight configurations of start and end positions on a 0.7 m table.
In VR, the objects and positions appeared on a virtual table with the same dimensions as the real setup. We recorded headset and controller poses at 90 Hz; recording started when an object spawned and stopped when it was placed. Pick-up and place events were logged so we could segment approach, movement, and release.
For the real-world side, we had the same table with tape marking the eight configurations. Participants were cued on a laptop and raised their hands above their head before and after each trial — a clean start/end signal. We used two Azure Kinects for color, depth, body tracking, and segmentation. Point clouds were reconstructed and aligned (checkerboard calibration plus ICP on tracked joints), then temporally aligned and manually trimmed to match the VR structure: neutral pose → grab → move → release → neutral.
We ended up with VR data from two people (4 trials per object–configuration pair, 192 recordings each) and real-world data from one (2 trials per pair, 96 recordings). The paper draft also laid out a bigger five-session protocol — within-modality, cross-modality (same object/config in VR then real or vice versa), and a “ tracing” session with qualitative questions — that we never got to run.
What the analysis showed
We started with consistency within VR: same person, same configuration, same vs different object. We used dynamic time warping (DTW) on right-hand controller position over the movement segment. The early results were interesting — people do measurably differently when handling different objects in the same configuration, even in VR. Same-object pairs had lower DTW distance than different-object pairs, and a lot of the variance came from the grab and release ends of the motion (we tried trimming the first and last 10% of the path to see if that held). The plan was to do the same analysis within real-world and cross-modality (VR hand vs Kinect hand after non-scaled ICP alignment), but the cross-modality and real-world sections stayed as placeholders — “to be filled in when manual marking of real-data is complete.”
Where it stands
We drafted the paper for a conference (IEEE-style, vgtc class) but never submitted. The draft had a solid related-work section (VR behavioral biometrics, haptic devices, gait, cross-modal re-ID), the full dataset and protocol description, and proposed improvements — more stable object spawns in Unity, a progress UI, breaks, an L-shaped table for more varied motion, better camera coverage for real capture. The idea is there, the pipeline and initial data are there; the write-up is incomplete.
Picking this back up would mean remaking the Unity scene (fixing object physics so tall objects don't , adding a progress UI and breaks), finishing the real-world trimming, and completing the cross-modality DTW — and maybe orientation metrics. I go back and forth on whether to push it toward a publication or just let the page document what we built. Either way, this is where VR research started for me — Robert and TARS — and I wanted it visible.
Draft paper (2021)
The draft we never finished is below. You can open the PDF in a new tab if the embed doesn't display well.