40.7°N  73.8°W
Pratt Institute · INFO 680 · AR interaction design · 2026

Shapes that listen.

A browser-based AR instrument made of nothing but hands. It reads hand motion and turns it into sound and shape at the same time, built across three rounds of testing with eight participants.

I designed the interaction and built the gesture detection and hand tracking in MediaPipe. This project shows how I use testing to decide what to keep, what to cut, and what to build next.

AetherPlay splash screen on a phone: Gesture · Sound · AR, the pixel AETHERPLAY logo and Tap to begin
RoleInteraction Designer
Gesture detection & hand tracking
Team3 designers
Yuxuan Xia, Zhen Chu, me
MethodsGesture prototyping
3 rounds of user testing
Iterative redesign
Timeline4 weeks + live final demo
Delivered May 2026

It started as a joke on a Zoom call.

Late one night, the team joked about controlling each other’s audio from across the call. The joke stuck. Our first prototype was a cat sprite that bounced when you pinched: one gesture, one sound.

That joke hid the real challenge. Gesture tracking, live audio and augmented reality each worked on their own. The question was how to fuse all three into one complete AR experience.

One gesture drives sound and shape.

Four tools, one loop. MediaPipe reads the hand, and a single detected gesture fires Tone.js and p5.js together, so what you hear and what you see never drift apart.

Augmented realityDigital content layered over the real world, straight from the browser and webcam.
MediaPipeGoogle’s ML model finds 21 landmark points on each hand, on-device.
Tone.jsThe audio engine. Turns each gesture into a synthesized note.
p5.jsThe visual engine. Draws the shapes that answer each gesture.

One pinch, one sound, one character.

The first build was deliberately small: pinch, and a cartoon character bounced with a note. We tested it with eight participants to see whether instant feedback alone could carry the experience.

FIG. 01Build 1 · the original cat prototype, live on webcam
86%
said it looked childish for how complex the interaction was
77%
wanted more visual reactivity
50%
wanted more interactions per session

The instant loop worked: nobody needed instructions. The cartoon look was the problem.

Characters inside geometric auras.

Build 2 wrapped the characters in geometric auras and added a gesture that flips them. We tested again with eight participants, some returning from round one.

FIG. 02Build 2 · the real prototype, characters flipping on gesture
93%
said flips were the strongest new feature
100%
still couldn’t tell if it was a toy or an instrument
65%
wanted more sound options
47%
said the animal and geometry mix felt incomplete

Flips landed. The hybrid didn’t. The next build had to commit to one visual language.

Pure geometry, played together.

The final build dropped the characters entirely and committed to sacred geometry, then added the two things testers kept asking for: more sound, and someone to play with.

🔊 Sound on — unmute
FIG. 03Final build · trace with your fingers, hear it play. Unmute for sound
Feature 01Full sacred geometryNo more characters. Each of six shapes calls up its own glowing 3D form, from a flower of life to a star tetrahedron, tesseract and hexagram.
Under the hood · p5.js
shapes.jsJavaScript
const shape = SHAPES[state.shape];
shape.draw(p, { rot: t * 0.4,
scale: 1 + meter.getValue() * 0.3 });

Each picker icon maps to its own p5.js draw function. The live level from a Tone.js meter scales the shape, so it pulses with the music.

Feature 02Duet modeTwo players join with a six-character lobby code. Each person’s hands are tracked locally and play into the same shared shape.
Under the hood · PeerJS
duet.jsJavaScript
const peer = new Peer(lobbyCode);
conn.send({ hand: results.landmarks });

PeerJS opens a direct WebRTC link from the six-character code. Only the 21 landmark points travel, never video, so each camera stays on its own device.

Feature 03Keepsake traceAt the end of a session, players download an image of their gesture trails. People value what they helped make: the IKEA effect.
Under the hood · p5.js
trace.jsJavaScript
trace.circle(tip.x, tip.y, r);
saveCanvas(trace, `trace_${seed}`, 'png');

Fingertips draw onto an offscreen p5 layer for the whole session. Save Everything exports that layer as a PNG named with the session seed.

Feature 04Layered soundOne synth patch became six instruments, one per shape: Strings, Dubstep, Rhodes, Wine Glass, Music Box and Choir, over warm pads.
Under the hood · Tone.js
sound.jsJavaScript
const synth = new Tone.PolySynth(INSTR[shape]);
synth.triggerAttackRelease(scale[finger], '8n');

Each shape swaps the Tone.js instrument preset. The finger that taps picks a note from a pentatonic scale, so any combination still sounds good.

What three builds taught us.

INSIGHT 01Instant feedback winsHand tracking that answered immediately got people playing without a single instruction.
INSIGHT 02Looks decide firstA cartoon look made a technically complex instrument read as a toy. People thin-slice before they touch anything.
INSIGHT 03Shared play landedLobby codes turned a solo demo into something people did together.
QR code that opens AetherPlay
Scan me to play
next case studyWikipedia iOS auditWCAG 2.1/2.2 audit for the Wikimedia FoundationView case studynext case studyCardStoryA marketplace collectors can trustView case study