MusicTraces
A collaborative system that blends music-making and painting into a single synesthetic experience, designed to foster social communication and emotional regulation in adults with neurodevelopmental conditions.
Art-based therapies for NDDs rarely go beyond a single art form or a single condition.
Adults with neurodevelopmental conditions (NDDs) face persistent challenges in social communication and emotional regulation. Art-based therapies can help, but most existing technologies focus on one art form, target only one condition, or measure narrow skill metrics rather than broader outcomes like well-being. Very few systems have explored combining music and painting in an immersive, shared environment for people with moderate-to-severe conditions.
Every visual element has a sound. Every sound can be touched.
MusicTraces is a two-player activity inside the Magic Room, an interactive multisensory environment. Participants use tangible smart controllers to paint on a projected canvas — and every visual element they create has a corresponding sound. Paint spots become musical notes; lines become melodies that play back when touched. Colors merge when two players' lines cross.
Floor pads trigger background harmonies generated from Markov Chains trained on popular songs. Three interaction modes — Explore, Create, and Play — let participants move between improvisation and composition. A companion tablet gives caregivers real-time session control without interrupting the experience.
Three iterations of the tangible controller — driven by real user feedback.
My primary contribution focused on the design, engineering, and iterative redesign of the tangible controllers — the physical bridge between the user and the virtual canvas.
Brush + eraser. Some participants struggled with size; switching objects caused confusion.
Two objects → confusionReduced size, rounded edges, simplified button layout. Internal testing showed partial improvement.
Smaller → still confusing
Beyond the controllers, I contributed to the overall interaction design of the activity and participated in the co-design process with caregivers and individuals with NDDs.
Four phases involving participants, caregivers, and domain experts.
Brainstorming with 3 adults with NDDs and 5 caregivers. Participants used concept sheets to propose ideas through writing or drawing. Seven design themes emerged: full-body interaction, music visualization, collaboration, support, gamification, understanding, and expressivity.
The core activity was designed around "sound painting", translating workshop themes into concrete mechanics: synesthetic mapping, three interaction modes, floor-based music progression, and the hint system.
Three two-week design cycles with a psychologist specializing in ASD. Features were added, removed, or modified based on video walkthroughs. Key additions: closed shapes auto-fill, floor blobs for limited mobility, lines-without-sound option.
Individual sessions with 3 educators and a psychologist who hadn't been involved earlier. Validated the interaction design; refined the hint system, tablet UI density, and tutorial structure.
Two field studies, 18 adults with moderate-to-severe NDDs.
Caregivers noted that achieving comparable levels of social engagement typically takes over a year in traditional group art therapy. MusicTraces reached similar levels in a single session.
Social interaction and autonomy increased across sessions. The redesigned single-controller contributed to lower workload scores.
What we learned about designing for neurodivergent users.
Every visual element had a corresponding sound and vice versa, creating a shared creative space accessible to participants with limited verbal abilities. Social interaction emerged through the canvas — crossing lines, imitating drawings, making music together — without requiring verbal communication. Inspired by sound painting and projects like Reactable, collaboration happened through doing, not just talking.
The system’s flexibility let each participant find their own way into the activity. Repetitive circular gestures functioned as self-regulation strategies, not errors to correct. Some participants scribbled to produce sounds, others painted precisely. This variety positively surprised caregivers and aligns with neurodiversity-affirming design principles that value unintended uses.
The evolution from two separate controllers (brush + eraser) to a single tapered object — narrower at the center for ergonomic grip, with two distinct ends: colored LED for drawing, felt tip for erasing — reduced cognitive and motor load. In the first study, some participants struggled to manage two objects and remember which button did what. The unified design made the distinction between actions physical and visual, likely contributing to the improved usability reported in the second study.