HackStack

Project Info

Harmony

Devpost

Inspiration

Most people will never know what it’s like to truly see sound or hear color. For millions with sensory or cognitive challenges—and for anyone craving deeper therapy, connection, or creativity—access to new senses has always been a dream. We wanted to turn that dream into reality. Harmony was born from a single question: What if technology could give everyone new senses? We set out to build a platform that bridges art, therapy, and accessibility—where sound becomes color, emotion becomes music, and everyone can experience the world in entirely new ways.

What it does

Harmony lets users see, hear, and feel the world differently. Users can describe their experiences, upload songs, or share photos—and Harmony generates immersive, multi-sensory outputs in real time. Deaf users can visualize music and sound as moving color. Blind users can hear images and artwork as layered ambient soundscapes. Anyone can translate emotions into visual and musical experiences for therapy, mindfulness, or creativity. Harmony blends text, audio, and images into a coherent 3D sensory world—bridging accessibility, art, and emotion.

How we built it

Our architecture brings together multiple cutting-edge tools for multimodal generation and rendering: Frontend → FastAPI: The user describes an immersive experience through text or images. The frontend sends this data to the backend via FastAPI. Storage + Memory: All user data is saved to MongoDB for persistence and Letta for contextual memory tracking. Orchestration Layer: The Harmony Orchestrator receives input data and calls Claude to interpret text and images into a coherent “story” that defines mood, emotion, and sensory tone. 3D World Generation: Claude produces a structured 3D world JSON, describing spatial layout, color palette, and movement. This is passed to Tripo3D for world generation. Immersive Music Creation: In parallel, Google Lyria Realtime composes dynamic, emotion-matched music based on the same story context. Frontend Visualization: The generated 3D world is rendered interactively in the frontend using Three.js, syncing sound, color, and movement into a single living experience. Together, these components make Harmony feel alive—reacting instantly to emotions, sounds, and visuals in real time.

Challenges we ran into

Translating abstract emotion into both sound and color required careful multimodal alignment and experimentation with embeddings. Managing real-time performance across Claude, Lyria, and Tripo3D while maintaining low latency was extremely difficult. Making Harmony accessible for both blind and deaf users pushed us to think beyond standard interfaces—how can someone feel color or see rhythm? Building a coherent orchestration layer that merged APIs and models into one narrative pipeline took extensive debugging and fine-tuning.

Accomplishments we're proud of

Built a fully functional pipeline that fuses Claude, Google Lyria, and Three.js into an end-to-end multimodal system. Successfully generated real-time 3D worlds and matching ambient soundscapes from just text and images. Designed Harmony to serve both accessibility and creativity—not just assistive tech, but expressive tech. Created something that feels human—users described it as “a new way to experience emotion.”

What we learned

True innovation in accessibility happens when we design for emotion, not limitation. AI can act as a sensory bridge—translating modalities like sound and sight in deeply meaningful ways. Building multimodal orchestration pipelines (text → 3D → sound) taught us how important synchronization and latency control are for immersive experiences. Working with creative AI systems like Claude and Lyria showed us that narrative and technical design go hand in hand.

What's next

Launching Harmony Studio, a creative toolkit where artists, therapists, and educators can design their own multi-sensory worlds. Integrating wearable haptics to add touch feedback—letting users feel sound and light. Expanding real-time accessibility features for deafblind users through vibration and braille devices. Partnering with accessibility organizations and mental health practitioners to bring Harmony into therapy and classrooms. Harmony turns synesthesia into a superpower—making therapy, creativity, and accessibility open to everyone.

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only
  • CSSIn code
  • MongoDBIn code
  • Next.jsIn code
  • ReactIn code
  • Tailwind CSSIn code
  • TypeScriptIn code
  • FastAPIClaimed
  • PythonClaimed

6 of 8 appear in the indexed code. 2 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

19 KB

Source files

9

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

0 stars