# Project export: Harmony This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Harmony lets anyone see sounds, hear images, and feel emotions through color—bringing new senses, accessibility, and creativity to everyone. - Devpost: https://devpost.com/software/harmony-a0wdlv - GitHub: https://github.com/VishnuK1947/Harmony - Video: https://www.youtube.com/embed/gPDgn7NQQ94?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Rodin Sh (9 commits), Vishnu Kadaba (4 commits), dhruvywuvy (2 commits) ## Devpost submission (written by the team) ### Inspiration Most people will never know what it’s like to truly see sound or hear color. For millions with sensory or cognitive challenges—and for anyone craving deeper therapy, connection, or creativity—access to new senses has always been a dream. We wanted to turn that dream into reality. Harmony was born from a single question: What if technology could give everyone new senses? We set out to build a platform that bridges art, therapy, and accessibility—where sound becomes color, emotion becomes music, and everyone can experience the world in entirely new ways. ### What it does Harmony lets users see, hear, and feel the world differently. Users can describe their experiences, upload songs, or share photos—and Harmony generates immersive, multi-sensory outputs in real time. Deaf users can visualize music and sound as moving color. Blind users can hear images and artwork as layered ambient soundscapes. Anyone can translate emotions into visual and musical experiences for therapy, mindfulness, or creativity. Harmony blends text, audio, and images into a coherent 3D sensory world—bridging accessibility, art, and emotion. ### How we built it Our architecture brings together multiple cutting-edge tools for multimodal generation and rendering: Frontend → FastAPI: The user describes an immersive experience through text or images. The frontend sends this data to the backend via FastAPI. Storage + Memory: All user data is saved to MongoDB for persistence and Letta for contextual memory tracking. Orchestration Layer: The Harmony Orchestrator receives input data and calls Claude to interpret text and images into a coherent “story” that defines mood, emotion, and sensory tone. 3D World Generation: Claude produces a structured 3D world JSON, describing spatial layout, color palette, and movement. This is passed to Tripo3D for world generation. Immersive Music Creation: In parallel, Google Lyria Realtime composes dynamic, emotion-matched music based on the same story context. Frontend Visualization: The generated 3D world is rendered interactively in the frontend using Three.js, syncing sound, color, and movement into a single living experience. Together, these components make Harmony feel alive—reacting instantly to emotions, sounds, and visuals in real time. ### Challenges we ran into Translating abstract emotion into both sound and color required careful multimodal alignment and experimentation with embeddings. Managing real-time performance across Claude, Lyria, and Tripo3D while maintaining low latency was extremely difficult. Making Harmony accessible for both blind and deaf users pushed us to think beyond standard interfaces—how can someone feel color or see rhythm? Building a coherent orchestration layer that merged APIs and models into one narrative pipeline took extensive debugging and fine-tuning. ### Accomplishments we're proud of Built a fully functional pipeline that fuses Claude, Google Lyria, and Three.js into an end-to-end multimodal system. Successfully generated real-time 3D worlds and matching ambient soundscapes from just text and images. Designed Harmony to serve both accessibility and creativity—not just assistive tech, but expressive tech. Created something that feels human—users described it as “a new way to experience emotion.” ### What we learned True innovation in accessibility happens when we design for emotion, not limitation. AI can act as a sensory bridge—translating modalities like sound and sight in deeply meaningful ways. Building multimodal orchestration pipelines (text → 3D → sound) taught us how important synchronization and latency control are for immersive experiences. Working with creative AI systems like Claude and Lyria showed us that narrative and technical design go hand in hand. ### What's next Launching Harmony Studio, a creative toolkit where artists, therapists, and educators can design their own multi-sensory worlds. Integrating wearable haptics to add touch feedback—letting users feel sound and light. Expanding real-time accessibility features for deafblind users through vibration and braille devices. Partnering with accessibility organizations and mental health practitioners to bring Harmony into therapy and classrooms. Harmony turns synesthesia into a superpower—making therapy, creativity, and accessibility open to everyone. ## README (from the GitHub repository) # 🌈 SenseSync - Universal Sensory Translation Platform > **See sounds. Hear colors. Experience synesthesia through AI.** ## 🎯 Objective **SenseSync democratizes synesthesia through AI, making multisensory experiences accessible to everyone.** ### **The Problem:** - **48M** deaf Americans cannot experience music emotionally - **8M** blind Americans cannot experience visual art - Mental health conditions need multisensory therapeutic tools that don't exist - Only **4%** of people naturally experience synesthesia ### **Our Solution:** An AI platform built on **Model Context Protocol (MCP)** that translates between senses in real-time: - 🎵 **Audio → Visual**: See music as real-time colors - 🎨 **Visual → Audio**: Hear paintings as soundscapes - 💬 **Text → Multisensory**: Transform emotions into colors + sounds - ♿ **Accessibility**: Deaf users "see" sounds, blind users "hear" images - 🧠 **Therapy**: PTSD, anxiety, autism treatment tools - 🎭 **Creative**: Compose with colors, paint with sounds ### **Impact:** - **56M Americans** with sensory disabilities gain new experiences - **Clinical applications** for mental health (40% improvement in studies) - **Educational enhancement** (multisensory learning = 40% better retention) - **New creative medium** for artists, musicians, game developers **Total Market**: $8.2B (accessibility + mental health + creative tools) --- ## 🛠️ Tech Stack ### **Frontend** ``` Next.js 14 + TypeScript + TailwindCSS Three.js (3D visualizations) WebAudio API (sound synthesis) Framer Motion (animations) ``` ### **Backend & AI (MCP Architecture)** ⭐ ``` Claude Sonnet 4.5 ⭐ → MCP orchestrator, semantic understanding Letta Cloud ⭐ → Stateful memory, learns preferences Groq ⭐ → Ultra-fast inference (<100ms) ``` ### **Custom MCP Servers** ``` 1. audio-analysis → Librosa, Essentia (extract audio features) 2. color-generation → Groq (real-time color palettes) 3. sound-synthesis → Vapi ⭐ + fish.audio ⭐ (generate audio) 4. image-analysis → Claude Vision ⭐ + Reka ⭐ 5. memory → Letta ⭐ (user preferences) ``` ### **Data & Storage** ⭐ ``` Elastic Cloud ⭐ → Search, Agent Builder with MCP Chroma ⭐ → Vector database (similarity search) Conway ⭐ → Real-time streaming PostgreSQL → User data Redis → Caching ``` ### **Integrations** ⭐ ``` Composio Toolrouter ⭐ → MCP server orchestration Bright Data ⭐ → Research data scraping CodeRabbit ⭐ → AI code reviews Warp ⭐ → AI terminal ``` ### **Architecture** ``` User → Next.js Frontend ↓ Claude (MCP Orchestrator) ⭐ ↓ 5 Custom MCP Servers → Composio Router ⭐ ↓ Letta Memory ⭐ + Elastic ⭐ + Chroma ⭐ ↓ Real-time Output ``` --- ## 📦 Key Dependencies **Frontend:** ```json "next": "^14.0.0" "three": "^0.159.0" "@anthropic-ai/sdk": "^0.20.0" "tone": "^14.7.77" ``` **Backend:** ```txt anthropic>=0.20.0 letta>=0.1.0 librosa>=0.10.0 fastapi>=0.109.0 elasticsearch>=8.11.0 chromadb>=1.7.0 ``` --- ## Detected evidence (automated analysis) Indexed codebase: 9 recognized source files, 19 KB. - CSS (language) — detected in the code - MongoDB (technology) — detected in the code - Next.js (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - FastAPI (technology) — claimed on Devpost, not found in the code - Python (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (14 of 14) ``` README.md src/sense-sync-frontend/.gitignore src/sense-sync-frontend/app/globals.css src/sense-sync-frontend/app/layout.tsx src/sense-sync-frontend/app/page.tsx src/sense-sync-frontend/app/sign-in/[[...sign-in]]/page.tsx src/sense-sync-frontend/app/sign-up/[[...sign-up]]/page.tsx src/sense-sync-frontend/eslint.config.mjs src/sense-sync-frontend/middleware.ts src/sense-sync-frontend/next.config.ts src/sense-sync-frontend/package.json src/sense-sync-frontend/postcss.config.mjs src/sense-sync-frontend/README.md src/sense-sync-frontend/tsconfig.json ``` ### Dependencies - src/sense-sync-frontend/package.json: @clerk/nextjs@^6.34.0, @specy/liquid-glass-react@^1.0.2, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, eslint@^9, eslint-config-next@16.0.0, liquid-glass-react@^1.1.1, lucide-react@^0.548.0, mongodb@^6.20.0, next@16.0.0, react@19.2.0, react-dom@19.2.0, tailwindcss@^4, typescript@^5 ### Recent commits (newest first) - Merge pull request #3 from VishnuK1947/revert-2-rodins-mods - Revert "Rodins mods" - Merge pull request #2 from VishnuK1947/rodins-mods - Update run_all_servers.py - Merge pull request #1 from VishnuK1947/frontend - Update page.tsx - Authenticated JWT token sent to API - Merge remote-tracking branch 'origin/backend' into backend - music ready for connecting and shit - world and asset generator - With Authentication - backend mcp server skeleton - Next JS Setup - start off readme - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### src/sense-sync-frontend/package.json ``` { "name": "sense-sync-frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint" }, "dependencies": { "@clerk/nextjs": "^6.34.0", "@specy/liquid-glass-react": "^1.0.2", "liquid-glass-react": "^1.1.1", "lucide-react": "^0.548.0", "mongodb": "^6.20.0", "next": "16.0.0", "react": "19.2.0", "react-dom": "19.2.0" }, "devDependencies": { "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "16.0.0", "tailwindcss": "^4", "typescript": "^5" } } ``` ### src/sense-sync-frontend/app/layout.tsx ```typescript // app/layout.tsx import { type Metadata } from 'next' import { ClerkProvider, SignInButton, SignUpButton, SignedIn, SignedOut, UserButton, } from '@clerk/nextjs' import { Geist, Geist_Mono } from 'next/font/google' import './globals.css' const geistSans = Geist({ variable: '--font-geist-sans', subsets: ['latin'], }) const geistMono = Geist_Mono({ variable: '--font-geist-mono', subsets: ['latin'], }) export const metadata: Metadata = { title: 'Harmony', description: 'AI-powered creative assistant', } export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode }>) { return (
{children} ) } ``` ### src/sense-sync-frontend/app/page.tsx ```typescript // app/page.tsx 'use client' import { useState, useRef } from 'react'; import { Plus, Send, Music, X } from 'lucide-react'; import { useAuth } from '@clerk/nextjs'; interface AttachedFile { id: string; file: File; name: string; type: string; preview: string | null; } export default function Home() { const [textInput, setTextInput] = useState(''); const [attachedFiles, setAttachedFiles] = useState([]); const fileInputRef = useRef(null); const { getToken } = useAuth(); const handleFileSelect = (e: React.ChangeEvent) => { const files = Array.from(e.target.files || []); const newFiles: AttachedFile[] = files.map(file => ({ id: Math.random().toString(36).substr(2, 9), file: file, name: file.name, type: file.type, preview: file.type.startsWith('image/') ? URL.createObjectURL(file) : null })); setAttachedFiles([...attachedFiles, ...newFiles]); }; const removeFile = (id: string) => { setAttachedFiles(attachedFiles.filter(f => f.id !== id)); }; const handleSend = async () => { if (!textInput.trim() && attachedFiles.length === 0) { return; } try { const formData = new FormData(); formData.append('prompt', textInput); attachedFiles.forEach((fileData, index) => { formData.append(`files`, fileData.file); }); const token = await getToken(); const response = await fetch('/api/harmony', { method: 'POST', headers: { Authorization: `Bearer ${token}`, }, body: formData, }); if (response.ok) { const result = await response.json(); console.log('Response:', result); setTextInput(''); setAttachedFiles([]); } else { console.error('Failed to send:', response.statusText); } } catch (error) { console.error('Error sending data:', error); } }; return ( <>
{/* Main Content */}
{/* Title */}

Harmony

{/* Input Section */}
{/* Plus Button */} {/* Text Input */} setTextInput(e.target.value)} onKeyDown={(e) => e.key === 'Enter' && handleSend()} placeholder="Describe your immersive experience..." className="flex-1 h-14 px-6 rounded-full bg-white/10 border-2 border-white/15 text-white placeholder-white/40 focus:outline-none focus:border-white/40 hover:border-white/25 transition-colors" /> {/* Send Button */}
{/* File Previews */} {attachedFiles.length > 0 && (
{attachedFiles.map(file => (
{file.type.startsWith('image/') && file.preview ? ( {file.name} ) : ( )}
))}
)}
); } ``` ### src/sense-sync-frontend/app/sign-in/[[...sign-in]]/page.tsx ```typescript import { SignIn } from '@clerk/nextjs' export default function SignInPage() { return (
) } ``` ### src/sense-sync-frontend/app/sign-up/[[...sign-up]]/page.tsx ```typescript import { SignUp } from '@clerk/nextjs' export default function SignUpPage() { return (
) } ``` ### src/sense-sync-frontend/next.config.ts ```typescript import type { NextConfig } from "next"; const nextConfig: NextConfig = { /* config options here */ }; export default nextConfig; ``` ### src/sense-sync-frontend/middleware.ts ```typescript import { clerkMiddleware } from '@clerk/nextjs/server'; export default clerkMiddleware(); export const config = { matcher: [ // Skip Next.js internals and all static files, unless found in search params '/((?!_next|[^?]*\\.(?:html?|css|js(?!on)|jpe?g|webp|png|gif|svg|ttf|woff2?|ico|csv|docx?|xlsx?|zip|webmanifest)).*)', // Always run for API routes '/(api|trpc)(.*)', ], }; ``` ### src/sense-sync-frontend/app/globals.css ```css @import "tailwindcss"; :root { --background: #ffffff; --foreground: #171717; } @theme inline { --color-background: var(--background); --color-foreground: var(--foreground); --font-sans: var(--font-geist-sans); --font-mono: var(--font-geist-mono); } @media (prefers-color-scheme: dark) { :root { --background: #0a0a0a; --foreground: #ededed; } } body { background: var(--background); color: var(--foreground); font-family: Arial, Helvetica, sans-serif; } @keyframes gradient-shift { 0% { background-position: 0% 0%; } 25% { background-position: 100% 0%; } 50% { background-position: 100% 100%; } 75% { background-position: 0% 100%; } 100% { background-position: 0% 0%; } } @keyframes rotate { 0% { transform: rotate(0deg); } 100% { transform: rotate(360deg); } } .animated-gradient-bg { position: relative; background: #0a0a0a; overflow: hidden; } .animated-gradient-bg::before { content: ''; position: absolute; top: -100%; left: -100%; width: 300%; height: 300%; background: radial-gradient(circle at 20% 50%, rgba(147, 51, 234, 0.9) 0%, transparent 25%), radial-gradient(circle at 80% 30%, rgba(14, 165, 233, 0.8) 0%, transparent 30%), radial-gradient(circle at 60% 70%, rgba(236, 72, 153, 0.9) 0%, transparent 25%), radial-gradient(circle at 30% 80%, rgba(99, 102, 241, 0.7) 0%, transparent 35%); animation: rotate 20s linear infinite; filter: blur(60px); } .animated-gradient-bg>* { position: relative; z-index: 10; } ```