HackStack

Project Info

DAWZY

Devpost

Inspiration

🎧 Music production software is incredibly powerful, but often intimidating. Many aspiring creators hear melodies in their heads but lack the technical knowledge or time to bring those ideas to life. We wanted to change that. Our goal was to build an assistant that empowers anyone, beginners & seasoned artists, to turn their musical ideas into reality with just their voice or a simple prompt.

What it does

🤖 DAWZY is an AI music production assistant that integrates with any DAW (digital audio workstation). It supports: Voice-to-Voice Communication: Have a natural conversation with the assistant about your music project. Text-to-Music Prompting: Describe what you want and let DAWZY generate instruments, tracks, and effects. Humming-to-MIDI Conversion: Hum a melody and have it transcribed into a playable MIDI part. DAW Control via Language: Tell DAWZY to add reverb, adjust EQs, or create new instruments, and it writes scripts to control your DAW environment directly. Contextual Memory: DAWZY keeps track of your session and helps build on ideas progressively.

How we built it

🛠 Frontend: A React-based chat interface with support for audio messages, speech input, and real-time feedback. Backend: A Flask server acting as a bridge between user input, language models, and Reaper (our current DAW prototype). LLMs: OpenAI models power the conversational and action logic, while Claude handles DAW scripting. Music Generation: We integrated Beatoven.ai for prompt-based music generation. Speech-to-Speech: We designed the architecture to allow natural voice dialogue with OpenAI's new speech model. Audio Conversion Tools: We implemented humming-to-MIDI by recording audio, converting it to WAV, and sending it to our MIDI processing logic. Reaper Integration: Using python-reapy, we remotely manipulate Reaper sessions from user commands.

Challenges we ran into

⚠️ Audio Handling: Getting consistent recording formats and playback behavior across browsers was surprisingly complex. Audio Handling: Getting consistent recording formats and playback behavior across browsers was surprisingly complex. LLM Integration: Coordinating different models (chat, voice, and action) while maintaining session continuity took careful planning. LLM Integration: Coordinating different models (chat, voice, and action) while maintaining session continuity took careful planning. DAW Control: Reaper exposes a lot of functionality, but translating natural language into precise DAW scripting was a major challenge. DAW Control: Reaper exposes a lot of functionality, but translating natural language into precise DAW scripting was a major challenge. Latency and Feedback: Ensuring that the assistant felt responsive, even when handling complex tasks like MIDI conversion, required efficient message handling and is something we wish to improve on in the future Latency and Feedback: Ensuring that the assistant felt responsive, even when handling complex tasks like MIDI conversion, required efficient message handling and is something we wish to improve on in the future

Accomplishments we're proud of

🏆 A Working End-to-End Flow: From voice input to MIDI playback, we created a seamless creative loop. A Working End-to-End Flow: From voice input to MIDI playback, we created a seamless creative loop. Intelligent DAW Manipulation: DAWZY can write and run scripts to make edits inside a live DAW session. Intelligent DAW Manipulation: DAWZY can write and run scripts to make edits inside a live DAW session. Multimodal Interaction: Users can interact by typing, speaking, or even humming. Multimodal Interaction: Users can interact by typing, speaking, or even humming. User-Friendly UI: The chat interface feels intuitive and responsive even when performing behind-the-scenes audio and AI processing. User-Friendly UI: The chat interface feels intuitive and responsive even when performing behind-the-scenes audio and AI processing.

What we learned

📚 How to integrate multiple AI services together in real time. How to integrate multiple AI services together in real time. The importance of UX when building tools for creative professionals. The importance of UX when building tools for creative professionals. Audio processing and format compatibility challenges across platforms. Audio processing and format compatibility challenges across platforms. Designing systems that adapt LLMs to real-world control tasks beyond text generation. Designing systems that adapt LLMs to real-world control tasks beyond text generation.

What's next

🚀 Full DAW Support: We plan to expand DAWZY beyond Reaper to support other popular DAWs like Ableton, Logic Pro, and FL Studio. Full DAW Support: We plan to expand DAWZY beyond Reaper to support other popular DAWs like Ableton, Logic Pro, and FL Studio. Real-Time MIDI Feedback: Users will be able to edit melodies or chord progressions conversationally while hearing live previews. Real-Time MIDI Feedback: Users will be able to edit melodies or chord progressions conversationally while hearing live previews. On-Device Voice Looping and Jamming: Think of DAWZY as your AI bandmate. On-Device Voice Looping and Jamming: Think of DAWZY as your AI bandmate. Plugin Suggestion and Effect Chain Design: DAWZY will recommend and auto-configure effect chains for genres or moods. Plugin Suggestion and Effect Chain Design: DAWZY will recommend and auto-configure effect chains for genres or moods. Deeper VAPI Integration: Fully enable two-way natural voice conversations. Deeper VAPI Integration: Fully enable two-way natural voice conversations. Our long-term vision is to make DAWZY the ultimate collaborator for music producers—creative, technical, and always ready to jam.

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only
  • CSSIn code
  • FlaskIn code
  • HTMLIn code
  • JavaScriptIn code
  • OpenAIIn code
  • PythonIn code
  • ReactIn code
  • Tailwind CSSIn code

8 of 8 appear in the indexed code.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

187 KB

Source files

25

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

0 stars