HackStack

Project Info

PLAI

Devpost

This project did not submit a demo video on Devpost.

Inspiration

There are many songs out there that do not have dedicated sheet music, and we wanted to create a tool that could let us play any song on the Internet on the piano.

What it does

We are using AI to transcribe music. It converts audio files in mp3, wav, ogg, flac, or m4a format into sheet music by using Google's Gemini 2.5 Pro AI model.

How we built it

PLAI was built using with TypeScript, but we also integrated Google's Gemini 2.5 Pro AI model for audio analysis. We implemented Web Audio API for file processing, and used VexFlow to convert AI-detected notes into sheet music notation. The development involved establishing the foundation, implementing audio processing with drag-and-drop functionality, integrating AI capabilities with error handling, building sheet music generation, and polishing the user experience with custom branding.

Challenges we ran into

It was difficult getting the notes to look like the sheet music we saw online, and it was even more difficult getting the notes to be actually accurate to the ones being played on the mp3. We had a hard time getting the Gemini API model that we wanted.

Accomplishments we're proud of

We managed to figure out which API works the best for generating sheet music that is easy to read and is also accurate to the song that was put in.

What we learned

We learned that there are a lot of challenges when it comes to analyzing audio files, as not many API's are trained on how to do more complex analyzing tasks yet. We learned how to transform complex audio analysis into an intuitive music transcription tool.

What's next

We are planning on expanding what instruments PLAI covers and the length of the sheet music. We would ideally like to expand from snippets of music to covering the entire song, and we would like to improve the complexity of songs that it can analyze. We want PLAI to be able to do a variety of different time signatures, lengths of notes, and clefs. We also want to implement Essentia, which is an open-source C++ library for audio analysis, to make our audio processing even more accurate.

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only
  • CSSIn code
  • Google GeminiIn code
  • Next.jsIn code
  • ReactIn code
  • Tailwind CSSIn code
  • TypeScriptIn code

6 of 6 appear in the indexed code.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

243 KB

Source files

68

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

0 stars