Project Info

SongSense

This project did not submit a demo video on Devpost.

Inspiration

Apps like Shazam are great at one thing: identifying a song you're currently hearing. But that's not how most "what's that song" moments actually happen. More often, you're left with a fragment — a half-remembered lyric, a mood, a genre, the way a melody felt — and no audio to feed into a fingerprinting algorithm. There was no good tool for that gap: searching for a song by description rather than by recording. SongSense was built to fill it.

What it does

SongSense lets users search for a song using any combination of four loose, fragmentary inputs — a hummed/recorded melody, a genre, a lyric snippet, or free-text mood/description — and returns the top 5 closest matches. None of the inputs need to be exact or complete; the whole point is supporting the "I don't really remember, but it felt like..." search.

How we built it

Frontend: React Native (Expo SDK 54), with three tabs (Search, History, Saved) built using React Navigation, running cross-platform on iOS, Android, and web. Backend: FastAPI serving a local REST API on port 8000, backed by SQLite (a songs catalog, search_history, and saved_songs tables). AI matching: every song is flattened into one descriptive string (title, artist, genre, mood keywords, lyric snippet, description) and embedded with all-MiniLM-L6-v2 (sentence-transformers) at startup. A user's query is embedded the same way, and ranked against all song vectors by cosine similarity: $$\text{sim}(q, s) = \frac{q \cdot s}{|q| |s|}$$ Since every embedding is pre-normalized to unit length, this reduces to a single dot product, so the whole catalog can be ranked with one matrix-vector multiply (song_matrix @ q) instead of a per-song loop. Audio: 30-second previews are pulled from Wikimedia Commons (public domain, for classical pieces) and the iTunes Search API (for modern songs), played inline via expo-av.

Challenges we ran into

The hardest part wasn't the plumbing — it was getting the matching itself to feel right. A few specific issues: Field weighting. Concatenating title, artist, genre, mood, lyric, and description into one string means a strong match on, say, mood keywords can get diluted by irrelevant noise elsewhere in the string. Tuning what goes into the descriptive text (and how it's phrased) mattered more than expected. Score interpretability. Cosine similarity scores don't have an obvious "good match" threshold — a 0.45 might be a strong match for one query and a weak one for another, so deciding how to rank/display confidence took iteration. Designing for a second signal in advance. Knowing that hum-based melody matching was coming in Phase 2, the matching engine had to be architected to blend two independent similarity scores (text + audio) via a weighted sum, without knowing yet what scale or distribution the audio scores would have. That meant building the normalization and blending logic defensively, ahead of having real data to test it against.

What we learned

*How to turn a fuzzy, human "vibe" query into something a vector space can actually rank. That embedding-based search needs content design, not just model selection — what you embed matters as much as which model embeds it. How to architect a multi-signal ranking system (text now, audio later) so a new scoring signal can be dropped in without reworking the pipeline.

What's next

Implementing real melody/hum matching to replace the score_hum() stub, tuning the text/hum blend weights once both signals are live, migrating the databases to more formal database instead of sqlite, and expanding the song database.

SongSense

A Shazam-style mobile app that finds songs from a hummed tune, lyrics, genre, or freeform keywords — matched against a 334-song catalog using AI semantic search.

Frontend: Expo / React Native — runs on iOS, Android, and web
Backend: FastAPI + SQLite
Matching: sentence-transformers (all-MiniLM-L6-v2) text embeddings with cosine similarity
Search labels: auto-generated using semantic vibe/era matching (e.g. "Romantic Jazz", "Energetic 80s Pop")
Audio previews: 30-second clips via iTunes (modern) and Wikimedia Commons (classical)
Album artwork: fetched from iTunes / Deezer APIs

Project layout

backend/
  main.py            FastAPI app & endpoints
  db.py              SQLite schema, seeding, history/saved persistence
  matcher.py         embedding-based ranking + search label generation
  songs_seed.py      original 20-song seed catalog
  expand_db.py       script that expanded catalog to 334 songs + iTunes previews
  fetch_artwork.py   script that backfilled album artwork URLs
  requirements.txt
  songfinder.db      SQLite database (gitignored)

frontend/
  App.js             tab navigator (Search · History · Saved)
  screens/           SearchScreen, HistoryScreen, SavedScreen
  components/        ResultRow (artwork + play overlay + bookmark) · SavedCard
  lib/
    api.js           backend HTTP client
    useAudioPlayer.js  shared audio playback hook (expo-av)
  app.json           Expo config (iOS · Android · Web)
  package.json

Song catalog

334 songs across pop, rock, hip-hop, R&B, electronic, jazz, classical, country, Latin, K-pop, and metal. Previews come from two sources:

Type	Preview source	Artwork source
Classical (public domain)	Wikimedia Commons MP3 transcodes	iTunes / Deezer
Modern (copyrighted)	iTunes 30-second preview clips	iTunes / Deezer

Running the backend

cd backend
/opt/anaconda3/bin/python3 -m uvicorn main:app --reload --host 0.0.0.0 --port 8000

First launch downloads the embedding model (~80 MB), runs the DB migration, and pre-computes embeddings for all 334 songs. API available at http://localhost:8000 — interactive docs at http://localhost:8000/docs.

Running the frontend

cd frontend
npx expo start

Key	Action
`i`	Open in iOS Simulator
`a`	Open in Android Emulator
`w`	Open in browser
Scan QR	Open in Expo Go on your phone

Physical device: lib/api.js sets BASE_URL to your Mac's LAN IP. Update it if your IP changes, and make sure both devices are on the same Wi-Fi.

How search works

Each song is converted to a text string: title · artist · genres · mood keywords · lyric snippet · description
All strings are embedded with all-MiniLM-L6-v2 at startup and stored as a normalized matrix
The user's query (genre + lyric + extra) is embedded the same way
Cosine similarity is computed against all songs; top 5 are returned
The search history label is generated by matching the query against vibe and era descriptor embeddings (e.g. "melancholic", "80s") and combining with the genre

API endpoints

Method	Path	Purpose
POST	`/search`	Run a search (multipart: genre, lyric, extra, optional audio), store in history, return top 5
GET	`/history`	List past searches
GET	`/history/{id}`	Re-fetch a past search's top 5 results
GET	`/saved`	List saved songs
POST	`/saved/{id}`	Bookmark a song
DELETE	`/saved/{id}`	Remove a bookmark
GET	`/songs`	Full catalog (debug)

Phase 2 — query by humming

The microphone input is wired up but melody matching is stubbed. Plan:

Reference melodies — add a melody_contour (normalized pitch sequence) to each song, extracted from MIDI or hand-annotated
Hum → contour — extract pitch curve from uploaded audio with librosa or CREPE, normalize for key/tempo
Compare — use DTW (dynamic time warping) to score hum against each contour
Blend — implement score_hum() in matcher.py and raise w_hum in the weighted blend

Analysis

Compare with all teams

View

Metric

Ian Liu
4

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

FastAPIIn code
JavaScriptIn code
PythonIn code
ReactIn code

4 of 4 appear in the indexed code.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

136 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Feature verification

8 Verified4 Code-supported1 Claimed only

AI semantic matching via sentence-transformers embeddings + cosine similarityVerified
Every song is embedded with all-MiniLM-L6-v2 and ranked by cosine similarity via a single matrix-vector multiply
Claimed on Devposthigh confidence
- backend/matcher.py:61 — SentenceTransformer('all-MiniLM-L6-v2') loads model and encodes song texts
- backend/matcher.py:83 — sims = _song_matrix @ q computes cosine similarity via single dot product since vectors are normalized
Auto-generated search labels via semantic vibe/era matchingVerified
Search labels auto-generated using semantic vibe/era matching (e.g. 'Romantic Jazz', 'Energetic 80s Pop')
Claimed on readmehigh confidence
- backend/matcher.py:88 — generate_label() embeds the query and picks best-matching vibe and era descriptors from fixed lists, combined with genre
- backend/main.py:64 — matcher.generate_label() is called and stored as the history label on every search
FastAPI backend on port 8000 with SQLite (songs, search_history, saved_songs)Verified
FastAPI serving a local REST API on port 8000, backed by SQLite with a songs catalog, search_history, and saved_songs tables
Claimed on Devposthigh confidence
- backend/main.py:25 — FastAPI app defined; docstring states uvicorn run on port 8000
- backend/db.py:31 — CREATE TABLE songs, search_history, saved_songs all present
Multi-input fuzzy song search (hum/genre/lyric/mood text)Verified
Search by any combination of hummed melody, genre, lyric snippet, or free-text mood/description, returns top 5 matches
Claimed on Devposthigh confidence
- backend/matcher.py:122 — search() combines genre/lyric/extra text embedding and optional audio, returns top_k ranked songs
- frontend/screens/SearchScreen.js:46 — runSearch collects genre, lyric, extra, and audioUri and posts to /search
- backend/main.py:44 — POST /search endpoint accepts genre, lyric, extra, and optional audio file
Multi-signal ranking architecture designed to blend text + future hum scoreVerified
Matching engine architected to blend two independent similarity scores (text + audio) via a weighted sum, ready for Phase 2 hum matching
Claimed on Devposthigh confidence
- backend/matcher.py:122 — search() computes w_text * text_scores + w_hum * hum_scores with w_hum currently 0.0, matching the described defensive blending design
React Native (Expo) frontend with Search / History / Saved tabsVerified
Frontend built with React Native (Expo SDK 54) and React Navigation, three tabs (Search, History, Saved), cross-platform iOS/Android/web
Claimed on Devposthigh confidence
- frontend/App.js:12 — createBottomTabNavigator with Search, History, Saved tab screens
- frontend/screens/SearchScreen.js:1 — SearchScreen implemented with React Native components
Saved songs (bookmarking)Verified
Users can bookmark and remove songs (Saved tab, POST/DELETE /saved/{id})
Claimed on readmehigh confidence
- backend/main.py:93 — POST /saved/{song_id} and DELETE /saved/{song_id} endpoints implemented, backed by db.save_song/unsave_song
- frontend/screens/SavedScreen.js:20 — remove() calls unsaveSong and updates local state; SavedCard renders bookmarked songs
Search history: list past searches and re-fetch a past search's resultsVerified
GET /history and GET /history/{id} let users list and re-view past searches
Claimed on readmehigh confidence
- backend/main.py:72 — GET /history and GET /history/{history_id} endpoints implemented
- frontend/screens/HistoryScreen.js:41 — open() calls getHistoryResults and displays the re-fetched top 5
30-second audio previews via iTunes and Wikimedia CommonsCode-supported
30-second previews pulled from Wikimedia Commons (public domain, classical) and the iTunes Search API (modern songs), played inline via expo-av
Claimed on Devpostmedium confidence
- backend/expand_db.py:363 — fetch_itunes_preview() fetches 30s clips from iTunes Search API for all new songs including classical ones
- frontend/lib/useAudioPlayer.js:31 — expo-av Audio.Sound.createAsync plays song.preview_url inline
- backend/songs_seed.py:23 — Original classical seed entries use a 'PD_AUDIO' placeholder string, not a real Wikimedia URL, and no code anywhere in the repo fetches from Wikimedia Commons (grep for 'wikimedia' across backend/frontend found no matches outside README.md)
334-song catalogCode-supported
Song matching is done against a 334-song catalog spanning many genres
Claimed on readmemedium confidence
- backend/songs_seed.py:1 — 20-song original seed catalog
- backend/expand_db.py:14 — NEW_SONGS list adds roughly 320 more song tuples across many genres, close to but not exactly verified as 334 (INSERT OR IGNORE could skip duplicates); exact count could not be confirmed without running the DB
Album artwork from iTunes / Deezer APIsCode-supported
Album artwork fetched from iTunes / Deezer APIs
Claimed on readmemedium confidence
- backend/fetch_artwork.py:14 — fetch_artwork() only queries api.deezer.com; no iTunes artwork lookup exists in this file or elsewhere in backend
- frontend/components/SongComponents.js:71 — song.artwork_url is rendered as an Image in result rows
Hum/record melody input (audio capture)Code-supported
Users can hum a few notes / record a melody as one of the search inputs
Claimed on Devposthigh confidence
- frontend/screens/SearchScreen.js:26 — toggleRecord uses Audio.Recording to capture a hum and attach it as audioUri
- backend/matcher.py:117 — score_hum() is an explicit stub that returns [] so recorded audio never actually contributes to the match, only text signals do
Real melody/hum matching (score_hum implementation)Claimed only
Implementing real melody/hum matching to replace the score_hum() stub is listed under What's Next, i.e. explicitly not yet built
Claimed on Devposthigh confidence

An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.

Project Info

SongSense

Inspiration

What it does

How we built it

Challenges we ran into

What we learned

What's next

SongSense

Project layout

Song catalog

Running the backend

Running the frontend

How search works

API endpoints

Phase 2 — query by humming

Analysis

Technology

AI coding agents

Codebase size

Structure

Interface

Application logic

Languages

Dependencies

Feature verification