# Project export: Sonoverse This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Protecting small artists' work through a decentralized on-chain ML-driven platform, integrating automatic DMCA claims. - Devpost: https://devpost.com/software/sonoverse - GitHub: https://github.com/gashon/treehacks-2024 - Video: https://www.youtube.com/embed/sYLGCz354as?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Taisu Ventures: Creative Web3 Platform Award ($500 Cash)) - Team: 3 GitHub contributor(s) — gashon (55 commits), Ameya Jadhav (34 commits), Varun Sahni (7 commits) ## Devpost submission (written by the team) ### Overview TL; DR Music piracy costs the U.S. economy $12.5 billion annually. Independent artists are the fastest growing segment in the music industry, yet lack the funds and reach to enforce the Digital Millennium Copyright Act (DMCA). We let artists OWN their work (stored on InterPlanetary File System) by tracking it on our own Sonoverse Ethereum L2 chain (powered by Caldera). Artists receive Authenticity Certificates of their work in the form of Non-Fungible Tokens (NFTs), powered by Crossmint’s Minting API. We protect against parodies and remixes with our custom dual-head LSTM neural network model trained from scratch which helps us differentiate these fraudulent works from originals. We proactively query YouTube through their API to constantly find infringing work. We’ve integrated with DMCA Services, LLC. to automate DMCA claim submissions. Interested? Keep reading! ### Inspiration Music piracy, including illegal downloads and streaming, costs the U.S. economy $12.5 billion annually. Independent artists are the fastest growing segment in the music industry, yet lack the funds to enforce DMCA. We asked “Why hasn’t this been solved?” and took our hand at it. Enter Sonoverse, a platform to ensure small musicians can own their own work by automating DMCA detection using deep learning and on-chain technologies. The Issue Is it even possible to automate DMCA reports? How can a complex piece of data like an audio file be meaningfully compared? How do we really know someone OWNS an audio file? and more... These are questions we had too, but by making custom DL models and chain algorithms, we have taken our hand at answering them. What we’ve made We let artists upload their original music to our platform where we store it on decentralized storage (IPFS) and our blockchain to track ownership. We also issue Authenticity Certificates to the original artists in the form of Non-Fungible Tokens. We compare uploaded music with all music on our blockchain to detect if it is a parody, remix, or other fraudulent copy of another original song, using audio processing and an LSTM deep learning model built and trained from scratch. We proactively query YouTube through their API for “similar” music (based on our lyric hashes, frequency analysis, and more) to constantly find infringing work. For detected infringing work, we’ve integrated with DMCA Services, LLC. to automate DMCA claim submissions. ### How we built it All together, we used… NextJS Postgres AWS SES AWS S3 IPFS Caldera Crossmint AssemblyAI Cohere YouTube API DMCA Services It’s a lot, but we were able to split up the work between our team. Gashon built most of the backend routes, an email magic link Auth platform, DB support, and AWS integrations. At the same time, Varun spent his hours collecting hours of audio clips, training and improving the deep LSTM model, and writing several sound differentiation/identification algorithms. Here’s Varun’s explanation of his algorithms: “To detect if a song is a remix, we first used a pre-trained speech to text model to extract lyrics from mp3 files and then analyzed the mel-frequency cepstral coefficients, tempo, melody, and semantics of the lyrics to determine if any songs are very similar. Checking whether a song is a parody is much more nuanced, and we trained a dual-head LSTM neural network model in PyTorch to take in vectorized embeddings of lyrics and output the probability of one of the songs being a parody of the other.” While Varun was doing that, Ameya built out the blockchain services with Caldera and Crossmint, and integrated DMCA Services. Ameya ran a Ethereum L2 chain specific for this project (check it out here) using Caldera. He built out significant infrastructure to upload audio files to IPFS (decentralized storage) and interact with the Caldera chain. He also created the Authenticity Certificate using Crossmint that’s delivered directly to each Sonoverse user’s account. Ameya and Gashon came together at the end to create the Sonoverse frontend, while Varun pivoted to create our YouTube API jobs that query through recently uploaded videos to find infringing content. Challenges we overcame We couldn’t find existing models to detect parodies and had to train a custom model from scratch on training data we had to find ourselves. Of course, this was quite challenging, but with audio files each being unique, we had to create a dataset of hours of audio clips. And, like always, integration was difficult. The power of a team was a huge plus, but also a challenge. Ameya’s blockchain infrastructure had Solidity compilation challenges when porting into Gashon’s platform (which took some precious hours to sort out). Varun’s ML algorithms ran on a Python backend which had to be hosted alongside our NextJS platform. You can imagine what else we had to change and fix and update, so I won’t bore you. Another major challenge was something we brought on ourselves, honestly. We set our aim high so we had to use several different frameworks, services, and technologies to add all the features we wanted. This included several hours of us learning new technologies and services, and figuring out how to implement them in our project. ### Accomplishments we're proud of Blockchain has a lot of cool and real-world applications, but we’re excited to have settled on Sonoverse. We identified a simple (yet technically complex) way to solve a problem that affects many small artists. We also made a sleek web platform, in just a short amount of time, with scalable endpoints and backend services. We also designed and trained a deep learning LSTM model to identify original audios vs fraudulent ones (remixes, speed ups, parodies, etc) that achieved 93% accuracy. ### What we learned About DMCA We learned how existing DMCA processes are implemented and the large capital costs associated with them. We became experts on digital copyrights and media work! Blockchain We learned how to combine centralized and decentralized infrastructure solutions to create a cohesive end-to-end project. ### What's next We're looking forward to incorporating on-chain royalties for small artists by detecting when users consume their music and removing the need for formal contracts with big companies to earn revenue. We’re excited to also add support for more public APIs in addition to YouTube API! ## README (from the GitHub repository) # Sonoverse Protecting small artists' work through a decentralized on-chain ML-driven platform, integrating automatic DMCA claims. ## TL; DR - Music piracy costs the U.S. economy [$12.5 billion annually](https://www.riaa.com/wp-content/uploads/2015/09/20120515_SoundRecordingPiracy.pdf). - Independent artists are the [fastest growing segment in the music industry](https://www.forbes.com/sites/melissamdaniels/2019/07/10/for-independent-musicians-goingyour-own-way-is-finally-starting-to-pay-off/), yet lack the funds and reach to enforce the Digital Millennium Copyright Act (DMCA). - We let artists **OWN** their work (stored on InterPlanetary File System) by tracking it on our own Sonoverse Ethereum L2 chain (powered by Caldera). - Artists receive **Authenticity Certificates** of their work in the form of Non-Fungible Tokens (NFTs), powered by Crossmint’s Minting API. - We protect against parodies and remixes with our **custom dual-head LSTM neural network model** trained from scratch which helps us differentiate these fraudulent works from originals. - We proactively query YouTube through their API to constantly find infringing work. - We’ve integrated with **DMCA Services**, LLC. to automate DMCA claim submissions. Interested? Keep reading! ## Inspiration Music piracy, including illegal downloads and streaming, costs the U.S. economy $12.5 billion annually. Independent artists are the fastest growing segment in the music industry, yet lack the funds to enforce DMCA. We asked “Why hasn’t this been solved?” and took our hand at it. Enter Sonoverse, a platform to ensure small musicians can own their own work by automating DMCA detection using deep learning and on-chain technologies. ## The Issue - Is it even possible to automate DMCA reports? - How can a complex piece of data like an audio file be meaningfully compared? - How do we really know someone OWNS an audio file? - and more... These are questions we had too, but by making custom DL models and chain algorithms, we have taken our hand at answering them. ## What we’ve made We let artists upload their original music to our platform where we store it on decentralized storage (IPFS) and our blockchain to **track ownership**. We also issue Authenticity Certificates to the original artists in the form of Non-Fungible Tokens. We compare uploaded music with all music on our blockchain to **detect** if it is a parody, remix, or other fraudulent copy of another original song, using audio processing and an LSTM deep learning model built and trained from scratch. We proactively query YouTube through their API for “similar” music (based on our **lyric hashes**, **frequency analysis**, and more) to constantly find infringing work. For detected infringing work, we’ve integrated with DMCA Services, LLC. to **automate DMCA claim submissions**. ## How we built it All together, we used… - NextJS - Postgres - AWS SES - AWS S3 - IPFS - Caldera - Crossmint - AssemblyAI - Cohere - YouTube API - DMCA Services It’s a **lot**, but we were able to split up the work between our team. Gashon built most of the backend routes, an email magic link Auth platform, DB support, and AWS integrations. At the same time, Varun spent his hours collecting hours of audio clips, training and improving the deep LSTM model, and writing several sound differentiation/identification algorithms. Here’s Varun’s **explanation** of his algorithms: “To detect if a song is a remix, we first used a pre-trained speech to text model to extract lyrics from mp3 files and then analyzed the mel-frequency cepstral coefficients, tempo, melody, and semantics of the lyrics to determine if any songs are very similar. Checking whether a song is a parody is much more nuanced, and we trained a dual-head LSTM neural network model in PyTorch to take in vectorized embeddings of lyrics and output the probability of one of the songs being a parody of the other.” While Varun was doing that, Ameya built out the blockchain services with Caldera and Crossmint, and integrated DMCA Services. Ameya ran a Ethereum L2 chain specific for this project (check it out [here](https://treehacks-2024.explorer.caldera.xyz)) using Caldera. He built out significant infrastructure to upload audio files to IPFS (decentralized storage) and interact with the Caldera chain. He also created the Authenticity Certificate using Crossmint that’s delivered directly to each Sonoverse user’s account. Ameya and Gashon came together at the end to create the Sonoverse frontend, while Varun pivoted to create our YouTube API jobs that query through recently uploaded videos to find infringing content. ## Challenges we overcame We couldn’t find existing models to detect parodies and had to train a custom model from scratch on training data we had to find ourselves. Of course, this was quite challenging, but with audio files each being unique, we had to create a dataset of hours of audio clips. And, like always, integration was difficult. The power of a team was a huge plus, but also a challenge. Ameya’s blockchain infrastructure had Solidity compilation challenges when porting into Gashon’s platform (which took some precious hours to sort out). Varun’s ML algorithms ran on a Python backend which had to be hosted alongside our NextJS platform. You can imagine what else we had to change and fix and update, so I won’t bore you. Another major challenge was something we brought on ourselves, honestly. We set our aim high so we had to use several different frameworks, services, and technologies to add all the features we wanted. This included several hours of us learning new technologies and services, and figuring out how to implement them in our project. ## Accomplishments that we're proud of Blockchain has a lot of cool and real-world applications, but we’re excited to have settled on Sonoverse. We identified a simple (yet technically complex) way to solve a problem that affects many small artists. We also made a sleek web platform, in just a short amount of time, with scalable endpoints and backend services. We also designed and trained a deep learning LSTM model to identify original audios vs fraudulent ones (remixes, speed ups, parodies, etc) that achieved **93% accuracy**. ## What we learned ### About DMCA We learned how existing DMCA processes are implemented and the large capital costs associated with them. We became **experts** on digital copyrights and media work! ### Blockchain We learned how to combine centralized and decentralized infrastructure solutions to create a cohesive **end-to-end** project. ## What's next for Sonoverse We're looking forward to incorporating on-chain **royalties** for small artists by detecting when users consume their music and removing the need for formal contracts with big companies to earn revenue. We’re excited to also add support for more public APIs in addition to YouTube API! ## Detected evidence (automated analysis) Indexed codebase: 67 recognized source files, 313 KB. - Cohere (technology) — detected in the code - CSS (language) — detected in the code - Flask (technology) — detected in the code - JavaScript (language) — detected in the code - MongoDB (technology) — detected in the code - Next.js (technology) — detected in the code - PostgreSQL (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code - Solidity (language) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (85 of 85) ``` client/.env.example client/.eslintrc.json client/.gitignore client/next.config.mjs client/package.json client/postcss.config.js client/README.md client/src/components/perlin.tsx client/src/consts/index.ts client/src/db/index.ts client/src/db/lib/index.ts client/src/db/lib/kysely.ts client/src/db/migrations/logs/001-create-user-table.ts client/src/db/migrations/logs/002-create-song-table.ts client/src/db/migrations/migrator.ts client/src/db/migrations/scripts/drop-all.ts client/src/db/migrations/scripts/to-latest.ts client/src/db/migrations/scripts/util.ts client/src/db/types/generated.types.ts client/src/db/types/index.ts client/src/features/auth/api/index.ts client/src/features/auth/api/login-email.ts client/src/features/auth/api/register.ts client/src/features/auth/index.ts client/src/features/song/api/chain.ts client/src/features/song/api/copyright.ts client/src/features/song/api/dmca.ts client/src/features/song/api/get-songs.ts client/src/features/song/api/index.ts client/src/features/song/api/mint.ts client/src/features/song/api/presigned.ts client/src/features/song/api/upload.ts client/src/features/song/index.ts client/src/lib/aws-sdk.ts client/src/lib/jwt.ts client/src/lib/react-query.ts client/src/lib/s3.ts client/src/pages/_app.tsx client/src/pages/_document.tsx client/src/pages/api/auth/magic.ts client/src/pages/api/auth/register.ts client/src/pages/api/hello.ts client/src/pages/api/s3/chain.ts client/src/pages/api/s3/dmca.ts client/src/pages/api/s3/mint.ts client/src/pages/api/s3/presigned/index.ts client/src/pages/api/songs/compare/index.ts client/src/pages/api/songs/index.ts client/src/pages/api/strikes/index.ts client/src/pages/auth/login.tsx client/src/pages/auth/register.tsx client/src/pages/dashboard.tsx client/src/pages/index.tsx client/src/styles/globals.css client/src/templates/index.ts client/src/types/index.ts client/src/util/send-email.ts client/src/util/verify-original.ts client/src/utils/blockchain/AudioContract.json client/src/utils/blockchain/complied_contracts/contracts_audio_sol_AudioContract.abi client/src/utils/blockchain/contracts/AudioContract.sol client/src/utils/blockchain/contracts/AudioContractABI.json client/src/utils/blockchain/contracts/AudioContractBytecode.json client/src/utils/blockchain/hardhat.config.js client/src/utils/blockchain/nft/IPFS_upload.js client/src/utils/blockchain/nft/mint.js client/src/utils/blockchain/scripts/get_metadata.js client/src/utils/blockchain/scripts/new_contract_deploy.js client/src/utils/blockchain/secrets.json client/src/utils/dmca/create_case.js client/tailwind.config.ts client/tsconfig.json ml/.DS_Store ml/.gitignore ml/lyric_extraction.py ml/main.py ml/mod_match.py ml/model.pth ml/og_false.txt ml/original.txt ml/parody_false.txt ml/parody_match.py ml/parody.txt ml/requirements.txt README.md ``` ### Dependencies - client/package.json: @aws-sdk/client-s3@^3.515.0, @aws-sdk/s3-request-presigner@^3.515.0, @nomiclabs/hardhat-ethers@^2.2.3, @tanstack/react-query@^5.20.5, @types/jsonwebtoken@^9.0.5, @types/node@^20, @types/p5@^1.7.6, @types/pg@^8.11.0, @types/react@^18, @types/react-dom@^18, autoprefixer@^10.0.1, aws-sdk@^2.1560.0, axios@^1.6.7, concurrently@^8.2.2, dotenv@^16.4.4, esbuild@^0.20.0, esbuild-runner@^2.2.2, eslint@^8, eslint-config-next@14.1.0, ethers@5.4, fs@^0.0.1-security, hardhat@^2.20.1, jsonwebtoken@^9.0.2, kysely@^0.27.2, kysely-codegen@^0.11.0, mongoose@^8.1.3, next@14.1.0, node-fetch@^3.3.2, p5@^1.9.0, pg@^8.11.3, postcss@^8, react@^18, react-audio-visualize@^1.1.3, react-dom@^18, react-dropzone@^14.2.3, react-icons@^5.0.1, react-p5@^1.4.1, react-type-animation@^3.2.0, solc@^0.7.3, sonner@^1.4.0, tailwindcss@^3.3.0, typescript@^5, wavesurfer-react@^3.0.2 - ml/requirements.txt: aiohttp@==3.9.1, aiosignal@==1.3.1, annotated-types@==0.6.0, async-timeout@==4.0.3, attrs@==23.2.0, audioread@==3.0.1, autofaiss@==2.17.0, backoff@==2.2.1, blinker@==1.7.0, blis@==0.7.11, catalogue@==2.0.10, certifi@==2023.11.17, cffi@==1.16.0, charset-normalizer@==3.3.2, click@==8.1.7, cloudpathlib@==0.16.0, cohere@==4.45, confection@==0.1.4, coverage@==5.5, cymem@==2.0.8, decorator@==5.1.1, deprecation@==2.1.0, distlib@==0.3.8, embedding-reader@==1.7.0, en-core-web-sm@==3.7.1, eyed3@==0.9.7, faiss-cpu@==1.7.4, fastavro@==1.9.3, filelock@==3.13.1, filetype@==1.2.0, fire@==0.5.0, flask@==3.0.1, Flask-Cors@==4.0.0, frozenlist@==1.4.1, fsspec@==2023.12.2, idna@==3.6, importlib-metadata@==6.11.0, itsdangerous@==2.1.2, Jinja2@==3.1.3, joblib@==1.3.2, langcodes@==3.3.0, lazy-loader@==0.3, librosa@==0.10.1, llvmlite@==0.41.1, MarkupSafe@==2.1.4, mpmath@==1.3.0, msgpack@==1.0.7, multidict@==6.0.4, murmurhash@==1.0.10, networkx@==3.1, numba@==0.58.1, numpy@==1.24.4, packaging@==23.2, pandas@==2.0.3, platformdirs@==4.1.0, pooch@==1.8.0, preshed@==3.0.9, pyarrow@==15.0.0, pycparser@==2.21, pydantic@==2.6.1, pydantic-core@==2.16.2, python-dateutil@==2.8.2, pytz@==2023.3.post1, requests@==2.31.0, scikit-learn@==1.3.2, scipy@==1.10.1, six@==1.16.0, smart-open@==6.4.0, soundfile@==0.12.1, soxr@==0.3.7, spacy@==3.7.4, spacy-legacy@==3.0.12, spacy-loggers@==1.0.5, srsly@==2.4.8, sympy@==1.12, termcolor@==2.4.0, thinc@==8.2.3, threadpoolctl@==3.3.0, toml@==0.10.2, torch@==2.2.0, tqdm@==4.66.1, typer@==0.9.0, typing-extensions@==4.9.0, tzdata@==2023.4, urllib3@==2.1.0, virtualenv@==20.25.0, wasabi@==1.1.2, weasel@==0.3.4, werkzeug@==3.0.1, yarl@==1.9.4, zipp@==3.17.0 ### Recent commits (newest first) - chore: fix ts errors - fix: p5 ts errors - chore: add p5 types - Update README.md - Update README.md - Update README.md - Create README.md - feat: get strikes over net req - chore: clean files - feat: add timestamp to display - fix: opacity on blur - fix: add node-fetch - nft fixes - feat: modal uploading state - feat: modal uploading state - feat: mintNFT route - feat: collision notifications - feat: modal styling - fix: mg conflicts - updates ## Key source files (fetched from GitHub, selected and truncated for size) ### ml/requirements.txt ``` aiohttp==3.9.1 aiosignal==1.3.1 annotated-types==0.6.0 async-timeout==4.0.3 attrs==23.2.0 audioread==3.0.1 autofaiss==2.17.0 backoff==2.2.1 blinker==1.7.0 blis==0.7.11 catalogue==2.0.10 certifi==2023.11.17 cffi==1.16.0 charset-normalizer==3.3.2 click==8.1.7 cloudpathlib==0.16.0 cohere==4.45 confection==0.1.4 coverage==5.5 cymem==2.0.8 decorator==5.1.1 deprecation==2.1.0 distlib==0.3.8 embedding-reader==1.7.0 en-core-web-sm==3.7.1 eyed3==0.9.7 faiss-cpu==1.7.4 fastavro==1.9.3 filelock==3.13.1 filetype==1.2.0 fire==0.5.0 flask==3.0.1 Flask-Cors==4.0.0 frozenlist==1.4.1 fsspec==2023.12.2 idna==3.6 importlib-metadata==6.11.0 itsdangerous==2.1.2 Jinja2==3.1.3 joblib==1.3.2 langcodes==3.3.0 lazy-loader==0.3 librosa==0.10.1 llvmlite==0.41.1 MarkupSafe==2.1.4 mpmath==1.3.0 msgpack==1.0.7 multidict==6.0.4 murmurhash==1.0.10 networkx==3.1 numba==0.58.1 numpy==1.24.4 packaging==23.2 pandas==2.0.3 platformdirs==4.1.0 pooch==1.8.0 preshed==3.0.9 pyarrow==15.0.0 pycparser==2.21 pydantic==2.6.1 pydantic-core==2.16.2 python-dateutil==2.8.2 pytz==2023.3.post1 requests==2.31.0 scikit-learn==1.3.2 scipy==1.10.1 six==1.16.0 smart-open==6.4.0 soundfile==0.12.1 soxr==0.3.7 spacy==3.7.4 spacy-legacy==3.0.12 spacy-loggers==1.0.5 srsly==2.4.8 sympy==1.12 termcolor==2.4.0 thinc==8.2.3 threadpoolctl==3.3.0 toml==0.10.2 torch==2.2.0 tqdm==4.66.1 typer==0.9.0 typing-extensions==4.9.0 tzdata==2023.4 urllib3==2.1.0 virtualenv==20.25.0 wasabi==1.1.2 weasel==0.3.4 werkzeug==3.0.1 yarl==1.9.4 zipp==3.17.0 ``` ### client/package.json ``` { "name": "treehacks", "version": "0.1.0", "private": true, "scripts": { "dev": "concurrently \"next dev\" \"pnpm run db:start\" ", "build": "next build", "start": "next start", "lint": "next lint", "format:check": "prettier --check .", "format:write": "prettier --write .", "db:start": "docker run --rm -v data:/var/lib/postgresql/data --name treehacks-db -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=treehacks_db -e POSTGRES_USER=postgres -p 5431:5432 postgres", "db:stop": "docker stop treehacks-db && docker rm treehacks-db", "db:reset": "docker exec -it treehacks-db psql -U postgres -c 'DROP DATABASE IF EXISTS treehacks_db'", "db:migrate": "node -r esbuild-runner/register src/db/migrations/scripts/to-latest.ts && pnpm db:types:generate", "db:migrate:undo:all": "node -r esbuild-runner/register src/db/migrations/scripts/drop-all.ts && pnpm db:types:generate", "db:types:generate": "kysely-codegen --dialect postgres --camel-case --out-file src/db/types/generated.types.ts", "lhci:mobile": "lhci autorun", "lhci:desktop": "lhci autorun --collect.settings.preset=desktop" }, "dependencies": { "@aws-sdk/client-s3": "^3.515.0", "@aws-sdk/s3-request-presigner": "^3.515.0", "@nomiclabs/hardhat-ethers": "^2.2.3", "@tanstack/react-query": "^5.20.5", "@types/p5": "^1.7.6", "aws-sdk": "^2.1560.0", "axios": "^1.6.7", "dotenv": "^16.4.4", "ethers": "5.4", "fs": "^0.0.1-security", "jsonwebtoken": "^9.0.2", "kysely": "^0.27.2", "mongoose": "^8.1.3", "next": "14.1.0", "node-fetch": "^3.3.2", "p5": "^1.9.0", "pg": "^8.11.3", "react": "^18", "react-audio-visualize": "^1.1.3", "react-dom": "^18", "react-dropzone": "^14.2.3", "react-icons": "^5.0.1", "react-p5": "^1.4.1", "react-type-animation": "^3.2.0", "solc": "^0.7.3", "sonner": "^1.4.0", "wavesurfer-react": "^3.0.2" }, "devDependencies": { "@types/jsonwebtoken": "^9.0.5", "@types/node": "^20", "@types/pg": "^8.11.0", "@types/react": "^18", "@types/react-dom": "^18", "autoprefixer": "^10.0.1", "concurrently": "^8.2.2", "esbuild": "^0.20.0", "esbuild-runner": "^2.2.2", "eslint": "^8", "eslint-config-next": "14.1.0", "hardhat": "^2.20.1", "kysely-codegen": "^0.11.0", "postcss": "^8", "tailwindcss": "^3.3.0", "typescript": "^5" } } ``` ### ml/main.py ```python from flask_cors import CORS, cross_origin from flask import Flask, flash, redirect, request, session, url_for, jsonify from mod_match import final_similarity, load_audio_features from parody_match import predict, ParodyLSTM from lyric_extraction import transcribe_audio import requests app = Flask(__name__) CORS(app) FILE_PATH1 = 'local1.mp3' FILE_PATH2 = 'local2.mp3' FILE_PATH3 = 'local3.mp3' @app.route('/ml/', methods=['POST']) def entry(): url1 = request.get_json()['url1'] url2 = request.get_json()['url2'] response1 = requests.get(url1) response2 = requests.get(url2) if response1.status_code == 200 == response2.status_code: # Write the audio content to a local file with open(FILE_PATH1, 'wb') as audio_file: audio_file.write(response1.content) print(f"Audio file downloaded and saved as {FILE_PATH1}") with open(FILE_PATH2, 'wb') as audio_file: audio_file.write(response2.content) print(f"Audio file downloaded and saved as {FILE_PATH2}") else: print(f"Failed to download audio file. Status code: {response.status_code}") return lyrics1 = transcribe_audio(FILE_PATH1) lyrics2 = transcribe_audio(FILE_PATH2) features1 = load_audio_features(FILE_PATH1, lyrics1) features2 = load_audio_features(FILE_PATH2, lyrics2) similarity_mod = final_similarity(features1, features2) similarity_parody = predict(lyrics1, lyrics2).item() with open('new_songs.txt', 'a') as file: file.write(lyrics1 + '\n---\n`') file.write(lyrics2) return jsonify( { 'similarity_parody': similarity_parody, 'similarity_mod': similarity_mod } ) @app.route('/dmca/', methods=['POST']) def enforce(): url = request.get_json()['url'] response = requests.get(url) if response.status_code == 200: # Write the audio content to a local file with open(FILE_PATH3, 'wb') as audio_file: audio_file.write(response.content) print(f"Audio file downloaded and saved as {FILE_PATH3}") return check_youtube(FILE_PATH3) if __name__ == '__main__': # Threaded option to enable multiple instances for multiple user access support app.run(threaded=True, port=5000) ``` ### client/src/db/index.ts ```typescript export * from './lib'; ``` ### client/src/templates/index.ts ```typescript import { EmailTemplate } from "@/types"; import { senderEmail } from "@/lib/aws-sdk"; export const loginTemplate: EmailTemplate = { from: senderEmail, subject: "Login to ChainGuard", content: `
Your login link for ChainGuard

Continue to ChainGuard

This link and code will only be valid for the next 5 minutes. If the link does not work, contact support at @ ChainGuard.
`, args: { login_link: undefined, }, }; ``` ### client/src/consts/index.ts ```typescript export const AUTH_COOKIE = "tree-auth-token"; export const YOUTUBE_LINKS = [ "https://www.youtube.com/watch?v=R7-ApK63X_U&pp=ygUXaGFwcHkgY2xhcCBhbG9uZyBwYXJvZHk%3D", "https://www.youtube.com/watch?v=dM12YTZvQGw&pp=ygUXaGFwcHkgY2xhcCBhbG9uZyBwYXJvZHk%3D", "https://www.youtube.com/watch?v=Go_p6oD7AIE&pp=ygUWaGFwcHkgY2xhcCBhbG9uZyByZW1peA%3D%3D", "https://www.youtube.com/watch?v=Sj0DF0rRF18&pp=ygUWaGFwcHkgY2xhcCBhbG9uZyByZW1peA%3D%3D", "https://www.youtube.com/watch?v=BbWGlDi7M7M&pp=ygUWaGFwcHkgY2xhcCBhbG9uZyByZW1peA%3D%3D", "https://www.youtube.com/watch?v=SqJGCnfr7rc&pp=ygUWaGFwcHkgY2xhcCBhbG9uZyByZW1peA%3D%3D", ]; export const COLLIDING_PARODY = "happy_parody.mp3"; export const COLLIDING_ALTERATION = "happy_speed_up.mp3"; ``` ### client/src/types/index.ts ```typescript import { Song } from "@/db/types"; export type AuthToken = { email: string; created_at: number; user_id: string; }; export type RegistrationToken = { email: string; created_at: number; }; export type LoginPostRequest = { email: string; }; export type RegisterPostRequest = { token: string; }; export type ChainPostRequest = { s3_key: string; file_name: string; file_type: string; }; export type DMCAClaimRequest = { song_id: string; file_name: string; audio_link: string; }; export type MintPostRequest = { file_name: string; }; export type EmailTemplate = { from: string; subject: string; content: string; args?: Record; }; export type S3PresignedGetRequest = { file_name: string; file_type: string; readonly?: boolean; }; type Response = { data: T; }; export type GetPresignedUrlResponse = Response<{ url: string; key: string; }>; export type PutChainResponse = | Response<{ song: Song; }> | { message: string }; export type SongsGetResponse = Response<{ songs: Pick< Song, "s3Key" | "chainAddress" | "id" | "fileName" | "createdAt" >[]; }>; ``` ### client/src/pages/index.tsx ```typescript import React, { useState } from 'react'; // Import useState import dynamic from 'next/dynamic'; import { Inter } from 'next/font/google'; import { TypeAnimation } from 'react-type-animation'; import Head from 'next/head'; const inter = Inter({ subsets: ['latin'] }); const PerlinSketchNoSSR = dynamic(() => import('@/components/perlin'), { ssr: false, }); // Updated Simple Modal Component for larger size const Modal = ({ isOpen, onClose, children }) => { if (!isOpen) return null; return ( // Increased z-index for modal overlay to ensure it's above everything
{' '} {/* Adjusted size */} {children}
); }; export default function Home() { const [isModalOpen, setIsModalOpen] = useState(false); // State to control modal visibility const handleOpenModal = () => setIsModalOpen(true); const handleCloseModal = () => setIsModalOpen(false); return ( <>
Sonoverse
{/* Button container with increased padding between buttons */}
{/* Modal with adjusted size */}

Sonoverse

We operate our own custom Sonoverse Ethereum L2 chain{' '} to track ownership of content our artists upload, with the help of Caldera.

We use our own custom LSTM deep learning model to proactively detect parodies and fraudulent remixes.

); } ``` ### client/src/features/auth/index.ts ```typescript export * from "./api"; ``` ### client/src/features/song/index.ts ```typescript export * from "./api"; ``` [55 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]