# Project export: NonoSubtitles This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: OpenAI Build Week - Tagline: NonoSubtitles creates live subtitles for local videos and application audio, letting people enjoy foreign media with language learning lessons created by an AI teacher built in! - Devpost: https://devpost.com/software/nonoprojectcoordinator - GitHub: https://github.com/nikokuma/NonoSub - Video: https://www.youtube.com/embed/_FySMBY3sks?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — nikokuma (65 commits), Claude Fable 5 (6 commits), Codex gpt-5.6-sol (1 commits) ## Devpost submission (written by the team) ### Inspiration I love Japanese media and am actively learning the language, but many of the online YouTube videos, livestreams and shows I want to watch don't have subtitles in languages other than Japanese. So when I watch these videos I can only understand about half of what is being said since I'm still learning the language. Sometimes it's great practice, but I often miss words, meanings and cultural relevance to what is being said. NonoSubtitles was created to solve these problems since it can provide a transcription of the original language, a translation to the user's language and a lesson about why things are said a certain way and the culture behind it. For example, 今日はちょっと... literally translates as "Today is a little..." From an English speakers perspective this doesn't make full sense. ちょっと means little, but in this context it becomes "I can't do this today". The culture is trying to be polite by not outright saying "I cannot do this today", ちょっと being that bridge word. Without the explanation is it hard to understand so the AI teacher's lesson would be perfect for these situations. It's possible to copy this into an LLM, but with more intense situations it is hard to understand every word when you don't know it exactly, as well as harder to put the context of the whole situation inside. Since NonoSubtitles is already has the transcript in Japanese, this can all be done automatically with a click. I've been having fun watch some of my favorite streamers and finally understanding everything they say, as well as learning about niche topics and words I would otherwise miss. ### What it does NonoSubtitles is a macOS (with windows and linux as the next implementations) than can process local videos or listen to audio from a program, such as a web browser, through Apple's native screen sharing system. It produces subtitles in the original language as well as a translation into a target language. Currently it is only tested with Japanese source to English translation, but since it uses OpenAI models for the transcriptions, translations and lessons, it will be able to support any language supported by the OpenAI models. The subtitles are displayed always on top, without a formal window so it can take up little space just like normal subtitles. The subtitles have different themes, ranging from all the classic subtitle styles to more stylized interesting subtitles depending on the user's preferences. Every finalized subtitle can be clicked to bring up an AI teacher where you can ask any question about the subtitles. There are presets for "Break this sentence down" and "Explain the cultural significance" to gain a better understanding based on the users needs. The lesson has options for beginner, intermediate and advanced. ### How we built it Codex and ChatGPT were used for 90-95% of development including planning, creating the specs, technical implementation, code, subtitle styles via lunacy, chalkboard 3d set created via blender, Nono's teacher outfit and help of design of all the workflows of agents called, subtitles and lessons. The stack uses Tauri with Svelte and Rust. I prefer this stack since it typically uses low RAM and can be used in Linux, Windows and Mac. I chose Mac as my target for this hackathon. I was responsible for product direction, features, visual design and figuring how each piece would work together. All animations and 3d models were refined by me as well. I already has Nono's, the teacher mascot, base 3d model before the hackathon since it is the mascot I use across many of my applications. The NonoSubtitles app, teaching system, subtitle pipeline, animations, integration and infrastructure were all created inside of the hackathon window. Claude Fable and Moonshot Kimi were also used as occasional reviewers and orchestrators using chatgpt-5.6-sol as subagents, when ChatGPT/Codex had troubles identifying bugs alone. They were also used for help with animation integration into three.js. Technical Pipeline For local video, NonoSubtitles extracts audio from video files, then sends it to OpenAi's models via API. gpt-4o-transcribe-diarize model is used in local videos since it is able to seperate speech from four different speakers allowing subtitles to be labeled by who is saying the lines as well as timestamps for the transcription. For source only subtitles, only gpt-realtime-whisper is used, for speed and cost since the other features of gpt-4o-transcribe-diarize are less required for native speakers. For translations, both gpt-5.6-terra and gpt-realtime-translate are options for use. Both have positives and negatives. gpt-5.6-terra takes the source transcription and translates often being slower but accurate to the source text. gpt-realtime-translate is quick and great but can only process audio so when it returns source + translation, a techincal limitation is these may differ. For example I once saw Japanese showing twelve and English showing eleven. gpt-5.6-sol is used for lesson creation because it has the highest success rate for correct formatting and choosing correct animations to go with the lessons. Why It Matters Language barriers are still prevalent online. The most common language is English which is alienating to non speakers. For me I am learning Japanese, but can't fully understand every conversation. The internet brings interest into foreign films, videos, livestreams, educational content, but subtitles can be missing, inaccurate or unavailable in one's native language. A boon of NonoSubtitles is that education is optional. It can be used for source language transcriptions, translations alone or learning. That makes it versatile and easy to enter. Why Nono? I included Nono because I wanted the app to feel more lighthearted, fun and personable instead of a utility only app which can scare off users. Nono is a mascot I use for all of my apps I am making. Her name comes from my mission statement. "No tracking. No ads. No subscriptions. No accounts. No anti-consumer behavior." the only one that can change is accounts since NonoSubtitles would be a pay as you go and needs a way to pay or track. But I read No... No... No... and thought, aww Nono is cute! I prefer apps with lots of expressiveness and personality. This is off-putting to some users, but my target audience is those who share that feeling with me! ### Challenges we ran into The biggest challenge I had was agentic testing not working because of Mac's Keychain. Since the API keys were stored this way, the agents had no way of inputting my password and accepting. While I worked or cared for my mother, I could not have the agent check for issues autonomously. Next would be my windows computer broke during a lightning storm which are common in Tennessee. I had planned to port NonoSubtitles to Windows, as well as do animation work on this desktop but was unable to. I could only build for Mac with no way of testing windows and use the Mac for animations in blender only. ### Accomplishments we're proud of Accomplishments I am proud of is the speed of the subtitles in both languages with an average of 1.7 second delay. For livestreams this is fast enough to keep up with the chat which I've been testing and having a lot of fun with! Second is the AI teacher simulation. Having it display a lesson and be able to point to and underline words/concepts at appropriate times almost feels like magic. ### What we learned I learned how to use Github! I usually use a local git on my home server. I learned about licensing for apps and software. I learned about animation and how to apply it to three.js I learned about MacOS keychain to store API keys for security, rather than keeping it in a file in app. ### What's next Next for NonoSubtitles will be to bug fix first and foremost. With only a week to work and a busy schedule I was not able to fix or figure out every bug. The biggest one is source language sentences being split up in the subtitle window, creating a jarring effect. Next would be to extensively test all of the OpenAI API models to figure the best pipeline for accuracy, speed and cost. This was done a bit, but to save time I stopped after an hour of testing. Nono's animations will be refined. This was my first attempt at animation so the character can look a tad goofy sometimes. The toon shader also needs a lot of work! Syncing animations from Nono's body and tails with the lesson, and delivering a multi step lesson is important. Since subtitles can be generated with timestamps, I would like to add the capability to add subtitles to your videos to upload online. I wanted to do this for the hackathon but found myself short on time. Lastly, it will be ported to Linux and Windows. Since it's on Tauri/Svelte/Rust, it already has functionality on these two systems, with the main connector being the audio capture needing different code and testing. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 126 recognized source files, 1302 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - Rust (language) — detected in the code - Svelte (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers - AI coding agent: Codex — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 146) ``` .claude/launch.json .github/workflows/ci.yml .gitignore .vscode/extensions.json ASSET_RIGHTS.md demo/README.md docs/AI_CONTRIBUTIONS.md docs/ARCHITECTURE.md docs/BUILD_WEEK_LOG.md docs/JUDGE_INSTRUCTIONS.md docs/LUNA_LIVE_TRANSLATION_BENCHMARK.md docs/NONO_ANIMATION_HANDOFF.md docs/NONO_CANDIDATE_PROMOTION.md docs/nono-arm-fix-session.diff docs/plans/ATOMIC_FILE_RETRANSLATION.md docs/plans/AUTONOMOUS_QA_CAMPAIGN.md docs/plans/BOUNDED_LONG_SESSIONS.md docs/plans/FILE_BOUNDARY_ID_STABILITY.md docs/plans/INVISIBLE_PRODUCT_SHELL.md docs/plans/LIVE_CLAUSE_LIFECYCLE.md docs/plans/LIVE_SUBTITLE_SAFETY_ENVELOPE.md docs/plans/MULTI_WINDOW_STATE_CONVERGENCE.md docs/plans/PINNED_LESSON_IDENTITY.md docs/plans/PLAYBACK_OWNERSHIP.md docs/plans/REALTIME_SESSION_HARDENING.md docs/plans/RELEASE_STATE_AND_ACCEPTANCE.md docs/plans/SESSION_GENERATION_ISOLATION.md docs/plans/TRANSLATION_FAILURE_CONTAINMENT.md docs/plans/TRANSPORT_AND_MEDIA_CLOCKS.md docs/PRIVACY.md docs/SESSION_REPORT_NONO_ARM_FIX.md docs/SESSION_REPORT_NONO_FACEKEYS.md docs/SUBMISSION_CHECKLIST.md docs/SUBMISSION_SPRINT.md LICENSE licenses/KLEE_ONE_OFL.txt licenses/NONTOON_ZLIB.txt package.json README.md rust-toolchain.toml scripts/audit_nono_glb.mjs scripts/build_english_fixture.sh scripts/build_indirect_refusal_fixture.sh scripts/build_two_speaker_fixture.sh scripts/cut_nono_clips.py scripts/export_nono_final.py scripts/prepare_nono_production.py scripts/prepare_nono_release.py scripts/render_chalkboard_asset.py scripts/render_nono_candidate.py scripts/retarget_nono_capture.py scripts/run_unattended_qa.sh scripts/save_qa_api_key.sh scripts/strip_nono_glb.mjs src-tauri/.gitignore src-tauri/build.rs src-tauri/capabilities/default.json src-tauri/Cargo.lock src-tauri/Cargo.toml src-tauri/icons/android/mipmap-anydpi-v26/ic_launcher.xml src-tauri/icons/android/values/ic_launcher_background.xml src-tauri/icons/icon.icns src-tauri/Info.plist src-tauri/src/chunking.rs src-tauri/src/contracts.rs src-tauri/src/lib.rs src-tauri/src/live.rs src-tauri/src/luna_benchmark.rs src-tauri/src/main.rs src-tauri/src/media_keys.rs src-tauri/src/media.rs src-tauri/src/openai.rs src-tauri/src/pipeline.rs src-tauri/tauri.conf.json src/app.css src/app.html src/lib/ArcadeSubtitleCard.svelte src/lib/BroadcastSubtitleCard.svelte src/lib/captureSources.test.ts src/lib/captureSources.ts src/lib/ChalkDemo.svelte src/lib/ChalkPhrase.svelte src/lib/ChalkStepNumber.svelte src/lib/contracts.ts src/lib/CyberiaSubtitleCard.svelte src/lib/fixtures.ts src/lib/floatingPlacement.test.ts src/lib/floatingPlacement.ts src/lib/hairMotion.test.ts src/lib/hairMotion.ts src/lib/LauncherSurface.svelte src/lib/lesson.test.ts src/lib/lesson.ts src/lib/lessonIdentity.test.ts src/lib/lessonIdentity.ts src/lib/lessonLayout.test.ts src/lib/lessonLayout.ts src/lib/LessonQuestionComposer.svelte src/lib/LessonSurface.svelte src/lib/lessonThreads.test.ts src/lib/lessonThreads.ts src/lib/LiveSubtitleStack.svelte src/lib/macosPrivacyPackaging.test.ts src/lib/MomentoSubtitleCard.svelte src/lib/NonoScene.svelte src/lib/nonoToon.test.ts src/lib/nonoToon.ts src/lib/overlayGeometry.test.ts src/lib/overlayGeometry.ts src/lib/OverlaySurface.svelte src/lib/playbackOwnership.test.ts src/lib/playbackOwnership.ts src/lib/preferences.test.ts src/lib/preferences.ts src/lib/runtime.test.ts src/lib/runtime.ts src/lib/session.test.ts src/lib/session.ts src/lib/sessionLaunch.test.ts src/lib/sessionLaunch.ts [26 more files omitted for size] ``` ### Dependencies - package.json: @fontsource/dotgothic16@^5.2.8, @fontsource/jetbrains-mono@^5.2.8, @fontsource/klee-one@^5.2.8, @fontsource/share-tech-mono@^5.2.7, @sveltejs/adapter-static@^3.0.6, @sveltejs/kit@^2.9.0, @sveltejs/vite-plugin-svelte@^5.0.0, @tailwindcss/vite@^4.3.2, @tauri-apps/api@^2, @tauri-apps/cli@^2, @tauri-apps/plugin-dialog@^2.7.1, @types/three@^0.185.1, svelte@^5.0.0, svelte-check@^4.0.0, tailwindcss@^4.3.2, three@^0.185.1, typescript@~5.6.2, vite@^6.0.3, vitest@^4.1.10 - src-tauri/Cargo.toml: base64@0.22, futures-util@0.3, hound@3.5, keyring@3, reqwest@0.12, serde@1, serde_json@1, symphonia@0.6, tauri@2, tauri-build@2, tauri-plugin-dialog@2, tempfile@3, tokio@1, tokio-tungstenite@0.28, unicode-segmentation@1.12 ### Recent commits (newest first) - License NonoSub code under MIT - Document end-to-end Codex collaboration - Stabilize Nono face and lesson tail routing - Tighten Nono lesson stage composition - Polish Nono anime toon shading - Repair lesson direction and Accurate live startup - Promote face-layer Nono to release slot - Load candidate Nono in Tauri development - Add Nono face layer: blink/smile morphs, >.< squint, shape-key export pipeline - Benchmark and lock Luna live translation - Build transcript-locked live translation engine - Add live translation engine contracts - Make capture privacy regression test portable - Repair macOS capture permission packaging - Document AI contributions and production Nono handoff - Enlarge and restructure Nono lesson stage - Promote repaired teacher Nono model - Align Claude launch config with NonoSub dev port - Checkpoint Fable Nono model repair pipeline - Rename Fallout subtitle preset to Arcade ## Key source files (fetched from GitHub, selected and truncated for size) Source contents could not be fetched from GitHub right now. [122 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]