# Project export: Tonguekeeper This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Every 2 weeks, a language dies. TongueKeeper deploys autonomous AI agents that compile and cross-reference vocabulary and grammar from scattered sources, forging them into a living language atlas. - Devpost: https://devpost.com/software/tonguekeeper - GitHub: https://github.com/lourdrickvalsote/tonguekeeper - Demo: https://tonguekeeper.vercel.app/ - Video: https://www.youtube.com/embed/nCSd-ZjfpkI?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner ([Perplexity] Best Use of Perplexity's Sonar API ($500 per team member and trip to Perplexity office)) - Team: 1 GitHub contributor(s) — lourdrickvalsote (4 commits) ## Devpost submission (written by the team) ### Inspiration Here's the thing that keeps me up at night: a language dies every two weeks. I've self-studied over 10 languages — from Korean to Portuguese, Mandarin to Arabic. Language learning isn't just a hobby for me, it's how I understand other people's worlds. UNESCO estimates that half of the world's ~7,000 languages will be extinct by 2100. When a language dies, it doesn't just take words with it — it takes entire ways of thinking, centuries of oral history, indigenous knowledge systems, and irreplaceable cultural identity. The data to preserve these languages exists — scattered across obscure academic papers, government archives, YouTube videos of elderly speakers, community-built dictionaries, and linguistic databases behind paywalls and anti-bot protections. But it's fragmented, hard to find, and disappearing alongside the languages themselves. I asked myself: what if AI agents could do in minutes what would take a linguist months? What if I could point an autonomous system at any of the 5,000+ endangered languages on Earth and have it discover, extract, cross-reference, and archive every fragment of linguistic data it can find — before it's too late? That's TongueKeeper. ### What it does TongueKeeper is an AI-powered platform that autonomously preserves endangered languages from across the internet. Select any of 5,352 endangered languages from the Glottolog database, hit "Preserve," and watch as AI agents go to work — discovering resources, crawling pages, extracting vocabulary and grammar, cross-referencing entries across sources, and building a searchable, interactive archive. All in real-time. Core features: Autonomous Multi-Agent Pipeline — Discovery, Extraction, and Cross-Reference agents work in concert, streaming results as they find them. A real-time activity feed lets you watch every agent action as it happens. 5,352+ Endangered Language Browser — Interactive world map and filterable table of every endangered language on Earth (Glottolog CLDF), with endangerment levels, speaker counts, geographic coordinates, and language family classification. Interactive Knowledge Graph — Force-directed visualization of linguistic relationships: related terms, semantic clusters, and embedding-based similarity edges. Explore how words connect across a language. Hybrid Search Archive — BM25 + kNN vector search with cross-encoder reranking. Search vocabulary in any language — English, the target language, or even a contact language like Korean or Spanish. AI Pronunciation Videos — HeyGen AI avatars that pronounce vocabulary entries, bringing endangered words to life. Conversational AI — Ask questions about the preserved archive in natural language. The system generates Elasticsearch queries on the fly and streams answers back. Grammar Pattern Extraction — Beyond vocabulary, the system identifies and catalogs grammar rules across 9 categories (verb conjugation, particle usage, sentence structure, honorifics, and more). Command Palette — Cmd+K unified search across languages, vocabulary, and grammar simultaneously. ### How we built it TongueKeeper is a multi-service architecture — a Next.js 16 frontend on Vercel, a standalone Express 5 + Socket.io backend on Railway, and an ecosystem of sponsor APIs that each solve a critical piece of the puzzle. The Agent Pipeline The core of TongueKeeper is a multi-agent pipeline orchestrated by a streaming worker pool. When you click "Preserve," here's what happens: 1. Discovery (Perplexity Sonar + BrightData SERP API) The Discovery Agent generates up to 24 search queries across 6 tiers — core queries, native name queries, alternate name queries, contact language queries (in 13 languages including Korean, Japanese, Mandarin, Spanish, and French), country-specific queries, and language family queries. Each query hits Perplexity Sonar for real-time web search with academic domain filtering. Simultaneously, BrightData's SERP API runs geo-targeted searches with CAPTCHA bypass to surface region-locked resources that standard search engines miss. Sources stream in as they're found — no waiting for the full discovery phase to finish. 2. Crawling (Cheerio → BrightData Web Unlocker → Browserbase/Stagehand) Every discovered URL goes through a three-tier crawling cascade, ordered by cost: Cheerio — Fast, lightweight HTML parsing for simple pages BrightData Web Unlocker — When Cheerio fails or the domain is known to be protected (.edu, .gov, JSTOR, ResearchGate, Forvo, Glosbe), BrightData's MCP integration strips anti-bot protections and returns clean markdown Browserbase + Stagehand — For JavaScript-heavy single-page apps, a cloud headless Chrome session spins up with AI observability. Stagehand classifies the page structure (paginated? searchable? scrollable?) and executes the right strategy — clicking through pages, typing into search bars, or scrolling to load content. Live debug URLs stream to the activity feed so I can watch the browser work in real-time. For PDFs, the system tries pdf-parse, falls back to pdfjs-dist, and if the text is too sparse (scanned documents), sends the PDF as Base64 to Claude's Vision API for OCR extraction. 3. Extraction (Anthropic Claude — Haiku 4.5 / Sonnet 4.5) The Extraction Agent runs a manual Claude tool_use loop — calling Claude with the crawled content, processing tool calls (save_entries for vocabulary, save_grammar_patterns for grammar), building tool results, and repeating until the agent signals "done." Each vocabulary entry captures 35+ fields: headword (native + romanized), IPA, part of speech, multi-language definitions, example sentences, conjugations, morphology, usage register, cultural notes, semantic domain, and more. Claude Haiku 4.5 handles text sources for speed and cost efficiency. Claude Sonnet 4.5 steps in for degraded scans and vision tasks. Content is chunked at 50K characters with 3 concurrent extraction workers, and a Semaphore limits total Claude API calls to prevent rate limiting. A contamination filter drops entries where language_confidence: "low" — preventing contact language words from polluting the archive. 4. Cross-Reference (Anthropic Claude — Haiku 4.5) The Cross-Reference Agent searches Elasticsearch for potential duplicates by headword, then uses Claude to intelligently merge entries — unioning definitions, combining example sentences, and merging metadata. Conservative bias: a false merge is worse than a missed duplicate. 5. Search & Retrieval (Elasticsearch + JINA AI) All data lands in Elasticsearch Serverless across 4 indexes (language_resources, grammar_patterns, languages, source_outcomes). Search uses a hybrid pipeline: $$\text{Score} = \text{RRF}(\text{BM25}(q), \text{kNN}(\text{JINA}{embed}(q))) \rightarrow \text{JINA}{rerank}(\text{top-50})$$ JINA Embeddings v3 (1024 dimensions) powers the kNN leg — crucially, these are multilingual embeddings, so searching "grandmother" in English finds "할머니" in Korean and the Jeju equivalent. JINA Reranker v2 (multilingual cross-encoder) does a second pass on the top 50 results for final ordering. This two-stage retrieval is what makes cross-lingual search actually work. 6. Post-Pipeline (HeyGen + Cloudflare + Perplexity Sonar) After the main pipeline: HeyGen generates AI avatar pronunciation videos for the top 5 vocabulary entries, with language-aware voice selection and audio lip-sync Cloudflare R2 stores pipeline artifacts (run logs, audio files) with edge-cached delivery across 150+ countries Cloudflare KV caches HeyGen videos (6-day TTL), Perplexity responses (24h TTL), and search results (30min TTL) at the edge Perplexity Sonar enriches high-priority entries with cultural context and scores source reliability Frontend The UI is built with Next.js 16 (App Router, React Server Components), React 19, and Tailwind CSS 4. The design language is warm and archival — cream backgrounds, DM Serif Display headings, burnt amber accents — evoking the feeling of preserving something precious. Framer Motion handles stagger animations, scroll-triggered counters, and the splash page logo reveal. The knowledge graph uses react-force-graph-2d (Canvas-based) with D3 utilities, and the world map uses React Leaflet with heatmap and choropleth layers. Real-time updates flow through Socket.io — the frontend connects to the WebSocket server and receives agent events as they happen, populating the activity feed and auto-refreshing search results mid-pipeline. Deployment Vercel — Frontend (Next.js 16, edge functions, API routes) Railway — WebSocket server (Express 5 + Socket.io, persistent process) Cloudflare Workers — R2 storage + KV cache + language detection API ### Challenges we ran into Anti-bot protection on linguistic resources. The most valuable endangered language data lives behind anti-bot walls — university archives, government language databases, academic publishers. A basic HTTP client gets blocked instantly. This is where BrightData's Web Unlocker was essential — without it, roughly 40% of discovered sources would have been inaccessible. For the remaining JavaScript-heavy sites, Browserbase/Stagehand provided cloud headless Chrome with AI-powered navigation. Cross-lingual search is hard. When someone searches "water" in English, they expect to find the Jeju word "물" and the Ainu word "ワッカ." Traditional keyword search fails here completely. The combination of JINA's multilingual embeddings for vector similarity and JINA's cross-encoder reranker for precision scoring was the breakthrough — it made search work across scripts and language families. Rate limiting and cost control. With 3 AI APIs (Claude, Perplexity, JINA), a cloud browser service, and a web unlocking proxy, costs compound fast. I implemented a semaphore for concurrent Claude calls, KV caching for Perplexity and HeyGen responses via Cloudflare KV, content-hash deduplication (SHA-256) to skip identical pages, and a 25-source cap per pipeline run. Scanned PDFs. Many endangered language resources are scanned academic papers from the 1970s–90s. pdf-parse returns gibberish, pdfjs-dist returns nothing. The solution: detect sparse text (< 500 chars), classify scan quality, and route degraded scans to Claude Sonnet 4.5's Vision API as Base64 images. Contamination filtering. When extracting vocabulary from a source about Jeju (a Korean language), the agent would sometimes extract Korean words instead of Jeju words. I added a language_confidence field and a contamination filter that drops low-confidence entries — better to miss a word than to pollute the archive with the wrong language. ### Accomplishments we're proud of Built a fully autonomous language preservation system — solo, in 36 hours — that can process any of 5,352+ endangered languages Achieved real-time, end-to-end pipeline execution: from zero data to a searchable, browsable, pronounceable vocabulary archive in under 10 minutes per language Integrated 8 sponsor/partner APIs into a cohesive pipeline where each service solves a distinct, critical problem Cross-lingual hybrid search that actually works — finding words across scripts and language families using JINA's multilingual embeddings and reranker An interactive world map of every endangered language on Earth, built from Glottolog CLDF data (5,352 languages with coordinates, endangerment levels, and family classification) Real-time agent activity feed with live Browserbase session thumbnails — you can literally watch the AI browse the web ### What we learned The MCP protocol is powerful. BrightData's MCP integration made web scraping feel like calling a function — no proxy configuration, no CAPTCHA solving, no IP rotation. Just scrape_as_markdown(url) and get clean content back. This is the future of tool integrations for AI agents. Multilingual embeddings change everything. Before JINA, search was language-locked. After JINA, a single query retrieves results across all languages the embedding model has seen. For endangered language work — where resources exist in multiple contact languages — this is transformative. Claude's tool_use loop is incredibly flexible. The manual loop pattern (call → process tool blocks → build results → append → repeat) let me build agents that stream results mid-extraction, save in batches, and gracefully handle failures without crashing the pipeline. Edge caching matters. Moving HeyGen video URLs and Perplexity responses to Cloudflare KV eliminated redundant API calls and made the UI feel instant on repeat visits. The 6-day TTL for pronunciation videos alone saved dozens of HeyGen API calls during development. Real-time feedback changes how you build. Watching the agent activity feed during development — seeing exactly which sources succeed, which fail, and why — made debugging 10x faster than reading logs. Give yourself more time than you think you will need. Self-evident. Plan for delays. ### What's next for TongueKeeper Community contributions — Allow native speakers to verify, correct, and add entries to the archive. The AI gets us 80% of the way; human speakers complete the last 20%. Audio preservation at scale — Automatically download and archive audio recordings of endangered language speakers before they disappear from the web. Comparative linguistics — Use the knowledge graph to surface cross-language patterns: shared roots, borrowed words, and grammatical similarities across language families. Offline-first mobile app — Many endangered language communities lack reliable internet. A downloadable, offline-capable archive would bring the data to where it's needed most. Educational material generation — Use the preserved vocabulary, grammar patterns, and pronunciation videos to automatically construct lesson plans, flashcard decks, and phrasebooks — turning raw linguistic data into tools that help communities teach and revitalize their languages. Each language contains centuries of history and culture. TongueKeeper keeps them alive. ## README (from the GitHub repository) # TongueKeeper **Every language is a universe of thought. We keep them alive.** An AI-powered platform that autonomously discovers, aggregates, and cross-references endangered language content scattered across the internet — transforming fragments into a unified, searchable, living linguistic archive. **[Live Demo](https://tonguekeeper.vercel.app)** --- ## The Problem A language dies every two weeks. By 2100, UNESCO estimates half of the world's ~7,000 languages will be extinct — each taking with it centuries of irreplaceable knowledge, oral history, and cultural identity. The resources to preserve these languages exist, but they're scattered across obscure PDFs, YouTube videos, academic papers, dictionary websites, and government archives in dozens of disconnected sources. A linguist would need months to even *find* them all, let alone cross-reference and synthesize them. TongueKeeper deploys a swarm of AI agents that autonomously crawl the web, discover these scattered fragments, extract linguistic data — vocabulary, grammar, audio, cultural context — and synthesize everything into a unified, searchable archive. In minutes, not months. --- ## How It Works ``` ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────────┐ ┌─────────────┐ │ Discovery │───▶│ Crawl │───▶│ Extraction │───▶│ Cross-Reference │───▶│ Archive │ │ │ │ │ │ │ │ │ │ │ │ Perplexity │ │ Cheerio │ │ Claude AI │ │ Claude AI │ │ Elastic │ │ BrightData │ │ Stagehand │ │ Vision API │ │ Merge & verify │ │ Jina embed │ │ SERP API │ │ BrightData │ │ PDF parsing │ │ Deduplication │ │ Semantic │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────────┘ └─────────────┘ ``` 1. **Discovery** — AI agents search across Perplexity Sonar and BrightData SERP API with 6-tier dynamic queries (core terms, native names, alternate names, contact languages, country-specific, language family), generating up to 24 targeted queries per language 2. **Crawl** — Each discovered source is fetched through a 3-tier cascade: specialized crawlers (YouTube, Wikipedia, ELAR, dictionaries) → BrightData Web Unlocker (CAPTCHA bypass) → Cheerio/Stagehand (headless browser). PDFs are parsed with text extraction or Claude Vision for scanned documents 3. **Extraction** — Claude processes each source in a manual tool-use loop, extracting structured vocabulary entries (headword, definitions, IPA, conjugations, morphology, examples) and grammar patterns (9 categories) into Elasticsearch 4. **Cross-Reference** — A second Claude agent searches for duplicate entries across sources, merging definitions, examples, and cross-references while calculating reliability scores based on source count 5. **Archive** — All data flows into Elasticsearch with Jina v3 embeddings (1024-dim) for semantic search, reranking, and knowledge graph generation --- ## Features - **Multi-Agent Pipeline** — Orchestrated Discovery, Extraction, and Cross-Reference agents with real-time event streaming via Socket.io - **Real-Time Dashboard** — Split-panel UI with live agent activity feed and a 200-event ring buffer for late-joining clients - **Semantic Search** — Elasticsearch multi-match queries with Jina AI embeddings and reranking for vocabulary and grammar patterns - **Knowledge Graph** — Force-directed graph visualization (react-force-graph-2d) with 3 edge types: related terms, semantic clusters, and embedding similarity - **Grammar Reference** — Browsable grammar patterns across 9 categories (verb conjugation, particles, sentence structure, honorifics, negation, questions, phonological rules, morphological rules) - **Language Browser** — 5,352 endangered languages from Glottolog CLDF with filtering by endangerment status, macroarea, language family, and speaker count - **Interactive Maps** — Leaflet maps with marker, heatmap, and choropleth modes showing global language endangerment - **Audio Pipeline** — YouTube audio extraction with Whisper transcription (RunPod serverless), word-level timestamps, and pronunciation avatar generation (HeyGen) - **PDF & Scan Extraction** — Text extraction via pdf-parse with automatic Vision API fallback for scanned/degraded documents - **Adaptive Web Crawling** — Domain-specific crawlers with BrightData Web Unlocker for geo-blocked and CAPTCHA-protected sources --- ## Tech Stack | Layer | Technologies | |---|---| | **Frontend** | Next.js 16, React 19, Tailwind CSS 4, shadcn/ui, Framer Motion, socket.io-client | | **Visualization** | react-force-graph-2d, Leaflet + react-leaflet, Recharts, D3.js | | **Backend** | Express 5, Socket.io, Node.js (tsx runtime) | | **AI Agents** | Anthropic Claude (Haiku 4.5 + Sonnet 4.5), manual tool-use loops | | **Search & Discovery** | Perplexity Sonar API, BrightData SERP API + Web Unlocker | | **Embeddings** | Jina AI v3 (embeddings) + v2 (reranking) | | **Data Store** | Elasticsearch 9 (serverless) — vocabulary, grammar, languages, pipeline runs | | **Web Crawling** | Cheerio, Browserbase + Stagehand, pdf-parse, pdfjs-dist | | **Audio/ML** | Python FastAPI, RunPod (Whisper transcription), HeyGen (avatar videos) | | **Infrastructure** | Vercel (frontend), Cloudflare Workers (R2 storage + KV cache) | | **Validation** | TypeScript 5 (strict), Zod 4 | --- ## Architecture ``` ┌──────────────────────────────────────────────────────────────┐ │ Browser (:3000) │ │ Next.js 16 App Router · React 19 · Socket.io Client │ └────────────┬──────────────────────────────┬──────────────────┘ │ HTTP (API Routes) │ WebSocket ▼ ▼ ┌────────────────────────┐ ┌────────────────────────────────┐ │ Next.js API Routes │ │ Express + Socket.io (:3001) │ │ │ │ │ │ /api/search │ │ Pipeline Orchestrator │ │ /api/grammar │ │ Discovery Agent │ │ /api/languages │ │ Extraction Agent │ │ /api/graph │ │ Cross-Reference Agent │ │ /api/preserve ────────┼───▶│ Enrichment Agent │ │ │ │ Event Emitter (ring buffer) │ └────────┬───────────────┘ └──────┬─────────────────────────┘ │ │ ▼ ▼ ┌────────────────────┐ ┌────────────────────────────────────┐ │ Elasticsearch 9 │ │ Python FastAPI (:3003) │ │ (Serverless) │ │ │ │ │ │ YouTube audio extraction │ │ vocabulary │ │ Whisper transcription (RunPod) │ │ grammar_patterns │ │ Word-level timestamps │ │ languages (5,352) │ └────────────────────────────────────┘ │ language_resources│ │ pipeline_runs │ └────────────────────┘ ``` --- ## Getting Started ### Prerequisites - Node.js 18+ - Python 3.10+ (for ML service) - npm ### Install ```bash git clone https://github.com/lourdrickvalsote/tonguekeeper.git cd tonguekeeper npm install --legacy-peer-deps ``` ### Environment Variables Create a `.env.local` file in the project root: | Variable | Required | Description | |---|---|---| | `ELASTIC_URL` | Yes | Elasticsearch cluster URL | | `ELASTIC_API_KEY` | Yes | Elasticsearch API key | | `ANTHROPIC_API_KEY` | Yes | Claude API key (agents) | | `PERPLEXITY_API_KEY` | Yes | Perplexity Sonar API key (discovery search) | | `JINA_API_KEY` | Yes | Jina AI key (embeddings + reranking) | | `NEXT_PUBLIC_WS_URL` | No | WebSocket server URL (default: `http://localhost:3001`) | | `BROWSERBASE_API_KEY` | No | Browserbase API key (headless browsing) | | `BROWSERBASE_PROJECT_ID` | No | Browserbase project ID | | `BRIGHTDATA_API_TOKEN` | No | Br [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 186 recognized source files, 1355 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - Express (technology) — detected in the code - FastAPI (technology) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 203) ``` .env.example .gitignore .npmrc app/(main)/about/page.tsx app/(main)/dashboard/page.tsx app/(main)/error.tsx app/(main)/languages/[glottocode]/page.tsx app/(main)/languages/page.tsx app/(main)/layout.tsx app/(splash)/page.tsx app/api/agent-chat/route.ts app/api/audio/download/route.ts app/api/audio/route.ts app/api/brightdata/metrics/route.ts app/api/crawl/route.ts app/api/events/route.ts app/api/export/route.ts app/api/grammar/route.ts app/api/grammar/stats/route.ts app/api/graph/route.ts app/api/heygen/pronounce/check/route.ts app/api/heygen/pronounce/route.ts app/api/index/route.ts app/api/insights/route.ts app/api/languages/[glottocode]/overview/route.ts app/api/languages/[glottocode]/route.ts app/api/languages/geo/route.ts app/api/languages/geojson/route.ts app/api/languages/route.ts app/api/poke/notify/route.ts app/api/preserve/route.ts app/api/pronunciations/route.ts app/api/runs/[languageCode]/[runId]/route.ts app/api/runs/[languageCode]/route.ts app/api/search/route.ts app/api/sources/entries/route.ts app/api/sources/route.ts app/api/stats/route.ts app/api/transcript/[videoId]/route.ts app/global-error.tsx app/globals.css app/layout.tsx app/not-found.tsx app/thumbnail/page.tsx components.json components/agent-builder/AgentChat.tsx components/agent-feed/agent-event-card.tsx components/agent-feed/agent-feed.tsx components/agent-feed/browser-session-panel.tsx components/agent-feed/counter-bar.tsx components/agent-feed/use-browser-sessions.ts components/command-palette/CommandPalette.tsx components/dashboard/BrightDataImpact.tsx components/dashboard/LanguageHealth.tsx components/dashboard/sponsor-footer.tsx components/dashboard/stats-bar.tsx components/grammar/GrammarPatternCard.tsx components/grammar/GrammarPatternDetail.tsx components/grammar/GrammarReference.tsx components/graph/graph-constants.ts components/graph/graph-types.ts components/graph/GraphCanvas.tsx components/graph/GraphContextMenu.tsx components/graph/GraphControls.tsx components/graph/GraphEmptyState.tsx components/graph/GraphLegend.tsx components/graph/GraphSearch.tsx components/graph/GraphTooltip.tsx components/graph/KnowledgeGraph.tsx components/graph/useGraphData.ts components/graph/useGraphInteractions.ts components/insights/DistinctiveVocabulary.tsx components/languages/EndangermentBadge.tsx components/languages/FilterBar.tsx components/languages/LanguageBrowserStats.tsx components/languages/LanguageOverview.tsx components/languages/LanguageTable.tsx components/languages/map/ChoroplethLayer.tsx components/languages/map/HeatmapLayer.tsx components/languages/map/MapControls.tsx components/languages/map/MapLegend.tsx components/languages/map/MapVisualizationToggle.tsx components/languages/MiniMap.tsx components/languages/WorldMap.tsx components/navigation/Sidebar.tsx components/navigation/TongueKeeperLogo.tsx components/pipeline/RunCard.tsx components/pipeline/RunHistory.tsx components/results/SearchResults.tsx components/results/vocabulary-card.tsx components/results/VocabularyDetail.tsx components/search/ArchiveBuilding.tsx components/search/ArchiveFilters.tsx components/search/PreservationDialog.tsx components/search/search-bar.tsx components/search/search-panel.tsx components/search/SourcesList.tsx components/transcripts/TranscriptViewer.tsx components/ui/badge.tsx components/ui/button.tsx components/ui/card.tsx components/ui/command.tsx components/ui/dialog.tsx components/ui/input.tsx components/ui/progress.tsx components/ui/resizable-layout.tsx components/ui/scroll-area.tsx components/ui/separator.tsx components/ui/skeleton.tsx components/ui/tabs.tsx eslint.config.mjs hooks/use-debounce.ts infra/package.json infra/tsconfig.json infra/worker/src/index.ts infra/wrangler.toml lib/active-language.tsx lib/agents/cross-reference-agent.ts lib/agents/discovery-agent.ts lib/agents/enrichment.ts [83 more files omitted for size] ``` ### Dependencies - infra/package.json: @cloudflare/workers-types@^4.20241205.0, typescript@^5.0.0, wrangler@^3.99.0 - ml/fetchai/requirements.txt: python-dotenv@>=1.1.0, requests@>=2.31.0, uagents@>=0.17.0, uagents-core@>=0.5.0 - ml/requirements.txt: anthropic@>=0.40.0, fastapi@>=0.115.0, faster-whisper@~=1.1.0, pydub@>=0.25.1, python-dotenv@>=1.1.0, requests@~=2.31.0, runpod@~=1.7.0, uvicorn[standard]@>=0.34.0, yt-dlp@>=2024.1.0 - package.json: @anthropic-ai/sdk@^0.74.0, @browserbasehq/stagehand@^3.0.8, @elastic/elasticsearch@^9.3.1, @modelcontextprotocol/sdk@^1.26.0, @react-leaflet/core@^3.0.0, @tailwindcss/postcss@^4, @tanstack/react-table@^8.21.3, @tanstack/react-virtual@^3.13.18, @types/d3@^7.4.3, @types/express@^5.0.6, @types/leaflet@^1.9.21, @types/node@^20, @types/react@^19, @types/react-dom@^19, cheerio@^1.2.0, class-variance-authority@^0.7.1, clsx@^2.1.1, cmdk@^1.1.1, concurrently@^9.2.1, csv-parse@^6.1.0, d3@^7.9.0, dotenv@^17.3.1, eslint@^9, eslint-config-next@16.1.6, express@^5.2.1, express-rate-limit@^8.2.1, framer-motion@^12.34.0, leaflet@^1.9.4, leaflet.heat@^0.2.0, lucide-react@^0.564.0, next@16.1.6, pdf-parse@^2.4.5, pdfjs-dist@^5.4.624, radix-ui@^1.4.3, react@19.2.3, react-dom@19.2.3, react-force-graph-2d@^1.29.1, react-leaflet@^5.0.0, react-markdown@^10.1.0, react-resizable-panels@^4.6.4, recharts@^3.7.0, shadcn@^3.8.4, socket.io@^4.8.3, socket.io-client@^4.8.3, tailwind-merge@^3.4.0, tailwindcss@^4, tsx@^4.21.0, tw-animate-css@^1.4.0, typescript@^5, undici@^7.22.0, zod@^4.3.6 ### Recent commits (newest first) - bruh no more time... - i have like an hour left lol - lowk might be - am i goated? ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "tonguekeeper", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint", "server": "tsx server/ws-server.ts", "ml": "cd ml && python3 server.py", "pregenerate": "tsx scripts/pregenerate-pronunciations.ts", "fetchai": "cd ml/fetchai && pip3 install -r requirements.txt && python3 discovery_agent.py", "backup": "tsx scripts/backup-elastic.ts", "restore": "tsx scripts/backup-elastic.ts --restore", "reindex": "tsx scripts/reindex-with-analyzers.ts", "ingest:glottolog": "tsx scripts/ingest-glottolog.ts", "migrate:multilang": "tsx scripts/migrate-multilang.ts", "prescan": "tsx scripts/prescan-languages.ts", "generate:map": "tsx scripts/generate-map-data.ts", "generate:countries": "tsx scripts/generate-countries-geojson.ts", "enrich:speakers": "tsx scripts/enrich-speaker-counts.ts", "setup:data": "npm run ingest:glottolog && npm run generate:map && npm run prescan", "dev:all": "concurrently -n next,ws,ml -c blue,green,magenta \"npm run dev\" \"npm run server\" \"npm run ml\"" }, "dependencies": { "@anthropic-ai/sdk": "^0.74.0", "@browserbasehq/stagehand": "^3.0.8", "@elastic/elasticsearch": "^9.3.1", "@modelcontextprotocol/sdk": "^1.26.0", "@react-leaflet/core": "^3.0.0", "@tanstack/react-table": "^8.21.3", "@tanstack/react-virtual": "^3.13.18", "@types/d3": "^7.4.3", "cheerio": "^1.2.0", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "cmdk": "^1.1.1", "csv-parse": "^6.1.0", "d3": "^7.9.0", "dotenv": "^17.3.1", "express": "^5.2.1", "express-rate-limit": "^8.2.1", "framer-motion": "^12.34.0", "leaflet": "^1.9.4", "leaflet.heat": "^0.2.0", "lucide-react": "^0.564.0", "next": "16.1.6", "pdf-parse": "^2.4.5", "pdfjs-dist": "^5.4.624", "radix-ui": "^1.4.3", "react": "19.2.3", "react-dom": "19.2.3", "react-force-graph-2d": "^1.29.1", "react-leaflet": "^5.0.0", "react-markdown": "^10.1.0", "react-resizable-panels": "^4.6.4", "recharts": "^3.7.0", "socket.io": "^4.8.3", "socket.io-client": "^4.8.3", "tailwind-merge": "^3.4.0", "undici": "^7.22.0", "zod": "^4.3.6" }, "devDependencies": { "@tailwindcss/postcss": "^4", "@types/express": "^5.0.6", "@types/leaflet": "^1.9.21", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "concurrently": "^9.2.1", "eslint": "^9", "eslint-config-next": "16.1.6", "shadcn": "^3.8.4", "tailwindcss": "^4", "tsx": "^4.21.0", "tw-animate-css": "^1.4.0", "typescript": "^5" } } ``` ### ml/requirements.txt ``` runpod~=1.7.0 faster-whisper~=1.1.0 requests~=2.31.0 yt-dlp>=2024.1.0 pydub>=0.25.1 anthropic>=0.40.0 fastapi>=0.115.0 uvicorn[standard]>=0.34.0 python-dotenv>=1.1.0 ``` ### infra/package.json ``` { "name": "tonguekeeper-worker", "version": "1.0.0", "private": true, "scripts": { "dev": "wrangler dev", "deploy": "wrangler deploy", "tail": "wrangler tail" }, "devDependencies": { "@cloudflare/workers-types": "^4.20241205.0", "typescript": "^5.0.0", "wrangler": "^3.99.0" } } ``` ### ml/Dockerfile ``` FROM --platform=linux/amd64 runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04 WORKDIR /app # Install system dependencies RUN apt-get update && \ apt-get install -y --no-install-recommends ffmpeg && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* # Install Python dependencies (upgrade filelock first — base image version clashes with huggingface_hub) COPY requirements.txt . RUN pip install --no-cache-dir --upgrade filelock huggingface_hub && \ pip install --no-cache-dir -r requirements.txt # Download the Whisper large-v3 model at build time (not runtime) RUN python -c "from huggingface_hub import snapshot_download; snapshot_download('Systran/faster-whisper-large-v3', local_dir='/app/models/large-v3')" # Copy handler COPY runpod_handler.py . ENV MODEL_PATH=/app/models/large-v3 CMD ["python", "-u", "/app/runpod_handler.py"] ``` ### ml/fetchai/requirements.txt ``` uagents>=0.17.0 uagents-core>=0.5.0 requests>=2.31.0 python-dotenv>=1.1.0 ``` ### app/layout.tsx ```typescript import type { Metadata } from "next"; import { DM_Serif_Display, Plus_Jakarta_Sans, JetBrains_Mono } from "next/font/google"; import { CommandPalette } from "@/components/command-palette/CommandPalette"; import "./globals.css"; const serif = DM_Serif_Display({ weight: "400", variable: "--font-dm-serif", subsets: ["latin", "latin-ext"], }); const sans = Plus_Jakarta_Sans({ variable: "--font-plus-jakarta", subsets: ["latin", "latin-ext"], }); const mono = JetBrains_Mono({ variable: "--font-jetbrains", subsets: ["latin", "latin-ext"], }); export const metadata: Metadata = { title: "TongueKeeper — AI-Powered Endangered Language Preservation", description: "Autonomous AI agents that discover, extract, and cross-reference endangered language content from across the internet. Preserving the world's endangered languages for future generations.", keywords: [ "endangered languages", "language preservation", "linguistic diversity", "AI agents", "NLP", ], }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### ml/server.py ```python """ TongueKeeper — ML FastAPI Service Exposes the audio processing pipeline over HTTP. """ import os import sys # Add project root to path for lib imports PROJECT_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) sys.path.insert(0, PROJECT_ROOT) # Load .env.local from project root from dotenv import load_dotenv load_dotenv(os.path.join(PROJECT_ROOT, ".env.local")) from fastapi import FastAPI, HTTPException from fastapi.middleware.cors import CORSMiddleware from pydantic import BaseModel from typing import Optional from audio_pipeline import ( process_video, _runpod_request, RUNPOD_ENDPOINT_ID, RUNPOD_API_KEY, ) app = FastAPI(title="TongueKeeper ML Service", version="1.0.0") _cors_origins = [ "http://localhost:3000", "http://localhost:3001", ] if os.environ.get("FRONTEND_URL"): _cors_origins.append(os.environ["FRONTEND_URL"]) if os.environ.get("WS_URL"): _cors_origins.append(os.environ["WS_URL"]) app.add_middleware( CORSMiddleware, allow_origins=_cors_origins, allow_methods=["*"], allow_headers=["*"], ) # --------------------------------------------------------------------------- # Request / Response models # --------------------------------------------------------------------------- class ProcessVideoRequest(BaseModel): video_url: str language: str = "en" # legacy, ignored when contact_languages present chunk_seconds: int = 30 language_name: str = "Unknown" language_code: str = "und" contact_languages: Optional[list[str]] = None known_vocabulary: Optional[list[str]] = None class TranscribeRequest(BaseModel): audio_url: str language: str = "en" # --------------------------------------------------------------------------- # Endpoints # --------------------------------------------------------------------------- @app.get("/health") async def health(): return { "service": "tonguekeeper-ml", "status": "ok", "runpod_configured": bool(RUNPOD_ENDPOINT_ID and RUNPOD_API_KEY), } @app.post("/process-video") async def handle_process_video(req: ProcessVideoRequest): try: result = process_video( video_url=req.video_url, language=req.language, chunk_seconds=req.chunk_seconds, known_vocabulary=req.known_vocabulary, language_name=req.language_name, language_code=req.language_code, contact_languages=req.contact_languages, ) return result except ValueError as e: raise HTTPException(status_code=400, detail=str(e)) except RuntimeError as e: raise HTTPException(status_code=502, detail=str(e)) except Exception as e: raise HTTPException(status_code=500, detail=str(e)) @app.post("/transcribe") async def handle_transcribe(req: TranscribeRequest): if not RUNPOD_ENDPOINT_ID or not RUNPOD_API_KEY: raise HTTPException( status_code=503, detail="RUNPOD_ENDPOINT_ID and RUNPOD_API_KEY must be set", ) try: result = _runpod_request(req.audio_url, req.language) if result.get("status") == "FAILED": raise HTTPException( status_code=502, detail=result.get("error", "RunPod transcription failed"), ) output = result.get("output", {}) return { "text": output.get("text", ""), "segments": output.get("segments", []), "detected_language": output.get("detected_language"), "confidence": output.get("confidence"), } except TimeoutError: raise HTTPException(status_code=504, detail="Transcription timed out") except HTTPException: raise except Exception as e: raise HTTPException(status_code=500, detail=str(e)) if __name__ == "__main__": import uvicorn port = int(os.environ.get("PORT", "3003")) uvicorn.run(app, host="0.0.0.0", port=port) ``` ### app/(main)/layout.tsx ```typescript import { Sidebar } from "@/components/navigation/Sidebar"; import { AgentEventsProvider } from "@/lib/websocket"; import { ActiveLanguageProvider } from "@/lib/active-language"; export default function MainLayout({ children, }: { children: React.ReactNode; }) { return (
{children}
 ); } ``` ### app/(splash)/page.tsx ```typescript "use client"; import { useState, useEffect, useRef } from "react"; import { motion, useInView } from "framer-motion"; import Link from "next/link"; import { LogoIcon } from "@/components/navigation/TongueKeeperLogo"; import { SponsorFooter } from "@/components/dashboard/sponsor-footer"; import { fetchLanguages } from "@/lib/api"; import { Search, BookOpen, GitMerge, ChevronDown } from "lucide-react"; // ── Animation Variants ─────────────────────────────────────────────────── const stagger = { hidden: {}, visible: { transition: { staggerChildren: 0.15, delayChildren: 0.3 } }, }; const fadeUp = { hidden: { opacity: 0, y: 24 }, visible: { opacity: 1, y: 0, transition: { duration: 0.6, ease: [0.25, 0.1, 0.25, 1] as const }, }, }; const logoReveal = { hidden: { opacity: 0, scale: 0.4 }, visible: { opacity: 1, scale: 1, transition: { type: "spring" as const, stiffness: 120, damping: 14 }, }, }; // ── Animated Counter ───────────────────────────────────────────────────── function AnimatedCount({ target, suffix = "", }: { target: number; suffix?: string; }) { const [count, setCount] = useState(0); const ref = useRef(null); const inView = useInView(ref, { once: true, margin: "-60px" }); useEffect(() => { if (!inView || target === 0) return; const duration = 1800; const start = performance.now(); function step(now: number) { const elapsed = now - start; const progress = Math.min(elapsed / duration, 1); const eased = 1 - Math.pow(1 - progress, 3); setCount(Math.floor(eased * target)); if (progress < 1) requestAnimationFrame(step); } requestAnimationFrame(step); }, [inView, target]); return ( {count.toLocaleString()} {suffix} ); } // ── Features ───────────────────────────────────────────────────────────── const FEATURES = [ { icon: Search, title: "Discover", description: "Autonomous agents scour the web for dictionaries, grammars, recordings, and academic papers in endangered languages.", color: "#1E40AF", }, { icon: BookOpen, title: "Extract", description: "AI-powered extraction pulls vocabulary, grammar patterns, and audio from diverse sources into structured archives.", color: "#047857", }, { icon: GitMerge, title: "Cross-Reference", description: "Intelligent verification links entries across sources, validating accuracy and building comprehensive language records.", color: "#6D28D9", }, ]; // ── Page ───────────────────────────────────────────────────────────────── export default function SplashPage() { const [stats, setStats] = useState({ totalEndangered: 0, criticallyEndangered: 0, preserved: 0, }); useEffect(() => { fetchLanguages({ limit: 1 }).then((data) => { setStats({ totalEndangered: data.stats.total_endangered, criticallyEndangered: data.stats.critically_endangered, preserved: data.stats.with_preservation_data, }); }); }, []); return (
{/* ── Hero ─────────────────────────────────────────────── */}
{/* Ambient glow — warm radiance emanating from center */}
{/* Grain texture overlay */}
{/* Hero content */} {/* Logo with glow halo */}
{/* Wordmark */} TongueKeeper {/* Tagline */}

5,000+ languages are disappearing.

We're keeping the flame alive.

{/* CTAs */}