# Project export: Voices This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: What if you could practice your presentation with an audience that never judges you, an audience that always shows up, and an audience that asks the right questions to prepare you for the real thing? - Devpost: https://devpost.com/software/voices-hw23r0 - GitHub: https://github.com/dev-alto/voices-FINAL - Video: https://www.youtube.com/embed/-guKl11Q0eU?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 0 GitHub contributor(s) — ## Devpost submission (written by the team) No Devpost description available. ## README (from the GitHub repository) This is a [Next.js](https://nextjs.org) project bootstrapped with [`create-next-app`](https://nextjs.org/docs/app/api-reference/cli/create-next-app). ## ✨ New Feature: AI Interview Practice Practice interviews with an AI interviewer using real-time voice conversation! The system uses: - **Browser Speech Recognition** to capture your voice - **Google Gemini AI** as an intelligent interviewer - **Fish Audio TTS** to speak responses aloud See [IMPLEMENTATION_SUMMARY.md](./IMPLEMENTATION_SUMMARY.md) for complete details. **Quick Start**: 1. Set up your `.env.local` file (see `.env.example`) 2. Navigate to `/interview-practice` 3. Click "Start Interview" and begin speaking! --- ## Getting Started First, run the development server: ```bash npm run dev # or yarn dev # or pnpm dev # or bun dev ``` Open [http://localhost:3000](http://localhost:3000) with your browser to see the result. You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file. This project uses [`next/font`](https://nextjs.org/docs/app/building-your-application/optimizing/fonts) to automatically optimize and load [Geist](https://vercel.com/font), a new font family for Vercel. ## Learn More To learn more about Next.js, take a look at the following resources: - [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API. - [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial. You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js) - your feedback and contributions are welcome! ## Deploy on Vercel The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js. Check out our [Next.js deployment documentation](https://nextjs.org/docs/app/building-your-application/deploying) for more details. ## Detected evidence (automated analysis) Indexed codebase: 86 recognized source files, 561 KB. - CSS (language) — detected in the code - Google Gemini (technology) — detected in the code - Next.js (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (93 of 93) ``` .env.example .env.local .gitignore AI_RULES.md components.json environment.d.ts FIXES_APPLIED.md GEMINI_PRACTICE_INTEGRATION.md IMPLEMENTATION_SUMMARY.md INTERVIEW_PRACTICE.md LICENSE next-env.d.ts next.config.ts package.json postcss.config.mjs PRD-Template.md README.md src/app/api/analyze/flow/route.ts src/app/api/analyze/talking-points/route.ts src/app/api/extract-text/route.ts src/app/api/gemini/interview/route.ts src/app/api/tts/route.ts src/app/globals.css src/app/icon.tsx src/app/interview-practice/page.tsx src/app/layout.tsx src/app/opengraph-image.tsx src/app/page.tsx src/app/practice/[scenarioId]/page.tsx src/app/practice/[scenarioId]/setup/page.tsx src/app/practice/[scenarioId]/voice-analysis/page.tsx src/app/practice/[scenarioId]/voice/page.tsx src/app/scenarios/page.tsx src/components/animated-title.tsx src/components/IntroFade.tsx src/components/made-with-dyad.tsx src/components/navbar.tsx src/components/ui/accordion.tsx src/components/ui/alert-dialog.tsx src/components/ui/alert.tsx src/components/ui/aspect-ratio.tsx src/components/ui/avatar.tsx src/components/ui/badge.tsx src/components/ui/breadcrumb.tsx src/components/ui/button.tsx src/components/ui/calendar.tsx src/components/ui/card.tsx src/components/ui/carousel.tsx src/components/ui/chart.tsx src/components/ui/checkbox.tsx src/components/ui/collapsible.tsx src/components/ui/command.tsx src/components/ui/context-menu.tsx src/components/ui/dialog.tsx src/components/ui/drawer.tsx src/components/ui/dropdown-menu.tsx src/components/ui/form.tsx src/components/ui/hover-card.tsx src/components/ui/input-otp.tsx src/components/ui/input.tsx src/components/ui/label.tsx src/components/ui/menubar.tsx src/components/ui/navigation-menu.tsx src/components/ui/pagination.tsx src/components/ui/popover.tsx src/components/ui/progress.tsx src/components/ui/radio-group.tsx src/components/ui/resizable.tsx src/components/ui/scroll-area.tsx src/components/ui/select.tsx src/components/ui/separator.tsx src/components/ui/sheet.tsx src/components/ui/sidebar.tsx src/components/ui/skeleton.tsx src/components/ui/slider.tsx src/components/ui/sonner.tsx src/components/ui/switch.tsx src/components/ui/table.tsx src/components/ui/tabs.tsx src/components/ui/textarea.tsx src/components/ui/toggle-group.tsx src/components/ui/toggle.tsx src/components/ui/tooltip.tsx src/components/voice-analysis.tsx src/hooks/use-mobile.tsx src/lib/agent-responses.ts src/lib/scenarios.ts src/lib/tts-service.ts src/lib/utils.ts src/types/index.ts tailwind.config.ts tsconfig.json VOICE_PRACTICE_INTEGRATION.md ``` ### Dependencies - package.json: @dyad-sh/nextjs-webpack-component-tagger@^0.8.0, @google/generative-ai@^0.24.1, @hookform/resolvers@^5.0.1, @radix-ui/react-accordion@^1.2.11, @radix-ui/react-alert-dialog@^1.1.14, @radix-ui/react-aspect-ratio@^1.1.7, @radix-ui/react-avatar@^1.1.10, @radix-ui/react-checkbox@^1.3.2, @radix-ui/react-collapsible@^1.1.11, @radix-ui/react-context-menu@^2.2.15, @radix-ui/react-dialog@^1.1.14, @radix-ui/react-dropdown-menu@^2.1.15, @radix-ui/react-hover-card@^1.1.14, @radix-ui/react-label@^2.1.7, @radix-ui/react-menubar@^1.1.15, @radix-ui/react-navigation-menu@^1.2.13, @radix-ui/react-popover@^1.1.14, @radix-ui/react-progress@^1.1.7, @radix-ui/react-radio-group@^1.3.7, @radix-ui/react-scroll-area@^1.2.9, @radix-ui/react-select@^2.2.5, @radix-ui/react-separator@^1.1.7, @radix-ui/react-slider@^1.3.5, @radix-ui/react-slot@^1.2.3, @radix-ui/react-switch@^1.2.5, @radix-ui/react-tabs@^1.1.12, @radix-ui/react-toggle@^1.1.9, @radix-ui/react-toggle-group@^1.1.10, @radix-ui/react-tooltip@^1.2.7, @types/node@^20, @types/pdf-parse@^1.1.5, @types/react@^19, @types/react-dom@^19, class-variance-authority@^0.7.1, clsx@^2.1.1, cmdk@^1.1.1, date-fns@^3.6.0, embla-carousel-react@^8.6.0, input-otp@^1.4.2, lucide-react@^0.511.0, motion@^12.23.24, next@15.3.4, next-themes@^0.4.6, pdf-lib@^1.17.1, pdf-parse@^2.4.5, pdfjs-dist@^5.4.296, postcss@^8, react@^19.0.0, react-day-picker@^8.10.1, react-dom@^19.0.0, react-hook-form@^7.56.4, react-resizable-panels@^3.0.2, recharts@^2.15.3, sonner@^2.0.3, tailwind-merge@^3.3.0, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5, vaul@^1.1.2, zod@^3.25.28 ### Recent commits (newest first) - feat: delete - first commit ## Key source files (fetched from GitHub, selected and truncated for size) ### AI_RULES.md ```markdown # AI Development Rules This document outlines the technology stack and specific library usage guidelines for this Next.js application. Adhering to these rules will help maintain consistency, improve collaboration, and ensure the AI assistant can effectively understand and modify the codebase. ## Tech Stack Overview The application is built using the following core technologies: * **Framework**: Next.js (App Router) * **Language**: TypeScript * **UI Components**: Shadcn/UI - A collection of re-usable UI components built with Radix UI and Tailwind CSS. * **Styling**: Tailwind CSS - A utility-first CSS framework for rapid UI development. * **Icons**: Lucide React - A comprehensive library of simply beautiful SVG icons. * **Forms**: React Hook Form for managing form state and validation, typically with Zod for schema validation. * **State Management**: Primarily React Context API and built-in React hooks (`useState`, `useReducer`). * **Notifications/Toasts**: Sonner for displaying non-intrusive notifications. * **Charts**: Recharts for data visualization. * **Animation**: `tailwindcss-animate` and animation capabilities built into Radix UI components. ## Library Usage Guidelines To ensure consistency and leverage the chosen stack effectively, please follow these rules: 1. **UI Components**: * **Primary Choice**: Always prioritize using components from the `src/components/ui/` directory (Shadcn/UI components). * **Custom Components**: If a required component is not available in Shadcn/UI, create a new component in `src/components/` following Shadcn/UI's composition patterns (i.e., building on Radix UI primitives and styled with Tailwind CSS). * **Avoid**: Introducing new, third-party UI component libraries without discussion. 2. **Styling**: * **Primary Choice**: Exclusively use Tailwind CSS utility classes for all styling. * **Global Styles**: Reserve `src/app/globals.css` for base Tailwind directives, global CSS variable definitions, and minimal base styling. Avoid adding component-specific styles here. * **CSS-in-JS**: Do not use CSS-in-JS libraries (e.g., Styled Components, Emotion). 3. **Icons**: * **Primary Choice**: Use icons from the `lucide-react` library. 4. **Forms**: * **Management**: Use `react-hook-form` for all form logic (state, validation, submission). * **Validation**: Use `zod` for schema-based validation with `react-hook-form` via `@hookform/resolvers`. 5. **State Management**: * **Local State**: Use React's `useState` and `useReducer` hooks for component-level state. * **Shared/Global State**: For state shared between multiple components, prefer React Context API. * **Complex Global State**: If application state becomes significantly complex, discuss the potential introduction of a dedicated state management library (e.g., Zustand, Jotai) before implementing. 6. **Routing**: * Utilize the Next.js App Router (file-sys [truncated — 1352 more characters] ``` ### PRD-Template.md ```markdown --- title: Product Requirements Document app: calhacks-2025 created: 2025-10-25T09:47:38.815Z version: 1 --- # calhacks-2025 - Product Requirements Document ## Executive Summary **Product Vision:** [Describe your product vision here] **Core Purpose:** [What problem does this solve?] **Target Users:** [Who will use this product?] **Key Features:** - [Feature 1] - [Feature 2] - [Feature 3] **Complexity Assessment:** Simple - **State Management:** Local - **External Integrations:** 0 - **Business Logic:** Simple - **Data Synchronization:** None **MVP Success Metrics:** - Users can complete core workflow - System handles expected concurrent users - Core features work without errors ## 1. USERS & PERSONAS **Primary Persona:** - **Name:** [Primary User] - **Context:** [Their situation] - **Goals:** [What they want to achieve] - **Needs:** [Specific to this product] ## 2. FUNCTIONAL REQUIREMENTS ### 2.1 User-Requested Features **FR-001: [Core Feature Name]** - **Description:** [What it does functionally] - **Entity Type:** User-Generated Content - **User Benefit:** [Why user wants this] - **Primary User:** Primary Persona - **Lifecycle Operations:** - **Create:** [How users create] - **View:** [How users view/access] - **Edit:** [How users modify] - **Delete:** [How users remove] - **List/Search:** [How users find] - **Acceptance Criteria:** - [ ] Given [context], when user creates [entity], then [expected result] - [ ] Given [entity exists], when user views it, then [what they see] - [ ] Given [entity exists], when user edits it, then [what happens] - [ ] Given [entity exists], when user deletes it, then [what happens] - [ ] Users can search/filter their [entities] by [criteria] ### 2.2 Essential Market Features **FR-002: User Authentication** - **Description:** Secure user login and session management - **Entity Type:** Configuration/System - **User Benefit:** Protects user data and personalizes experience - **Primary User:** All personas - **Lifecycle Operations:** - **Create:** Register new account - **View:** View profile information - **Edit:** Update profile and preferences - **Delete:** Account deletion option (with data export) - **Additional:** Password reset, session management - **Acceptance Criteria:** - [ ] Given valid credentials, when user logs in, then access is granted - [ ] Given invalid credentials, when user attempts login, then access is denied with clear error - [ ] Users can reset forgotten passwords - [ ] Users can update their profile information - [ ] Users can delete their account (with confirmation and data export option) ## 3. USER WORKFLOWS ### 3.1 Primary Workflow: [Main User Journey] - **Trigger:** [What starts this workflow] - **Outcome:** [What user achieves] - **Steps:** 1. User [does something] 2. System [responds with] 3. User sees [result] 4. User [next action] 5. System [validates/processes] 6. User receives [feedback] ### 3.2 Entity Management [truncated — 4132 more characters] ``` ### package.json ``` { "name": "next-template", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@google/generative-ai": "^0.24.1", "@hookform/resolvers": "^5.0.1", "@radix-ui/react-accordion": "^1.2.11", "@radix-ui/react-alert-dialog": "^1.1.14", "@radix-ui/react-aspect-ratio": "^1.1.7", "@radix-ui/react-avatar": "^1.1.10", "@radix-ui/react-checkbox": "^1.3.2", "@radix-ui/react-collapsible": "^1.1.11", "@radix-ui/react-context-menu": "^2.2.15", "@radix-ui/react-dialog": "^1.1.14", "@radix-ui/react-dropdown-menu": "^2.1.15", "@radix-ui/react-hover-card": "^1.1.14", "@radix-ui/react-label": "^2.1.7", "@radix-ui/react-menubar": "^1.1.15", "@radix-ui/react-navigation-menu": "^1.2.13", "@radix-ui/react-popover": "^1.1.14", "@radix-ui/react-progress": "^1.1.7", "@radix-ui/react-radio-group": "^1.3.7", "@radix-ui/react-scroll-area": "^1.2.9", "@radix-ui/react-select": "^2.2.5", "@radix-ui/react-separator": "^1.1.7", "@radix-ui/react-slider": "^1.3.5", "@radix-ui/react-slot": "^1.2.3", "@radix-ui/react-switch": "^1.2.5", "@radix-ui/react-tabs": "^1.1.12", "@radix-ui/react-toggle": "^1.1.9", "@radix-ui/react-toggle-group": "^1.1.10", "@radix-ui/react-tooltip": "^1.2.7", "@types/pdf-parse": "^1.1.5", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "cmdk": "^1.1.1", "date-fns": "^3.6.0", "embla-carousel-react": "^8.6.0", "input-otp": "^1.4.2", "lucide-react": "^0.511.0", "motion": "^12.23.24", "next": "15.3.4", "next-themes": "^0.4.6", "pdf-lib": "^1.17.1", "pdf-parse": "^2.4.5", "pdfjs-dist": "^5.4.296", "react": "^19.0.0", "react-day-picker": "^8.10.1", "react-dom": "^19.0.0", "react-hook-form": "^7.56.4", "react-resizable-panels": "^3.0.2", "recharts": "^2.15.3", "sonner": "^2.0.3", "tailwind-merge": "^3.3.0", "tailwindcss-animate": "^1.0.7", "vaul": "^1.1.2", "zod": "^3.25.28" }, "devDependencies": { "@dyad-sh/nextjs-webpack-component-tagger": "^0.8.0", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "^5" } } ``` ### src/app/layout.tsx ```typescript import type { Metadata } from "next"; import { Inter } from "next/font/google"; import "./globals.css"; const inter = Inter({ subsets: ['latin'], variable: '--font-inter', }); export const metadata: Metadata = { title: "Public Speaking Practice Platform", description: "Practice public speaking with AI-powered scenarios", icons: { icon: 'data:image/svg+xml,🎤', }, }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### src/types/index.ts ```typescript export type DifficultyLevel = 'easy' | 'medium' | 'hard'; export type Vibe = 'casual' | 'academic' | 'professional' | 'tense' | 'formal'; export type ScenarioType = | 'party' | 'classroom' | 'job-interview' | 'de-escalation' | 'presentation'; export interface Scenario { id: string; title: string; description: string; type: ScenarioType; participantCount: number; duration: number; // in seconds icon: string; difficulty: DifficultyLevel; basePrompt: string; vibe: Vibe; presentational: boolean; } export interface Agent { id: string; name: string; personality: string; avatar: string; role?: string; voiceId?: string; emotionPrefix?: string; } export interface Message { id: string; agentId: string; agentName: string; content: string; timestamp: Date; isUser?: boolean; } export interface PracticeSession { scenarioId: string; startTime: Date; messages: Message[]; score: number; difficulty: DifficultyLevel; } // Presentational flow types for structured presentation plans export interface FlowSection { id: string; title: string; goals: string[]; } export interface PresentationalFlow { intro: FlowSection; sections: FlowSection[]; conclusion: FlowSection; qa: FlowSection; } ``` ### src/app/page.tsx ```typescript import Link from 'next/link'; import { Button } from '@/components/ui/button'; import { Mic, Users, TrendingUp, Zap } from 'lucide-react'; import { AnimatedTitle } from '@/components/animated-title'; import { IntroFade } from '@/components/IntroFade'; import { Navbar } from '@/components/navbar'; export default function HomePage() { return (
{/* Video Background */} {/* Overlay to darken video */}
{/* Content */}
{/* Hero Section */}
{/* Announcement Banner */}
Practice real-world social interactions with AI-powered agents. Read more →

Practice real-world social interactions with AI-powered agents. Build confidence in any scenario, from parties to presentations.

{/* Features Grid */}

Realistic Scenarios

Practice with AI agents in job interviews, presentations, parties, and more

Track Progress

Get scored on your performance and see your improvement over time

Adaptive Difficulty

Choose your challenge level and face increasingly complex interactions

{/* Hero Image - Practice Session Mockup */}
{/* Mockup container */}
{/* Top bar */}
{/* Main content area - 3 column layout */}
{/* Left sidebar - User video & controls */}
{/* User video */}
You
{/* Control buttons */}
[truncated — 3584 more characters] ``` ### src/app/scenarios/page.tsx ```typescript import Link from 'next/link'; import { Button } from '@/components/ui/button'; import { Card, CardHeader, CardTitle, CardDescription, CardContent } from '@/components/ui/card'; import { scenarios } from '@/lib/scenarios'; import { ArrowLeft, Clock, Users, Mic } from 'lucide-react'; import { IntroFade } from '@/components/IntroFade'; export default function ScenariosPage() { return (
{/* Video Background */} {/* Overlay to darken video */}
{/* Content */}
{/* Header */}

Choose Your Scenario

Select a practice scenario to begin improving your public speaking skills

{/* Quick Access to Interview Practice */}
AI Interview Practice (Voice) Practice with an AI interviewer using real-time voice conversation
{/* Scenarios Grid */}
{scenarios.map((scenario) => (
{scenario.icon}
{scenario.title} {scenario.difficulty.toUpperCase()} {scenario.description}
{scenario.participantCount} participants
{Math.floor(scenario.duration / 60)} min
))}
); } ``` ### src/app/interview-practice/page.tsx ```typescript "use client"; import { useState, useEffect, useRef } from "react"; import { useRouter } from "next/navigation"; import { Button } from "@/components/ui/button"; import { Card, CardContent, CardHeader, CardTitle, CardDescription } from "@/components/ui/card"; import { Textarea } from "@/components/ui/textarea"; import { Label } from "@/components/ui/label"; import { Switch } from "@/components/ui/switch"; import { ttsService } from "@/lib/tts-service"; import { Mic, MicOff, Volume2, VolumeX, ArrowLeft, MessageSquare, User, Bot, Loader2, AlertCircle, PlayCircle, StopCircle } from "lucide-react"; // Browser Speech Recognition types declare global { interface Window { SpeechRecognition: any; webkitSpeechRecognition: any; } } interface ConversationMessage { role: 'user' | 'model'; parts: [{ text: string }]; } interface DisplayMessage { id: string; role: 'user' | 'assistant'; content: string; timestamp: Date; } export default function InterviewPracticePage() { const router = useRouter(); // Speech Recognition State const [isListening, setIsListening] = useState(false); const [speechSupported, setSpeechSupported] = useState(false); const [interimTranscript, setInterimTranscript] = useState(""); const [speechError, setSpeechError] = useState(null); // Conversation State const [messages, setMessages] = useState([]); const [conversationHistory, setConversationHistory] = useState([]); const [isProcessing, setIsProcessing] = useState(false); const [sessionActive, setSessionActive] = useState(false); // TTS State const [ttsEnabled, setTtsEnabled] = useState(() => { if (typeof window !== 'undefined') { const saved = localStorage.getItem("interview-tts-enabled"); return saved ? JSON.parse(saved) : true; } return true; }); // Custom System Prompt State const [customSystemPrompt, setCustomSystemPrompt] = useState(""); const [useCustomPrompt, setUseCustomPrompt] = useState(false); // Refs const recognitionRef = useRef(null); const lastTranscriptTimeRef = useRef(Date.now()); const messagesEndRef = useRef(null); const processingTimeoutRef = useRef(null); // Check for Speech Recognition support useEffect(() => { const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition; if (SpeechRecognition) { setSpeechSupported(true); } else { setSpeechSupported(false); setSpeechError("Speech recognition is not supported in your browser. Please use Chrome or Edge."); } }, []); // Update TTS service when enabled state changes useEffect(() => { ttsService.setEnabled(ttsEnabled); localStorage.setItem("interview-tts-enabled", JSON.stringify(ttsEnabled)); }, [ttsEnabled]); // Auto-scroll to bottom of messages useEffect(() => { messagesEndRef.current?.scrollIntoView({ behavior: "smooth" }); }, [messages]); // Cleanup on unmount useEffect(() => { return () => { if (recognitionRef.current) { recognitionRef.current.stop(); } if (processingTimeoutRef.current) { clearTimeout(processingTimeoutRef.current); } ttsService.clearQueue(); }; }, []); // Initialize Speech Recognition const initializeSpeechRecognition = () => { const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition; if (!SpeechRecognition) return null; const recognition = new SpeechRecognition(); recognition.continuous = true; recognition.interimResults = true; recognition.lang = "en-US"; recognition.maxAlternatives = 1; recognition.onstart = () => { console.log("Speech recognition started"); setIsListening(true); setSpeechError(null); }; recognition.onresult = (event: any) => { let interimText = ""; let finalText = ""; for (let i = event.resultIndex; i < event.results.length; i++) { const transcript = event.results[i][0].transcript; if (event.results[i].isFinal) { finalText += transcript + " "; } else { interimText += transcript; } } setInterimTranscript(interimText); if (finalText.trim()) { const now = Date.now(); // Debounce: only process if enough time has passed if (now - lastTranscriptTimeRef.current > 1500) { handleUserSpeech(finalText.trim()); lastTranscriptTimeRef.current = now; } } }; recognition.onerror = (event: any) => { console.error("Speech recognition error:", event.error); if (event.error === "no-speech") { setSpeechError("No speech detected. Please speak clearly."); } else if (event.error === "aborted") { setSpeechError("Speech recognition was aborted."); } else if (event.error === "not-allowed") { setSpeechError("Microphone access denied. Please allow microphone access."); } else { setSpeechError(`Speech recognition error: ${event.error}`); } }; recognition.onend = () => { console.log("Speech recognition ended"); setIsListening(false); setInterimTranscript(""); // Restart if session is still active if (sessionActive && recognitionRef.current) { try { recognitionRef.current.start(); } catch (error) { console.error("Failed to restart recognition:", error); } } }; return recognition; }; // Handle user speech input const handleUserSpeech = async (text: string) => { if (!text.trim() || isProcessing) return; const userMessage: DisplayMessage = { id: `user-${Date.now()}`, role: 'user', content: text, timestamp: new Date() }; setMessages(prev => [...prev, userMessage]); setIsProcessing(true); try { [truncated — 13825 more characters] ``` ### src/app/api/tts/route.ts ```typescript // TODO: // add different voices per person by adding voiceId (see reference_id below) in agent/scenarios // add tones for specific agents by prefixing their emotion control to messages // ex for agent personality 'Adam', {name: 'Adam' personality: '(quiet) (sad)'} ---> message: (quiet) (sad) Blah blah blah // adding random yawning import { NextRequest, NextResponse } from 'next/server'; const endpoint = process.env.TTS_ENDPOINT const ttsEnabled = process.env.TTS_ENABLED const headers = { 'Authorization': 'Bearer ' + process.env.TTS_FISH_AUDIO_API_KEY, 'Content-Type': 'application/json', 'model': 's1' } export async function POST(request: NextRequest) { try { if (!ttsEnabled) { return NextResponse.json( { error: 'TTS is disabled' }, { status: 500 } ); } if (!endpoint) { return NextResponse.json( { error: 'TTS endpoint not configured' }, { status: 500 } ); } // Parse request body const body = await request.json(); const { text, agentName, voiceId } = body; if (!text) { return NextResponse.json( { error: 'Text is required' }, { status: 400 } ); } // Prepare the request body for the external TTS API const ttsRequestBody = { "text": text, "format": "mp3", "reference_id": voiceId || "bf322df2096a46f18c579d0baa36f41d" } // Make request to external TTS API const response = await fetch(endpoint, { method: 'POST', headers: headers, body: JSON.stringify(ttsRequestBody), }); if (!response.ok) { console.error(`TTS API error: ${response.status} ${response.statusText}`); return NextResponse.json( { error: `TTS API error: ${response.status}` }, { status: response.status } ); } // Get the audio blob from the TTS API const audioBlob = await response.blob(); // Return the audio blob to the client return new NextResponse(audioBlob, { status: 200, headers: { 'Content-Type': response.headers.get('Content-Type') || 'audio/mpeg', 'Content-Length': audioBlob.size.toString(), }, }); } catch (error) { console.error('Error in TTS API route:', error); return NextResponse.json( { error: 'Internal server error' }, { status: 500 } ); } } ``` ### src/app/api/extract-text/route.ts ```typescript import { NextRequest, NextResponse } from "next/server"; import { GoogleGenerativeAI } from "@google/generative-ai"; export const runtime = "nodejs"; // Initialize Gemini API const genAI = new GoogleGenerativeAI("AIzaSyAttsNk7ExqMPfTKwzZK7UImfWW9sbEHpM"); export async function POST(request: NextRequest) { try { // API key is now hardcoded const form = await request.formData(); const file = form.get("file"); if (!file || !(file instanceof File)) { return NextResponse.json( { error: "A 'file' field is required" }, { status: 400 } ); } const filename = (file as File).name || "upload.file"; const contentType = (file as File).type || ""; // Check supported file types const isPdf = contentType === "application/pdf" || /\.pdf$/i.test(filename); const isPptx = contentType === "application/vnd.openxmlformats-officedocument.presentationml.presentation" || /\.pptx$/i.test(filename); const isImage = contentType.startsWith("image/") || /\.(jpg|jpeg|png|gif|webp)$/i.test(filename); const isText = contentType.startsWith("text/") || /\.(txt|md|csv)$/i.test(filename); // For text files, just read directly if (isText) { const text = await file.text(); return NextResponse.json({ text, meta: { pages: 1, chars: text.length, fileType: 'text' }, }); } // For other file types, use Gemini API try { const arrayBuffer = await file.arrayBuffer(); const bytes = new Uint8Array(arrayBuffer); // Get the appropriate model - using Gemini 2.0 Flash (latest available) const model = genAI.getGenerativeModel({ model: "gemini-2.0-flash-exp" // Latest Gemini Flash model }); // Prepare the file for Gemini const base64Data = Buffer.from(bytes).toString('base64'); let mimeType = contentType; if (!mimeType) { // Infer MIME type from extension if not provided if (isPdf) mimeType = "application/pdf"; else if (isPptx) mimeType = "application/vnd.openxmlformats-officedocument.presentationml.presentation"; else if (/\.png$/i.test(filename)) mimeType = "image/png"; else if (/\.jpe?g$/i.test(filename)) mimeType = "image/jpeg"; else mimeType = "application/octet-stream"; } // Create the prompt based on file type let prompt = ""; if (isPdf || isPptx) { prompt = `Please extract and return ALL the text content from this document. Include all slides, pages, headers, bullet points, and any text visible in the document. If there are images with text, describe them briefly. Format the output as plain text, maintaining the document's structure with clear separations between sections/slides. Do not summarize - extract everything.`; } else if (isImage) { prompt = `Please extract any text visible in this image. If there is no text, describe what you see in the image. If there are charts or diagrams, describe their content.`; } else { prompt = `Please extract and return all text content from this file.`; } // Call Gemini API const result = await model.generateContent([ prompt, { inlineData: { mimeType: mimeType, data: base64Data } } ]); const response = await result.response; const extractedText = response.text(); // Estimate pages (rough approximation) const estimatedPages = isPdf || isPptx ? Math.max(1, Math.ceil(extractedText.length / 3000)) : 1; return NextResponse.json({ text: extractedText, meta: { pages: estimatedPages, chars: extractedText.length, fileType: isPdf ? 'pdf' : isPptx ? 'pptx' : isImage ? 'image' : 'other', processedBy: 'gemini-2.0-flash-exp' }, }); } catch (geminiError: any) { console.error("Gemini API error:", geminiError); // Provide more specific error messages if (geminiError.message?.includes("API key")) { return NextResponse.json( { error: "Invalid Gemini API key. Please check your GEMINI_API_KEY environment variable." }, { status: 401 } ); } else if (geminiError.message?.includes("quota")) { return NextResponse.json( { error: "Gemini API quota exceeded. Please try again later." }, { status: 429 } ); } else if (geminiError.message?.includes("File too large")) { return NextResponse.json( { error: "File is too large for processing. Please use a smaller file." }, { status: 413 } ); } return NextResponse.json( { error: `Failed to process document with Gemini: ${geminiError.message || 'Unknown error'}` }, { status: 500 } ); } } catch (error) { console.error("Error in extract-text API route:", error); return NextResponse.json( { error: "Failed to extract text from document" }, { status: 500 } ); } } ``` [72 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]