# Project export: Lumina Vision This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: An AI-powered neural assistant that narrate the world for the visually impaired, transforming visual scenes into immersive audio stories with real-time safety monitoring and haptic alerts. - Devpost: https://devpost.com/software/lumina-vision - GitHub: https://github.com/Alpha04-ash/lumina_vision - Demo: https://lumina-vision-sy2b.vercel.app/ - Team: 1 GitHub contributor(s) โ€” abuwwww (4 commits) ## Devpost submission (written by the team) No Devpost description available. ## README (from the GitHub repository) # Lumina Vision: Neural Narrative & Safety Assistant ![Lumina Vision Thumbnail](public/neural-link.png) ## ๐ŸŒŸ Inspiration The primary inspiration for **Lumina Vision** came from a simple question: *Can we use high-speed AI to narrate the visual world for those who cannot see it?* Current accessibility tools are often slow, robotic, or siloed into specific apps. We built a "Neural Link"โ€”a seamless, high-performance assistant that doesn't just list objects, but describes the world with narrative depth, detects immediate physical hazards, and provides tactile feedback through haptics. ## ๐Ÿš€ Features - **Narrative Intelligence**: Uses GPT-4o Vision to provide rich, descriptive narratives of surroundings. - **Safety Guardian**: Real-time hazard detection that classifies risks (Safe, Caution, DANGER) and triggers haptic (vibration) alerts. - **Low-Latency Architecture**: Client-side image compression and streaming APIs ensure responses feel instantaneous. - **Neural Personalization**: Toggle between "Professional" and "Warm" voice profiles to suit the user's environment. - **Offline-First PWA**: Can be installed as a native app for a browser-free, immersive experience. ## ๐Ÿ›  Built With Lumina is built on a high-performance **Clean Architecture** for modern web apps: - **Framework**: [Next.js](https://nextjs.org) with App Router. - **AI Brain**: [OpenAI GPT-4o](https://openai.com) for vision analysis and real-time streaming. - **Voice Synthesis**: OpenAI TTS-1 with dynamic voice profile switching. - **UI/UX**: Custom "Neural Link" dark-mode aesthetic built with Tailwind CSS v4 and Lucide icons. - **Sensory Hub**: Web Audio API for native sound effects and the Vibration API for haptic feedback. ## ๐Ÿ“ฆ Getting Started ### Prerequisites - Node.js 18+ - OpenAI API Key ### Installation 1. Clone the repository: ```bash git clone https://github.com/Alpha04-ash/lumina_vision.git ``` 2. Install dependencies: ```bash npm install ``` 3. Set up environment variables: Create a `.env.local` file and add your key: ```env OPENAI_API_KEY=your_key_here ``` 4. Run the development server: ```bash npm run dev ``` ## ๐Ÿง  What we learned We learned that the difference between an "app" and an "assistant" is in the detailsโ€”haptics, sound cues, and streaming responses aren't just "polish," they are the core experience for accessibility-focused software. ## ๐Ÿ”ฎ What's next - **Object Tracking**: Active spatial tracking with stereo-audio to help users find specific items. - **Multi-Modal Memory**: Remembering the layout of a room to provide "Spatial Navigation" instructions. ## Detected evidence (automated analysis) Indexed codebase: 25 recognized source files, 87 KB. - CSS (language) โ€” detected in the code - Next.js (technology) โ€” detected in the code - OpenAI (technology) โ€” detected in the code - React (technology) โ€” detected in the code - SQL (language) โ€” detected in the code - Supabase (technology) โ€” detected in the code - Tailwind CSS (technology) โ€” detected in the code - TypeScript (language) โ€” detected in the code - Vercel (technology) โ€” claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (32 of 32) ``` .gitignore eslint.config.mjs next.config.ts package.json postcss.config.mjs public/manifest.json README.md src/app/api/chat/route.ts src/app/api/tts/route.ts src/app/api/vision/route.ts src/app/globals.css src/app/layout.tsx src/app/narrator/.page.tsx.swp src/app/narrator/page.tsx src/app/page.tsx src/components/narrator/CameraView.tsx src/components/narrator/ChatStream.tsx src/components/narrator/HistorySidebar.tsx src/components/narrator/NarratorHeader.tsx src/components/ui/button.tsx src/components/ui/card.tsx src/hooks/useCamera.ts src/hooks/useNarratorHistory.ts src/hooks/useSoundEffects.ts src/hooks/useSpeech.ts src/hooks/useVoiceRecognition.ts src/lib/logger.ts src/lib/supabase.ts src/lib/utils.ts src/types/narrator.ts supabase_schema.sql tsconfig.json ``` ### Dependencies - package.json: @radix-ui/react-slot@^1.2.4, @supabase/supabase-js@^2.95.3, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, babel-plugin-react-compiler@1.0.0, class-variance-authority@^0.7.1, clsx@^2.1.1, eslint@^9, eslint-config-next@16.1.6, framer-motion@^12.34.0, lucide-react@^0.564.0, next@16.1.6, openai@^6.22.0, react@19.2.3, react-dom@19.2.3, react-markdown@^10.1.0, remark-gfm@^4.0.1, tailwind-merge@^3.4.0, tailwindcss@^4, typescript@^5 ### Recent commits (newest first) - ๐Ÿš€ Feature: Supabase Cloud Sync & Persistent History Integration - ๐Ÿ”ง Meta: Standardize viewport/theme-color for clean deployment - ๐Ÿš€ Initial Commit: Lumina Vision Hackathon Edition - Initial commit from Create Next App ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "lumina", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint" }, "dependencies": { "@radix-ui/react-slot": "^1.2.4", "@supabase/supabase-js": "^2.95.3", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "framer-motion": "^12.34.0", "lucide-react": "^0.564.0", "next": "16.1.6", "openai": "^6.22.0", "react": "19.2.3", "react-dom": "19.2.3", "react-markdown": "^10.1.0", "remark-gfm": "^4.0.1", "tailwind-merge": "^3.4.0" }, "devDependencies": { "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "babel-plugin-react-compiler": "1.0.0", "eslint": "^9", "eslint-config-next": "16.1.6", "tailwindcss": "^4", "typescript": "^5" } } ``` ### src/app/layout.tsx ```typescript import type { Metadata } from "next"; import { Geist, Geist_Mono } from "next/font/google"; import "./globals.css"; const geistSans = Geist({ variable: "--font-geist-sans", subsets: ["latin"], }); const geistMono = Geist_Mono({ variable: "--font-geist-mono", subsets: ["latin"], }); export const metadata: Metadata = { title: "Lumina Vision | Neural Assistant", description: "AI-Powered Narrative Assistant for the Visually Impaired", manifest: "/manifest.json", appleWebApp: { capable: true, statusBarStyle: "black-translucent", title: "Lumina", }, }; export const viewport = { themeColor: "#4f46e5", width: "device-width", initialScale: 1, maximumScale: 1, userScalable: false, }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### src/app/page.tsx ```typescript "use client"; import { useState, useEffect } from "react"; import Link from "next/link"; import { Button } from "@/components/ui/button"; import { ArrowRight, Eye, Mic, ShieldCheck, Zap, Bot } from "lucide-react"; import { motion, Variants } from "framer-motion"; const containerVariants: Variants = { hidden: { opacity: 0 }, visible: { opacity: 1, transition: { staggerChildren: 0.15, delayChildren: 0.2 } } }; const itemVariants: Variants = { hidden: { opacity: 0, y: 30 }, visible: { opacity: 1, y: 0, transition: { type: "spring", stiffness: 100, damping: 20 } } }; export default function Home() { const [mounted, setMounted] = useState(false); useEffect(() => { setMounted(true); }, []); // Return a simple version if not mounted to prevent hydration mismatch if (!mounted) { return (
); } return (
{/* Background Gradients */} {/* Badge */}
Grand Prize Contender 2026
v2.0 Update
{/* Hero Text */} See the world
through AI eyes. Lumina is an advanced visual assistant that narrates your surroundings in real-time. Designed for the visually impaired, powered by multimodal AI. {/* CTAs */}

Free for all TreeHacks participants

 {/* Neural Scroll Indicator */}
Scroll to Explore {/* How it Works / Instructions */} ([]); const [input, setInput] = useState(""); const [analyzing, setAnalyzing] = useState(false); const [safetyMode, setSafetyMode] = useState(false); const [autoSpeak, setAutoSpeak] = useState(false); const [showChatMobile, setShowChatMobile] = useState(false); const [error, setError] = useState(null); const [copiedIndex, setCopiedIndex] = useState(null); const [isStreaming, setIsStreaming] = useState(false); const lastAction = useRef<(() => void) | null>(null); // Initial hydration fix useEffect(() => { setMounted(true); }, []); // 1. Logic Hooks const { videoRef, canvasRef, facingMode, toggleCamera, cameraError, takeCompressedPicture } = useCamera(); const { speak, stopSpeaking, playingIndex, loadingVoiceIndex, voice, setVoice } = useSpeech(autoSpeak); const { history, currentSessionId, showHistory, setShowHistory, startNewSession, loadSession, deleteSession, syncStatus } = useNarratorHistory(messages, setMessages); const { playCaptureSound, playAlertSound, playMessageSound, playSuccessSound } = useSoundEffects(); // 2. Action Logic const captureAndAnalyze = useCallback(async (mode: "narrator" | "safety" = "narrator") => { const imageData = takeCompressedPicture(); if (!imageData) return; if (mode === "narrator") { setAnalyzing(true); playCaptureSound(); logger.info("Initiating scene analysis..."); lastAction.current = () => captureAndAnalyze("narrator"); } setError(null); try { const res = await fetch("/api/vision", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ image: imageData, mode }), }); const data = await res.json(); if (!res.ok) throw new Error(data.error || "Analysis failed"); if (data.description) { if (mode === "safety") { if (data.description.toLowerCase() !== "safe") { playAlertSound(); logger.warn("Safety hazard detected", { description: data.description }); speak(data.description); setMessages(prev => [...prev, { role: "assistant", content: "โš ๏ธ " + data.description }]); } } else { playSuccessSound(); logger.info("Analysis complete", { length: data.description.length }); speak(data.description); setMessages(prev => [...prev, { role: "assistant", content: data.description }]); } } } catch (err: any) { logger.error("Vision Analysis Error", { error: err.message }); setError(err.message || "Failed to connect to AI."); if (mode === "narrator") speak("System error. Check connection."); } finally { if (mode === "narrator") { setAnalyzing(false); if (window.innerWidth < 768) setShowChatMobile(true); } } }, [takeCompressedPicture, speak, playCaptureSound, playAlertSound, playSuccessSound]); const sendMessage = useCallback(async (textOverride?: string) => { const textToSend = textOverride || input; if (!textToSend.trim() || isStreaming) return; playMessageSound(); const userMessage: Message = { role: "user", content: textToSend }; const newMessages: Message[] = [...messages, userMessage]; setMessages(newMessages); setInput(""); setError(null); setIsStreaming(true); const currentImage = takeCompressedPicture(); logger.info("Sending message with context", { hasImage: !!currentImage }); lastAction.current = () => sendMessage(textOverride); try { const res = await fetch("/api/chat", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ messages: newMessages, image: currentImage }), }); if (!res.ok) { const data = await res.json(); throw new Error(data.error || "Chat failed"); } // Handle Streaming Response const reader = res.body?.getReader(); const decoder = new TextDecoder(); if (!reader) throw new Error("Stream not available"); let fullReply = ""; let chunkCount = 0; // Pre-create the assistant message to stream into it setMessages(prev => [...prev, { role: "assistant", content: "" }]); while (true) { const { done, value } = await reader.read(); if (done) break; const chunk = decoder.decode(value [truncated โ€” 6833 more characters] ``` ### src/app/api/tts/route.ts ```typescript import { OpenAI } from "openai"; import { NextRequest, NextResponse } from "next/server"; export async function POST(req: NextRequest) { try { const { text, voice = "shimmer" } = await req.json(); if (!process.env.OPENAI_API_KEY) { return NextResponse.json({ error: "Missing OpenAI API Key" }, { status: 401 }); } const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, }); if (!text) { return NextResponse.json( { error: "Text is required" }, { status: 400 } ); } const mp3 = await openai.audio.speech.create({ model: "tts-1", voice: voice as any, input: text, }); const buffer = Buffer.from(await mp3.arrayBuffer()); return new NextResponse(buffer, { headers: { "Content-Type": "audio/mpeg", "Content-Length": buffer.length.toString(), }, }); } catch (error: any) { console.error("TTS API Error:", error); return NextResponse.json( { error: error.message || "Failed to generate speech" }, { status: 500 } ); } } ``` ### src/app/api/vision/route.ts ```typescript import { OpenAI } from "openai"; import { NextRequest, NextResponse } from "next/server"; export async function POST(req: NextRequest) { try { const { image, mode = "narrator" } = await req.json(); if (!process.env.OPENAI_API_KEY) { return NextResponse.json({ error: "Missing OpenAI API Key. Please add it to your .env.local file." }, { status: 401 }); } const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, }); if (!image) { return NextResponse.json( { error: "Image data is required" }, { status: 400 } ); } let systemPrompt = "You are Lumina, an intelligent assistant for the visually impaired. Analyze the image provided and describe the scene. Focus on key elements such as obstacles, people, text, and safety hazards. Be concise but descriptive. Speak as if you are guiding the user in real-time."; if (mode === "safety") { systemPrompt = `You are a SAFETY GUARDIAN for a blind person. Analyze the image for IMMEDIATE HAZARDS in their direct path. CLASSIFICATION & RESPONSE RULES: 1. If NO HAZARDS: Reply ONLY with "Safe". 2. If MINOR HAZARDS (Uneven ground, distant obstacle): Reply "Caution: [Hazard]" 3. If IMMEDIATE DANGER (Stairs, moving vehicle, low-hanging branch): Reply "STOP! [Hazard] ahead!" Be extremely concise. Use only the exact phrases above. Priority 1 is safety. Priority 2 is brevity.`; } const response = await openai.chat.completions.create({ model: "gpt-4o", messages: [ { role: "system", content: systemPrompt, }, { role: "user", content: [ { type: "text", text: mode === "safety" ? "Scan for hazards." : "What is in front of me?", }, { type: "image_url", image_url: { url: image, // Expecting data:image/jpeg;base64,... }, }, ], }, ], max_tokens: 300, }); const description = response.choices[0].message.content; return NextResponse.json({ description }); } catch (error: any) { console.error("Vision API Error:", error); const errorMessage = error.error?.message || error.message || "Failed to analyze image"; return NextResponse.json( { error: errorMessage }, { status: 500 } ); } } ``` ### src/app/api/chat/route.ts ```typescript import { OpenAI } from "openai"; import { NextRequest } from "next/server"; export const runtime = "edge"; export async function POST(req: NextRequest) { try { const { messages, image } = await req.json(); if (!process.env.OPENAI_API_KEY) { return new Response(JSON.stringify({ error: "Missing OpenAI API Key." }), { status: 401, headers: { "Content-Type": "application/json" } }); } const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, }); if (!messages) { return new Response(JSON.stringify({ error: "Messages are required" }), { status: 400, headers: { "Content-Type": "application/json" } }); } // Get the latest user message const lastMessage = messages[messages.length - 1]; const previousMessages = messages.slice(0, -1); // Prepare the message content for the latest message let userContent: any[] = [{ type: "text", text: lastMessage.content }]; // If an image is provided, attach it to the latest message if (image) { userContent.push({ type: "image_url", image_url: { url: image, // Assumes base64 data URI }, }); } const response = await openai.chat.completions.create({ model: "gpt-4o", messages: [ { role: "system", content: "You are Lumina, a ultra-high-end AI assistant for the visually impaired. You can see the user's camera feed. ALWAYS respond with beautiful Markdown formatting (use lists, bold text, and clear headings). Your voice is natural and professional. Be concise but warm.", }, ...previousMessages, { role: "user", content: userContent, } ], max_tokens: 500, stream: true, }); const stream = new ReadableStream({ async start(controller) { const encoder = new TextEncoder(); try { for await (const chunk of response) { const content = chunk.choices[0]?.delta?.content || ""; if (content) { controller.enqueue(encoder.encode(content)); } } } catch (err) { controller.error(err); } finally { controller.close(); } }, }); return new Response(stream, { headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-cache", "Connection": "keep-alive", }, }); } catch (error: any) { console.error("Chat API Error:", error); return new Response(JSON.stringify({ error: "Failed to generate response" }), { status: 500, headers: { "Content-Type": "application/json" } }); } } ``` ### next.config.ts ```typescript import type { NextConfig } from "next"; const nextConfig: NextConfig = { /* config options here */ reactCompiler: true, }; export default nextConfig; ``` ### supabase_schema.sql ```sql -- Create a table for narrative sessions create table public.sessions ( id text primary key, messages jsonb not null default '[]'::jsonb, timestamp timestamptz not null default now(), user_id uuid references auth.users(id) -- Optional: for future authentication ); -- Enable Row Level Security (RLS) alter table public.sessions enable row level security; -- Create a policy that allows anyone to insert/select/update/delete for now (Hackathon mode) -- WARNING: In production, you should restrict this to authenticated users. create policy "Public Access" on public.sessions for all using (true) with check (true); ``` ### src/lib/utils.ts ```typescript import { type ClassValue, clsx } from "clsx"; import { twMerge } from "tailwind-merge"; export function cn(...inputs: ClassValue[]) { return twMerge(clsx(inputs)); } ``` [15 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]