# Project export: Relay It This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: CruzHacks 2026 - Tagline: "Relay It" is a research assistant that turns screenshots into an interactive notepad. Capture anything while browsing, then chat with AI to organize insights, compare options, and make decisions. - Devpost: https://devpost.com/software/relay-it - GitHub: https://github.com/AndrewMahran7/Relay-It-Server - Demo: https://github.com/Gary0302/relay-it-webapp - Video: https://www.youtube.com/embed/Nj_kPc68u9o?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Andrew Mahran (18 commits), dez03 (2 commits) ## Devpost submission (written by the team) ### Inspiration As naturally curious people, we often find ourselves researching multiple topics or juggling different tasks at the same time. Over time, it becomes increasingly difficult to keep track of scattered information and to extract what truly matters while filtering out the noise. This process is further slowed by the constant, time-consuming context switching between note-taking apps and web browsers, which fragments attention and disrupts productive research flow. We wanted an easy way to capture an instance that matters, across multiple sources, and bring them into a single, organized space. We realized screenshots are a powerful tool for capturing information instantly. Tools like Notion and Google Docs are excellent for note-taking, and AI can generate insights when given context. Relay it was incepted from the combination of all these ideas into one seamless workflow. Just like pointing at something and saying, “Relay it!” AI takes over the heavy lifting. From that moment on, AI handles the capture and organization of information, allowing you to keep browsing without breaking your flow. ### What it does Relay It is an AI-powered research workspace that turns visual browsing into organized, actionable notes. The Core Workflow: Capture - Take screenshots of anything interesting while researching (This is already the most time comsuming process of the whole workflow!!!!) Intelligent Analysis - Each screenshot is analyzed by AI to understand: What information it contains (OCR text extraction) Why you captured it (user intent detection) How it relates to your other screenshots (context clues) Smart Note Generation - Instead of showing raw data or JSON, Relay It generates context-aware markdown notes tailored to what you're doing: Trip planning? Get hotel comparisons, decision frameworks, and itinerary suggestions Job searching? Get position comparisons, application trackers, and interview prep notes Content creation? Get structured outlines, key talking points, and source citations Learning? Get concept summaries, connections between ideas, and study guides Your notes aren't static, you can: Talk with AI to add more information or ask questions Talk with AI to add more information or ask questions Directly edit the markdown for full control Directly edit the markdown for full control Add new screenshots that automatically integrate into existing notes Add new screenshots that automatically integrate into existing notes The Result: Seemingly disorganized screenshots become a living document that helps you understand, compare, and act on your research— seamlessly. ### How we built it A stroke of luck, ~1050mg of caffeine. ### Challenges we ran into Our biggest problem was not one of technicality but one of collaboration and a lack of communication. Creative differences led to arguments and nearly led us to quitting altogether. Thankfully we were able to use our words to come to a better conclusion than that and were able to produce a final product we were all proud of in the end. ### Accomplishments we're proud of The accomplishment we’re most proud of is turning messy, passive information into something interactive and useful. Instead of just storing screenshots, we built a system that understands context, generates insights, and lets users edit their notes through conversation. Technically, we’re proud of how the frontend, backend, and AI reasoning work together seamlessly under hackathon time pressure. Conceptually, we’re proud that it genuinely saves cognitive load and helps people think, not just collect data. ## README (from the GitHub repository) # Relay It Backend AI-powered backend API for screenshot analysis and intelligent note management. Built for research sessions where users capture screenshots and receive AI-generated insights, summaries, and interactive note editing. ## What It Does This backend provides RESTful API endpoints that power the Relay It application: - **Screenshot Analysis** - Analyzes uploaded images using Google's Gemini 2.0 Flash to extract text, identify entities (hotels, restaurants, products, etc.), and generate structured insights - **Session Management** - Organizes screenshots into sessions with metadata tracking and retrieval - **AI Summarization** - Creates intelligent summaries from multiple screenshots and entities within a session - **Interactive Chat** - Enables users to edit markdown notes via natural language commands or ask questions about their research - **Regeneration** - Re-analyzes screenshots with refined prompts for improved results ## Tech Stack - **Next.js 16** - API routes with Edge Runtime for fast global deployment - **TypeScript** - Full type safety across all endpoints - **Gemini 2.0 Flash Exp** - Google's latest AI model for vision and text generation - **Supabase** - Authentication and PostgreSQL database - **Vercel** - Serverless deployment platform ## API Endpoints - `POST /api/analyze` - Analyze screenshot and extract entities - `POST /api/summarize` - Generate session summary from entities - `POST /api/regenerate` - Re-analyze with custom prompts - `POST /api/chat` - Interactive note editing and Q&A - `GET /api/sessions` - List all sessions - `GET /api/sessions/[id]` - Get session details See [API.md](./API.md) for detailed endpoint documentation. ## Quick Start ```bash npm install npm run dev ``` Environment variables required: - `GEMINI_API_KEY` - Google AI API key Deployed at: https://relay-that-backend.vercel.app ## Detected evidence (automated analysis) Indexed codebase: 16 recognized source files, 95 KB. - Next.js (technology) — detected in the code - React (technology) — detected in the code - Supabase (technology) — detected in the code - TypeScript (language) — detected in the code - Flask (technology) — claimed on Devpost, not found in the code - Node.js (technology) — claimed on Devpost, not found in the code - PostgreSQL (technology) — claimed on Devpost, not found in the code - Python (language) — claimed on Devpost, not found in the code - Swift (language) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (31 of 31) ``` .gitignore analysis-result.json API.md app/api/analyze/route.ts app/api/chat/route.ts app/api/regenerate/route.ts app/api/sessions/[id]/regenerate/route.ts app/api/sessions/[id]/route.ts app/api/sessions/route.ts app/api/summarize/route.ts app/layout.tsx app/page.tsx eslint.config.mjs lib/supabaseServer.ts next.config.ts notes.md package.json postcss.config.mjs README-API.md README.md regenerate-result.json REGENERATE.md response-format.json sample-responses.json test-analyze.mjs test-chat.mjs test-regenerate.mjs test-session-detail.mjs test-sessions.mjs test-summarize.mjs tsconfig.json ``` ### Dependencies - package.json: @supabase/supabase-js@^2.90.1, @types/node@^20, @types/react@^19, @types/react-dom@^19, eslint@^9, eslint-config-next@16.1.3, next@16.1.3, react@19.2.3, react-dom@19.2.3, typescript@^5 ### Recent commits (newest first) - updated readme - Updated the prompt for api/chat endpoint - fixed the prompt for api/chat - updated api.md for api/chat - Added /chat api endpoint - push for gary - updated analyze and regenerate prompts - Update API documentation and root page - Fix build: Add required React dependencies and minimal layout for Next.js - Refactor: Separate backend from frontend - Archive frontend files and implement summarize endpoint - Fix Next.js 15 params compatibility and update branding to Relay It - Add supabase-js dependency - Added web application frontend and backend - updated regenerate api - Added api endpoint 'regenerate' - root update - cros update - updated response package - Initial backend with /api/analyze mock - Initial commit from Create Next App ## Key source files (fetched from GitHub, selected and truncated for size) ### notes.md ```markdown # Los Cabos Trip Planning ## Accommodation Options ### Los Cabos - **One&Only Palmilla** - ✅ Pros: Exceptional 9.8/10 rating from 652 reviews, highlighted as having a top-rated beach location, VIP Access available via Expedia. - ⚠️ Cons: (None identified in this screenshot, pricing not yet visible). - Best for: Luxury beach getaway, highly-rated experience, short romantic trip (based on 2 travelers, 1 night). ## Decision Framework - Prioritize **luxury & high ratings** → One&Only Palmilla is a strong contender due to its 9.8/10 'Exceptional' rating. - Prioritize **beach access** → One&Only Palmilla is highlighted for its top-rated beach location. ## Next Steps - [ ] Check specific pricing for Jan 31 - Feb 1 for the One&Only Palmilla. - [ ] Research other comparable luxury hotels in Los Cabos for alternative options. - [ ] Look into activities and dining options in the Los Cabos area for a 1-night stay. ``` ### README-API.md ```markdown # Screenshot Analysis API Next.js API endpoints for analyzing screenshots and summarizing sessions with Google Gemini. ## Endpoints ### 1. Analyze Screenshot ``` POST /api/analyze ``` ### 2. Summarize Session ``` POST /api/summarize ``` ## Request ```typescript { image: string; // PNG data URL: "data:image/png;base64,..." } ``` ## Response ```typescript { rawText: string; // Full OCR text summary: string; // 1-3 sentence description category: string; // "trip-planning" | "shopping" | "job-search" | "research" | "content-writing" | "productivity" | "other" entities: Array<{ type: string; // e.g. "hotel", "product", "job", "article" title: string | null; // Main name/title attributes: Record; // Key-value metadata }>; suggestedNotebookTitle: string | null; // Optional notebook title suggestion } ``` ## Example Response ```json { "rawText": "Hotel Deluxe\n5 stars\n$299/night\nSan Francisco, CA", "summary": "User is browsing hotel options in San Francisco with pricing and ratings.", "category": "trip-planning", "entities": [ { "type": "hotel", "title": "Hotel Deluxe", "attributes": { "price": "$299/night", "rating": "5 stars", "location": "San Francisco, CA" } } ], "suggestedNotebookTitle": "San Francisco Hotels" } ``` ## Setup 1. Create `.env.local`: ```bash GEMINI_API_KEY=your_api_key_here ``` 2. Run dev server: ```bash npm run dev ``` 3. Test with hotel.png: ```bash node test-analyze.mjs ``` ## Categories - **trip-planning**: Hotels, flights, destinations, itineraries - **shopping**: Products, comparisons, reviews - **job-search**: Job postings, applications, company research - **research**: Articles, papers, documentation - **content-writing**: Drafts, notes, writing tools - **productivity**: Tasks, calendars, project management - **other**: Everything else ## Entity Types Common entity types extracted: - `hotel` - Hotels with price, rating, location - `product` - Products with price, specs, brand - `job` - Job postings with company, salary, location - `article` - Articles with author, date, source - `flight` - Flights with airline, time, price - `restaurant` - Restaurants with cuisine, rating, location - `generic` - Other structured data ## Error Handling The API never throws errors. If: - `GEMINI_API_KEY` is missing - Gemini API fails - Response parsing fails Returns fallback response: ```json { "rawText": "", "summary": "", "category": "other", "entities": [], "suggestedNotebookTitle": null } ``` ## Deployment Deploy to Vercel: 1. Add environment variable in Vercel dashboard: - `GEMINI_API_KEY` 2. Deploy: ```bash vercel --prod ``` ## Testing ### Local Test ```bash node test-analyze.mjs ``` ### cURL Test ```bash curl -X POST http://localhost:3000/api/analyze \ -H "Content-Type: application/json" \ -d '{"im [truncated — 3304 more characters] ``` ### package.json ``` { "name": "relay-that-backend", "version": "0.1.0", "private": true, "description": "Backend API server for Relay It - Screenshot analysis and session management", "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint", "test:analyze": "node test-analyze.mjs", "test:summarize": "node test-summarize.mjs", "test:sessions": "node test-sessions.mjs" }, "dependencies": { "@supabase/supabase-js": "^2.90.1", "next": "16.1.3", "react": "19.2.3", "react-dom": "19.2.3" }, "devDependencies": { "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "16.1.3", "typescript": "^5" } } ``` ### app/layout.tsx ```typescript export default function RootLayout({ children, }: { children: React.ReactNode; }) { return ( {children} ); } ``` ### app/page.tsx ```typescript export default function Home() { return (

Relay It Backend API

This is a backend API server. Available endpoints:

See API.md for full documentation.

); } ``` ### app/api/sessions/route.ts ```typescript import { NextRequest, NextResponse } from 'next/server'; import { getSupabaseServerClient, getUserFromAuth } from '@/lib/supabaseServer'; type Entity = { type: string; title: string | null; attributes: Record; }; type Suggestion = | { type: 'question'; text: string } | { type: 'ranking'; basis: string; items: { entityTitle: string; reason: string }[]; } | { type: 'next-step'; text: string }; type RegenerateState = { sessionSummary: string; sessionCategory: string; entities: Entity[]; suggestedNotebookTitle: string | null; suggestions: Suggestion[]; }; type SessionListItem = { id: string; name: string; description: string | null; createdAt: string; updatedAt: string; screenshotCount: number; regenerateState: RegenerateState | null; }; export async function GET(request: NextRequest) { try { // Authenticate user const authHeader = request.headers.get('Authorization'); const userId = await getUserFromAuth(authHeader); // For development: allow without auth (remove this in production) // if (!userId) { // return NextResponse.json({ error: 'Unauthorized' }, { status: 401 }); // } const supabase = getSupabaseServerClient(); // Fetch sessions for this user (or all if no userId in dev mode) const query = supabase .from('sessions') .select('*') .order('updated_at', { ascending: false }); // Only filter by user_id if we have one const { data: sessions, error: sessionsError } = userId ? await query.eq('user_id', userId) : await query; if (sessionsError) { console.error('Error fetching sessions:', sessionsError); return NextResponse.json({ error: 'Internal server error' }, { status: 500 }); } // For each session, fetch screenshot count and regenerate state const sessionList: SessionListItem[] = await Promise.all( sessions.map(async (session) => { // Get screenshot count const { count, error: countError } = await supabase .from('screenshots') .select('*', { count: 'exact', head: true }) .eq('session_id', session.id); if (countError) { console.error('Error counting screenshots:', countError); } // Get regenerate state const { data: regenerateStateRow, error: regenerateError } = await supabase .from('regenerate_state') .select('*') .eq('session_id', session.id) .single(); if (regenerateError && regenerateError.code !== 'PGRST116') { // PGRST116 = no rows returned, which is fine console.error('Error fetching regenerate state:', regenerateError); } let regenerateState: RegenerateState | null = null; if (regenerateStateRow) { regenerateState = { sessionSummary: regenerateStateRow.session_summary, sessionCategory: regenerateStateRow.session_category, entities: regenerateStateRow.entities, suggestedNotebookTitle: regenerateStateRow.suggested_notebook_title, suggestions: regenerateStateRow.suggestions, }; } return { id: session.id, name: session.name, description: session.description, createdAt: session.created_at, updatedAt: session.updated_at, screenshotCount: count || 0, regenerateState, }; }) ); return NextResponse.json(sessionList, { status: 200 }); } catch (error) { console.error('Error in GET /api/sessions:', error); return NextResponse.json({ error: 'Internal server error' }, { status: 500 }); } } ``` ### app/api/summarize/route.ts ```typescript import { NextRequest, NextResponse } from 'next/server'; export const runtime = 'edge'; interface Entity { type: string; title: string | null; attributes: Record; } interface SummarizeRequest { sessionId: string; sessionName: string; entities: Entity[]; } interface SummarizeResponse { condensedSummary: string; keyHighlights: string[]; recommendations: string[]; mergedEntities: Entity[]; suggestedTitle: string; } async function summarizeSession(reqBody: SummarizeRequest): Promise { const apiKey = process.env.GEMINI_API_KEY; if (!apiKey) { console.warn('GEMINI_API_KEY not set, returning fallback response'); return { condensedSummary: '', keyHighlights: [], recommendations: [], mergedEntities: reqBody.entities, suggestedTitle: reqBody.sessionName, }; } console.log('API key present, calling Gemini for summarization...'); try { const prompt = `You are a research assistant that creates condensed summaries. Given a collection of entities the user has gathered, create: 1. A 2-4 sentence overview summarizing the research 2. 3-5 key highlights as bullet points 3. 2-3 actionable recommendations 4. Merged/deduplicated entities (combine similar items into more concise representations) 5. A suggested title for this collection STRICT OUTPUT FORMAT (JSON ONLY): { "condensedSummary": "2-4 sentence overview of the entire research/collection", "keyHighlights": [ "Highlight 1 - most important finding or item", "Highlight 2 - second most important", "Highlight 3 - third most important", "..." ], "recommendations": [ "Actionable recommendation 1", "Actionable recommendation 2", "..." ], "mergedEntities": [ { "type": "comparison" | "merged" | "group" | original type, "title": "meaningful title for merged entity", "attributes": { "key": "value" } } ], "suggestedTitle": "Clear, concise title for this collection" } RULES: - Return ONLY valid JSON, no markdown, no explanations - condensedSummary should provide context and insights, not just list items - keyHighlights should be 3-5 bullet points (strings) - recommendations should be 2-3 actionable suggestions based on the content - mergedEntities should combine similar items, create comparisons, or group related entities - If entities can't be meaningfully merged, return them as-is - suggestedTitle should be descriptive but concise (max 6-8 words) EXAMPLES: Input: Hotels in Taipei Output: { "condensedSummary": "Planning a trip to Taipei with focus on luxury hotels in Xinyi District. The Grand Hyatt offers better value at $250/night with a higher rating than W Taipei. Din Tai Fung is a must-visit for authentic Taiwanese cuisine.", "keyHighlights": [ "Grand Hyatt Taipei: Best value at $250/night, 4.8 rating", "W Taipei: Premium option at $300/night", "Din Tai Fung: Top-rated Taiwanese restaurant (4.9★)" ], "recommendations": [ "Book Grand Hyatt for best price-to-quality ratio", "Make Din Tai Fung reservation in advance - very popular", "Both hotels are in Xinyi District - convenient for exploring" ], "mergedEntities": [ { "type": "hotel-comparison", "title": "Taipei Hotels Comparison", "attributes": { "best_value": "Grand Hyatt ($250, 4.8★)", "premium_option": "W Taipei ($300, 4.6★)", "location": "Xinyi District" } }, { "type": "restaurant", "title": "Din Tai Fung", "attributes": { "cuisine": "Taiwanese", "rating": "4.9", "note": "Must-visit" } } ], "suggestedTitle": "Taipei Trip: Hotels & Dining" } Input: Job listings Output: { "condensedSummary": "Evaluating software engineering positions at tech companies. Salaries range from $120K-$180K, with remote and hybrid options available. Most positions require 3-5 years experience with React and TypeScript.", "keyHighlights": [ "Tech Corp: $180K, fully remote, best compensation", "StartUp Inc: $120K, hybrid, equity included", "DevShop: $150K, on-site, excellent benefits" ], "recommendations": [ "Tech Corp offers highest salary and full remote flexibility", "Consider StartUp Inc if interested in equity upside", "Compare benefits packages before making final decision" ], "mergedEntities": [ { "type": "job-comparison", "title": "Software Engineering Positions", "attributes": { "salary_range": "$120K-$180K", "top_pick": "Tech Corp ($180K, remote)", "equity_option": "StartUp Inc", "common_requirements": "React, TypeScript, 3-5 years" } } ], "suggestedTitle": "Software Engineering Job Search" } NOW ANALYZE THIS SESSION: Session Name: ${reqBody.sessionName} Session ID: ${reqBody.sessionId} Entities: ${JSON.stringify(reqBody.entities, null, 2)} Return ONLY valid JSON matching the format above.`; const response = await fetch( `https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash-exp:generateContent?key=${apiKey}`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ contents: [ { parts: [{ text: prompt }], }, ], generationConfig: { temperature: 0.7, topK: 40, topP: 0.95, maxOutputTokens: 2048, }, }), } ); if (!response.ok) { const errorText = await response.text(); console.error('Gemini API error:', response.status, errorText); throw new Error(`Gemini API returned ${response.status}`); } const data = await response.json(); const rawResponse = data.candidates?.[0]?.content?.parts?.[0]?.text || ''; console.log('Raw Gemini response:', rawResponse); // Parse the JSON response const cl [truncated — 1670 more characters] ``` ### app/api/analyze/route.ts ```typescript import { NextRequest, NextResponse } from 'next/server'; export const runtime = 'edge'; interface AnalyzeRequest { image: string; } interface Entity { type: string; title: string | null; attributes: Record; } interface AnalyzeResponse { rawText: string; summary: string; userIntent: string; // NEW category: string; entities: Entity[]; suggestedNotebookTitle: string | null; contextClues: { // NEW isComparison: boolean; decisionPoint: string | null; relatedTopics: string[]; }; } const FALLBACK_RESPONSE: AnalyzeResponse = { rawText: '', summary: '', userIntent: '', // NEW category: 'other', entities: [], suggestedNotebookTitle: null, contextClues: { // NEW isComparison: false, decisionPoint: null, relatedTopics: [] } }; async function analyzeScreenshot(imageData: string): Promise { const apiKey = process.env.GEMINI_API_KEY; if (!apiKey) { console.warn('GEMINI_API_KEY not set, returning fallback response'); return FALLBACK_RESPONSE; } console.log('API key present, calling Gemini...'); try { // Strip data URL prefix if present const base64Data = imageData.includes(',') ? imageData.split(',')[1] : imageData; const prompt = `You are an AI assistant helping someone take notes while browsing the web. ANALYZE THIS SCREENSHOT TO EXTRACT STRUCTURED INFORMATION: 1. Extract all visible text (OCR) 2. Understand what the user might be trying to accomplish/take away with this screenshot 3. Extract structured data for search/filtering 4. Provide context about this screenshot's role in their research RETURN THIS JSON FORMAT: { "rawText": "full OCR text", "summary": "What is this screenshot and why did the user capture it? (1-3 sentences)", "userIntent": "What is the user trying to figure out, decide, or understand?", "category": "trip-planning|shopping|job-search|research|content-writing|productivity|projects|general-planning|brainstorming|study-guides|academic-research|other", "entities": [ { "type": "hotel|product|job|article|flight|restaurant|etc", "title": "name", "attributes": { "key": "value" } } ], "suggestedNotebookTitle": "title or null", "contextClues": { "isComparison": true/false, "decisionPoint": "what they're deciding, or null", "relatedTopics": ["topic1", "topic2"] } } EXAMPLES OF GOOD userIntent: - "Comparing luxury hotels in Tokyo for February trip" - "Researching AI trends for TikTok video about GPT-4" - "Evaluating software engineering roles at big tech companies" - "Looking for affordable laptops with good battery life" - "Doing research on a specific topic for class" EXAMPLES OF GOOD contextClues: For hotel screenshot: { "isComparison": true, "decisionPoint": "location vs price tradeoff", "relatedTopics": ["Tokyo accommodation", "budget travel", "Shibuya nightlife"] } For article screenshot (TikTok research, brainstorming, studying, etc.): { "isComparison": false, "decisionPoint": null, "relatedTopics": ["AI trends", "GPT-4 capabilities", "prompt engineering"] } IMPORTANT: - userIntent should capture the user's GOAL, not just describe the screenshot - contextClues help connect this screenshot to others in the session - Entities are still structured (for search), but userIntent adds meaning - Return ONLY valid JSON, no markdown, no explanations - If uncertain, use empty string for userIntent - If no clear decision point, use null NOW ANALYZE THE SCREENSHOT:`; const requestBody = { contents: [ { role: 'user', parts: [ { text: prompt }, { inlineData: { mimeType: 'image/png', data: base64Data, }, }, ], }, ], generationConfig: { temperature: 0.3, // Slightly higher for userIntent inference responseMimeType: 'application/json', }, }; const response = await fetch( `https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent?key=${apiKey}`, { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify(requestBody), } ); if (!response.ok) { const errorBody = await response.text(); console.error(`Gemini API error: ${response.status} ${response.statusText}`, errorBody); return FALLBACK_RESPONSE; } const data = await response.json(); const textContent = data.candidates?.[0]?.content?.parts?.[0]?.text; if (!textContent) { console.error('No content returned from Gemini'); return FALLBACK_RESPONSE; } let parsed: any; try { parsed = JSON.parse(textContent); } catch (e) { console.error('Failed to parse Gemini response as JSON:', e); console.error('Raw response:', textContent); return FALLBACK_RESPONSE; } // Validate and normalize response with NEW FIELDS const result: AnalyzeResponse = { rawText: parsed.rawText || '', summary: parsed.summary || '', userIntent: parsed.userIntent || '', // NEW category: parsed.category || 'other', entities: Array.isArray(parsed.entities) ? parsed.entities : [], suggestedNotebookTitle: parsed.suggestedNotebookTitle || null, contextClues: { // NEW - with validation isComparison: parsed.contextClues?.isComparison === true, decisionPoint: parsed.contextClues?.decisionPoint || null, relatedTopics: Array.isArray(parsed.contextClues?.relatedTopics) ? parsed.contextClues.relatedTopics : [] } }; console.log('Analysis complete with userIntent:', result.userIntent); return result; } catch (error) { console.error('Error in analyzeScreenshot:', error); return FALLBACK_RESPONSE; } } export async function POST [truncated — 1837 more characters] ``` ### app/api/chat/route.ts ```typescript import { NextRequest, NextResponse } from 'next/server'; export const runtime = 'edge'; // Type definitions type ScreenshotContext = { id: string; rawText: string; summary: string; }; type ChatContext = { screenshots?: ScreenshotContext[]; sessionName?: string; sessionCategory?: string; }; type ChatRequest = { sessionId: string; userMessage: string; currentNote: string; context?: ChatContext; }; type ChatResponse = { reply: string; updatedNote?: string; noteWasModified: boolean; }; /** * Builds the prompt for the LLM */ function buildPrompt(userMessage: string, currentNote: string, context?: ChatContext): string { let prompt = `You are an AI assistant helping a user manage their markdown notes and research sessions. You have access to: - The current markdown note - Screenshot context (if provided) - Your general knowledge and training data Your job is to: 1. Decide if the user's message is an EDIT COMMAND or a QUESTION 2. If EDIT COMMAND: modify the note and return the full updated markdown 3. If QUESTION: answer using the note, screenshots, AND your general knowledge 4. Always respond in STRICT JSON ## Identifying Command Type **EDIT COMMANDS** modify the note. Look for: - Instruction verbs: remove, delete, add, insert, rewrite, shorten, expand, change, update, fix, clean up, make concise, rephrase, edit, include - References to note parts: title, summary, section, bullet, recommendation, item, examples, etc. - Requests to add new content: "add examples of...", "include information about...", "create a section for..." Examples: - "Remove the third recommendation" - "Rewrite the summary to be shorter" - "Add a section about budget" - "Delete the second hotel" - "Add some examples of Java input and output" - "Include more details about Python loops" **QUESTIONS** ask for information without changing anything. Look for: - Question words: what, why, how, where, who, which, when, can you tell me, explain - Informational requests without modification intent: "tell me about...", "explain..." Examples: - "What hotels did I look at?" - "Summarize what's in my notes" - "What was the price of the second hotel?" - "How many recommendations do I have?" - "Explain how Java I/O works" - "What is recursion?" **When ambiguous, treat as QUESTION (do not modify).** ## Current Note (Markdown): ${currentNote} ## User Message: "${userMessage}" `; // Add context if available if (context) { if (context.sessionName) { prompt += `\n\n## Session Info:\n- Name: ${context.sessionName}`; } if (context.sessionCategory) { prompt += `\n- Category: ${context.sessionCategory}`; } if (context.screenshots && context.screenshots.length > 0) { prompt += `\n\n## Screenshot Context:`; context.screenshots.forEach((screenshot, idx) => { prompt += `\n\n### Screenshot ${idx + 1} (${screenshot.id}):`; prompt += `\nSummary: ${screenshot.summary}`; if (screenshot.rawText) { const text = screenshot.rawText.substring(0, 500); prompt += `\nOCR Text: ${text}${screenshot.rawText.length > 500 ? '...' : ''}`; } }); } } prompt += ` ## Response Format You MUST respond with ONLY a JSON object (no markdown, no explanations): { "reply": "Your message to the user", "updatedNote": "Full markdown note (required if modified)", "noteWasModified": boolean } ## Rules: **If EDIT COMMAND:** - Set noteWasModified: true - Return FULL modified note in updatedNote - Use information from: the existing note, screenshot context, AND your general knowledge - Add relevant, accurate content even if not present in screenshots - Preserve markdown structure and existing content - Provide short confirmation in reply: "Done! I've [what you did]." **If QUESTION:** - Set noteWasModified: false - Keep note unchanged (updatedNote = original note) - Answer using: the note content, screenshot context, AND your general knowledge - Provide helpful, accurate information from your training data - Don't say "I cannot" unless truly impossible CRITICAL: Return ONLY the JSON object. No backticks, no explanations, no extra text.`; return prompt; } /** * Calls Gemini API with the prompt */ async function callGemini(prompt: string, apiKey: string): Promise { try { const response = await fetch( `https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash-exp:generateContent?key=${apiKey}`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ contents: [ { parts: [{ text: prompt }], }, ], generationConfig: { temperature: 0.7, topK: 40, topP: 0.95, maxOutputTokens: 4096, }, }), } ); if (!response.ok) { const errorText = await response.text(); console.error('Gemini API error:', response.status, errorText); throw new Error(`Gemini API returned ${response.status}`); } const data = await response.json(); const rawResponse = data.candidates?.[0]?.content?.parts?.[0]?.text || ''; console.log('Raw Gemini response:', rawResponse); // Clean up the response - remove markdown code blocks if present const cleanedResponse = rawResponse .replace(/```json\n?/g, '') .replace(/```\n?/g, '') .trim(); // Parse JSON const result: ChatResponse = JSON.parse(cleanedResponse); // Validate required fields if (typeof result.reply !== 'string' || typeof result.noteWasModified !== 'boolean') { throw new Error('Invalid response structure from LLM'); } // If noteWasModified is true, updatedNote must be present if (result.noteWasModified && !result.updatedNote) { throw new Error('updatedNote is required when noteWasModified is true'); } console.log('Parsed chat result:', { [truncated — 2704 more characters] ``` ### app/api/regenerate/route.ts ```typescript import { NextRequest, NextResponse } from 'next/server'; export const runtime = 'edge'; interface Entity { type: string; title: string | null; attributes: Record; } interface AnalyzeResponse { rawText: string; summary: string; category: string; entities: Entity[]; suggestedNotebookTitle: string | null; userIntent?: string; // NEW: Why user took this screenshot contextClues?: { // NEW: Helps connect screenshots isComparison: boolean; decisionPoint: string | null; relatedTopics: string[]; }; } interface PreviousSession { sessionSummary: string; sessionCategory: string; entities: Entity[]; formattedNotes?: string; // NEW: Previous markdown notes } interface ScreenInput { id: string; analysis: AnalyzeResponse; } interface RegenerateRequest { sessionId: string; previousSession?: PreviousSession; screens: ScreenInput[]; userQuery?: string; } type Suggestion = | { type: 'question'; text: string; } | { type: 'ranking'; basis: string; items: { entityTitle: string; reason: string; }[]; } | { type: 'next-step'; text: string; }; interface RegenerateResponse { sessionId: string; sessionSummary: string; sessionCategory: string; formattedNotes: string; // NEW: Markdown-formatted notes noteStyle: string; // NEW: Type of notes generated entities: Entity[]; // Keep for search/filtering suggestedNotebookTitle: string | null; suggestions: Suggestion[]; } async function analyzeSession(reqBody: RegenerateRequest): Promise { const apiKey = process.env.GEMINI_API_KEY; if (!apiKey) { console.warn('GEMINI_API_KEY not set, returning fallback response'); return { sessionId: reqBody.sessionId, sessionSummary: '', sessionCategory: 'other', formattedNotes: '', noteStyle: 'general', entities: [], suggestedNotebookTitle: null, suggestions: [], }; } console.log('API key present, calling Gemini for session analysis...'); try { // NEW PROMPT - Context-aware note generation let prompt = `You are an AI note-taking assistant. A user has taken multiple screenshots while researching something. YOUR JOB: Synthesize these screenshots into USEFUL, ACTIONABLE notes. --- SCREENSHOT ANALYSIS --- `; // Add each screenshot's analysis reqBody.screens.forEach((screen, i) => { const analysis = screen.analysis; prompt += ` Screenshot ${i + 1} (ID: ${screen.id}): - Summary: ${analysis.summary} - User Intent: ${analysis.userIntent || 'Not specified'} - Category: ${analysis.category} - Entities: ${JSON.stringify(analysis.entities)} - Context Clues: ${JSON.stringify(analysis.contextClues || {})} - Raw Text Preview: ${analysis.rawText.substring(0, 150)}... `; }); // Add previous session context if it exists if (reqBody.previousSession) { prompt += `\n--- PREVIOUS SESSION CONTEXT --- Summary: ${reqBody.previousSession.sessionSummary} Category: ${reqBody.previousSession.sessionCategory} ${reqBody.previousSession.formattedNotes ? `Previous Notes:\n${reqBody.previousSession.formattedNotes.substring(0, 500)}...` : ''} `; } else { prompt += `\n--- FIRST SCREENSHOTS IN THIS SESSION ---\n`; } // Add user query if present if (reqBody.userQuery) { prompt += `\n--- USER QUESTION --- "${reqBody.userQuery}" Address this question directly in your notes. `; } // Main instruction prompt prompt += ` --- YOUR TASK --- 1. Determine the SESSION TYPE based on what the user is doing: - "trip-planning" → Travel research/booking - "content-creation" → Making TikTok/blog/video content - "shopping" → Product comparison/purchase decision - "job-search" → Career research/applications - "learning" → Educational research - "project-planning" → Work/personal project - "general" → Mixed/unclear purpose 2. Generate CONTEXTUAL MARKDOWN NOTES based on session type: FOR TRIP-PLANNING: \`\`\`markdown # [Destination] Trip Planning ## Accommodation Options ### [City 1] - **[Hotel Name]** - $X/night - ✅ Pros: [specific benefits] - ⚠️ Cons: [specific drawbacks] - Best for: [use case] ## Decision Framework - Prioritize [factor] → Choose [option] because [reason] - Prioritize [factor] → Choose [option] because [reason] ## Next Steps - [ ] [Specific action item] - [ ] [Specific action item] \`\`\` FOR CONTENT-CREATION (TikTok/Video/Blog): \`\`\`markdown # [Content Topic] ## Content Outline 1. **Hook (0-5s)**: [Attention-grabbing opener] 2. **Main Point 1 (5-20s)**: [Key message] 3. **Main Point 2 (20-35s)**: [Supporting point] 4. **Call to Action (35-60s)**: [What viewer should do] ## Key Talking Points - [Stat/fact from source with citation] - [Quote/insight from source] - [Example from source] ## Script Notes [How to present this information naturally] ## Visual Ideas - B-roll: [specific footage needed] - Graphics: [text overlays, animations] - Transitions: [between sections] ## Sources 1. [Source name] - [Key takeaway] 2. [Source name] - [Key takeaway] ## Production Checklist - [ ] Write full script - [ ] Record B-roll - [ ] Create graphics \`\`\` FOR ACADEMIC-RESEARCH / STUDY-GUIDES: \`\`\`markdown # [Topic/Course] Study Notes ## Key Concepts ### [Concept 1]: [Name] **Definition**: [Clear explanation] **Why it matters**: [Practical relevance] **Key details**: - [Important point 1] - [Important point 2] **Related to**: [Other concepts from session] ### [Concept 2]: [Name] ... ## Summary & Connections [How all these concepts fit together - the "big picture"] ## Study Checklist - [ ] Understand [core concept] - [ ] Review [practice problems/examples] - [ ] Memorize [key definitions/formulas] ## Quick Reference | Term | Definition | Example | |------|------------|---------| | [X] | [Def] | [Ex] | ## Sources 1. [Textbook/Article] - [Chapter/Pages] 2. [ [truncated — 11193 more characters] ``` [6 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]