# Project export: idontwantanipadbaby This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: An AI that recognizes children's hand drawings and give suggestions on how they can improve on their drawing skills based on their voices' inputs. - Devpost: https://devpost.com/software/idontwantanipadbaby - GitHub: https://github.com/sudip0789/DrawingTutor - Team: 4 GitHub contributor(s) — doxatin (5 commits), suhaas (2 commits), sudip0789 (1 commits), doxatin (1 commits) ## Devpost submission (written by the team) ### Inspiration The idea for idontwantanipadbaby came from a desire to provide children with a fun, interactive, and educational drawing experience that doesn’t just rely on screens for entertainment. We wanted to create a virtual teacher that goes beyond just instructing; it engages with kids in a meaningful way, helps them express their creativity through drawing, and provides constructive feedback to improve their skills. The goal was to design an AI-powered tool that encourages hands-on activity while fostering artistic development. ### What it does idontwantanipadbaby is an AI-powered virtual drawing tutor that interacts with children to help them learn how to draw. The process involves: The AI asking the child what they would like to draw. Taking the child's voice input to understand the drawing concept. Waiting for the child to draw the chosen concept. Analyzing the hand-drawn sketch and providing feedback on its accuracy and aesthetic qualities using a LLM. Offering suggestions on how the child can improve their drawing skills using Voice AI text-to-speech. ### How we built it We utilized a variety of tools and technologies to bring this project to life: Deepgram API for live voice transcription, converting the child's spoken input into text. Groq API to process the transcribed text and understand the drawing concept. JavaScript and React for the front-end interface, providing an intuitive and interactive experience. Node.js on the back end to integrate various API calls and manage real-time interactions. ### Challenges we ran into Real-time Voice Processing: Getting the live voice transcription to work smoothly was challenging, especially handling background noise and ensuring accurate speech recognition. Drawing Analysis: It was tricky to evaluate freehand drawings in a meaningful way. We had to fine-tune the drawing analysis to balance between providing helpful feedback and encouraging creativity. Integrating Multiple APIs: Combining different services and ensuring smooth communication between them was difficult. Each API had its own requirements, and coordinating these services to work seamlessly took some effort. User Experience for Young Children: Designing a user interface that is simple enough for children to use, yet robust enough to support our AI’s functionalities, required careful consideration. ### Accomplishments we're proud of End-to-End Functionality: We successfully built a prototype that guides a child through the entire drawing process, from voice input to drawing analysis and feedback. Seamless Voice Interaction: We achieved a smooth voice-to-text and text-to-feedback loop using the Deepgram and Agora APIs, enhancing the interactivity of the project. Impactful User Interface: The user interface was designed with young users in mind, making it visually appealing and easy to navigate. ### What we learned AI Integration in Education: We gained valuable insights into the challenges of using AI to teach creative skills like drawing and how to provide constructive feedback in a way that encourages rather than discourages. API Communication: Working with multiple APIs taught us how to effectively manage API calls, error handling, and asynchronous programming in a real-world project. Voice and Visual Processing: We improved our understanding of voice recognition and image analysis techniques, especially in adapting these technologies for young users. ### What's next Enhanced Drawing Analysis: We plan to improve our drawing analysis algorithms to better assess sketches based on style, color use, and overall composition. Expanded Educational Content: Adding more topics, fun facts, and interactive lessons could make the experience richer and more educational. Voice Customization: We aim to personalize the virtual tutor's voice, allowing children to choose different voices and even languages for a more tailored experience. Parental Dashboard: We are considering adding a dashboard where parents can monitor their child’s progress and see how their drawing skills are developing over time. AR Integration with Snap AR: We want to explore augmented reality to make drawing even more interactive, such as projecting a 3D model of the concept for children to follow. ## README (from the GitHub repository) This is a [Next.js](https://nextjs.org/) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app). ## Getting Started First, run the development server: ```bash npm run dev # or yarn dev # or pnpm dev # or bun dev ``` Open [http://localhost:3000](http://localhost:3000) with your browser to see the result. You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file. This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font. ## Learn More To learn more about Next.js, take a look at the following resources: - [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API. - [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial. You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome! ## Deploy on Vercel The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js. Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details. ## Detected evidence (automated analysis) Indexed codebase: 37 recognized source files, 111 KB. - CSS (language) — detected in the code - Next.js (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - JavaScript (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (43 of 43) ``` .gitignore biome.json components.json next.config.mjs package.json postcss.config.mjs README.md src/app/api/groq-chat.tsx src/app/components/draw-canvas.tsx src/app/components/enter-key.tsx src/app/components/let-me-guess.tsx src/app/components/palette-bar.tsx src/app/components/pen-and-eraser-bar.tsx src/app/components/top-chrome.tsx src/app/components/top-commands.tsx src/app/components/TutorLLM.tsx src/app/context/UserChoiceContext.tsx src/app/globals.css src/app/layout.tsx src/app/page.tsx src/components/groq-logo.tsx src/components/slider-vertical.tsx src/components/ui/button.tsx src/components/ui/input.tsx src/components/ui/separator.tsx src/components/ui/siriwave.tsx src/components/ui/slider.tsx src/components/ui/textarea.tsx src/hooks/use-keyboard-shortcut.ts src/hooks/use-llama-vision.ts src/hooks/use-llava.ts src/hooks/use-llm.ts src/hooks/use-localstorage.ts src/hooks/use-undo-redo.ts src/hooks/use-viewport-size.ts src/lib/provider-factory.tsx src/lib/utils.ts src/models/index.ts src/providers/draw-editor-provider.tsx src/providers/keys-provider.tsx src/providers/let-me-guess-provider.tsx tailwind.config.ts tsconfig.json ``` ### Dependencies - package.json: @biomejs/biome@1.7.3, @deepgram/sdk@^3.8.1, @radix-ui/react-icons@^1.3.0, @radix-ui/react-separator@^1.0.3, @radix-ui/react-slider@^1.1.2, @radix-ui/react-slot@^1.0.2, @types/node@^20, @types/react@^18, @types/react-dom@^18, axios@^1.7.7, class-variance-authority@^0.7.0, clsx@^2.1.1, cross-fetch@^4.0.0, groq-sdk@^0.4.0, lucide-react@^0.379.0, next@^14.2.15, postcss@^8, react@^18.3.1, react-dom@^18.3.1, siriwave@^2.4.0, tailwind-merge@^2.3.0, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5 ### Recent commits (newest first) - more changes - Added llm output - Delete unused files - Added transcription using Deepgram - Merge pull request #1 from sudip0789/dev - Delete .env - Add siriwave while recording - Update page.tsx - Update page.tsx - Add Deepgram and Microphone Context Provider - Added intial blocks for user input - Add llama-3.2-11b-vision-preview - Quick! let's put something together ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "groq-llava", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@deepgram/sdk": "^3.8.1", "@radix-ui/react-icons": "^1.3.0", "@radix-ui/react-separator": "^1.0.3", "@radix-ui/react-slider": "^1.1.2", "@radix-ui/react-slot": "^1.0.2", "axios": "^1.7.7", "class-variance-authority": "^0.7.0", "clsx": "^2.1.1", "cross-fetch": "^4.0.0", "groq-sdk": "^0.4.0", "lucide-react": "^0.379.0", "next": "^14.2.15", "react": "^18.3.1", "react-dom": "^18.3.1", "siriwave": "^2.4.0", "tailwind-merge": "^2.3.0", "tailwindcss-animate": "^1.0.7" }, "devDependencies": { "@biomejs/biome": "1.7.3", "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "^5" } } ``` ### src/models/index.ts ```typescript interface Message { role: string; content: string | null; image?: string; } interface Keys { groqApiKey: string; } ``` ### src/app/layout.tsx ```typescript import type { Metadata, Viewport } from "next"; import { Inter } from "next/font/google"; import "./globals.css"; import { KeysProvider } from "@/providers/keys-provider"; const inter = Inter({ subsets: ["latin"] }); export const metadata: Metadata = { title: "Drawing Tutor", description: "Draw and guess.", openGraph: { images: ["/og-image.png"], }, }; export const viewport: Viewport = { width: "device-width", initialScale: 1, maximumScale: 1, userScalable: false, }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### src/app/page.tsx ```typescript "use client"; import React, { useState, useEffect } from 'react'; import { createClient, LiveTranscriptionEvents, ListenLiveClient } from '@deepgram/sdk'; import { Button } from "@/components/ui/button"; import SiriWaveComponent from "@/components/ui/siriwave"; import { LogOut } from "lucide-react"; import { useKeys } from "@/providers/keys-provider"; import { LetMeGuessProvider, useLetMeGuess } from "@/providers/let-me-guess-provider"; import { LetMeGuess } from "./components/let-me-guess"; import { EnterKey } from "./components/enter-key"; import { createGridTile } from "@/lib/utils"; import TutorLLM from "./components/TutorLLM"; // Import the new component import { UserChoiceProvider, useUserChoice } from './context/UserChoiceContext'; const GRID_TILE = createGridTile(10, 10); export default function Home() { const { keys, setKeys } = useKeys(); const [inited, setInited] = useState(false); const [isRecording, setIsRecording] = useState(false); const [mediaRecorder, setMediaRecorder] = useState(null); const [audioChunks, setAudioChunks] = useState([]); const [showWelcomeMessage, setShowWelcomeMessage] = useState(true); const [recordingCompleted, setRecordingCompleted] = useState(false); const [transcript, setTranscript] = useState(''); // To store the transcript in real-time const [deepgramClient, setDeepgramClient] = useState(null); const [isDeepgramReady, setIsDeepgramReady] = useState(false); // To track if Deepgram is ready const [llmResponse, setLlmResponse] = useState(""); const [llmGuessInput, setLlmGuessInput] = useState(""); const { interpretation } = useLetMeGuess(); let setUserChoice: any; // Ensure that useUserChoice is called only within UserChoiceProvider try { ({ setUserChoice } = useUserChoice()); } catch (error) { console.warn("UserChoiceProvider is not yet initialized."); } const deepgramApiKey = 'e6fe8a89b6c360fd972f8c713a6967a8ce725e06'; // Replace with your Deepgram API key useEffect(() => { if (isRecording) { initializeDeepgram(); } }, [isRecording]); useEffect(() => { if (inited) return; setInited(true); }, [inited]); useEffect(() => { if (setUserChoice) { console.log("setUserChoice presenttttttttttt") setUserChoice(transcript); console.log("setting usechoice", transcript); // Ensure setUserChoice is called only if it's available } }, [transcript]); const initializeDeepgram = () => { const deepgram = createClient(deepgramApiKey); const connection = deepgram.listen.live({ smart_format: true, model: 'nova', language: 'en-US', }); // Handle Deepgram events connection.on(LiveTranscriptionEvents.Open, () => { console.log('Deepgram connection opened.'); setIsDeepgramReady(true); // Set the connection as ready }); connection.on(LiveTranscriptionEvents.Transcript, (data) => { if (data.channel.alternatives[0].transcript) { console.log('Transcription received:', data.channel.alternatives[0].transcript); // Debugging setTranscript((prevTranscript) => prevTranscript + ' ' + data.channel.alternatives[0].transcript); } else { console.log('No transcription received'); // Debugging } }); connection.on(LiveTranscriptionEvents.Close, () => { console.log('Deepgram connection closed.'); setIsDeepgramReady(false); // Mark as no longer ready }); connection.on(LiveTranscriptionEvents.Error, (error) => { console.error('Deepgram connection error:', error); }); setDeepgramClient(connection); // Store the connection }; const handleRecord = () => { setShowWelcomeMessage(false); // Hide the welcome message if (!isRecording) { navigator.mediaDevices.getUserMedia({ audio: true }).then((stream) => { const recorder = new MediaRecorder(stream); recorder.ondataavailable = (event) => { if (deepgramClient && deepgramClient.getReadyState() === 1) { console.log('Sending audio data to Deepgram'); // Debugging deepgramClient.send(event.data); // Send the audio data to Deepgram } else { console.log('Deepgram connection not ready'); // Debugging } }; recorder.start(250); // Send audio every 250ms for real-time transcription setMediaRecorder(recorder); console.log('Recording started...'); }).catch(err => { console.error('Error accessing microphone:', err); }); } else if (mediaRecorder) { mediaRecorder.stop(); mediaRecorder.onstop = () => { console.log('Recording stopped.'); setRecordingCompleted(true); // Trigger the LLM when recording is completed and transcription is ready if (transcript) { askLLM(); // Process the transcript when user stops speaking } }; } setIsRecording(!isRecording); }; if (!inited) { return (
Loading...
); } if (!keys) { return setKeys(k)} />; } return ( {/* Ensure UserChoiceProvider is wrapping the whole component */}
{!recordingCompleted && showWelcomeMessage && (
Hey! How are you doing?
{ const [keys, setKeys] = useLocalStorage("kingdom_keys", null); return { keys, setKeys, }; }); export { KeysProvider, useKeys }; ``` ### src/hooks/use-viewport-size.ts ```typescript import { useState, useEffect } from "react"; export function useViewportSize() { const [width, setWidth] = useState(window.innerWidth); const [height, setHeight] = useState(window.innerHeight); useEffect(() => { const handleResize = () => { setWidth(window.innerWidth); setHeight(window.innerHeight); }; window.addEventListener("resize", handleResize); return () => { window.removeEventListener("resize", handleResize); }; }, []); return { width, height }; } ``` ### src/hooks/use-llava.ts ```typescript import Groq from "groq-sdk"; export function useLlava(apiKey: string) { const groq = new Groq({ apiKey, dangerouslyAllowBrowser: true, }); function callLlava(prompt: string, image_url: string) { return groq.chat.completions.create({ messages: [ { role: "user", content: [ { type: "text", text: prompt }, { type: "image_url", image_url: { url: image_url }, // biome-ignore lint/suspicious/noExplicitAny: SDK not updated } as any, ], }, ], temperature: 0, model: "llava-v1.5-7b-4096-preview", max_tokens: 300, }); } return { callLlava, }; } ``` ### src/hooks/use-llama-vision.ts ```typescript import Groq from "groq-sdk"; export function useLlamaVision(apiKey: string) { const groq = new Groq({ apiKey, dangerouslyAllowBrowser: true, }); function callLlamaVision(prompt: string, image_url: string) { return groq.chat.completions.create({ messages: [ { role: "user", content: [ { type: "text", text: prompt }, { type: "image_url", image_url: { url: image_url }, // biome-ignore lint/suspicious/noExplicitAny: SDK not updated } as any, ], }, ], temperature: 0, model: "llama-3.2-11b-vision-preview", max_tokens: 300, }); } return { callLlamaVision, }; } ``` ### src/lib/provider-factory.tsx ```typescript import { type Context, createContext, useContext } from "react"; import type React from "react"; export function providerFactory( hook: (options?: unknown) => T, ): [({ children }: { children: React.ReactNode }) => JSX.Element, () => T] { // 1. Create Context const context = createContext({} as T); // 2. Create Provider const Provider = ({ children }: { children: React.ReactNode }) => { const state = hook(); return {children}; }; // 3. Create Hook const useProvider = () => useContextOrDie(context); // Expose Provider and Hook return [Provider, useProvider]; } function useContextOrDie(context: Context): T { const value = useContext(context); if (!value) { throw new Error("trying to use context outside a valid context"); } return value; } ``` [26 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]