# Project export: DeeR This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Deep Retention of Concepts using research-backed techniques! - Devpost: https://devpost.com/software/deer - GitHub: https://github.com/Priyansh4444/DeeR - Video: https://www.youtube.com/embed/1LN9q6boel8?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Agora: Best Practice of TEN) - Team: 1 GitHub contributor(s) β€” Priyansh Shah (55 commits) ## Devpost submission (written by the team) ### Inspiration Jumping from study habit to study habit is a loop hole, and I had been stuck in it for a long time! I wanted to build something which had intersection in the domains of studying, neuroscience, and computer science, so I built DeeR. I can finally scrap all the useless apps that I have and be proud to use my own :D Heavily reccommend watching this video: https://youtube.com/shorts/icfI_iVLeRs?si=1AhrOj9470H4WgDW since its the basis of this entire project ### What it does DeeR is an AI-powered study companion that: Uses AI to detect emotional conditions which I have a hard time recognizing about myself Implements the Feynman technique, one of the best approaches in learning techniques that has served me well for the past few weeks, which actively makes me understand rather than brute force remembrance Adds Study Cycles Creates workflows Generates nice summaries from PDFs, PPTs, and other document formats Emotional Analysis to see when you are distressed and reccommending you to take a break Hume Model to talk about what you learned about to improve your recalling and retention ### Challenges we ran into Cartesia API credits reaching 662% of the free usage limit 🀭 Hume API requiring manual screenshots for each inference rather than connecting through WebSockets Managing time and presenting solo ### Accomplishments we're proud of Integrating multiple technologies and participating in various tracks, especially as a solo developer Overcoming anxiety by participating in competitions like these Successfully building a functional prototype of DeeR ### What we learned Solo hackathons are long but fun No merge conflicts when working alone! Discovered interesting aspects of Hume and Deepgram's model training approaches Improved skills in working with WebSockets and fetch requests Enhanced ability to read documentation and manage time effectively ### What's next Adding authentication for user accounts Adding advice on what to do when the person is distressed Implementing multi-user support with individual LLM preferences Adding a feature to skip voice lines by Cartesia Fine-tuning models for better outcomes Made with ❀️ ## README (from the GitHub repository) # DeeR (Deep Retention) 🦌 ![DeeR Logo](public/DeeR.png) ![Cal Hacks 11.0 Winner](https://img.shields.io/badge/Cal%20Hacks%2011.0-Winner-gold) ![Python](https://img.shields.io/badge/Python-3.8%2B-blue) ![Next.js](https://img.shields.io/badge/Next.js-14.0%2B-black) [Watch Demo Video](https://youtu.be/1LN9q6boel8) | [DevPost](https://devpost.com/software/deer) ## About DeeR DeeR is an emotionally intelligent AI study companion that helps students learn and retain information. Born from the intersection of neuroscience, computer science, and educational psychology, DeeR implements the renowned Richard Feynman technique while actively monitoring and responding to students' emotional states during their learning journey. ## The Challenge Being Addressed In today's fast-paced academic environment, an alarming 70% of students develop mental health issues due to academic pressure. The rapid scaling of course difficulty often leaves students struggling to keep pace, leading many to abandon their passions or drop out entirely. Traditional solutions often focus on external study techniques, overlooking a crucial internal factor: emotions. Research shows that we are constantly emotional beings, with blood flowing through our brain's emotional circuitry at all times, significantly impacting our ability to learn and retain information. ![Home Page](public/HomePage.png) ## Our Solution DeeR takes a revolutionary approach by combining emotional intelligence with proven learning techniques. Studies have demonstrated that positive emotional states like concentration and calmness can increase retention by up to 50% during study sessions. Conversely, distress can decrease learning efficiency by 30% and trigger negative feedback loops that erode self-confidence and academic performance. Our platform actively monitors emotional states through advanced AI, detecting subtle changes in concentration, calmness, and distress levels. When the system identifies optimal learning conditions, it encourages continued engagement. If signs of distress appear, DeeR provides timely interventions to prevent negative spirals and maintain effective learning patterns. ![User Interface](public/UI.png) ## Key Features DeeR seamlessly integrates multiple technologies to create a comprehensive learning environment. The platform performs real-time emotion detection using sophisticated AI models (DeepGram to record stutters and Hume for getting Emotions through voice!), while implementing the Feynman technique to enhance understanding and retention. Users can create customized study cycles and workflows, while the system automatically generates concise summaries from various document formats. When users show signs of distress, DeeR intelligently suggests breaks and adjustments to maintain optimal learning conditions. ![Tech Stack](public/TechStack.png) ## Getting Started To run DeeR locally, follow these simple steps: ```bash # Clone the repository git clone https://github.com/yourusername/deer.git cd deer # Install dependencies pip install -r requirements.txt npm install # Run the application python server.py # Backend npm run dev # Frontend ``` ## The Team DeeR was developed as a solo project by me🦌, a Computer Science student at UCI, during Cal Hacks 11.0. This project represents a fusion of personal experience with academic challenges and a desire to create meaningful solutions for students worldwide. I would also like to thank HealthyGamerGG, since without their inspiration I couldn't come up with this idea and I would also be stuck in that Negative Feedback Loop! ## Acknowledgments This project wouldn't have been possible without the support of Cal Hacks 11.0, our API providers, and the broader neuroscience research community. Their contributions have been instrumental in creating a tool that we believe will make a real difference in students' academic journeys. --- ### Made with ❀️ at Cal Hacks 11.0 ## Detected evidence (automated analysis) Indexed codebase: 30 recognized source files, 93 KB. - CSS (language) β€” detected in the code - FastAPI (technology) β€” detected in the code - Next.js (technology) β€” detected in the code - Python (language) β€” detected in the code - React (technology) β€” detected in the code - Tailwind CSS (technology) β€” detected in the code - TypeScript (language) β€” detected in the code - Google Gemini (technology) β€” claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (41 of 41) ``` .env.example .eslintrc.json .gitignore bun.lockb components.json LICENSE next.config.mjs package.json perplexity.py postcss.config.mjs README.md requirements.txt server.py src/app/channel/[channelName]/page.tsx src/app/globals.css src/app/layout.tsx src/app/page.tsx src/components/Book.tsx src/components/Controls.tsx src/components/EmotionalAnalysis.tsx src/components/Experience.tsx src/components/GlobalContext.tsx src/components/HumeVoiceComponent.tsx src/components/RagUpload.tsx src/components/Timer.tsx src/components/UI.tsx src/components/ui/button.tsx src/components/ui/card.tsx src/components/ui/input.tsx src/components/ui/scroll-area.tsx src/components/ui/toast.tsx src/components/ui/toaster.tsx src/components/VideoCall.tsx src/hooks/api.ts src/hooks/SpeechToText.ts src/hooks/types.ts src/hooks/use-toast.ts src/hooks/useMessages.ts src/lib/utils.ts tailwind.config.ts tsconfig.json ``` ### Dependencies - package.json: @cartesia/cartesia-js@^1.1.0, @deepgram/sdk@^3.8.1, @humeai/voice-react@^0.1.16, @radix-ui/react-icons@^1.3.0, @radix-ui/react-scroll-area@^1.2.0, @radix-ui/react-slot@^1.1.0, @radix-ui/react-toast@^1.2.2, @react-three/drei@^9.114.6, @react-three/fiber@^8.17.10, @types/node@^20, @types/react@^18, @types/react-dom@^18, agora-rtc-react@^2.3.0, agora-rtc-sdk-ng@^4.22.1, class-variance-authority@^0.7.0, clsx@^2.1.1, cross-fetch@^4.0.0, dotenv@^16.4.5, eslint@^8, eslint-config-next@14.2.15, formidable@^3.5.2, framer-motion@^11.11.9, hume@^0.9.1, jotai@^2.10.1, lucide-react@^0.453.0, next@14.2.15, postcss@^8, react@^18, react-dom@^18, reconnecting-websocket@^4.4.0, styled-components@^6.1.13, tailwind-merge@^2.5.4, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5, typewriter-effect@^2.21.0 - requirements.txt: chromadb, fastapi, google-generativeai, groq, hume, numpy, opencv-python, perplexity, pydantic, PyPDF2, python-dotenv, requests, uvicorn ### Recent commits (newest first) - Create LICENSE - Correcting and Adding things to the Readme - adding readme and pictures - set game and finish - Cleaning code for readibility - I won!!! :D - making it autumn colours! - Partially done but I dont wanna touch the repo anymore - adding requirements to run the code - changin g it back temporarily - changing chain of though! - finishing dummy ui step - removing a temp file - adding timeline / learnign path for nextJS - UI UI UI UI UI - Doing some UI tings here and there - fixing multiple things - hydrated error disappear - DeeR - feat: finishing home page ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` fastapi uvicorn pydantic python-dotenv requests google-generativeai groq PyPDF2 chromadb opencv-python numpy hume perplexity ``` ### package.json ``` { "name": "calhacks", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@cartesia/cartesia-js": "^1.1.0", "@deepgram/sdk": "^3.8.1", "@humeai/voice-react": "^0.1.16", "@radix-ui/react-icons": "^1.3.0", "@radix-ui/react-scroll-area": "^1.2.0", "@radix-ui/react-slot": "^1.1.0", "@radix-ui/react-toast": "^1.2.2", "@react-three/drei": "^9.114.6", "@react-three/fiber": "^8.17.10", "agora-rtc-react": "^2.3.0", "agora-rtc-sdk-ng": "^4.22.1", "class-variance-authority": "^0.7.0", "clsx": "^2.1.1", "cross-fetch": "^4.0.0", "dotenv": "^16.4.5", "formidable": "^3.5.2", "framer-motion": "^11.11.9", "hume": "^0.9.1", "jotai": "^2.10.1", "lucide-react": "^0.453.0", "next": "14.2.15", "react": "^18", "react-dom": "^18", "reconnecting-websocket": "^4.4.0", "styled-components": "^6.1.13", "tailwind-merge": "^2.5.4", "tailwindcss-animate": "^1.0.7", "typewriter-effect": "^2.21.0" }, "devDependencies": { "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "eslint": "^8", "eslint-config-next": "14.2.15", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "^5" } } ``` ### server.py ```python from functools import lru_cache import io from typing import List from PyPDF2 import PdfReader from fastapi import FastAPI, HTTPException, UploadFile, File, WebSocket, WebSocketDisconnect from fastapi.middleware.cors import CORSMiddleware from pydantic import BaseModel import chromadb import uvicorn import logging import cv2 import numpy as np from hume import AsyncHumeClient from hume.expression_measurement.stream import Config from hume.expression_measurement.stream.socket_client import StreamConnectOptions from hume.expression_measurement.stream.types import StreamFace import asyncio from perplexity import multi_model_learning_chain from dotenv import load_dotenv import os load_dotenv() HUME_API_KEY = os.getenv("NEXT_PUBLIC_HUME_API_KEY") # Configure logging logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') app = FastAPI() # Add CORS middleware app.add_middleware( CORSMiddleware, allow_origins=["*"], # Allows all origins allow_credentials=True, allow_methods=["*"], # Allows all methods allow_headers=["*"], # Allows all headers ) # Add CORS middleware app.add_middleware( CORSMiddleware, allow_origins=["*"], # Allows all origins allow_credentials=True, allow_methods=["*"], # Allows all methods allow_headers=["*"], # Allows all headers ) # Initialize Chromadb client client = chromadb.PersistentClient(path="./chroma_db") # Get or create a collection for our documents collection = client.get_or_create_collection(name="documents") # List of emotions we're interested in RELEVANT_EMOTIONS = ['Calmness', 'Concentration', 'Distress'] class QueryItem(BaseModel): query: str class ContentItem(BaseModel): content: str @app.post("/add-to-chroma") async def add_to_chroma(item: ContentItem): collection.add( documents=[item.content], ids=[f"doc_{collection.count() + 1}"] ) return {"status": "success", "message": "Content added to Chromadb"} @app.post("/query-chroma") async def query_chroma(item: QueryItem): results = collection.query( query_texts=[item.query], n_results=5 ) return {"results": results['documents'][0]} @app.post("/upload-file") async def upload_file(file: UploadFile = File(...)): try: contents = await file.read() if file.filename.lower().endswith('.pdf'): # Handle PDF files pdf_reader = PdfReader(io.BytesIO(contents)) text_content = "" for page in pdf_reader.pages: text_content += page.extract_text() + "\n" elif file.filename.lower().endswith(('.txt', '.md', '.py', '.js', '.html', '.css')): # Handle text-based files text_content = contents.decode('utf-8', errors='ignore') else: # For other file types, store file info or handle as needed text_content = f"File uploaded: {file.filename} (binary file)" # Add file content to Chromadb collection.add( documents=[text_content], ids=[f"file_{collection.count() + 1}"] ) return {"status": "success", "message": f"File {file.filename} uploaded and added to Chromadb"} except Exception as e: logging.error(f"Error uploading file: {str(e)}") return {"status": "error", "message": str(e)} @lru_cache(maxsize=1) def get_hume_client(): return AsyncHumeClient(api_key=HUME_API_KEY) async def process_image_stream(websocket: WebSocket): client = get_hume_client() model_config = Config(face=StreamFace()) stream_options = StreamConnectOptions(config=model_config) async with client.expression_measurement.stream.connect(options=stream_options) as socket: while True: try: image_data = await asyncio.wait_for(websocket.receive_bytes(), timeout=5.0) # Convert raw frame data to numpy array nparr = np.frombuffer(image_data, np.uint8) image = nparr.reshape((480, 640, 4)) image_bgr = cv2.cvtColor(image, cv2.COLOR_RGBA2BGR) # Save the image to a temporary file temp_file = "temp_image.png" cv2.imwrite(temp_file, image_bgr) # Send the file to Hume result = await socket.send_file(temp_file) # Clean up the temporary file os.remove(temp_file) # Process the result if hasattr(result, 'face') and result.face.predictions: predictions = result.face.predictions[0] processed_emotions = process_emotions(predictions.emotions) await websocket.send_json({ "emotions": processed_emotions, "face_detected": True, "face_probability": predictions.prob }) else: await websocket.send_json({ "emotions": [], "face_detected": False, "error": "No face detected" }) except asyncio.TimeoutError: logging.warning("WebSocket receive timeout") except Exception as e: logging.error(f"Error in image processing: {str(e)}") await websocket.send_json({"error": str(e)}) break def process_emotions(emotions): relevant_emotions = { emotion.name: emotion.score for emotion in emotions if emotion.name in RELEVANT_EMOTIONS } return [{"name": name, "score": score} for name, score in relevant_emotions.items()] @app.websocket("/ws") async def websocket_endpoint(websocket: WebSocket): await websocket.accept() try: await process_image_stream(websocket) except WebSocketDisconnect: logging.info("WebSocket disconnected") finally: [truncated β€” 1386 more characters] ``` ### src/app/layout.tsx ```typescript import type { Metadata } from "next"; import localFont from "next/font/local"; import "./globals.css"; import { GlobalStateProvider } from "@/components/GlobalContext"; import { Toaster } from '../components/ui/toaster'; const JetBrainsMono = localFont({ src: "./fonts/JetBrainsMono-Regular.ttf", variable: "--font-geist-sans", weight: "100 900", }); export const metadata: Metadata = { title: "Create Next App", description: "Generated by create next app", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return (
{children}
); } ``` ### src/app/page.tsx ```typescript "use client"; import { useRouter } from "next/navigation"; import { Loader, PointMaterial, Points } from "@react-three/drei"; import { Canvas, PointsProps, useFrame } from "@react-three/fiber"; import { Suspense, useRef, useState } from "react"; import { Experience } from "@/components/Experience"; import { UI } from "@/components/UI"; // @ts-expect-error - no types available import * as random from "maath/random/dist/maath-random.esm"; import { BufferGeometry, Material, Points as ThreePoints, } from "three/src/Three.js"; export default function Home() { const router = useRouter(); const [isOpen, setIsOpen] = useState(false); return (
{/* Blurred background */}
{/* Foreground form */}

DeeR

{ e.preventDefault(); const target = e.target as typeof e.target & { channel: { value: string }; }; router.push(`/channel/${target.channel.value}`); }} className="flex flex-col items-center space-y-4" >
{/* Button to open form */} {!isOpen && ( )}
); } type PointsInstance = ThreePoints; function Stars(props: PointsProps) { const ref = useRef(null); const [sphere] = useState(() => random.inSphere(new Float32Array(3000), { radius: 3 }) ); useFrame((state, delta) => { ref.current!.rotation.x -= delta / 10; ref.current!.rotation.y -= delta / 15; }); return ( ); } ``` ### src/app/channel/[channelName]/page.tsx ```typescript import Call from "@/components/VideoCall"; export default function Page({ params }: { params: { channelName: string } }) { return (

{params.channelName!}

); } ``` ### tailwind.config.ts ```typescript import type { Config } from "tailwindcss"; const { default: flattenColorPalette, } = require("tailwindcss/lib/util/flattenColorPalette"); const config: Config = { darkMode: ["class"], content: [ "./src/pages/**/*.{js,ts,jsx,tsx,mdx}", "./src/components/**/*.{js,ts,jsx,tsx,mdx}", "./src/app/**/*.{js,ts,jsx,tsx,mdx}", ], theme: { extend: { animation: { shimmer: "shimmer 2s linear infinite", }, keyframes: { shimmer: { from: { backgroundPosition: "0 0", }, to: { backgroundPosition: "-200% 0", }, }, }, colors: { background: "hsl(var(--background))", foreground: "hsl(var(--foreground))", card: { DEFAULT: "hsl(var(--card))", foreground: "hsl(var(--card-foreground))", }, popover: { DEFAULT: "hsl(var(--popover))", foreground: "hsl(var(--popover-foreground))", }, primary: { DEFAULT: "hsl(var(--primary))", foreground: "hsl(var(--primary-foreground))", }, secondary: { DEFAULT: "hsl(var(--secondary))", foreground: "hsl(var(--secondary-foreground))", }, muted: { DEFAULT: "hsl(var(--muted))", foreground: "hsl(var(--muted-foreground))", }, accent: { DEFAULT: "hsl(var(--accent))", foreground: "hsl(var(--accent-foreground))", }, destructive: { DEFAULT: "hsl(var(--destructive))", foreground: "hsl(var(--destructive-foreground))", }, border: "hsl(var(--border))", input: "hsl(var(--input))", ring: "hsl(var(--ring))", chart: { "1": "hsl(var(--chart-1))", "2": "hsl(var(--chart-2))", "3": "hsl(var(--chart-3))", "4": "hsl(var(--chart-4))", "5": "hsl(var(--chart-5))", }, }, borderRadius: { lg: "var(--radius)", md: "calc(var(--radius) - 2px)", sm: "calc(var(--radius) - 4px)", }, }, }, plugins: [require("tailwindcss-animate"), addVariablesForColors], }; export default config; function addVariablesForColors({ addBase, theme }: any) { let allColors = flattenColorPalette(theme("colors")); let newVars = Object.fromEntries( Object.entries(allColors).map(([key, val]) => [`--${key}`, val]) ); addBase({ ":root": newVars, }); } ``` ### perplexity.py ```python import os from groq import Groq import google.generativeai as genai import requests from dotenv import load_dotenv def multi_model_learning_chain(concept): print("Starting multi_model_learning_chain") load_dotenv() groq_api_key = os.environ.get("GROQ_API_KEY") gemini_api_key = os.environ.get("GEMINI_API_KEY") hyperbolic_api_key = os.environ.get("NEXT_PUBLIC_HYPERBOLIC_API_KEY") print("API keys loaded") # Stage 1: Initial Analysis (Hyperbolic) print("Starting Hyperbolic analysis") url = "https://api.hyperbolic.xyz/v1/chat/completions" headers = { "Content-Type": "application/json", "Authorization": f"Bearer {hyperbolic_api_key}" } data = { "messages": [ { "role": "user", "content": f"Provide a comprehensive and deep analysis of the concept: {concept}. Include key principles, theories, and any relevant scientific or academic context." } ], "model": "meta-llama/Meta-Llama-3.1-70B-Instruct", "max_tokens": 2048, "temperature": 0.7, "top_p": 0.9 } print("Sending request to Hyperbolic API") response = requests.post(url, headers=headers, json=data) print(f"Received response from Hyperbolic API. Status code: { response.status_code}") response_data = response.json() print("Response data:", response_data) initial_analysis = response_data['choices'][0]['message']['content'] print("Hyperbolic analysis completed") # Stage 2: Simplification and Feynman Technique (Gemini) print("Starting Gemini simplification and Feynman technique") genai.configure(api_key=gemini_api_key) model = genai.GenerativeModel("gemini-1.5-flash") print("Generating simple explanation") simplify_prompt = f"Simplify the following analysis for easy understanding:\n\n{ initial_analysis} with the initial analysis of {concept} as the input." simple_response = model.generate_content(simplify_prompt) print("Simple explanation generated") print("Generating Feynman explanation") feynman_prompt = f"Using the Feynman Technique, explain the concept as if teaching to someone else:\n\n{ simple_response.text}" feynman_response = model.generate_content(feynman_prompt) print("Feynman explanation generated") simple_explanation = simple_response.text feynman_explanation = feynman_response.text print("Gemini process completed") # Stage 3: Analogies and Final Summary (Groq) print("Starting Groq analogies and summary") client = Groq(api_key=groq_api_key) print("Generating analogies") analogy_prompt = f"Generate insightful analogies to explain the concept of { concept} based on this analysis:\n\n{initial_analysis} please priotiize requests from the concept of {concept}." analogy_completion = client.chat.completions.create( messages=[{"role": "user", "content": analogy_prompt}], model="llama3-8b-8192", ) analogies = analogy_completion.choices[0].message.content print("Analogies generated") print("Generating final summary") summary_prompt = f"Integrate and summarize the following information about {concept}, please follow the request of {concept} into a comprehensive learning resource:\n\nAnalysis: { initial_analysis}\n\nSimple Explanation: {simple_explanation}\n\nFeynman Technique Explanation: {feynman_explanation}\n\nAnalogies: {analogies}" summary_completion = client.chat.completions.create( messages=[{"role": "user", "content": summary_prompt}], model="llama3-8b-8192", ) final_summary = summary_completion.choices[0].message.content print("Final summary generated") return final_summary # Example usage ``` ### src/hooks/types.ts ```typescript export interface Message { role: "user" | "assistant"; content: string; } ``` ### src/lib/utils.ts ```typescript import { clsx, type ClassValue } from "clsx" import { twMerge } from "tailwind-merge" export function cn(...inputs: ClassValue[]) { return twMerge(clsx(inputs)) } ``` [21 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]