# Project export: CogniBoost This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: We create individualized AI-powered cognitive games for the elders with Alzheimer's, utilizing predictive analysis to create interactive AR learning tasks. - Devpost: https://devpost.com/software/cogiboost - GitHub: https://github.com/rg0727/cogtaskapp - Video: https://www.youtube.com/embed/vhkVVpppN8k?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Zixin Xu (23 commits), Nathan Chow (21 commits), Raksha Govind (7 commits), Isita (3 commits) ## Devpost submission (written by the team) ### Inspiration Have you ever worried about your grandma after something she said? Don't worry, we have too. Over 7 million Americans don’t realize they have mild cognitive decline (MCI), and studies show that by age 70, two out of three will experience some level of cognitive impairment. Nearly 37% of women and 24% of men are at risk for long-term cognitive issues, making brain health a growing concern—not just for older adults, but for seemingly healthy individuals too. Early diagnosis and proactive care are key to preventing our loved one's long-term damage. For those already showing signs, it's crucial to provide accurate diagnostics and connect them with smart, easy-to-use digital tools that fit into our everyday lives. With the rise of GenAI and advanced visual-intelligence & databases for predictive healthcare, we went out to create a novel multimodal way for individuals to participate in cognitive and neuroscience-backed tasks through their own household objects. ### What it does CogniBoost uses your household objects and advanced computer-vision, augmented reality, and RAGs to process multimodal sources of input, mostly visual data, and spatiotemporal marked tasks to generate dynamic short-length tasks for your everyday acute-cognition strength. The goalWe focus on three of the major contributors to growth in sharpness and their useful visuospatial tasks: Users upload photos of objects around them, and CogniBoost uses our built vector-search & database (on top of InterSystems IRIS) and generative AI to pull users' personal similar yet contextually different multimodal data. These image activities are used to strengthen their understanding of logical differences in their everyday objects and their personal belongings (task backed by NIH), which are then validated and scored by an LLM. Users upload photos of objects around them, and CogniBoost uses our built vector-search & database (on top of InterSystems IRIS) and generative AI to pull users' personal similar yet contextually different multimodal data. These image activities are used to strengthen their understanding of logical differences in their everyday objects and their personal belongings (task backed by NIH), which are then validated and scored by an LLM. CogniBoost directs users through neurophysical tasks (i.e. classification, logic puzzles, height-ordering, garden and species identification) generated through spatiotemporal scene understanding and LLMs. CogniBoost directs users through neurophysical tasks (i.e. classification, logic puzzles, height-ordering, garden and species identification) generated through spatiotemporal scene understanding and LLMs. CogniBoost also looks at the diagnostic side of preventative healthcare, using feature capabilities for quick diagnostics with basic predictive analytics through hand-tracking computer vision tasks. CogniBoost also looks at the diagnostic side of preventative healthcare, using feature capabilities for quick diagnostics with basic predictive analytics through hand-tracking computer vision tasks. ### How we built it See our diagram below. ### Challenges we ran into We had a lot of core functionalities we were super pumped about building out with some great APIs. However, a lot of my time was spent figuring out how to efficiently route our data processing to our three servers, which all had varying LLM, RAG, or CV pipelines. It was really useful to have cool vector stores + search capabilities through our data that made this process way better and centralized to our case. Definitely challenging trying to get our advanced visual backend to render well in this short 36 hours - but it worked out! ### Accomplishments we're proud of We were able to integrate various types of APIs and create a really immersive application that touched on different pain points we wanted to hit traditional cognitive/physical task systems (hardware costs, not immersive, not engaging). ### What we learned Merging with so many APIs + pipelines can be hard but its doable! ### What's next Picture this: Grandma steps into her garden, pulls out CogniBoost, and we ask her to take a photo. Our scene recognition system identifies her gorgeous garden, suggests a flower arrangement, and guides her to organize them by flower types. Your grandma probably doesn’t even know it, but she’s doing what she loves with the items closest to her and taking steps towards her long-term neurogenic future. ## README (from the GitHub repository) # CogniBoost Users upload photos of objects around them, and CogniBoost uses our built vector-search & database (on top of InterSystems IRIS) and generative AI to pull users' personal similar yet contextually different multimodal data. These image activities are used to strengthen their understanding of logical differences in their everyday objects and their personal belongings (task backed by NIH), which are then validated and scored by an LLM. CogniBoost directs users through neurophysical tasks (i.e. classification, logic puzzles, height-ordering, garden and species identification) generated through spatiotemporal scene understanding and LLMs. ## Detected evidence (automated analysis) Indexed codebase: 47 recognized source files, 122 KB. - CSS (language) — detected in the code - Express (technology) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - Hugging Face (technology) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code - Swift (language) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Vercel (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (88 of 88) ``` .DS_Store .env .gitignore api/.gitignore api/app.py api/backend/.DS_Store api/backend/app2.py api/backend/clip.py api/backend/db.py api/mistral_image_understand.py api/openai_image_understand.py api/requirements.txt AR-Comp/.DS_Store AR-Comp/AR-Cog.xcodeproj/project.pbxproj AR-Comp/AR-Cog.xcodeproj/project.xcworkspace/contents.xcworkspacedata AR-Comp/AR-Cog.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist AR-Comp/AR-Cog.xcodeproj/project.xcworkspace/xcuserdata/cocox.xcuserdatad/UserInterfaceState.xcuserstate AR-Comp/AR-Cog.xcodeproj/xcshareddata/xcschemes/CoreML in ARKit.xcscheme AR-Comp/AR-Cog.xcodeproj/xcuserdata/cocox.xcuserdatad/xcdebugger/Breakpoints_v2.xcbkptlist AR-Comp/AR-Cog.xcodeproj/xcuserdata/cocox.xcuserdatad/xcschemes/xcschememanagement.plist AR-Comp/AR-Cog.xcodeproj/xcuserdata/hanleyweng.xcuserdatad/xcschemes/xcschememanagement.plist AR-Comp/AR-Cog/.DS_Store AR-Comp/AR-Cog/AppDelegate.swift AR-Comp/AR-Cog/art.scnassets/ship.scn AR-Comp/AR-Cog/Assets.xcassets/AppIcon.appiconset/Contents.json AR-Comp/AR-Cog/Base.lproj/LaunchScreen.storyboard AR-Comp/AR-Cog/Base.lproj/Main.storyboard AR-Comp/AR-Cog/FastViTMA36F16.mlpackage/Data/com.apple.CoreML/model.mlmodel AR-Comp/AR-Cog/FastViTMA36F16.mlpackage/Manifest.json AR-Comp/AR-Cog/Inceptionv3.mlmodel AR-Comp/AR-Cog/Info.plist AR-Comp/AR-Cog/MobileNet.mlmodel AR-Comp/AR-Cog/Resnet50.mlmodel AR-Comp/AR-Cog/SpeechRecognizer.swift AR-Comp/AR-Cog/ViewController.swift AR-Detection/.DS_Store AR-Detection/.gitignore AR-Detection/index.html AR-Detection/package.json AR-Detection/server.js audiorecord.py frontend/.gitignore frontend/components.json frontend/eslint.config.mjs frontend/next.config.ts frontend/package.json frontend/pages/api/get-image.js frontend/postcss.config.mjs frontend/README.md frontend/src/app/(dashboard)/activity/game/game1/page.tsx frontend/src/app/(dashboard)/activity/game/game2/page.tsx frontend/src/app/(dashboard)/activity/game/game3/page.tsx frontend/src/app/(dashboard)/activity/game/layout.tsx frontend/src/app/(dashboard)/activity/page.tsx frontend/src/app/(dashboard)/dashboard/page.tsx frontend/src/app/(dashboard)/layout.tsx frontend/src/app/api/image/route.ts frontend/src/app/api/iris/route.ts frontend/src/app/globals.css frontend/src/app/layout.tsx frontend/src/app/page.tsx frontend/src/components/BackgroundPaths.tsx frontend/src/components/BrainAnalyticsData.tsx frontend/src/components/BrainNeuralNetwork.tsx frontend/src/components/Chat.tsx frontend/src/components/CognitiveMetrics.tsx frontend/src/components/navbar.tsx frontend/src/components/NeuralBackgroundLayout.tsx frontend/src/components/NeuralChallenge.tsx frontend/src/components/NeuralInterface.tsx frontend/src/components/NeuralInterface2.tsx frontend/src/components/ui/button.tsx frontend/src/components/ui/card.tsx frontend/src/components/ui/chart.tsx frontend/src/components/ui/input.tsx frontend/src/components/ui/progress.tsx frontend/src/components/VideoCapture.js frontend/src/lib/utils.ts frontend/tailwind.config.ts frontend/tsconfig.json install/intersystems_irispython-5.0.1-8026-cp38.cp39.cp310.cp311.cp312-cp38.cp39.cp310.cp311.cp312-macosx_10_9_universal2.whl install/intersystems_irispython-5.0.1-8026-cp38.cp39.cp310.cp311.cp312-cp38.cp39.cp310.cp311.cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl install/intersystems_irispython-5.0.1-8026-cp38.cp39.cp310.cp311.cp312-cp38.cp39.cp310.cp311.cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl install/intersystems_irispython-5.0.1-8026-cp38.cp39.cp310.cp311.cp312-cp38.cp39.cp310.cp311.cp312-win_amd64.whl install/intersystems_irispython-5.0.1-8026-cp38.cp39.cp310.cp311.cp312-cp38.cp39.cp310.cp311.cp312-win32.whl query_chat.py README.md requirements.txt ``` ### Dependencies - api/requirements.txt: blinker@==1.9.0, click@==8.1.8, Flask@==3.1.0, Flask-Cors@==5.0.0, itsdangerous@==2.2.0, Jinja2@==3.1.5, MarkupSafe@==3.0.2, Werkzeug@==3.1.3 - AR-Detection/package.json: @tensorflow-models/coco-ssd@^2.2.3, @tensorflow/tfjs@^4.0.0, express@^4.18.2 - frontend/package.json: @eslint/eslintrc@^3, @radix-ui/react-progress@^1.1.2, @radix-ui/react-slot@^1.1.2, @types/node@^20.17.19, @types/react@^19, @types/react-dom@^19, class-variance-authority@^0.7.1, clsx@^2.1.1, eslint@^9, eslint-config-next@15.1.7, framer-motion@^12.4.3, fs@^0.0.1-security, lucide-react@^0.475.0, next@^15.1.7, path@^0.12.7, postcss@^8, react@^19.0.0, react-dom@^19.0.0, recharts@^2.15.1, socket.io-client@^4.8.1, tailwind-merge@^3.0.1, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5 - requirements.txt: flask, flask_cors, flask_socketio, iris, mediapipe, numpy, openai, openai-clip, opencv-python, pillow, pyaudio, python-dotenv, torch, transformers ### Recent commits (newest first) - Update naming - Update README.md - AR-cog - Merge pull request #9 from rg0727/isita_capture - Merge branch 'isita_capture' of https://github.com/rg0727/cogtaskapp into isita_capture - add: iris integration - Remote Image query finished - fix: merge conflicts - add: video capture logic - Merge branch 'isita_capture' of https://github.com/rg0727/cogtaskapp into isita_capture - add: image retreival from db - Finished Game 3 - Merge pull request #8 from rg0727/isita_capture - added more - added logic for handling game1 and game2 with video capture - Merge pull request #7 from rg0727/feat/receive-audio-input - Fixed leak - Fixed issue - fix: audio changes - capture works; still need to pass to parent ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` torch transformers openai-clip pillow numpy iris opencv-python mediapipe flask flask_socketio flask_cors pyaudio python-dotenv openai ``` ### api/requirements.txt ``` blinker==1.9.0 click==8.1.8 Flask==3.1.0 Flask-Cors==5.0.0 itsdangerous==2.2.0 Jinja2==3.1.5 MarkupSafe==3.0.2 Werkzeug==3.1.3 ``` ### AR-Detection/package.json ``` { "name": "tensorflow-ar-detection", "version": "1.0.0", "description": "AR object detection using TensorFlow.js and COCO-SSD", "main": "server.js", "scripts": { "start": "node server.js" }, "dependencies": { "@tensorflow/tfjs": "^4.0.0", "@tensorflow-models/coco-ssd": "^2.2.3", "express": "^4.18.2" }, "devDependencies": {}, "engines": { "node": ">=14.0.0" } } ``` ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev --turbopack", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@radix-ui/react-progress": "^1.1.2", "@radix-ui/react-slot": "^1.1.2", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "framer-motion": "^12.4.3", "fs": "^0.0.1-security", "lucide-react": "^0.475.0", "next": "^15.1.7", "path": "^0.12.7", "react": "^19.0.0", "react-dom": "^19.0.0", "recharts": "^2.15.1", "socket.io-client": "^4.8.1", "tailwind-merge": "^3.0.1", "tailwindcss-animate": "^1.0.7" }, "devDependencies": { "@eslint/eslintrc": "^3", "@types/node": "^20.17.19", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "15.1.7", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "^5" } } ``` ### AR-Detection/server.js ```javascript const express = require('express'); const path = require('path'); const app = express(); const port = 3000; // Serve static files from the current directory app.use(express.static(path.join(__dirname, '/'))); // Start the server app.listen(port, () => { console.log(`Server running at http://localhost:${port}`); }); ``` ### api/app.py ```python from flask import Flask from flask_cors import CORS from flask_socketio import SocketIO from openai_image_understand import ask_openai import pyaudio import wave import speech_recognition as sr import threading import time from backend.db import handle_capture_iris from openai_image_understand import ask_openai app = Flask(__name__) CORS(app) socketio = SocketIO(app, cors_allowed_origins="*") # Audio Parameters FORMAT = pyaudio.paInt16 CHANNELS = 1 RATE = 44100 CHUNK = 2048 OUTPUT_FILENAME = "output.wav" audio = pyaudio.PyAudio() stream = None frames = [] recording_thread = None recording = False # Flag to control recording state recognizer = sr.Recognizer() # Function to record audio in a background thread def record_audio_chunks(): global stream, frames, recording recording = True print("Recording started...") try: while recording: try: data = stream.read(CHUNK, exception_on_overflow=False) frames.append(data) except Exception as e: print(f"Error while reading audio chunk: {e}") break except Exception as e: print(f"Recording thread error: {e}") # Start Recording def start_recording(): global stream, frames, recording_thread frames = [] stream = audio.open(format=FORMAT, channels=CHANNELS, rate=RATE, input=True, frames_per_buffer=CHUNK) recording_thread = threading.Thread(target=record_audio_chunks) recording_thread.daemon = True recording_thread.start() # Stop Recording def stop_recording(): global stream, frames, recording, recording_thread if stream is None: return "No active recording to stop.", 400 recording = False # Stop recording loop time.sleep(0.1) # Ensure buffer flush before closing stream.stop_stream() stream.close() if recording_thread: recording_thread.join() # Save the recorded audio to a WAV file with wave.open(OUTPUT_FILENAME, 'wb') as wavefile: wavefile.setnchannels(CHANNELS) wavefile.setsampwidth(audio.get_sample_size(FORMAT)) wavefile.setframerate(RATE) wavefile.writeframes(b''.join(frames)) print(f"Recording stopped and saved to {OUTPUT_FILENAME}") return OUTPUT_FILENAME # Transcribe Audio def transcribe_audio(): with sr.AudioFile(OUTPUT_FILENAME) as source: print("Transcribing audio...") audio_data = recognizer.record(source) try: text = recognizer.recognize_google(audio_data) print("Transcription:", text) return text except sr.UnknownValueError: print("Could not understand the audio") return "Could not understand the audio" except sr.RequestError: print("Error with the speech recognition service") return "Error with the speech recognition service" # WebSocket Handlers @socketio.on('start_recording') def handle_start_recording(): start_recording() socketio.emit('recording_status', {"status": "Recording started"}) @socketio.on('stop_recording') def handle_stop_recording(): result = stop_recording() socketio.emit('recording_status', {"status": "Recording stopped", "filename": result}) @socketio.on('transcribe_audio') def handle_transcribe_audio(): transcription = transcribe_audio() socketio.emit('transcription', {"transcription": transcription}) # Ensure cleanup on exit def cleanup(): global stream, audio if stream: stream.stop_stream() stream.close() audio.terminate() print("Cleaned up resources.") def process_frame(msg): ask_openai(img_url=msg) @socketio.on('message') def handle_message(msg): if msg['id'] == 1: print("game1") image = msg['image'] most_sim = handle_capture_iris(image) SocketIO.emit('response', most_sim, room=request.sid) elif msg['id'] == 2: print("game2") process_frame(msg['image']) import atexit atexit.register(cleanup) if __name__ == '__main__': socketio.run(app, port=8080, debug=True) ``` ### frontend/src/app/layout.tsx ```typescript import type { Metadata } from "next"; import { Geist, Geist_Mono } from "next/font/google"; import "./globals.css"; const geistSans = Geist({ variable: "--font-geist-sans", subsets: ["latin"], }); const geistMono = Geist_Mono({ variable: "--font-geist-mono", subsets: ["latin"], }); export const metadata: Metadata = { title: "Create Next App", description: "Generated by create next app", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### frontend/src/app/page.tsx ```typescript import Image from "next/image"; import Link from "next/link"; import { Button } from "@/components/ui/button"; import { Brain, Zap, BarChart, Users } from "lucide-react"; export default function Home() { return (
CogniBoost

Boost Your Cognitive Health

Enhance your mental clarity, focus, and memory with our scientifically-backed cognitive health app.

Key Features

} title="Brain Training Exercises" description="Engaging games and puzzles designed to improve cognitive function." /> } title="Progress Tracking" description="Monitor your cognitive improvements over time with detailed analytics." /> } title="Community Support" description="Connect with others on their cognitive health journey for motivation and tips." />
App screenshot

Your Path to Better Cognitive Health

Our app provides personalized brain training programs, mindfulness exercises, and nutrition tips to help you achieve optimal cognitive health. Start your journey to a sharper mind today.

Ready to Boost Your Brain Power?

Download CogniBoost now and start your journey to better cognitive health. Available on iOS and Android.