# Project export: Neurodog This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Neurodog provides therapy and essential services to those with neurological disorders who need but can't care for live animals using brain activity from EEGs, by offering immediate and reliable care. - Devpost: https://devpost.com/software/neurodog-d2cufp - GitHub: https://github.com/GhostWalker562/neurodog - Demo: https://neurodog.vercel.app/ - Video: https://www.youtube.com/embed/5GLrz6Up_lk?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Philip Vu (11 commits) ## Devpost submission (written by the team) ### Inspiration For individuals suffering from conditions like mental health or physical disabilities, service animals can offer immense improvement to a person’s quality of life. Yet, tragically, many are unable to own a service animal due to the limitations imposed by their conditions. In 2023, 74% of pet owners believed their mental health improved after getting a pet; however, less than 1% of Americans with disabilities own service dogs due to cognitive/physical incapacities that may result in animal neglect and differences in the attributes of dogs. We began to think: How can people with disabilities receive emotional support and physical assistance from service animals, without needing to shoulder the responsibility of caring for the animals' needs? ### What it does Designed as a self-functioning robotic service animal, Neurodog has the capability to recognize familiar brainwave patterns, including emotional states like "sadness" in its user, and is equipped to automatically execute corresponding commands based on those emotions, such as approaching the user for comforting interactions through play or engaging in entertaining tricks. Neurodog is a service dog that offers therapy and a range of essential medical services. By merging AI with voice commands we were able to implement into Neurodog to recognize natural language voice prompts— providing a more realistic feel while extending the range of voice functionalities available. ### How we built it Frontend was built using Next.js (React) with Shadcn UI for the components. We included a Github OAuth for dashboard login, and Neurosity OAuth to gain access to the tab containing brainwave scans and functionalities. The backend consisted of serverless functions using NextJS’s app router. This app router allowed us to develop with SSR that improved the performance of the application. We used Drizzle as the ORM and lucia-auth for the session management. Through Lucia’s documentation, we were able to spin up our oauth quickly. In addition to those tools, we used Kirimase which bootstraps NextJS applications with all the tools above with one command. For natural language comprehension, we used OpenAI’s whisper to understand speech. We then processed the response into ChatGPT’s function calls to allow us to derive a command and run it on the frontend or backend. This enabled us to do Voice Actions on the frontend. With Spot, we used Elevenlabs to generate the speech and ChatGPT’s whisper to understand recorded audio. We also used Neurosity’s SDK to allow Spot to receive brain waves from our EEG. ### Challenges we ran into This project encountered a significant number of integration challenges. Here are some interesting tidbits: Kinesis, the Motor Imagery Brain-Computer Interaction (BCI) method that allows us to trigger actions through trained brainwave patterns, needs to be consistently trained to maintain the accuracy of the brain scans and work on command. Sharing a singular Boston Dynamic SPOT dog between numerous teams, significantly reduced the time we had to execute our programs. Figuring out the Neurosity SDK, and connecting it with the SPOT SDK proved to be difficult because of the complex intricacies in both documentations. ### What's next Unlike competitor products such as Tombot which is currently only touch-activated and real live dogs, Neurodog will interact and safeguard users through voice and based on their brain activity using an electroencephalogram (EEG) allowing immediate care reliably. Neurodog will include precautionary SOS features to ensure it is protecting its user in case of emergencies We plan to target households, hospitals, and nursing homes to assist with caretaking. Neurodog will have computer vision capabilities so it will act as the eyes for its user. Neurodog will be able to detect these dangerous situations and analyze traffic signals allowing them to alert the user while protecting them. ## README (from the GitHub repository) # Neurodog Treehacks 2024 Hackathon Submission ## Prequisites Install the following tools to get started. ### Installing Runtimes - [mise-en-place](https://github.com/jdx/mise): CLI tool that can manage multiple runtime versions on a per-project basis. ```bash mise install ``` ### Installing Dependencies - [bun](https://bun.sh/): A modern package manager. ```bash bun install ``` ## Environment Variables Create a `.env` file in the root of each app and add the following environment variables (copied from `.env.example`). ```bash # When using the PlanetScale driver with Drizzle, your connection string must end with ?ssl={"rejectUnauthorized":true} instead of ?sslaccept=strict. DATABASE_URL= GITHUB_CLIENT_ID= GITHUB_CLIENT_SECRET= OPENAI_API_KEY= NEXT_PUBLIC_POSTHOG_KEY=phc_Wtvh51O7cSudfjCDoxnGhmF2OPoSs44rKeoiOBBTcH9 NEXT_PUBLIC_POSTHOG_HOST=https://app.posthog.com ``` ## Tech Stack ### Hardware - Boston Dynamics Spot - Petoi Bittle - Neurosity Crown EEG - Docker - Python ### Frontend - NextJS - TailwindCSS - Turbo - Posthog - Bun - Typescript - React ### npm Tooling - Kirimase - @tanstack/react-query - shadcn/ui - radix-ui - zod - zustand ### Backend - PlanetScale - Drizzle - Github OAuth ### APIs - OpenAI - Elevenlabs ### Hosting - Vercel - Merklebot ## Detected evidence (automated analysis) Indexed codebase: 83 recognized source files, 146 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - SQL (language) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Docker (technology) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (102 of 102) ``` .gitignore .mise.toml .npmrc apps/spot/.gitignore apps/spot/constants.py apps/spot/Dockerfile apps/spot/elevenlabs_actions.py apps/spot/main.py apps/spot/openai_actions.py apps/spot/README.md apps/spot/requirements.txt apps/spot/spot_actions.py apps/spot/spot_controller.py apps/web/.env.example apps/web/.eslintrc.json apps/web/.gitignore apps/web/app.d.ts apps/web/app/(app)/dashboard/page.tsx apps/web/app/(app)/dog/kinesis/page.tsx apps/web/app/(app)/dog/layout.tsx apps/web/app/(app)/dog/page.tsx apps/web/app/(app)/information/page.tsx apps/web/app/(app)/layout.tsx apps/web/app/(app)/map/page.tsx apps/web/app/(auth)/layout.tsx apps/web/app/(auth)/login/github/callback/route.ts apps/web/app/(auth)/login/github/route.ts apps/web/app/(auth)/login/page.tsx apps/web/app/api/transcribe-action/route.ts apps/web/app/globals.css apps/web/app/layout.tsx apps/web/app/page.tsx apps/web/components.json apps/web/components/ActionsList.tsx apps/web/components/assets/GithubIcon.tsx apps/web/components/assets/Logo.tsx apps/web/components/AvailableActions.tsx apps/web/components/Footer.tsx apps/web/components/GuidedTutorial.tsx apps/web/components/Header.tsx apps/web/components/Hero.tsx apps/web/components/Logout.tsx apps/web/components/MainNav.tsx apps/web/components/ModeToggle.tsx apps/web/components/NeurosityDashboard.tsx apps/web/components/Overview.tsx apps/web/components/PosthogWrapper.tsx apps/web/components/ProbabilityPie.tsx apps/web/components/QueryClientWrapper.tsx apps/web/components/ServiceDogForm.tsx apps/web/components/ShapeBackground.tsx apps/web/components/shapes/Shape1.tsx apps/web/components/shapes/Shape2.tsx apps/web/components/shapes/Shape3.tsx apps/web/components/shapes/Shape4.tsx apps/web/components/Spinner.tsx apps/web/components/ThemeProvider.tsx apps/web/components/ui/avatar.tsx apps/web/components/ui/button.tsx apps/web/components/ui/card.tsx apps/web/components/ui/command.tsx apps/web/components/ui/dialog.tsx apps/web/components/ui/dropdown-menu.tsx apps/web/components/ui/form.tsx apps/web/components/ui/input.tsx apps/web/components/ui/label.tsx apps/web/components/ui/menubar.tsx apps/web/components/ui/separator.tsx apps/web/components/ui/sheet.tsx apps/web/components/ui/sonner.tsx apps/web/components/ui/tabs.tsx apps/web/components/ui/ThemeToggle.tsx apps/web/components/UserNav.tsx apps/web/components/VoiceActions.tsx apps/web/config/nav.ts apps/web/drizzle.config.ts apps/web/kirimase.config.json apps/web/lib/api/mutations/logout.ts apps/web/lib/api/mutations/useTranscribeAction.ts apps/web/lib/api/queries/useAudioPermissions.ts apps/web/lib/api/stores/neurosity.ts apps/web/lib/api/stores/recordings.ts apps/web/lib/api/useRecord.ts apps/web/lib/db/index.ts apps/web/lib/db/migrate.ts apps/web/lib/db/migrations/0000_absent_gamora.sql apps/web/lib/db/migrations/meta/_journal.json apps/web/lib/db/migrations/meta/0000_snapshot.json apps/web/lib/db/schema/auth.ts apps/web/lib/env.ts apps/web/lib/lucia.ts apps/web/lib/utils.ts apps/web/next.config.mjs apps/web/package.json apps/web/postcss.config.js apps/web/README.md apps/web/tailwind.config.ts apps/web/tsconfig.json bun.lockb package.json README.md turbo.json ``` ### Dependencies - apps/spot/requirements.txt: elevenlabs, neurosity, openai, opencv-python, python-dotenv, requests, setuptools, tenacity - apps/web/package.json: @gsap/react@^2.1.0, @hookform/resolvers@^3.3.4, @lucia-auth/adapter-drizzle@1.0.2, @neurosity/sdk@^6.5.10, @planetscale/database@^1.16.0, @radix-ui/react-avatar@^1.0.4, @radix-ui/react-dialog@^1.0.5, @radix-ui/react-dropdown-menu@^2.0.6, @radix-ui/react-label@^2.0.2, @radix-ui/react-menubar@^1.0.4, @radix-ui/react-separator@^1.0.3, @radix-ui/react-slot@^1.0.2, @radix-ui/react-tabs@^1.0.4, @svgr/webpack@^8.1.0, @t3-oss/env-nextjs@^0.9.2, @tanstack/react-query@^5.20.5, @types/node@^20, @types/react@^18.2.55, @types/react-dom@^18.2.19, arctic@^1.1.6, autoprefixer@^10.4.17, class-variance-authority@^0.7.0, clsx@^2.1.0, cmdk@^0.2.1, dotenv@^16.4.4, drizzle-kit@^0.20.14, drizzle-orm@^0.29.3, drizzle-zod@^0.5.1, eslint@^8.56.0, eslint-config-next@14.1.0, gsap@^3.12.5, lucia@3.0.1, lucide-react@^0.331.0, mysql2@^3.9.1, nanoid@^5.0.5, next@14.1.0, next-themes@^0.2.1, openai@^4.28.0, postcss@^8.4.35, posthog-js@^1.106.0, react@^18.2, react-dom@^18.2, react-hook-form@^7.50.1, recharts@^2.6.2, sonner@^1.4.0, tailwind-merge@^2.2.1, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, tsx@^4.7.1, typescript@^5.3.3, zod@^3.22.4, zustand@^4.5.1 - package.json: parcel@^2.11.0, prettier@^3.1.1, turbo@latest ### Recent commits (newest first) - Add analytics - Add guided actions - Add animations - Add voice action shortcut - Revamped pages - More UI work - Finish service dog UI - Add spot to repo - Remove server for server actions - Add voice commands - first commit ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "neurodog", "private": true, "scripts": { "build": "turbo build", "dev": "turbo dev", "lint": "turbo lint", "format": "prettier --write \"**/*.{ts,tsx,md}\"" }, "devDependencies": { "parcel": "^2.11.0", "prettier": "^3.1.1", "turbo": "latest" }, "engines": { "node": ">=18" }, "packageManager": "bun@1.0.26", "workspaces": [ "apps/*", "packages/*" ] } ``` ### apps/spot/requirements.txt ``` python-dotenv openai neurosity opencv-python requests setuptools elevenlabs tenacity ``` ### apps/spot/Dockerfile ``` FROM ghcr.io/merklebot/hackathon-arm-image:master as build ARG OPENAI_API_KEY ARG XI_KEY ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1 ENV OPENAI_API_KEY $OPENAI_API_KEY ENV XI_KEY $XI_KEY ENV NEUROSITY_DEVICE_ID $NEUROSITY_DEVICE_ID ENV NEUROSITY_EMAIL $NEUROSITY_EMAIL ENV NEUROSITY_PASSWORD $NEUROSITY_PASSWORD ARG TARGETPLATFORM ARG BUILDPLATFORM ARG TARGETOS ARG TARGETARCH ARG Version ARG GitCommit RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM" COPY requirements.txt requirements.txt RUN python3.8 -m pip install -r requirements.txt COPY . . CMD ["python3.8", "main.py"] ``` ### apps/web/package.json ``` { "name": "web", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint", "db:generate": "drizzle-kit generate:mysql", "db:migrate": "tsx lib/db/migrate.ts", "db:drop": "drizzle-kit drop", "db:pull": "drizzle-kit introspect:mysql", "db:push": "drizzle-kit push:mysql", "db:studio": "drizzle-kit studio", "db:check": "drizzle-kit check:mysql" }, "dependencies": { "@gsap/react": "^2.1.0", "@hookform/resolvers": "^3.3.4", "@lucia-auth/adapter-drizzle": "1.0.2", "@neurosity/sdk": "^6.5.10", "@planetscale/database": "^1.16.0", "@radix-ui/react-avatar": "^1.0.4", "@radix-ui/react-dialog": "^1.0.5", "@radix-ui/react-dropdown-menu": "^2.0.6", "@radix-ui/react-label": "^2.0.2", "@radix-ui/react-menubar": "^1.0.4", "@radix-ui/react-separator": "^1.0.3", "@radix-ui/react-slot": "^1.0.2", "@radix-ui/react-tabs": "^1.0.4", "@svgr/webpack": "^8.1.0", "@t3-oss/env-nextjs": "^0.9.2", "@tanstack/react-query": "^5.20.5", "arctic": "^1.1.6", "class-variance-authority": "^0.7.0", "clsx": "^2.1.0", "cmdk": "^0.2.1", "drizzle-orm": "^0.29.3", "drizzle-zod": "^0.5.1", "gsap": "^3.12.5", "lucia": "3.0.1", "lucide-react": "^0.331.0", "mysql2": "^3.9.1", "nanoid": "^5.0.5", "next": "14.1.0", "next-themes": "^0.2.1", "openai": "^4.28.0", "posthog-js": "^1.106.0", "react": "^18.2", "react-dom": "^18.2", "react-hook-form": "^7.50.1", "recharts": "^2.6.2", "sonner": "^1.4.0", "tailwind-merge": "^2.2.1", "tailwindcss-animate": "^1.0.7", "zod": "^3.22.4", "zustand": "^4.5.1" }, "devDependencies": { "@types/node": "^20", "@types/react": "^18.2.55", "@types/react-dom": "^18.2.19", "autoprefixer": "^10.4.17", "dotenv": "^16.4.4", "drizzle-kit": "^0.20.14", "eslint": "^8.56.0", "eslint-config-next": "14.1.0", "postcss": "^8.4.35", "tailwindcss": "^3.4.1", "tsx": "^4.7.1", "typescript": "^5.3.3" } } ``` ### apps/spot/main.py ```python import os import time from neurosity import NeurositySDK from elevenlabs_actions import listen, speak from openai_actions import get_audio_transcript, request_action from spot_actions import bow, sit, stand_up from spot_controller import SpotController from dotenv import load_dotenv load_dotenv() ROBOT_IP = "10.0.0.3" # os.environ['ROBOT_IP'] SPOT_USERNAME = "admin" # os.environ['SPOT_USERNAME'] SPOT_PASSWORD = "2zqa8dgw7lor" # os.environ['SPOT_PASSWORD'] # Neurosity SDK last_time = 0 is_forward = False is_sitting = False neurosity = NeurositySDK( { "device_id": os.environ["NEUROSITY_DEVICE_ID"], } ) neurosity.login( { "email": os.environ["NEUROSITY_EMAIL"], "password": os.environ["NEUROSITY_PASSWORD"], } ) controller = SpotController( username=SPOT_USERNAME, password=SPOT_PASSWORD, robot_ip=ROBOT_IP ) def main(): with controller as spot: time.sleep(2) # Move head to specified positions with intermediate time.sleep spot.move_head_in_points( yaws=[0.2, 0], pitches=[0.3, 0], rolls=[0.4, 0], sleep_after_point_reached=1 ) time.sleep(3) action_count = 0 listening = True while listening: speak("I'm listening, talk now...") listen() # Get the transcription print("Getting audio transcript") user_prompt = get_audio_transcript() action_count += 1 print(user_prompt) if action_count > 5: speak("I'm tired of listening to you, goodbye!") print("Stopping") listening = False break if "stop" in user_prompt.lower(): speak("Goodbye!") print("Stopping") listening = False break if user_prompt == "": continue else: actionRequest = request_action(user_prompt) print(f"Action Request: {actionRequest}") command = actionRequest["action"] if command == "sit": sit(spot) continue elif command == "stand-up": stand_up(spot) continue elif command == "listen-to-brain": speak("Listening to your brain...") print("logged in") info = neurosity.get_info() def callback(data): global last_time global is_forward global is_sitting if (time.time() - last_time) > 3: print(data) if is_sitting: speak("Grr!") spot.move_by_velocity_control( v_x=-0.3, v_y=0, v_rot=0, cmd_duration=2 ) is_forward = False is_sitting = False elif is_forward: speak("Woof!") spot.bow(20) spot.move_head_in_points( yaws=[1], pitches=[-20], rolls=[0] ) spot.move_head_in_points( yaws=[0], pitches=[-20], rolls=[0] ) is_sitting = True else: speak("Bark!") spot.move_to_goal(goal_x=0.5, goal_y=0) is_forward = True last_time = time.time() try: unsubscribe = neurosity.kinesis("tongue", callback) time.sleep(30) finally: unsubscribe() print("done with neurosity") speak("I'm done listening to your brain!") elif command == "bow": bow(spot) continue elif command == "stop": speak("Goodbye!") print("Stopping") listening = False break else: speak("I don't know that command! Try again.") print("dont recognize this command") continue if __name__ == "__main__": main() ``` ### apps/web/app/layout.tsx ```typescript import type { Metadata } from "next"; import { Inter } from "next/font/google"; import "./globals.css"; import { ThemeProvider } from "@/components/ThemeProvider"; import { Toaster } from "@/components/ui/sonner"; import QueryClientWrapper from "@/components/QueryClientWrapper"; import Head from "next/head"; import { CSPostHogProvider } from "@/components/PosthogWrapper"; const inter = Inter({ subsets: ["latin"] }); export const metadata: Metadata = { title: "Neurodog", description: "Emotional robotic service dogs assisting those in need", metadataBase: new URL("https://neurodog.vercel.app/"), openGraph: { images: "/opengraph-image.png" }, }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### apps/web/app/page.tsx ```typescript /** * v0 by Vercel. * @see https://v0.dev/t/PmwTvNfrVgf * Documentation: https://v0.dev/docs#integrating-generated-code-into-your-nextjs-app */ import { Hero } from "@/components/Hero"; import ShapeBackground from "@/components/ShapeBackground"; import Logo from "@/components/assets/Logo"; import { Button } from "@/components/ui/button"; import { Separator } from "@/components/ui/separator"; import Link from "next/link"; import Image from "next/image"; import Footer from "@/components/Footer"; export default function LandingPage() { return (
Neurodog
Key Features

Deeper connections. More innovation.

This platform aims to provide an automated experience towards caretaking and compassion for the elderly. Developed for an accessible and seamless user experiencie.

Robot Dog looking at guy

  • Voice Control

    Our platforms and features all come with voice capabilities enabling a more accessible experience.

  • Automation

    Our service dogs understand your emotions and will come to your side when you need it the most.

  • Power by AI

    Our platforms allow us a more personalized experience through natural language comprehension.

); } ``` ### apps/web/lib/db/index.ts ```typescript import { drizzle } from "drizzle-orm/planetscale-serverless"; import { connect } from "@planetscale/database"; import { env } from "@/lib/env"; export const connection = connect({ url: env.DATABASE_URL }); export const db = drizzle(connection); ``` ### apps/web/app/(auth)/layout.tsx ```typescript import ShapeBackground from "@/components/ShapeBackground"; import { validateRequest } from "@/lib/lucia"; import { redirect } from "next/navigation"; interface AuthLayoutProps { children: React.ReactNode; } export default async function AuthLayout({ children }: AuthLayoutProps) { const { session } = await validateRequest(); if (session) return redirect("/dashboard"); return (
{children}
); } ``` ### apps/web/app/(app)/layout.tsx ```typescript import Footer from "@/components/Footer"; import Header from "@/components/Header"; import { validateRequest } from "@/lib/lucia"; import { redirect } from "next/navigation"; interface AppLayoutProps { children: React.ReactNode; } export default async function AppLayout({ children }: AppLayoutProps) { const { session, user } = await validateRequest(); if (!session) return redirect("/login"); return (
{children}
); } ``` [73 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]