# Project export: Insight OS This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: We envisioned to build a pipeline for EEG-controlled Spot movement. We would be able to control Spot to perform different dynamic movements based on visual entrainment captured in real-time EEG data. - Devpost: https://devpost.com/software/insight-os - GitHub: https://github.com/LukeKulm/BrainRobotConnect - Video: https://www.youtube.com/embed/94X22F_zibE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Laurence Liang (16 commits), Luke Kulm (3 commits), jheyu0309 (3 commits), isita (2 commits) ## Devpost submission (written by the team) ### Inspiration We noticed that some people are either unable to navigate or move around their spaces easily. We wanted to empower these people by giving them the tools to embody a robot using brain control. ### What it does It allows users to control a robot like spot using their brain signals. ### How we built it eeg part wrote a visual entrainment script on psychopy (using python) wrote a real-time eeg processing script that takes in continuous voltage data from 8 channels, cleans/filters it with butterworth and independent component analysis, complete fast fourier transform, and wrote an algorithm to classify which frequency had the largest increase in power within the past five seconds. ### Challenges we ran into visual entrainment script is supposed to have four images flashing at different frequencies but both psychopy and react cannot achieve this. should've tried matlab. real-time eeg processing is difficult to debug as it requires constant streaming of data and using multiple functions at the same time. ### Accomplishments we're proud of debugging nonstop for several hours + finally understanding most of lsl ### What we learned need to conduct a full literature review for eeg projects in a short amount of time + understood more in depth about real-time processing ## README (from the GitHub repository) # BrainRobotConnect Robot control using a brain computer interface ### Folder Directory * `EEG`: contains files for the brain-computer interface * `client-controller`: contains sample code to control Spot using a BCI * `hackathon-spot-control-fork`: contains the forked repository (with our modifications) that allow us to control Spot using HTTP * `raspberry-pi`: some code that we wrote for a Raspberry Pi while evaluating different options * `server`: NextJS server used to connect the client controller to the Spot receiver code ## Detected evidence (automated analysis) Indexed codebase: 15 recognized source files, 23 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Supabase (technology) — detected in the code - Tailwind CSS (technology) — detected in the code ## Codebase structure (from repository index) ### Files (28 of 28) ``` .gitignore client-controller/example.py EEG/live_proc.ipynb EEG/main.ipynb EEG/sample.csv EEG/save_sample.csv hackathon-spot-control-fork/.github/workflows/build.yml hackathon-spot-control-fork/.gitignore hackathon-spot-control-fork/Dockerfile hackathon-spot-control-fork/main.py hackathon-spot-control-fork/README.md hackathon-spot-control-fork/requirements.txt hackathon-spot-control-fork/spot_controller.py LICENSE raspberry-pi/robot_control.py README.md server/.gitignore server/jsconfig.json server/next.config.mjs server/package.json server/pages/api/hello.js server/pages/api/spot.js server/postcss.config.js server/README.md server/src/app/globals.css server/src/app/layout.js server/src/app/page.js server/tailwind.config.js ``` ### Dependencies - hackathon-spot-control-fork/requirements.txt: supabase - server/package.json: @supabase/supabase-js@^2.39.6, autoprefixer@^10.0.1, next@14.1.0, postcss@^8, react@^18, react-dom@^18, tailwindcss@^3.3.0 ### Recent commits (newest first) - Add files via upload - Add files via upload - Delete server/pages/api/forward.js - Delete server/pages/api/stop.js - Delete server/pages/api/right.js - Delete server/pages/api/left.js - Delete server/pages/api/kick.js - Update README.md - Merge pull request #2 from LukeKulm/dev/main - Update README.md - Add code for forked repo - Delete moved files - Add NextJS server that connects the client with Spot - Add forked repo for Spot code - Add WIP code for Raspberry Pi - Add client controller for Spot using HTTP - Move files to EEG folder - Merge pull request #1 from LukeKulm/isita/liveproc - updated main - robot control addition ## Key source files (fetched from GitHub, selected and truncated for size) ### hackathon-spot-control-fork/requirements.txt ``` supabase ``` ### hackathon-spot-control-fork/Dockerfile ``` FROM ghcr.io/merklebot/hackathon-arm-image:master as build ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1 ARG TARGETPLATFORM ARG BUILDPLATFORM ARG TARGETOS ARG TARGETARCH ARG Version ARG GitCommit RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM" COPY requirements.txt requirements.txt RUN python3.8 -m pip install -r requirements.txt COPY . . CMD ["python3.8", "main.py"] ``` ### server/package.json ``` { "name": "reflector-server-spot", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@supabase/supabase-js": "^2.39.6", "next": "14.1.0", "react": "^18", "react-dom": "^18" }, "devDependencies": { "autoprefixer": "^10.0.1", "postcss": "^8", "tailwindcss": "^3.3.0" } } ``` ### hackathon-spot-control-fork/main.py ```python import os import time from spot_controller import SpotController import requests ROBOT_IP = "10.0.0.3"#os.environ['ROBOT_IP'] SPOT_USERNAME = "admin"#os.environ['SPOT_USERNAME'] SPOT_PASSWORD = "2zqa8dgw7lor"#os.environ['SPOT_PASSWORD'] # from supabase import create_client, Client # url: str = os.environ.get("SUPABASE_URL") # key: str = os.environ.get("SUPABASE_KEY") # supabase: Client = create_client(url, key) def main(): #example of using micro and speakers print("Start recording audio") sample_name = "aaaa.wav" cmd = f'arecord -vv --format=cd --device={os.environ["AUDIO_INPUT_DEVICE"]} -r 48000 --duration=10 -c 1 {sample_name}' print(cmd) os.system(cmd) print("Playing sound") os.system(f"ffplay -nodisp -autoexit -loglevel quiet {sample_name}") # Capture image import cv2 camera_capture = cv2.VideoCapture(0) rv, image = camera_capture.read() print(f"Image Dimensions: {image.shape}") cv2.imwrite("img.png", image) camera_capture.release() # with open("img.png", 'rb') as f: # supabase.storage.from_("testbucket").upload(file=f,path=path_on_supastorage) # Use wrapper in context manager to lease control, turn on E-Stop, power on the robot and stand up at start # and to return lease + sit down at the end with SpotController(username=SPOT_USERNAME, password=SPOT_PASSWORD, robot_ip=ROBOT_IP) as spot: for i in range(200): resp = requests.get("https://spot-rest-api.vercel.app/api/spot") command = resp.json()["message"] if command == "rest": time.sleep(3) if command == "forward": spot.move_to_goal(goal_x=0.1, goal_y=0) if command == "left": spot.move_by_velocity_control(v_x=0,v_y=0,v_rot=0.3,cmd_duration=5) if command == "right": spot.move_by_velocity_control(v_x=0,v_y=0,v_rot=-0.3,cmd_duration=5) time.sleep(1) # time.sleep(2) # spot.move_to_goal(goal_x=0.5, goal_y=0) # time.sleep(1) # spot.move_to_goal(goal_x=-0.5,goal_y=0) # time.sleep(1) # spot.move_by_velocity_control(v_x=0,v_y=0,v_rot=0.3,cmd_duration=5) # time.sleep(1) # spot.move_by_velocity_control(v_x=0,v_y=0,v_rot=-0.3,cmd_duration=5) # time.sleep(1) # # Move head to specified positions with intermediate time.sleep # spot.move_head_in_points(yaws=[0.2, 0], # pitches=[0.3, 0], # rolls=[0.4, 0], # sleep_after_point_reached=1) # time.sleep(3) # # Make Spot to move by goal_x meters forward and goal_y meters left # spot.move_to_goal(goal_x=0.5, goal_y=0) # time.sleep(3) # # Control Spot by velocity in m/s (or in rad/s for rotation) # spot.move_by_velocity_control(v_x=-0.3, v_y=0, v_rot=0, cmd_duration=2) # time.sleep(3) if __name__ == '__main__': main() ``` ### server/src/app/layout.js ```javascript import { Inter } from "next/font/google"; import "./globals.css"; const inter = Inter({ subsets: ["latin"] }); export const metadata = { title: "Create Next App", description: "Generated by create next app", }; export default function RootLayout({ children }) { return ( {children} ); } ``` ### server/src/app/page.js ```javascript import Image from "next/image"; export default function Home() { return (

Get started by editing  src/app/page.js

By{" "} Vercel Logo
Next.js Logo

Docs{" "} ->

Find in-depth information about Next.js features and API.

Learn{" "} ->

Learn about Next.js in an interactive course with quizzes!

Templates{" "} ->

Explore starter templates for Next.js.

Deploy{" "} ->

Instantly deploy your Next.js site to a shareable URL with Vercel.

); } ``` ### server/postcss.config.js ```javascript module.exports = { plugins: { tailwindcss: {}, autoprefixer: {}, }, }; ``` ### client-controller/example.py ```python import requests import time URL = "https://spot-rest-api.vercel.app/api/spot" r = requests.post(URL, json={"data":"rest"}) time.sleep(3) r = requests.post(URL, json={"data":"left"}) time.sleep(3) r = requests.post(URL, json={"data":"right"}) time.sleep(3) r = requests.post(URL, json={"data":"forward"}) time.sleep(3) r = requests.post(URL, json={"data":"rest"}) time.sleep(3) ``` ### server/tailwind.config.js ```javascript /** @type {import('tailwindcss').Config} */ module.exports = { content: [ "./src/pages/**/*.{js,ts,jsx,tsx,mdx}", "./src/components/**/*.{js,ts,jsx,tsx,mdx}", "./src/app/**/*.{js,ts,jsx,tsx,mdx}", ], theme: { extend: { backgroundImage: { "gradient-radial": "radial-gradient(var(--tw-gradient-stops))", "gradient-conic": "conic-gradient(from 180deg at 50% 50%, var(--tw-gradient-stops))", }, }, }, plugins: [], }; ``` ### raspberry-pi/robot_control.py ```python from gpiozero import Robot import time # Define the robot with motor pins robot = Robot(left=(17, 18), right=(22, 23)) # Function to drive the robot forward def drive_forward(): robot.forward() # Function to drive the robot backward def drive_backward(): robot.backward() # Function to turn the robot left def turn_left(): robot.left() # Function to turn the robot right def turn_right(): robot.right() # Function to stop the robot def stop_robot(): robot.stop() # Example usage try: drive_forward() time.sleep(4) turn_left() time.sleep(2) stop_robot() time.sleep(2) turn_right() time.sleep(2) stop_robot() time.sleep(2) drive_backward() time.sleep(4) stop_robot() except KeyboardInterrupt: stop_robot() ``` [5 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]