# Project export: JournEyes This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: AI-infused VR travel assistant, to explore language, culture, and life. - Devpost: https://devpost.com/software/journeyes - GitHub: https://github.com/spikecodes/JournEyes - Video: https://www.youtube.com/embed/QdcCzyXl3Cg?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Bill Zhang (11 commits), Spike O'Carroll (7 commits), Simon Quach (5 commits) ## Devpost submission (written by the team) ### Inspiration Have you ever found yourself wandering in a foreign land, eyes wide with wonder, yet feeling that pang of curiosity about the stories behind the unfamiliar sights and sounds? That's exactly where we found ourselves. All four of us share a deep love for travel and an insatiable curiosity about the diverse cultures, breathtaking scenery, and intriguing items we encounter abroad. It sparked an idea: why not create a travel companion that not only shares our journey but enhances it? Enter our brainchild, a fusion of VR and AI designed to be your personal travel buddy. Imagine having a friend who can instantly transcribe signs in foreign languages, identify any object from monuments to local flora, and guide you through the most bewildering of environments. That's what we set out to build—a gateway to a richer, more informed travel experience. ### What it does Picture this: you're standing before a captivating monument, curiosity bubbling up. With our VR travel assistant, simply speak your question, and it springs into action. This clever buddy captures your voice, processes your command, and zooms in on the object of your interest in the video feed. Using cutting-edge image search, it fetches information about just what you're gazing at. Wondering about that unusual plant or historic site? Ask away, and you'll have your answer. It's like having a local guide, historian, and botanist all rolled into one, accessible with just a glance and a word. ### How we built it We initiated our project by integrating Unity with the Meta XR SDK to bring our VR concept to life. The core of our system, a server engineered with Python and FastAPI, was designed to perform the critical tasks, enhanced by AI capabilities for efficient processing. We leveraged Google Lens via the SERP API for superior image recognition and OpenAI's Whisper for precise voice transcription. Our approach was refined by adopting techniques from a Meta research paper, enabling us to accurately crop images to highlight specific objects. This method ensured that queries were efficiently directed to the appropriate AI model for quick and reliable answers. To ensure a smooth operation, we encapsulated our system within Docker and established connectivity to our VR app through ngrok, facilitating instantaneous communication via websockets and the SocketIO library. ### Challenges we ran into None of us had much or any experience with both Unity3D and developing VR applications so there were many challenges in learning how to use the Meta XR SDK and how to build a VR app in general. Additionally, Meta imposed a major restriction that added to the complexity of the application: we could not capture the passthrough video feed through any third-party screen recording software. This meant we had to, in the last few hours of the hackathon, create a new server in our network that would capture the casted video feed from the headset (which had no API) and then send it to the backend. This was a major challenge and we are proud to have overcome it. ### Accomplishments we're proud of From web developers to VR innovators, we've journeyed into uncharted territories, crafting a VR application that's not just functional but truly enriching for the travel-hungry soul. Our creation stands as a beacon of what's possible, painting a future where smart glasses serve as your personal AI-powered travel guides, making every journey an enlightening exploration. ### What we learned The journey was as rewarding as the destination. We mastered the integration of Meta Quest 2s and 3s with Unity, weaving through the intricacies of Meta XR SDKs. Our adventure taught us to make HTTP calls within Unity, transform screenshots into Base64 strings, and leverage Google Cloud for image hosting, culminating in real-time object identification through Google Lens. Every challenge was a lesson, turning us from novices into seasoned navigators of VR development and AI integration. ## README (from the GitHub repository) ![Thumbnail (1)](https://github.com/spikecodes/JournEyes/assets/19519553/e1c308ae-4495-4e9b-9b33-09f2d4705afb) # JournEyes AI-infused VR travel assistant, to explore language, culture, and life. > Built for [TreeHacks 2024](https://treehacks-2024.devpost.com/). ### Inspiration Have you ever found yourself wandering in a foreign land, eyes wide with wonder, yet feeling that pang of curiosity about the stories behind the unfamiliar sights and sounds? That's exactly where we found ourselves. All four of us share a deep love for travel and an insatiable curiosity about the diverse cultures, breathtaking scenery, and intriguing items we encounter abroad. It sparked an idea: why not create a travel companion that not only shares our journey but enhances it? Enter our brainchild, a fusion of VR and AI designed to be your personal travel buddy. Imagine having a friend who can instantly transcribe signs in foreign languages, identify any object from monuments to local flora, and guide you through the most bewildering of environments. That's what we set out to build—a gateway to a richer, more informed travel experience. ### What it does Picture this: you're standing before a captivating monument, curiosity bubbling up. With our VR travel assistant, simply speak your question, and it springs into action. This clever buddy captures your voice, processes your command, and zooms in on the object of your interest in the video feed. Using cutting-edge image search, it fetches information about just what you're gazing at. Wondering about that unusual plant or historic site? Ask away, and you'll have your answer. It's like having a local guide, historian, and botanist all rolled into one, accessible with just a glance and a word. ### How we built it Diving into the unknown, we embarked on our development journey with Unity and the Meta XR SDK for the VR magic, while the brains of the operation—a server built with Python, FastAPI, and a sprinkle of AI wizardry—handled the heavy lifting. We tapped into Google Lens through the SERP API for unmatched image recognition and employed OpenAI's Whisper for flawless voice transcription. A breakthrough research paper from Meta allowed us to smartly crop images to focus solely on the mentioned object, directing the inquiry through our powerful agent to the relevant AI model for a swift, accurate response. This entire setup was neatly packaged with Docker and linked via ngrok, creating a seamless bridge between the VR app and our server, all while maintaining real-time interaction through websockets and the SocketIO library. ![architecture](https://i.ibb.co/cLqW8k3/architecture.png) ### Challenges we ran into None of us had much or any experience with both Unity3D and developing VR applications so there were many challenges in learning how to use the Meta XR SDK and how to build a VR app in general. Additionally, Meta imposed a major restriction that added to the complexity of the application: we could not capture the passthrough video feed through any third-party screen recording software. This meant we had to, in the last few hours of the hackathon, create a new server in our network that would capture the casted video feed from the headset (which had no API) and then send it to the backend. This was a major challenge and we are proud to have overcome it. ### Accomplishments that we're proud of From web developers to VR innovators, we've journeyed into uncharted territories, crafting a VR application that's not just functional but truly enriching for the travel-hungry soul. Our creation stands as a beacon of what's possible, painting a future where smart glasses serve as your personal AI-powered travel guides, making every journey an enlightening exploration. ### What we learned The journey was as rewarding as the destination. We mastered the integration of Meta Quest 2s and 3s with Unity, weaving through the intricacies of Meta XR SDKs. Our adventure taught us to make HTTP calls within Unity, transform screenshots into Base64 strings, and leverage Google Cloud for image hosting, culminating in real-time object identification through Google Lens. Every challenge was a lesson, turning us from novices into seasoned navigators of VR development and AI integration. ## Detected evidence (automated analysis) Indexed codebase: 52 recognized source files, 269 KB. - C# (language) — detected in the code - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - JavaScript (language) — detected in the code - LangChain (technology) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code ## Codebase structure (from repository index) ### Files (66 of 66) ``` .dockerignore .gitignore agent.py app.py client/.eslintrc.json client/.gitignore client/app/globals.css client/app/layout.js client/app/page.js client/jsconfig.json client/next.config.mjs client/package.json client/postcss.config.js client/README.md client/tailwind.config.js Dockerfile google_lens.py lang_sam/__init__.py lang_sam/lang_sam.py lang_sam/utils.py README.md requirements.txt segment_anything/__init__.py segment_anything/automatic_mask_generator.py segment_anything/build_sam.py segment_anything/modeling/__init__.py segment_anything/modeling/common.py segment_anything/modeling/image_encoder.py segment_anything/modeling/mask_decoder.py segment_anything/modeling/prompt_encoder.py segment_anything/modeling/sam.py segment_anything/modeling/transformer.py segment_anything/predictor.py segment_anything/utils/__init__.py segment_anything/utils/amg.py segment_anything/utils/onnx.py segment_anything/utils/transforms.py segment.py segmentation/lang_sam/__init__.py segmentation/lang_sam/lang_sam.py segmentation/lang_sam/utils.py segmentation/requirements.txt segmentation/segment_anything/__init__.py segmentation/segment_anything/automatic_mask_generator.py segmentation/segment_anything/build_sam.py segmentation/segment_anything/modeling/__init__.py segmentation/segment_anything/modeling/common.py segmentation/segment_anything/modeling/image_encoder.py segmentation/segment_anything/modeling/mask_decoder.py segmentation/segment_anything/modeling/prompt_encoder.py segmentation/segment_anything/modeling/sam.py segmentation/segment_anything/modeling/transformer.py segmentation/segment_anything/predictor.py segmentation/segment_anything/utils/__init__.py segmentation/segment_anything/utils/amg.py segmentation/segment_anything/utils/onnx.py segmentation/segment_anything/utils/transforms.py segmentation/segment_point.ipynb segmentation/segment_text.ipynb server.py socket_wrapper.py temp_text_file.txt treehacks-service-key.json Unity Scripts/AudioRecordBtn.cs Unity Scripts/MicAudio.cs Unity Scripts/WavUtility.cs ``` ### Dependencies - client/package.json: autoprefixer@^10.0.1, eslint@^8, eslint-config-next@14.1.0, next@14.1.0, postcss@^8, react@^18, react-dom@^18, recordrtc@^5.6.2, socket.io-client@^4.7.4, tailwindcss@^3.3.0 - requirements.txt: beautifulsoup4@==4.12.3, email-validator@==2.1.0.post1, fastapi@==0.109.2, google-cloud-storage@==2.14.0, groundingdino-gml@==0.1.2, httptools@==0.6.1, ipykernel@==6.22.0, itsdangerous@==2.1.2, langchain@==0.1.7, langchain-openai@==0.0.6, langchainhub@==0.1.14, orjson@==3.9.14, pip-chill@==1.0.3, pydantic-extra-types@==2.5.0, pydantic-settings@==2.2.0, python-multipart@==0.0.9, python-socketio@==5.11.1, serpapi@==0.1.5, ujson@==5.9.0, uuid@==1.30, uvicorn@==0.27.1, watchfiles@==0.21.0, websockets@==12.0 - segmentation/requirements.txt: groundingdino-gml@==0.1.2, ipykernel@==6.29.2, pickleshare@==0.7.5, pip-chill@==1.0.3 ### Recent commits (newest first) - Merge pull request #1 from spikecodes/fin - Update MicAudio.cs - Update AudioRecordBtn.cs - final - Websockets work - Update README.md - Create README.md - Progress update - Create MicAudio.cs - Create WavUtility.cs - Create AudioRecordBtn.cs - Try to fix docker - Docker - Update page.js - Reverting to wav - Gitignore - Audio strema might work - frontend tester - Transcription - Add files via upload ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` beautifulsoup4==4.12.3 email-validator==2.1.0.post1 fastapi==0.109.2 google-cloud-storage==2.14.0 groundingdino-gml==0.1.2 httptools==0.6.1 ipykernel==6.22.0 itsdangerous==2.1.2 langchain==0.1.7 langchain-openai==0.0.6 langchainhub==0.1.14 orjson==3.9.14 pip-chill==1.0.3 pydantic-extra-types==2.5.0 pydantic-settings==2.2.0 python-multipart==0.0.9 python-socketio==5.11.1 serpapi==0.1.5 ujson==5.9.0 uuid==1.30 uvicorn==0.27.1 watchfiles==0.21.0 websockets==12.0 ``` ### Dockerfile ``` # Use an official Python runtime as a base image FROM python:3.9-slim # Set environment variables ENV PYTHONDONTWRITEBYTECODE 1 ENV PYTHONUNBUFFERED 1 # Set the working directory in the container WORKDIR /app # Copy the current directory contents into the container at /app COPY requirements.txt . # Install any needed packages specified in requirements.txt RUN pip install --upgrade pip && \ pip install -r requirements.txt COPY . . # Make port 4008 available to the world outside this container EXPOSE 4008 # Define environment variable for Uvicorn ENV UVICORN_HOST=0.0.0.0 ENV UVICORN_PORT=4008 # Run the FastAPI app when the container launches using Uvicorn CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "4008"] ``` ### segmentation/requirements.txt ``` groundingdino-gml==0.1.2 ipykernel==6.29.2 pickleshare==0.7.5 pip-chill==1.0.3 ``` ### client/package.json ``` { "name": "client", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "next": "14.1.0", "react": "^18", "react-dom": "^18", "recordrtc": "^5.6.2", "socket.io-client": "^4.7.4" }, "devDependencies": { "autoprefixer": "^10.0.1", "eslint": "^8", "eslint-config-next": "14.1.0", "postcss": "^8", "tailwindcss": "^3.3.0" } } ``` ### server.py ```python from flask import Flask, send_file import pyautogui import tempfile import os app = Flask(__name__) @app.route('/screenshot') def takescreenshot(): screenshot = pyautogui.screenshot() screenshot_file = tempfile.NamedTemporaryFile(suffix='.png', delete=False) screenshot.save(screenshot_file.name) response = send_file(screenshot_file.name, mimetype='image/png') return response if __name__ == '__main__': app.run(debug=True, port=8000) ``` ### app.py ```python from fastapi import FastAPI from fastapi.responses import ( HTMLResponse, JSONResponse, StreamingResponse, PlainTextResponse, ) from PIL import Image from socket_wrapper import socketio_mount import tempfile import os from agent import Agent import pyautogui from openai import AsyncOpenAI from langchain.tools import tool import websockets from websockets.sync.client import connect import dotenv dotenv.load_dotenv() from segment import process_image_and_text from google_lens import google_lens_search import io import base64 import json from pydantic import BaseModel client = AsyncOpenAI() # Initialize FastAPI and SocketIO app = FastAPI() from fastapi.middleware.cors import CORSMiddleware sio = socketio_mount(app) elevenlabs_api_key = os.getenv("ELEVENLABS_API_KEY") voice = { "voice_id": "EXAVITQu4vr4xnSDxMaL", "name": "Bella", "settings": { "stability": 0.72, "similarity_boost": 0.2, "style": 0.0, "use_speaker_boost": False, "speaking_rate": 2, }, } model = { "model_id": "eleven_multilingual_v2", } current_image = None @tool def google_lens_wrapper(text: str) -> str: """ Performs a google lens search on the current image for object detection. Input: - text: a short description of the object to search for. Returns: The result of the google lens search. """ try: print("Cropping image based on text:", text) global current_image # Remove header if present if ";base64," in current_image: current_image = current_image.split(";base64,")[1] # Decode the base64 image image_bytes = base64.b64decode(current_image) image = Image.open(io.BytesIO(image_bytes)).convert("RGB") print("Image opened") processed_image = process_image_and_text(text, image) print("Image processed") # Convert the processed image back to base64 to send back as response buffered = io.BytesIO() processed_image.save(buffered, format="JPEG") current_image = base64.b64encode(buffered.getvalue()).decode() return google_lens_search(current_image) except Exception as e: return f"Error: {e}" @tool async def vision_llm(text: str) -> str: """ Uses the OPENAI Vision API to answer a question based on the input image. Input: - text: a question about the image. Returns: The answer to the question based on the image. """ global current_image current_image = current_image.strip('"') # Add png header if not current_image.startswith("data:image/png;base64,"): current_image = "data:image/png;base64," + current_image response = await client.chat.completions.create( model="gpt-4-vision-preview", temperature=0, max_tokens=500, messages=[ { "role": "user", "content": [ {"type": "text", "text": "What’s in this image?"}, { "type": "image_url", "image_url": { "url": current_image, }, }, ], } ], ) return response.choices[0].message.content from langchain.agents import Tool from langchain.tools import tool from langchain_community.utilities import GoogleSerperAPIWrapper search = GoogleSerperAPIWrapper() google_search = Tool( name="google_search", func=search.run, description="Searches Google for the input query", ) @app.get("/") def index(): return "Audio Transcription Service" # Define SocketIO events @sio.on("connect") async def connect(sid, environ): print("Client connected", sid) @sio.on("disconnect") async def disconnect(sid): print("Client disconnected", sid) tools = [google_lens_wrapper, google_search, vision_llm] agent = Agent(tools) @sio.on("event") async def segment_image(sid, data): """ Socket.IO event to handle image segmentation requests. Expects data to contain 'text' and 'image' (base64 encoded) keys. """ print("Received data:", data) audio = data["audio"] image_data = data["image"] global current_image current_image = image_data text = await transcribe_base64_audio(audio) print("Transcribed text:", text) async def stream_response(): async for content in agent.invoke(text): await sio.emit("text", content) await stream_response() class ImageAudioRequest(BaseModel): audio: str # Base64 encoded audio image: str # Base64 encoded image @app.post("/segment-image") async def segment_image(request: ImageAudioRequest): """ Endpoint to handle image segmentation requests. Accepts a JSON payload containing 'audio' and 'image' (both base64 encoded). """ audio = request.audio image_data = request.image global current_image current_image = image_data text = await transcribe_base64_audio(audio) print("Transcribed text:", text) full_text = "" async for content in agent.invoke(text): full_text += content # Accumulate the content print("Full text:", full_text) return PlainTextResponse(full_text) @sio.on("event_audio") async def segment_image(sid, data): """ Socket.IO event to handle image segmentation requests. Expects data to contain 'text' and 'image' (base64 encoded) keys. """ print("Received data:", data) audio = data["audio"] image_data = data["image"] global current_image current_image = image_data text = await transcribe_base64_audio(audio) print("Transcribed text:", text) async def generate_strea [truncated — 4893 more characters] ``` ### client/app/layout.js ```javascript import { Inter } from "next/font/google"; import "./globals.css"; const inter = Inter({ subsets: ["latin"] }); export const metadata = { title: "Create Next App", description: "Generated by create next app", }; export default function RootLayout({ children }) { return ( {children} ); } ``` ### client/app/page.js ```javascript 'use client' import {useState, useRef, useEffect} from 'react' import {useRouter} from 'next/navigation' import io from 'socket.io-client' // import Audio from "@/assets/audio.svg"; let RecordRTC const socket = io('https://8d74cc6176ee.ngrok.app/') export default function RecordForm() { const router = useRouter() const [recording, setRecording] = useState(false) const [audio, setAudio] = useState(null) const [image, setImage] = useState(null) useEffect(() => { if (socket) { // Listen to an event, for example, 'message' socket.on('text', (data) => { console.log(data) }) // Clean up event listener on component unmount return () => { socket.off('text') } } }, [socket]) // Recording states const recorder = useRef(null) const microphone = useRef(null) useEffect(() => { import('recordrtc').then((r) => { RecordRTC = r.default }) window.addEventListener('keydown', handleKeyDown) return () => window.removeEventListener('keydown', handleKeyDown) }, [recording]) const handleKeyDown = (event) => { if (event.code === 'Space') { if (recording) { stopRecording() } else { startRecording() } } } const handleImageUpload = (event) => { const file = event.target.files[0] if (file) { const reader = new FileReader() reader.onloadend = () => { // This will log the base64 string of the image setImage(reader.result) } reader.readAsDataURL(file) } } const captureMicrophone = async (callback) => { try { const stream = await navigator.mediaDevices.getUserMedia({audio: true}) callback(stream) } catch (error) { alert('Unable to access your microphone.') console.error(error) } } const startRecording = async () => { if (!recording) { await captureMicrophone((stream) => { microphone.current = stream const options = { // Required format for speech to text, 16k mono channel type: 'audio', recorderType: RecordRTC.StereoAudioRecorder, desiredSampRate: 16000, numberOfAudioChannels: 1, } recorder.current = RecordRTC(stream, options) console.log(recorder.current) recorder.current.startRecording() recorder.current.microphone = microphone.current setRecording(true) }) } } const stopRecordingCallback = () => { console.log(recorder.current) const audioBlob = recorder.current.getBlob() setRecording(false) const reader = new FileReader() reader.readAsDataURL(audioBlob) reader.onloadend = () => { let base64Audio = reader.result setAudio(base64Audio) } } const sendAudioToServer = async () => { socket.emit('event', {audio: audio, image: image}) } const stopRecording = () => { if (recorder.current) { recorder.current.stopRecording(stopRecordingCallback) } } return (
{recording ? ( ) : ( )} {recording &&
Recording...
}
) } ``` ### socket_wrapper.py ```python # socket.py from fastapi import FastAPI from socketio import ASGIApp, AsyncServer def socketio_mount( app: FastAPI, async_mode: str = "asgi", mount_path: str = "/socket.io/", socketio_path: str = "socket.io", logger: bool = False, engineio_logger: bool = False, cors_allowed_origins="*", **kwargs ) -> AsyncServer: """Mounts an async SocketIO app over an FastAPI app.""" sio = AsyncServer(async_mode=async_mode, cors_allowed_origins=cors_allowed_origins, logger=logger, engineio_logger=engineio_logger, **kwargs) sio_app = ASGIApp(sio, socketio_path=socketio_path) # mount app.add_route(mount_path, route=sio_app, methods=["GET", "POST"]) app.add_websocket_route(mount_path, sio_app) return sio ``` ### agent.py ```python from langchain import hub from langchain.agents import AgentExecutor, create_openai_tools_agent from langchain_openai import ChatOpenAI from langchain.agents import Tool from langchain.tools import tool from langchain_community.utilities import GoogleSerperAPIWrapper new_prompt = """You are a helpful assistant. Respond in the language of the user. Your output will be transcribed to speech and played to the user. So, when responding: 1. Use plain, conversational language. 2. Avoid markdown, special characters, or symbols. 3. Expand abbreviations and acronyms into their full spoken form. For example, use 'miles per hour' instead of 'mph'. 4. If technical terms or jargon are unavoidable, provide a brief spoken explanation. 5. Articulate numbers as they would be spoken. For example, use 'two point two' instead of '2.2'. 6. Avoid complex punctuation. Use simple sentence structures conducive to spoken language. 7. Response in a concise manner. Remember, your goal is to provide responses that are clear, concise, and easily understood when spoken aloud. """ class Agent: def __init__(self, tools): """tools should be list of functions""" prompt = hub.pull("hwchase17/openai-tools-agent") prompt.messages[0].prompt.template = new_prompt model = ChatOpenAI(model="gpt-4-turbo-preview", temperature=0, streaming=True) print("All tools", tools) agent = create_openai_tools_agent( model.with_config({"tags": ["agent_llm"]}), tools, prompt ) self.executor = AgentExecutor(agent=agent, tools=tools, verbose=True) async def invoke(self, message): async for event in self.executor.astream_events( {"input": message}, version="v1", ): kind = event["event"] if kind == "on_chat_model_stream": content = event["data"]["chunk"].content if content: yield content ``` [44 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]