# Project export: eHealth
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2025
- Tagline: Multimodal agentic robot which aims to be a companion and aid to everyone.
- Devpost: https://devpost.com/software/r2d2-ltwyxd
- GitHub: https://github.com/sohvasa/treehacks
- Team: 2 GitHub contributor(s) — Soham Vasanawala (15 commits), Jay Khemchandani (1 commits)
## Devpost submission (written by the team)
### Inspiration
Telehealth services are expected to grow at an impressive rate of 39% year over year until 2030, a growth, in part, driven by AI assistants, chatbots, and modern distribution technologies. At the same time, we're facing a growing elderly care crisis. The global population of people aged 65 and older is set to reach 1.5 billion by 2050 – double what it is today. This dramatic demographic shift coincides with a severe healthcare staffing shortage. In the United States alone, we're expecting a shortfall of 86,000 physicians by 2036. This shortage is especially concerning in elderly care facilities, which are already struggling with low nurse-to-patient ratios. However, we believe there's hope. By strategically implementing AI technology in areas and ways where it's needed most, we can create lasting positive change that touches lives across the world. Though we had only 36 hours, we wondered if we could build an autonomous robotic assistant which harnessed the accessibility of telehealth, the overwhelming need for elderly care, and the efficiency of AI-powered nursing support? eHealth was born, a robot which we hope can inspire the world to create tools which will stand for the benefit of everyone.
### What it does
eHealth is a multimodal agentic system which has the autonomy to move itself simply given a request to pick up and drop off anything in its environment. It is powered by two Visual Language Models, or VLMs, hosted locally on an Nvidia Jetson, and uses a Raspberry Pi to make API calls to both secondary devices and cloud computed LLMs. Beyond its tech, eHealth was made to be as helpful as possible to as many people as possible. In the context of disability support, it autonomously identifies and retrieves objects in the environment based on natural language requests or app-based instructions. The agent's ability to be controlled via a joystick provides an additional layer of accessibility for users with different abilities, or for remote home and family safety. In the field of healthcare, eHealth can help with basic patient care tasks, freeing up healthcare workers for more complex duties or more emotionally impactful encounters. The remote monitoring capability allows healthcare providers to oversee multiple rooms efficiently while maintaining patient privacy and safety. In mental health settings, the agent serves as both a physical assistant and a consistent presence, helping patients maintain routines and complete daily tasks while providing data about activity patterns through its monitoring system.
### How we built it
At the beginning of TreeHacks, our team scrounged up spare parts and within a few hours had built out a basic chassis to hold our robot. Having attended an Nvidia talk where they demoed their impressive Jetson Nano micro AI computers as well as NanoOwl, a live object detection VLM, we realized that this was a perfect opportunity to create what would become eHealth. We positioned webcams to act as the eyes of a dual VLM system that gave our robot a complete understanding of its environment so it could be as helpful as possible. We wanted eHealth to be a light in the dark, a helpful friend to anyone who wants to use it. By combining both VLMs to contextualize its environment through multi-layered natural language processing and calculating the position of objects in 3 dimensional space utilizing hitbox depth software, we apply multiple layers of LLM inference to allow eHealth the ability to parse user inputs as a task or query, going through different decision trees for each of the two inputs. The task utilizes image recognition software string parsing from Gemini, feeds the data from this parsing into the VLM (NanoOwl) to do hitbox detection on “important” objects, wherein the 3d displacement process occurs with both movement and claw tracking.
### Challenges we ran into
The first challenge that we ran into was optimizing the latency between the web interface and the physical robot actions as the web interface ran gen-ai image and video inference based off of robotic actions. From more of a UI/UX design perspective, creating an intuitive user interface for different user groups so our product could be accessible in the healthtech sector was of utmost importance. A smaller but pivotal challenge was making sense of documentation that consumed tens of hours from our team, as we had to implement robust security for remote control, ensuring reliable object recognition across various lighting conditions, and balancing autonomous operation with human oversight added to the complexity. Finally, the cost of development with such a hardware and software intensive project was exorbitant as we had to navigate the challenge of integrating siloed AI backend systems with the frontend project while effectively utilizing Jetson Nano and Raspberry Pi as both computational and physical detection and action objects.
### Accomplishments we're proud of
After countless hours of troubleshooting, debugging, and refining, our team finally built the ecosystem that is eHealth successfully, developing a multi-modal control system that adapted to different user needs, which ensured that the robot had a variety of use cases that would help assist the healthcare sector along with elderly care for years to come. A team favorite integration was a secure and responsive web interface allowing for smooth remote monitoring, while real-time video streaming minimized latency, making interactions feel natural. We were incredibly excited with such a system as it took over 30 hours of straight coding and hardware development for everyone on the team. The cherry on top was this level of inherent reliable communication between the web application and the physical robot tied everything together. At 5 AM, exhausted but exhilarated, we celebrated our success in the only way that made sense—a victorious 10-minute nap before diving back into the next challenge.
### What we learned
This project was more than just a technical challenge: it was a deep dive into the complexities of designing assistive technology that truly works for people. We learned firsthand how user-centered design shapes interactions, with UI/UX playing a crucial role in making systems intuitive and accessible. One of our team's biggest learnings stemmed from optimizing real-time communication in robotics, which pushed us to explore AI backend integrations. As a whole, eHealth really allowed us to understand the power of large language models and image detection systems. Balancing automation with human control also came with ethical considerations, particularly in ensuring AI doesn’t interfere with high-risk prompts while also recognizing its potential impact on livelihoods. Making complex systems accessible to the diverse user groups who are involved in receiving healthcare required creative problem-solving, as did integrating multiple technologies into a single, cohesive platform. Each challenge taught us something new, reinforcing the importance of designing with both technology and people in mind.
### What's next
The future of our Robotic Agent is filled with possibilities, with a strong focus on transforming healthcare support. We plan to expand its integration with existing telehealth platforms, enabling seamless remote care. More sophisticated healthcare monitoring capabilities will enhance real-time assessments, while specialized routines and behaviors tailored for elderly care will provide personalized assistance. A network of connected devices will allow for large-scale institutional deployment, ensuring consistent support across hospitals, nursing homes, and assisted living facilities. AI-powered predictive care capabilities will further strengthen its ability to anticipate patient needs and improve outcomes. eHealth’s vision is to create a scalable solution that not only eases the healthcare staffing crisis but also enhances patient care while preserving the essential human connection through remote monitoring. Designed for flexibility, eHealth’s ecosystem will continue evolving with user needs and technological advancements, ensuring long-term sustainability in the ever-changing healthcare landscape.
## README (from the GitHub repository)
# treehacks
2025
## Detected evidence (automated analysis)
Indexed codebase: 25 recognized source files, 85 KB.
- CSS (language) — detected in the code
- Flask (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Google Gemini (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (35 of 35)
```
archive/gemini.py
client/.gitignore
client/package.json
client/public/index.html
client/public/manifest.json
client/public/robots.txt
client/README.md
client/src/App.css
client/src/App.js
client/src/App.test.js
client/src/components/Controls.js
client/src/components/Login.js
client/src/components/RobotCamera.js
client/src/components/VideoSection.js
client/src/components/VoiceAgent.js
client/src/index.css
client/src/index.js
client/src/reportWebVitals.js
client/src/setupTests.js
cv/.gitignore
cv/cv.py
cv/cv2.py
cv/cv3.py
rasp/drive.py
rasp/server.py
README.md
server/.gitignore
server/app.py
server/requirements.txt
voice/__init__.py
voice/.env.example
voice/.gitignore
voice/luma_input.py
voice/requirements.txt
voice/voice.py
```
### Dependencies
- client/package.json: @mdi/js@^7.4.47, @mdi/react@^1.6.1, @reduxjs/toolkit@^2.5.1, @testing-library/dom@^10.4.0, @testing-library/jest-dom@^6.6.3, @testing-library/react@^16.2.0, @testing-library/user-event@^13.5.0, @zoom/videosdk@^2.1.5, nipplejs@^0.10.2, react@^19.0.0, react-dom@^19.0.0, react-redux@^9.2.0, react-scripts@5.0.1, socket.io-client@^4.8.1, styled-components@^6.1.15, web-vitals@^2.1.4
- server/requirements.txt: elevenlabs@>=0.3.0, flask@==2.3.3, flask-cors@==4.0.0, google-generativeai@>=0.3.0, openai@>=1.12.0, pymongo@==4.6.1, python-dotenv@==1.0.0, SpeechRecognition@>=3.10.0, werkzeug@==2.3.7
- voice/requirements.txt: elevenlabs@>=0.3.0, openai@>=1.12.0, PyAudio@>=0.2.14, python-dotenv@>=1.0.0, SpeechRecognition@>=3.10.0
### Recent commits (newest first)
- Add raspberry pi and Nvidia Nano
- Sunday
- Add joystick controller
- Integrate microphone with avatar
- Save raspberry pi code
- Save raspberry pi code
- Fix avatar
- Add voice recording
- Create frontend
- Update gitignore
- Add gitignore to server
- Autoplay avatar videos
- Switch to Gemini
- Merge branch 'main' of https://github.com/sohvasa/treehacks
- Create avatar
- switched to a gemini and a diff elevenlabs model
- Update .gitignore
- Integrate ElevenLabs
- Set up frontend
- Initial commit
## Key source files (fetched from GitHub, selected and truncated for size)
### voice/requirements.txt
```
openai>=1.12.0
SpeechRecognition>=3.10.0
elevenlabs>=0.3.0
python-dotenv>=1.0.0
PyAudio>=0.2.14
```
### server/requirements.txt
```
flask==2.3.3
flask-cors==4.0.0
pymongo==4.6.1
python-dotenv==1.0.0
werkzeug==2.3.7
google-generativeai>=0.3.0
elevenlabs>=0.3.0
SpeechRecognition>=3.10.0
openai>=1.12.0
```
### client/package.json
```
{
"name": "client",
"version": "0.1.0",
"private": true,
"dependencies": {
"@mdi/js": "^7.4.47",
"@mdi/react": "^1.6.1",
"@reduxjs/toolkit": "^2.5.1",
"@testing-library/dom": "^10.4.0",
"@testing-library/jest-dom": "^6.6.3",
"@testing-library/react": "^16.2.0",
"@testing-library/user-event": "^13.5.0",
"@zoom/videosdk": "^2.1.5",
"nipplejs": "^0.10.2",
"react": "^19.0.0",
"react-dom": "^19.0.0",
"react-redux": "^9.2.0",
"react-scripts": "5.0.1",
"socket.io-client": "^4.8.1",
"styled-components": "^6.1.15",
"web-vitals": "^2.1.4"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
"eslintConfig": {
"extends": [
"react-app",
"react-app/jest"
]
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
}
}
```
### rasp/server.py
```python
import RPi.GPIO as GPIO
import time
from flask import Flask, jsonify, request
from flask_cors import CORS
import logging # Add this import
# Set up logging
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
# Define the Motor class
class Motor:
def __init__(self, in1, in2, en):
logger.debug(f"Initializing motor with pins: in1={in1}, in2={in2}, en={en}")
self.in1 = in1
self.in2 = in2
self.en = en
try:
GPIO.setup(self.in1, GPIO.OUT)
GPIO.setup(self.in2, GPIO.OUT)
GPIO.setup(self.en, GPIO.OUT)
logger.debug(f"GPIO setup successful for pins: {in1}, {in2}, {en}")
self.pwm = GPIO.PWM(self.en, 1000)
self.pwm.start(0)
logger.debug("PWM initialized successfully")
except Exception as e:
logger.error(f"Error setting up motor: {str(e)}")
raise
def set_speed(self, speed, forward=True):
try:
logger.debug(f"Setting speed to {speed}, forward={forward}")
GPIO.output(self.in1, GPIO.HIGH if forward else GPIO.LOW)
GPIO.output(self.in2, GPIO.LOW if forward else GPIO.HIGH)
self.pwm.ChangeDutyCycle(abs(speed))
logger.debug("Speed set successfully")
except Exception as e:
logger.error(f"Error setting speed: {str(e)}")
raise
def stop(self):
GPIO.output(self.in1, GPIO.LOW)
GPIO.output(self.in2, GPIO.LOW)
self.pwm.ChangeDutyCycle(0) # Stop the motor
# Initialize GPIO
try:
logger.debug("Initializing GPIO...")
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)
logger.debug("GPIO initialized successfully")
except Exception as e:
logger.error(f"Error initializing GPIO: {str(e)}")
raise
# Define motors with BCM GPIO pin numbers
try:
logger.info("Creating motor instances...")
front_left_motor = Motor(27, 17, 22)
front_right_motor = Motor(10, 9, 11)
rear_left_motor = Motor(5, 6, 13)
rear_right_motor = Motor(26, 19, 21)
logger.info("All motors initialized successfully")
except Exception as e:
logger.error(f"Error creating motors: {str(e)}")
raise
def move_forward(speed):
front_left_motor.set_speed(speed, True)
front_right_motor.set_speed(speed, True)
rear_left_motor.set_speed(speed, True)
rear_right_motor.set_speed(speed, True)
print(f"Moving forward at speed: {speed}")
def move_backward(speed):
front_left_motor.set_speed(speed, False)
front_right_motor.set_speed(speed, False)
rear_left_motor.set_speed(speed, False)
rear_right_motor.set_speed(speed, False)
print(f"Moving backward at speed: {speed}")
def turn_right(speed):
front_left_motor.set_speed(speed, False)
front_right_motor.set_speed(speed, True)
rear_left_motor.set_speed(speed, False)
rear_right_motor.set_speed(speed, True)
print(f"Turning right at speed: {speed}")
def turn_left(speed):
front_left_motor.set_speed(speed, True)
front_right_motor.set_speed(speed, False)
rear_left_motor.set_speed(speed, True)
rear_right_motor.set_speed(speed, False)
print(f"Turning left at speed: {speed}")
def stop_all_motors():
front_left_motor.stop()
front_right_motor.stop()
rear_left_motor.stop()
rear_right_motor.stop()
print("All motors stopped.")
# Initialize Flask app
app = Flask(__name__)
CORS(app)
@app.route('/move', methods=['POST'])
def handle_move():
logger.info('Received POST request to /move')
try:
data = request.get_json()
logger.debug(f"Received data: {data}")
if not data or 'speed' not in data or 'direction' not in data:
logger.error("Missing speed or direction in request")
return jsonify({'error': 'Missing speed or direction'}), 400
speed = int(data['speed'])
direction = data['direction'].lower()
logger.debug(f"Processing movement: direction={direction}, speed={speed}")
if not (0 <= speed <= 100):
return jsonify({'error': 'Speed must be between 0 and 100'}), 400
if direction == "center" or speed <= 10:
stop_all_motors()
elif direction == "forward":
move_backward(speed) # the directions are reversed
elif direction == "backward":
move_forward(speed)
elif direction == "left":
turn_left(speed)
elif direction == "right":
turn_right(speed)
else:
return jsonify({'error': 'Invalid direction'}), 400
return jsonify({'status': 'success', 'message': f'Moving {direction} at speed {speed}'})
except Exception as e:
logger.error(f"Error in handle_move: {str(e)}")
return jsonify({'error': str(e)}), 500
@app.route('/stop', methods=['POST'])
def handle_stop():
try:
stop_all_motors()
return jsonify({'status': 'success', 'message': 'All motors stopped'})
except Exception as e:
return jsonify({'error': str(e)}), 500
def cleanup():
stop_all_motors()
GPIO.cleanup()
if __name__ == "__main__":
try:
# Register cleanup handler
import atexit
atexit.register(cleanup)
# Run the Flask app
app.run(host='0.0.0.0', port=5000)
except KeyboardInterrupt:
cleanup()
```
### server/app.py
```python
from flask import Flask, request, jsonify, make_response, Response, send_from_directory
from flask_cors import CORS
from pymongo.mongo_client import MongoClient
from pymongo.server_api import ServerApi
from pymongo.errors import DuplicateKeyError
from werkzeug.security import generate_password_hash, check_password_hash
from dotenv import load_dotenv
import os
from bson import ObjectId
from datetime import datetime
from flask_socketio import SocketIO, emit
from twilio.rest import Client
import base64
import tempfile
import sys
import json
import time
sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'voice'))
from voice import get_gemini_chat_response, speak_text, transcribe_speech_to_text, generate_lipsync_video
# Load environment variables
load_dotenv()
# Initialize OpenAI client
from openai import OpenAI
openai_client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))
if not os.getenv('OPENAI_API_KEY'):
raise ValueError("OPENAI_API_KEY not found in environment variables")
app = Flask(__name__, static_folder='static')
CORS(app, resources={
r"/*": {
"origins": ["http://localhost:3000"],
"methods": ["GET", "POST", "OPTIONS"],
"allow_headers": ["Content-Type"]
}
})
socketio = SocketIO(app, cors_allowed_origins="*")
# MongoDB connection
uri = os.getenv('MONGODB_URI')
client = MongoClient(uri, tlsAllowInvalidCertificates=True)
db = client.robot_control # database name
users = db.users # collection name
# Create indexes
users.create_index('username', unique=True)
# Test MongoDB connection
try:
client.admin.command('ping')
print("Successfully connected to MongoDB!")
except Exception as e:
print(f"Error connecting to MongoDB: {e}")
# Create test user if it doesn't exist
test_user = {
"username": "admin",
"password": generate_password_hash("password123"),
"phone_number": "1234567890",
"created_at": datetime.utcnow()
}
try:
# Only insert if user doesn't exist
if not users.find_one({"username": "admin"}):
users.insert_one(test_user)
print("Test user created successfully")
except Exception as e:
print(f"Error creating test user: {e}")
# Initialize Twilio client
twilio_client = Client(
os.getenv('TWILIO_ACCOUNT_SID'),
os.getenv('TWILIO_AUTH_TOKEN')
)
def send_sms_notification(message):
"""Send SMS notification using Twilio"""
try:
twilio_client.messages.create(
body=message,
from_=os.getenv('TWILIO_PHONE_NUMBER'),
to=os.getenv('USER_PHONE_NUMBER')
)
print("SMS sent successfully: ", message)
return True
except Exception as e:
print(f"Error sending SMS: {str(e)}")
return False
def process_audio(audio_data):
"""Process audio data using OpenAI Whisper"""
try:
# Save base64 audio data to temporary file
with tempfile.NamedTemporaryFile(suffix=".wav", delete=True) as temp_audio:
audio_bytes = base64.b64decode(audio_data.split(',')[1])
temp_audio.write(audio_bytes)
temp_audio.flush()
# Transcribe audio using Whisper
with open(temp_audio.name, "rb") as audio_file:
transcript = openai_client.audio.transcriptions.create(
model="whisper-1",
file=audio_file
)
print("Transcript: ", transcript.text)
return transcript.text
except Exception as e:
print(f"Error processing audio: {str(e)}")
return None
def generate_summary(transcript):
"""Generate a summary using OpenAI GPT"""
try:
response = openai_client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[
{"role": "system", "content": "You are a helpful assistant that summarizes completed tasks."},
{"role": "user", "content": f"Please provide a brief, clear summary of this completed task: {transcript}"}
],
max_tokens=150)
print('summary: ', response.choices[0].message.content)
return response.choices[0].message.content
except Exception as e:
print(f"Error generating summary: {str(e)}")
return None
@socketio.on('voice_data')
def handle_voice_data(data):
"""Handle incoming voice data"""
try:
# Process the audio
transcript = process_audio(data['audio'])
if not transcript:
emit('voice_response', {'error': 'Failed to process audio'})
return
# Generate summary
summary = generate_summary(transcript)
if not summary:
emit('voice_response', {'error': 'Failed to generate summary'})
return
# Send SMS notification
print('summary: ', summary)
sms_sent = send_sms_notification(summary)
# Send response back to client
emit('voice_response', {
'success': True,
'transcript': transcript,
'summary': summary,
'sms_sent': sms_sent
})
except Exception as e:
emit('voice_response', {'error': str(e)})
@app.route('/api/register', methods=['POST', 'OPTIONS'])
def register():
if request.method == 'OPTIONS':
response = make_response()
response.headers.add('Access-Control-Allow-Origin', 'http://localhost:3000')
response.headers.add('Access-Control-Allow-Headers', 'Content-Type')
response.headers.add('Access-Control-Allow-Methods', 'POST')
return response
data = request.get_json()
username = data.get('username')
password = data.get('password')
phone = data.get('phone_number')
if not username or not password or not phone:
return jsonify({"error": "Missing required fields"}), 400
# Basic phone number validation
phone = ''.join(filter(str.isdigit, phone))
if len(phone) < 10:
return jsonify({'error': 'Invalid phone number'}), 400
# Hash the password
hashed
[truncated — 3842 more characters]
```
### client/src/index.js
```javascript
import React from 'react';
import ReactDOM from 'react-dom/client';
import './index.css';
import App from './App';
import reportWebVitals from './reportWebVitals';
const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
);
// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();
```
### client/src/App.js
```javascript
import React, { useState } from 'react';
import styled from 'styled-components';
import VideoSection from './components/VideoSection';
import RobotCamera from './components/RobotCamera';
import Controls from './components/Controls';
import Login from './components/Login';
const AppContainer = styled.div`
height: 100vh;
height: -webkit-fill-available;
width: 100vw;
position: fixed;
background: #0A0A0B;
overflow: hidden;
padding-bottom: env(safe-area-inset-bottom, 0px);
`;
const MainCameraSection = styled.div`
position: absolute;
top: 0;
left: 0;
right: 0;
bottom: calc(200px + env(safe-area-inset-bottom, 0px));
overflow: hidden;
`;
const ZoomBubble = styled.div`
position: absolute;
top: 24px;
right: 24px;
width: 180px;
height: 180px;
border-radius: 24px;
overflow: hidden;
box-shadow: 0 8px 32px rgba(0, 0, 0, 0.4);
border: 1px solid rgba(255, 255, 255, 0.1);
z-index: 10;
`;
const SafeArea = styled.div`
position: fixed;
bottom: 0;
left: 0;
right: 0;
height: calc(200px + env(safe-area-inset-bottom, 0px));
padding-bottom: calc(env(safe-area-inset-bottom, 0px) + 40px);
background: linear-gradient(
to bottom,
transparent,
rgba(0, 0, 0, 0.8) 30%
);
display: flex;
align-items: flex-start;
justify-content: center;
padding-top: 24px;
`;
const ControlsOverlay = styled.div`
background: rgba(18, 18, 20, 0.8);
backdrop-filter: blur(12px);
padding: 20px 32px;
border-radius: 20px;
border: 1px solid rgba(255, 255, 255, 0.1);
`;
function App() {
const [isLoggedIn, setIsLoggedIn] = useState(false);
const handleJoystickMove = (data) => {
console.log('Joystick moved:', data);
};
const handleMicToggle = (isActive) => {
console.log('Microphone:', isActive ? 'on' : 'off');
};
// Add viewport height fix for iOS
React.useEffect(() => {
const setViewportHeight = () => {
document.documentElement.style.setProperty(
'--vh',
`${window.innerHeight * 0.01}px`
);
// Reset scroll position
window.scrollTo(0, 0);
};
setViewportHeight();
window.addEventListener('resize', setViewportHeight);
return () => window.removeEventListener('resize', setViewportHeight);
}, []);
// Reset scroll position after login
React.useEffect(() => {
if (isLoggedIn) {
window.scrollTo(0, 0);
}
}, [isLoggedIn]);
return (
{!isLoggedIn ? (
setIsLoggedIn(true)} />
) : (
<>
)}
);
}
export default App;
```
### voice/__init__.py
```python
from .voice import get_gemini_chat_response, speak_text, transcribe_speech_to_text, generate_lipsync_video
__all__ = [
'get_gemini_chat_response',
'speak_text',
'transcribe_speech_to_text',
'generate_lipsync_video'
]
```
### cv/cv3.py
```python
import requests
from PIL import Image
import torch
from transformers import OwlViTProcessor, OwlViTForObjectDetection
processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch32")
model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32")
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
texts = [["a photo of a cat", "a photo of a dog"]]
inputs = processor(text=texts, images=image, return_tensors="pt")
outputs = model(**inputs)
# Target image sizes (height, width) to rescale box predictions [batch_size, 2]
target_sizes = torch.Tensor([image.size[::-1]])
# Convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax)
results = processor.post_process_object_detection(outputs=outputs, target_sizes=target_sizes, threshold=0.1)
i = 0 # Retrieve predictions for the first image for the corresponding text queries
text = texts[i]
boxes, scores, labels = results[i]["boxes"], results[i]["scores"], results[i]["labels"]
for box, score, label in zip(boxes, scores, labels):
box = [round(i, 2) for i in box.tolist()]
print(f"Detected {text[label]} with confidence {round(score.item(), 3)} at location {box}")
```
### voice/luma_input.py
```python
import google.generativeai as genai
import speech_recognition as sr
from elevenlabs import generate, play
from elevenlabs import set_api_key
from dotenv import load_dotenv
import os
# Load environment variables
load_dotenv()
# Get API Keys from environment variables
GEMINI_API_KEY = os.getenv('GEMINI_API_KEY')
ELEVENLABS_API_KEY = os.getenv('ELEVENLABS_API_KEY')
if not GEMINI_API_KEY or not ELEVENLABS_API_KEY:
raise ValueError("Missing API keys in .env file")
# Configure Gemini
genai.configure(api_key=GEMINI_API_KEY)
set_api_key(ELEVENLABS_API_KEY)
# Initialize Gemini model
model = genai.GenerativeModel('gemini-1.5-flash')
# Context window to keep track of conversation
conversation_history = [
{"role": "assistant", "content": "You are a friendly and natural-sounding AI assistant. Keep responses simple and engaging, like a human conversation."}
]
def transcribe_speech_to_text():
"""
Listens to microphone input and returns recognized text using Google's Speech Recognition.
"""
recognizer = sr.Recognizer()
with sr.Microphone() as source:
print("\nListening... (Say 'end process' to stop)")
recognizer.adjust_for_ambient_noise(source) # Reduces background noise
audio_data = recognizer.listen(source)
try:
text = recognizer.recognize_google(audio_data).lower()
print(f"You said: {text}")
return text
except sr.UnknownValueError:
print("Didn't catch that. Can you say it again?")
return None
except sr.RequestError as e:
print(f"Speech Recognition error: {e}")
return None
def get_gemini_chat_response(prompt):
"""
Sends a prompt to Gemini API and returns a human-like response.
"""
global conversation_history
# Add the new user message to conversation history
conversation_history.append({"role": "user", "content": prompt})
try:
# Create the chat request
response = model.generate_content([
{"text": msg["content"]} for msg in conversation_history
])
ai_response = response.text.strip()
# Add AI response to conversation history
conversation_history.append({"role": "assistant", "content": ai_response})
return ai_response
except Exception as e:
print(f"Error calling Gemini API: {e}")
return "Hmm... I ran into a problem. Can you try again?"
def speak_text(text):
"""
Uses ElevenLabs API (Eric - Eleven Multilingual v2) to generate a more human-like voice response.
"""
try:
audio = generate(
text=text,
voice="Matthew",
model="eleven_flash_v2_5"
)
play(audio)
except Exception as e:
print(f"Error using ElevenLabs API: {e}")
def main():
"""
Runs a continuous conversation loop until the user says "end process".
"""
print("Chat started! Say 'end process' anytime to stop.")
while True:
# Listen to user input
user_input = transcribe_speech_to_text()
if user_input:
if user_input == "end process":
print("Ending chat...")
speak_text("Goodbye! Have a great day!")
break
# Get AI response using Gemini
ai_response = get_gemini_chat_response(user_input)
print(f"AI: {ai_response}")
# Convert response to speech
speak_text(ai_response)
if __name__ == "__main__":
main()
```
[16 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]