# Project export: Orbit This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: From “I should” to “I did.”Our workflow listens, understands, and drives real progress from every conversation. - Devpost: https://devpost.com/software/orbit-n97hqz - GitHub: https://github.com/Ganeshmohank/orbit.git - Demo: https://drive.google.com/drive/folders/1bx7zZMYRjNHUiyFbiJHbxRaMBweDjGtI?usp=drive_link - Video: https://www.youtube.com/embed/-H-pOgb0LvM?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Fetch AI: Best Use of Fetch AI) - Team: 1 GitHub contributor(s) — Ganeshmohank (11 commits) ## Devpost submission (written by the team) ### Overview Turning every conversation into measurable progress. The Problem Every meeting ends with a list of things to do, but most of them never get done. Professionals spend an average of 11.3 hours per week in meetings, and nearly 57% of their workweek goes into communication instead of creation. This inefficiency costs companies over $29,000 per employee annually. Existing tools like Fellow and Tactiq only capture notes or action items - they “record” work, but they don’t move it forward. Teams need automation that goes beyond transcription - one that truly acts. Our Solution Orbit is an AI-powered productivity agent that listens to meetings, understands context, and executes real actions across your workflow tools - automatically. It connects your conversations with your workflows, transforming talk into tangible results. How it works: Listens to meetings (Zoom, Google Meet) and identifies key tasks, deadlines, and owners. Extracts and executes actions: creates Jira tickets, sends follow-up emails, and schedules events. Supports voice-triggered commands via the Omi Mic e.g., “Book a cab to SFO” or “Remind the team about the release.” Powered by a multi-agent (We call it NeXus) system using Sim.ai, Composio MCP, and Fetch.ai Agentverse. Orbit doesn’t just summarize your meetings, it gets things done. How We Built It Sim.ai - reasoning and intent extraction from raw meeting transcripts, with multiple triggers Composio MCP Integrations - automation for Jira, Gmail, Calendar, and more. Omi Mic - enables natural, voice-triggered automation. Fetch.ai Agentverse - gives Orbit autonomy, persistence, and collaboration across agents. Under the hood, Orbit transforms unstructured dialogue into structured JSON actions, validates them with Sim.ai, and triggers automation sequences through Composio MCP - with full transparency and user review. Challenges We Ran Into Combining multiple inputs (Zoom + Voice + Email) into a single cohesive workflow. Translating ambiguous natural language into clear, executable tasks. Handling authentication, API rate limits, and user trust. Balancing autonomy with control - ensuring Orbit acts responsibly. Accomplishments We’re Proud Of Built a full end-to-end AI workflow that connects meeting transcripts to task automation. Successfully hosted the autonomous agent on Fetch.ai Agentverse. Enabled real-time voice control with Omi Mic, Did any app can book a ride for you from voice command Designed a unified brand identity - Orbit: your AI orbit that keeps life in motion. Market Insight Every employee loses nearly one-third of their week to communication overhead. Automation of post-meeting tasks shows clear ROI and faster adoption. Competitors like Fellow and Tactiq stop at transcription - Orbit extends into execution. Our middleware, Nexus, integrates with any MCP, expanding Orbit’s reach into scheduling, logistics, and enterprise automation. Go-To-Market Strategy Inbound content: Blog posts & case studies on “lost meeting actions,” “voice command productivity,” and “meeting-to-execution automation.” Demo-led sales: Live demo showing post-meeting automation in real time. Partner integrations: Zoom, Jira, Slack, and hardware partnerships with Omi Mic. Pricing model: Freemium: Basic meeting summaries + limited integrations. Team/SMB ($20–30/user/month): Full automation suite + Nexus integration. Pricing model: Freemium: Basic meeting summaries + limited integrations. Team/SMB ($20–30/user/month): Full automation suite + Nexus integration. What’s Next Expand to Microsoft Teams, Slack, and Notion integrations. Introduce contextual memory for ongoing projects. Enable multi-agent collaboration between team members. Launch as a desktop widget and browser extension for instant accessibility. Orbit - Turning every conversation into measurable progress. From “I should” → “I did.” Links Review more about our NeXes here GitHub Repository Contact Us ## README (from the GitHub repository) # Orbit - Agentic Workflow Automation System Orbit is an intelligent agent designed to streamline daily workflows by integrating Jira, Google Calendar, and other productivity tools. It automates task management, meeting scheduling, progress tracking, and status reporting, enabling teams and individuals to focus on high-value work instead of manual coordination. ## Architecture Overview Orbit consists of three core components: ### 1. **Omi** - Entry Point Voice capture device that records conversations and sends voice commands to the system. - Listens for voice commands like "Book an Uber", "Schedule a meeting", "Update my Jira tickets" - Sends audio segments to the webhook for processing - Receives task confirmation and status updates ### 2. **Nexus** - Middleware Agent Intelligent agent that processes voice commands and communicates with external services. - Parses voice commands using LLM - Routes commands to appropriate MCP (Model Context Protocol) servers - Maintains context from conversation flow - Executes multi-step workflows ### 3. **MCP Servers** - Service Integrations Extensible protocol for connecting to external services: - **Jira** - Update tickets, create issues, manage sprints - **Slack** - Send messages, create channels, post notifications - **Teams** - Send messages, schedule meetings - **Calendar** - Schedule meetings, check availability - **PR Systems** - Create pull requests, manage code reviews - *(More services can be added)* ## Project Structure ``` orbit/ ├── main.py # FastAPI app with webhook and endpoints ├── ride_detector.py # LLM-powered command extraction ├── simple_storage.py # File-based user storage ├── auth_manager.py # Authentication and session management ├── uber_automation.py # Browser automation for ride booking │ ├── middleware/ │ ├── agent.py # Nexus agent - MCP server communication │ ├── mcp_client.py # MCP protocol client │ └── AGENTVERSE_README.md # Nexus architecture documentation │ ├── requirements.txt # Python dependencies ├── .env.example # Environment variables template ├── Dockerfile # Docker configuration ├── docker-compose.yml # Multi-container setup ├── railway.toml # Railway deployment config └── README.md # This file ``` ## Features ### Voice Command Processing - **Sliding Window Collection** - Batches voice segments with 5 seconds of silence detection - **LLM-Powered Extraction** - Understands natural language and corrects spelling mistakes - **Multi-Service Routing** - Routes commands to appropriate MCP servers ### Supported Commands **Ride Booking (via Uber automation)** ``` "Book an Uber to Pier 39" "Get me a ride from SJSU to the airport" "Call an Uber to downtown" ``` **Jira Integration (via Nexus + MCP)** ``` "Update my Jira ticket PROJ-123 to done" "Create a new ticket for bug fix" "Show me my assigned tickets" ``` **Calendar Integration (via Nexus + MCP)** ``` "Schedule a meeting with the team tomorrow at 2pm" "Check my availability next week" "Add this to my calendar" ``` **Extensible to more services:** - Slack notifications - Teams messaging - PR creation and reviews - And more... ## Setup ### 1. Install ```bash git clone cd orbit python3 -m venv venv source venv/bin/activate pip install -r requirements.txt playwright install chromium ``` ### 2. Configure ```bash cp .env.example .env # Add your API keys: # - OPENAI_API_KEY (for LLM processing) # - JIRA_API_TOKEN (for Jira integration) # - Other service credentials as needed ``` ### 3. Run ```bash uvicorn main:app --reload ``` Visit `http://localhost:8000` ## Configuration ### Environment Variables - `OPENAI_API_KEY` - OpenAI API key for LLM processing - `JIRA_API_TOKEN` - Jira API token for ticket management - `JIRA_DOMAIN` - Your Jira instance domain - `PORT` - Server port (default: 8000) - `AUTO_REQUEST` - Auto-book rides (default: false) **Example .env:** ``` OPENAI_API_KEY=sk-proj-xxxxx JIRA_API_TOKEN=your-jira-token JIRA_DOMAIN=your-domain.atlassian.net PORT=8000 AUTO_REQUEST=false ``` ## API Endpoints ### GET `/` Home page showing system status and authentication. ### GET `/auth` Authentication flow for Uber account connection. ### POST `/webhook` Main entry point for voice commands from Omi device. **Request Body:** ```json { "segments": [ { "text": "Book an Uber to Pier 39", "speaker": "user" } ], "gps_lat": 37.7749, "gps_lon": -122.4194 } ``` **Response:** ```json { "message": "Processing command...", "command_type": "ride_booking|jira_update|calendar_event", "status": "processing|completed|failed" } ``` ### GET `/health` Health check endpoint. ## Workflow Examples ### Example 1: Voice-to-Uber Booking ``` User (via Omi): "Book an Uber to Pier 39" ↓ Webhook receives segments ↓ Sliding window waits for 5s silence ↓ LLM extracts: destination="Pier 39" ↓ Browser automation books ride ↓ Confirmation sent back to Omi ``` ### Example 2: Voice-to-Jira Update ``` User (via Omi): "Update ticket PROJ-123 to done" ↓ Webhook receives segments ↓ LLM extracts: ticket="PROJ-123", status="done" ↓ Nexus agent connects to Jira MCP server ↓ Jira ticket updated ↓ Confirmation sent back to Omi ``` ### Example 3: Voice-to-Calendar Event ``` User (via Omi): "Schedule a meeting tomorrow at 2pm" ↓ Webhook receives segments ↓ LLM extracts: event details from conversation context ↓ Nexus agent connects to Calendar MCP server ↓ Meeting scheduled ↓ Confirmation sent back to Omi ``` ## Documentation For detailed architecture and system design documentation, see: - **documentation/START_HERE.md** - Quick start guide with system overview and architecture diagrams - **documentation/architectures/ARCHITECTURE_UNIVERSE.md** - Complete system architecture and data flow - **documentation/architectures/ARCHITECTURE_OMI.md** - Voice entry point (Omi device) documentation - **documentation/architectures/ARCHITECTURE_NEXUS.md** - Middleware brain (Nexus agent) documentation - **documentation/architectures/ARCHITECTURE_MCP.md** - Service integrations (Jira MCP, Google Calendar MCP, Uber Service) - **documentation/architectures/ARCHITECTURE_GUIDE.md** - Navigation guide for all architecture documentation ## Nexus Agent (Middleware) The Nexus agent is the intelligent middleware that: - Maintains conversation context - Routes commands to appropriate services - Handles multi-step workflows - Manages service integrations See `middleware/AGENTVERSE_README.md` for detailed architecture. ## Deployment ### Docker ```bash docker-compose up ``` ### Railway ```bash railway link railway variables set OPENAI_API_KEY=your_key railway up ``` ### Heroku ```bash heroku create your-app-name heroku config:set OPENAI_API_KEY=your_key git push heroku main ``` ## Security - ✅ Session files stored locally (not in version control) - ✅ API keys stored in environment variables - ✅ HTTPS enforced in production - ✅ Rate limiting on endpoints - ✅ User authentication validation ## Extensibility To add a new service: 1. Create an MCP server for the service 2. Add connection logic to `middleware/agent.py` 3. Update `ride_detector.py` to recognize commands for the service 4. Add environment variables for service credentials 5. Test with voice commands via the webhook ## Development ### Running Tests ```bash pytest tests/ ``` ### Code Style ```bash black *.py flake8 *.py ``` ## License Proprietary License - Approval Required This project is proprietary and requires explicit written approval from the author before use, modification, or distribution. **To request approval:** - Email: mohankancherla519@gmail.com - Phone: +1 6693257754 - Include: Project details, intended use case, and timeline - Wait for written [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 20 recognized source files, 191 KB. - FastAPI (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code ## Codebase structure (from repository index) ### Files (25 of 25) ``` .gitignore auth_manager.py browser_pool.py debug_booking.py docker-compose.yml Dockerfile documentation/architectures/ARCHITECTURE_GUIDE.md documentation/architectures/ARCHITECTURE_MCP.md documentation/architectures/ARCHITECTURE_NEXUS.md documentation/architectures/ARCHITECTURE_OMI.md documentation/architectures/ARCHITECTURE_UNIVERSE.md documentation/START_HERE.md main.py middleware/agent.py middleware/AGENTVERSE_README.md Procfile PROPRIETARY_LICENSE.md railway.toml README.md requirements.txt ride_detector.py setup.sh simple_storage.py test_app.py uber_automation.py ``` ### Dependencies - requirements.txt: aiofiles@>=23.2.1, fastapi@>=0.110.0, httpx@>=0.25.0, openai@>=1.14.0, playwright@>=1.46.0, pydantic@>=2.7.0, python-dotenv@>=1.0.0, uvicorn@>=0.28.0 ### Recent commits (newest first) - docs: update licensing from MIT to proprietary - docs: reorganize architecture documentation into subdirectories - refactor: reorganize documentation and remove unused images - chore: update gitignore patterns for test snapshots - docs: remove outdated API documentation files - feat: replace regex patterns with LLM for ride request detection - feat: optimize ride booking flow and reduce timeouts - feat: add geolocation and GPS coordinate support for ride pickup - docs: add comprehensive API documentation and changelog - feat: enhance voice-based Uber booking with segment batching - first commit ## Key source files (fetched from GitHub, selected and truncated for size) ### PROPRIETARY_LICENSE.md ```markdown # Proprietary License Agreement ## Orbit - Agentic Workflow Automation System **Copyright (c) 2024 Mohan Kancherla. All rights reserved.** --- ## 1. Grant of License This software and associated documentation files (the "Software") are proprietary and confidential. No license is granted to any person or entity except as expressly provided in this agreement. ## 2. Restrictions You may NOT: - Use the Software without explicit written approval from the copyright holder - Copy, modify, or distribute the Software - Reverse engineer, decompile, or disassemble the Software - Create derivative works based on the Software - Use the Software for commercial purposes without a commercial license agreement - Sublicense or transfer rights to the Software ## 3. Permitted Use Use of this Software is permitted ONLY: - With explicit written approval from the copyright holder - For the specific purpose stated in the approval - Under the terms specified in the approval letter - For the duration specified in the approval letter ## 4. Approval Process To request approval for use: - Email: mohankancherla519@gmail.com - Phone: +1 6693257754 - Provide: Project details, intended use case, and timeline - Wait for written approval before proceeding ## 5. Intellectual Property All intellectual property rights, including but not limited to: - Source code - Documentation - Architecture designs - Algorithms - Concepts remain the exclusive property of Mohan Kancherla. ## 6. Disclaimer THE SOFTWARE IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. ## 7. Limitation of Liability IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ## 8. Termination Any unauthorized use of this Software will result in immediate termination of any implied license and may result in legal action. ## 9. Governing Law This license is governed by applicable law and jurisdiction where the copyright holder resides. --- **For licensing inquiries, contact:** - Email: mohankancherla519@gmail.com - Phone: +1 6693257754 **Last Updated:** October 2024 ``` ### middleware/AGENTVERSE_README.md ```markdown # Meeting Analysis Agent ## Description An AI-powered agent that analyzes meeting transcripts and extracts: - **Action items** with assignees, priorities, and due dates - **Decisions** made during the meeting with rationale - **Unresolved questions** that need follow-up - **Participant summaries** showing contributions - **Overall sentiment** analysis of the meeting Powered by **Google Gemini 2.5 Flash** for fast, accurate, and cost-effective analysis. ## Input Format Send a JSON object with the following structure: ```json { "meeting_title": "Sprint Planning Q4", "meeting_date": "2024-10-25T14:00:00Z", "participants": ["Alice", "Bob", "Carol"], "transcript": [ { "speaker": "Alice", "text": "Let's discuss the Q4 roadmap. We need to prioritize the API redesign.", "timestamp": "14:00:00" }, { "speaker": "Bob", "text": "I'll handle the backend API by Friday. It's critical for the release.", "timestamp": "14:05:30" }, { "speaker": "Carol", "text": "Should we also update the documentation?", "timestamp": "14:10:15" } ] } ``` ### Required Fields: - `meeting_title` (string): Title or subject of the meeting - `meeting_date` (string): ISO 8601 format date (e.g., "2024-10-25T14:00:00Z") - `participants` (array): List of participant names - `transcript` (array): List of transcript entries with: - `speaker` (string): Name of the person speaking - `text` (string): What was said - `timestamp` (string, optional): When it was said ### Optional Fields: - `metadata` (object): Any additional metadata about the meeting ## Output Format Returns a formatted analysis including: ### 1. Meeting Summary Concise overview of what was discussed and decided ### 2. Action Items Each action item includes: - Task description - Assigned person - Priority level (critical, high, medium, low) - Due date (if mentioned) - Context from the conversation ### 3. Decisions Key decisions made with: - What was decided - Rationale behind the decision - Participants involved ### 4. Unresolved Questions Questions that need follow-up with: - The question text - Who asked it - Context ### 5. Sentiment Analysis Overall tone of the meeting (positive, neutral, negative, mixed) ## Example Output ``` 📊 **Meeting Analysis: Sprint Planning Q4** 📅 Date: 2024-10-25T14:00:00Z 🏷️ Type: Sprint Planning **Summary:** The team discussed Q4 priorities focusing on API redesign. Bob committed to completing the backend API by Friday, which is critical for the upcoming release. Documentation updates were raised as a potential concern. **✅ Action Items (1):** 1. **Complete backend API redesign** - Assignee: Bob - Priority: high - Due: Friday **🎯 Decisions (1):** 1. Prioritize API redesign for Q4 release - Rationale: Critical for product launch timeline **❓ Unresolved Questions (1):** 1. Should we also update the documentation? **😊 Sentiment:** Positive ``` ## Required Secrets To use thi [truncated — 2169 more characters] ``` ### requirements.txt ``` fastapi>=0.110.0 uvicorn>=0.28.0 playwright>=1.46.0 openai>=1.14.0 python-dotenv>=1.0.0 pydantic>=2.7.0 aiofiles>=23.2.1 httpx>=0.25.0 ``` ### docker-compose.yml ```yaml version: '3.8' services: app: build: . ports: - "8000:8000" environment: - OPENAI_API_KEY=${OPENAI_API_KEY} - PORT=8000 - PLAYWRIGHT_BROWSERS_PATH=/app/.cache/ms-playwright volumes: - ./sessions:/app/sessions - ./users:/app/users restart: unless-stopped healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8000/health"] interval: 30s timeout: 10s retries: 3 start_period: 5s ``` ### Dockerfile ``` FROM python:3.10-slim WORKDIR /app # Install system dependencies for Playwright RUN apt-get update && apt-get install -y \ wget \ gnupg \ apt-transport-https \ ca-certificates \ && rm -rf /var/lib/apt/lists/* # Copy requirements COPY requirements.txt . # Install Python dependencies RUN pip install --no-cache-dir -r requirements.txt # Install Playwright browsers RUN playwright install chromium # Copy application code COPY . . # Create directories for storage RUN mkdir -p sessions users # Expose port EXPOSE 8000 # Health check HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \ CMD python -c "import requests; requests.get('http://localhost:8000/health')" # Run application CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"] ``` ### main.py ```python import os import asyncio import time import json from typing import Optional from dotenv import load_dotenv from fastapi import FastAPI, HTTPException, BackgroundTasks, Request from fastapi.responses import HTMLResponse, RedirectResponse from fastapi.staticfiles import StaticFiles from pydantic import BaseModel import logging # Load environment variables FIRST before importing modules that use them load_dotenv() from auth_manager import auth_manager, active_browsers from uber_automation import uber_automation from ride_detector import detect_trigger_and_destinations, get_pickup_location_from_ip from simple_storage import ( load_user_data, update_user_status, load_session, ) # Setup logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) app = FastAPI(title="Omi Uber App", version="1.0.0") # Rate limiting to prevent bombarding Uber last_booking_time = {} MIN_BOOKING_INTERVAL = 30 # Minimum 15 seconds between bookings per user # Active bookings - only 1 per user active_bookings = {} # {uid: bool} # Segment buckets for collecting voice data segment_buckets = {} # {uid: [segments]} segment_last_arrival = {} # {uid: timestamp of last segment} bucket_timers = {} # {uid: asyncio.Task} BUCKET_WAIT_TIME = 5 # Wait 5 seconds from last segment before processing # Models class VoiceSegment(BaseModel): text: str speaker: str class WebhookRequest(BaseModel): uid: str segments: list[VoiceSegment] class TwoFARequest(BaseModel): uid: str code: str class AuthStatusResponse(BaseModel): status: str message: str # ============================================================================ # HEALTH CHECK # ============================================================================ @app.get("/health") async def health_check(): """Health check endpoint for deployment.""" return {"status": "ok", "service": "omi-uber-app"} # ============================================================================ # HOME PAGE # ============================================================================ @app.get("/", response_class=HTMLResponse) async def home(uid: Optional[str] = None): """App home page - shows authentication status and usage instructions.""" if not uid: uid = "default_user" user_data = load_user_data(uid) is_authenticated = user_data.get("uber_authenticated", False) auth_status = user_data.get("auth_status", "not_authenticated") if auth_status == "waiting_2fa": status_html = """

📱 Waiting for 2FA Verification

Please enter your verification code below

""" elif is_authenticated: status_html = """

✅ Uber Connected

Your Uber account is authenticated and ready to use.

How to use:

  • Say "Book an Uber to [destination]"
  • Or "Get me a ride to [destination]"
  • Your ride will be booked automatically
Re-authenticate
""" else: status_html = """

🔐 Connect Your Uber Account

Authenticate once to start booking rides with your voice.

Connect Uber Account
""" html = f""" Omi Uber App