# Project export: sheltr This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: Sheltr is an AI disaster response tool that analyzes tweets using text and image models to assess crises and alert users in danger zones via calls, texts, and a live help agent. - Devpost: https://devpost.com/software/sheltr-gyvk6f - GitHub: https://github.com/aditii-jain/calhacks - Video: https://www.youtube.com/embed/pezSakegb64?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) โ€” Aditi Jain (22 commits), suchi (13 commits), Aadi Biyani (11 commits) ## Devpost submission (written by the team) ### Inspiration Our whole team is from Los Angeles and the Los Angeles wildfires didn't just burn through our neighborhoods - they burned through our hearts. As we watched our community scramble for information, we witnessed something heartbreaking: while some of us frantically refreshed social media for updates, our grandparents, elderly neighbors, and loved ones with disabilities sat in silence, waiting for someone to tell them if they were safe. Information inequality in crisis isn't just inconvenient - it's deadly. We realized that the people who need disaster information the most are often the last to receive it. Your grandmother doesn't scroll Twitter during a wildfire. Your disabled neighbor might not have access to emergency apps. The elderly couple down the street doesn't know which news sources to trust. While the digitally connected world shares real-time updates, vulnerable populations remain in the dark, sometimes with devastating consequences. Sheltr was born from a simple but urgent belief: every person deserves to know when danger is approaching, regardless of their digital literacy or social media presence. ### What it does Sheltr is an AI-powered guardian that bridges the dangerous gap between digital crisis information and the people who need it most. Our platform continuously monitors social media streams, identifies emerging disasters through advanced multimodal AI analysis, and proactively reaches vulnerable individuals with personalized, life-saving communication. Here's how we protect those who matter most: ๐Ÿ” Intelligent Crisis Detection Real-time analysis of social media feeds using our fine-tuned multimodal AI model Fusion of text and image data to accurately identify and classify disaster events Severity scoring and geographic mapping to identify "Red Zones" requiring immediate action ๐Ÿ“ž Proactive, Personal Outreach Automated, empathetic phone calls to registered vulnerable individuals in affected areas Location-aware text alerts sent to emergency contacts Human-like conversation that provides guidance, resources, and most importantlyโ€”comfort ๐ŸŽฏ Designed for Digital Equity Specifically built for elderly, disabled, and digitally disconnected populations Uses familiar communication methods (phone calls, SMS) instead of requiring app downloads Pre-registered family members can ensure their loved ones are protected ### How we built it The AI Classifier: We fine-tuned a Llama 3 multimodal model using the CrisisMMD datasetโ€”a comprehensive collection of thousands of manually annotated disaster-related social media posts. This specialized training enables our AI to understand the nuanced relationship between crisis imagery and text, accurately identifying disaster types, information quality, and urgency levels. The Detection Engine: Our system continuously scrapes and analyzes social media feeds, classifying posts across multiple parameters: Disaster type and severity Information reliability and usefulness Geographic relevance and impact zones Urgency scoring for immediate response The Response Network: When threat levels reach critical thresholds in any geographic area, Sheltr's automated response system springs into action: Vapi-powered voice calls deliver personalized, location-specific guidance to registered vulnerable individuals Gemini-generated text alerts notify emergency contacts with status updates and safety confirmations Intelligent routing using MCP that browses the internet ensures the right information about safety shelters reaches the right people at the right time The Safety Net: Every interaction is designed to provide not just information, but reassurance. Our AI agent doesn't just warn - it guides, comforts, and connects people to resources, ensuring no one faces crisis alone. ### Challenges we ran into Beyond the usual rate limits and depleted API credits, our biggest challenge was supporting a team member who was new to hackathons. None of us had worked with Vapi before, and we had to quickly learn about Smithery to integrate MCP services - all while racing against the clock. The learning curve was steep, but it brought our team closer together. ### Accomplishments we're proud of Our biggest accomplishment is the clean system architecture we built. We successfully dockerized our entire API on Railway's cloud platform, orchestrated multiple AI models working in harmony, and delivered a polished frontend experience. Most importantly, our API pipeline ensures that vulnerable people receive critical information through both personalized phone calls and SMS alertsโ€”exactly what we set out to achieve. ### What we learned This hackathon pushed us to master new technologies: MCP integration, Gemini API, advanced AI coding tools, and Railway deployment. We also learned that the most impactful solutions often come from addressing overlooked problems with familiar tools. ### What's next for Sheltr Immediate Goals: Deploy a live Twitter bot for real-time disaster monitoring beyond historical data Add multilingual support to serve diverse communities Integrate with official emergency sources (FEMA, USGS) for enhanced accuracy and validation Future Vision: Improve personalization based on user demographics and accessibility needs Build a mobile app for easier family registration and alert customization Launch community outreach programs to connect with vulnerable populations Scale to a production-ready platform deployable in disaster-prone regions worldwide Our ultimate goal remains unchanged: ensuring that when disaster strikes, no one - regardless of age, ability, or digital access - gets left behind. ## README (from the GitHub repository) # ๐Ÿšจ Crisis-MMD: Intelligent Crisis Response System
![Crisis-MMD Banner](https://img.shields.io/badge/๐Ÿšจ_Crisis--MMD-Emergency_Response-FF6B35?style=for-the-badge) [![Built for CalHacks](https://img.shields.io/badge/Built_for-CalHacks_2024-1DA1F2?style=for-the-badge)](https://calhacks.io/) [![FastAPI](https://img.shields.io/badge/FastAPI-005571?style=for-the-badge&logo=fastapi)](https://fastapi.tiangolo.com/) [![Next.js](https://img.shields.io/badge/Next.js-000000?style=for-the-badge&logo=next.js&logoColor=white)](https://nextjs.org/) [![Supabase](https://img.shields.io/badge/Supabase-3ECF8E?style=for-the-badge&logo=supabase&logoColor=white)](https://supabase.com/) **When crisis strikes, we speak up.** *AI-powered multimodal disaster detection and emergency response system that saves lives through intelligent alerts and voice guidance.* [๐Ÿš€ Live Demo](#-demo) โ€ข [๐Ÿ“– Documentation](#-features) โ€ข [โšก Quick Start](#-quick-start) โ€ข [๐ŸŽฏ How It Works](#-how-it-works)
--- ## ๐ŸŽฏ The Problem We Solve In disasters, **every second counts**. Traditional emergency systems are slow, impersonal, and often fail when people need them most. Social media becomes flooded with crisis information, but there's no intelligent way to process it and turn it into actionable emergency responses. **Crisis-MMD** changes that. ## โœจ What Makes Us Different ๐Ÿง  **AI-Powered Detection**: Analyzes text and images from social media to detect real emergencies ๐Ÿ“ž **Intelligent Voice Calls**: Makes personalized emergency calls with calm, helpful guidance ๐Ÿ—บ๏ธ **Real-Time Crisis Map**: Live visualization of disasters as they unfold ๐Ÿ“ฑ **Smart Notifications**: Multi-channel alerts via SMS, voice, and app notifications ๐ŸŽฏ **Precise Targeting**: Only alerts people actually in danger zones ๐Ÿค– **Multimodal Analysis**: Understands both text and visual crisis indicators --- ## ๐Ÿš€ Demo ### ๐ŸŒ Live Applications | Component | Description | Link | Status | |-----------|-------------|------|--------| | **Sheltr WebApp** | Main crisis management dashboard | [๐Ÿ”— sheltr.app](#) | โœ… Live | | **Crisis Map** | Real-time disaster visualization | [๐Ÿ”— crisis-map.app](#) | โœ… Live | | **Mock Twitter** | Crisis reporting interface | [๐Ÿ”— localhost:8000](http://localhost:8000) | ๐Ÿƒ Local | | **API Backend** | Crisis processing engine | [๐Ÿ”— api.crisis-mmd.app](#) | โœ… Live | ### ๐Ÿ“ฑ Screenshots
๐Ÿ–ผ๏ธ Click to see our beautiful interfaces #### Sheltr WebApp - Crisis Dashboard ``` โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ ๐Ÿšจ sheltr ๐Ÿ”” ๐Ÿ‘ค JD โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ โ”‚ โ”‚ ๐Ÿ›ก๏ธ Your Status โ”‚ โ”‚ ๐ŸŸก Fire risk detected 2.1 mi from you โ”‚ โ”‚ โ”‚ โ”‚ ๐Ÿ“ž [Notify Contacts] ๐Ÿงญ [Find Shelter] โ”‚ โ”‚ โ”‚ โ”‚ ๐Ÿ—บ๏ธ [Interactive Crisis Map] โ”‚ โ”‚ โ”‚ โ”‚ ๐Ÿ“ž Call History โ”‚ โšก Quick Actions โ”‚ โ”‚ ๐Ÿ”ฅ Fire Alert โ”‚ ๐Ÿ‘ฅ Update Contacts โ”‚ โ”‚ ๐Ÿšจ Evacuation โ”‚ ๐Ÿ“ Change Radius โ”‚ โ”‚ โ›ˆ๏ธ Weather Alert โ”‚ ๐Ÿ”” Test Voice Alert โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ ``` #### Mock Twitter - Crisis Reporting ``` โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ ๐Ÿšจ MockTwitter - Crisis Alert System โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ โ”‚ โ”‚ ๐Ÿ“ What's happening? Report emergencies... โ”‚ โ”‚ ๐Ÿ“ท [Photo] ๐Ÿ“ [Add location] [280] [Post] โ”‚ โ”‚ โ”‚ โ”‚ ๐Ÿ”ด Live Crisis Feed โ”‚ โ”‚ โ”œโ”€ ๐Ÿ”ฅ Wildfire spotted near highway... โ”‚ โ”‚ โ”œโ”€ ๐ŸŒŠ Flooding reported downtown... โ”‚ โ”‚ โ””โ”€ โšก Power outage affecting... โ”‚ โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ ```
--- ## ๐Ÿ—๏ธ Architecture ```mermaid graph TB A[๐Ÿฆ Social Media Data] --> B[๐Ÿค– AI Classification Engine] B --> C[๐Ÿ“Š Crisis Aggregation] C --> D[๐ŸŽฏ Red Zone Detection] D --> E[๐Ÿ“ž Voice Agent System] D --> F[๐Ÿ“ฑ SMS Notifications] D --> G[๐ŸŒ Web Dashboard] H[๐Ÿ‘ฅ User Database] --> I[๐Ÿ“ Location Matching] I --> D J[๐Ÿ—บ๏ธ Crisis Map] --> K[๐Ÿ“ˆ Real-time Visualization] C --> J style A fill:#FF6B35 style B fill:#4B5D67 style D fill:#F4A261 style E fill:#2A9D8F style F fill:#E76F51 style G fill:#264653 ``` ## ๐Ÿ› ๏ธ Tech Stack ### ๐ŸŽจ Frontend - **Next.js 14** - React framework with App Router - **TypeScript** - Type-safe development - **Tailwind CSS** - Beautiful, responsive styling - **shadcn/ui** - Premium component library ### โšก Backend - **FastAPI** - High-performance Python API - **Supabase** - PostgreSQL database with real-time features - **Pydantic** - Data validation and serialization - **SQLAlchemy** - Database ORM ### ๐Ÿค– AI & Intelligence - **Google Gemini** - Multimodal AI for content analysis - **VAPI** - Voice AI for emergency calls - **Custom ML Models** - Crisis classification and sentiment analysis ### ๐Ÿ”ง Infrastructure - **Railway** - Backend deployment and hosting - **Vercel** - Frontend deployment - **Twilio** - SMS and voice communications - **Docker** - Containerized deployment --- ## ๐ŸŽฏ How It Works ### 1. ๐Ÿ” **Crisis Detection** ```python # AI analyzes social media posts result = await classify_crisis({ "text": "Massive wildfire spreading fast near Highway 101!", "image": "wildfire_photo.jpg", "location": "San Jose, CA" }) # Returns: {"severity": 0.89, "type": "wildfire", "urgent": true} ``` ### 2. ๐Ÿ“Š **Intelligent Aggregation** ```python # Combines multiple reports into crisis zones aggregate_score = calc_aggregate_score({ "reports": [report1, report2, report3], "location": "San Jose, CA", "time_window": "30_minutes" }) # Creates heat map of crisis intensity ``` ### 3. ๐ŸŽฏ **Red Zone Activation** ```python # Triggers emergency response for affected areas await trigger_red_zone({ "city": "San Jose, CA", "incident_data": { "type": "wildfire", "severity": "high", "evacuation_routes": ["Highway 280", "Highway 85"] } }) ``` ### 4. ๐Ÿ“ž **Personalized Emergency Calls** ```python # AI voice agent calls people in danger call_result = await voice_agent.emergency_call({ "phone": "+1234567890", "location": "Downtown San Jose", "disaster_type": "wildfire", "personalized_context": user_profile }) ``` --- ## โšก Quick Start ### ๐Ÿš€ One-Click Setup ```bash # Clone the repository git clone https://github.com/aditii-jain/calhacks.git cd calhacks # Set up environment variables cp .env.example .env # Add your API keys (Supabase, Gemini, VAPI, Twilio) # Start everything with Docker docker-compose up -d # Or run locally: # Backend cd backend && pip install -r requirements.txt && python main.py # Frontend cd frontend/webapp && npm install && npm run dev # Mock Twitter cd frontend/mock-twitter && python server.py ``` ### ๐Ÿ”‘ Required API Keys ```env # Supabase (Database) SUPABASE_URL=your_supabase_url SUPABASE_ANON_KEY=your_anon_key SUPABASE_SERVICE_KEY=your_service_key # AI Services GEMINI_API_KEY=your_gemini_key VAPI_API_KEY=your_vapi_key # Communications (Optional for testing) TWILIO_ACCOUNT_SID=your_twilio_sid TWILIO_AUTH_TOKEN=your_twilio_token ``` ### ๐ŸŽฎ Try It Out 1. **Report a Crisis**: Go to `localhost:8000` and post about an emergency 2. **Watch AI Analysis**: See real-time classification in the backend logs 3. **View Crisis Map**: Open the webapp to see crisis zones appear 4. **Test Voice Calls**: Trigger red zone to initiate emergency calls --- ## ๐ŸŒŸ Features ### ๐ŸŽจ **Sheltr WebApp** - ๐Ÿ“ฑ **Responsive Design** - Beautiful on all devices - ๐Ÿ”” **Real-time Alerts** - Instant notifications for nearby crises - ๐Ÿ‘ฅ **Contact Management** - Emergency [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 111 recognized source files, 498 KB. - CSS (language) โ€” detected in the code - FastAPI (technology) โ€” detected in the code - Flask (technology) โ€” detected in the code - HTML (language) โ€” detected in the code - JavaScript (language) โ€” detected in the code - Next.js (technology) โ€” detected in the code - Python (language) โ€” detected in the code - React (technology) โ€” detected in the code - SQL (language) โ€” detected in the code - Supabase (technology) โ€” detected in the code - Tailwind CSS (technology) โ€” detected in the code - TypeScript (language) โ€” detected in the code - Google Gemini (technology) โ€” claimed on Devpost, not found in the code - Node.js (technology) โ€” claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 128) ``` .gitignore api/classify_crisis.py api/get_aggregate.py api/orchestrate.py api/push_classification_db.py api/trigger_call_for_location.py backend/build.sh backend/cleanup_test_data.py backend/config.py backend/database_schema.sql backend/database.py backend/deploy-cli.sh backend/deploy.sh backend/main.py backend/models.py backend/nixpacks-simple.toml backend/nixpacks.toml backend/Procfile backend/RAILWAY_DEPLOYMENT.md backend/railway.toml backend/requirements.txt backend/routes/classify_crisis.py backend/routes/crisis_map.py backend/routes/get_aggregate.py backend/routes/orchestrate.py backend/routes/process.py backend/routes/push_classification_db.py backend/routes/red_zone.py backend/routes/trigger_call_for_location.py backend/routes/users.py backend/runtime.txt backend/setup_supabase.py backend/test_crisis_data.py backend/voice_agent/analyze_transcript.py backend/voice_agent/emergency_test.py backend/voice_agent/post_call_analysis.py backend/voice_agent/trigger_call.py backend/voice_agent/workflow.py data/create_test_data.py frontend/mock-twitter/index.html frontend/mock-twitter/script.js frontend/mock-twitter/server.py frontend/mock-twitter/setup_table.py frontend/mock-twitter/styles.css frontend/mock-twitter/test_integration.py frontend/webapp/app/auth/page.tsx frontend/webapp/app/dashboard/page.tsx frontend/webapp/app/globals.css frontend/webapp/app/layout.tsx frontend/webapp/app/onboarding/page.tsx frontend/webapp/app/page.tsx frontend/webapp/components.json frontend/webapp/components/CrisisMap.tsx frontend/webapp/components/theme-provider.tsx frontend/webapp/components/ui/accordion.tsx frontend/webapp/components/ui/alert-dialog.tsx frontend/webapp/components/ui/alert.tsx frontend/webapp/components/ui/aspect-ratio.tsx frontend/webapp/components/ui/avatar.tsx frontend/webapp/components/ui/badge.tsx frontend/webapp/components/ui/breadcrumb.tsx frontend/webapp/components/ui/button.tsx frontend/webapp/components/ui/calendar.tsx frontend/webapp/components/ui/card.tsx frontend/webapp/components/ui/carousel.tsx frontend/webapp/components/ui/chart.tsx frontend/webapp/components/ui/checkbox.tsx frontend/webapp/components/ui/collapsible.tsx frontend/webapp/components/ui/command.tsx frontend/webapp/components/ui/context-menu.tsx frontend/webapp/components/ui/dialog.tsx frontend/webapp/components/ui/drawer.tsx frontend/webapp/components/ui/dropdown-menu.tsx frontend/webapp/components/ui/form.tsx frontend/webapp/components/ui/hover-card.tsx frontend/webapp/components/ui/input-otp.tsx frontend/webapp/components/ui/input.tsx frontend/webapp/components/ui/label.tsx frontend/webapp/components/ui/menubar.tsx frontend/webapp/components/ui/navigation-menu.tsx frontend/webapp/components/ui/pagination.tsx frontend/webapp/components/ui/popover.tsx frontend/webapp/components/ui/progress.tsx frontend/webapp/components/ui/radio-group.tsx frontend/webapp/components/ui/resizable.tsx frontend/webapp/components/ui/scroll-area.tsx frontend/webapp/components/ui/select.tsx frontend/webapp/components/ui/separator.tsx frontend/webapp/components/ui/sheet.tsx frontend/webapp/components/ui/sidebar.tsx frontend/webapp/components/ui/skeleton.tsx frontend/webapp/components/ui/slider.tsx frontend/webapp/components/ui/sonner.tsx frontend/webapp/components/ui/switch.tsx frontend/webapp/components/ui/table.tsx frontend/webapp/components/ui/tabs.tsx frontend/webapp/components/ui/textarea.tsx frontend/webapp/components/ui/toast.tsx frontend/webapp/components/ui/toaster.tsx frontend/webapp/components/ui/toggle-group.tsx frontend/webapp/components/ui/toggle.tsx frontend/webapp/components/ui/tooltip.tsx frontend/webapp/components/ui/use-mobile.tsx frontend/webapp/components/ui/use-toast.ts frontend/webapp/hooks/use-mobile.tsx frontend/webapp/hooks/use-toast.ts frontend/webapp/next-env.d.ts frontend/webapp/next.config.mjs frontend/webapp/package.json frontend/webapp/postcss.config.mjs frontend/webapp/README.md frontend/webapp/styles/globals.css frontend/webapp/tailwind.config.ts frontend/webapp/tsconfig.json model/classification.py model/compare_classification_conf.py model/compare_conf_error.py model/download_and_classify.py model/extract_sample.py model/model_finetuning.ipynb [8 more files omitted for size] ``` ### Dependencies - backend/requirements.txt: fastapi, google-generativeai@==0.8.3, Pillow, pydantic-settings, python-dotenv, requests, supabase, uvicorn[standard], vapi_server_sdk - frontend/webapp/package.json: @hookform/resolvers@^3.9.1, @radix-ui/react-accordion@1.2.2, @radix-ui/react-alert-dialog@1.1.4, @radix-ui/react-aspect-ratio@1.1.1, @radix-ui/react-avatar@1.1.2, @radix-ui/react-checkbox@1.1.3, @radix-ui/react-collapsible@1.1.2, @radix-ui/react-context-menu@2.2.4, @radix-ui/react-dialog@1.1.4, @radix-ui/react-dropdown-menu@2.1.4, @radix-ui/react-hover-card@1.1.4, @radix-ui/react-label@2.1.1, @radix-ui/react-menubar@1.1.4, @radix-ui/react-navigation-menu@1.2.3, @radix-ui/react-popover@1.1.4, @radix-ui/react-progress@1.1.1, @radix-ui/react-radio-group@1.2.2, @radix-ui/react-scroll-area@1.2.2, @radix-ui/react-select@2.1.4, @radix-ui/react-separator@1.1.1, @radix-ui/react-slider@1.2.2, @radix-ui/react-slot@1.1.1, @radix-ui/react-switch@1.1.2, @radix-ui/react-tabs@1.1.2, @radix-ui/react-toast@1.2.4, @radix-ui/react-toggle@1.1.1, @radix-ui/react-toggle-group@1.1.1, @radix-ui/react-tooltip@1.1.6, @types/node@^22, @types/react@^19, @types/react-dom@^19, autoprefixer@^10.4.20, class-variance-authority@^0.7.1, clsx@^2.1.1, cmdk@1.0.4, date-fns@4.1.0, embla-carousel-react@8.5.1, input-otp@1.4.1, lucide-react@^0.454.0, next@15.2.4, next-themes@^0.4.4, postcss@^8.5, react@^19, react-day-picker@8.10.1, react-dom@^19, react-hook-form@^7.54.1, react-resizable-panels@^2.1.7, recharts@2.15.0, sonner@^1.7.1, tailwind-merge@^2.5.5, tailwindcss@^3.4.17, tailwindcss-animate@^1.0.7, typescript@^5, vaul@^0.9.6, zod@^3.24.1 - requirements.txt: flask, google-generativeai@==0.8.3, Pillow, python-dotenv, requests@==2.31.0, supabase, vapi_server_sdk ### Recent commits (newest first) - Update README.md - Update README.md - Add Supabase config endpoint for mock-twitter integration - meoww - sms meow - sms - deploy - updated api shi - updated api shi - final deployment - meow - Add comprehensive logging for emergency contacts and SMS - Enhanced debugging for emergency contact lookup and SMS sending to identify why messages aren't being sent - Replace vapi-python with REST API implementation - Removes pyaudio dependency issues by using direct HTTP requests to Vapi API - Fix voice agent imports and add VAPI dependencies - Added vapi-python package, fixed import paths, enhanced error reporting for missing VAPI_API_KEY - Merge branch 'main' of https://github.com/aditii-jain/calhacks - Fix orchestrate workflow - Use direct function calls instead of HTTP requests to avoid timeouts, make voice agent optional with simulation mode - Refactor transcript analysis - Enhanced shelter extraction to include emergency contact intent and updated API integration for SMS notifications. - Fix import errors - Added get_supabase_client function to database.py and fixed logger import order - Convert all API endpoints from Vercel to Railway - Added FastAPI routes for classify-crisis, push-classification-db, get-aggregate, and trigger-call-for-location endpoints - Add orchestrate endpoint to Railway backend ## Key source files (fetched from GitHub, selected and truncated for size) ### backend/RAILWAY_DEPLOYMENT.md ```markdown # Railway Deployment Guide This guide will help you deploy the Crisis-MMD Backend to Railway. ## Prerequisites 1. **Railway Account**: Sign up at [railway.app](https://railway.app) 2. **GitHub Account**: Your code should be in a GitHub repository 3. **Supabase Project**: You'll need your Supabase credentials ## Required Environment Variables Set these in your Railway project settings: ### Required Variables ``` SUPABASE_URL=https://your-project.supabase.co SUPABASE_ANON_KEY=your-anon-key SUPABASE_SERVICE_KEY=your-service-role-key-here USE_SUPABASE=true DEBUG=false ``` ### Optional Variables ``` VAPI_API_KEY=your-vapi-api-key-here CLAUDE_API_KEY=your-claude-api-key-here GEMINI_API_KEY=your-gemini-api-key-here SERIOUSNESS_THRESHOLD=0.7 MAX_BATCH_SIZE=1000 ``` ## Deployment Steps ### Method 1: Deploy from GitHub (Recommended) 1. **Push your code to GitHub** ```bash git add . git commit -m "Prepare for Railway deployment" git push origin main ``` 2. **Create Railway Project** - Go to [railway.app](https://railway.app) - Click "New Project" - Select "Deploy from GitHub repo" - Choose your repository - Select the `backend` folder as the root directory 3. **Configure Environment Variables** - In Railway dashboard, go to your project - Click on your service - Go to "Variables" tab - Add all the required environment variables listed above 4. **Deploy** - Railway will automatically detect the Python app and deploy it - The deployment will use the `Procfile` and `nixpacks.toml` we created ### Method 2: Deploy from CLI 1. **Install Railway CLI** ```bash npm install -g @railway/cli # or curl -fsSL https://railway.app/install.sh | sh ``` 2. **Login to Railway** ```bash railway login ``` 3. **Initialize and Deploy** ```bash cd backend railway init railway up ``` 4. **Set Environment Variables** ```bash railway variables set SUPABASE_URL=https://your-project.supabase.co railway variables set SUPABASE_ANON_KEY=your-anon-key railway variables set SUPABASE_SERVICE_KEY=your-service-key railway variables set USE_SUPABASE=true railway variables set DEBUG=false # Add other variables as needed ``` ## Configuration Files Explained - **`Procfile`**: Tells Railway how to start your app - **`nixpacks.toml`**: Specifies Python version and build commands - **`railway.json`**: Railway-specific configuration - **`requirements.txt`**: Python dependencies ## Post-Deployment Steps 1. **Get your Railway URL**: Railway will provide a public URL like `https://your-app.railway.app` 2. **Test the deployment**: ```bash curl https://your-app.railway.app/health ``` 3. **Update your frontend**: Update your frontend to use the Railway URL instead of `localhost:8000` 4. **Monitor logs**: Check Railway dashboard for deployment logs and any errors ## Troubleshooting ### Common Issues 1. **"pip: command not found" Error** If you get this error du [truncated โ€” 1299 more characters] ``` ### requirements.txt ``` google-generativeai==0.8.3 requests==2.31.0 Pillow supabase vapi_server_sdk python-dotenv flask ``` ### backend/requirements.txt ``` fastapi uvicorn[standard] python-dotenv vapi_server_sdk supabase pydantic-settings requests google-generativeai==0.8.3 Pillow ``` ### frontend/webapp/package.json ``` { "name": "my-v0-project", "version": "0.1.0", "private": true, "scripts": { "build": "next build", "dev": "next dev", "lint": "next lint", "start": "next start" }, "dependencies": { "@hookform/resolvers": "^3.9.1", "@radix-ui/react-accordion": "1.2.2", "@radix-ui/react-alert-dialog": "1.1.4", "@radix-ui/react-aspect-ratio": "1.1.1", "@radix-ui/react-avatar": "1.1.2", "@radix-ui/react-checkbox": "1.1.3", "@radix-ui/react-collapsible": "1.1.2", "@radix-ui/react-context-menu": "2.2.4", "@radix-ui/react-dialog": "1.1.4", "@radix-ui/react-dropdown-menu": "2.1.4", "@radix-ui/react-hover-card": "1.1.4", "@radix-ui/react-label": "2.1.1", "@radix-ui/react-menubar": "1.1.4", "@radix-ui/react-navigation-menu": "1.2.3", "@radix-ui/react-popover": "1.1.4", "@radix-ui/react-progress": "1.1.1", "@radix-ui/react-radio-group": "1.2.2", "@radix-ui/react-scroll-area": "1.2.2", "@radix-ui/react-select": "2.1.4", "@radix-ui/react-separator": "1.1.1", "@radix-ui/react-slider": "1.2.2", "@radix-ui/react-slot": "1.1.1", "@radix-ui/react-switch": "1.1.2", "@radix-ui/react-tabs": "1.1.2", "@radix-ui/react-toast": "1.2.4", "@radix-ui/react-toggle": "1.1.1", "@radix-ui/react-toggle-group": "1.1.1", "@radix-ui/react-tooltip": "1.1.6", "autoprefixer": "^10.4.20", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "cmdk": "1.0.4", "date-fns": "4.1.0", "embla-carousel-react": "8.5.1", "input-otp": "1.4.1", "lucide-react": "^0.454.0", "next": "15.2.4", "next-themes": "^0.4.4", "react": "^19", "react-day-picker": "8.10.1", "react-dom": "^19", "react-hook-form": "^7.54.1", "react-resizable-panels": "^2.1.7", "recharts": "2.15.0", "sonner": "^1.7.1", "tailwind-merge": "^2.5.5", "tailwindcss-animate": "^1.0.7", "vaul": "^0.9.6", "zod": "^3.24.1" }, "devDependencies": { "@types/node": "^22", "@types/react": "^19", "@types/react-dom": "^19", "postcss": "^8.5", "tailwindcss": "^3.4.17", "typescript": "^5" } } ``` ### backend/main.py ```python from fastapi import FastAPI, HTTPException from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import JSONResponse import uvicorn import logging from contextlib import asynccontextmanager # Configure logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) # Global variable to store loaded models models = {} @asynccontextmanager async def lifespan(app: FastAPI): """Handle startup and shutdown events""" # Startup import os logger.info("๐Ÿš€ Crisis-MMD Backend starting up...") logger.info(f"๐Ÿ” PORT env var: {os.environ.get('PORT', 'NOT SET')}") logger.info(f"๐Ÿ” SUPABASE_URL: {os.environ.get('SUPABASE_URL', 'NOT SET')[:50]}...") logger.info(f"๐Ÿ” USE_SUPABASE: {os.environ.get('USE_SUPABASE', 'NOT SET')}") logger.info("๐Ÿ“Š Initializing classified data storage and retrieval system...") logger.info("๐Ÿ‘ฅ Initializing user authentication and management system...") # TODO: Initialize ML models here in future phases # models["text_classifier"] = load_text_model() # models["image_classifier"] = load_image_model() # models["multimodal_fusion"] = load_fusion_model() logger.info("โœ… Startup complete - Crisis-MMD system ready") yield # Shutdown logger.info("๐Ÿ›‘ Crisis-MMD Backend shutting down...") models.clear() logger.info("โœ… Cleanup complete") # Create FastAPI app with lifespan events app = FastAPI( title="Crisis-MMD API", description="Multimodal Disaster Analysis - Store and retrieve classified tweet data with user authentication for crisis alert system", version="2.0.0", docs_url="/docs", redoc_url="/redoc", lifespan=lifespan ) # Configure CORS app.add_middleware( CORSMiddleware, allow_origins=["*"], # Configure more restrictively in production allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # Global exception handler @app.exception_handler(HTTPException) async def http_exception_handler(request, exc): return JSONResponse( status_code=exc.status_code, content={ "error": exc.detail, "status_code": exc.status_code, "path": str(request.url) } ) @app.exception_handler(Exception) async def general_exception_handler(request, exc): logger.error(f"Unexpected error: {exc}") return JSONResponse( status_code=500, content={ "error": "Internal server error", "status_code": 500, "path": str(request.url) } ) # Basic health check endpoint @app.get("/") async def root(): """Root endpoint - API status""" import os return { "message": "Crisis-MMD API is running", "status": "healthy", "version": "2.0.0", "description": "Multimodal Disaster Analysis with user authentication and crisis alert system", "debug_info": { "port": os.environ.get("PORT", "NOT SET"), "host": "0.0.0.0", "environment": "production" if not os.environ.get("DEBUG", "true").lower() == "true" else "development" } } @app.get("/debug") async def debug_info(): """Debug endpoint to check environment""" import os return { "environment_variables": { "PORT": os.environ.get("PORT", "NOT SET"), "DEBUG": os.environ.get("DEBUG", "NOT SET"), "USE_SUPABASE": os.environ.get("USE_SUPABASE", "NOT SET"), "SUPABASE_URL_SET": bool(os.environ.get("SUPABASE_URL")), "VAPI_API_KEY_SET": bool(os.environ.get("VAPI_API_KEY")) }, "server_info": { "uvicorn_accessible": True, "fastapi_running": True, "timestamp": "2024-01-01" } } @app.get("/health") async def health_check(): """Health check endpoint""" import os return { "status": "healthy", "models_loaded": len(models), "port": os.environ.get("PORT", "NOT SET"), "supabase_configured": bool(os.environ.get("SUPABASE_URL")), "use_supabase": os.environ.get("USE_SUPABASE", "NOT SET"), "environment_check": "OK", "available_endpoints": [ "/", "/health", "/docs", "/redoc", # Data processing endpoints "/api/v1/classified-data/store", "/api/v1/classified-data/all", "/api/v1/classified-data/filter", "/api/v1/health", # User authentication endpoints "/api/v1/users/auth/send-otp", "/api/v1/users/auth/verify-otp", "/api/v1/users/profile", "/api/v1/users/", "/api/v1/users/location-radius", # Red Zone emergency endpoints "/api/v1/red-zone/trigger", # Crisis Map endpoints "/api/v1/crisis-map/data", "/api/v1/crisis-map/summary", "/api/v1/crisis-map/disaster-types", "/api/v1/crisis-map/health", # Orchestrate endpoints "/api/v1/orchestrate", "/api/v1/orchestrate/health", # Classification endpoints "/api/v1/classify-crisis", "/api/v1/classify-crisis/health", "/api/v1/push-classification-db", "/api/v1/push-classification-db/health", # Aggregation endpoints "/api/v1/get-aggregate", "/api/v1/get-aggregate/health", # Emergency endpoints "/api/v1/trigger-call-for-location", "/api/v1/trigger-call-for-location/health", # Config endpoints "/api/v1/config/supabase" ] } @app.get("/api/v1/config/supabase") async def get_supabase_config(): """Get Supabase configuration for frontend applications""" import os return { "supabase_url": os.environ.get("SUPABASE_URL"), "supabase_anon_key": os.environ.get("SUPABASE_ANON_KEY") [truncated โ€” 1438 more characters] ``` ### frontend/mock-twitter/server.py ```python from http.server import HTTPServer, SimpleHTTPRequestHandler import json import os import sys from datetime import datetime # Simple fallback configuration for when backend config is not available class SimpleSettings: def __init__(self): self.supabase_url = os.environ.get("SUPABASE_URL") self.supabase_service_key = os.environ.get("SUPABASE_SERVICE_KEY") # Add the parent backend directory to Python path to import config and supabase backend_path = os.path.join(os.path.dirname(__file__), '..', '..', 'backend') sys.path.insert(0, backend_path) # Insert at beginning to prioritize our config try: # Try to import supabase first from supabase import create_client # Try to import our backend config try: import config settings = config.settings print("โœ… Backend config loaded successfully") except ImportError as backend_import_error: print(f"โš ๏ธ Backend config not available: {backend_import_error}") print(" Using simple environment-based config") settings = SimpleSettings() SUPABASE_AVAILABLE = True print("โœ… Supabase integration available") except ImportError as e: SUPABASE_AVAILABLE = False print(f"โš ๏ธ Supabase not available: {e}") print(" Install with: pip install supabase") print(" Falling back to JSON-only storage") settings = SimpleSettings() class PostsHandler(SimpleHTTPRequestHandler): def __init__(self, *args, **kwargs): # Initialize Supabase client if available self.supabase_client = None if SUPABASE_AVAILABLE and settings and hasattr(settings, 'supabase_url') and hasattr(settings, 'supabase_service_key'): if settings.supabase_url and settings.supabase_service_key: try: # Use minimal options to avoid compatibility issues self.supabase_client = create_client( settings.supabase_url, settings.supabase_service_key ) print("โœ… Supabase client initialized in server") except Exception as e: print(f"โš ๏ธ Failed to initialize Supabase client: {e}") self.supabase_client = None super().__init__(*args, **kwargs) def end_headers(self): # Add CORS headers self.send_header('Access-Control-Allow-Origin', '*') self.send_header('Access-Control-Allow-Methods', 'GET, POST, OPTIONS') self.send_header('Access-Control-Allow-Headers', 'Content-Type') SimpleHTTPRequestHandler.end_headers(self) def do_OPTIONS(self): # Handle preflight requests self.send_response(200) self.end_headers() def do_GET(self): if self.path == '/': # Serve the main HTML file self.path = '/index.html' return SimpleHTTPRequestHandler.do_GET(self) elif self.path == '/get-posts': # Fetch posts from database self._handle_get_posts() return else: return SimpleHTTPRequestHandler.do_GET(self) def _handle_get_posts(self): """Handle GET request for fetching posts from database""" try: # Try to get posts from Supabase first posts_from_db = self._get_posts_from_supabase() # If Supabase fails, fall back to local JSON if not posts_from_db: posts_from_db = self._get_posts_from_json() # Send response self.send_response(200) self.send_header('Content-type', 'application/json') self.end_headers() response = { 'status': 'success', 'posts': posts_from_db, 'total_count': len(posts_from_db), 'source': 'supabase' if self.supabase_client and posts_from_db else 'local_json' } self.wfile.write(json.dumps(response).encode()) except Exception as e: print(f"โŒ Error fetching posts: {e}") self.send_response(500) self.send_header('Content-type', 'application/json') self.end_headers() error_response = { 'status': 'error', 'message': 'Failed to fetch posts', 'posts': [], 'total_count': 0 } self.wfile.write(json.dumps(error_response).encode()) def _get_posts_from_supabase(self): """Fetch posts from Supabase database""" if not self.supabase_client: print("โš ๏ธ Supabase client not available for fetching posts") return [] try: # Fetch all posts ordered by timestamp (newest first) response = self.supabase_client.table('mock_twitter_posts').select('*').order('timestamp', desc=True).execute() if response.data: print(f"โœ… Fetched {len(response.data)} posts from Supabase") # Convert database format to frontend format posts = [] for db_post in response.data: post = { 'text': db_post.get('text', ''), 'image': db_post.get('image'), 'timestamp': db_post.get('timestamp', ''), 'location': None # Extract location from text if needed } posts.append(post) return posts else: print("๐Ÿ“ญ No posts found in Supabase") return [] except Exception as e: print(f"โŒ Failed to fetch from Supabase: {e}") return [] def _get_posts_from_json(self): """Fetch posts from local JSON file (fallback)""" try: p [truncated โ€” 6623 more characters] ``` ### frontend/webapp/app/layout.tsx ```typescript import type React from "react" import type { Metadata } from "next" import { Inter, Poppins } from "next/font/google" import "./globals.css" const inter = Inter({ subsets: ["latin"], variable: "--font-inter", }) const poppins = Poppins({ subsets: ["latin"], weight: ["400", "500", "600", "700"], variable: "--font-poppins", }) export const metadata: Metadata = { title: "sheltr - Crisis assistance that speaks up", description: "Get early warnings and voice guidance in emergencies. Smart alerts. Calm voices. Real help.", generator: 'v0.dev' } export default function RootLayout({ children, }: { children: React.ReactNode }) { return ( {children} ) } ``` ### frontend/webapp/app/page.tsx ```typescript import Link from "next/link" import { Button } from "@/components/ui/button" import { MapPin, Phone, Shield, Users } from "lucide-react" export default function LandingPage() { return (
{/* Header */}
{/* Hero Section */}

When crisis strikes,
we speak up.

Get early warnings and voice guidance in emergencies.

{/* Mini Crisis Map Preview */}

Real-time Crisis Monitoring

{/* Simulated map with crisis zones */}
{/* Features Section */}

Crisis assistance that speaks up

Smart alerts. Calm voices. Real help.

Early Detection

AI-powered monitoring detects emergencies before they escalate

Voice Guidance

Receive calm, clear instructions through personalized voice calls

Emergency Contacts

Automatically notify your loved ones when you're in danger

{/* Footer */}
) } ``` ### frontend/webapp/app/auth/page.tsx ```typescript "use client" import type React from "react" import { useState } from "react" import { Button } from "@/components/ui/button" import { Input } from "@/components/ui/input" import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card" import { ArrowLeft, Phone } from "lucide-react" import Link from "next/link" export default function AuthPage() { const [step, setStep] = useState<"phone" | "otp">("phone") const [phoneNumber, setPhoneNumber] = useState("") const [otp, setOtp] = useState("") const [isLoading, setIsLoading] = useState(false) const handlePhoneSubmit = async (e: React.FormEvent) => { e.preventDefault() setIsLoading(true) // Simulate API call await new Promise((resolve) => setTimeout(resolve, 1000)) setStep("otp") setIsLoading(false) } const handleOtpSubmit = async (e: React.FormEvent) => { e.preventDefault() setIsLoading(true) // Simulate API call await new Promise((resolve) => setTimeout(resolve, 1000)) window.location.href = "/onboarding" } return (
Back to home
sheltr
{step === "phone" ? "Get Started" : "Verify Your Number"} {step === "phone" ? "We'll use this to send you voice alerts only during a verified emergency." : `We sent a code to ${phoneNumber}`} {step === "phone" ? (
setPhoneNumber(e.target.value)} className="pl-10 border-[#4B5D67]/20 focus:border-[#4B5D67] focus:ring-[#4B5D67]" required />
) : (
setOtp(e.target.value)} className="text-center text-2xl tracking-widest border-[#4B5D67]/20 focus:border-[#4B5D67] focus:ring-[#4B5D67]" maxLength={6} required />
)}
) } ``` ### frontend/webapp/app/onboarding/page.tsx ```typescript "use client" import type React from "react" import { useState } from "react" import { Button } from "@/components/ui/button" import { Input } from "@/components/ui/input" import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card" import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select" import { MapPin, Users, Check } from "lucide-react" export default function OnboardingPage() { const [alertRadius, setAlertRadius] = useState("") const [contacts, setContacts] = useState([ { name: "", phone: "" }, { name: "", phone: "" }, { name: "", phone: "" }, ]) const [isLoading, setIsLoading] = useState(false) const updateContact = (index: number, field: "name" | "phone", value: string) => { const newContacts = [...contacts] newContacts[index][field] = value setContacts(newContacts) } const handleSubmit = async (e: React.FormEvent) => { e.preventDefault() setIsLoading(true) // Simulate API call await new Promise((resolve) => setTimeout(resolve, 1500)) window.location.href = "/dashboard" } const isValidPhone = (phone: string) => { return phone.length >= 10 } return (
sheltr

Let's set up your emergency preferences

{/* Location Preferences */} Location Preferences Choose how wide an area you want to monitor for emergencies {/* Emergency Contacts */} Emergency Contacts Who should we notify if you're in a dangerous zone? {contacts.map((contact, index) => (
updateContact(index, "name", e.target.value)} className="border-[#4B5D67]/20 focus:border-[#4B5D67] focus:ring-[#4B5D67]" />
updateContact(index, "phone", e.target.value)} className="border-[#4B5D67]/20 focus:border-[#4B5D67] focus:ring-[#4B5D67] pr-10" /> {isValidPhone(contact.phone) && ( )}
))}
) } ``` [101 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]