# Project export: VetAI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Hands-free, multimodal AI diagnostic assistant for veterinarians VetAI enables veterinarians to diagnose animals through voice and vision while keeping their hands on the patient. - Devpost: https://devpost.com/software/vetai - GitHub: https://github.com/janprasad/HealthDetection - Video: https://www.youtube.com/embed/GdgnwVsUHIY?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Zini Chakraborty (20 commits), Claude Opus 4.6 (6 commits), Janani (6 commits), atrayav (6 commits) ## Devpost submission (written by the team) ### Inspiration As pet owners ourselves and from talking to other pet owners, we realised that pet owners often don’t know whether a symptom is minor or life-threatening. They also find it hard to connect with other pet owners who have gone through a similar issue. We kept thinking there had to be a better in-between, something that helps people make sense of what they’re seeing before panic sets in. VetAI came from wanting to reduce that anxiety and give both pet owners and professionals information that actually feels helpful and grounded. ### What it does VetAI is a veterinary diagnostic assistant mobile app that uses AI to help both veterinary professionals and pet owners analyze pet health conditions through images and voice. Users can capture a photo of a pet condition and receive a diagnosis with confidence scores, risk levels, and extracted visual indicators like asymmetry, border irregularity, color variation, and diameter. The app supports both Professional Mode (clinical terminology, research citations, detailed protocols) and Consumer Mode (plain English explanations, urgency levels, cost estimates, and emergency auto-detection). It includes hands-free voice interaction, allowing users to speak naturally while the AI responds with text-to-speech. If confidence is low, VetAI asks targeted follow-up questions and provides peer-reviewed veterinary research citations in real time. It also includes community features where pet owners can connect with others facing similar conditions. ### How we built it React Native with Expo and TypeScript (Mobile App) Python backend (FastAPI) Anthropic's Claude Sonnet 4 model (image analysis and conversational reasoning) OpenAI’s Whisper (voice integration, speech-to-text) Perplexity Sonar Pro (fetch peer-reviewed veterinary research citations in real time) The app includes camera capture, real-time analysis screens, confidence visualizations, animated UI components, and tab-based navigation. ### Challenges we ran into One of the biggest hurdles we ran into was the lack of accessible, reliable veterinary data for our specific use case. We initially assumed there would be usable public datasets available, but most of what we found was either too limited, poorly structured, or not relevant to what we were building. So we had to create our own dataset by sourcing images of pet diseases online. We also reached out directly to pet owners to understand their experiences, symptoms they observed, and the outcomes their pets had. It was a slower, more hands-on process than we expected, but it helped us collect more realistic and nuanced data. ### Accomplishments we're proud of VetAI is more than just an image classifier. It feels like a real, multi-modal AI assistant. It brings together vision, voice, research citations, tool use, and even community support into one smooth experience. The dual-mode setup makes sure vets get clinical depth while pet owners get clear, understandable guidance that actually helps in the moment. We also built in smart follow-up questions, which makes the results feel more thoughtful and reliable. And on top of that, we shipped a fully working mobile app with polished UI, live confidence visuals, emergency detection, and multi-turn conversation memory all within a hackathon timeframe. ### What's next We're looking for pilot partnerships with vet clinics and funding to launch our public beta. VetAI makes pet healthcare accessible, transparent, and connected. ## README (from the GitHub repository) # VetAI **Hands-free, multimodal AI diagnostic assistant for veterinarians and pet owners** VetAI enables veterinarians to diagnose animals through voice and vision while keeping their hands on the patient — and gives pet owners plain-English explanations with cost estimates. Built for TreeHacks 2026. --- ## The Problem Veterinarians examine 20-30+ animals daily in high-pressure, hands-on environments. Current diagnostic tools require: - Stopping the examination to type queries - Navigating complex databases with dirty hands - Separate workflows for image analysis - No conversational back-and-forth with AI Pet owners face a different problem: they get technical diagnoses they can't understand, don't know if it's urgent, and have no way to connect with others going through the same thing. **VetAI solves both sides.** --- ## Our Solution VetAI is a **voice-first, vision-enabled** diagnostic assistant with two modes: ### For Veterinarians (Professional Mode) - **Voice Queries** - Ask questions hands-free via Whisper STT, get spoken responses via OpenAI TTS - **Image Analysis** - Snap photos mid-exam for AI-powered visual diagnosis with Claude Sonnet - **Agentic Reasoning** - Claude searches veterinary databases and builds differential diagnoses with tool use - **Evidence-Based Research** - Perplexity Sonar retrieves peer-reviewed citations for every diagnosis - **Clarifying Questions** - When confidence is low (<85%), the AI generates targeted follow-up questions to refine the diagnosis ### For Pet Owners (Consumer Mode) - **Plain English Diagnoses** - Technical terms converted to simple language ("atopic dermatitis" -> "allergies causing skin irritation") - **Urgency Assessment** - Clear guidance: emergency, high confidence, moderate, or low confidence - **Cost Estimates** - Estimated vet visit cost ranges (routine $80-250, specialist $200-800, emergency $500-2000) - **Emergency Detection** - Automatic flagging of conditions like bloat, seizures, poisoning - **Community Support** - Find other pet owners with similar conditions nearby - **Activity Feed** - See community engagement, trending conditions, and success stories --- ## Demo Flow ``` Vet examining dog with skin rash: 1. Press mic: "What causes red patches on dog abdomens?" 2. Snap photo of the affected area 3. AI responds (voice): "Based on the image, this appears to be atopic dermatitis with 65% confidence..." 4. AI shows clarifying questions: - "Is the rash seasonal or year-round?" - "Are the paws and face also affected?" - "Did symptoms start before age 3?" 5. Vet answers via voice: "Yes, it's seasonal and the paws are red too" 6. AI refines: "With seasonal presentation and paw involvement, atopic dermatitis is confirmed. Recommending allergy testing..." Pet owner at home: 1. Takes photo of their dog's rash 2. Gets: "Allergies Causing Skin Irritation" (not "Atopic Dermatitis") 3. Urgency: "This is a likely diagnosis, but a vet should confirm." 4. Cost estimate: $80-$250 (routine visit) 5. Finds 3 other dog owners with allergies in their state ``` --- ## Features ### Implemented - **Voice Conversation**: OpenAI Whisper (STT) + TTS for hands-free interaction - **Vision Analysis**: Claude Sonnet for multimodal image diagnosis - **Agentic Backend**: Tool-calling architecture with disease database search, treatment protocols, and differential diagnosis - **Research Citations**: Perplexity Sonar API retrieves peer-reviewed veterinary literature with inline citations - **Clarifying Questions**: When diagnosis confidence < 85%, generates 2-3 targeted yes/no questions to help narrow down the diagnosis - **Consumer Mode**: Plain English diagnoses with urgency levels, cost estimates, and emergency detection for pet owners - **Community Support Groups**: Find relevant support groups by condition and species with fuzzy matching - **Pet Profile Matching**: Connect pet owners with similar conditions in the same area (same species, overlapping conditions, same state) - **Activity Feed**: Community engagement metrics, trending conditions, and statistics - **Multi-Turn Voice Sessions**: Clarifying question answers carry context across follow-up voice interactions - **Mobile App**: Cross-platform React Native (Expo) with camera integration - **Structured Outputs**: Confidence scores, risk levels, recommendations, and cited sources ### In Progress - **Frontend UI for Community Features**: Screens for profiles, matching, groups, and activity feed - **Consumer Mode Toggle**: UI switch between professional and consumer views - **Fine-Tuned VLM**: Custom vision model trained on veterinary imagery - **Multi-Image Context**: Accumulate photos across conversation for progressive diagnosis --- ## Tech Stack | Layer | Technology | |-------|-----------| | **Frontend** | React Native (Expo), TypeScript, expo-av, expo-image-picker | | **Backend** | FastAPI (Python 3.11+), Pydantic v2 | | **Vision AI** | Anthropic Claude Sonnet (claude-sonnet-4-20250514) | | **Speech** | OpenAI Whisper (STT) + OpenAI TTS (tts-1, alloy voice) | | **Research** | Perplexity Sonar Pro API | | **Tunneling** | ngrok (for Expo Go device testing) | --- ## Architecture ``` +---------------------+ | Mobile App | | React Native/Expo | | - Voice Input | | - Camera Capture | | - Results Display | +---------+-----------+ | +---------v-----------+ | FastAPI Backend | | | | POST /analyze | <- Image -> diagnosis (professional or consumer) | POST /chat | <- Agentic chat with tools | POST /voice/query | <- Audio + optional image | GET /voice/audio | <- Cached TTS playback | GET /health | <- Status check | | | Community & Feed | | POST /api/community/profile | GET /api/community/groups/{condition}/{species} | GET /api/community/matches/{id} | GET /api/activity/feed | GET /api/activity/stats | GET /api/activity/trending +---------+-----------+ | +-----------+-------+-------+--------------+ v v v v +-----------+ +--------+ +------------+ +----------+ | Claude | |Whisper | | Perplexity | | OpenAI | | Sonnet | | STT | | Sonar | | TTS | |(Vision+ | | | | (Research) | | (Voice) | | Agent) | | | | | | | +-----------+ +--------+ +------------+ +----------+ ``` ### Data Flow: Image Analysis ``` Photo -> /analyze -> Claude Vision -> Structured diagnosis JSON -> Perplexity Sonar -> Research citations -> Claude (if conf < 85%) -> Clarifying questions -> Consumer mode? -> Simplified response + cost estimate -> Combined AnalysisResult response ``` ### Data Flow: Voice Query ``` Audio -> /voice/query -> Whisper STT -> Transcribed text -> Claude Agent (with tool use) -> Text response -> OpenAI TTS -> Audio response -> Session tracking for multi-turn context ``` --- ## Setup ### Prerequisites - Node.js 18+ - Python 3.11+ - Expo CLI (`npm install -g expo-cli`) - API keys: Anthropic, OpenAI, Perplexity - ngrok account ([sign up](https://dashboard.ngrok.com/signup)) ### Backend ```bash cd backend # Inst [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 73 recognized source files, 418 KB. - Anthropic (technology) — detected in the code - FastAPI (technology) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (86 of 86) ``` .agents/skills/baseline-ui/SKILL.md .agents/skills/fixing-accessibility/SKILL.md .agents/skills/fixing-metadata/SKILL.md .agents/skills/fixing-motion-performance/SKILL.md .gitignore App.js app.json backend/.env.example backend/app.py backend/config.py backend/main.py backend/models/__init__.py backend/models/community.py backend/models/consumer_scheme.py backend/models/schemas.py backend/README.md backend/requirements.txt backend/routes/__init__.py backend/routes/activity_routes.py backend/routes/analyze.py backend/routes/chat.py backend/routes/community_routes.py backend/routes/voice_routes.py backend/services/__init__.py backend/services/activity_service.py backend/services/agent.py backend/services/community_service.py backend/services/consumer_mode.py backend/services/image_validation.py backend/services/research_service.py backend/services/vlm.py backend/services/voice_service.py HealthDetect/.env.example HealthDetect/.gitignore HealthDetect/.vscode/extensions.json HealthDetect/.vscode/settings.json HealthDetect/app.json HealthDetect/app/_layout.tsx HealthDetect/app/(tabs)/_layout.tsx HealthDetect/app/(tabs)/community.tsx HealthDetect/app/(tabs)/history.tsx HealthDetect/app/(tabs)/index.tsx HealthDetect/app/(tabs)/learn.tsx HealthDetect/app/(tabs)/profile.tsx HealthDetect/app/camera.tsx HealthDetect/app/group-chat.tsx HealthDetect/app/photo-review.tsx HealthDetect/app/processing.tsx HealthDetect/app/results.tsx HealthDetect/app/role-select.tsx HealthDetect/components/external-link.tsx HealthDetect/components/haptic-tab.tsx HealthDetect/components/hello-wave.tsx HealthDetect/components/parallax-scroll-view.tsx HealthDetect/components/themed-text.tsx HealthDetect/components/themed-view.tsx HealthDetect/components/ui/collapsible.tsx HealthDetect/components/ui/icon-symbol.ios.tsx HealthDetect/components/ui/icon-symbol.tsx HealthDetect/components/VoiceButton.tsx HealthDetect/constants/theme.ts HealthDetect/constants/types.ts HealthDetect/context/AnalysisContext.tsx HealthDetect/context/PetProfileContext.tsx HealthDetect/eslint.config.js HealthDetect/hooks/use-color-scheme.ts HealthDetect/hooks/use-color-scheme.web.ts HealthDetect/hooks/use-theme-color.ts HealthDetect/package.json HealthDetect/README.md HealthDetect/scripts/reset-project.js HealthDetect/services/analysis-service.ts HealthDetect/services/community-service.ts HealthDetect/services/voice-service.ts HealthDetect/tsconfig.json index.js package.json README.md src/components/Card.js src/components/MetricCard.js src/constants/theme.js src/screens/HealthTipsScreen.js src/screens/HomeScreen.js src/screens/ProfileScreen.js src/screens/SymptomsScreen.js tsconfig.json ``` ### Dependencies - backend/requirements.txt: anthropic@>=0.42.0, fastapi@>=0.115.0, openai@>=1.12.0, packaging@>=23.2,<26, Pillow@>=10.0.0, pydantic@>=2.10.0, pydub@>=0.25.1, python-dotenv@>=1.0.1, python-multipart@>=0.0.12, uvicorn@>=0.32.0 - HealthDetect/package.json: @expo/ngrok@^4.1.3, @expo/vector-icons@^15.0.3, @react-navigation/bottom-tabs@^7.4.0, @react-navigation/elements@^2.6.3, @react-navigation/native@^7.1.8, @types/node@^25.2.3, @types/react@~19.1.0, eslint@^9.25.0, eslint-config-expo@~10.0.0, expo@~54.0.33, expo-av@^16.0.8, expo-camera@~17.0.10, expo-constants@~18.0.13, expo-file-system@^19.0.21, expo-font@~14.0.11, expo-haptics@~15.0.8, expo-image@~3.0.11, expo-image-picker@~17.0.10, expo-linear-gradient@~15.0.8, expo-linking@~8.0.11, expo-router@~6.0.23, expo-splash-screen@~31.0.13, expo-status-bar@~3.0.9, expo-symbols@~1.0.8, expo-system-ui@~6.0.9, expo-web-browser@~15.0.10, react@19.1.0, react-dom@19.1.0, react-native@0.81.5, react-native-gesture-handler@~2.28.0, react-native-reanimated@~4.1.1, react-native-safe-area-context@~5.6.0, react-native-screens@~4.16.0, react-native-web@~0.21.0, react-native-worklets@0.5.1, typescript@~5.9.2 - package.json: @expo/vector-icons@^15.0.3, @react-navigation/bottom-tabs@^7.13.0, @react-navigation/native@^7.1.28, @types/react@~19.1.10, expo@~54.0.33, expo-status-bar@~3.0.9, react@19.1.0, react-native@0.81.5, react-native-safe-area-context@~5.6.0, react-native-screens@~4.16.0, typescript@~5.9.2 ### Recent commits (newest first) - vet info - split up vets and users - make it multiturn - more agentic - group chat - make links clickable - fix vet experience visual - require analysis - shorten audio - Merge pull request #8 from janprasad/feat/ui-improvements - rest of files - Merge pull request #7 from janprasad/feat/ui-improvements - consumer mode - community tab - profile info included - Add activity feed backend for community engagement metrics - Add community support features for connecting pet owners - index edits - audio pause + faster audio - create .env for ngrok ## Key source files (fetched from GitHub, selected and truncated for size) ### .agents/skills/baseline-ui/SKILL.md ```markdown --- name: baseline-ui description: Enforces an opinionated UI baseline to prevent AI-generated interface slop. --- # Baseline UI Enforces an opinionated UI baseline to prevent AI-generated interface slop. ## How to use - `/baseline-ui` Apply these constraints to any UI work in this conversation. - `/baseline-ui ` Review the file against all constraints below and output: - violations (quote the exact line/snippet) - why it matters (1 short sentence) - a concrete fix (code-level suggestion) ## Stack - MUST use Tailwind CSS defaults unless custom values already exist or are explicitly requested - MUST use `motion/react` (formerly `framer-motion`) when JavaScript animation is required - SHOULD use `tw-animate-css` for entrance and micro-animations in Tailwind CSS - MUST use `cn` utility (`clsx` + `tailwind-merge`) for class logic ## Components - MUST use accessible component primitives for anything with keyboard or focus behavior (`Base UI`, `React Aria`, `Radix`) - MUST use the project’s existing component primitives first - NEVER mix primitive systems within the same interaction surface - SHOULD prefer [`Base UI`](https://base-ui.com/react/components) for new primitives if compatible with the stack - MUST add an `aria-label` to icon-only buttons - NEVER rebuild keyboard or focus behavior by hand unless explicitly requested ## Interaction - MUST use an `AlertDialog` for destructive or irreversible actions - SHOULD use structural skeletons for loading states - NEVER use `h-screen`, use `h-dvh` - MUST respect `safe-area-inset` for fixed elements - MUST show errors next to where the action happens - NEVER block paste in `input` or `textarea` elements ## Animation - NEVER add animation unless it is explicitly requested - MUST animate only compositor props (`transform`, `opacity`) - NEVER animate layout properties (`width`, `height`, `top`, `left`, `margin`, `padding`) - SHOULD avoid animating paint properties (`background`, `color`) except for small, local UI (text, icons) - SHOULD use `ease-out` on entrance - NEVER exceed `200ms` for interaction feedback - MUST pause looping animations when off-screen - SHOULD respect `prefers-reduced-motion` - NEVER introduce custom easing curves unless explicitly requested - SHOULD avoid animating large images or full-screen surfaces ## Typography - MUST use `text-balance` for headings and `text-pretty` for body/paragraphs - MUST use `tabular-nums` for data - SHOULD use `truncate` or `line-clamp` for dense UI - NEVER modify `letter-spacing` (`tracking-*`) unless explicitly requested ## Layout - MUST use a fixed `z-index` scale (no arbitrary `z-*`) - SHOULD use `size-*` for square elements instead of `w-*` + `h-*` ## Performance - NEVER animate large `blur()` or `backdrop-filter` surfaces - NEVER apply `will-change` outside an active animation - NEVER use `useEffect` for anything that can be expressed as render logic ## Design - NEVER use gradients unless explicitly requested - NEV [truncated — 343 more characters] ``` ### .agents/skills/fixing-accessibility/SKILL.md ```markdown --- name: fixing-accessibility description: Fix accessibility issues. --- # fixing-accessibility Fix accessibility issues. ## how to use - `/fixing-accessibility` Apply these constraints to any UI work in this conversation. - `/fixing-accessibility ` Review the file against all rules below and report: - violations (quote the exact line or snippet) - why it matters (one short sentence) - a concrete fix (code-level suggestion) Do not rewrite large parts of the UI. Prefer minimal, targeted fixes. ## when to apply Reference these guidelines when: - adding or changing buttons, links, inputs, menus, dialogs, tabs, dropdowns - building forms, validation, error states, helper text - implementing keyboard shortcuts or custom interactions - working on focus states, focus trapping, or modal behavior - rendering icon-only controls - adding hover-only interactions or hidden content ## rule categories by priority | priority | category | impact | |----------|----------|--------| | 1 | accessible names | critical | | 2 | keyboard access | critical | | 3 | focus and dialogs | critical | | 4 | semantics | high | | 5 | forms and errors | high | | 6 | announcements | medium-high | | 7 | contrast and states | medium | | 8 | media and motion | low-medium | | 9 | tool boundaries | critical | ## quick reference ### 1. accessible names (critical) - every interactive control must have an accessible name - icon-only buttons must have aria-label or aria-labelledby - every input, select, and textarea must be labeled - links must have meaningful text (no “click here”) - decorative icons must be aria-hidden ### 2. keyboard access (critical) - do not use div or span as buttons without full keyboard support - all interactive elements must be reachable by Tab - focus must be visible for keyboard users - do not use tabindex greater than 0 - Escape must close dialogs or overlays when applicable ### 3. focus and dialogs (critical) - modals must trap focus while open - restore focus to the trigger on close - set initial focus inside dialogs - opening a dialog should not scroll the page unexpectedly ### 4. semantics (high) - prefer native elements (button, a, input) over role-based hacks - if a role is used, required aria attributes must be present - lists must use ul or ol with li - do not skip heading levels - tables must use th for headers when applicable ### 5. forms and errors (high) - errors must be linked to fields using aria-describedby - required fields must be announced - invalid fields must set aria-invalid - helper text must be associated with inputs - disabled submit actions must explain why ### 6. announcements (medium-high) - critical form errors should use aria-live - loading states should use aria-busy or status text - toasts must not be the only way to convey critical information - expandable controls must use aria-expanded and aria-controls ### 7. contrast and states (medium) - ensure sufficient contrast for text and icons - [truncated — 926 more characters] ``` ### package.json ``` { "name": "healthdetectiontemp", "version": "1.0.0", "main": "index.js", "scripts": { "start": "expo start", "android": "expo start --android", "ios": "expo start --ios", "web": "expo start --web" }, "dependencies": { "@expo/vector-icons": "^15.0.3", "@react-navigation/bottom-tabs": "^7.13.0", "@react-navigation/native": "^7.1.28", "expo": "~54.0.33", "expo-status-bar": "~3.0.9", "react": "19.1.0", "react-native": "0.81.5", "react-native-safe-area-context": "~5.6.0", "react-native-screens": "~4.16.0" }, "private": true, "devDependencies": { "@types/react": "~19.1.10", "typescript": "~5.9.2" } } ``` ### backend/requirements.txt ``` fastapi>=0.115.0 uvicorn>=0.32.0 anthropic>=0.42.0 python-multipart>=0.0.12 pydantic>=2.10.0 python-dotenv>=1.0.1 openai>=1.12.0 pydub>=0.25.1 Pillow>=10.0.0 packaging>=23.2,<26 ``` ### HealthDetect/package.json ``` { "name": "healthdetect", "main": "expo-router/entry", "version": "1.0.0", "scripts": { "start": "expo start", "reset-project": "node ./scripts/reset-project.js", "android": "expo start --android", "ios": "expo start --ios", "web": "expo start --web", "lint": "expo lint" }, "dependencies": { "@expo/ngrok": "^4.1.3", "@expo/vector-icons": "^15.0.3", "@react-navigation/bottom-tabs": "^7.4.0", "@react-navigation/elements": "^2.6.3", "@react-navigation/native": "^7.1.8", "expo": "~54.0.33", "expo-av": "^16.0.8", "expo-camera": "~17.0.10", "expo-constants": "~18.0.13", "expo-file-system": "^19.0.21", "expo-font": "~14.0.11", "expo-haptics": "~15.0.8", "expo-image": "~3.0.11", "expo-image-picker": "~17.0.10", "expo-linear-gradient": "~15.0.8", "expo-linking": "~8.0.11", "expo-router": "~6.0.23", "expo-splash-screen": "~31.0.13", "expo-status-bar": "~3.0.9", "expo-symbols": "~1.0.8", "expo-system-ui": "~6.0.9", "expo-web-browser": "~15.0.10", "react": "19.1.0", "react-dom": "19.1.0", "react-native": "0.81.5", "react-native-gesture-handler": "~2.28.0", "react-native-reanimated": "~4.1.1", "react-native-safe-area-context": "~5.6.0", "react-native-screens": "~4.16.0", "react-native-web": "~0.21.0", "react-native-worklets": "0.5.1" }, "devDependencies": { "@types/node": "^25.2.3", "@types/react": "~19.1.0", "eslint": "^9.25.0", "eslint-config-expo": "~10.0.0", "typescript": "~5.9.2" }, "private": true } ``` ### index.js ```javascript import { registerRootComponent } from 'expo'; import App from './App'; // registerRootComponent calls AppRegistry.registerComponent('main', () => App); // It also ensures that whether you load the app in Expo Go or in a native build, // the environment is set up appropriately registerRootComponent(App); ``` ### App.js ```javascript import React from 'react'; import { StatusBar } from 'expo-status-bar'; import { NavigationContainer } from '@react-navigation/native'; import { createBottomTabNavigator } from '@react-navigation/bottom-tabs'; import { Ionicons } from '@expo/vector-icons'; import { View, Platform } from 'react-native'; import HomeScreen from './src/screens/HomeScreen'; import SymptomsScreen from './src/screens/SymptomsScreen'; import HealthTipsScreen from './src/screens/HealthTipsScreen'; import ProfileScreen from './src/screens/ProfileScreen'; const Tab = createBottomTabNavigator(); const TAB_ICON = { Home: { focused: 'home', unfocused: 'home-outline' }, Symptoms: { focused: 'search', unfocused: 'search-outline' }, Tips: { focused: 'bulb', unfocused: 'bulb-outline' }, Profile: { focused: 'person', unfocused: 'person-outline' }, }; export default function App() { return ( ({ headerShown: false, tabBarIcon: ({ focused, color, size }) => { const iconName = focused ? TAB_ICON[route.name].focused : TAB_ICON[route.name].unfocused; return ; }, tabBarActiveTintColor: '#0EA5E9', tabBarInactiveTintColor: '#94A3B8', tabBarStyle: { backgroundColor: '#FFFFFF', borderTopWidth: 0, elevation: 20, shadowColor: '#0F172A', shadowOffset: { width: 0, height: -4 }, shadowOpacity: 0.06, shadowRadius: 12, height: Platform.OS === 'ios' ? 88 : 64, paddingBottom: Platform.OS === 'ios' ? 28 : 8, paddingTop: 8, }, tabBarLabelStyle: { fontSize: 11, fontWeight: '600', }, })} > ); } ``` ### backend/main.py ```python import os from dotenv import load_dotenv from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware from routes.analyze import router as analyze_router from routes.chat import router as chat_router from routes.voice_routes import router as voice_router from routes.community_routes import router as community_router from routes.activity_routes import router as activity_router load_dotenv() app = FastAPI(title="VetAI Backend", version="1.0.0") app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) app.include_router(analyze_router) app.include_router(chat_router) app.include_router(voice_router) app.include_router(community_router, prefix="/api", tags=["community"]) app.include_router(activity_router, prefix="/api", tags=["activity"]) @app.get("/health") async def health(): return { "status": "ok", "anthropic_key_set": bool(os.getenv("ANTHROPIC_API_KEY")), "openai_key_set": bool(os.getenv("OPENAI_API_KEY")), } ``` ### backend/app.py ```python """ PetHealth AI - Modal Backend Serves a fine-tuned pet disease classification model + Anthropic-powered agent. Deploy: modal deploy app.py Dev: modal serve app.py """ import modal import io import os import json import base64 import time from datetime import datetime from config import ( DISEASE_CLASSES, CLASS_NAMES, AGENT_SYSTEM_PROMPT, AGENT_TOOLS, BREED_HEALTH_INFO, DEFAULT_BREED_INFO, IMAGE_SIZE, IMAGE_MEAN, IMAGE_STD, CORS_ORIGINS, ANTHROPIC_MODEL, ) # ─── Modal App Setup ───────────────────────────────────────────────────────── app = modal.App("pethealth-api") # Container image with all dependencies image = ( modal.Image.debian_slim(python_version="3.11") .pip_install( "torch", "torchvision", "pillow", "fastapi[standard]", "python-multipart", "anthropic", ) ) # Volume for storing model weights # Upload your model: modal volume put pethealth-model-weights model.pth model_volume = modal.Volume.from_name("pethealth-model-weights", create_if_missing=True) # ─── FastAPI Application ───────────────────────────────────────────────────── @app.function( image=image, gpu="T4", volumes={"/model-weights": model_volume}, secrets=[modal.Secret.from_name("anthropic-api-key")], timeout=120, allow_concurrent_inputs=10, ) @modal.asgi_app() def fastapi_app(): """Mount the full FastAPI application on Modal.""" from fastapi import FastAPI, UploadFile, File, HTTPException from fastapi.middleware.cors import CORSMiddleware from pydantic import BaseModel from typing import Optional import torch import torchvision.transforms as transforms from PIL import Image import anthropic # ── FastAPI Setup ── api = FastAPI( title="PetHealth AI API", description="AI-powered pet skin condition analysis with agentic follow-up", version="1.0.0", ) api.add_middleware( CORSMiddleware, allow_origins=CORS_ORIGINS, allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # ── Model Loading ── model = None demo_mode = True model_path = "/model-weights/model.pth" if os.path.exists(model_path): try: model = torch.load(model_path, map_location="cuda" if torch.cuda.is_available() else "cpu") model.eval() demo_mode = False print(f"✓ Model loaded from {model_path}") except Exception as e: print(f"⚠ Failed to load model: {e}. Running in demo mode.") else: print("⚠ No model weights found. Running in demo mode.") print(" Upload weights: modal volume put pethealth-model-weights your_model.pth") # Image preprocessing pipeline (adjust to match your model's training) preprocess = transforms.Compose([ transforms.Resize((IMAGE_SIZE, IMAGE_SIZE)), transforms.ToTensor(), transforms.Normalize(mean=IMAGE_MEAN, std=IMAGE_STD), ]) # Anthropic client claude = anthropic.Anthropic() # ── Request/Response Models ── class AnalyzeRequest(BaseModel): image_base64: str pet_name: Optional[str] = None pet_breed: Optional[str] = None pet_age: Optional[str] = None pet_species: Optional[str] = "dog" class ChatRequest(BaseModel): messages: list[dict] # Conversation history analysis_context: Optional[dict] = None # Model results for context pet_profile: Optional[dict] = None class AnalyzeResponse(BaseModel): success: bool diagnosis: dict agent_message: str metadata: dict class ChatResponse(BaseModel): message: str tool_results: Optional[list] = None # ── Helper Functions ── def run_inference(image: Image.Image) -> dict: """Run the model on a preprocessed image.""" if demo_mode: return demo_inference() device = "cuda" if torch.cuda.is_available() else "cpu" tensor = preprocess(image).unsqueeze(0).to(device) with torch.no_grad(): output = model(tensor) probabilities = torch.softmax(output, dim=1)[0] # Build results sorted by confidence results = [] for idx, prob in enumerate(probabilities): if idx in DISEASE_CLASSES: results.append({ **DISEASE_CLASSES[idx], "confidence": round(prob.item(), 4), }) results.sort(key=lambda x: x["confidence"], reverse=True) primary = results[0] alternatives = results[1:4] # Top 3 alternatives return { "primary": { "condition": primary["condition"], "common_name": primary["common_name"], "confidence": primary["confidence"], "risk_level": primary["risk_level"], }, "alternatives": [ {"condition": alt["condition"], "confidence": alt["confidence"]} for alt in alternatives ], "all_predictions": {r["condition"]: r["confidence"] for r in results}, } def demo_inference() -> dict: """Simulated inference for demo/testing when no model is loaded.""" import random # Weighted random selection (favor benign for demo) weights = [0.25, 0.15, 0.10, 0.15, 0.10, 0.08, 0.05, 0.04, 0.05, 0.03] idx = random.choices(range(len(weights)), weights=weights, k=1)[0] primary = DISEASE_CLASSES[idx] primary_conf = round(0.70 + random.random() * 0.25, 4) # Generate alternatives remaining = 1.0 - primary_conf alt_indices = [i for i in range(len(DISEASE_CLASSES)) if i != idx] random.shuffle(alt_indices) alt_indices = alt_indices[:3] raw_confs = [random.random() for _ in alt_indices] total = [truncated — 10259 more characters] ``` ### HealthDetect/app/_layout.tsx ```typescript import { DarkTheme, DefaultTheme, ThemeProvider } from '@react-navigation/native'; import { Stack, useRouter, useSegments } from 'expo-router'; import { StatusBar } from 'expo-status-bar'; import { useEffect } from 'react'; import 'react-native-reanimated'; import { useColorScheme } from '@/hooks/use-color-scheme'; import { AnalysisProvider } from '@/context/AnalysisContext'; import { PetProfileProvider, usePetProfile } from '@/context/PetProfileContext'; import { AppColors } from '@/constants/theme'; // Custom theme that matches our design system const LightTheme = { ...DefaultTheme, colors: { ...DefaultTheme.colors, primary: AppColors.primary.teal, background: AppColors.neutral.backgroundLight, card: AppColors.neutral.white, text: AppColors.neutral.textPrimary, border: AppColors.neutral.border, }, }; const DarkAppTheme = { ...DarkTheme, colors: { ...DarkTheme.colors, primary: AppColors.primary.tealLight, background: AppColors.neutral.backgroundDark, card: AppColors.neutral.surfaceDark, text: '#ECEDEE', border: AppColors.neutral.borderDark, }, }; export const unstable_settings = { anchor: '(tabs)', }; function InnerLayout() { const { petProfile, isLoaded } = usePetProfile(); const router = useRouter(); const segments = useSegments(); useEffect(() => { if (!isLoaded) return; const onRoleSelect = segments[0] === 'role-select'; if (!petProfile.hasChosenRole && !onRoleSelect) { router.replace('/role-select'); } else if (petProfile.hasChosenRole && onRoleSelect) { router.replace('/(tabs)'); } }, [isLoaded, petProfile.hasChosenRole, segments]); return ( <> ); } export default function RootLayout() { const colorScheme = useColorScheme(); return ( ); } ``` [58 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]