# Project export: VetAI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Hands-free, multimodal AI diagnostic assistant for veterinarians VetAI enables veterinarians to diagnose animals through voice and vision while keeping their hands on the patient. - Devpost: https://devpost.com/software/vetai - GitHub: https://github.com/janprasad/HealthDetection - Video: https://www.youtube.com/embed/GdgnwVsUHIY?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Zini Chakraborty (20 commits), Claude Opus 4.6 (6 commits), Janani (6 commits), atrayav (6 commits) ## Devpost submission (written by the team) ### Inspiration As pet owners ourselves and from talking to other pet owners, we realised that pet owners often don’t know whether a symptom is minor or life-threatening. They also find it hard to connect with other pet owners who have gone through a similar issue. We kept thinking there had to be a better in-between, something that helps people make sense of what they’re seeing before panic sets in. VetAI came from wanting to reduce that anxiety and give both pet owners and professionals information that actually feels helpful and grounded. ### What it does VetAI is a veterinary diagnostic assistant mobile app that uses AI to help both veterinary professionals and pet owners analyze pet health conditions through images and voice. Users can capture a photo of a pet condition and receive a diagnosis with confidence scores, risk levels, and extracted visual indicators like asymmetry, border irregularity, color variation, and diameter. The app supports both Professional Mode (clinical terminology, research citations, detailed protocols) and Consumer Mode (plain English explanations, urgency levels, cost estimates, and emergency auto-detection). It includes hands-free voice interaction, allowing users to speak naturally while the AI responds with text-to-speech. If confidence is low, VetAI asks targeted follow-up questions and provides peer-reviewed veterinary research citations in real time. It also includes community features where pet owners can connect with others facing similar conditions. ### How we built it React Native with Expo and TypeScript (Mobile App) Python backend (FastAPI) Anthropic's Claude Sonnet 4 model (image analysis and conversational reasoning) OpenAI’s Whisper (voice integration, speech-to-text) Perplexity Sonar Pro (fetch peer-reviewed veterinary research citations in real time) The app includes camera capture, real-time analysis screens, confidence visualizations, animated UI components, and tab-based navigation. ### Challenges we ran into One of the biggest hurdles we ran into was the lack of accessible, reliable veterinary data for our specific use case. We initially assumed there would be usable public datasets available, but most of what we found was either too limited, poorly structured, or not relevant to what we were building. So we had to create our own dataset by sourcing images of pet diseases online. We also reached out directly to pet owners to understand their experiences, symptoms they observed, and the outcomes their pets had. It was a slower, more hands-on process than we expected, but it helped us collect more realistic and nuanced data. ### Accomplishments we're proud of VetAI is more than just an image classifier. It feels like a real, multi-modal AI assistant. It brings together vision, voice, research citations, tool use, and even community support into one smooth experience. The dual-mode setup makes sure vets get clinical depth while pet owners get clear, understandable guidance that actually helps in the moment. We also built in smart follow-up questions, which makes the results feel more thoughtful and reliable. And on top of that, we shipped a fully working mobile app with polished UI, live confidence visuals, emergency detection, and multi-turn conversation memory all within a hackathon timeframe. ### What's next We're looking for pilot partnerships with vet clinics and funding to launch our public beta. VetAI makes pet healthcare accessible, transparent, and connected. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 73 recognized source files, 418 KB. - Anthropic (technology) — detected in the code - FastAPI (technology) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (86 of 86) ``` .agents/skills/baseline-ui/SKILL.md .agents/skills/fixing-accessibility/SKILL.md .agents/skills/fixing-metadata/SKILL.md .agents/skills/fixing-motion-performance/SKILL.md .gitignore App.js app.json backend/.env.example backend/app.py backend/config.py backend/main.py backend/models/__init__.py backend/models/community.py backend/models/consumer_scheme.py backend/models/schemas.py backend/README.md backend/requirements.txt backend/routes/__init__.py backend/routes/activity_routes.py backend/routes/analyze.py backend/routes/chat.py backend/routes/community_routes.py backend/routes/voice_routes.py backend/services/__init__.py backend/services/activity_service.py backend/services/agent.py backend/services/community_service.py backend/services/consumer_mode.py backend/services/image_validation.py backend/services/research_service.py backend/services/vlm.py backend/services/voice_service.py HealthDetect/.env.example HealthDetect/.gitignore HealthDetect/.vscode/extensions.json HealthDetect/.vscode/settings.json HealthDetect/app.json HealthDetect/app/_layout.tsx HealthDetect/app/(tabs)/_layout.tsx HealthDetect/app/(tabs)/community.tsx HealthDetect/app/(tabs)/history.tsx HealthDetect/app/(tabs)/index.tsx HealthDetect/app/(tabs)/learn.tsx HealthDetect/app/(tabs)/profile.tsx HealthDetect/app/camera.tsx HealthDetect/app/group-chat.tsx HealthDetect/app/photo-review.tsx HealthDetect/app/processing.tsx HealthDetect/app/results.tsx HealthDetect/app/role-select.tsx HealthDetect/components/external-link.tsx HealthDetect/components/haptic-tab.tsx HealthDetect/components/hello-wave.tsx HealthDetect/components/parallax-scroll-view.tsx HealthDetect/components/themed-text.tsx HealthDetect/components/themed-view.tsx HealthDetect/components/ui/collapsible.tsx HealthDetect/components/ui/icon-symbol.ios.tsx HealthDetect/components/ui/icon-symbol.tsx HealthDetect/components/VoiceButton.tsx HealthDetect/constants/theme.ts HealthDetect/constants/types.ts HealthDetect/context/AnalysisContext.tsx HealthDetect/context/PetProfileContext.tsx HealthDetect/eslint.config.js HealthDetect/hooks/use-color-scheme.ts HealthDetect/hooks/use-color-scheme.web.ts HealthDetect/hooks/use-theme-color.ts HealthDetect/package.json HealthDetect/README.md HealthDetect/scripts/reset-project.js HealthDetect/services/analysis-service.ts HealthDetect/services/community-service.ts HealthDetect/services/voice-service.ts HealthDetect/tsconfig.json index.js package.json README.md src/components/Card.js src/components/MetricCard.js src/constants/theme.js src/screens/HealthTipsScreen.js src/screens/HomeScreen.js src/screens/ProfileScreen.js src/screens/SymptomsScreen.js tsconfig.json ``` ### Dependencies - backend/requirements.txt: anthropic@>=0.42.0, fastapi@>=0.115.0, openai@>=1.12.0, packaging@>=23.2,<26, Pillow@>=10.0.0, pydantic@>=2.10.0, pydub@>=0.25.1, python-dotenv@>=1.0.1, python-multipart@>=0.0.12, uvicorn@>=0.32.0 - HealthDetect/package.json: @expo/ngrok@^4.1.3, @expo/vector-icons@^15.0.3, @react-navigation/bottom-tabs@^7.4.0, @react-navigation/elements@^2.6.3, @react-navigation/native@^7.1.8, @types/node@^25.2.3, @types/react@~19.1.0, eslint@^9.25.0, eslint-config-expo@~10.0.0, expo@~54.0.33, expo-av@^16.0.8, expo-camera@~17.0.10, expo-constants@~18.0.13, expo-file-system@^19.0.21, expo-font@~14.0.11, expo-haptics@~15.0.8, expo-image@~3.0.11, expo-image-picker@~17.0.10, expo-linear-gradient@~15.0.8, expo-linking@~8.0.11, expo-router@~6.0.23, expo-splash-screen@~31.0.13, expo-status-bar@~3.0.9, expo-symbols@~1.0.8, expo-system-ui@~6.0.9, expo-web-browser@~15.0.10, react@19.1.0, react-dom@19.1.0, react-native@0.81.5, react-native-gesture-handler@~2.28.0, react-native-reanimated@~4.1.1, react-native-safe-area-context@~5.6.0, react-native-screens@~4.16.0, react-native-web@~0.21.0, react-native-worklets@0.5.1, typescript@~5.9.2 - package.json: @expo/vector-icons@^15.0.3, @react-navigation/bottom-tabs@^7.13.0, @react-navigation/native@^7.1.28, @types/react@~19.1.10, expo@~54.0.33, expo-status-bar@~3.0.9, react@19.1.0, react-native@0.81.5, react-native-safe-area-context@~5.6.0, react-native-screens@~4.16.0, typescript@~5.9.2 ### Recent commits (newest first) - vet info - split up vets and users - make it multiturn - more agentic - group chat - make links clickable - fix vet experience visual - require analysis - shorten audio - Merge pull request #8 from janprasad/feat/ui-improvements - rest of files - Merge pull request #7 from janprasad/feat/ui-improvements - consumer mode - community tab - profile info included - Add activity feed backend for community engagement metrics - Add community support features for connecting pet owners - index edits - audio pause + faster audio - create .env for ngrok ## Key source files (fetched from GitHub, selected and truncated for size) Source contents could not be fetched from GitHub right now. [68 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]