Project Info

Sproutify - Table 244

Inspiration

28 million Americans struggle with eating disorders, where every meal becomes a source of anxiety and fear. We discovered a powerful insight from clinical research: people with eating disorders often care for others before themselves. This sparked an idea—what if we could redirect that nurturing instinct to support their own recovery? We wanted to transform one of the most challenging aspects of recovery—eating meals—from a moment of stress into something rewarding and positive. By combining the emotional power of pet companionship with cutting-edge AR technology and AI agents, we created Sproutify to make recovery feel less like a battle and more like raising a friend.

What it does

Sproutify is a gamified AR companion system that transforms eating disorder recovery into a rewarding experience. For Patients: AR Cat Companion "Sprout" appears through Snap Spectacles during meals Real-time AI Detection: Uses depth sensing and Google Gemini to identify food items, track macros, and measure consumption Gamification System: Earn points for healthy eating behaviors: Eating speed and consistency, Meal completion rates, Nutritional balance (macros), Total time taken to eat Rewards & Customization: Spend points on toys and cosmetics for Sprout Intelligent Support: Sprout provides adaptive encouragement when hesitation is detected Predictive AI: Detects concerning patterns and provides gentle interventions before relapse For Doctors: Automated Progress Reports generated by Fetch.ai agents Long-term Behavior Analysis: Track eating patterns, nutritional progress, and consistency Early Warning System: AI flags concerning behaviors for clinical intervention Data-Driven Decisions: Objective metrics to inform treatment adjustments The Result: A complete support ecosystem that makes recovery feel rewarding instead of restrictive.

How we built it

We used Spectacles and Lens Studio as our main platform for AR interactivity with the outside world. Our algorithm accesses our custom Fetch.ai endpoint to access the long-term data of the user and returns a JSON object to our Lens client. We also use local object recognition algorithms to identity objects locally.

Challenges we ran into

One challenge is real-time Object Detection: Achieving accurate food recognition and portion estimation through Spectacles’ limited onboard compute required optimization and hybrid inference (local + cloud via Google Gemini). Another challenge is balancing gamification with clinical sensitivity was difficult—we had to ensure the app motivates recovery without triggering anxiety or unhealthy behaviors.

Accomplishments we're proud of

Functional AR Prototype: We successfully created an interactive AR companion that detects food items in real-time, reacts emotionally, and evolves with the user’s progress. We also built a full End-to-End AI Pipeline: from food detection to behavior analysis, our Fetch.ai agents and Gemini-powered models enable seamless monitoring and insights for clinicians. We are also proud of our clinically inspired design: We conducted psychological research on recovery motivation, turning empathy and care into measurable progress through gamified behavior reinforcement.

What we learned

Technical Skills AR Development: How to build immersive experiences for Snap Spectacles Depth Sensing: Understanding and leveraging 3D computer vision Multi-agent Systems: Orchestrating specialized AI agents with Fetch.ai Multimodal AI: Using Gemini's vision capabilities for food analysis Real-time Systems: Balancing latency, accuracy, and user experience Design Insights Gamification Psychology: How to motivate behavior change without triggering Healthcare UX: Building interfaces that work for both patients and clinicians AR Interaction: Creating intuitive controls in 3D space

What's next

for Sproutify Sproutify hopes to continue the transition o making good from a negative to a positive experience for all patients struggling with anorexia. We hope to fully integrate nutritional data in the form of subtle suggestions through Sprout into Sprout’s verbal and visual cues. We also hope to export our nutritional analysis in a format suitable to be used in medical reports to track and predict past and future recovery trends to assist medical professionals.

Patient Nutrition Analysis System

A comprehensive system for analyzing patient eating patterns and nutritional intake using Fetch.ai agents, Supabase storage, and Gemini Vision API.

🏥 System Overview

This system provides healthcare professionals with detailed analysis of patient meal patterns, including:

Eating Pattern Analysis: Regular vs irregular eating schedules
Nutritional Assessment: Calorie intake, food categories, portion analysis
Professional Recommendations: Evidence-based suggestions for nutritionists/doctors
Historical Data: Analysis of all stored meal images

🏗️ Architecture

Spectacles AR → Storage Agent → Analysis Agent → Frontend Web App
     ↓              ↓              ↓              ↓
Image Capture   Supabase Upload  Gemini Analysis  Doctor Dashboard

Components:

Storage Agent (storage_agent.py) - Handles image uploads to Supabase
Analysis Agent (nutrition_analysis_agent.py) - Performs comprehensive analysis
Frontend Web App (nutrition_frontend.py) - Professional dashboard
Batch Upload (upload_assets_folder.py) - Uploads multiple images

🚀 Quick Start

1. Prerequisites

Python 3.11+
Supabase account and project
Gemini API key
Fetch.ai Agentverse account

2. Environment Setup

Create .env file:

SUPABASE_URL=https://your-project.supabase.co
SUPABASE_KEY=your-anon-public-key
GEMINI_API_KEY=your-gemini-api-key

3. Install Dependencies

cd backend
pip install -r requirements.txt

4. Supabase Setup

Run this SQL in Supabase SQL Editor:

-- Create meal_images table
CREATE TABLE meal_images (
    id SERIAL PRIMARY KEY,
    session_id TEXT NOT NULL,
    frame_id TEXT NOT NULL,
    file_path TEXT NOT NULL,
    url TEXT NOT NULL,
    uploaded_at BIGINT NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

-- Disable RLS for testing
ALTER TABLE meal_images DISABLE ROW LEVEL SECURITY;

-- Create storage bucket
-- Go to Storage → Create bucket: "meals" (set to public)

5. Run the System

python setup_and_run.py

📋 Usage Instructions

Option 1: Complete System

python setup_and_run.py
# Choose option 4 to run everything

Option 2: Individual Components

Upload Assets Folder

python upload_assets_folder.py

Uploads all images from assets/ folder
30-second intervals between uploads
Creates session tracking

Start Analysis Agent

python nutrition_analysis_agent.py

Runs on port 8003
Provides REST API for analysis
Integrates with Gemini Vision API

Start Frontend Web App

python nutrition_frontend.py

Runs on port 5000
Professional dashboard for healthcare workers
Access at: http://127.0.0.1:5000

🔍 Analysis Features

Eating Pattern Analysis

Meal Frequency: Number of meals per day
Timing Patterns: Regular vs irregular eating schedules
Consumption Patterns: Portion sizes and eating pace
Interval Analysis: Time between meals

Nutritional Assessment

Food Identification: AI-powered food recognition
Calorie Estimation: Estimated caloric intake per meal
Food Categories: Protein, carbs, vegetables, etc.
Portion Analysis: Consumption percentage per meal

Professional Recommendations

Eating Schedule: Suggestions for regular meal timing
Portion Control: Recommendations for appropriate serving sizes
Nutritional Balance: Advice on food variety and nutrients
Behavioral Patterns: Insights into eating habits

🌐 Frontend Dashboard

Features:

Patient Analysis: Enter patient ID and date range
Real-time Results: Live analysis of stored images
Professional Interface: Designed for healthcare workers
Comprehensive Reports: Detailed eating pattern analysis

Usage:

Open http://127.0.0.1:5000
Enter patient ID
Select date range (optional)
Choose analysis type
View comprehensive results

📊 API Endpoints

Analysis Agent (Port 8003)

POST /analyze - Analyze patient data
GET /health - Health check

Frontend (Port 5000)

GET / - Main dashboard
POST /analyze_patient - Patient analysis request
GET /health - System health check

🔧 Configuration

Agent Configuration

Storage Agent: Port 8001
Analysis Agent: Port 8003
Frontend: Port 5000

Supabase Configuration

Bucket: "meals" (public access)
Table: "meal_images"
RLS: Disabled for testing

Gemini Configuration

Model: gemini-2.0-flash-exp
Analysis: Food recognition and nutritional assessment

🧪 Testing

Test Individual Components

# Test Supabase upload
python test_supabase_upload.py

# Test custom image analysis
python upload_custom_image.py

# Query uploaded data
python query_uploads.py

Test Complete Flow

Upload images using batch script
Start analysis agent
Start frontend
Access dashboard and run analysis

📁 File Structure

backend/
├── storage_agent.py              # Storage Agent
├── nutrition_analysis_agent.py  # Analysis Agent
├── nutrition_frontend.py        # Flask Frontend
├── upload_assets_folder.py      # Batch Upload
├── test.py                      # Core Analysis Logic
├── templates/
│   └── nutrition_dashboard.html  # Web Interface
├── assets/                      # Image Storage
├── requirements.txt             # Dependencies
├── setup_and_run.py            # Setup Script
└── .env                        # Environment Variables

🚨 Troubleshooting

Common Issues:

Supabase Upload Fails
- Check bucket exists and is public
- Verify SUPABASE_URL and SUPABASE_KEY
- Disable RLS: ALTER TABLE meal_images DISABLE ROW LEVEL SECURITY;
Gemini Analysis Fails
- Verify GEMINI_API_KEY in .env
- Check API quota and limits
- Ensure images are accessible via URL
Agent Communication Issues
- Check agent addresses and ports
- Verify all agents are running
- Check network connectivity
Frontend Not Loading
- Ensure Flask is installed
- Check port 5000 is available
- Verify analysis agent is running

🎯 Hackathon Requirements Met

✅ Fetch.ai Agentverse Integration: Agents registered and deployed ✅ Chat Protocol: All agents use Fetch.ai messaging ✅ LLM Integration: Gemini Vision API for food analysis ✅ Real-world Actions: File uploads, database operations, API calls ✅ Professional Interface: Web dashboard for healthcare workers ✅ Multi-agent Architecture: Specialized agents for different tasks ✅ External API Integration: Supabase, Gemini, Flask

📞 Support

For issues or questions:

Check troubleshooting section
Verify all dependencies are installed
Ensure .env file is properly configured
Check Supabase and Gemini API access

🏆 Features Delivered

Complete Fetch.ai Integration: Pure chat protocols, no REST endpoints
Professional Healthcare Interface: Designed for nutritionists/doctors
Comprehensive Analysis: Eating patterns, nutritional assessment, recommendations
Scalable Architecture: Microservices with specialized agents
Real-world Application: Solving eating disorder support challenges

Analysis

Compare with all teams

View

Metric

Irenka Ni
8
EvelynFung
4

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

FlaskIn code
HTMLIn code
PythonIn code
SupabaseIn code
TypeScriptIn code

5 of 5 appear in the indexed code.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

166 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.