# Project export: HungryFlow

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 12.0
- Tagline: Staring at your fridge, don't know what to eat, here's the solution: take a pic and this app will help find something.
- Devpost: https://devpost.com/software/hungryoverflow
- GitHub: https://github.com/KhoaNguyen706/CalHack12.0
- Video: https://www.youtube.com/embed/O6Mta3VZrSQ?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 2 GitHub contributor(s) — Quoc Anh Khoa Nguyen (13 commits), hung98po (12 commits)

## Devpost submission (written by the team)

### Inspiration

As someone who often stares at their fridge. Deciding what they're going to eat takes 1 to 10 minutes.

### What it does

You upload an image of your fridge and add the description of what taste you're currently feeling. Once complete, the AI agent's process will commence. The AI summarizes your description and ingredients, transfers that info to another AI to give a recipe, then it will talk to an AI that is programmed to judge the food if it is the right quality the user is asking, if not, Cooking AI and Food Adviser AI will go back and forward until a recipe is found.

### How we built it

We used Fetch AI(uAgent), Groq, Genimini 2.5 flash, Postman, FastAPI, React with Tailwind, and Vite to build this project. Genimini 2.5 flash we use to scan the photo for ingredients. React with Tailwind and Vite as the frontend framework. Fetch AI and Groq to create an AI Agent to complete our goal.

### Challenges we ran into

The challenge we ran into was that initially, we had an idea with YOLO for this project; however, we tried to implement YOLO, but it wouldn't work. The next challenge we came across was connecting our backend with our AI agents server, to was a difficult task. The big challenge was getting a good internet connection while we were working on our project.

### Accomplishments we're proud of

We finally got our AI agents all talking together, and we have two that are talking until a condition is met.

### What we learned

We learned

### What's next

HungryFlow's next step will be adding voice recognition to allow user to input their text by speech, add a database for food, and

## README (from the GitHub repository)

# Hungry Flow - CalHack12.0

Hungry Flow is your AI-powered multi-agent assistant for generating recipes from images and text. Upload a food photo and describe your cravings in plain English—Hungry Flow will detect ingredients, summarize your request, cook up a recipe, and advise on its quality, all with visible step-by-step progress.

## Features
- **Image-to-Recipe AI**: Upload a food image and get a custom recipe generated using advanced AI agents.
- **Multi-Agent System**: Detect, summarize, cook, and advise—each step handled by a dedicated agent for transparency and modularity.
- **Step-by-Step Progress**: See each stage of the recipe creation process in real time, with retry logic for quality improvement.
- **Sponsor Image Integration**: Easily fetch and display sponsor images using Google API, Bright Data, or Gemini-generated images.
- **Frontend/Backend Separation**: Modern React frontend and FastAPI/uAgents backend for scalability and clarity.

## Diagram
```
[User Uploads Image/Text]
        |
   [Detect Ingredients]
        |
   [Summarize Request]
        |
   [Cook Recipe]
        |
   [Advisor Quality Check]
        |
   [Retry (if needed)]
        |
   [Final Result]
```

## Instructions

### Frontend
1. Install Node.js and npm.
2. Change directory to `frontend` and run:
   ```powershell
   npm install
   npm run dev
   ```
3. Access the app at `http://localhost:5173` (or as shown in your terminal).

### Backend
1. Install Python (minimum version 3.10).
2. Change directory to `backend` and set up your environment:
   ```powershell
   python -m venv venv
   .\venv\Scripts\activate
   pip install -r requirements.txt
   ```
3. Add your API keys to a `.env` file (see below).
4. Open two terminals:
   - Terminal 1: Run the agent system:
     ```powershell
     python agents.py
     ```
   - Terminal 2: Start the FastAPI server:
     ```powershell
     python server.py
     ```

### Environment Variables
Create a `.env` file in `backend/` with your API keys:
```
GROQ_API_KEY=your_groq_api_key
GOOGLE_API_KEY=your_google_api_key

```

### Use Cases
- Generate recipes from food images and text descriptions
- Visualize each step of the agent workflow in real time
- Fetch and display sponsor images for demo or branding
- Retry recipe generation for improved quality

### Audience
Anyone who wants to turn food photos and cravings into actionable recipes—home cooks, food bloggers, hackathon teams, and more. No technical expertise required!

## Demo
Watch a demo: https://www.youtube.com/watch?v=O6Mta3VZrSQ

## Quick Troubleshooting
- If you see connection errors, make sure both backend terminals are running and ports are not blocked.
- If sponsor images do not appear, check your API keys and backend logs.
- For step-by-step progress, ensure the frontend is polling the backend and delays are set for visibility.

---
For more details, see the code and comments in `frontend/` and `backend/`. Contributions and feedback welcome!


## Detected evidence (automated analysis)

Indexed codebase: 15 recognized source files, 65 KB.
- CSS (language) — detected in the code
- FastAPI (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code

## Codebase structure (from repository index)

### Files (24 of 24)

```
.gitignore
backend/.env.example
backend/agentFunction.py
backend/agents.py
backend/DetectService.py
backend/requirements.txt
backend/server.py
frontend/.gitignore
frontend/eslint.config.js
frontend/index.html
frontend/package.json
frontend/README.md
frontend/src/App.tsx
frontend/src/index.css
frontend/src/main.tsx
frontend/src/pages/HomePage.tsx
frontend/src/pages/InputPage.tsx
frontend/src/pages/ResultPage.tsx
frontend/tsconfig.app.json
frontend/tsconfig.json
frontend/tsconfig.node.json
frontend/vite.config.ts
package.json
README.md
```

### Dependencies

- backend/requirements.txt: fastapi@==0.104.1, google-generativeai, groq, langchain-groq, python-dotenv, uagents@==0.22.10, uvicorn[standard]@>=0.30.1
- frontend/package.json: @eslint/js@^9.36.0, @types/node@^24.6.0, @types/react@^19.1.16, @types/react-dom@^19.1.9, @vitejs/plugin-react@^5.0.4, eslint@^9.36.0, eslint-plugin-react-hooks@^5.2.0, eslint-plugin-react-refresh@^0.4.22, globals@^16.4.0, react@^19.1.1, react-dom@^19.1.1, react-router@^7.9.4, typescript@~5.9.3, typescript-eslint@^8.45.0, vite@^7.1.7
- package.json: @tailwindcss/vite@^4.1.16, autoprefixer@^10.4.21, postcss@^8.5.6, react@^19.2.0, react-dom@^19.2.0, tailwindcss@^4.1.16

### Recent commits (newest first)

- polish readme
- polish backend
- Stop tracking recipe_results and ignore runtime outputs
- Merge branch 'main' of https://github.com/KhoaNguyen706/CalHack12.0
- bug fix
- Modify README title to 'CalHack12.0 - Hungry Flow'
- Update README with setup instructions
- Add demo link to README
- before judges
- add frontend
- Merge branch 'main' of https://github.com/KhoaNguyen706/CalHack12.0
- add Agent family and fix bug
- save
- save
- basic frontend
- update
- add component
- Merge branch 'main' of https://github.com/KhoaNguyen706/CalHack12.0
- add family_agent
- frontend

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "dependencies": {
    "react": "^19.2.0",
    "react-dom": "^19.2.0"
  },
  "devDependencies": {
    "@tailwindcss/vite": "^4.1.16",
    "autoprefixer": "^10.4.21",
    "postcss": "^8.5.6",
    "tailwindcss": "^4.1.16"
  }
}

```

### backend/requirements.txt

```
fastapi==0.104.1
uvicorn[standard]>=0.30.1
uagents==0.22.10
langchain-groq
python-dotenv
google-generativeai
groq
```

### frontend/package.json

```
{
  "name": "frontend",
  "private": true,
  "version": "0.0.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "tsc -b && vite build",
    "lint": "eslint .",
    "preview": "vite preview"
  },
  "dependencies": {
    "react": "^19.1.1",
    "react-dom": "^19.1.1",
    "react-router": "^7.9.4"
  },
  "devDependencies": {
    "@eslint/js": "^9.36.0",
    "@types/node": "^24.6.0",
    "@types/react": "^19.1.16",
    "@types/react-dom": "^19.1.9",
    "@vitejs/plugin-react": "^5.0.4",
    "eslint": "^9.36.0",
    "eslint-plugin-react-hooks": "^5.2.0",
    "eslint-plugin-react-refresh": "^0.4.22",
    "globals": "^16.4.0",
    "typescript": "~5.9.3",
    "typescript-eslint": "^8.45.0",
    "vite": "^7.1.7"
  }
}

```

### backend/server.py

```python
from typing import List
from fastapi import FastAPI, UploadFile, File, HTTPException,Form
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from DetectService import read_image_bytes, detect_ingredients
import asyncio
import uuid
import httpx
import os
import json


app = FastAPI(title="Recipe Generator API")
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"]
)

pending_requests = {}

class RecipeRequest(BaseModel):
    text: str
    ingredients: List[str]

class RecipeResponse(BaseModel):
    status: str
    data: dict
class DetectResponse(BaseModel):
    ingredients: List[str]
    confidence: float  
    raw_items: List[dict]  

@app.get("/api/recipe/result/{request_id}")
async def get_recipe_status(request_id: str):
    """
    Get the current status of a recipe generation request
    Returns: processing, retrying, completed, or not_found
    """
    try:
        file_path = f"recipe_results/{request_id}.json"
        
        if os.path.exists(file_path):
            with open(file_path, 'r') as f:
                recipe_data = json.load(f)
            
            
            if "status" in recipe_data:
                return recipe_data  
            else:
                
                return {
                    "status": "completed",
                    "data": recipe_data
                }
        else:
            return {
                "status": "not_found",
                "message": "Recipe not yet available"
            }
            
    except Exception as e:
        print(f"Error getting recipe status: {e}")
        return {
            "status": "error",
            "message": str(e)
        }

@app.post("/detect", response_model=DetectResponse)
async def detect(image: UploadFile = File(...)):
    """Detect ingredients from an uploaded image."""
    if image.content_type not in {"image/jpeg", "image/jpg", "image/png", "image/webp", "image/heic", "image/heif"}:
        raise HTTPException(400, "Please upload a JPEG/PNG/WEBP/HEIC image")

    
    img_bytes = read_image_bytes(image)
    
    
    result = detect_ingredients(img_bytes)
    
    return DetectResponse(**result)

@app.post("/recipe/generate")
async def generate_recipe(
    image: UploadFile = File(...),
    text: str = Form(...)
):
    """
    Start recipe generation: detect ingredients, write initial status file,
    start Action Agent in background and return request_id immediately.
    The frontend should poll /api/recipe/result/{request_id}.
    """
    try:
        # Step 1: Detect ingredients from image
        img_bytes = await image.read()
        detection = detect_ingredients(img_bytes)
        ingredients = detection["ingredients"]
        if not ingredients:
            raise HTTPException(400, "No ingredients detected in image")

        request_id = str(uuid.uuid4())
        pending_requests[request_id] = {
            "status": "processing",
            "ingredients": ingredients,
            "text": text
        }

        # Ensure result dir
        RESULTS_DIR = "recipe_results"
        os.makedirs(RESULTS_DIR, exist_ok=True)

        # Write initial detect status so frontend can show "Detecting" immediately
        detect_status = {
            "status": "processing",
            "stage": "detect",
            "step": "detect",
            "message": "Ingredients detected. Starting summarization...",
            "data": {
                "ingredients": ingredients,
                "confidence": detection.get("confidence", 0),
                "raw_items": detection.get("raw_items", [])
            }
        }
        file_path = os.path.join(RESULTS_DIR, f"{request_id}.json")
        with open(file_path, "w") as f:
            json.dump(detect_status, f, indent=2)

        # Start Action Agent in background so this endpoint returns immediately
        async def start_action_agent():
            try:
                async with httpx.AsyncClient(timeout=180.0) as client:
                    # adjust URL/port to your Action Agent service
                    resp = await client.post(
                        "http://localhost:8000/api/recipe/start",
                        json={
                            "text": text,
                            "ingredients": ingredients,
                            "session": request_id
                        }
                    )
                    # log response for debugging; agent itself should update status file
                    print(f"[background] Action Agent start status: {resp.status_code}")
                    print(f"[background] Action Agent body: {resp.text}")
            except Exception as e:
                print(f"[background] Failed to call Action Agent: {e}")
                # write an error status to results so frontend sees failure
                error_status = {
                    "status": "error",
                    "stage": "detect",
                    "step": "detect",
                    "message": f"Failed to start action agent: {str(e)}"
                }
                with open(file_path, "w") as f:
                    json.dump(error_status, f, indent=2)

        asyncio.create_task(start_action_agent())

        # Return immediately with request_id - frontend will poll for step updates
        return {"status": "accepted", "request_id": request_id}

    except HTTPException:
        raise
    except Exception as e:
        import traceback
        traceback.print_exc()
        raise HTTPException(500, f"Error: {str(e)}")
        
@app.post("/recipe/webhook/{request_id}")
async def recipe_webhook(request_id: str, data: dict):
    """
    Webhook for agents to send results back
    Called by Advisor Agent when recipe is approved/rejected
    """
    print(f" Webhook received for {request_id}")
    
    if request_id in pending_requests:
        pending_requests[request_id].update(data)
 
[truncated — 375 more characters]
```

### frontend/src/main.tsx

```typescript
import { StrictMode } from 'react'
import { createRoot } from 'react-dom/client'
import './index.css'
import App from './App.tsx'

createRoot(document.getElementById('root')!).render(
  <StrictMode>
    <App />
  </StrictMode>,
)

```

### frontend/src/App.tsx

```typescript
import { BrowserRouter as Router, Routes, Route } from 'react-router'
import HomePage from './pages/HomePage'
import InputPage from './pages/InputPage'
import ResultPage from './pages/ResultPage'



function App() {

  return (
    <>
      <Router>
      <Routes>
        <Route path="/" element={<HomePage />} />
        <Route path="/input" element={<InputPage />} />
        <Route path="/result" element={<ResultPage />} />
      </Routes>
    </Router>
    </>
  )
}

export default App

```

### frontend/vite.config.ts

```typescript
import { defineConfig } from 'vite';
    import react from '@vitejs/plugin-react';
    import tailwindcss from '@tailwindcss/vite'; // Import the Tailwind CSS plugin

    export default defineConfig({
      plugins: [
        react(),
        tailwindcss(), // Add the Tailwind CSS plugin to your plugins array
      ],
    });

```

### frontend/index.html

```html
<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" type="image/svg+xml" href="/vite.svg" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>frontend</title>
  </head>
  <body>
    <div id="root"></div>
    <script type="module" src="/src/main.tsx"></script>
  </body>
</html>

```

### frontend/eslint.config.js

```javascript
import js from '@eslint/js'
import globals from 'globals'
import reactHooks from 'eslint-plugin-react-hooks'
import reactRefresh from 'eslint-plugin-react-refresh'
import tseslint from 'typescript-eslint'
import { defineConfig, globalIgnores } from 'eslint/config'

export default defineConfig([
  globalIgnores(['dist']),
  {
    files: ['**/*.{ts,tsx}'],
    extends: [
      js.configs.recommended,
      tseslint.configs.recommended,
      reactHooks.configs['recommended-latest'],
      reactRefresh.configs.vite,
    ],
    languageOptions: {
      ecmaVersion: 2020,
      globals: globals.browser,
    },
  },
])

```

### backend/DetectService.py

```python
import os
import io
import json
from typing import List, Dict
from fastapi import UploadFile, HTTPException
from PIL import Image
from dotenv import load_dotenv
from rapidfuzz import process as rf_process, fuzz as rf_fuzz
import google.generativeai as genai

load_dotenv()
api_key = os.getenv("GEMINI_API_KEY")
if not api_key:
    raise RuntimeError("Set GEMINI_API_KEY in .env")

genai.configure(api_key=api_key)
gem_model = genai.GenerativeModel("gemini-2.5-flash")


CANON = [
    "apple", "banana", "orange", "lemon", "lime", "grape", "strawberry", "blueberry", "mango",
    "avocado", "tomato", "cucumber", "carrot", "broccoli", "lettuce", "spinach", "onion", "garlic",
    "potato", "egg", "milk", "cheese", "yogurt", "butter", "rice", "pasta", "bread", "chicken", "beef", "tofu"
]


def best_match(token: str, choices: List[str], thr: int = 85):
    """Match a token to the best canonical ingredient name."""
    token = (token or "").strip().lower()
    if not token:
        return None
    match, score, _ = rf_process.extractOne(token, choices, scorer=rf_fuzz.WRatio)
    return match if score >= thr else None


def read_image_bytes(upload: UploadFile) -> bytes:
    """Read and normalize image to JPEG bytes for Gemini."""
    raw = upload.file.read()
    try:
        img = Image.open(io.BytesIO(raw)).convert("RGB")
        buf = io.BytesIO()
        img.save(buf, format="JPEG", quality=90)
        return buf.getvalue()
    except Exception:
        
        return raw


def detect_ingredients(image_bytes: bytes) -> Dict:
    """
    Detect ingredients from image using Gemini Vision API.
    
    Args:
        image_bytes: JPEG image bytes
        
    Returns:
        Dict with keys: ingredients (List[str]), confidence (float), raw_items (List[dict])
    """
    prompt = (
        "You are an ingredient detector for fridge photos or for table photos. "
        "List visible edible ingredients/foods (not brands/containers). "
        "Return ONLY JSON: {\"ingredients\":[{\"name\":\"apple\",\"confidence\":0.92}, ...]} "
        "Use lowercase common names and include 5–15 items max."
    )

    image_part = {"inline_data": {"mime_type": "image/jpeg", "data": image_bytes}}
    
    try:
        resp = gem_model.generate_content(
            [prompt, image_part],
            generation_config={"response_mime_type": "application/json"}
        )
        data = json.loads(resp.text)
        items = data.get("ingredients", [])
    except Exception as e:
        raise HTTPException(500, f"Vision model error: {e}")

    
    names, top_conf = [], 0.0
    for it in items:
        n = str(it.get("name", "")).lower().strip()
        c = float(it.get("confidence", 0.0))
        m = best_match(n, CANON, 80) or n  
        names.append(m)
        top_conf = max(top_conf, c)

    cleaned = sorted(set(names))
    
    return {
        "ingredients": cleaned,
        "confidence": float(top_conf),
        "raw_items": items
    }
```

[6 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]