# Project export: Refrigerator Ramsay This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: AI-powered kitchen assistant that reduces food waste with smart recipes and real-time interaction. - Devpost: https://devpost.com/software/refrigerator-ramsay - GitHub: https://github.com/sherwinvishesh/Refrigerator-Ramsay - Video: https://www.youtube.com/embed/bNv-EJtd40s?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Google: Most useful app) - Team: 1 GitHub contributor(s) — Sherwin Vishesh Jathanna (15 commits) ## Devpost submission (written by the team) ### Inspiration The idea for Refrigerator Ramsay came from a common problem we all face: food waste. So many times, we forget about ingredients in the fridge, or we don't know how to use them, and they go bad. We wanted to create something that could help us not only use those ingredients but also inspire us to try new recipes and reduce food waste in the process. ### What it does Refrigerator Ramsay is an AI-powered kitchen assistant that helps users make the most of their groceries. You can take a picture of your fridge or pantry, and the AI will identify the ingredients. Then, it suggests recipes based on what you have. It also has an interactive voice feature, so you can ask for cooking tips or get help with recipes in real-time. ### How we built it We used Google Gemini AI for image recognition to identify ingredients from photos. and then, to come up with creative meal ideas. We also added Hume EVI, which is an emotionally intelligent voice AI, to make the experience more interactive. This lets users chat with the AI to get tips, learn new cooking techniques, or personalize recipes. ### Challenges we ran into One challenge we faced was working with Hume EVI for the first time. It was really fun to explore, but there was definitely a learning curve. We had to figure out how to make the voice assistant not just smart but also engaging, making sure it could deliver cooking tips naturally and understand what users were asking. It took some trial and error to get the right balance between information and interaction. Another challenge was training the Google Gemini AI to be as accurate as possible when recognizing ingredients. It wasn't easy to ensure the AI could reliably detect different foods, especially when items were stacked or grouped together in photos. We had to spend extra time fine-tuning the model so that it could handle a variety of lighting conditions and product packaging. ### Accomplishments we're proud of We're proud of creating a solution that not only helps reduce food waste but also makes cooking more fun. The AI's ability to suggest creative recipes from random ingredients is really exciting. We're also proud of how user-friendly (and funny) the voice assistant turned out to be, making it feel like you're cooking with a friend. ### What we learned We learned a lot about how AI can be used to solve everyday problems, like reducing food waste. Working with Hume EVI taught us about building conversational AI, which was new for us. It was really fun, but there was definitely a challenge in figuring out how to make the voice interaction feel natural and helpful at the same time. We also learned about the importance of training AI models, especially with the Gemini AI. Getting the image recognition to accurately identify different ingredients required us to experiment a lot with the data and train the model to work in a variety of environments. This taught us that accuracy is key when it comes to creating a seamless user experience. ### What's next Next, we want to improve the AI’s ability to recognize even more ingredients and maybe even offer nutritional advice. We’re also thinking about adding features where the AI can help you plan meals for the whole week based on what you have. Eventually, we’d love to partner with grocery stores to suggest recipes based on store deals, helping users save money too! ## README (from the GitHub repository) # Refrigerator-Ramsay AI-powered kitchen assistant reducing food waste with smart recipes and real-time interaction ## Detected evidence (automated analysis) Indexed codebase: 41 recognized source files, 194 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (50 of 50) ``` .gitignore backend/main.py backend/Readme.md backend/requirements.txt evi-files/app/error.tsx evi-files/app/globals.css evi-files/app/layout.tsx evi-files/app/page.tsx evi-files/components.json evi-files/components/Chat.tsx evi-files/components/Controls.tsx evi-files/components/Expressions.tsx evi-files/components/logos/GitHub.tsx evi-files/components/logos/Hume.tsx evi-files/components/Messages.tsx evi-files/components/MicFFT.tsx evi-files/components/Nav.tsx evi-files/components/StartCall.tsx evi-files/components/ui/button.tsx evi-files/components/ui/toggle.tsx evi-files/next-env.d.ts evi-files/package.json evi-files/postcss.config.mjs evi-files/README.md evi-files/tailwind.config.ts evi-files/tsconfig.json evi-files/utils/expressionColors.ts evi-files/utils/expressionLabels.ts evi-files/utils/getHumeAccessToken.ts evi-files/utils/index.ts frontend/package.json frontend/public/index.html frontend/public/manifest.json frontend/public/robots.txt frontend/Readme.md frontend/src/App.css frontend/src/App.js frontend/src/App.test.js frontend/src/components/CallPage.js frontend/src/components/ChefChat.js frontend/src/components/EntryDescription.js frontend/src/components/HomePage.js frontend/src/components/index.js frontend/src/components/Page3.js frontend/src/index.css frontend/src/index.js frontend/src/reportWebVitals.js frontend/src/setupTests.js LICENSE README.md ``` ### Dependencies - backend/requirements.txt: fastapi, google-generativeai, markdown, Pillow, python-dotenv, python-multipart, uvicorn - evi-files/package.json: @humeai/voice-react@^0.1.15, @radix-ui/react-slot@^1.0.2, @radix-ui/react-switch@^1.0.3, @radix-ui/react-toggle@^1.0.3, @types/node@^20, @types/react@^18, @types/react-dom@^18, @types/react-virtualized@^9.21.30, class-variance-authority@^0.7.0, clsx@^2.1.1, eslint@^8, eslint-config-next@14.2.3, framer-motion@^11.2.10, geist@^1.3.0, hume@^0.8.10, lucide-react@^0.394.0, next@14.2.3, postcss@^8, react@^18, react-dom@^18, react-virtualized@^9.22.5, remeda@^2.0.9, server-only@^0.0.1, tailwind-merge@^2.3.0, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, ts-pattern@^5.1.2, typescript@^5 - frontend/package.json: @fortawesome/free-solid-svg-icons@^6.6.0, @fortawesome/react-fontawesome@^0.2.2, @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, react@^18.3.1, react-dom@^18.3.1, react-dropzone@^14.2.10, react-markdown@^9.0.1, react-router-dom@^6.26.2, react-scripts@5.0.1, react-webcam@^7.2.0, rehype-raw@^7.0.0, styled-components@^6.1.13, web-vitals@^2.1.4 ### Recent commits (newest first) - Update HomePage.js - Add files via upload - Update requirements.txt - Merge pull request #1 from sherwinvishesh/Hume - Added files - Rename CalHacks PPT.pdf to Public/CalHacks PPT.pdf - Add files via upload - Update HomePage.js - Add files via upload - Add files via upload - Create Readme.md - Add files via upload - Create Readme.md - Create .gitignore - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### backend/requirements.txt ``` fastapi uvicorn google-generativeai python-dotenv Pillow markdown python-multipart ``` ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "dependencies": { "@fortawesome/free-solid-svg-icons": "^6.6.0", "@fortawesome/react-fontawesome": "^0.2.2", "@testing-library/jest-dom": "^5.17.0", "@testing-library/react": "^13.4.0", "@testing-library/user-event": "^13.5.0", "react": "^18.3.1", "react-dom": "^18.3.1", "react-dropzone": "^14.2.10", "react-markdown": "^9.0.1", "react-router-dom": "^6.26.2", "react-scripts": "5.0.1", "react-webcam": "^7.2.0", "rehype-raw": "^7.0.0", "styled-components": "^6.1.13", "web-vitals": "^2.1.4" }, "scripts": { "start": "react-scripts start", "build": "react-scripts build", "test": "react-scripts test", "eject": "react-scripts eject" }, "eslintConfig": { "extends": [ "react-app", "react-app/jest" ] }, "browserslist": { "production": [ ">0.2%", "not dead", "not op_mini all" ], "development": [ "last 1 chrome version", "last 1 firefox version", "last 1 safari version" ] } } ``` ### evi-files/package.json ``` { "name": "hume-evi-next-js-starter", "version": "0.1.0", "private": true, "homepage": "https://github.com/humeai/hume-evi-next-js-starter", "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@humeai/voice-react": "^0.1.15", "hume": "^0.8.10", "@radix-ui/react-slot": "^1.0.2", "@radix-ui/react-switch": "^1.0.3", "@radix-ui/react-toggle": "^1.0.3", "@types/react-virtualized": "^9.21.30", "class-variance-authority": "^0.7.0", "clsx": "^2.1.1", "framer-motion": "^11.2.10", "geist": "^1.3.0", "hume": "^0.8.10", "lucide-react": "^0.394.0", "next": "14.2.3", "react": "^18", "react-dom": "^18", "react-virtualized": "^9.22.5", "remeda": "^2.0.9", "server-only": "^0.0.1", "tailwind-merge": "^2.3.0", "tailwindcss-animate": "^1.0.7", "ts-pattern": "^5.1.2" }, "devDependencies": { "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "eslint": "^8", "eslint-config-next": "14.2.3", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "^5" } } ``` ### backend/main.py ```python import os from fastapi import FastAPI, HTTPException, UploadFile, File, Form from fastapi.middleware.cors import CORSMiddleware import google.generativeai as genai from dotenv import load_dotenv from PIL import Image import base64 import io # Load environment variables load_dotenv() # Configure FastAPI app = FastAPI() # Enable CORS for frontend-backend communication app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # Configure Google Gemini AI api_key = os.getenv("GEMINI_API_KEY") genai.configure(api_key=api_key) # Allowed file types for images ALLOWED_EXTENSIONS = {'png', 'jpg', 'jpeg', 'gif'} def allowed_file(filename): return '.' in filename and filename.rsplit('.', 1)[1].lower() in ALLOWED_EXTENSIONS # Temporary uploads directory uploads_dir = os.path.join(os.getcwd(), 'uploads') os.makedirs(uploads_dir, exist_ok=True) @app.post("/analyze_image") async def analyze_image(image_file: UploadFile = File(...), description: str = Form(...), language: str = Form('en')): if not allowed_file(image_file.filename): return {"error": "File type not allowed"}, 400 try: # Save the uploaded image image_path = os.path.join(uploads_dir, image_file.filename) with open(image_path, "wb") as f: f.write(image_file.file.read()) # Open the saved image for processing with Image.open(image_path) as img: # Step 1: Identify groceries from the image prompt_identify = "Identify all groceries from this image." response_identify = genai.GenerativeModel("gemini-1.5-flash").generate_content([prompt_identify, img]) # Extract identified grocery items groceries = [] for candidate in response_identify.candidates: if candidate.content.parts: grocery_item = candidate.content.parts[0].text.strip().lower() groceries.append(grocery_item) if not groceries: print("No groceries identified.") return {"error": "No groceries identified"}, 500 # Log recognized groceries to terminal print(f"Recognized Groceries: {groceries}") # Step 2: Generate recipe using identified groceries and user description groceries_list = ', '.join(groceries) prompt_recipe = f"Using these groceries: {groceries_list}, generate a recipe with the following instructions: {description}" response_recipe = genai.GenerativeModel('gemini-1.5-pro').generate_content([prompt_recipe]) # Final recipe response if hasattr(response_recipe, 'text'): recipe = response_recipe.text.strip() print(f"Generated Recipe: {recipe}") # Log the generated recipe to the terminal else: recipe = "No recipe generated." print("No recipe generated.") return {"groceries": groceries, "recipe": recipe} except Exception as e: print(f"Error processing the image: {e}") raise HTTPException(status_code=500, detail=f"Error processing the image: {e}") ``` ### evi-files/utils/index.ts ```typescript import { type ClassValue, clsx } from "clsx" import { twMerge } from "tailwind-merge" export function cn(...inputs: ClassValue[]) { return twMerge(clsx(inputs)) } ``` ### evi-files/app/page.tsx ```typescript import { getHumeAccessToken } from "@/utils/getHumeAccessToken"; import dynamic from "next/dynamic"; const Chat = dynamic(() => import("@/components/Chat"), { ssr: false, }); export default async function Page() { const accessToken = await getHumeAccessToken(); if (!accessToken) { throw new Error(); } return (
); } ``` ### frontend/src/index.js ```javascript import React from 'react'; import ReactDOM from 'react-dom/client'; import './index.css'; import App from './App'; import reportWebVitals from './reportWebVitals'; const root = ReactDOM.createRoot(document.getElementById('root')); root.render( ); // If you want to start measuring performance in your app, pass a function // to log results (for example: reportWebVitals(console.log)) // or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals reportWebVitals(); ``` ### evi-files/app/layout.tsx ```typescript import type { Metadata } from "next"; import { GeistSans } from "geist/font/sans"; import { GeistMono } from "geist/font/mono"; import "./globals.css"; import { Nav } from "@/components/Nav"; import { cn } from "@/utils"; export const metadata: Metadata = { title: "Hume AI - EVI - Next.js Starter", description: "A Next.js starter using Hume AI's Empathic Voice Interface", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return (