# Project export: Grocery Tracker This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Do you have trouble keeping track of all your groceries? With Grocery Tracker, you can make a grocery list of all your items with their shelf lives and generate recipes with a quick scan of a receipt! - Devpost: https://devpost.com/software/grocery-tracker-nchs87 - GitHub: https://github.com/jparkk04/grocery-tracker - Team: 2 GitHub contributor(s) — jparkk04 (8 commits), Noah Han (2 commits) ## Devpost submission (written by the team) ### Inspiration Our inspiration for Grocery Tracker is that feeling when you realize you've forgotten the lettuce in the fridge for more than just a couple of weeks and by now, it's definitely gone bad. ### What it does Grocery Tracker solves this problem by keeping a list of all your groceries with a countdown of the expiration date of each item with just a quick picture of your receipt! Grocery Tracker also generates recipes with your grocery list items for inspiration to get use of all your ingredients! ### How we built it We used React Native and Flask to build a mobile app that can take pictures of receipts and display grocery lists and recipes! With Google Gemini generative AI, grounded in Google Search, we were able to identify food items from receipts, generate the estimated shelf life for each item, and find recipe inspiration! ### Challenges we ran into The most difficult part of the project was getting our React Native front end to communicate with the Flask backend which handled communication with the Google Gemini script. Communicating with the backend was difficult because of the Flask app configuration required for our specific project and the changes in various versions of Flask and React Native made it harder to find helpful resources. Additionally, the asynchronous functions in React Native that were used for communicating with the backend Flask App were difficult to use. However, we did not give up and built a fully functional app, learning more about app development and generative AI along the way! ### Accomplishments we're proud of We’re proud of our ability to persevere and create a functional app to the best of our abilities given the limited time we had. At certain points, we felt like giving up and cutting corners when we faced bugs that we didn't understand while working on the project until 5 am, but we refused to do either. As a result, we created an app we are proud of, an app that utilizes AI to solve the problem of food waste, and an app that we want to continue developing in the future! ### What we learned We learned how generative AI has improved in recent developments with technologies such as Gemini's grounding and the RAG technique. Generative AI's ability to 'hallucinate' answers is a concerning limiting factor for its application and we saw firsthand how new developments can reduce this concern! We also gained a ton of experience using Native React and Flask for our full-stack application! Additionally, we learned skills just as important as these tools such as taking a brainstormed project idea and starting the development process to create a real product, how to be patient despite (very frustrating) bugs, and how to utilize the tools we have to create something amazing! ### What's next In the short term, we want to improve Grocery Tracker's UI to make it more attractive and accessible for users. For the long term, we want to continue experimenting with developing generative AI models and techniques to improve the accuracy and add features for Grocery Tracker including an idea to generate shopping lists based on missing ingredients in recipes that users want to use and prioritizing recipe generation for grocery items that are about to go bad! ## README (from the GitHub repository) Grocery Tracker provides a simple, fast solution to keeping track of all your groceries! With just a picture of your receipts, Grocery Tracker will identify your food items and list them with a countdown of their expected shelf life using Gemini's Generative AI Grounded with Google Search. After adding some items to the grocery list, Grocery Tracker can generate example recipes using the ingredients in your list! These features are implemented in a React Native App with Flask for the backend (The actual backend is in a folder named 'backend' in the GroceryTracker folder). ![1000009306](https://github.com/user-attachments/assets/f11bb5b6-a1fa-4909-9e38-dd92108ee0ab) ![1000009305](https://github.com/user-attachments/assets/07a00328-b199-44a2-b612-afd0486f4d3f) ![1000009307](https://github.com/user-attachments/assets/8176575c-1cf0-46de-ae41-741d27475e37) ## Detected evidence (automated analysis) Indexed codebase: 33 recognized source files, 55 KB. - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - Flask (technology) — claimed on Devpost, not found in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (38 of 38) ``` Backend/server.py GroceryTracker/.gitignore GroceryTracker/app-example/_layout.tsx GroceryTracker/app-example/(tabs)/_layout.tsx GroceryTracker/app-example/(tabs)/explore.tsx GroceryTracker/app-example/(tabs)/index.tsx GroceryTracker/app-example/+html.tsx GroceryTracker/app-example/+not-found.tsx GroceryTracker/app.json GroceryTracker/app/_layout.tsx GroceryTracker/app/(tabs)/_layout.tsx GroceryTracker/app/(tabs)/grocery-list.tsx GroceryTracker/app/(tabs)/index.tsx GroceryTracker/app/(tabs)/recipes.tsx GroceryTracker/app/+not-found.tsx GroceryTracker/babel.config.js GroceryTracker/backend/main.py GroceryTracker/backend/server.py GroceryTracker/components/__tests__/__snapshots__/ThemedText-test.tsx.snap GroceryTracker/components/__tests__/ThemedText-test.tsx GroceryTracker/components/Button.tsx GroceryTracker/components/Collapsible.tsx GroceryTracker/components/ExternalLink.tsx GroceryTracker/components/HelloWave.tsx GroceryTracker/components/ImageViewer.tsx GroceryTracker/components/navigation/TabBarIcon.tsx GroceryTracker/components/ParallaxScrollView.tsx GroceryTracker/components/ThemedText.tsx GroceryTracker/components/ThemedView.tsx GroceryTracker/constants/Colors.ts GroceryTracker/hooks/useColorScheme.ts GroceryTracker/hooks/useColorScheme.web.ts GroceryTracker/hooks/useThemeColor.ts GroceryTracker/package.json GroceryTracker/README.md GroceryTracker/scripts/reset-project.js GroceryTracker/tsconfig.json README.md ``` ### Dependencies - GroceryTracker/package.json: @babel/core@^7.20.0, @expo/vector-icons@^14.0.2, @react-native-async-storage/async-storage@^1.23.1, @react-navigation/native@^6.0.2, @types/jest@^29.5.12, @types/react@~18.2.45, @types/react-test-renderer@^18.0.7, axios@^1.7.7, expo@^51.0.38, expo-constants@~16.0.2, expo-font@~12.0.9, expo-image@~1.13.0, expo-image-picker@~15.0.7, expo-linking@~6.3.1, expo-router@~3.5.23, expo-splash-screen@~0.27.5, expo-status-bar@~1.12.1, expo-system-ui@~3.0.7, expo-web-browser@~13.0.3, jest@^29.2.1, jest-expo@~51.0.3, react@18.2.0, react-dom@18.2.0, react-native@0.74.5, react-native-gesture-handler@~2.16.1, react-native-reanimated@~3.10.1, react-native-safe-area-context@4.10.5, react-native-screens@3.31.1, react-native-web@~0.19.10, react-test-renderer@18.2.0, typescript@~5.3.3 ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md with pictures - Create README.md - Finishedgit add . - Merge branch 'master' of https://github.com/jparkk04/grocery-tracker - updated styles - Finished uploading image - Merge branch 'master' of https://github.com/jparkk04/grocery-tracker - to pull - fixed loading - Fixed Flask, working on adding grocery list items - Merge branch 'master' of https://github.com/jparkk04/grocery-tracker - changed things on my end - improved frontend - finished grocery-list - commit to merge - commit to pull - Started implementing picture taking - commit to pull ## Key source files (fetched from GitHub, selected and truncated for size) ### GroceryTracker/package.json ``` { "name": "grocerytracker", "main": "expo-router/entry", "version": "1.0.0", "scripts": { "start": "expo start", "reset-project": "node ./scripts/reset-project.js", "android": "expo start --android", "ios": "expo start --ios", "web": "expo start --web", "test": "jest --watchAll", "lint": "expo lint" }, "jest": { "preset": "jest-expo" }, "dependencies": { "@expo/vector-icons": "^14.0.2", "@react-native-async-storage/async-storage": "^1.23.1", "@react-navigation/native": "^6.0.2", "axios": "^1.7.7", "expo": "^51.0.38", "expo-constants": "~16.0.2", "expo-font": "~12.0.9", "expo-image": "~1.13.0", "expo-image-picker": "~15.0.7", "expo-linking": "~6.3.1", "expo-router": "~3.5.23", "expo-splash-screen": "~0.27.5", "expo-status-bar": "~1.12.1", "expo-system-ui": "~3.0.7", "expo-web-browser": "~13.0.3", "react": "18.2.0", "react-dom": "18.2.0", "react-native": "0.74.5", "react-native-gesture-handler": "~2.16.1", "react-native-reanimated": "~3.10.1", "react-native-safe-area-context": "4.10.5", "react-native-screens": "3.31.1", "react-native-web": "~0.19.10" }, "devDependencies": { "@babel/core": "^7.20.0", "@types/jest": "^29.5.12", "@types/react": "~18.2.45", "@types/react-test-renderer": "^18.0.7", "jest": "^29.2.1", "jest-expo": "~51.0.3", "react-test-renderer": "18.2.0", "typescript": "~5.3.3" }, "private": true } ``` ### Backend/server.py ```python from flask import Flask app = Flask(__name__) ``` ### GroceryTracker/app/_layout.tsx ```typescript import { Stack } from "expo-router"; export default function RootLayout() { return ( ); } ``` ### GroceryTracker/backend/server.py ```python from flask import Flask, jsonify, request from flask_cors import CORS from main import food_expiration, generate_recipe import base64 import os # Initializing flask app app = Flask(__name__) cors = CORS(app, origins='*') @app.route('/') def hello_world(): return "hello world!" @app.route('/upload', methods=['POST']) def upload_image(): try: print('hi') data = request.get_json() image_base64 = data['image'] image_data = base64.b64decode(image_base64) filename = 'image.jpg' with open(filename, 'wb') as f: f.write(image_data) return jsonify({'message': 'Upload success!'}), 200 except Exception as e: return jsonify({'error', str(e)}), 500 @app.route('/generate_recipe', methods=['GET']) def script0(): s = request.args.get('s') print("generating recipes...") result = generate_recipe(s) return result @app.route('/food_expiration/', methods=['GET']) def script1(s): print("generating food expiration times...") result = food_expiration(s) return result # Running app if __name__ == '__main__': app.run(debug=True, host='0.0.0.0', port=5000) # from flask import Flask, jsonify # from flask_cors import CORS # app = Flask(__name__) # cors = CORS(app, origins='*') # @app.route('/') # def hello_wrld(): # return jsonify({"output": ["a", "b", "c"]}) # @app.route('/api/data') # def hello_world(): # print('asdf') # return jsonify({"output": ["a", "b", "c"]}) # if __name__ == '__main__': # app.run(debug=True, host='0.0.0.0', port=5000) ``` ### GroceryTracker/backend/main.py ```python import google.generativeai as genai import os import json import vertexai from vertexai.generative_models import ( GenerationConfig, GenerativeModel, Tool, grounding, Part, ) def food_expiration(image_file): genai.configure(api_key=os.environ["API_KEY"]) myfile = genai.upload_file("C:\\Users\\noahm\\OneDrive\\Desktop\\Calhacks\\GroceryTrackerRepo\\grocery-tracker\\GroceryTracker\\" + image_file, mime_type='image/jpeg') model = genai.GenerativeModel("gemini-1.5-flash") result = model.generate_content( [myfile, "\n\n", "Identify the food items from the text of this grocery receipt. If adjectives come after the food items, write the food items with the adjectives in front. If there are abbreviations to food items, write the full name of the food item. Return a list of the food items."] ) # print(result.text) # response1 = model.generate_content([result.text, "\n\n", "Can you make a list of the estimated shelf life for each food item along with the best method of storing these items? Make sure the shelf life value is a single integer representing the number of days and that the storing method is either Refrigerator, Countertop, or Freezer?"]) # print(response1.text) PROJECT_ID = "winter-inquiry-439119-v8" vertexai.init(project=PROJECT_ID, location="us-central1") model = GenerativeModel("gemini-1.5-flash-001") # image_file = Part.from_uri( # image_file, mime_type # ) # result = model.generate_content( # [image_file, "Identify the food items from the text of this grocery receipt. If adjectives come after the food items, write the food items with the adjectives in front. If there are abbreviations to food items, write the full name of the food item. Return a list of the food items."] # ) tool = Tool.from_google_search_retrieval(grounding.GoogleSearchRetrieval()) prompt1 = ("Make a list of the estimated shelf life for each food item, making sure that the shelf life " + "is a single number representing the average number of days, for the following food items. Return " + "it in a python dictionary format with each food item mapping to an integer representing the estimated shelf life" + result.text) response1 = model.generate_content( prompt1, tools=[tool], generation_config=GenerationConfig( temperature=0.0, ), ) # print(response1.text) return response1.text def generate_recipe(s): PROJECT_ID = "winter-inquiry-439119-v8" vertexai.init(project=PROJECT_ID, location="us-central1") tool = Tool.from_google_search_retrieval(grounding.GoogleSearchRetrieval()) model = GenerativeModel("gemini-1.5-flash-001") prompt2 = "Give 1 recipe including some but not necessarily all ingredients given in this list. Return it in a python dictionary format with 'title' mapping to the name of the recipe and 'description' mapping to the ingredients and directions of the recipe." + s response2 = model.generate_content( prompt2, tools=[tool], generation_config=GenerationConfig( temperature=0.0, ), ) # print(response2.text) return response2.text # def main(): # x = food_expiration("receipt.jpeg") # s = generate_recipe(x) # # data = { # # "message": s, # # "status": "success" # # } # return x, s # if __name__ == "__main__": # print(main()) # from PIL import Image # import pytesseract # pytesseract.pytesseract.tesseract_cmd = r'C:\Program Files\Tesseract-OCR\tesseract' # print(pytesseract.image_to_boxes(Image.open('images/receipt.jpeg'))) # # Get verbose data including boxes, confidences, line and page numbers # print(pytesseract.image_to_data(Image.open('images/receipt.jpeg'))) # # Get information about orientation and script detection # print(pytesseract.image_to_osd(Image.open('images/receipt.jpeg'))) # s = pytesseract.image_to_string(Image.open('images/receipt.jpeg')) # service_account_file_name = 'service_account_key.json' # from google.oauth2 import service_account # credentials = service_account.Credentials.from_service_account_file(service_account_file_name) # scoped_credentials = credentials.with_scopes( # ['https://www.googleapis.com/auth/cloud-platform', 'https://www.googleapis.com/auth/generative-language.retriever']) # import google.ai.generativelanguage as glm # generative_service_client = glm.GenerativeServiceClient(credentials=scoped_credentials) # retriever_service_client = glm.RetrieverServiceClient(credentials=scoped_credentials) # permission_service_client = glm.PermissionServiceClient(credentials=scoped_credentials) # example_corpus = glm.Corpus(display_name="Recipe Generator") # create_corpus_request = glm.CreateCorpusRequest(corpus=example_corpus) # # Make the request # create_corpus_response = retriever_service_client.create_corpus(create_corpus_request) # # Set the `corpus_resource_name` for subsequent sections. # corpus_resource_name = create_corpus_response.name # get_corpus_request = glm.GetCorpusRequest(name=corpus_resource_name) # # Make the request # get_corpus_response = retriever_service_client.get_corpus(get_corpus_request) # # Create a document with a custom display name. # example_document = glm.Document(display_name="TasteOfHomeRecipes") # # Add metadata. # # Metadata also supports numeric values not specified here # document_metadata = [ # glm.CustomMetadata(key="url", string_value="https://www.tasteofhome.com/collection/our-100-highest-rated-recipes-ever")] # example_document.custom_metadata.extend(document_metadata) # # Make the request # # corpus_resource_name is a variable set in the "Create a corpus" section. # create_document_request = glm.CreateDocumentRequest(parent=corpus_resource_name, document=example_document) # create_document_response = retriever_service_client.create_document(create_document_request) # # Set t [truncated — 2233 more characters] ``` ### GroceryTracker/app/(tabs)/_layout.tsx ```typescript import { Tabs } from 'expo-router'; import Ionicons from '@expo/vector-icons/Ionicons'; export default function TabLayout() { return ( ( ), }} /> ( ), }} /> ( ), }} /> ); } ``` ### GroceryTracker/app-example/(tabs)/index.tsx ```typescript import { Image, StyleSheet, Platform } from 'react-native'; import { HelloWave } from '@/components/HelloWave'; import ParallaxScrollView from '@/components/ParallaxScrollView'; import { ThemedText } from '@/components/ThemedText'; import { ThemedView } from '@/components/ThemedView'; export default function HomeScreen() { return ( }> Welcome! Step 1: Try it Edit app/(tabs)/index.tsx to see changes. Press{' '} {Platform.select({ ios: 'cmd + d', android: 'cmd + m' })} {' '} to open developer tools. Step 2: Explore Tap the Explore tab to learn more about what's included in this starter app. Step 3: Get a fresh start When you're ready, run{' '} npm run reset-project to get a fresh{' '} app directory. This will move the current{' '} app to{' '} app-example. ); } const styles = StyleSheet.create({ titleContainer: { flexDirection: 'row', alignItems: 'center', gap: 8, }, stepContainer: { gap: 8, marginBottom: 8, }, reactLogo: { height: 178, width: 290, bottom: 0, left: 0, position: 'absolute', }, }); ``` ### GroceryTracker/app/(tabs)/index.tsx ```typescript import { Text, View, StyleSheet } from "react-native"; import { Link } from "expo-router"; import Button from '@/components/Button'; import ImageViewer from '@/components/ImageViewer'; import * as ImagePicker from 'expo-image-picker'; import * as FileSystem from 'expo-file-system'; import { ThemedText } from '@/components/ThemedText'; import { ThemedView } from '@/components/ThemedView'; import { useState, useEffect } from 'react'; import { useFocusEffect} from '@react-navigation/native'; import React from 'react'; import axios from 'axios'; import AsyncStorage from '@react-native-async-storage/async-storage'; export default function Index({}) { const [selectedImage, setSelectedImage] = useState(undefined); // const [output, setOutput] = useState(''); // let i = 1; const [groceryList, setGroceryList] = useState( [] ) let nextId = groceryList.length==0 ? 0 : Math.max(...groceryList.map((item) => item.id)) + 1; useFocusEffect( React.useCallback(() => { loadList(); }, []) ); useEffect(() => { saveList(); }, [groceryList]); useEffect(() => { sendImage().then(() => runScript0()); }, [selectedImage]); // useEffect(() => { // runScript0(); // }, [selectedImage]); const sendImage = async () => { if (selectedImage) { try { const base64 = await FileSystem.readAsStringAsync(selectedImage, { encoding: 'base64' }); const response = await fetch('http://192.168.212.103:5000/upload', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ image: base64 }) }); if (response.ok) { // Handle successful upload console.log('Image uploaded successfully!'); } else { // Handle upload error console.error('Image upload failed.'); } } catch (error) { console.error('Error sending image:', error); } } } const AddToList = (item: string, days: number) => { if (item && days) { setGroceryList(groceryList => {const newList = [...groceryList, { name: item, days: days, id: nextId }]; // console.log("newList:" + newList); // Log the new list return newList;} ); nextId += 1; } } const runScript0 = async () => { if (selectedImage) { try { const url = 'http://192.168.212.103:5000/' + `food_expiration/image.jpg`; const response = await axios.get(url); // setOutput(response.data); // console.log(response.data); let x = response.data.split("{")[1].split("}")[0] for (const line of x.split(",")) { let y = line.split(":"); AddToList(y[0].replace(/['"]+/g, ''), parseInt(y[1])); } } catch (error) { console.error(error); } } }; const saveList = async () => { try { const jsonList = JSON.stringify(groceryList); await AsyncStorage.setItem('groceryList', jsonList); } catch (error) { console.error('Error saving list:', error); } }; const loadList = async () => { try { const jsonList = await AsyncStorage.getItem('groceryList'); if (jsonList !== null) { const loadedList = JSON.parse(jsonList); setGroceryList(loadedList); } else { setGroceryList([]); } } catch (error) { console.error('Error loading list:', error); } }; // const runScript0 = async () => { // try { // const url = 'http://10.41.22.170:5000/' + `food_expiration/receipt${i}.jpg`; // const options = {method: 'GET'}; // console.log(url); // const response = await fetch(url, options); // Use your Flask server URL // if (!response.ok) { // console.error('An error occured when requesting to confirm a referal'); // return; // } // console.log(response); // const data = await response.json(); // setOutput(data.output); // console.log(data.output); // } catch (error) { // console.error(error); // } // }; // const uploadImageAsync = async () => { // let formData = new FormData(); // //append created photo{} to formdata // formData.append('image', selectedImage); // //use axios to POST // axios({ // method: 'POST', // url: api_url + 'customer/upload-avatar', // data: formData, // headers: { // 'Authorization': "Bearer " + YOUR_BEARER_TOKEN, // 'Accept': 'application/json', // 'Content-Type': 'multipart/form-data;' // }}) .then(function (response) { console.log(response)}) // .catch(function (error) { console.log(error.response) // }); // } const takeImageAsync = async () => { let result = await ImagePicker.launchCameraAsync({ allowsEditing: true, quality: 1, cameraType: ImagePicker.CameraType.back, }); if (!result.canceled) { setSelectedImage(result.assets[0].uri); console.log(selectedImage); } }; const pickImageAsync = async () => { let result = await ImagePicker.launchImageLibraryAsync({ allowsEditing: true, quality: 1, }); if (!result.canceled) { setSelectedImage(result.assets[0].uri); console.log(selectedImage); } }; return ( Scan your receipt!