# Project export: Vision Mama: LLMs + Vision Pro + Agents = Cooking Magic

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2024
- Tagline: Remember Cooking Mama? We turned it into a Conversational Agent for Vision Pro that teaches cooking! We pretrained & fine-tuned recipe making LLMs, made a food ordering agent, semantic recipes,...
- Devpost: https://devpost.com/software/vision-mama-llm-vision-pro-agents-fun-learning
- GitHub: https://github.com/andrewgcodes/treehacks2024
- Demo: https://recipes.reflex.run/
- Video: https://www.youtube.com/embed/hyWJAuR7EVY?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Codegen: Best Use of AI Agents ($3k Cash); Best Use of Monster Generative AI APIs (4x XBox Series S [1st] & 1 million Monster API credits [2nd] & $400 Cash [3rd]))
- Team: 1 GitHub contributor(s) — Andrew Kean Gao (32 commits)

## Devpost submission (written by the team)

### Overview

The future of computing 🍎 👓 ⚙️ 🤖 🍳 👩‍🍳 How could Mixed Reality, Spatial Computing, and Generative AI transform our lives? And what happens when you combine Vision Pro and AI? (spoiler: magic! 🔮) Our goal was to create an interactive VisionOS app 🍎 powered by AI. While our app could be applied towards many things (like math tutoring, travel planning, etc.), we decided to make the demo use case fun. We loved playing the game Cooking Mama 👩‍🍳 as kids so we made a voice-activated conversational AI agent that teaches you to cook healthy meals, invents recipes based on your preferences, and helps you find and order ingredients. Overall, we want to demonstrate how the latest tech advances could transform our lives. Food is one of the most important, basic needs so we felt that it was an interesting topic. Additionally, many people struggle with nutrition so our project could help people eat healthier foods and live better, longer lives. What we created Conversational Vision Pro app that lets you talk to an AI nutritionist that speaks back to you in a realistic voice with low latency. Built-in AI agent that will create a custom recipe according to your preferences, identify the most efficient and cheapest way to purchase necessary ingredients in your area (least stores visited, least cost), and finally creates Instacart orders using their simulated API. Web version of agent at recipes.reflex.run in a chat interface InterSystems IRIS vector database of 10k recipes with HyDE enabled semantic search Pretrained 40M LLM from scratch to create recipes Fine-tuned Mistral-7b using MonsterAPI to generate recipes

### How we built it

We divided tasks efficiently given the time frame to make sure we weren't bottlenecked by each other. For instance, Gao's first priority was to get a recipe LLM deployed so Molly and Park could use it in their tasks. While we split up tasks, we also worked together to help each other debug and often pair programmed and swapped tasks if needed. Various tools used: Xcode, Cursor, OpenAI API, MonsterAI API, IRIS Vector Database, Reflex.dev, SERP API,... Vision OS Talk to Vision Mama by running Whisper fully on device using CoreML and Metal Chat capability powered by GPT-3.5-turbo, our custom recipe-generating LLM (Mistral-7b backbone), and our agent endpoint. To ensure that you are able to see both Vision Mama's chats and her agentic skills, we have a split view that shows your conversation and your generated recipes Lastly, we use text-to-speech synthesis using ElevenLabs API for Vision Mama's voice AI Agent Pipeline for Recipe Generation, Food Search, and Instacart Ordering We built an endpoint that we hit from our Vision Pro and our Reflex site. Basically what happens is we submit a user's desired food such as "banana soup". We pass that to our fine-tuned Mistral-7b LLM to generate a recipe. Then, we quickly use GPT-4-turbo to parse the recipe and extract the ingredients. Then we use the SERP API on each ingredient to find where it can be purchased nearby. We prioritize cheaper ingredients and use an algorithm to try to visit the least number of stores to buy all ingredients. Finally, we populate an Instacart Order API call to purchase the ingredients (simulated for now since we do not have actual partner access to Instacart's API) Pre-training (using nanogpt architecture): Created large dataset of recipes. Tokenized our recipe dataset using BPE (GPT2 tokenizer) Dataset details (9:1 split): train: 46,826,468 tokens val: 5,203,016 tokens Trained for 1000 iterations with settings: layers = 12 attention heads = 12 embedding dimension = 384 batch size = 32 In total, the LLM had 40.56 million parameters! It took several hours to train on an M3 Mac with Metal Performance Shaders. Fine-tuning While the pre-trained LLM worked ok and generated coherent (but silly) English recipes for the most part, we couldn't figure out how to deploy it in the time frame and it still wasn't good enough for our agent. So, we tried fine-tuning Mistral-7b, which is 175 times bigger and is much more capable. We curated fine-tuning datasets of several sizes (10k recipes, 50k recipes, 250k recipes). We prepared them into a specific prompt/completion format: We fine-tuned and deployed the 250k-fine-tuned model on the MonsterAPI platform, one of the sponsors of TreeHacks. We observed that using more fine-tuning data led to lower loss, but at diminishing returns. Reflex.dev Web Agent Most people don't have Vision Pros so we wrapped our versatile agent endpoint into a Python-based Reflex app that you can chat with! Try here Note that heavy demand may overload our agent. IRIS Semantic Recipe Discovery We used the IRIS Vector Database, running it on a Mac with Docker. We embedded 10,000 unique recipes from diverse cuisines using OpenAI's text-ada-002 embedding. We stored the embeddings and the recipes in an IRIS Vector Database. Then, we let the user input a "vibe", such as "cold rainy winter day". We use Mistral-7b to generate three Hypothetical Document Embedding (HyDE) prompts in a structured format. We then query the IRIS DB using the three Mistral-generated prompts. The key here is that regular semantic search does not let you search by vibe effectively. If you do semantic search on "cold rainy winter day", it is more likely to give you results that are related to cold or rain, rather than foods. Our prompting encourages Mistral to understand the vibe of your input and convert it to better HyDE prompts. Real example: User input: something for a chilly winter day Generated Search Queries: {'queries': ['warming winter dishes recipes', 'comfort food recipes for cold days', 'hearty stews and soups for chilly weather']} Result: recipes that match the intent of the user rather than the literal meaning of their query

### Challenges we ran into

Programming for the Vision Pro, a new way of coding without that much documentation available Two of our team members wear glasses so they couldn't actually use the Vision Pro :( Figuring out how to work with Docker Package version conflicts :(( Cold starts on Replicate API A lot of tutorials we looked at used the old version of the OpenAI API which is no longer supported

### Accomplishments we're proud of

Learning how to hack on Vision Pro! Making the Vision Mama 3D model blink Pretraining a 40M parameter LLM Doing fine-tuning experiments Using a variant of HyDE to turn user intent into better semantic search queries

### What we learned

How to pretrain LLMs and adjust the parameters How to use the IRIS Vector Database How to use Reflex How to use Monster API How to create APIs for an AI Agent How to develop for Vision Pro How to do Hypothetical Document Embeddings for semantic search How to work under pressure

### What's next

for Vision Mama: LLM + Vision Pro + Agents = Fun & Learning Improve the pre-trained LLM: MORE DATA, MORE COMPUTE, MORE PARAMS!!! Host the InterSystems IRIS Vector Database online and let the Vision Mama agent query it Implement the meal tracking photo analyzer into VisionOs app Complete the payment processing for the Instacart API once we get developer access Impacts Mixed reality and AI could enable more serious use cases like: Assisting doctors with remote robotic surgery Making high quality education and tutoring available to more students Amazing live concert and event experiences remotely Language learning practice partner Concerns Vision Pro is very expensive so most people can't afford it for the time being. Thus, edtech applications are limited. Data privacy Thanks for checking out Vision Mama!

## README (from the GitHub repository)

# Vision Mama!
TreeHacks 2024 project.
**Scroll down** for details.
![VisionMama Photo](https://github.com/andrewgcodes/treehacks2024/blob/main/visionmama-photo.png?raw=true)

## Table of contents
| Section | Description |
|---------|-------------|
| [**Vision OS App**](#vision-os-app) | Instructions for downloading and extracting the visionmama.zip file for the Vision OS App. |
| [**AI Agent Pipeline for Recipe Generation, Food Search, and Instacart Ordering**](#ai-agent-pipeline-for-recipe-generation-food-search-and-instacart-ordering) | Details on the pipeline from a user's food request to generating a recipe, identifying ingredients, finding purchase locations, and creating an Instacart order. Utilizes a fine-tuned Mistral-7b LLM, GPT-4-turbo, SERP API, and a sophisticated ranking algorithm. |
| [**Pre-training**](#pre-training) | Information on dataset preparation, pre-training process, and decision-making regarding the use of a fine-tuned Mistral-7b model. |
| [**Fine-tuning**](#fine-tuning) | Describes LORA fine-tuning of Mistral-7b with 250k recipes, settings used, and dataset size impact on performance. |
| [**Reflex.dev Web Chat Agent**](#reflexdev-web-chat-agent) | Using Reflex.dev to create a chat interface for interacting with the AI agent, including triggers for recipe generation and ingredient identification. |
| [**InterSystems IRIS Vector Database for Semantic Recipe Discovery**](#intersystems-iris-vector-database-for-semantic-recipe-discovery) | Use of the IRIS Vector Database for recipe embeddings and semantic searches based on user "vibe" inputs. |

## Vision OS App
In the visionmama.zip file, please download and extract [this](https://github.com/andrewgcodes/treehacks2024/blob/main/VisionMama.zip)

## AI Agent Pipeline for Recipe Generation, Food Search, and Instacart Ordering
We built an endpoint that we hit from our Vision Pro and our Reflex site.
Basically what happens is we submit a user's desired food such as "banana soup". We pass that to our fine-tuned Mistral-7b LLM to generate a recipe. Then, we quickly use GPT-4-turbo to parse the recipe and extract the ingredients. Then we use the SERP API on each ingredient to find where it can be purchased nearby. We prioritize cheaper ingredients and use an algorithm to try to visit the least number of stores to buy all ingredients. Finally, we populate an Instacart Order API call to purchase the ingredients (simulated for now since we do not have actual partner access to Instacart's API)

## Pre-training:
We found a dataset online of 250,000 recipes. We preprocessed them and split and tokenized them for pretraining.
We used the GPT2 Byte Pair Encoding tokenizer.
We trained our **40M parameter LLM** using modified [nanogpt implementation](https://github.com/karpathy/nanoGPT)
We didn't have time to figure out how to deploy the LLM so we went with our fine-tuned Mistral-7b model (which also performed better).
More details on our devpost.

## Fine-tuning:
We **LORA fine-tuned Mistral-7b** using MonsterAPI's online platform: MonsterAPI.ai. (Thank you to the team for giving us free credits!)
Settings: one epoch, Lora R = 8, Lora Alpha = 16, Dropout = 0, Bias = none, Gradient accumulation steps = 32, Lr = 0.0002, warmup steps = 100

Before fine-tuning, we prepared **250k recipes** we got from online into a standard instruct format using this script: prepareRecipesForFinetuning.py
The format is:
You are an expert chef. You know about a lot of diverse cuisines. You write helpful tasty recipes.\n\n###Instruction: please think step by step and generate a detailed recipe for {prompt}\n\n###Response:{completion}

We also lowercased all prompts and completions.
We experimented with **fine-tuning using 10k, 50k, and 250k recipes.**
We observed that using more data led to lower loss, but at diminishing returns.
We deployed our fine-tuned Mistral-7b (250k examples) using MonsterAPI.ai
The script finetuned-mistral7b-monsterapi.py demonstrates how we call the fine-tuned model as well as process the output into a standardized format using regex and string processing methods.

## Reflex.dev Web Chat Agent
We used Reflex.dev, which is like React but entirely in Python, to create a simple chat interface to interact with our agent, because most people do not have a Vision Pro.
We run GPT-3.5-turbo that is prompt engineered to provide nutritional information to the user if they ask a question. However, if the user begins their chat message with "get me " and then an imaginary food, it **triggers our AI agent pipeline** which then calls our **fine-tuned Mistral-7b** to generate a recipe, **GPT-4-turbo** to process and extract ingredients from the recipe, and then **Google Search via SERP API** and a sophisticiated **multiobjective ranking algorithm** to identify the cheapest and best ingredients from the minimal number of stores, and finally populates **Instacart order API** calls.
We hosted it on reflex.dev which was easy. We just did reflex deploy and put in our env variable from the terminal! Thank you to reflex.

## InterSystems IRIS Vector Database for Semantic Recipe Discovery:
We used the early access version of the **IRIS Vector Database**, running it on a Mac with Docker.
We embedded 10,000 unique recipes from diverse cuisines using OpenAI's **text-ada-002 embedding**.
We stored the embeddings and the recipes in an IRIS Vector Database.
Then, we let the user input a "vibe", such as "cold rainy winter day".
We use Mistral-7b to generate three **Hypothetical Document Embedding** (HyDE) prompts in a structured format.
We then query the IRIS DB using the three Mistral-generated prompts.
The key here is that regular semantic search **does not** let you search by vibe effectively.
If you do semantic search on "cold rainy winter day", it is more likely to give you results that are related to cold or rain, rather than foods.
Our prompting encourages Mistral to understand teh vibe of your input and convert it to better HyDE prompts.
Real example:
User input: something for a chilly winter day
Generated Search Queries: {'queries': ['warming winter dishes recipes', 'comfort food recipes for cold days', 'hearty stews and soups for chilly weather']}


## Detected evidence (automated analysis)

Indexed codebase: 10 recognized source files, 28 KB.
- Python (language) — detected in the code
- Mistral AI (technology) — claimed on Devpost, not found in the code
- OpenAI (technology) — claimed on Devpost, not found in the code
- Swift (language) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (10 of 10)

```
finetuned-mistral7b-monsterapi.py
food_agent_chain_1.py
food_agent_chain_2.py
food_agent_chain_3.py
IRIS_VectorDatabase_MagicRecipeRecommender.py
loss_curve_visualizer.py
prepareRecipesForFinetuning.py
README.md
reflex_state.py
reflex_webui.py
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- Update README.md
- Update README.md
- Update README.md
- visionOS code (xcode project)
- Add VisionMama folder
- Create loss_curve_visualizer.py
- Update README.md
- Update README.md
- Create reflex_state.py
- Create reflex_webui.py
- Create food_agent_chain_3.py
- Create food_agent_chain_2.py
- Create food_agent_chain_1.py
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Create IRIS_VectorDatabase_MagicRecipeRecommender.py

## Key source files (fetched from GitHub, selected and truncated for size)

### loss_curve_visualizer.py

```python
import pandas as pd
import re
import matplotlib.pyplot as plt

path = 'recipeLoss2.txt'

iterations = []
losses = []

pattern = re.compile(r'iter (\d+): loss ([\d.]+),')

with open(path, 'r') as file:
    for line in file:
        match = pattern.search(line)
        if match:
            iterations.append(int(match.group(1)))
            losses.append(float(match.group(2)))

df = pd.DataFrame({
    'Iteration': iterations,
    'Loss': losses,
})

plt.figure(figsize=(12, 8))
plt.plot(df['Iteration'], df['Loss'], marker='o', linestyle='-', color='tab:red', markersize=1)
plt.title('Mini Recipe LM Training Loss Over Iterations')
plt.xlabel('Iteration')
plt.ylabel('Loss')
plt.grid(True)
plt.show()

```

### reflex_webui.py

```python
"""based on their chat template"""

import reflex as rx

from webui import styles
from webui.components import chat, modal, navbar, sidebar
from webui.state import State


def index() -> rx.Component:
    return rx.chakra.vstack(
        navbar(),
        rx.chakra.text("Built and hosted with Reflex.dev", text_align="center", font_size = "2em"), 
        rx.chakra.text("Under the hood: recipe-fine-tuned Mistral-7b and an AI Agent that creates recipes, searches for the cheapest ingredients near you, makes a detailed shopping list, and purchases the ingredients using Instacart API.", text_align="center", font_size = "1em"),
        rx.chakra.text("Say 'get me' + a creative food to trigger the agent.", text_align="center", font_size = "1em"),  
        chat.chat(),
        chat.action_bar(),
        sidebar(),
        modal(),
        bg=styles.bg_dark_color,
        color=styles.text_light_color,
        min_h="100vh",
        align_items="stretch",
        spacing="0",
    )


app = rx.App(style=styles.base_style)
app.add_page(index)

```

### prepareRecipesForFinetuning.py

```python
import pandas as pd

file_path = 'RAW_recipes 2.csv'
data = pd.read_csv(file_path)

def safe_lower(s):
    if pd.isna(s):
        return ""
    return str(s).lower()

def format_recipe_lowercase(row):
    ingredients = eval(safe_lower(row['ingredients']))
    formatted_ingredients = '\n'.join([f"{idx+1}. {ingredient}" for idx, ingredient in enumerate(ingredients)])

    steps = eval(safe_lower(row['steps']))
    formatted_steps = '\n'.join([f"{idx+1}. {step}" for idx, step in enumerate(steps)])

    recipe_name = safe_lower(row['name'])
    
    description = safe_lower(row['description'])
    recipe = (f"here is your recipe!\n"
              f"recipe name: {recipe_name}\n"
              f"description: {description}\n"
              f"minutes to make: {row['minutes']}\n"
              f"ingredients:\n{formatted_ingredients}\n"
              f"steps:\n{formatted_steps}\n").replace("  ", " ")
    return recipe

data['recipe'] = data.apply(format_recipe_lowercase, axis=1)

def create_prompt(row):
    recipe_name = safe_lower(row['name'])
    prompt_text = f"please think step by step and generate a detailed recipe for {recipe_name.replace('  ', ' ')}"
    return prompt_text

data['prompt'] = data.apply(create_prompt, axis=1)

print(data.loc[0, 'recipe'])
print("\nPrompt Example:\n", data.loc[0, 'prompt'])

top_10000_shortest_recipes = data.sort_values(by='minutes').head(50000)

top_10000_shortest_recipes.head()

```

### food_agent_chain_3.py

```python
from datetime import datetime

def handler(pd: "pipedream"):
    shopping_lists = pd.steps["python_1"]["$return_value"]["shopping_lists"]
    orders = []

    def generate_order_items(items):
        order_items = []
        for index, item in enumerate(items, start=1):
            order_item = {
                "line_num": str(index),
                "count": 1, 
                "special_instructions": "",
                "replacement_policy": "shoppers_choice",
                "item": {"upc": item["product_id"]}  
            }
            order_items.append(order_item)
        return order_items

    timestamp = datetime.now().strftime("%Y%m%d%H%M%S")

    for store, items in shopping_lists.items():
        order = {
            "order_id": f"order_{store.lower()}_{timestamp}",  
            "service_option_hold_id": 1, 
            "initial_tip_cents": 499, 
            "leave_unattended": False,
            "special_instructions": f"Deliver to front door please!  Items from {store}. Thank you",
            "location_code": store.lower(),
            "paid_with_ebt": False,
            "locale": "en-US",
            "applied_instacartplus": False,
            "user": {
                "birthday": "2003-11-05", 
                "phone_number": "+123456789",
                "sms_opt_in": True
            },
            "address": {
                "address_line_1": "459 Lagunita Drive",
                "address_line_2": "",
                "address_type": "residential",
                "postal_code": "94305"
            },
            "items": generate_order_items(items)
        }
        orders.append(order)

    return_value = {
        "orders": orders,
        "original_shopping_list": shopping_lists
    }
    
    return {"output":str(return_value).replace("'", '"')}

```

### finetuned-mistral7b-monsterapi.py

```python
import requests
import json

def generate_recipe(prompt):
    url = "https://f40e171a-ade7-40c5-913b-84db8e5fd1f8.monsterapi.ai/generate"
    api_auth_token = #AUTH TOKEN HERE
    
    headers = {
        "Authorization": f"Bearer {api_auth_token}",
        "Content-Type": "application/json"
    }

    data = {
        "prompt": f"You are an expert chef. You know about a lot of diverse cuisines. You write helpful tasty recipes.\n\n###Instruction: please think step by step and generate a detailed recipe for {prompt}\n\n###Response:",
        "stream": False,
        "max_tokens": 256,
        "n": 1,
        "best_of": 1,
        "presence_penalty": 0,
        "frequency_penalty": 0,
        "repetition_penalty": 1,
        "temperature": 0.7,
        "top_p": 1,
        "top_k": -1,
        "min_p": 0,
        "use_beam_search": False,
        "length_penalty": 1,
        "early_stopping": False
    }

    response = requests.post(url, headers=headers, data=json.dumps(data), verify=False)

    if response.status_code == 200:
        response_text = response.text
        start_str = "recipe name: "
        end_str = "token_counts"
        start_idx = response_text.find(start_str) + len(start_str)
        end_idx = response_text.find(end_str)
        recipe_text = response_text[start_idx:end_idx]
        recipe_text = recipe_text.replace("\\n", "\n").replace('\\"', '"')
        replaced_string = recipe_text.replace("\\\n", "\n")
        lines = replaced_string.split('\n')
        reversed_lines = list(reversed(lines))
        for line in reversed_lines:
            if any(char.isdigit() for char in line):
                index_to_remove = lines.index(line)
                lines = lines[:index_to_remove]
                break
        cleaned_text = '\n'.join(lines)
        final_string = cleaned_text.replace("\\\n", "\n")
        print(final_string)
        return final_string
    else:
        print(f"Error: {response.status_code}, Message: {response.text}")
        return "Error"

result = generate_recipe("grape spaghetti")

```

### food_agent_chain_2.py

```python
import json
import serpapi

def handler(pd: "pipedream"):
    chat_content = pd.steps["chat"]["$return_value"]["choices"][0]["message"]["content"]
    ingredients = json.loads(chat_content)["ingredients"]
    
    results = []
    all_items = {} 
    store_coverage = {}  

    for ingredient in ingredients:
        query = f"{ingredient} near me"
        params = {
            "engine": "google",
            "q": query,
            "tbm": "shop",
            "location": "Stanford, California",
            "api_key": # SERP API KEY
        }
        
        search = serpapi.search(params)
        search_results = search.as_dict()
        shopping_results = search_results.get("shopping_results", [])
        
        sorted_results = sorted(shopping_results, key=lambda x: x.get("extracted_price", float('inf')))[:10]
        all_items[ingredient] = sorted_results
        
        for item in sorted_results:
            source = item["source"]
            if source not in store_coverage:
                store_coverage[source] = set()
            store_coverage[source].add(ingredient)

    ingredients_needed = set(ingredients)
    optimal_stores = {}
    while ingredients_needed:
        best_store = None
        best_coverage = set()
        for store, coverage in store_coverage.items():
            current_coverage = coverage & ingredients_needed
            if len(current_coverage) > len(best_coverage):
                best_store = store
                best_coverage = current_coverage
        
        if not best_store:
            break 
        
        optimal_stores[best_store] = list(best_coverage)
        ingredients_needed -= best_coverage

    shopping_lists = {}
    for store, ingredients_list in optimal_stores.items():
        shopping_lists[store] = []
        for ingredient in ingredients_list:
            for item in all_items[ingredient]:
                if item["source"] == store:
                    shopping_lists[store].append(item)
                    break  

    return {
        "shopping_lists": shopping_lists
    }

```

### food_agent_chain_1.py

```python
import requests
import json

def generate_recipe(prompt):
    url = "https://f40e171a-ade7-40c5-913b-84db8e5fd1f8.monsterapi.ai/generate"
    api_auth_token = #AUTH TOKEN

    headers = {
        "Authorization": f"Bearer {api_auth_token}",
        "Content-Type": "application/json"
    }

    data = {
        "prompt": f"You are an expert chef. You know about a lot of diverse cuisines. You write helpful tasty recipes.\n\n###Instruction: please think step by step and generate a detailed recipe for {prompt}\n\n###Response:",
        "stream": False,
        "max_tokens": 256,
        "n": 1,
        "best_of": 1,
        "presence_penalty": 0,
        "frequency_penalty": 0,
        "repetition_penalty": 1,
        "temperature": 0.7,
        "top_p": 1,
        "top_k": -1,
        "min_p": 0,
        "use_beam_search": False,
        "length_penalty": 1,
        "early_stopping": False
    }

    response = requests.post(url, headers=headers, data=json.dumps(data), verify=False)

    if response.status_code == 200:
        response_text = response.text
        start_str = "recipe name: "
        end_str = "token_counts"
        start_idx = response_text.find(start_str) + len(start_str)
        end_idx = response_text.find(end_str)
        recipe_text = response_text[start_idx:end_idx]
        recipe_text = recipe_text.replace("\\n", "\n").replace('\\"', '"')
        replaced_string = recipe_text.replace("\\\n", "\n")
        lines = replaced_string.split('\n')
        reversed_lines = list(reversed(lines))
        for line in reversed_lines:
            if any(char.isdigit() for char in line):
                index_to_remove = lines.index(line)
                lines = lines[:index_to_remove]
                break
        cleaned_text = '\n'.join(lines)
        final_string = cleaned_text.replace("\\\n", "\n")
        print(final_string)
        return final_string
    else:
        print(f"Error: {response.status_code}, Message: {response.text}")
        return "Error"

def handler(pd: "pipedream"):
    print(pd.steps["trigger"]["event"]["body"]["desiredFood"])
    result = generate_recipe(pd.steps["trigger"]["event"]["body"]["desiredFood"])
    return {"generatedRecipe": result}

```

### IRIS_VectorDatabase_MagicRecipeRecommender.py

```python
import csv
from langchain_iris import IRISVector
from langchain.docstore.document import Document
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.embeddings import HuggingFaceEmbeddings, FastEmbedEmbeddings
import getpass
import os
import json
from dotenv import load_dotenv
load_dotenv(override=True)
from langchain.document_loaders import TextLoader
from langchain.text_splitter import CharacterTextSplitter
from langchain.embeddings.fastembed import FastEmbedEmbeddings
from langchain_iris import IRISVector
import replicate
from replicate.client import Client


os.environ["OPENAI_API_KEY"] = #API KEY

csv_file_path = "1000shortrecipes.csv"

documents = []
with open(csv_file_path, mode='r', encoding='utf-8') as file:
    reader = csv.reader(file)
    for row in reader:
        recipe_text = row[13]
        #print(recipe_text)
        documents.append(Document(page_content=recipe_text))

embeddings = OpenAIEmbeddings()

collection_name = 'recipes_collection2'
connection_string = "iris://SuperUser:SYS2@localhost:1972/USER"

db = IRISVector.from_documents(
    embedding=embeddings,
    documents=documents,
    collection_name=collection_name,
    connection_string=connection_string,
)

def generate_search_queries(user_query):
    prompt = f"I want to find some food that matches this description or would fit the vibe: {user_query}. Please come up with exactly three possible queries that could lead to recipes with a similar vibe to what I want. Use this format {{\"queries\": [\"query1\", \"query2\", \"query3\"]}}. Do not say ANYTHING else other than the JSON format. Your response should begin with {{\"queries\":"
    replicate = Client(api_token=#API)
    print("Getting response from Mistral-7b...")
    try:
        response = replicate.run(
            "mistralai/mistral-7b-instruct-v0.2",
            input={
                "debug": False,
                "top_k": 50,
                "top_p": 0.9,
                "prompt": prompt,
                "temperature": 0.6,
                "max_new_tokens": 128,
                "min_new_tokens": -1,
                "prompt_template": "<s>[INST] {prompt} [/INST] ",
                "repetition_penalty": 1.15
            },
        )

        content_str = ''.join(list(response)).replace(', ]', ']') 
        print(content_str)

        content = json.loads(content_str) 
        print("Generated Search Queries:", content)
        search_queries = content["queries"]

    except json.JSONDecodeError:
        print("Failed to decode JSON from GPT response.")
        search_queries = []
    except KeyError:
        print("JSON does not contain 'queries' key.")
        search_queries = []

    return search_queries

def main():
    user_query = input("Enter your query: ")
    search_queries = generate_search_queries(user_query)
    print(search_queries)
    
        
    for query in search_queries:
        print(f"Results for query: '{query}'")
        
        docs_with_score = db.similarity_search_with_score(query)
        
        for doc, score in docs_with_score:
            if len(doc.page_content) > 30:
                print("-" * 80)
                print(f"Score: {score}")
                print(doc.page_content)
                print("-" * 80)

if __name__ == "__main__":
    main()

```

### reflex_state.py

```python
import os
import requests
import json
import openai
import reflex as rx
import re
openai.api_key = os.getenv("OPENAI_API_KEY")
openai.api_base = os.getenv("OPENAI_API_BASE", "https://api.openai.com/v1")


def generate_recipe(prompt):
    url = "https://f40e171a-ade7-40c5-913b-84db8e5fd1f8.monsterapi.ai/generate"
    api_auth_token = #AUTH TOKEN

    headers = {
        "Authorization": f"Bearer {api_auth_token}",
        "Content-Type": "application/json"
    }

    data = {
        "prompt": f"You are an expert chef. You know about a lot of diverse cuisines. You write helpful tasty recipes.\n\n###Instruction: please think step by step and generate a detailed recipe for {prompt}\n\n###Response:",
        "stream": False,
        "max_tokens": 256,
        "n": 1,
        "best_of": 1,
        "presence_penalty": 0,
        "frequency_penalty": 0,
        "repetition_penalty": 1,
        "temperature": 0.7,
        "top_p": 1,
        "top_k": -1,
        "min_p": 0,
        "use_beam_search": False,
        "length_penalty": 1,
        "early_stopping": False
    }

    response = requests.post(url, headers=headers, data=json.dumps(data), verify=False)

    if response.status_code == 200:
        response_text = response.text
        start_str = "recipe name: "
        end_str = "token_counts"
        start_idx = response_text.find(start_str) + len(start_str)
        end_idx = response_text.find(end_str)
        recipe_text = response_text[start_idx:end_idx]
        recipe_text = recipe_text.replace("\\n", "\n").replace('\\"', '"')
        replaced_string = recipe_text.replace("\\\n", "\n")
        lines = replaced_string.split('\n')
        reversed_lines = list(reversed(lines))
        for line in reversed_lines:
            if any(char.isdigit() for char in line):
                index_to_remove = lines.index(line)
                lines = lines[:index_to_remove]
                break
        cleaned_text = '\n'.join(lines)
        final_string = cleaned_text.replace("\\\n", "\n")
        print(final_string)
        return final_string
    else:
        print(f"Error: {response.status_code}, Message: {response.text}")
        return "Error"
    
def get_access_token():
    """
    :return: access_token
    """
    url = "https://aip.baidubce.com/oauth/2.0/token"
    params = {
        "grant_type": "client_credentials",
        "client_id": BAIDU_API_KEY,
        "client_secret": BAIDU_SECRET_KEY,
    }
    return str(requests.post(url, params=params).json().get("access_token"))


class QA(rx.Base):
    """A question and answer pair."""

    question: str
    answer: str


DEFAULT_CHATS = {
    "Recipe Wizard": [],
}


class State(rx.State):
    """The app state."""

    chats: dict[str, list[QA]] = DEFAULT_CHATS

    current_chat = "Recipe Wizard"

    question: str

    processing: bool = False

    new_chat_name: str = ""

    drawer_open: bool = False

    modal_open: bool = False

    api_type: str = "baidu" if BAIDU_API_KEY else "openai"

    dialog_open: bool = False

    def create_chat(self):
        """Create a new chat."""
        self.current_chat = self.new_chat_name
        self.chats[self.new_chat_name] = []

        self.modal_open = False

    def toggle_modal(self):
        """Toggle the new chat modal."""
        self.modal_open = not self.modal_open

    def toggle_drawer(self):
        """Toggle the drawer."""
        self.drawer_open = not self.drawer_open

    def delete_chat(self):
        """Delete the current chat."""
        del self.chats[self.current_chat]
        if len(self.chats) == 0:
            self.chats = DEFAULT_CHATS
        self.current_chat = list(self.chats.keys())[0]
        self.toggle_drawer()

    def set_chat(self, chat_name: str):
        """Set the name of the current chat.

        Args:
            chat_name: The name of the chat.
        """
        self.current_chat = chat_name
        self.toggle_drawer()

    @rx.var
    def chat_titles(self) -> list[str]:
        """Get the list of chat titles.

        Returns:
            The list of chat names.
        """
        return list(self.chats.keys())
    async def process_question(self, form_data: dict[str, str]):
        question = form_data["question"]

        if question == "":
            return
        self.dialog_open = True

        if question.lower().startswith("make me"):
            food_item = question[21:].strip()
            
            recipe = generate_recipe(food_item)
            qa = QA(question=question, answer=recipe)
            self.chats[self.current_chat].append(qa)
            self.chats = self.chats
            return
        if question.lower().startswith("get me"):
            food_item = question[7:].strip()
            
            response = requests.post(
                "https://eofrhbmhilleja4.m.pipedream.net",
                headers={"Content-Type": "application/json"},
                data=json.dumps({"desiredFood": food_item}),
            )

            
            data = response.json()
            recipe = data["generated_recipe"]
            ingredients = data["ingredients"]
            shopping_list_data = data["instacart_and_shopping_list"]["original_shopping_list"]

            qa_recipe = QA(question=question, answer=recipe)
            qa_ingredients = QA(question=question, answer=ingredients)
            self.chats[self.current_chat].extend([qa_recipe, qa_ingredients])

            for store, items in shopping_list_data.items():
                store_message = f"Store: {store},\n"
                for item in items:
                    store_message += f" [Item: {item['title']}, Price: {item['price']}...]\n"
                qa_store = QA(question=question, answer=store_message)
                self.chats[self.current_chat].append(qa_store)

            self.chats = self.chats
            return
        if self.api_type == "openai":
            model = self.openai_process_question

        async for value in model(question):
         
[truncated — 1603 more characters]
```