# Project export: Sprout This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: Where imagination shapes learning - Devpost: https://devpost.com/software/sprout-io - GitHub: https://github.com/sdoshi4/Treehacks2025 - Demo: https://treehackssprout.flutterflow.app/ - Video: https://www.youtube.com/embed/GbGJ_mDDeCE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Flutterflow: Best Use of FlutterFlow ($2,000 Cash + meet w/ founders [1st] & $1,500 Cash [2nd] & iPad 10.9"s [3rd])) - Team: 3 GitHub contributor(s) — rheiformes (28 commits), Shaan Doshi (26 commits), sirihaasanallamothu28 (1 commits) ## Devpost submission (written by the team) ### Inspiration It all started with a TikTok video: a teacher shared how she turned her kid’s doodles into adventure books, which sparked curiosity and made the excitement in the classroom undeniable. ✨ What if we could bring that magic to every child, everywhere? The Challenge: Kids are at their peak in creativity, but are often bored and unengaged by the traditional method of passive reading. Parents and teachers need ways to keep students engaged in learning and gauge their understanding. The Solution: Sprout is the future of storytelling, interactive learning, and self expression. Through Sprout kids can: Upload their drawings from their imagination Read a Sprout generated interactive choose-your-own adventure story Be engaged through vivid imagery Learn vocabulary while having fun Get tested on their comprehension, adapted to their grade level With Sprout, children aren’t just reading stories—they’re living them. They upload their own drawings, personalize their adventures, and build language skills through play. Teachers can tailor stories to lessons, integrate vocabulary, and expand beyond just reading into science, history, and more. Sprout.io is a win-win. ### What it does Sprout starts with your student’s drawing. From there, we build an engaging, choose-your-own story, with generative visuals based on your input. The adventure is filled with grade-level vocabulary to hit comprehension goals, with a quick comprehension check at the end. ✨ Unleash your creativity Draw something magical- characters, a wild adventure, animals, your wildest dreams. 📸 Upload Your Drawing Snap a picture and upload it to Sprout.io. 📚 Pick Reading Level Pick grade level so the story and included vocabulary matches your reading comprehension. 🔎 Choose Your Own Adventure Read as the story unfolds with interactive choices, engaging visuals, and exciting vocabulary blended in along the way. Choose your own adventure 🧠 Test Your Knowledge Take a quiz at the end to see how far you have come and check your comprehension of the vocab words. How it works Simply go to: demo select your grade level, upload a drawing of your wildest creations, and click “Begin Your Story”! Enjoy this seamless learning experience and story as it unfolds, with vocab words integrated along the way. ### How we built it Sprout.io was developed using Flutterflow, for a seamless user experience. We integrated several AI-driven APIs: Google Cloud - Gemini Flash 2.0 AI Studio generates dynamic, age-appropriate storylines with the student’s image and a prompt we integrated. We specifically chose Gemini for the Multimodal 2.0 Flash model, which let us combine both visual and text inputs. We also utilized the multi-category content filtering Gemini provides to prioritize safety for kids on this platform. Luma API powers our image generation, bringing stories to life in the most creative way possible. We used the Dream Lab API and tinkered with the reference and styling weights to ensure that we could maintain consistent visual storylines, while also injecting a bit of creativity and unexpected cliff-hangers for students. FlutterFlow for our front-end! This tool was really helpful, since it let us visually diagram our control flow while also integrating our custom FastAPI endpoints for reactive generative AI, all with a seamless integration and clean UI perfect for kids! Imgur API handles quick image uploads within the FlutterFlow platform, passing them along to Luma’s API. Custom FastAPI Endpoints for our text and image generation and modifications, since our control-flow was pretty complex. Google Cloud and Docker, for persistent hosting of the backend solution. We also used ngrok and uvicorn for local testing. Perplexity for help with expanding our prompt engineering with its strong natural language and research capabilities Lots of free food, swag, and entertainment from the Stanford TreeHacks organizers and sponsors! Thank you! By combining these resources, Sprout creates a high value learning experience with personalized and interactive storytelling. Project link with code ### Challenges we ran into Front-End via Flutterflow: At the end of the day, we all know that a technical feat is not very impactful without consideration of the users. While our team had some strong back-end background, we are all beginners to FrontEnd. Thankfully, we found out about FlutterFlow. We spent time understanding how the platform works, figuring out how to add our API calls to our custom FastAPI endpoints, and getting our data formatted, encoded, and displayed onto each page. This took significant time, but led us to create a seamless product that runs on multiple devices, from phones to web screens. Front-End via Flutterflow: At the end of the day, we all know that a technical feat is not very impactful without consideration of the users. While our team had some strong back-end background, we are all beginners to FrontEnd. Thankfully, we found out about FlutterFlow. We spent time understanding how the platform works, figuring out how to add our API calls to our custom FastAPI endpoints, and getting our data formatted, encoded, and displayed onto each page. This took significant time, but led us to create a seamless product that runs on multiple devices, from phones to web screens. Hiding User-Visible Latency: We faced issues with latency of generative AI within our platform. Kids have short attention spans, and our goal is to keep them engaged as long as possible in an educational state of mind. This comes in conflict with the long time frames required for generative AI; for example, we noticed that each Luma Labs API query could take up to 10 seconds, and our Gemeni Flash 2.0 API queries via Studio also took roughly 8 seconds. To fix this issue, we generated possibilities of all future story trajectories 1 chapter ahead, allowing us to use the time the user spends reading to cover part of the future generation time. Even though we will not choose them all, doing some of this work concurrently while students are reading allows us to make our project faster for the end user. Hiding User-Visible Latency: We faced issues with latency of generative AI within our platform. Kids have short attention spans, and our goal is to keep them engaged as long as possible in an educational state of mind. This comes in conflict with the long time frames required for generative AI; for example, we noticed that each Luma Labs API query could take up to 10 seconds, and our Gemeni Flash 2.0 API queries via Studio also took roughly 8 seconds. To fix this issue, we generated possibilities of all future story trajectories 1 chapter ahead, allowing us to use the time the user spends reading to cover part of the future generation time. Even though we will not choose them all, doing some of this work concurrently while students are reading allows us to make our project faster for the end user. Protections & Safety via Gemini Filters: We’ve all seen Generative AI go off-the-rails with its creations, and that is particularly a situation we want to avoid when presenting to younger audiences. As a result, we specifically chose Gemini Flash 2.0 for the API’s adjustable safety filtering weights. We set our levels to explicitly filter out dangerous or harmful content for minors. Protections & Safety via Gemini Filters: We’ve all seen Generative AI go off-the-rails with its creations, and that is particularly a situation we want to avoid when presenting to younger audiences. As a result, we specifically chose Gemini Flash 2.0 for the API’s adjustable safety filtering weights. We set our levels to explicitly filter out dangerous or harmful content for minors. Maintaining Consistent Context: It's important that our stories balance consistency with creativity as students go through the choose-your-own adventure process, but this can be hard to do with the reactiveness of generative AI, especially when considering both visual and text input AND output. We picked multi-model tools that let us manage context (Gemini Flash 2.0's context capabilities) and focused on maintaining strong references to past information by tuning LumaLabs Dream Lab API's adjustable styling and reference weightage components. Maintaining Consistent Context: It's important that our stories balance consistency with creativity as students go through the choose-your-own adventure process, but this can be hard to do with the reactiveness of generative AI, especially when considering both visual and text input AND output. We picked multi-model tools that let us manage context (Gemini Flash 2.0's context capabilities) and focused on maintaining strong references to past information by tuning LumaLabs Dream Lab API's adjustable styling and reference weightage components. ### Accomplishments we're proud of Working together well! Collaboration across our different backgrounds let us connect at the event and build out a really creative idea. Figuring out how to use FlutterFlow from square one! Being adaptive and persistent in debugging Livesharing code and pair programming when things got tough Building a project with technical depth, social impact, and having a fun time through all of it! Our use of multiple different technologies and optimizations for our use case, from our focus on safety, to integrating multi-modal creativity, and even a focus on user-experience with reducing visible latency. ### What we learned We learned a TON about FlutterFlow. Not knowing much about frontend design, FlutterFlow was a great tool that we spent the majority of our hackathon working in. We managed to get our app to work seamlessly cross-platform and in sync with our dynamic backend endpoints, which took a ton of time, but was a great learning experience for creating a full-stack app from nothing but an idea. Special shoutout to the FlutterFlow team for staying late into the night with hackers to help debug ❤️. We learned a ton about front-end and had great conversations. We also developed a ton of API routes in FastAPI, which we then interfaced within FlutterFlow, adding actions to route data across the frontend, and sending and receiving information from our Generative AI models. Working with these generative AI models was very new to us as well, and we picked up a lot on how to use Gemini’s Multimodal 2.0 Flash API and Luma’s API to generate the content we need, doing a ton of prompt engineering along the way. We also learned the value of discussion and diagraming our workflows and API connections using visual tools like excalidraw for a better design process. ### What's next We are planning on expanding Sprout.io’s functionality to better benefit teachers when creating personalized learning plans for students! Implement a teacher side flow to better monitor individual student progress, see what kids are drawing, and quiz results for their class In addition, us giving out comprehension quizzes allows us to create tons of meaningful data that can be used for better data-driven, or even AI-driven strategies in the classroom We’d love to add ElevenLabs or similar API to read the text-to-speech to enhance the user experience, especially for kids who struggle with text-reading comprehension. Gemini Flash 2.0 is also releasing audio capabilities at the end of the month; we'd love to integrate them. Bold, highlight, and underline key words throughout the story for better immersion. We were thinking of including Markdown support (just like Devpost!) but didn't have enough time to fully flesh out this idea. Make Sprout.io a more kid focused application in terms of adding animation to their drawings (further Luma integration) Add a wider selection of quiz questions and topics that quizzes can cover! Adding a point system (something like Duolingo's Streaks) to keep kids engaged in learning! ## README (from the GitHub repository) # Treehacks2025 Shaan Doshi, Rhea Rai, and Sirihaasa Nallamothu's TreeHacks 2025 Hackathon Project ## Detected evidence (automated analysis) Indexed codebase: 10 recognized source files, 36 KB. - FastAPI (technology) — detected in the code - Python (language) — detected in the code - Dart (language) — claimed on Devpost, not found in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (16 of 16) ``` .gitignore bytes.txt dictionary.py image.json install_dependencies.sh integrated.py luma.py main.py output.txt README.md requirements.txt requirements.yml rhea-testing.py test.py upload_image_test.py vercel.json ``` ### Dependencies - requirements.txt: fastapi, google-generativeai, lumaai, pillow, pydantic, pyyaml, uvicorn ### Recent commits (newest first) - Final Changes - push quiz changes - harcode to debug - merge - Merge branch 'main' of https://github.com/sdoshi4/Treehacks2025 - get requestion - push rhea's code - add post call for dictionary - Merge branch 'main' of https://github.com/sdoshi4/Treehacks2025 - add dictionary and quiz questions - add final story api call - prompt engineering - Merge branch 'main' of https://github.com/sdoshi4/Treehacks2025 - Add title commit 2 - add title - more context prompt engineering - luma prompt engineering - Merge branch 'main' of https://github.com/sdoshi4/Treehacks2025 - prompt engineering + luma style - Merge branch 'main' of https://github.com/sdoshi4/Treehacks2025 ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` fastapi uvicorn pillow pydantic lumaai google-generativeai pyyaml ``` ### main.py ```python from lumaai import LumaAI import os import time import random import requests import yaml from google import genai from PIL import Image from google.genai import types from pydantic import BaseModel from concurrent.futures import ThreadPoolExecutor from typing import List, Optional, Dict, Any from fastapi import FastAPI, HTTPException from fastapi.responses import FileResponse from typing import Optional from fastapi import FastAPI, Body, Request, Form import requests from io import BytesIO import json from fastapi import FastAPI, UploadFile, File from fastapi.responses import JSONResponse import math # from PyDictionary import PyDictionary import nltk nltk.download('wordnet') from nltk.corpus import wordnet as wn app = FastAPI() comprehension_vocab = { "grade_3": [ "abbreviation", "adverb", "biography", "character", "chronological", "conjunction", "context", "declarative", "encyclopedia", "fact", "glossary", "index", "inference", "interrogative", "main", "modern", "opinion", "persuasion", "possessive", "revise", "subject", "theme", "attentive", "destination", "emerge", "fragrance", "habitation", "illuminate", "jargon", "knoll", "misgiving", "pertinent", "serenity", "stamina", "vigilant", "waft" ], "grade_4": [ "almanac", "analyze", "audience", "compare", "contrast", "evaluate", "genre", "legend", "metaphor", "outline", "paraphrase", "publish", "research", "sentence", "simile", "supporting", "text", "transition", "adversary", "banish", "bluff", "cunning", "defiance", "dispel", "egregious", "falter", "grueling", "headway", "imperious", "malleable", "momentum", "obscure", "precipice", "replenish", "scarcity", "somber", "swagger", "tactic" ], "grade_5": [ "caption", "conflict", "figurative", "idiom", "interjection", "minor", "onomatopoeia", "outline", "poetic", "reference", "resolution", "stereotypical", "superlative", "supporting", "word", "barricade", "circumference", "concoction", "contortion", "disdain", "exasperation", "foresight", "gusto", "ignite", "jut", "meander", "monotonous", "muster", "outlandish", "pristine", "quell", "recuperate", "restitution", "subside", "translucent", "unsightly", "weather" ], "grade_6": [ "affix", "analogy", "allusion", "appositive", "clause", "dialect", "literal", "mythology", "narrative", "phrases", "plagiarism", "predicate", "propaganda", "relevant", "sentence", "synthesize", "aplomb", "apprehensive", "brackish", "debris", "deft", "diminish", "dismal", "ember", "engross", "exhilarate", "furtive", "hasten", "impending", "jabber", "jostle", "kindle", "luminous", "materialize", "meticulous", "multitude", "narrate", "ominous", "persistent", "potential", "repugnant", "sabotage", "scurry", "sociable", "specimen", "swarm", "terse", "uncanny", "versatile", "vulnerable", "waver", "zeal" ], "grade_7": [ "assumption", "clause", "convention", "description", "exposition", "flashback", "fluency", "foreshadowing", "imagery", "interpretation", "irony", "nominative", "prose", "types", "viewpoint", "brandish", "commotion", "conspicuous", "counter", "eavesdrop", "engross", "exasperation", "falter", "fragrance", "grueling", "headway", "ignite", "imperious", "jargon", "jut", "luminous", "meander", "momentum", "muster", "obscure", "pertinent", "potential", "pristine", "recuperate", "repugnant", "scarcity", "serenity", "swagger", "tactic", "translucent", "unsightly", "vigilant", "waft" ], "grade_8": [ "agreement", "argument", "bias", "coherence", "debate", "derivation", "dramatization", "elaboration", "gerund", "inference", "infinitive", "parallel", "persuasive", "sensory", "thesis", "adversary", "apprehensive", "barricade", "banish", "bluff", "circumference", "concoction", "contortion", "cunning", "defiance", "destination", "disdain", "dispel", "egregious", "emerge", "exasperation", "falter", "foresight", "gusto", "habitation", "headway", "illuminate", "imperious", "jargon", "knoll", "malleable", "meander", "misgiving", "momentum", "muster", "obscure", "outlandish", "pertinent", "precipice", "pristine", "recluse", "replenish", "restitution", "scarcity", "serenity", "somber", "stamina", "swagger", "tactic", "translucent", "unsightly", "vigilant", "waft", "zeal" ] } chosen_vocab = [] chosen_grade_level_key = "grade_4" used_word2 = False with open('keys.yaml', 'r') as file: keys = yaml.safe_load(file) luma_client = LumaAI(auth_token=keys['lumaai_api_key']) gemini_client = genai.Client(api_key=keys['gemini_api_key']) class ImageData(BaseModel): name: str bytes: List[int] # Expecting a list of integers representing image bytes class RequestModel(BaseModel): data: ImageData class StoryOutput(BaseModel): story: str title: str image_prompt: str options: list[str] class StoryRequest(BaseModel): # This is asking for the image_path: Optional[str] = None story: Optional[str] = None title: Optional[str] = None choice: Optional[str] = None class StoryResponse(BaseModel): # This is the output of gemini story: str title: str image_prompt: str options: list[str] image_path: str def get_definition(word): synsets = wn.synsets(word) if not synsets: return None # Get the first definition as a simple approach return synsets[0].definition() def upload_to_imgur(image_path: str) -> str: """ Uploads an image to Imgur and returns the image URL. """ client_id = "40c6711dcccaf03" # TODO : fix bc this is hardcoded url = "https://api.imgur.com/3/upload" headers = {"Authorization": f"Client-ID {client_id}"} with open(image_path, 'rb') as image_file: response = requests.post( url [truncated — 18404 more characters] ``` ### rhea-testing.py ```python # temp for testing ``` ### requirements.yml ```yaml name: lumaai_env channels: - defaults dependencies: - python=3.11 # Adjust the version if needed - pip - pip: - requests - lumaai - time #conda env create -f requirements.yml ``` ### install_dependencies.sh ```shell # make sure conda on py 3.9 pip install lumaai; pip install requests; pip install pyyaml; pip install google; pip install pillow; pip install pydantic; pip install google-genai; pip install fastapi; pip install uvicorn; ``` ### upload_image_test.py ```python import requests import json # Read image as bytes # with open("kids_drawing.jpg", "rb") as f: # image_bytes = f.read() # with open("bytes.txt", 'wb') as f2: # f2.write(image_bytes) # print(image_bytes) with open('image.json', 'r') as file: image_bytes = json.load(file) # print(image_bytes) # print(data["data"]["bytes"]) # image_bytes = bytes(data["data"]["bytes"]) # Send the request response = requests.post( # "https://385a-68-65-164-29.ngrok-free.app/upload_image/", "https://9b1f-68-65-164-139.ngrok-free.app/upload_image_flutterflow", json=image_bytes, headers={"Content-Type": "multipart/form-data"} ) # response = requests.get("https://385a-68-65-164-29.ngrok-free.app/generate_story") # Print response print(response.content) ``` ### luma.py ```python from lumaai import LumaAI import os import time import requests import yaml # Load API key from keys.yaml with open('keys.yaml', 'r') as file: keys = yaml.safe_load(file) client = LumaAI( auth_token=keys['lumaai_api_key'], ) generation = client.generations.image.create( prompt="this is a happy family show the happy family ", image_ref=[ { "url": "https://yourelc.com.au/wp-content/uploads/2022/07/Blog.png", "weight": 0.85 } ] ) completed = False while not completed: generation = client.generations.get(id=generation.id) if generation.state == "completed": completed = True elif generation.state == "failed": raise RuntimeError(f"Generation failed: {generation.failure_reason}") print("Dreaming") time.sleep(2) image_url = generation.assets.image # download the image response = requests.get(image_url, stream=True) with open(f'{generation.id}.jpg', 'wb') as file: file.write(response.content) print(f"File downloaded as {generation.id}.jpg") ``` ### test.py ```python # AIzaSyDq-age16qrYQOssIusF87S4zGU1I9N17A from google import genai from PIL import Image from google.genai import types from pydantic import BaseModel, TypeAdapter class StoryOutput(BaseModel): story: str image_prompt: str options: list[str] # sys_instruct="You are an author of children's stories" client = genai.Client(api_key="AIzaSyDq-age16qrYQOssIusF87S4zGU1I9N17A") image = Image.open("kids_drawing.jpg") response = client.models.generate_content( model="gemini-2.0-flash", # config=types.GenerateContentConfig(system_instruction=sys_instruct), contents=[image, '''This is an image of a child's drawing. Please generate the first chapter of a children's book (around 500 words) from this image, leaving it where the main character has to make one of two choices. Store this story as the story parameter, and the two choices in the options list. Next, generate a description of an image that would best describe this chapter of the story. Make this description descriptive enough such that one could draw an accurate image of the story without ever reading it. Store this as the image_prompt.'''], config={ 'response_mime_type': 'application/json', 'response_schema': StoryOutput, }, ) output: StoryOutput = response.parsed print(response.text) response = client.models.generate_content( model="gemini-2.0-flash", # config=types.GenerateContentConfig(system_instruction=sys_instruct), contents=['''Generate the next chapter of the story given the first choice occurs.''', f"Chapter: {output.story}", f"Option: {output.options[0]}"], # config={ # 'response_mime_type': 'application/json', # 'response_schema': StoryOutput, # }, ) print(response.text) # output: StoryOutput = response.parsed # print(response.text) # print() # print(output.story) # print(output.image_prompt) # print(output.options) # luma-014b98b0-cb5f-428e-bc1a-45e32698a433-191e7d4c-c751-40a5-bd90-f4c045659a0e ``` ### dictionary.py ```python comprehension_vocab = { "grade_3": [ "abbreviation", "adverb", "biography", "character", "chronological", "conjunction", "context", "declarative", "encyclopedia", "fact", "glossary", "index", "inference", "interrogative", "main", "modern", "opinion", "persuasion", "possessive", "revise", "subject", "theme", "attentive", "destination", "emerge", "fragrance", "habitation", "illuminate", "jargon", "knoll", "misgiving", "pertinent", "serenity", "stamina", "vigilant", "waft" ], "grade_4": [ "almanac", "analyze", "audience", "compare", "contrast", "evaluate", "genre", "legend", "metaphor", "outline", "paraphrase", "publish", "research", "sentence", "simile", "supporting", "text", "transition", "adversary", "banish", "bluff", "cunning", "defiance", "dispel", "egregious", "falter", "grueling", "headway", "imperious", "malleable", "momentum", "obscure", "precipice", "replenish", "scarcity", "somber", "swagger", "tactic" ], "grade_5": [ "caption", "conflict", "figurative", "idiom", "interjection", "minor", "onomatopoeia", "outline", "poetic", "reference", "resolution", "stereotypical", "superlative", "supporting", "word", "barricade", "circumference", "concoction", "contortion", "disdain", "exasperation", "foresight", "gusto", "ignite", "jut", "meander", "monotonous", "muster", "outlandish", "pristine", "quell", "recuperate", "restitution", "subside", "translucent", "unsightly", "weather" ], "grade_6": [ "affix", "analogy", "allusion", "appositive", "clause", "dialect", "literal", "mythology", "narrative", "phrases", "plagiarism", "predicate", "propaganda", "relevant", "sentence", "synthesize", "aplomb", "apprehensive", "brackish", "debris", "deft", "diminish", "dismal", "ember", "engross", "exhilarate", "furtive", "hasten", "impending", "jabber", "jostle", "kindle", "luminous", "materialize", "meticulous", "multitude", "narrate", "ominous", "persistent", "potential", "repugnant", "sabotage", "scurry", "sociable", "specimen", "swarm", "terse", "uncanny", "versatile", "vulnerable", "waver", "zeal" ], "grade_7": [ "assumption", "clause", "convention", "description", "exposition", "flashback", "fluency", "foreshadowing", "imagery", "interpretation", "irony", "nominative", "prose", "types", "viewpoint", "brandish", "commotion", "conspicuous", "counter", "eavesdrop", "engross", "exasperation", "falter", "fragrance", "grueling", "headway", "ignite", "imperious", "jargon", "jut", "luminous", "meander", "momentum", "muster", "obscure", "pertinent", "potential", "pristine", "recuperate", "repugnant", "scarcity", "serenity", "swagger", "tactic", "translucent", "unsightly", "vigilant", "waft" ], "grade_8": [ "agreement", "argument", "bias", "coherence", "debate", "derivation", "dramatization", "elaboration", "gerund", "inference", "infinitive", "parallel", "persuasive", "sensory", "thesis", "adversary", "apprehensive", "barricade", "banish", "bluff", "circumference", "concoction", "contortion", "cunning", "defiance", "destination", "disdain", "dispel", "egregious", "emerge", "exasperation", "falter", "foresight", "gusto", "habitation", "headway", "illuminate", "imperious", "jargon", "knoll", "malleable", "meander", "misgiving", "momentum", "muster", "obscure", "outlandish", "pertinent", "precipice", "pristine", "recluse", "replenish", "restitution", "scarcity", "serenity", "somber", "stamina", "swagger", "tactic", "translucent", "unsightly", "vigilant", "waft", "zeal" ] } ``` ### integrated.py ```python #moved into backend.py to wrap with fastapi from lumaai import LumaAI import os import time import requests import yaml from google import genai from PIL import Image from google.genai import types from pydantic import BaseModel from concurrent.futures import ThreadPoolExecutor #api keys with open('keys.yaml', 'r') as file: keys = yaml.safe_load(file) luma_client = LumaAI(auth_token=keys['lumaai_api_key']) gemini_client = genai.Client(api_key=keys['gemini_api_key']) class StoryOutput(BaseModel): story: str image_prompt: str options: list[str] # this just generates an image from str promtp def generate_image(prompt): generation = luma_client.generations.image.create(prompt=prompt) while generation.state != "completed": generation = luma_client.generations.get(id=generation.id) if generation.state == "failed": raise RuntimeError(f"Image generation failed: {generation.failure_reason}") time.sleep(2) image_url = generation.assets.image response = requests.get(image_url, stream=True) filename = f'images/{generation.id}.jpg' os.makedirs('images', exist_ok=True) #make dir for images if it doesnt exist already with open(filename, 'wb') as file: file.write(response.content) return filename # this tkaes in an image and generates the start of the story via gemini multimodeal input def generate_story_from_image(image_path): image = Image.open(image_path) response = gemini_client.models.generate_content( model="gemini-2.0-flash", contents=[image, "This is an image of a child's drawing. Generate the first chapter of a children's book (around 100 words) where the main character has to make a choice. Return 'story', 'image_prompt', and 'options' as described previously."], config={ 'response_mime_type': 'application/json', 'response_schema': StoryOutput, }, ) return response.parsed # genereates the next chapter based on previous choices def generate_next_story(story, choice): response = gemini_client.models.generate_content( model="gemini-2.0-flash", contents=[f"Chapter: {story}", f"Choice: {choice}", "Generate the next chapter based on this choice."], config={ 'response_mime_type': 'application/json', 'response_schema': StoryOutput, }, ) return response.parsed # main loop for storytelling def storytelling_loop(start_image_path, num_panels=4): current_image_path = start_image_path for panel in range(num_panels): if panel == 0: story_output = generate_story_from_image(current_image_path) else: choice = input(f"Choose option 1 or 2: {story_output.options[0]} / {story_output.options[1]}: ") choice_text = story_output.options[int(choice) - 1] story_output = generate_next_story(story_output.story, choice_text) print(f"Panel {panel + 1} Story: {story_output.story}") print(f"Options: 1. {story_output.options[0]} 2. {story_output.options[1]}") current_image_path = generate_image(story_output.image_prompt) # get the 2 images as the child is reading choice1_prompt = f"Make an illustration for the story if the character chooses: {story_output.options[0]}" choice2_prompt = f"Make an illustration for the story if the character chooses: {story_output.options[1]}" # trying to do concurrent image generation via python threads (not sure if this will work when we do full integrate) with ThreadPoolExecutor() as executor: future1 = executor.submit(generate_image, choice1_prompt) future2 = executor.submit(generate_image, choice2_prompt) choice1_image = future1.result() choice2_image = future2.result() print(f"generated images for choices saved as: {choice1_image}, {choice2_image}") # main running storytelling_loop('kids_drawing.jpg', num_panels=4) ```