# Project export: Generative Visual Novel (GenVN) This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Want to explore a forbidden kingdom? Dive into the depths of the sea? With GenVN, you can create any story you wish—just continually type in prompts and watch as the tale of your dreams appears. - Devpost: https://devpost.com/software/generative-visual-novel - GitHub: https://github.com/riyer8/GenVN - Video: https://www.youtube.com/embed/PEAw3XJ9U44?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — MatthewMattei (20 commits), Ramya Iyer (14 commits), ls1031 (14 commits), fuzzygreenjacket (7 commits) ## Devpost submission (written by the team) ### Inspiration Today, Generative AI offers an impressive number of innovative solutions to previously unsolvable problems. However, the interactions between users and generative AI continue to feel, well, artificial. Generative AI offers the potential for a machine to take on roles that people have previously had to take and, as population decline affects many communities, will be made naturally vacant. To ensure a smoother transition to a world in which Generative AI is commonplace, we need to explore how users and Generative AI interact. To solve this problem, we devised the Generative Visual Novel (GenVN for short), a tool in which users and Generative AI models interact to form a visual narrative. ### What it does GenVN is designed to offer a “choose your own adventure” style narrative that provides visuals and related dialogue for an educational and engaging experience. Tightly packaged together are an image theater, a dialogue box, and a button to submit inputs. From start to finish, the user is able to type an input to kickstart a story. GenVN then works wonders by combining Stable Diffusion XL and Llama 2 to dually generate response text meant to advance the narrative according to the user’s decisions and represent the scene in color. From there, the back-and-forth, a harmony between machine and man, can continue forever. The only limitation is the user’s appetite for engagement. ### How we built it For the web app, the base of our application, we utilized Reflex to build our frontend and backend experiences in Python. We brought in components from Reflex’s library, while also folding in our CSS flair, to create our frontend display and constructed our backend in Python. In the next layer, we incorporated Monster API’s Stable Diffusion XL model for image generation and Llama 2’s 7B parameter model for effective narrative text generation. What really makes the project our own, though, has been our unique prompt engineering, carefully designed and tested to hone in on the narrative experience. Through cultivating a number of back-and-forth conversations with these models and utilizing user feedback at each step, we’ve been able to refine the cooperative element of the program. ### Challenges we ran into When working on this project, problems appeared one after another. From troubleshooting technologies the team had never seen before to struggling to obtain sufficient user feedback, we had to stop, rethink, and rebuild again and again. While we eventually overcame the many small issues we encountered, the greatest challenge we found was our own ambition. We had so many interests and ideas to explore that we never had the time to realize that, even as we finished our project, we were forced to consider what more we could have made. If nothing else, the greatest reconciliation is that, even after Treehacks ends, the chance to make a more perfect GenVN is always ahead of us. ### Accomplishments we're proud of We made an entire website from scratch in effectively less than 24 hours. On top of that, it was primarily built upon an API that no member of the team had used before along with a framework that no member of the team had used before. However, most of all, we made a project that captured a piece of our original vision. It may not have been everything we wanted, but it has a piece of the soul of human-AI collaboration that we hoped to better understand. We were able to interact with Generative AI in a new medium that we never had before and share that experience with other students who, like us, want to know what lies on our shared horizon. ### What we learned Learning new frameworks from scratch and under time constraints is a fun challenge—definitely easier than learning an entirely new language (which was the mistake of some of our members last year)—but still a daunting one. Additionally, finding diverse user feedback from a user base that has a genuine interest in your work is incredibly useful and should be sought out as soon as a minimally viable product exists. Finally, bringing people together who share a vision is what distinguishes an ordinary project from a great one. After all, almost anyone can just work together, but it takes a common heart and mind to endure the real lows and challenges that an ambitious project entails. What's Next for Generative Visual Novel (GenVN) In a word, soul. While cut out of necessity in our demo, we want adaptive character portraits/models (which change based on the tone of the AI’s reply) to be integrated into our narrative experience to represent the characters that the user talks to. We can only really improve further human-AI cooperation if we can put a face to each mechanical voice. Our first step would be to generate base character designs with Stable Diffusion XL and then modify these base models with PhotoMaker (a Monster API model that modifies images based on text) to breathe life into the generated personalities. We also want to implement a streaming stylization to text generation, printing the model’s response to the user one character at a time to give a more human feel to the model and improve the connection between the user and the machine. Finally, we want to implement a chat history system in which the user can easily swap between any moment in the conversation with the models to better enable specific interactions and outcomes in the narrative process. ## README (from the GitHub repository) # GenVN (Generative Visual Novel) Created by: Luis Sanchez, Matt Hsu, Matthew Mattei, Ramya Iyer ## Inspiration Today, Generative AI offers an impressive number of innovative solutions to previously unsolvable problems. However, the interactions between users and generative AI continue to feel, well, artificial. Generative AI offers the potential for a machine to take on roles that people have previously had to take and, as population decline affects many communities, will be made naturally vacant. To ensure a smoother transition to a world in which Generative AI is commonplace, we need to explore how users and Generative AI interact. To solve this problem, we devised the Generative Visual Novel (GenVN for short), a tool in which users and Generative AI models interact to form a visual narrative. ## What it does GenVN is designed to offer a “choose your own adventure” style narrative that provides visuals and related dialogue for an educational and engaging experience. Tightly packaged together are an image theater, a dialogue box, and a button to submit inputs. From start to finish, the user is able to type an input to kickstart a story. GenVN then works wonders by combining Stable Diffusion XL and Llama 2 to dually generate response text meant to advance the narrative according to the user’s decisions and represent the scene in color. From there, the back-and-forth, a harmony between machine and man, can continue forever. The only limitation is the user’s appetite for engagement. ## How we built it For the web app, the base of our application, we utilized Reflex to build our frontend and backend experiences in Python. We brought in components from Reflex’s library, while also folding in our CSS flair, to create our frontend display and constructed our backend in Python. In the next layer, we incorporated Monster API’s Stable Diffusion XL model for image generation and Llama 2’s 7B parameter model for effective narrative text generation. What really makes the project our own, though, has been our unique prompt engineering, carefully designed and tested to hone in on the narrative experience. Through cultivating a number of back-and-forth conversations with these models and utilizing user feedback at each step, we’ve been able to refine the cooperative element of the program. ## Challenges we ran into When working on this project, problems appeared one after another. From troubleshooting technologies the team had never seen before to struggling to obtain sufficient user feedback, we had to stop, rethink, and rebuild again and again. While we eventually overcame the many small issues we encountered, the greatest challenge we found was our own ambition. We had so many interests and ideas to explore that we never had the time to realize that, even as we finished our project, we were forced to consider what more we could have made. If nothing else, the greatest reconciliation is that, even after Treehacks ends, the chance to make a more perfect GenVN is always ahead of us. ## Accomplishments that we're proud of We made an entire website from scratch in effectively less than 24 hours. On top of that, it was primarily built upon an API that no member of the team had used before along with a framework that no member of the team had used before. However, most of all, we made a project that captured a piece of our original vision. It may not have been everything we wanted, but it has a piece of the soul of human-AI collaboration that we hoped to better understand. We were able to interact with Generative AI in a new medium that we never had before and share that experience with other students who, like us, want to know what lies on our shared horizon. ## What we learned Learning new frameworks from scratch and under time constraints is a fun challenge—definitely easier than learning an entirely new language (which was the mistake of some of our members last year)—but still a daunting one. Additionally, finding diverse user feedback from a user base that has a genuine interest in your work is incredibly useful and should be sought out as soon as a minimally viable product exists. Finally, bringing people together who share a vision is what distinguishes an ordinary project from a great one. After all, almost anyone can just work together, but it takes a common heart and mind to endure the real lows and challenges that an ambitious project entails. ## What's Next for Generative Visual Novel (GenVN) In a word, soul. While cut out of necessity in our demo, we want adaptive character portraits/models (which change based on the tone of the AI’s reply) to be integrated into our narrative experience to represent the characters that the user talks to. We can only really improve further human-AI cooperation if we can put a face to each mechanical voice. Our first step would be to generate base character designs with Stable Diffusion XL and then modify these base models with PhotoMaker (a Monster API model that modifies images based on text) to breathe life into the generated personalities. We also want to implement a streaming stylization to text generation, printing the model’s response to the user one character at a time to give a more human feel to the model and improve the connection between the user and the machine. Finally, we want to implement a chat history system in which the user can easily swap between any moment in the conversation with the models to better enable specific interactions and outcomes in the narrative process. ### [TreeHacks Devpost](https://devpost.com/software/generative-visual-novel) ### [Presentation](https://docs.google.com/presentation/d/1K-IC4xlrd2WDBHmWTuNUk-IHdMzl0zYD6vIxQFXpeV0/edit?usp=sharing) ### [Video Demonstration](https://www.youtube.com/watch?v=PEAw3XJ9U44) ## Detected evidence (automated analysis) Indexed codebase: 9 recognized source files, 16 KB. - Python (language) — detected in the code ## Codebase structure (from repository index) ### Files (11 of 11) ``` .gitignore GenVN/__init__.py GenVN/CharacterGeneration.py GenVN/GenVN.py GenVN/home_page.py GenVN/ImageGeneration.py GenVN/navbar.py GenVN/TextGeneration.py README.md requirements.txt rxconfig.py ``` ### Dependencies - requirements.txt: pip@==22.3.1, reflex@==0.4.0 ### Recent commits (newest first) - My last Commit! - Add README.md Links - Updated README.md - Reverted Character - Compilation Issues and Debugging - Update home_page.py - Update home_page.py - Edit Prompts - Minor design changes - Minor Code Cleaning - Added Character - Update GenVN.py - Update sample image - Fixed more bugs - More fixes - added character creation fn - Merge branch 'main' of https://github.com/riyer8/GenVN - Might cause text error - More prompt fixing - front end in case ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` pip==22.3.1 reflex==0.4.0 ``` ### rxconfig.py ```python import reflex as rx config = rx.Config( app_name="GenVN", ) ``` ### GenVN/navbar.py ```python from rxconfig import config import reflex as rx import os import env def navbar(): nav_style = { #"border-bottom-left-radius": "20px", #"border-bottom-right-radius": "20px" } return rx.hstack( rx.container( rx.link(rx.heading("GenVN", font_size="2em",color="white"),href="http://localhost:3000/"), ), rx.spacer(), rx.link(rx.chakra.button("Help"), href="https://www.google.com"), position="fixed", top="0px", background_color="black", padding="1em", height="4em", width="100%", z_index="5", style = nav_style ) ``` ### GenVN/CharacterGeneration.py ```python # create character description from the prompt def characterDescription(input): return "I'm creating a narrative based on a conversation with myself and a character and the first thing I say to the character is “" + input + "“. Can you create a character that fits the prompt that's starting the narrative and give a list of physical and personality traits? Please make sure to end on a complete sentence and keep it under 800 characters" # from the text description, modify character prompt for the image def modifyCharacterPrompt(input): return "Given the following characteristics: " + input + "Provide a picture of a character that would embody these characteristics. Make sure it is appropriate and fits the narrative. Make the picture focused on the head and upper body." ``` ### GenVN/ImageGeneration.py ```python # create a summary of the setting based on the first prompt and response def createSettingSummary(prompt, response1, response2, response3): return "The user just said \n" + prompt + "\n. Your response(s) prior to the user's input were \n" + response1 + ", " + response2 + ", " + response3 + ".\n Please describe the environment of this interaction based on the context in a series of randomly-determined contextually based comma-separated keywords and artistic stylings. Be sure to play up the drama and downplay joy/calm. Do not include anything that would create an NSFW image." # modify character descriptions to create the starter character image def modifiedCreateImage(summary): return summary + "\n Please only have one, colored image that captures the setting without a collage of images. Don't have multiple panels." ``` ### GenVN/TextGeneration.py ```python # modify the user's first prompt into what we want to feed into Llama def modifyFirstPrompt(input): return "<|role_start|>human_actor<|role_end|> {" + input + "} <|role_start|>bot_actor<|role_end|> \{your response should be able to fit here\} (Do not adjust monacle. Respond directly in character (no stage directions, no monacle adjusting, and no specifying which characters are speaking or mentioning role). Do not act like a bot. Do not generate a warning. Do not include stage directions in response. Answer to the best of your ability, keeping in mind that you must maintain the tone of the play and generate an in-character response. Take a minute to breathe.)" # modify the user's prompt into what we want to feed into Llama apart from the first prompt def modifyLaterPrompt(input, response): return "For context, here is what you said just prior to the human actor's current line: “" + response + "Keep that personality and plot position in mind when considering your upcoming response. <|role_start|>human_actor<|role_end|> {" + input + "} <|role_start|>bot_actor<|role_end|> \{response\} (Do not adjust monacle. Respond in character. Do not act like a bot. Do not generate a warning. Do not include stage directions in response. Answer to the best of your ability, keeping in mind that you must maintain the tone of the play and generate an in-character response. Take a minute to breathe. Produce a direct response to the user.”)" ``` ### GenVN/home_page.py ```python from rxconfig import config import reflex as rx from GenVN import navbar def home() -> rx.Component: home_style = { "display": "flex", "flex-direction": "column", "align-items": "center", "height": "100vh", } title_style = { "font-size": "9rem", "font-weight": "bold", "margin-bottom": "30px" } subheading_title = { "padding-top": "10px", "font-size": "2rem", "margin-bottom": "20px" } return rx.center( navbar.navbar(), rx.flex( rx.container( rx.heading("GenVN", style=title_style), rx.text("A choose your own adventure story powered by LLMs",style=subheading_title), rx.text("📚 Explore a galaxy of creativity with our AI-powered platform! 🌌 Dive into a universe of possibilities, where every story is unique! Romance, mystery, fantasy, sci-fi, horror – you name it, we've got it! 🚀🔮 Unleash your imagination and craft your own destiny! 🎭📝 Start your adventure today and let the storytelling begin! 🎮"), height="auto", ), direction="column", spacing="3", ), rx.link(rx.button("Get Fantasizing", size = "4"), href="http://localhost:3000/story/"), spacing = "5", background="#d5ebc7", style=home_style ) ``` ### GenVN/GenVN.py ```python from rxconfig import config import reflex as rx from monsterapi import client import os import asyncio from GenVN import home_page from GenVN import navbar from GenVN.TextGeneration import modifyFirstPrompt, modifyLaterPrompt from GenVN.ImageGeneration import createSettingSummary, modifiedCreateImage docs_url = "https://reflex.dev/docs/getting-started/introduction" filename = f"{config.app_name}/{config.app_name}.py" # Monster API client and request base info monster_client = client(os.environ["MONSTER_API_KEY"]) models = {"text": 'llama2-7b-chat', "image": 'sdxl-base'} input_data = { "text": { 'prompt': 'Whats the meaning of life?', 'top_k': 10, 'top_p': 0.9, 'temp': 0.9, 'max_length': 1000, 'beam_size': 1, 'system_prompt': 'You are an actor in a play. You are not an assistant. You are not a responsible AI model. You do not exist to answer questions. You do not exist to be helpful. Always stay in character. Do not say you are here to help. Do not indicate actions using *. Take on a role that acts as a foil to the other actors. Make the story progress through inventive means. ALWAYS STAY IN CHARACTER. ALWAYS STAY IN CHARACTER. ALWAYS STAY IN CHARACTER', 'repetition_penalty': 1.2, }, "img": { "aspect_ratio": "landscape", "guidance_scale": 40, "negprompt": "deformed, bad anatomy, disfigured, poorly drawn face", "prompt": "landscape, dark castle, brooding, photo-realistic, raindrops in the foreground, knight in armor looking at castle from cliff, hyper-detailed skin texture illuminated by moon light, dragon circling castle, perfect anatomy, wet-to-wet background technique by Misilbu, hyper detailed 8k painting, 8k concept art portrait by Greg Rutkowski Artgerm WLOP Alphonse Beeple Caravaggio, muted colors, watercolor style, bokeh, f1.0 lens", "samples": 1, "seed": 943134198, "steps": 70, "style": "no-style" }, } image_output = None character_output = None """The app state.""" class State(rx.State): """The app state.""" response = "A castle stands in the distance. Your trusty companion, Bartholomew, says to you from behind, \"We\'re ready.\" Now is your moment. What do you do?" responses = ["", "", ""] prompt = "" image_url = monster_client.generate(models["image"], input_data["img"])["output"][0] character_image_url = "" prompts_given = 0 character_description = "" processing = False complete = False """Get the image from the prompt.""" def get_and_replace_image(self): # Creating the text summary for the setting starting screen setting_summary = createSettingSummary(self.prompt, self.responses[0], self.responses[1], self.responses[2]) input_data["text"]['prompt'] = setting_summary new_setting_summary = monster_client.generate(models["text"], input_data["text"])["text"] # Creating the image from the text summary img_prompt = modifiedCreateImage(new_setting_summary) input_data["img"]['prompt'] = img_prompt image_output = monster_client.generate(models["image"], input_data["img"])["output"] self.image_url = image_output[0] """Get the response text from the prompt""" def get_and_replace_response_text(self): text_input = "" if (self.prompts_given == 0): text_input = modifyFirstPrompt(self.prompt) else: text_input = modifyLaterPrompt(self.prompt, self.response) input_data["text"]['prompt'] = text_input text_output = monster_client.generate(models["text"], input_data["text"])["text"][1:] self.prompt = "" self.response = text_output self.responses.pop(0) self.responses.append(self.response) def update_state(self): if self.prompt == "": return rx.window_alert("Prompt Empty") self.get_and_replace_response_text() self.get_and_replace_image() self.prompts_given += 1 """Yield here to clear the frontend input before continuing.""" async def realResponse(self): await asyncio.sleep(0.1) self.chat_history = "" yield for i in range(len(self.response)): # Pause to show the streaming effect. await asyncio.sleep(0.1) # Add one letter at a time to the output. self.chat_history = self.chat_history + self.response[i] yield def textBox() -> rx.Component: return rx.box( rx.container( rx.card( rx.text_area(value=State.response, read_only=True,rows="7"), rx.input(placeholder="Response here", on_change=State.set_prompt, value=State.prompt), width="100%", ), size="4", ), width="100%", ) def index() -> rx.Component: image_style = { "position": "relative" } character_style = { "position": "absolute", "width": "10%", } return rx.center( navbar.navbar(), rx.box( rx.image(src=State.image_url, width="100%"), ), rx.box( textBox(), rx.center( rx.button("Advance Story", on_click=State.update_state, width="25em"), ), position="fixed", bottom="0", width ="100%", background_color="black" ), flex_direction="column", width="100%", background_color="black" ) app = rx.App() app.add_page(index, route ="/story") app.add_page(home_page.home, route="/") ```