# Project export: Sentiment Storybook This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Enabling children to nurture emotional literacy in children through interactive story-telling. - Devpost: https://devpost.com/software/sentiment-storybook - GitHub: https://github.com/YJC23/TreeHacksProj - Video: https://player.vimeo.com/video/914173904?byline=0&portrait=0&title=0#t= - Team: 2 GitHub contributor(s) — YJC23 (7 commits), Jenny Wei (6 commits) ## Devpost submission (written by the team) ### Inspiration Hi! We’re Jenny, Emma, Alice, and YJ, sophomores at Stanford University passionate about education and accessibility. We powered through this hackathon through Tejava Teas, Matcha Pocky, and Spotify Blends. ### What it does The COVID-19 pandemic had widespread impacts on human interaction, especially for young children. Research indicates that the pandemic's restrictions, such as virtual schooling, have significantly affected children's ability to learn social cues from external stimuli. This disruption in a child’s vital developmental process highlights the urgent need for innovative solutions to support children's emotional and social learning. We wanted to create an engaging education tool children would show a genuine interest in. Sentiment Storybook is envisioned to not only leverage technology to bridge the learning gap in a nurturing and accessible manner but also facilitate a deeper understanding among parents of their children’s needs while fostering parent-child quality time. We sought to teach children how to navigate situations that require different responses in a fun and engaging way. Functionality We introduce Sentiment Storybook, an interactive digital storybook that invites parents to embark on a journey of emotional learning alongside their children. Here's how it works: parents and children begin by selecting a core value or principle they wish to explore together — whether it be respect, compassion, honesty, or any other virtue pivotal to emotional and ethical growth. Using a fine-tuned Mistral LLM and Stable Diffusion technologies, Sentiment Storybook generates a unique picture book illustrating a scenario demonstrating how to practice this value. The scenarios feature colorful drawings that users can navigate through to simulate the experience of reading a physical picture book. The accompanying storyline provides context and depth to the illustrated scenarios. It's designed to prompt discussions between parents and children, encouraging them to reflect on the story, relate it to their own lives, and consider how they can apply the learned values in their daily interactions. What’s more — Sentiment Storybook enables parents and children to experience a unique story each time, ensuring a fresh learning adventure every time. Furthermore, children get the opportunity to reflect on diverse scenarios, deepening their understanding. ### How we built it On the high level, we hoped to pass in a value (such as “kindness”) as an input into our web application and receive a sequence of 4-6 illustrations, accompanied by text, to demonstrate that value. To generate the picturebook, we first generated 4-6 story sentences from the value, as well as the illustration descriptions corresponding to each sentence in the story. This required building an API endpoint to Monster API’s Llama2-7B-Chat model and several iterations of prompt engineering to receive our desired format of sentences and illustration descriptions. After processing the response to build a list of the illustration descriptions, we passed them into Monster API’s sdxl-base model (with another API endpoint) to generate our illustrations. Next, we uploaded the values, story sentences, and illustrations to Convex’s cloud-hosted database. We chose Convex to incorporate real time updates on the frontend, so that picturebooks would generate and immediately populate the frontend upon user input. We implemented the above generation process through Typescript as Convex actions and saved to our database with Convex mutations. Finally, for the front-end, we designed our picture book through Figma and built it through ReactJS + Tailwind. Note: Initially, we hoped to build our own LLM model for generating a story and illustrations by fine-tuning with the MonsterAPI. To do so, we first created our own custom dataset from the values as our input and the story as our output with GPT-4 and careful prompt engineering. With this dataset, we fine-tuned the Mistral 7B language model with MonsterAPI to generate our 4-6 step story sentences and illustration descriptions. Finally, we employed stable diffusion XL with MonsterAPI, passing in the illustration descriptions to generate our picture book illustrations! We successfully deployed the model and hit the endpoint to receive our picture book. However, our deployed model was terminated unexpectedly (potentially due to depleting our tokens), so we reverted to our original model for generation. In future steps, we hope to continue exploring an even more robust fine-tuned model! ### Challenges we ran into Our project involved several key phases and challenges, each requiring its own set of innovative solutions and collaborative efforts: Exploring New APIs: Our team delved into uncharted waters by integrating Convex DB and Monster API into our workflow. This introduced a steep learning curve for our team as we navigated the intricacies of these tools; however, we persevered until the end. Prompt Engineering Iterations: We began our journey by testing basic prompts with GPT-4; however, when we transitioned to open-source LLMs on Monster API, we encountered diverse outcomes. Many generated stories were unrelated to our intended content, lacking the desired structure. To address this, we iteratively refined our prompts, assessing the ideal balance of specificity and structure to elicit descriptive narratives effectively. We also fine-tuned and deployed Mixtral-8x7B-Instruct-v0.1 LLM for this purpose. Stable Diffusion Challenges: Attempting to leverage Stable Diffusion XL for image generation, we faced a lot of hurdles. For instance, inputting a prompt for a scene featuring both a turtle and a frog resulted in the model generating only one of the specified animals. To overcome such obstacles, we undertook a rigorous process of hyperparameter tuning and adapted our prompt structuring methods to enhance model comprehension. Custom Dataset Generation: Recognizing the importance of fine-tuning models for our specific requirements, we embarked on custom dataset generation. This process aimed to optimize model performance and accuracy for our application. Overcoming Deployment Hurdles: During the deployment phase, we encountered obstacles in fine-tuning and deploying models using Monster API. However, with invaluable assistance from the dedicated team at MonsterAPI—whose round-the-clock support on Discord proved instrumental—we successfully deployed a fine-tuned model capable of consistently generating desired outputs. Other Unexpected Results: Throughout our journey, we encountered unexpected results stemming from various factors such as NSFW issues and the necessity of including character names for accurate image descriptions. Additionally, we had to devise strategies for processing data with varying formats, ensuring consistency to facilitate smooth operation. ### Accomplishments we're proud of As students passionate about education, it was incredible to work towards solving an issue we all care about. Especially since some of us have had the opportunity to teach and interact with young kids, we kept them in mind as we built our product, which made the process so much more rewarding. We are also proud that we built a generative picturebook from scratch! We had little experience with the APIs, platforms, and fine-tuning LLMs, but through scrutinizing the documentation and running back and forth from Gates to Huang to seek help from mentors, we were able to pull all the pieces together. Especially when merging the pieces from the generation process to the database to the frontend, we were shocked and overjoyed that it worked. ### What we learned On the technical side, we learned how to work with completely new platforms and APIs such as Convex, MonsterAPI, and Hugging Face. We also struggled with creating a custom dataset from scratch and prompt engineering to receive our desired output formats, but we triumphed after much trial and error. Finally, many of us experienced our first hackathon! From idea brainstorming to debugging and more debugging to building our final product, we learned how to work as a team and organize our priorities. We delegated tasks according to each teammate’s passion or experience level and planned out our time strategically to ensure we had a MVP up and running first. ### What's next The following are a few of the next steps that we hope to take in the near future. Adding more interactive features, such as offering several options for the storyline or presenting positive and negative examples for the kids to choose between. Fine-tuning SDXL with Monster API once it’s released (or HuggingFace) so that we can get better, more consistent images for our storybook. Continuing to fine-tune the LLM to generate longer, more comprehensive stories for older age-groups. We can also have age-group as another variable. Applicable beyond just learning core values/principles. It can be stories that teach more nuanced morals – eg. importance of healthy lifestyle ## README (from the GitHub repository) # Convex Hack Pack Hello, hackers! Welcome to [Convex](https://convex.dev), a full-featured backend platform you can use to rapidly prototype any hackathon app you can think of! ## So you want to build an app, ASAP If you're building an app for a hackathon, there's no time to waste! Convex is the perfect partner for hackers on a deadline because: - you get a cloud-hosted database & serverless backend for free - your frontend clients get automatic, realtime data updates - you can flexibly change your database schema as your product evolves - you get file storage, text- and vector-based search, cron jobs, and lots more out of the box Whatever your idea is, Convex can help you make it happen! ## What is this hack pack? This repository is your starting point for getting hacking with Convex. It includes: - A simple Convex demo app to demonstrate the basics - Hands-on exercises to get you started (below) - Lots of further reading links & resources to keep you going (below) ## Before you begin - Sign up for a free Convex account at [convex.dev](https://convex.dev) - [highly recommended] Sign up for a [Github](https://github.com) account if you don't have one already - Get your development environment set up: - Make sure you have [git](https://github.com/git-guides/install-git) installed, or install it if needed - Install/update [Node.js](https://nodejs.org/en/download) (which includes [npm](https://www.npmjs.com/)) to the latest LTS version - [optional] Read up on the technologies we'll be using: - [JavaScript](https://developer.mozilla.org/en-US/docs/Learn/JavaScript) and [TypeScript](https://www.typescriptlang.org/) programming languages - [React](https://react.dev/) JS/TS frontend framework - [Vite](https://vitejs.dev/) JS/TS build & development tool - and of course, [Convex](https://docs.convex.dev)! ## Install the hack pack - Download a local copy of this repository from Github using `git`: ``` git clone https://github.com/get-convex/convex-hack-pack.git ``` (Note: if you prefer, you can also clone the repo with the [Github CLI](https://cli.github.com/) or [download a ZIP file](https://github.com/get-convex/convex-hack-pack/archive/refs/heads/main.zip) of the contents) - Install the project dependencies: ``` cd convex-hack-pack npm install ``` - Run the following command, then follow the prompts to configure a new project in your Convex account and start the development server for the demo site: ``` npm run dev ``` - You should see the demo app automatically open in your web browser (if not, navigate to [localhost:5173](http://localhost:5173)) - In the demo app, type in a new app idea and click "Save", and/or click the "Generate a random app idea" button, and you should see the ideas appear! You may have noticed that "Include random ideas" checkbox in the demo app doesn't work! Don't worry, we're going to fix it. But before we do, let's take a closer look at our data in the Convex dashboard. --- # Exercises ## Exercise 1: Update your data - In the browser, navigate to the `convex-hack-pack` project in your [Convex dashboard](https://dashboard.convex.dev) (if it didn't open automatically) - you'll be taken to the 'Data' tab where you should see the `ideas` table and any documents inside it - Edit data: - Double-click in the 'idea' field of any document and edit the text - Back in your demo app, you'll see the text has automatically updated! - Add new data: - In the dashboard `ideas` table, click the "Add documents" button on the top right - In the document editor that opens, type a new app idea to fill out the `idea` property (e.g. `"A brainstorming app for developers"` - in quotes, because it's a string value) - Click "Save" to save the new document - In both the dashboard and the demo app, you should now see your new idea! But we're not done yet - that "Include random ideas" checkbox in our app still doesn't work! Let's fix that. ## Exercise 2: Update your backend - Update your `listIdeas` function: - In your code editor, open `convex/myFunctions.ts` - In the `listIdeas` query function, add an additional argument named `includeRandom` to the `args` object, whose value is a boolean (`v.boolean()`). The `args` object should now look like this: ```js args: { includeRandom: v.boolean() }, ``` - In the `handler` function, add an `if` conditional based on the value of `args.includeRandom`: if true, return the same query results as before, but if false, filter the data to only return documents where the `random` field is _not_ equal to `true`, like so: ```js handler: async (ctx, args) => { if (args.includeRandom) { return await ctx.db.query("ideas").collect(); // Returns all documents in the 'ideas' table } else { return await ctx.db .query("ideas") .filter((q) => q.neq(q.field("random"), true)) // Only returns documents whose 'random' field is not equal to `true` .collect(); } }, ``` - Save the `myFunctions.ts` file, and in the terminal where you have `npm run dev` running, you should see a log line that says "Convex functions ready!" (this means your new function code has been successfully deployed) - Test out your updated function - Go to your [Convex dashboard](https://dashboard.convex.dev), navigate to the "Functions" tab (``) and open `myFunctions:listIdeas`. You should now see the new version of your code there! - Click the "Run function" button to try out your new function in the dashboard - In the "Arguments" panel, edit the value of `includeRandom` and verify that you see the correct results in the "Query outcome" panel! Great, we've confirmed in the Convex dashboard that our backend change was successful! But now when visiting [localhost:5173](http://localhost:5173) you'll see a whole lot of nothing. That's because our backend function change broke the frontend code that invokes that function! Let's fix it and get our ideas back. ## Exercise 3: Update your frontend - In your code editor, open `src/App.tsx` - In the `App` function, find the line near the top where `ideas` is defined using the `useQuery()` hook to call the `api.myFunctions.listIdeas` query function - The `useQuery` hook can take an optional second argument, an `args` object that matches the `args` validator of the given query function. Update the call to `useQuery()` to pass `{ includeRandom }` as the second argument, like so: ```js const ideas = useQuery(api.myFunctions.listIdeas, { includeRandom }); ``` - Now, not only are the ideas displaying properly, but when you (un)check the "Include random ideas" checkbox you should see the results update accordingly! ## Bonus: Challenge Exercises If you've got extra time, try your hand at implementing some new features for the app! ### Challenge 1: Delete idea button Currently, there is no way for a user to delete an idea from the page. Your challenge is to add a button that fixes that! Hints: - In `convex/myFunctions.ts` you'll need to create a new mutation function `deleteIdea`, which [deletes a document](https://docs.convex.dev/database/writing-data#deleting-documents) from the `ideas` table using its [Convex document ID](https://docs.convex.dev/database/document-ids) - In `src/App.tsx`, you'll need the `useMutation` hook to invoke your new `deleteIdea` function as needed from the frontend - In `src/App.tsx`, you can add a new button using the `Button` component (see the "Generate a random app idea" button for an example) ### Challenge 2: Prevent duplicates At the moment, the app doesn't prevent you from adding the same idea twice (try it!), so we might save duplicate ideas to the database. Your challenge is to fix that by making sure that we check for duplicates before saving a new idea! Hints: - In `convex/myFu [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 29 recognized source files, 67 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Mistral AI (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (37 of 37) ``` _gitignore .eslintrc.cjs .gitignore components.json convex/_generated/api.d.ts convex/_generated/api.js convex/_generated/dataModel.d.ts convex/_generated/server.d.ts convex/_generated/server.js convex/_schema.ts convex/images.ts convex/myFunctions.ts convex/README.md convex/tsconfig.json index.html package.json postcss.config.js README.md src/App.tsx src/components/typography/code.tsx src/components/typography/link.tsx src/components/ui/button.tsx src/components/ui/checkbox.tsx src/components/ui/header.tsx src/components/ui/input.tsx src/components/ui/label.tsx src/components/ui/splash.tsx src/components/ui/story-block.tsx src/components/ui/story.tsx src/index.css src/lib/utils.tsx src/main.tsx src/vite-env.d.ts tailwind.config.js tsconfig.json tsconfig.node.json vite.config.ts ``` ### Dependencies - package.json: @hookform/resolvers@^3.3.4, @radix-ui/react-checkbox@^1.0.4, @radix-ui/react-icons@^1.3.0, @radix-ui/react-label@^2.0.2, @radix-ui/react-slot@^1.0.2, @types/node@^20.11.19, @types/react@^18.2.55, @types/react-dom@^18.2.19, @typescript-eslint/eslint-plugin@^7.0.1, @typescript-eslint/parser@^7.0.1, @vitejs/plugin-react@^4.2.1, autoprefixer@^10.4.17, class-variance-authority@^0.7.0, clsx@^2.1.0, cmdk@^0.2.1, convex@^1.9.0, date-fns@^3.3.1, eslint@^8.56.0, eslint-plugin-react-hooks@^4.6.0, eslint-plugin-react-refresh@^0.4.5, monsterapi@^0.0.3, npm-run-all@^4.1.5, postcss@^8.4.35, react@^18.2.0, react-day-picker@^8.10.0, react-dom@^18.2.0, react-hook-form@^7.50.1, react-native-get-random-values@^1.10.0, tailwind-merge@^2.2.1, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5.3.3, vite@^5.1.3, zod@^3.22.4 ### Recent commits (newest first) - Final push? - Polished react - Merge pull request #1 from YJC23/frontend - Merge branch 'main' into frontend - pre-merge - add more ui elements - add header.tsx' - styling - Merge remote-tracking branch 'refs/remotes/origin/frontend' into frontend - add story.tsx - add images section - add images section - create splash - Emma's code pushed: - Changes - Merge pull request #4 from get-convex/updates - Format with Prettier 3.2.5 (add semicolons and trailing commas) - Enter works on app idea submitter - Update all dependencies - Update convex ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "convex-hack-pack", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "npm-run-all dev:init --parallel dev:frontend dev:backend", "dev:frontend": "vite --open", "dev:backend": "convex dev", "dev:init": "convex dev --until-success && convex dashboard", "build": "tsc && vite build", "lint": "eslint . --ext ts,tsx ", "preview": "vite preview" }, "dependencies": { "@hookform/resolvers": "^3.3.4", "@radix-ui/react-checkbox": "^1.0.4", "@radix-ui/react-icons": "^1.3.0", "@radix-ui/react-label": "^2.0.2", "@radix-ui/react-slot": "^1.0.2", "class-variance-authority": "^0.7.0", "clsx": "^2.1.0", "cmdk": "^0.2.1", "convex": "^1.9.0", "date-fns": "^3.3.1", "monsterapi": "^0.0.3", "react": "^18.2.0", "react-day-picker": "^8.10.0", "react-dom": "^18.2.0", "react-hook-form": "^7.50.1", "react-native-get-random-values": "^1.10.0", "tailwind-merge": "^2.2.1", "tailwindcss-animate": "^1.0.7", "zod": "^3.22.4" }, "devDependencies": { "@types/node": "^20.11.19", "@types/react": "^18.2.55", "@types/react-dom": "^18.2.19", "@typescript-eslint/eslint-plugin": "^7.0.1", "@typescript-eslint/parser": "^7.0.1", "@vitejs/plugin-react": "^4.2.1", "autoprefixer": "^10.4.17", "eslint": "^8.56.0", "eslint-plugin-react-hooks": "^4.6.0", "eslint-plugin-react-refresh": "^0.4.5", "npm-run-all": "^4.1.5", "postcss": "^8.4.35", "tailwindcss": "^3.4.1", "typescript": "^5.3.3", "vite": "^5.1.3" } } ``` ### src/main.tsx ```typescript import { ConvexProvider, ConvexReactClient } from "convex/react"; import React from "react"; import ReactDOM from "react-dom/client"; import App from "./App"; import "./index.css"; const convex = new ConvexReactClient(import.meta.env.VITE_CONVEX_URL as string); ReactDOM.createRoot(document.getElementById("root")!).render( , ); ``` ### src/App.tsx ```typescript import { useAction, useMutation, useQuery } from "convex/react"; import { api } from "../convex/_generated/api"; import { useState } from "react"; import { Button } from "@/components/ui/button"; import { Input } from "@/components/ui/input"; import { Splash } from "@/components/ui/splash"; import { Header } from "@/components/ui/header"; import { Story } from "@/components/ui/story"; import { StoryBlock } from "@/components/ui/story-block"; // import { Checkbox } from "@/components/ui/checkbox"; // import { Label } from "@/components/ui/label"; function App() { const [newValue, setNewValue] = useState(""); const [isLoading, setIsLoading] = useState(false); const [count, setCount] = useState(0); const handleIncrement = () => { setCount(count + 1); // Update the count state }; const handleDecrement = () => { setCount(count - 1); // Update the count state }; const sentencesIllustrations = useQuery( api.myFunctions.listSentencesIllustrations, {} ); // const sentenceStories = useQuery(api.myFunctions.listSentencesStory, {}); const fetchIllustrations = useAction(api.myFunctions.fetchIllustrations); const listDefault = useMutation(api.myFunctions.listDefault); const listDefault2 = useMutation(api.myFunctions.listDefault2); return ( <>
setNewValue(event.target.value)} placeholder="Choose a value: compassion, kindness, respect for a teacher..." />
{isLoading ? (

{" "} Generating a story that will change how you think ...{" "}

) : (
{count !== 0 && ( )} {sentencesIllustrations && count !== sentencesIllustrations[sentencesIllustrations.length - 1] .storySentences.length - 1 && ( )}
)}
); } export default App; // function App() { // const [newValue, setNewValue] = useState(""); // const sentencesIllustrations = useQuery( // api.myFunctions.listSentencesIllustrations, // {} // ); // const sentenceStories = useQuery(api.myFunctions.listSentencesStory, {}); // // const generateIdea = useAction(api.myFunctions.fetchRandomIdea); // const fetchIllustrations = useAction(api.myFunctions.fetchIllustrations); // return ( // <> //
//

// What value do you want to learn more about? //

//

Let's learn about the values!

//
// setNewValue(event.target.value)} // placeholder="Type your value here" // /> //