# Project export: mentis.ai This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: AI-powered tutor designed to be accessible, interactive and audiovisual. - Devpost: https://devpost.com/software/ai-thena - GitHub: https://github.com/jtcheng26/treehacks24 - Video: https://www.youtube.com/embed/AoTd_J_dEXY?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Best Use of Monster Generative AI APIs (4x XBox Series S [1st] & 1 million Monster API credits [2nd] & $400 Cash [3rd])) - Team: 4 GitHub contributor(s) — Jeff Cheng (14 commits), Kingsley Zhong (10 commits), ak2246@cornell.edu (6 commits), batyr0601 (1 commits) ## Devpost submission (written by the team) ### Inspiration The inspiration behind Mentis emerged from a realization of the vast potential that a personalized learning AI platform holds in transforming education. We envisioned an AI-driven mentor capable of adapting to individual learning styles and needs, making education more accessible, engaging, and effective for everyone. The idea was to create an AI that could dynamically update its teaching content based on live user questions, ensuring that every learner could find a path that suits them best, regardless of their background or level of knowledge. We wanted to build something that encompasses the intersection of: accessibility, interactivity and audiovisual. ### What it does Mentis is an AI-powered educational platform that offers personalized learning experiences across a wide range of topics. It is able to generate and teach animated lesson plans with both visuals and audio, as it generates checkpoint questions for the user and listens to the questions of its users and dynamically adjusts the remainder of the teaching content and teaching methods to suit their individual learning preferences. Whether it's mathematics, science, or economics, Mentis provides tailored guidance, ensuring that users not only receive answers to their questions but also a deep understanding of the subject matter. ### How we built it At its core, a fast API backend powers the intelligent processing and dynamic delivery of educational content, ensuring rapid response to user queries. This backend is complemented by our use of advanced Large Language Models (LLMs), which have been fine-tuned to understand a diverse range of educational topics and specialize in code generation for the best animation, enhancing the platform's ability to deliver tailored learning experiences. We curated a custom dataset in order to leverage LLMs to the fullest and reduce errors in both script and code generation. Using our curated datasets, we were able to fine-tune models using MonsterAPI tailoring our LLMs and improve accuracy.. We implemented several API calls to ensure a smooth and dynamic operation of our platform, for general organization of the lesson plan, script generation, audio generation with ElevenLabs, and code generation for the manim library we utilize to create the animations on our front end in Bun and Next.js. Fine tuned open source model Curated custom dataset ### Challenges we ran into Throughout the development of Mentis, we encountered significant challenges, particularly in setting up environments and installing various dependencies. These hurdles consumed a considerable amount of our time, persisting until the final stages of development. Every stage of our application had issues we had to address: generating dynamic sections for our video scripts, ensuring that the code is able to execute the animation, integrating the text-to-speech component to generate audio for our educational content all introduced layers of complexity, requiring precise tuning and a lot of playing with to set up. The number of API calls needed to fetch, update, and manage content dynamically, coupled with ensuring the seamless interaction between the user and our application, demanded a meticulous approach. We found ourselves in a constant battle to maintain efficiency and reliability, as we tried to keep our latency low for practicality and interactivity of our product. ### Accomplishments we're proud of Despite the setbacks, we are incredibly proud of: Technical Overcomes: Overcoming significant technical hurdles, learning from them and enhancing our problem-solving capabilities. Versatile System: Enabling our platform to cover a broad range of topics, making learning accessible to everyone. Adaptive Learning: Developing a system that can truly adapt to each user's unique learning style and needs. User-Friendly UI: Creating a user-friendly design and experience keeping our application as accessible as possible. API Management: Successfully managing numerous API calls, we smoothed the backend operation as much as possible for a seamless user experience. Fine tuned/Tailored Models: Going through the full process of data exploration & cleaning, model selection, and configuring the fine-tuned model. ### What we learned Throughout the backend our biggest challenge and learning point was the setup, coordination and training of multiple AI agents and APIs. For all of us, this was our first time fine-tuning a LLM and there were many things we learned through this process such as dataset selection, model selection, fine-tuning configuration. We gained an appreciation for all the great work that was being done by the many researchers. With careful tuning and prompting, we were able to greatly increase the efficiency and accuracy of the models. We also learned a lot about coordinating multi-agent systems and how to efficiently have them run concurrently and together. We tested many architectures and ended up settling for one that would optimize first for accuracy then for speed. To accomplish this, we set up an asynchronous query system where multiple “frames” can be generated at once and allow us not to be blocked by cloud computation time. ### What's next Looking ahead, Mentis.ai has exciting plans for improvement and expansion: Reducing Latency: We're committed to enhancing efficiency, aiming to minimize latency further and optimize performance across the platform. Innovative Features: Given more time, we plan to integrate cutting-edge features, like using HeyGen API to create natural videos of personalized AI tutors, combining custom images, videos, and audio for a richer learning experience. Classroom Integration: We're exploring opportunities to bring Mentis into classroom settings, testing its effectiveness in a real-world educational environment and tailoring its capabilities to support teachers and students alike. ## README (from the GitHub repository) # treehacks24 # Backend Create a venv `python3 -m venv .venv` Activate the venv Install requirements `pip3 install -r requirements.txt` Start server: `uvicorn main:app --reload --reload-exclude='scene_code/*'` ## Manim Bullshit `brew install cairo pango ffmpeg mactex pkg-config` `pip install pycairo` ## Detected evidence (automated analysis) Indexed codebase: 37 recognized source files, 87 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (53 of 53) ``` .gitignore backend/codegen.py backend/combined_data.json backend/fine_tuning.ipynb backend/i dont even fucking know.ipynb backend/main.py backend/manim_test.py backend/monster.ipynb backend/monsterapi.ipynb backend/processing.py backend/prompts.py backend/requirements.txt backend/scene_99e5afbf-a517-4ba0-951a-51e507b86901.py backend/scene_code/scene_71526399-84a9-4472-b993-3c61fe4b21a5.py backend/scene_code/scene_99e5afbf-a517-4ba0-951a-51e507b86901.py backend/scene_code/scene_idk.py backend/scenes.json backend/temp_manim_script_MatrixIntroduction.py backend/temp_manim_script_VectorIntroduction.py backend/temp.json backend/temp.py backend/together.ipynb frontend/.eslintrc.json frontend/.gitignore frontend/bun.lockb frontend/next.config.mjs frontend/package.json frontend/postcss.config.js frontend/README.md frontend/src/components/button/PrimaryButton.tsx frontend/src/components/button/ReplayButton.tsx frontend/src/components/button/SecondaryButton.tsx frontend/src/components/content/Content.tsx frontend/src/components/input/InlineInput.tsx frontend/src/components/input/QuestionInput.tsx frontend/src/components/player/Player.tsx frontend/src/components/practice/MCQuizQuestion.tsx frontend/src/components/practice/Skip.tsx frontend/src/components/practice/TextQuizQuestion.tsx frontend/src/components/progress/ProgressBar.tsx frontend/src/components/progress/ProgressBars.tsx frontend/src/components/requests/post.ts frontend/src/components/text/HeaderText.tsx frontend/src/hooks/useManim.tsx frontend/src/pages/_app.tsx frontend/src/pages/_document.tsx frontend/src/pages/api/hello.ts frontend/src/pages/index.tsx frontend/src/styles/globals.css frontend/tailwind.config.ts frontend/tsconfig.json README.md todo.md ``` ### Dependencies - backend/requirements.txt: aiohttp, fastapi, manimgl, pycairo, toml, uvicorn - frontend/package.json: @types/node@^20, @types/react@^18, @types/react-dom@^18, autoprefixer@^10.0.1, eslint@^8, eslint-config-next@14.1.0, next@14.1.0, postcss@^8, react@^18, react-dom@^18, react-player@^2.14.1, react-spinners@^0.13.8, tailwindcss@^3.3.0, typescript@^5 ### Recent commits (newest first) - Additions - fallbacks - code - Fix - openai api - Trying to add videos - Merge branch 'main' of https://github.com/jtcheng26/treehacks24 - Trying to make videos - fashjdklasdfasef - Video pipeline mostly - Basic audio generation - demo - fucking hell - fixes - added returning a file - work so far - asdfghjk - add verify parameter - Merge branch 'main' of https://github.com/jtcheng26/treehacks24 - Add temp monster file ## Key source files (fetched from GitHub, selected and truncated for size) ### todo.md ```markdown # Script Controller - API Endpoint - Save Data Somewhere (along with prompt and repsonse) # Animation Generator 1. What happens when it errors, we want to pipe the error back into the codegen 2. How do we run and save ``` ### backend/requirements.txt ``` fastapi uvicorn manimgl pycairo toml aiohttp ``` ### frontend/package.json ``` { "name": "treehacks", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "next": "14.1.0", "react": "^18", "react-dom": "^18", "react-player": "^2.14.1", "react-spinners": "^0.13.8" }, "devDependencies": { "typescript": "^5", "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "autoprefixer": "^10.0.1", "postcss": "^8", "tailwindcss": "^3.3.0", "eslint": "^8", "eslint-config-next": "14.1.0" } } ``` ### backend/main.py ```python import os import re from fastapi import FastAPI, Request from fastapi.middleware.cors import CORSMiddleware import json import fastapi import prompts import toml import requests import asyncio import uuid import aiohttp import pathlib import textwrap import subprocess import time from starlette.concurrency import run_in_threadpool from moviepy.editor import VideoFileClip, AudioFileClip from moviepy.editor import * # Event loop for querying each scene app = FastAPI() config = toml.load("config.toml") from elevenlabs import generate, play, save, set_api_key set_api_key(config["elevenlabs_api_key"]) async def generate_scene(item, i=0, err=''): global scenes if i > 10: return print("Generating scene: ", item["id"], item["title"]) if item["id"] not in scenes: print("Creating new scene: ", item["id"]) scenes[item["id"]] = {} # item has, key, title and description # Generate the scene information scene_query = prompts.SCENE_AGENT_PROMPT.format(item["description"]) + ( '' if not err else '\nThese were the errors from the last run please resolve them: ' + err) print(scene_query) scene_response = await run_in_threadpool(query_gpt, scene_query, False, True) print(scene_response) work = False while not work: try: print("Trying to parse JSON...") text = scene_response # remove everything before the first { text = text[text.index("{"):] # remove everything after the last } text = text[: text.rindex("}") + 1] print(text) scene_data = json.loads(text) work = True except json.JSONDecodeError as e: scene_query += ( "\n\n" + scene_response["text"][0] + "\n\nTHIS IS NOT VALID JSON. PLEASE FIX IT. RETURN ONLY A VALID JSON FORMAT." ) print("Invalid JSON") print(scene_query) scene_response = await run_in_threadpool(query_gpt, scene_query, False) scenes[item["id"]]["data"] = scene_data # Generate the animation code animation_query = prompts.ANIMATION_PROMPT.format( scene_data["narration"], scene_data["animation-description"] ) print(animation_query) animation_response = await run_in_threadpool(query_gpt, animation_query, False, True) # todo: deal with fine tuning = true print(animation_response) animation_code = animation_response scenes[item["id"]]["code"] = animation_code # Generate the video await generate_video(item["id"], i) # Write the data to a file with open("scenes.json", "w") as f: json.dump(scenes, f, indent=4) @app.get("/api/generate") async def generate_video(item_id: str, i=0, err = ''): if i > 10: print("FAILED TEN TIMES") return print("Generating video: ", item_id) global scenes # scenes = json.load(open("scenes.json")) # Generate the video from the code def extract_code_blocks(input_string, pattern): matches = re.findall(pattern, input_string, re.DOTALL) return matches with open(f"scene_code/scene_{item_id}.py", "w") as f: uh = textwrap.dedent(scenes[item_id]["code"]).rstrip() res = extract_code_blocks(uh, r'```python(.*?)```') res2 = extract_code_blocks(uh, r'```(.*?)```') if len(res) and len(res[0]): res = textwrap.dedent(res[0]).rstrip() f.write( 'from manim import *\nconfig.background_color = "#0F172A"\n' + res) elif len(res2) and len(res2[0]): res2 = textwrap.dedent(res2[0]).rstrip() f.write( 'from manim import *\nconfig.background_color = "#0F172A"\n' + res2) else: f.write('from manim import *\nconfig.background_color = "#0F172A"\n' + uh) # Run the manim command try: # # log all output # cmd_output = os.system(f"manim scene_{item_id}.py -o -lq scene_{item_id}.mp4") # os.system(f"manim scene_{item_id}.py -o -lq scene_{item_id}.mp4") print("new") command = [ "manim", f"scene_code/scene_{item_id}.py", "-ql", "-o", pathlib.Path(__file__).parent / "scene_code" / f"scene_{item_id}", ] async def run_command(command): process = await asyncio.create_subprocess_exec( *command, stdout=subprocess.PIPE, stderr=subprocess.PIPE ) stdout, stderr = await process.communicate() return process.returncode, stdout.decode(), stderr.decode() exitcode, stdout, stderr = await run_command(command) narration_text = scenes[item_id]["data"]["narration"] audio_filename = f"scene_audio/scene_{item_id}.mp3" # Check if the audio file already exists to avoid regenerating it if not os.path.exists(audio_filename): print("Generating audio narration for: ", item_id) # Assuming you have a function `generate_audio` that encapsulates the audio generation logic audio = generate( text=narration_text, voice="Charlie", model="eleven_multilingual_v1" ) # Assuming you have a function `save_audio` to save the generated audio save(audio, audio_filename) video_clip = VideoFileClip(f"scene_code/scene_{item_id}.mp4") audio_clip = AudioFileClip(audio_filename) final_clip = video_clip.set_audio(audio_clip) final_video_filename = f"scene_code/scene_{item_id}.mp4" final_clip.write_videofile(final_video_filename) print("EXITCODE", exitcode) print(stdout) print("STDERRRR", stderr) if (exitcode != 0): if stderr != '' or not os.path.exists(f"scene_code/s [truncated — 7436 more characters] ``` ### frontend/src/pages/index.tsx ```typescript import { Inter } from "next/font/google"; import HeaderText from "@/components/text/HeaderText"; import InlineInput from "@/components/input/InlineInput"; import { useState } from "react"; import Content from "@/components/content/Content"; const inter = Inter({ subsets: ["latin"] }); export default function Home() { const [user, setUser] = useState(""); const [topic, setTopic] = useState(""); const [started, setStarted] = useState(false); function handleStart() { setStarted(true); } function reset() { setUser(""); setTopic(""); setStarted(false); } return (
I am a{" "} Teach me about{" "}
{started ? : ""}
); } ``` ### frontend/postcss.config.js ```javascript module.exports = { plugins: { tailwindcss: {}, autoprefixer: {}, }, }; ``` ### backend/temp_manim_script_MatrixIntroduction.py ```python from manim import * import numpy as np class MatrixIntroduction(Scene): def construct(self): matrix = Matrix([[Symbol('a'), Symbol('b')], [Symbol('c'), Symbol('d')]], v_buff=0.5, h_buff=0.5).set_stroke(width=0.5) matrix_label = MathTex('A = ', font_size=24).next_to(matrix, LEFT) self.play(Write(matrix_label), Write(matrix)) self.wait(1) ``` ### backend/temp_manim_script_VectorIntroduction.py ```python from manim import * import numpy as np class VectorIntroduction(Scene): def construct(self): number_line = NumberLine(x_range=[-10, 10, 0.2], length=10, include_numbers=True) vector = Arrow(start=ORIGIN, end=[2, 0, 0], buff=0).set_color(RED) vector_label = MathTex('v', font_size=24).next_to(vector, UP, buff=0.1) self.play(Create(number_line)) self.play(GrowArrow(vector), Write(vector_label)) self.wait(1) self.play(vector.animate.scale(0.4, about_point=ORIGIN)) self.wait(1) ``` ### frontend/tailwind.config.ts ```typescript import type { Config } from "tailwindcss"; const config: Config = { content: [ "./src/pages/**/*.{js,ts,jsx,tsx,mdx}", "./src/components/**/*.{js,ts,jsx,tsx,mdx}", "./src/app/**/*.{js,ts,jsx,tsx,mdx}", ], theme: { extend: { backgroundImage: { "gradient-radial": "radial-gradient(var(--tw-gradient-stops))", "gradient-conic": "conic-gradient(from 180deg at 50% 50%, var(--tw-gradient-stops))", }, colors: { 'primary': '#4EB389', } }, }, plugins: [], }; export default config; ``` ### backend/scene_99e5afbf-a517-4ba0-951a-51e507b86901.py ```python from manim import * class VectorIntro(Scene): def construct(self): # Set up number line number_line = NumberLine( x_min=-10, x_max=10, unit_size=2, number_of_marks=5, mark_to_number_mappings={ 2: "$2$", 4: "$4$", 6: "$6$", 8: "$8$", } ) # Set up vector vector_label = Tex("v", font_size=25).shift(0.5 * RIGHT) vector_end = 5 * RIGHT vector_start = ORIGIN vector_config = Arrow( vector_start, vector_end, buff=0.2, stroke_width=3, color=YELLOW ).add_label(vector_label) # Set up shrinking vector shrunk_vector_end = 2 * RIGHT shrunk_vector_config = vector_config.copy().shift( shrunk_vector_end - vector_end ) # Display all objects self.play( Create(number_line), Create(vector_config), ) self.wait(1) self.play( vector_config.animate( target=(vector_start, shrunk_vector_end) ), ) self.wait(1) self.play( Uncreate(number_line), Uncreate(vector_config), ) ``` [26 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]