# Project export: GuideStone This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Meet Rocky, your AI Teacher who learns how you learn with self-improving videos - Devpost: https://devpost.com/software/guidestone - GitHub: https://github.com/owenonline/guidestone-functions - Demo: https://github.com/mbruckert/guidestone-frontend - Team: 1 GitHub contributor(s) — Owen Burns (32 commits) ## Devpost submission (written by the team) ### Inspiration Throughout the United States and internationally, there is a huge disparity in the educational opportunities afforded to underrepresented and underfunded communities, leading to significant gaps in academic achievement and access to higher education. These disparities are often compounded by factors such as socioeconomic status, race, and geographic location, which can limit the resources available to students, including access to quality teachers, extracurricular programs, and advanced coursework. On the other side of this problem, both primary/secondary schools and higher learning institutions alike are in desperate need of qualified educators and tutors to teach the next generation, but there is a shortage of teachers available, worsening the aforementioned problem as well as hurting students. When class sizes increase, the level of individual help and resources available to students diminish, leading students across the globe behind. I think we can all agree, at least Owen and I can, that it's easy to feel defeated in classes with hundreds of your closest peers, with little to no support available from teachers just because they are spread to thin. Our goal with GuideStone is to both empower students to truly understand the material they seek to learn, as well as provide a solution for educational systems who want to provide one-on-one teaching/tutoring to their students to see them succeed but don't have the resources or employees to do so. ### What it does GuideStone is an AI education platform that seeks to reinvent tutoring for students just like us. With Rocky as your tutor (the GuideStone mascot) you don't deal with the problems you often experience with services like Khan Academy or platforms like it. (OpenStax, IXL, etc.) These platforms are great, and we have used them countless times, but they are definitely a one-size-fits-all solution, when we all know that each and every one of us has our own learning preferences and we don't have to settle in the world of AI! Introducing GuideStone! We each have our own educational path, and GuideStone understands that. We build out a graph for each and every user that expands as you learn automatically. For example, if you want to learn integrals - but haven't yet learned derivatives, our platform will first generate a lesson on deviates to make sure you have a full grasp on the topic. I mentioned lessons, but you might be wondering.. what does a lesson look like on GuideStone? Well, every lesson comes complete with a video & quiz to test your understanding. How is that better than the earlier mentioned education platforms? Well... Generative AI of course!! Every video on our platform is generated from scratch by GPT-4. We'll dive into how a bit later, but the ability to create videos on topics for a specific user allows us to do some really special things like modifying example animations to be about things you care about and resonate with like hobbies/interests/passions. I also mentioned that GuideStone was self-improving. How does that work? Well, we incorporated a combination of eye-tracking & quizzes to modify your generated videos based on what kind of videos work to better your understanding. While you watch the video, GuideStone is gathering data on what parts of the video you are paying the most attention to without any required input from you. Then, when you finish the video, we use this eye-tracking data to both surface general lesson content, but also specific content we think you might have missed during the video. This creates a really special experience and helps to bring education directly to you like if you had a one-on-one tutor. We use an agentic arcitecture to then analyze this data, and plan a form of action on how to create better videos (one's which perform better on attention & quiz score) for you in the future. This is something that no other education software on the market can offer, and it allows for a much more fulfilling experience. However, that is not the only way that GuideStone is self-improving, we also designed and implemented our own memory architecture so that we are able to remember and track reasons why results were sub-par or even executions failed in the past, and improve upon this in the future. Way more information about this memory architecture is available in 'Things we are Proud of'. ### How we built it We built GuideStone using a combination of cutting-edge software both on the frontend and backend, all of which is hosted on a combination of Azure to ensure quick production of . Let's dive into how both work: Backend: Azure Function App Services to handle all of our cloud infrastructure, including: planning & generating videos, exposing endpoints to the frontend to access Graph & Postgres Databases Azure Apache Gremlin Cosmos DB Graph to manage user's lessons, these graphs are recursively added to by custom-written GPT-4 output parser to manage new nodes and edges between the nodes. PostgreSQL DB to manage user information, lesson information, recommendations for future video improvements, and more GPT-4 for many different reasons, including: scene planning, code generation, graph generation, graph traversal, and more. Langchain was at the center of a lot of our LLM usage, providing us an easy layer with which we could add output parsing, agentic bhavior, and more. We also <3 LangSmith, it made debugging LLMs a lot easier. ElevenLabs for Text-to-Speech Now let's dive in a bit more detail into how video generation & personalization works: We have a listener that is in charge of looking for new/modified nodes, and is responsible for putting these nodes in queue to generate lessons & videos for. Video generation start's with an agent that is tasked with planning out a video, a "director" of sorts, who splits the lessons into scenes, going into detail about what should be displayed on the screen. This agent is also in charge of developing a script which is going to be read aloud by ElevenLabs TTS later in the process. In parallel, these scenes are passed to another agent whose job it is to turn these scene descriptions into Manim (an animation library) code. We noticed in our testing that this agent would make mistakes causing the animation to fail, so we put it in a situation where the agent had a critique that responds to it's work and points out errors, causing the initial agent to rethink it's original code. At the same time as the Manim code generation, we pass the script to ElevenLabs to turn this text into speech. Once we have, both the Manim code and speech generations, we run code first to render the code into animations and then using FFMPEG, we put them together formulating the complete video, which is then put into an Azure storage bucket. Once a user finishes a lesson video & quiz, their results (both eye-tracking & quiz) are sent to the backend for processing. If a user get's a question wrong, a new lesson is generated focusing on this topic and any topics it was detected they might have missed. Regardless of whether they get the answers right or wrong, another agent is dispatched in order to reflect on the quality of the lesson. This agent questions the videos pace, scene types, animation types, etc. all in order to create better, more educational future videos for the user. Frontend: React for Frontend Framework WebGazer.js for Eye-tracking react-force-graph for Graph Creation Github Primer for UI Google OAuth for Authentication When a user is created, they go through an eye-tracking setup process where they are asked to follow their cursor with their eyes and click gradient circles placed around their screen. This entire process takes about 10-15 seconds, yet in the background is training a regression model on every click and by the end we have enough accuracy for our purpose. This model is then saved in the user's localstorage so users do not have to complete this setup every time. The user also is asked for their hobbies/interests at the next step of onboarding, which then generates DALL-E assets and places them in the user's videos in order to make the videos more engaging, relatable, and personable. When a user watches a video, their eye position is tracked multiple times a second and is paired with the timestamp of the video for processing on the backend. Additionally, the number & timestamps of pauses & rewinds are gathered and sent to the backend for additional data analysis. ### Challenges we ran into We ran into MANY problems - but the following were the biggest time-sinks: Recursive Graph Building - Our agent had a lot of problems connecting new nodes and populating parents in a way that connected with existing nodes, and this required A LOT of prompt engineering to improve. This ended up being the major downfall of our product, because while it worked sometimes, it is very inconsistent. Graph Rendering - We tried a number of solutions for this before we finally ended up using (discovering) the right solution. At one point, we had even written our own physics engine to simulate the graph in a package made for flowcharts, but found that to be too unsustainable. ### Accomplishments we're proud of We designed and build out a memory architecture that we are really proud of. We used the critic agent to enrich the logs before they wen't into memory, highlighting errors which could be useful in future runs. We very quickly saw a HUGE improvement in video output quality once we implemented this memory architecture. We knew that fine-tuning the model would require more high-quality data than we had access to or had time to generate/find, so instead we took advantage of the fact that while generating an answer to a question is a high-entropy task, verifying that answer is a much lower-entropy task by adding a critical agent posing as a user to the agent loop. We then summarized the entirety of the execution at the end of each run and saved all of the improvements that the critics feedback were able to cause. Eventually, we were able to disable the critic entirely, massively increasing speed while maintaining the same level of quality, which would not have been possible with GPT-4 without this context. I think also getting over the challenges listed above we are also really proud of- especially since we only got about 3 hours of sleep over the weekend :) ### What we learned We learned A LOT from Treehacks. One takeaway we both agreed on is how valuable incorporating long-term memory with a critic across runs to an LLM can be to performance. But here are some things we each were super excited to have learned: Mark: Graphs are fun, but making them in the frontend is harder than it looks, especially if you try to do it yourself What Owen said Owen: "I learned the reason that most apps only have one database" ### What's next GuideStone was already a continuation of an idea from a previous hackathon (idea transformed into something completely different, no code crossover whatsoever), so we are excited to see where it can go from here! ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 16 recognized source files, 76 KB. - LangChain (technology) — detected in the code - Python (language) — detected in the code - OpenAI (technology) — claimed on Devpost, not found in the code - PostgreSQL (technology) — claimed on Devpost, not found in the code - React (technology) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (21 of 21) ``` .funcignore .github/workflows/main_guidestone-functions.yml .gitignore function_app.py grader/__init__.py grader/grade.py grader/metrics.py graph/__init__.py graph/api.py graph/expand.py graph/traverse.py host.json lesson/__init__.py lesson/audio.py lesson/create.py lesson/video.py requirements.txt testing.ipynb users/__init__.py users/auth.py users/new.py ``` ### Dependencies - requirements.txt: aiohttp@==3.8.3, ansi2html@==1.9.1, azure-functions@==1.18.0, azure-storage-blob@==12.19.0, azure-storage-queue@==12.9.0, chromadb@==0.4.14, gremlinpython@==3.6.4, langchain@==0.1.0, langchain_community@==0.0.10, langchain_core@==0.1.22, langchain_openai@==0.0.2, langgraph@==0.0.24, moviepy@==1.10.3, psycopg2-binary@==2.9.9, pydantic@==1.10.2, Requests@==2.29.0 ### Recent commits (newest first) - r - final commit - we back - adaws - dasds - commented out most of the lesson create.py code - disabled add user function but left import - removed unused imports - addede everything except lesson creator back - re-enabled graph imports - disabled everything except for health check - fixed saving final video - added changed files - attempting to fix downtime - auth fix - potentially fixed onboarding - pushing changes - fixed token exchange - create user finally returns an id, and get graph structure returns node names - fixed token exchange and api ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` # DO NOT include azure-functions-worker in this file # The Python Worker is managed by Azure Functions platform # Manually managing azure-functions-worker may cause unexpected issues azure-functions==1.18.0 azure-storage-queue==12.9.0 azure-storage-blob==12.19.0 chromadb==0.4.14 gremlinpython==3.6.4 langchain==0.1.0 langchain_community==0.0.10 langchain_core==0.1.22 langchain_openai==0.0.2 langgraph==0.0.24 psycopg2-binary==2.9.9 pydantic==1.10.2 Requests==2.29.0 aiohttp==3.8.3 ansi2html==1.9.1 moviepy==1.10.3 ``` ### function_app.py ```python from typing import List import typing import azure.functions as func from gremlin_python.driver import client, serializer import os import logging from psycopg2 import pool import json from grader.grade import score_user from grader.metrics import calculate_attention from graph.api import get_graph_structure, get_node_details from graph.expand import expand_graph from graph.traverse import traverse_graph from lesson.create import create_lesson from users.auth import exchange_token from users.new import create_new_user app = func.FunctionApp(http_auth_level=func.AuthLevel.ANONYMOUS) @app.function_name("healthcheck") @app.route(route="healthcheck", auth_level=func.AuthLevel.ANONYMOUS, methods=['GET','POST']) def healthcheck(req: func.HttpRequest) -> func.HttpResponse: # verify gremlin connection graph_client = client.Client('wss://guidestone-gremlin.gremlin.cosmos.azure.com:443/','g', username=f"/dbs/guidestone/colls/knowledge-graph", password=os.getenv("KNOWLEDGE_GRAPH_KEY"), message_serializer=serializer.GraphSONSerializersV2d0()) # verify postgres connection postgreSQL_pool = pool.SimpleConnectionPool(1, int(os.getenv("PYTHON_THREADPOOL_THREAD_COUNT")), os.getenv("POSTGRES_CONN_STRING")) conn = postgreSQL_pool.getconn() postgreSQL_pool.putconn(conn) postgreSQL_pool.closeall() return func.HttpResponse( status_code=200 ) @app.function_name("createUser") @app.route(route="createUser", auth_level=func.AuthLevel.ANONYMOUS, methods=['POST']) def createUser(req: func.HttpRequest) -> func.HttpResponse: try: user_id = create_new_user(req.get_json()) return func.HttpResponse( status_code=200, body=json.dumps({"user_id": user_id}) ) except Exception as e: logging.exception(e) return func.HttpResponse( status_code=500 ) @app.function_name("exchangeToken") @app.route(route="exchangeToken", auth_level=func.AuthLevel.ANONYMOUS, methods=['POST']) def exchangeToken(req: func.HttpRequest) -> func.HttpResponse: return func.HttpResponse( status_code=200, body=json.dumps(exchange_token(req.get_json())) ) @app.function_name("getGraphStructure") @app.route(route="getGraphStructure", auth_level=func.AuthLevel.ANONYMOUS, methods=['POST']) def getGraphStructure(req: func.HttpRequest) -> func.HttpResponse: graph_structure = get_graph_structure(req.get_json()) return func.HttpResponse( status_code=200, body=json.dumps(graph_structure) ) @app.function_name("getNodeDetails") @app.route(route="getNodeDetails", auth_level=func.AuthLevel.ANONYMOUS, methods=['POST']) def getNodeDetails(req: func.HttpRequest) -> func.HttpResponse: node_details = get_node_details(req.get_json()) return func.HttpResponse( status_code=200, body=json.dumps(node_details) ) @app.function_name("expandGraph") @app.route(route="expandGraph", auth_level=func.AuthLevel.ANONYMOUS, methods=['POST']) @app.queue_output(arg_name='queue', queue_name='node-updated', connection="AzureWebJobsStorage") def expandGraph(req: func.HttpRequest, queue: func.Out[str]) -> func.HttpResponse: req_json: dict = req.get_json() expand_graph(req_json) # queue.set(req_json['user_id']) return func.HttpResponse( status_code=200 ) @app.function_name("traverseGraph") @app.queue_trigger(arg_name='queuein', queue_name='node-updated', connection="AzureWebJobsStorage") def traverseGraph(queuein: func.QueueMessage, context) -> None: traverse_graph(queuein.get_body().decode("utf-8")) @app.function_name("createLesson") @app.queue_trigger(arg_name='queuemessage', queue_name='lesson-regenerate', connection="AzureWebJobsStorage") def createLesson(queuemessage: func.QueueMessage, context) -> None: create_lesson(queuemessage.get_json()) @app.function_name("lessonDone") @app.route(route="lessonDone", auth_level=func.AuthLevel.ANONYMOUS, methods=['POST']) @app.queue_output(arg_name='queue', queue_name='quiz-taken', connection="AzureWebJobsStorage") def lessonDone(req: func.HttpRequest, queue: func.Out[str]) -> func.HttpResponse: graph_client = client.Client('wss://guidestone-gremlin.gremlin.cosmos.azure.com:443/','g', username=f"/dbs/guidestone/colls/knowledge-graph", password=os.getenv("KNOWLEDGE_GRAPH_KEY"), message_serializer=serializer.GraphSONSerializersV2d0()) postgreSQL_pool = pool.SimpleConnectionPool(1, int(os.getenv("PYTHON_THREADPOOL_THREAD_COUNT")), os.getenv("POSTGRES_CONN_STRING")) conn = postgreSQL_pool.getconn() cursor = conn.cursor() req_json = req.get_json() attention_score = calculate_attention(req_json['vision_points'], req_json['node_id']) queue.set(json.dumps({"user_id": req_json['user_id'], "node_id": req_json['node_id'], "attention_score": attention_score, "quiz_data": req_json["quiz_data"]})) # mark node status as grading graph_client.submit(f"g.V('{req_json['node_id']}').property('status', 'scoring')") return func.HttpResponse( status_code=200 ) @app.function_name("gradeQuiz") @app.queue_trigger(arg_name='queuein', queue_name='quiz-taken', connection="AzureWebJobsStorage") @app.queue_output(arg_name='queueout', queue_name='node-updated', connection="AzureWebJobsStorage [truncated — 1146 more characters] ``` ### users/auth.py ```python from typing import List import azure.functions as func from gremlin_python.driver import client, serializer import os import logging from psycopg2 import pool from pydantic import BaseModel, Field, ValidationError, root_validator, validator from enum import Enum from langchain_openai import AzureChatOpenAI from langchain.output_parsers import OutputFixingParser, PydanticOutputParser from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder from operator import itemgetter import requests from graph.api import get_graph_structure, get_node_details import re import json class TokenExchangeRequest(BaseModel): code: str redirect_uri: str def exchange_token(request_json: dict) -> dict[str, any]: try: req_json = TokenExchangeRequest(**request_json) except Exception as e: print("Could not parse token exchange request: " + str(e)) token_endpoint = 'https://www.googleapis.com/oauth2/v4/token' payload = { 'code': req_json.code, 'client_id': os.getenv("GOOGLE_CLIENT_ID"), 'client_secret': os.getenv("GOOGLE_CLIENT_SECRET"), 'redirect_uri': req_json.redirect_uri, 'grant_type': 'authorization_code', } response = requests.post(token_endpoint, data=payload).json() return response ``` ### lesson/audio.py ```python import requests from pydantic import BaseModel from langchain_openai import AzureChatOpenAI from langchain.output_parsers import OutputFixingParser, PydanticOutputParser from langchain.prompts import ChatPromptTemplate from langchain_openai import AzureChatOpenAI from langchain.output_parsers import OutputFixingParser, PydanticOutputParser from operator import itemgetter import os class Script(BaseModel): txt: str def create_audio(prompt: str, visuals: str, folder: str, i: int): gpt_4_llm = AzureChatOpenAI(deployment_name="gpt-4-turbo", api_version="2023-07-01-preview", model_name="gpt-4-1106-preview", temperature=0, max_retries=10) s_parser = PydanticOutputParser(pydantic_object=Script) lp_fixing_parser = OutputFixingParser.from_llm(parser=s_parser, llm=gpt_4_llm) script_writing_prompt = ChatPromptTemplate.from_messages( [ ('system', "Your job is to write a script for a scene of an educational video based on the information below. Your script must contain nothing but the words that are going to be spoken. Anything else will mess up the recording session."), ('user', "This is an overview of what the script needs to contain: {prompt}. While your script is being recited, this will be on stage, so keep that in mind: {visuals}") ] ) script_writing_chain = ( { "prompt": itemgetter("prompt"), "visuals": itemgetter("visuals"), } | script_writing_prompt | gpt_4_llm | lp_fixing_parser ) script = script_writing_chain.invoke({ "prompt": prompt, "visuals": visuals }) response = requests.post("https://api.elevenlabs.io/v1/text-to-speech/fJE3lSefh7YI494JMYYz", json={ "text": script.txt, "voice_settings": { "similarity_boost": 0.75, "stability": 0.5, } }, headers={ "Accept": "audio/mpeg", "Content-Type": "application/json", "xi-api-key": os.getenv("ELEVEN_LABS_API_KEY") }) with open(os.path.join(folder, f"voiceover_{i}.mp3"), 'wb') as f: for chunk in response.iter_content(chunk_size=1024): if chunk: f.write(chunk) return os.path.join(folder, "voiceover.mp3") ``` ### graph/traverse.py ```python from typing import List from gremlin_python.driver import client, serializer import os import logging from psycopg2 import pool from pydantic import BaseModel, Field from enum import Enum, auto from azure.storage.queue import QueueClient, TextBase64EncodePolicy, TextBase64DecodePolicy from langchain_openai import AzureChatOpenAI from langchain.output_parsers import OutputFixingParser, PydanticOutputParser from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder from operator import itemgetter import re import json def send_update_message(node_id, user_id): with QueueClient.from_connection_string(conn_str=os.environ['AzureWebJobsStorage'], queue_name="lesson-regenerate", message_encode_policy = TextBase64EncodePolicy(), message_decode_policy = TextBase64DecodePolicy()) as queue_client: queue_client.send_message(json.dumps({"node_id": node_id, "user_id": user_id})) def traverse_graph(user_id: str) -> list[str]: graph_client = client.Client('wss://guidestone-gremlin.gremlin.cosmos.azure.com:443/','g', username=f"/dbs/guidestone/colls/knowledge-graph", password=os.getenv("KNOWLEDGE_GRAPH_KEY"), message_serializer=serializer.GraphSONSerializersV2d0()) def update_node_status(node_id): # Fetch the node and its status cb = graph_client.submit(f"g.V('{node_id}').values('status')") node_status = cb.all().result()[0] if node_status in ['ready', 'scoring', 'regen', 'firstgen']: # Do not search children return elif node_status == 'completed': # Progress to children child_id_callback = graph_client.submit(f"g.V('{node_id}').out().id()") child_ids = child_id_callback.all().result() for child_id in child_ids: update_node_status(child_id) elif node_status == 'graded': # Send out a message to update the level content send_update_message(node_id, user_id) elif node_status == 'unstarted': # Check if all incoming nodes are 'completed' invals_callback = graph_client.submit(f"g.V('{node_id}').in().values('status')") invals = invals_callback.all().result() if all(status == 'completed' for status in invals): # Update status to 'firstgen' and send out a message graph_client.submit(f"g.V('{node_id}').property('status', 'firstgen')") send_update_message(node_id, user_id) gc_callback = graph_client.submit(f"g.V().hasLabel('start_node').has('user_id', '{user_id}').values('id')") root_node_id = gc_callback.all().result()[0] update_node_status(root_node_id) ``` ### graph/api.py ```python from gremlin_python.driver import client, serializer import os import logging from psycopg2 import pool from pydantic import BaseModel from azure.storage.blob import BlobServiceClient class GraphStructureRequest(BaseModel): user_id: int class NodeDetailRequest(BaseModel): node_id: str class GraphError(Exception): pass def get_graph_structure(req_json: dict) -> dict[str, any]: graph_client = client.Client('wss://guidestone-gremlin.gremlin.cosmos.azure.com:443/','g', username=f"/dbs/guidestone/colls/knowledge-graph", password=os.getenv("KNOWLEDGE_GRAPH_KEY"), message_serializer=serializer.GraphSONSerializersV2d0()) postgreSQL_pool = pool.SimpleConnectionPool(1, int(os.getenv("PYTHON_THREADPOOL_THREAD_COUNT")), os.getenv("POSTGRES_CONN_STRING")) conn = postgreSQL_pool.getconn() cursor = conn.cursor() try: get_graph_body = GraphStructureRequest(**req_json) except Exception as e: logging.error("Could not parse get graph structure request: " + str(e)) node_id_callback = graph_client.submit(f"g.V().has('user_id', '{get_graph_body.user_id}').values('id', 'table_id').fold()") node_table_ids = node_id_callback.all().result()[0] # base_ids_callback = graph_client.submit(f"g.V().hasLabel('start_node').has('user_id', '{get_graph_body.user_id}').out().values('id').fold()") # base_ids = base_ids_callback.all().result()[0] edges = [] nodes = [] node_names = {} for node_id, table_id in zip(node_table_ids[::2], node_table_ids[1::2]): # building a list of just node ids nodes.append(node_id) # building a list of edges edge_callback = graph_client.submit(f"g.V('{node_id}').outE().inV().values('id').fold()") edge_results = edge_callback.all().result()[0] edges.extend([(node_id, edge) for edge in edge_results]) # building a dictionary of names cursor.execute("SELECT public_name FROM nodes WHERE id = %s", (table_id,)) public_name, = cursor.fetchone() node_names[node_id] = public_name return { "nodes": nodes, # "bases": base_ids, "edges": edges, "names": node_names } def get_node_details(req_json: dict) -> dict[str, any]: graph_client = client.Client('wss://guidestone-gremlin.gremlin.cosmos.azure.com:443/','g', username=f"/dbs/guidestone/colls/knowledge-graph", password=os.getenv("KNOWLEDGE_GRAPH_KEY"), message_serializer=serializer.GraphSONSerializersV2d0()) postgreSQL_pool = pool.SimpleConnectionPool(1, int(os.getenv("PYTHON_THREADPOOL_THREAD_COUNT")), os.getenv("POSTGRES_CONN_STRING")) conn = postgreSQL_pool.getconn() cursor = conn.cursor() try: node_details_body = NodeDetailRequest(**req_json) except Exception as e: logging.error("Could not parse node details request: " + str(e)) node_details_callback = graph_client.submit(f"g.V('{node_details_body.node_id}').valueMap()") node_details_results = node_details_callback.all().result()[0] logging.info(node_details_results) # get the lesson info if node_details_results['lesson_id'][0] != '-1': cursor.execute("SELECT lesson_description, video_id, quiz FROM lessons WHERE id = %s", (node_details_results['lesson_id'][0],)) lesson_description, video_id, quiz = cursor.fetchone() else: lesson_description, video_id, quiz = None, None, None # get node info cursor.execute("SELECT public_name, learning_status, masteries, blurb FROM nodes WHERE id = %s", (node_details_results['table_id'][0],)) public_name, learning_status, masteries, blurb = cursor.fetchone() if video_id is None: video_url = None else: blob_service_client = BlobServiceClient.from_connection_string(os.getenv("AzureWebJobsStorage")) container_client = blob_service_client.get_container_client("videos") blob_client = container_client.get_blob_client(video_id) video_url = blob_client.url if not quiz is None: quiz = {question: (quiz[question]['choices'], quiz[question]['correct_index']) for question in quiz.keys()} if masteries == {}: masteries = None if learning_status == []: learning_status = None else: learning_status = learning_status[-1] return { "lesson_description": lesson_description, "node_name": public_name, "video_url": video_url, "quiz": quiz, "masteries": masteries, "blurb": blurb, "learning_status": learning_status } ``` ### grader/metrics.py ```python import cv2 from scipy.interpolate import CubicSpline from concurrent.futures import ThreadPoolExecutor, as_completed import numpy as np import requests from typing import List from gremlin_python.driver import client, serializer import os import logging from psycopg2 import pool from pydantic import BaseModel, Field from enum import Enum, auto from azure.storage.queue import QueueClient, TextBase64EncodePolicy, TextBase64DecodePolicy from langchain_openai import AzureChatOpenAI from langchain.output_parsers import OutputFixingParser, PydanticOutputParser from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder from operator import itemgetter import re from azure.storage.blob import BlobServiceClient import json from skimage.metrics import structural_similarity as compare_ssim def calculate_attention(points, node_id): graph_client = client.Client('wss://guidestone-gremlin.gremlin.cosmos.azure.com:443/','g', username=f"/dbs/guidestone/colls/knowledge-graph", password=os.getenv("KNOWLEDGE_GRAPH_KEY"), message_serializer=serializer.GraphSONSerializersV2d0()) postgreSQL_pool = pool.SimpleConnectionPool(1, int(os.getenv("PYTHON_THREADPOOL_THREAD_COUNT")), os.getenv("POSTGRES_CONN_STRING")) conn = postgreSQL_pool.getconn() cursor = conn.cursor() # get video table_id_callback = graph_client.submit(f"g.V('{node_id}').values('table_id')") table_id = table_id_callback.all().result()[0] cursor.execute("SELECT video_id FROM nodes WHERE id = %s", (table_id,)) video_id, = cursor.fetchone() blob_service_client = BlobServiceClient.from_connection_string(os.getenv("AzureWebJobsStorage")) container_client = blob_service_client.get_container_client("videos") blob_client = container_client.get_blob_client(video_id) stream = blob_client.download_blob() video_bytes = stream.readall() temp_video_path = 'temp_video.mp4' with open(temp_video_path, 'wb') as temp_video_file: temp_video_file.write(video_bytes) # smoothen motion times = [point['time'] for point in points] x_coords = [point['x'] for point in points] y_coords = [point['y'] for point in points] cs_x = CubicSpline(times, x_coords) cs_y = CubicSpline(times, y_coords) smooth_times = np.linspace(min(times), max(times), 100) smooth_x = cs_x(smooth_times) smooth_y = cs_y(smooth_times) smooth_points = [{"x": x, "y": y, "time": t} for x, y, t in zip(smooth_x, smooth_y, smooth_times)] # get the frames frames = [] cap = cv2.VideoCapture("temp_video.mp4") fps = cap.get(cv2.CAP_PROP_FPS) # Get frames per second of the video for timestamp in [timestamps['time'] for timestamps in smooth_points]: cap.set(cv2.CAP_PROP_POS_MSEC, timestamp*1000) ret, frame = cap.read() if ret: frames.append(frame) else: print(f"Frame for timestamp {timestamp} not found.") cap.release() ssim_values = [] position_differences = [] for i in range(len(frames) - 1): frame1 = frames[i] frame2 = frames[i + 1] # Convert frames to grayscale for SSIM calculation gray1 = cv2.cvtColor(frame1, cv2.COLOR_BGR2GRAY) gray2 = cv2.cvtColor(frame2, cv2.COLOR_BGR2GRAY) # Calculate SSIM between two consecutive frames ssim, _ = compare_ssim(gray1, gray2, full=True) ssim_values.append(ssim) # Calculate difference in position between coordinates of two consecutive frames coord1 = (smooth_points[i]['x'], smooth_points[i]['y']) coord2 = (smooth_points[i + 1]['x'], smooth_points[i + 1]['y']) position_diff = np.sqrt((coord2[0] - coord1[0]) ** 2 + (coord2[1] - coord1[1]) ** 2) position_differences.append(position_diff) ssim_derivatives = np.diff(ssim_values) position_derivatives = np.diff(position_differences) correlation_coefficient = np.corrcoef(ssim_derivatives, position_derivatives)[0, 1] attention_score = abs(correlation_coefficient) return attention_score def calculate_pace(rewinds, node_id): graph_client = client.Client('wss://guidestone-gremlin.gremlin.cosmos.azure.com:443/','g', username=f"/dbs/guidestone/colls/knowledge-graph", password=os.getenv("KNOWLEDGE_GRAPH_KEY"), message_serializer=serializer.GraphSONSerializersV2d0()) postgreSQL_pool = pool.SimpleConnectionPool(1, int(os.getenv("PYTHON_THREADPOOL_THREAD_COUNT")), os.getenv("POSTGRES_CONN_STRING")) conn = postgreSQL_pool.getconn() cursor = conn.cursor() # get video table_id_callback = graph_client.submit(f"g.V('{node_id}').values('table_id')") table_id = table_id_callback.all().result()[0] cursor.execute("SELECT video_id FROM nodes WHERE id = %s", (table_id,)) video_id, = cursor.fetchone() blob_service_client = BlobServiceClient.from_connection_string(os.getenv("AzureWebJobsStorage")) container_client = blob_service_client.get_container_client("videos") blob_client = container_client.get_blob_client(video_id) stream = blob_client.download_blob() video_bytes = stream.readall() temp_video_path = 'temp_video.mp4' with open(temp_video_path, 'wb') as temp_video_file: temp_video_file.write(video_bytes) cap = cv2.VideoCapture(temp_video_path) fps = cap.get(cv2.CAP_PROP_FPS) # Get the total number of frames in the video frame_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) # Calculate the duration of the video in seconds duration_seconds = frame_count / fps rewinds_per_second = len(rewinds) / duration_seconds secs_between_rewinds = [rewinds[i+1]['from'] - rewinds[i]['from'] for i in range(len(rewinds)-1)] # Calculate the average of these differences average_secs_between_rewinds = sum(secs_between_rewinds) / len(secs_be [truncated — 77 more characters] ``` ### grader/grade.py ```python import cv2 from scipy.interpolate import CubicSpline from concurrent.futures import ThreadPoolExecutor, as_completed import numpy as np import requests from typing import List from gremlin_python.driver import client, serializer import os import logging from psycopg2 import pool from pydantic import BaseModel, Field from enum import Enum, auto from azure.storage.queue import QueueClient, TextBase64EncodePolicy, TextBase64DecodePolicy from langchain_openai import AzureChatOpenAI from langchain.output_parsers import OutputFixingParser, PydanticOutputParser from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder from operator import itemgetter import re from azure.storage.blob import BlobServiceClient import json from skimage.metrics import structural_similarity as compare_ssim class AfterLessonReport(BaseModel): teaching_effectiveness_report: str = Field(description="A report on the effectiveness of the teaching in the video. This should be a summary of the impact the video had on the user, and should be based on the attention scores and the pause and rewind data. Be sure to include briefly what the video was about, what teaching techniques contributed to that, and which subtracted. Make recommendations for the future and summarize your predictions about the student's learning style") learning_state: str = Field(description="A summary of the user's learning state. This should include what topics they have mastered, what topics they have struggled with, and what topics they are currently learning. This should be based more heavily on the quiz data. Be sure to include what topics you think the user might have missed, and what topics they might have been confused about.") def score_user(node_id, quiz_data, attn_score, masteries): graph_client = client.Client('wss://guidestone-gremlin.gremlin.cosmos.azure.com:443/','g', username=f"/dbs/guidestone/colls/knowledge-graph", password=os.getenv("KNOWLEDGE_GRAPH_KEY"), message_serializer=serializer.GraphSONSerializersV2d0()) gpt_4_llm = AzureChatOpenAI(deployment_name="gpt-4-turbo", api_version="2023-07-01-preview", model_name="gpt-4-1106-preview", temperature=0, max_retries=10) postgreSQL_pool = pool.SimpleConnectionPool(1, int(os.getenv("PYTHON_THREADPOOL_THREAD_COUNT")), os.getenv("POSTGRES_CONN_STRING")) conn = postgreSQL_pool.getconn() cursor = conn.cursor() # get video table_id_callback = graph_client.submit(f"g.V('{node_id}').values('table_id')") table_id = table_id_callback.all().result()[0] cursor.execute("SELECT video_id FROM nodes WHERE id = %s", (table_id,)) video_id, = cursor.fetchone() quiz_data = [] for count, question in enumerate(quiz_data): choices_str = " | ".join(question["choices"]) quiz_data.append(f"{count+1}. Question: {question['question']}, Choices: {choices_str}, Correct Answer: {question['choices'][question['correct_index']]}") quiz_data_str = "\n".join(quiz_data) grading_prompt_template = ChatPromptTemplate( [ ("system", """You are a helpful, critiquing (but ultimately friendly) evaluator evaluating someone's performance on watching a video about and then answering questions on a topic. The topic is: {topic_name} You have been passed the following information about how engaged they were by the video: They got an attention score of {attention_score} on a scale from 0 to 1, with 1 being perfect attention and 0 being no attention. Some important things that you can point out based on this information is what information you think the person watching the video might have missed based on their attention score. This can be validated by the quiz that the user took after watching the video. In fact, you should take more relevance from this quiz than the attention score. Here are the questions, choices, and answers they selected: {quiz_data_str} If the user missed a question, a new video is going to be created going over what they missed, so keep in mind what you think is a gap in the user's understanding. You are also going to need to generate a summary of the user's progress, so also be thinking of a summary of their performance. Then, finally, you are going to need to evaluate the teaching received from the video, focusing on the impact it had specifically on the person watching it. Based on the attention scores, and the following pause and rewind data, think about the following questions: 1. Is the pace good or does it need to slow down to better accommodate the viewer? 2. What kind of scenes did this user prefer? Do they like to see a list of formulas on the screen? Prefer detailed animations? Like blocks of text? Predict what their learning style might be. 3. What kind of narration style did the user prefer? Did they like to hear a lot of anecdoates? Did they prefer to stick to the facts? Or did they want to have more time to think out solutions on their own? Predict what their learning style might be. Number of rewinds per second: {rewind_per_sec} Avg number of second before the student rewound again: {sec_b4_rewind} previously, they had mastered these topics: {mastered_topics} and struggled with these: {struggled_topics} and the state of their learning was: {learning_state} Output your response following this template: {formatting_instructions} """) ] ) dpe_parser = PydanticOutputParser(pydantic_object=AfterLessonReport) dpe_fixing_parser = OutputFixingParser.from_llm(parser=dpe_parser, llm=gpt_4_llm) grade_chain = ( { "topic_name": itemgetter("topic_name"), "attention_score": itemgetter, "quiz_data_str": itemgetter("quiz_data_str"), "rewind_per_sec": itemgetter("rewind_per_sec"), "sec_b4_rewind": itemgetter("sec_b4_rewind"), "mastered_topics": itemgetter("mastered_topics"), [truncated — 1071 more characters] ``` ### users/new.py ```python from gremlin_python.driver import client, serializer import os import logging from psycopg2 import pool from pydantic import BaseModel, Field from enum import Enum import json from stemtopics import Topics class GradeLevel(Enum): KINDERGARTEN = "K" FIRST_GRADE = "1" SECOND_GRADE = "2" THIRD_GRADE = "3" FOURTH_GRADE = "4" FIFTH_GRADE = "5" SIXTH_GRADE = "6" SEVENTH_GRADE = "7" EIGHTH_GRADE = "8" NINTH_GRADE = "9" TENTH_GRADE = "10" ELEVENTH_GRADE = "11" TWELFTH_GRADE = "12" FRESHMAN = "F" SOPHOMORE = "S" JUNIOR = "J" SENIOR = "SR" STARTING_KNOWLEDGE: dict[GradeLevel, Topics] = { GradeLevel.KINDERGARTEN: { Topics.COUNTING, Topics.BASIC_SHAPES, Topics.ADDITION, Topics.SUBTRACTION, Topics.COMPUTER_BASICS, }, GradeLevel.FIRST_GRADE: { Topics.COUNTING, Topics.ADDITION, Topics.SUBTRACTION, Topics.BASIC_GEOMETRY, Topics.SIMPLE_MACHINES, }, GradeLevel.SECOND_GRADE: { Topics.PLACE_VALUES, Topics.TIME_TELLING, Topics.MONEY_MATH, Topics.MULTIPLICATION, Topics.BASIC_PLANT_BIOLOGY, }, GradeLevel.THIRD_GRADE: { Topics.MULTIPLICATION, Topics.DIVISION, Topics.SIMPLE_FRACTIONS, Topics.MEASUREMENTS, Topics.PHOTOSYNTHESIS, }, GradeLevel.FOURTH_GRADE: { Topics.DECIMALS, Topics.FRACTIONS, Topics.AREA_AND_PERIMETER, Topics.ELECTRICITY_AND_MAGNETISM, Topics.ELEMENTS_AND_PERIODIC_TABLE, }, GradeLevel.FIFTH_GRADE: { Topics.VOLUME, Topics.FRACTIONS, Topics.MIXTURES_AND_SOLUTIONS, Topics.LIGHT_AND_OPTICS, Topics.CELL_STRUCTURE_AND_FUNCTION, }, GradeLevel.SIXTH_GRADE: { Topics.RATIOS, Topics.NEGATIVE_NUMBERS, Topics.ECOSYSTEMS_AND_BIOMES, Topics.ENERGY_TYPES_AND_CONVERSION, Topics.CHEMICAL_REACTIONS, }, GradeLevel.SEVENTH_GRADE: { Topics.PROBABILITY, Topics.ALGEBRAIC_EXPRESSIONS, Topics.HUMAN_BODY_SYSTEMS, Topics.THERMODYNAMICS_IN_PHYSICS, Topics.SOLUTIONS_AND_MIXTURES, }, GradeLevel.EIGHTH_GRADE: { Topics.LINEAR_EQUATIONS, Topics.FUNCTIONS, Topics.EVOLUTION_AND_NATURAL_SELECTION, Topics.WAVES_AND_SOUND, Topics.ATOMIC_AND_NUCLEAR_PHYSICS, }, GradeLevel.NINTH_GRADE: { Topics.QUADRATIC_EQUATIONS, Topics.FUNCTIONS, Topics.GENETICS, Topics.CHEMICAL_BONDING, Topics.MOTION_AND_FORCES, }, GradeLevel.TENTH_GRADE: { Topics.GEOMETRY_THEOREMS, Topics.ACID_BASE_REACTIONS, Topics.ELECTRICITY_AND_MAGNETISM, Topics.PLANT_AND_ANIMAL_CLASSIFICATION, Topics.CODING_SIMPLE_PROGRAMS, }, GradeLevel.ELEVENTH_GRADE: { Topics.ALGEBRAIC_EXPRESSIONS, Topics.CHEMICAL_EQUILIBRIUM, Topics.FLUID_DYNAMICS, Topics.GENETICS_AND_MOLECULAR_BIOLOGY, Topics.DATABASES_ADVANCED, }, GradeLevel.TWELFTH_GRADE: { Topics.CALCULUS_LIMITS, Topics.ECOLOGY_CONSERVATION, Topics.ELECTROCHEMISTRY, Topics.QUANTUM_THEORY_BASICS, Topics.NETWORKING_AND_SECURITY, }, GradeLevel.FRESHMAN: { Topics.DERIVATIVES, Topics.CELL_BIOLOGY, Topics.ORGANIC_CHEMISTRY_BASICS, Topics.GENERAL_PHYSICS_I, Topics.INTRODUCTION_TO_PROGRAMMING, }, GradeLevel.SOPHOMORE: { Topics.INTEGRALS, Topics.ECOLOGY, Topics.ORGANIC_CHEMISTRY, Topics.GENERAL_PHYSICS_II, Topics.DATA_STRUCTURES_COMPLEX, }, GradeLevel.JUNIOR: { Topics.MULTIVARIABLE_CALCULUS, Topics.MICROORGANISMS, Topics.PHYSICAL_CHEMISTRY, Topics.MODERN_PHYSICS, Topics.SOFTWARE_ENGINEERING, }, GradeLevel.SENIOR: { Topics.DIFFERENTIAL_EQUATIONS, Topics.GENETICS_AND_MOLECULAR_BIOLOGY, Topics.INORGANIC_CHEMISTRY, Topics.ADVANCED_PHYSICS_ELECTIVE, Topics.COMPUTER_NETWORKS_AND_SECURITY, }, } def get_starting_knowledge(grade_level: GradeLevel) -> list[str]: # for every grade level below or equal to current grade level # get the starting knowledge for the topic starting_knowledge = [] for gl in GradeLevel: if gl.value <= grade_level.value: try: vals = [x.value for x in STARTING_KNOWLEDGE[gl]] starting_knowledge.extend(vals) except Exception: logging.info(STARTING_KNOWLEDGE[gl]) pass return starting_knowledge class UserCreateRequest(BaseModel): name: str = Field(description="The name of the user") email: str = Field(description="The email of the user") profile_pic_url: str = Field(description="The profile picture of the user") grade_level: GradeLevel = Field(description="The grade level of the user") interests: list[str] = Field(description="The interests of the user") def create_new_user(req_json: dict) -> int: graph_client = client.Client('wss://guidestone-gremlin.gremlin.cosmos.azure.com:443/','g', username=f"/dbs/guidestone/colls/knowledge-graph", password=os.getenv("KNOWLEDGE_GRAPH_KEY"), message_serializer=serializer.GraphSONSerializersV2d0()) postgreSQL_pool = pool.SimpleConnectionPool(1, int(os.getenv("PYTHON_THREADPOOL_THREAD_COUNT")), os.getenv("POSTGRES_CONN_STRING")) conn = postgreSQL_pool.getconn() cursor = conn.cursor() try: create_user_body = UserCreateRequest(**req_json) except Exception as e: logging.error("Could not parse user creation request: " + str(e)) # add user to postgres try: name = create_user_body.name email = create_user_body.email profile_pic_url = create_user_body.profile_p [truncated — 2327 more characters] ``` [4 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]