# Project export: Elevate This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Take a picture of your notes and problem sets and seamlessly convert it to a beautifully-formatted LaTeX document, with feedback and an interactive study chat-bot. - Devpost: https://devpost.com/software/elevate-4s3rez - GitHub: https://github.com/wry0313/treehacks - Demo: https://treehacks-weld.vercel.app/ - Video: https://www.youtube.com/embed/3PxwzryMzo0?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — timothygao8710 (28 commits), Gavin Wang (25 commits), Rudy Arora (5 commits), LAVIK (3 commits) ## Devpost submission (written by the team) ### Inspiration Frustrated by the steep learning curve of LaTeX—a formatting language used by over 90% of students in scientific fields like math or physics—we, as engineering students, found ourselves spending more time converting handwritten notes into LaTeX than solving the actual problems. ### What it does Upload notes or problem sets as images, and Elevate converts them into LaTeX documents while annotating errors with explanations. This process not only simplifies formatting but also enhances learning by helping you identify and correct mistakes for improved academic performance. ### How we built it Convex helped us link our React frontend to the backend smoothly, making it quicker to change things and catch mistakes thanks to its type-safe coding. We organized our data using Convex's NoSQL database, adding structure and making searches faster with indexing. For security, we hooked up authentication easily with Convex's Clerk Adaptor. Since we needed to store images and LaTeX files, Convex was handy there too. Plus, we used Convex’s vector database to smarten up our chatbot, making everything work together nicely and more efficiently. OCR Preprocessing (Image2Latex): We converted images to grayscale via the following formula: We applied Gaussian Blur, which smooths the image by averaging the pixels based on their spatial closeness and intensity similarity, helping to reduce high-frequency noise for next steps. We utilized the OpenCV Library to compute a (5, 5) kernel with standard deviation 0, via the following formula for a two-dimensional Gaussian: Finally, we employed a custom sharpening kernel, which enhances edges by increasing the contrast between adjacent pixels. The convolution operation of the image with the kernel, I * K, is given by After the preprocessing stage, when then feed our resulting image into OpenAI’s Computer Vision API, to generate text describing what is in the image. We then feed this text into together.AI’s Mixtral-8x7B LLM, which is able to generate appropriate latex code for the text. Finally, once again using together.AI’s LLM and extensive prompt engineering and fine-tuning, we were able to highlight errors and generate corrections. Original Image Image After Gaussian Blur (Reduced Noise) Image After Sharpening Kernel Convolution Latex Code Generated by Together.AI Latex Code Corrected by Together.AI Database Vector Search: Using Intersystem’s Langchain-IRIS vector search technology, we were able to: Generate embedded vectors for latex notes Use cosine similarity to query, for a certain problem, the most relevant notes that contain information to answer this problem By representing each note as an edge and drawing an edge between nodes with high similarity (low vector degree distance), create a connection graph and mind map of all notes in the database For example, querying the question “How can I find a stable matching?” in a database of one of our team member's discrete math course notes results in the following responses from Intersystem’s API: Leveraging Together.AI’s Mixtral LLM, we also created an interactive ChatBot that is able to summarize ideas in the notes and answer questions from users about the notes. ### Challenges we ran into It took a while to figure out how to render LaTeX text in a browser, as no existing packages met our needs for clarity and precision in displaying complex formulas. Creating LLM prompts for annotating student solutions with feedback proved difficult. We had to not only ensure that the feedback was accurate but also that it was formatted and presented properly. The technical complexity of implementing the vector graph for note comparison presented a hard challenge. Perfecting an algorithm that seemlessly integrate Intersystem's Langchain vector embeddings into a graph structure, and generating appropriate adjacency lists required deep dives into ML, optimization, and Data Structures. Additionally, rendering the graph on our live website presented an additional challenge. ### Accomplishments we're proud of We successfully built a tool that we will use for the rest of our lives and are optimistic enough about the quality/market potential to go to market after the Hackathon. ### What we learned We learned the crucial importance of having a diverse team with varied skill sets, enabling us to effectively divide the workload across design, backend development, and machine learning tasks. ### What's next Get 1k users by posting on social media. Use feedback from users to iterate product and add features they would find helpful. ## README (from the GitHub repository) # Elevate ## Inspiration Every science/math student in college spends more time "TeXing up" their problem sets than actually solving the problems. The simple truth is that formatting problem sets is a huge time-suck. ## What it does Upload image(s) of your notes/problem sets and watch Elevate beautifully format them into a LaTeX document. An additional created document displays feedback to the work. ## How we built it Frontend with React. Backend with Convex. GPT-4 with Vision processes the image and generates LaTeX script. GPT-4 then processes that LaTeX and adds feedback within the LaTeX script. Finally, we load that file as the context for a Mistral-chatbot that can provide insights from the notes/problem set. ## Challenges we ran into Using LLMs to format and position feedback within the LaTeX scripts. ## Accomplishments that we're proud of ## What we learned ## What's next for Elevate Looking to expand into other realms of the EdTech space! ## Detected evidence (automated analysis) Indexed codebase: 36 recognized source files, 79 KB. - CSS (language) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - LangChain (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (58 of 58) ``` .eslintrc.cjs .gitignore bun.lockb chatbot.py convex/_generated/api.d.ts convex/_generated/api.js convex/_generated/dataModel.d.ts convex/_generated/server.d.ts convex/_generated/server.js convex/auth.config.js convex/http.ts convex/noteImages.ts convex/noteLatexPdf.ts convex/notes.ts convex/README.md convex/schema.ts convex/tasks.ts convex/tsconfig.json flask/.gitignore flask/app.py flask/requirements.txt flask/tmp13ope08e.aux flask/tmpkvr7v5vf.aux flask/tmpn1gwiqt1.aux flask/tmpp2zo41vq.aux graph.txt image_to_latex.py index.html package.json postcss.config.js README.md sampleData.jsonl src/components/ChatBot.tsx src/components/config.ts src/components/Graph.tsx src/components/UploadNotesImage.tsx src/index.css src/main.tsx src/pages/_layout.tsx src/pages/graph/index.tsx src/pages/index.tsx src/pages/note/[id].tsx src/pages/upload/index.tsx src/router.ts src/vite-env.d.ts tailwind.config.js test_demo.py tmp5vm7078v.aux tmpa36rv3wf.aux tmpzbv6vud8.aux tsconfig.json tsconfig.node.json Vector Embedding/CS70Notes.txt Vector Embedding/intersystem_vector_search_demo.ipynb Vector Embedding/intersystem_vector_search_large_graph.ipynb Vector Embedding/intersystem_vector_search.ipynb vite.config.ts word_embeddings.py ``` ### Dependencies - flask/requirements.txt: annotated-types@==0.6.0, anyio@==4.2.0, blinker@==1.7.0, certifi@==2024.2.2, charset-normalizer@==3.3.2, click@==8.1.7, distro@==1.9.0, Flask@==3.0.2, Flask-Cors@==4.0.0, h11@==0.14.0, httpcore@==1.0.3, httpx@==0.26.0, idna@==3.6, itsdangerous@==2.1.2, Jinja2@==3.1.3, MarkupSafe@==2.1.5, numpy@==1.26.4, openai@==1.12.0, opencv-python@==4.9.0.80, pydantic@==2.6.1, pydantic_core@==2.16.2, python-dotenv@==1.0.1, requests@==2.31.0, sniffio@==1.3.0, tqdm@==4.66.2, typing_extensions@==4.9.0, urllib3@==2.2.1, Werkzeug@==3.0.1 - package.json: @clerk/clerk-react@^4.30.5, @generouted/react-router@^1.18.2, @types/react@^18.2.55, @types/react-dom@^18.2.19, @typescript-eslint/eslint-plugin@^6.21.0, @typescript-eslint/parser@^6.21.0, @vitejs/plugin-react@^4.2.1, autoprefixer@^10.4.17, convex@^1.9.0, daisyui@latest, eslint@^8.56.0, eslint-plugin-react-hooks@^4.6.0, eslint-plugin-react-refresh@^0.4.5, install@^0.13.0, npm@^10.4.0, openai@^4.28.0, postcss@^8.4.35, react@^18.2.0, react-dom@^18.2.0, react-force-graph@^1.44.3, react-hot-toast@^2.4.1, react-query@^3.39.3, react-router-dom@^6.22.1, react-spinners@^0.13.8, tailwindcss@^3.4.1, typescript@^5.2.2, vite@^5.1.0 ### Recent commits (newest first) - adfad - changes - changes - changes - ghb,jn - graph - mvp graph - adsf - merge - nice - Merge branch 'master' of https://github.com/wry0313/treehacks - asdf - convex nice - changes - chatbot works - knjl - Merge branch 'master' of https://github.com/wry0313/treehacks - made bot pretty - readme - merged ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "frontend", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc && vite build", "lint": "eslint . --ext ts,tsx --report-unused-disable-directives --max-warnings 0", "preview": "vite preview" }, "dependencies": { "@clerk/clerk-react": "^4.30.5", "@generouted/react-router": "^1.18.2", "convex": "^1.9.0", "daisyui": "latest", "install": "^0.13.0", "npm": "^10.4.0", "openai": "^4.28.0", "react": "^18.2.0", "react-dom": "^18.2.0", "react-force-graph": "^1.44.3", "react-hot-toast": "^2.4.1", "react-query": "^3.39.3", "react-router-dom": "^6.22.1", "react-spinners": "^0.13.8" }, "devDependencies": { "@types/react": "^18.2.55", "@types/react-dom": "^18.2.19", "@typescript-eslint/eslint-plugin": "^6.21.0", "@typescript-eslint/parser": "^6.21.0", "@vitejs/plugin-react": "^4.2.1", "autoprefixer": "^10.4.17", "eslint": "^8.56.0", "eslint-plugin-react-hooks": "^4.6.0", "eslint-plugin-react-refresh": "^0.4.5", "postcss": "^8.4.35", "tailwindcss": "^3.4.1", "typescript": "^5.2.2", "vite": "^5.1.0" } } ``` ### flask/requirements.txt ``` annotated-types==0.6.0 anyio==4.2.0 blinker==1.7.0 certifi==2024.2.2 charset-normalizer==3.3.2 click==8.1.7 distro==1.9.0 Flask==3.0.2 Flask-Cors==4.0.0 h11==0.14.0 httpcore==1.0.3 httpx==0.26.0 idna==3.6 itsdangerous==2.1.2 Jinja2==3.1.3 MarkupSafe==2.1.5 numpy==1.26.4 openai==1.12.0 opencv-python==4.9.0.80 pydantic==2.6.1 pydantic_core==2.16.2 python-dotenv==1.0.1 requests==2.31.0 sniffio==1.3.0 tqdm==4.66.2 typing_extensions==4.9.0 urllib3==2.2.1 Werkzeug==3.0.1 ``` ### src/main.tsx ```typescript // src/main.tsx import { createRoot } from "react-dom/client"; import { Routes } from "@generouted/react-router"; import "./index.css"; import { ClerkProvider, useAuth } from "@clerk/clerk-react"; import { ConvexProviderWithClerk } from "convex/react-clerk"; import { ConvexReactClient } from "convex/react"; import { Toaster } from "react-hot-toast"; const convex = new ConvexReactClient(import.meta.env.VITE_CONVEX_URL as string); createRoot(document.getElementById("root")!).render( ); ``` ### flask/app.py ```python from flask import Flask, jsonify, request from flask_cors import CORS app = Flask(__name__) CORS(app) # This enables CORS for all routes and origins @app.route('/') def hello_world(): return 'Hello, World!' if __name__ == '__main__': app.run(debug=True) from urllib.parse import quote, unquote @app.route('/image_to_latex', methods=['POST']) def image_to_latex(): # Get the image url from the request boyd body = request.json image_url = body['image_url'] noteId = body['noteId'] noteImageId = body['noteImageId'] print(noteId) print(image_url) image_path = './tmp/image.jpg' os.makedirs(os.path.dirname(image_path), exist_ok=True) # Download the image try: response = requests.get(image_url) response.raise_for_status() # Check if the request was successful with open(image_path, 'wb') as f: f.write(response.content) print("Image downloaded") # Add your logic for processing the image to LaTeX here # return jsonify({'message': 'Image processed successfully'}), 200 except requests.RequestException as e: print(e) return jsonify({'error': 'Failed to download the image'}), 500 # Convert the image to LaTeX latext = ImageToLatex(image_path) print(latext) remove_first_page_in_place("feedback.pdf") with open("feedback.pdf", "rb") as f: pdf_bytes = f.read() # No need for base64 encoding header = { "Content-Type": "application/pdf" } # Send the raw PDF bytes # santize the latex for the url def sanitize_for_url(input_string): return quote(input_string) latext = sanitize_for_url(latext) response = requests.post("https://astute-cheetah-548.convex.site/uploadPdf?noteId=" + noteId + "&latextString=" + latext + "¬eImageId=" + noteImageId , headers=header, data=pdf_bytes) print(response) return jsonify({"latex": latext, "feedback_pdf": base64.b64encode(pdf_bytes).decode('utf-8')}), 200 # Return the LaTeX import subprocess from openai import OpenAI import os import requests from dotenv import load_dotenv import base64 import cv2 import numpy as np from tempfile import NamedTemporaryFile from openai import OpenAI # from pdf2image import convert_from_path from convex import ConvexClient load_dotenv(".env.local") client = ConvexClient(os.getenv("CONVEX_URL")) # print(client.query("tasks:get")) load_dotenv() TOGETHER_API_KEY = os.getenv("TOGETHER_API_KEY") OPENAI_API_KEY = os.getenv("OPENAI_API_KEY") print(OPENAI_API_KEY) def preprocess_image_for_ocr(image_path): image = cv2.imread(image_path) # Convert the image to grayscale gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) # Apply Gaussian Blur to reduce noise blur = cv2.GaussianBlur(gray, (5, 5), 0) # Apply a sharpening kernel sharpen_kernel = np.array([[-1, -1, -1], [-1, 9, -1], [-1, -1, -1]]) sharpened = cv2.filter2D(blur, -1, sharpen_kernel) # Optionally, apply edge detection (e.g., Canny) to further enhance edges for OCR edges = cv2.Canny(sharpened, 100, 200) # Save or return the processed image processed_image_path = 'processed_image.jpg' cv2.imwrite(processed_image_path, edges) return processed_image_path def image_to_text(image_path): # Function to encode the image def encode_image(image_path): with open(image_path, "rb") as image_file: return base64.b64encode(image_file.read()).decode('utf-8') # Getting the base64 string base64_image = encode_image(image_path) headers = { "Content-Type": "application/json", "Authorization": f"Bearer {OPENAI_API_KEY}" } payload = { "model": "gpt-4-vision-preview", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "This is an image of a student's notes. Transcribe it into latex code. Have a one-line gap between each line. This should compile as a pdf file later. Do not output anything except for the transcribed latex! I will tip you $100 if you make the latex concise." }, { "type": "image_url", "image_url": { "url": f"data:image/jpeg;base64,{base64_image}" } } ] } ], "max_tokens": 1024 } response = requests.post("https://api.openai.com/v1/chat/completions", headers=headers, json=payload) res = response.json() print(res) return res['choices'][0]['message']['content'] def latex_to_pdf(latex_code, output_dir='./', filename='output.pdf'): # Ensure the output directory exists if not os.path.exists(output_dir): os.makedirs(output_dir) # Full path for the output PDF pdf_output_path = os.path.join(output_dir, filename) # Create a temporary .tex file with NamedTemporaryFile(suffix=".tex", delete=False) as temp_tex_file: temp_tex_path = temp_tex_file.name # Write the LaTeX code to the temporary file temp_tex_file.write(latex_code.encode('utf-8')) temp_tex_file.flush() # Compile the LaTeX file into a PDF using pdflatex subprocess.run(["pdflatex", "-interaction=nonstopmode", "-output-directory", output_dir, temp_tex_path], cwd=os.path.dirname(temp_tex_path)) # Check if the PDF was successfully created at the specified output path final_pdf_path = os.path.join(output_dir, os.path.basename(temp_tex_path).replace(".tex", ".pdf")) if os.path.exists(final_pdf_path): # If a specific filename is provided, rename the generated PDF accordingly if filename: custom_pdf_path = pdf_output_path os.rename(final_pdf_ [truncated — 3723 more characters] ``` ### convex/_generated/server.js ```javascript /* eslint-disable */ /** * Generated utilities for implementing server-side Convex query and mutation functions. * * THIS CODE IS AUTOMATICALLY GENERATED. * * Generated by convex@1.9.0. * To regenerate, run `npx convex dev`. * @module */ import { actionGeneric, httpActionGeneric, queryGeneric, mutationGeneric, internalActionGeneric, internalMutationGeneric, internalQueryGeneric, } from "convex/server"; /** * Define a query in this Convex app's public API. * * This function will be allowed to read your Convex database and will be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const query = queryGeneric; /** * Define a query that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to read from your Convex database. It will not be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const internalQuery = internalQueryGeneric; /** * Define a mutation in this Convex app's public API. * * This function will be allowed to modify your Convex database and will be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const mutation = mutationGeneric; /** * Define a mutation that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to modify your Convex database. It will not be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const internalMutation = internalMutationGeneric; /** * Define an action in this Convex app's public API. * * An action is a function which can execute any JavaScript code, including non-deterministic * code and code with side-effects, like calling third-party services. * They can be run in Convex's JavaScript environment or in Node.js using the "use node" directive. * They can interact with the database indirectly by calling queries and mutations using the {@link ActionCtx}. * * @param func - The action. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped action. Include this as an `export` to name it and make it accessible. */ export const action = actionGeneric; /** * Define an action that is only accessible from other Convex functions (but not from the client). * * @param func - The function. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped function. Include this as an `export` to name it and make it accessible. */ export const internalAction = internalActionGeneric; /** * Define a Convex HTTP action. * * @param func - The function. It receives an {@link ActionCtx} as its first argument, and a `Request` object * as its second. * @returns The wrapped endpoint function. Route a URL path to this function in `convex/http.js`. */ export const httpAction = httpActionGeneric; ``` ### src/pages/index.tsx ```typescript /* eslint-disable @typescript-eslint/no-unused-vars */ import { SignInButton, SignOutButton } from "@clerk/clerk-react"; import { useConvexAuth, useMutation, useQuery } from "convex/react"; import { useUser } from "@clerk/clerk-react"; import { api } from "../../convex/_generated/api"; import { useNavigate } from "react-router-dom"; import toast from "react-hot-toast"; import { Id } from "../../convex/_generated/dataModel"; import logo from "../assets/logo.png"; export default function HomePage() { const { isAuthenticated } = useConvexAuth(); const { user } = useUser(); console.log(user); const userNotes = useQuery(api.notes.getNotesByUserId, { userId: user?.id ?? "", }); console.log(userNotes); return (
{/* Navbar */}
logo Elevate Notebooks
    {/* Navbar menu content here */}
  • Graph
  • {isAuthenticated ? (
    ) : (
    )}
{/* Page content here */}
{isAuthenticated ? (

Welcome {user?.fullName}

) : (

Welcome to the home page

)}
{userNotes?.map(({ _id, title, createdAt }) => ( ))}
); } const NoteCard = ({ id, title, date, isAdd = false, }: { id: string; title: string; date: string; isAdd: boolean; }) => { const navigate = useNavigate(); const getThumbNail = useQuery(api.noteImages.getOneImageByOneNoteId, { noteId: id as Id<"notes">, }); return ( ); }; const AddNoteCard = () => { const insertNewNotes = useMutation(api.notes.insertNewNotes); const { user } = useUser(); const { isAuthenticated } = useConvexAuth(); const navigate = useNavigate(); return ( ); } ``` ### postcss.config.js ```javascript export default { plugins: { tailwindcss: {}, autoprefixer: {}, }, } ``` ### vite.config.ts ```typescript // vite.config.ts import { defineConfig } from 'vite' import react from '@vitejs/plugin-react' import generouted from '@generouted/react-router/plugin' export default defineConfig({ plugins: [react(), generouted()] }) ``` [25 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]