# Project export: \LeTEXT This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: From math wizards, to CS freshman, LaTeX is loved and hated by many. With the power of Fine-Tuned LLMs, OCR and integrated development tools, we make LaTeX accessible and intuitive for everyone! - Devpost: https://devpost.com/software/letext - GitHub: https://github.com/derekmil/treehacks-2024 - Demo: https://www.loom.com/share/423e53c2ebdd4262ad1a72b372e02dd2?sid=80d13510-857a-4a89-8869-e3403bc40ac6 - Team: 4 GitHub contributor(s) — Derek (24 commits), Shane Liu (22 commits), William de Voest (6 commits), Michael Kim (5 commits) ## Devpost submission (written by the team) ### Inspiration As CS Students, math enthusiasts and amateur researches, we know the struggle and the beauty of watching code become artistic, organized writing. Yet, it is so difficult to become comfortable without extensive experience. We hope to enable anyone to quickly form perfect documents, quickly and intuitively. ### What it does The crux of our application is a Fine-Tuned LLM Copilot, OCR for handwritten math, and custom commands integrated with a blazingly fast native LaTeX IDE and Renderer in browser. We have built in commands for common things like logical-and, logical-or, and even Demorgans Law so that your paper or proof shouldn’t be hindered by a curly bracket. The main part of our product is an AI model using Together.ai with CodeLlama-2 trained on a large data set of 450 common prompts in Discrete Math and research papers. A prompt like “create a bullet point for my education section on my resume at the Stanford” would spit a perfectly formatted block of latex in our proprietary IDE. Or “perform DeMorgans law on Set A and Set B”. This is chips and guac for us. Have messy handwritten math? No problem at all! ### How we built it The backend was built using a TexLive renderer in a Docker container that is run to consistently render the Tex Document continuously so you can see your changes “live”. The frontend was built with Typescript + React with Vite, and the backend with FastAPI, Firebase Auth for authentication, and Postgres for database storage so that users can save to the cloud in the future. This was used in tandem with Monaco Editor from Microsoft for all the logic we embedded to deal with the autocomplete and code snippet replacement commands. For the ML Model we trained it using Together.ai for a fine-tuned CodeLlama-2. For OCR, we leveraged MathPix APIs. ### Challenges we ran into We found no shortage of challenges. Our biggest issue was giving the ML model permission to write into the IDE as the package configurations coupled with the extensive setup for the Tex Renderer to take this input. We decided to approach it using custom commands (such as /ai) to give the user command over when they want to get help. It was quite the challenge. Of course, building the container and integrating it with the backend was no small feat either. We put these different puzzle pieces together and realized there was more color to the bigger picture than we anticipated. ### Accomplishments we're proud of We are most proud of creating an integrated IDE. None of us have experience making developer tools so we were very proud with how we learned the tech to power our application. The key thing is in Monaco Editor Latex was not a supported language compatible with the IDE. As such we had to bruteforce produce mappings of the key words like \textbf{} and link smart highlighting with the intellisense that runs Monaco Editor to get a fully functioning IDE tailored to our use cases. As an extension we are especially proud of our Latex CoPilot. Utilizing Together to integrate a smart AI code generator that intakes English semantic was an enriching experience to accomplish, especially building the model from the ground up and linking it to the IDE. ### What we learned We see ourselves enhancing the ML model to implement more tailored Tex code and even make suggestions based on what the user wants. Furthermore, we see ourselves implementing our own syntactical language that overlaps Tex that makes it more user friendly maintaining a consistent lexical structure throughout, as to interpreting English input and immediately translating it into Tex. Finally, Nonetheless, we enjoyed this project and were able to see the fruit of our work taking our old Discrete Math work and research papers being written in this app of ours. ### What's next We see ourselves enhancing the ML model to implement more tailored Tex code and even make suggestions based on what the user wants. Furthermore, we see ourselves implementing our own syntactical language that overlaps Tex that makes it more user friendly maintaining a consistent lexical structure throughout, as to interpreting English input and immediately translating it into Tex. We also intend to add cloud storage for users. Nonetheless, we enjoyed this project and were able to see the fruit of our work taking our old Discrete Math work and research papers being written in this app of ours. ## README (from the GitHub repository) # Latex AI Dev Guide ## FastAPI Server Setup FastAPI is a modern, fast web framework for building APIs with Python 3.7+ based on standard Python type hints. ### Steps 1. **Clone the Repository** Ensure you have the project repository cloned to your local machine. 2. **Setup Poetry Environment** Install [Poetry](https://python-poetry.org/) with `pipx install poetry` Naviagate to `backend` and run `poetry install` and then `poetry shell` 3. **Run Server** Now, in the root dir, run `uvicorn backend.main:app --reload` ## Vite Server Setup Vite is a modern frontend build tool that provides a faster and leaner development experience. ### Steps 1. **Navigate…** …to `client`. 2. **Install npm** Install `npm` with `npm install` 3. **Run Server** Now, run `npm run dev` ## Detected evidence (automated analysis) Indexed codebase: 41 recognized source files, 88 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Firebase (technology) — claimed on Devpost, not found in the code - PostgreSQL (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (53 of 53) ``` .DS_Store backend/__init__.py backend/.gitignore backend/api_models.py backend/main.py backend/poetry.lock backend/pyproject.toml backend/search.py backend/sql/database.py backend/sql/models.py client/.eslintrc.cjs client/.gitignore client/components.json client/index.html client/package.json client/postcss.config.js client/public/pdf.worker.min.mjs client/README.md client/src/App.tsx client/src/assets/commands.json client/src/assets/data.js client/src/components/Box.tsx client/src/components/customToolTip.tsx client/src/components/defaultText.tsx client/src/components/mode-toggle.tsx client/src/components/monaco-editor.tsx client/src/components/navbar.tsx client/src/components/pdf.tsx client/src/components/ScrollBaby.tsx client/src/components/theme-provider.tsx client/src/components/ui/button.tsx client/src/components/ui/card.tsx client/src/components/ui/command.tsx client/src/components/ui/dialog.tsx client/src/components/ui/dropdown-menu.tsx client/src/components/ui/hover-card.tsx client/src/components/ui/navigation-menu.tsx client/src/components/ui/scroll-area.tsx client/src/components/ui/separator.tsx client/src/components/ui/skeleton.tsx client/src/components/ui/tooltip.tsx client/src/Docs.tsx client/src/index.css client/src/lib/latex/latex.ts client/src/lib/latex/setup.ts client/src/lib/utils.ts client/src/main.tsx client/src/vite-env.d.ts client/tailwind.config.js client/tsconfig.json client/tsconfig.node.json client/vite.config.ts README.md ``` ### Dependencies - backend/pyproject.toml: fastapi@^0.109.2, firebase-admin@^6.4.0, psycopg2-binary@^2.9.9, pyrebase4@^4.7.1, python-dotenv@^1.0.1, python-multipart@^0.0.9, setuptools@^69.1.0, sqlalchemy@^2.0.27, uvicorn@^0.27.1 - client/package.json: @monaco-editor/react@^4.6.0, @radix-ui/react-dialog@^1.0.5, @radix-ui/react-dropdown-menu@^2.0.6, @radix-ui/react-hover-card@^1.0.7, @radix-ui/react-icons@^1.3.0, @radix-ui/react-navigation-menu@^1.1.4, @radix-ui/react-scroll-area@^1.0.5, @radix-ui/react-separator@^1.0.3, @radix-ui/react-slot@^1.0.2, @radix-ui/react-tooltip@^1.0.7, @react-pdf/renderer@^3.3.8, @tanstack/react-query@^5.21.4, @types/lodash@^4.14.202, @types/node@^20.11.19, @types/react@^18.2.55, @types/react-dom@^18.2.19, @typescript-eslint/eslint-plugin@^6.21.0, @typescript-eslint/parser@^6.21.0, @vitejs/plugin-react@^4.2.1, autoprefixer@^10.4.17, class-variance-authority@^0.7.0, clsx@^2.1.0, cmdk@^0.2.1, eslint@^8.56.0, eslint-plugin-react-hooks@^4.6.0, eslint-plugin-react-refresh@^0.4.5, localforage@^1.10.0, lodash@^4.17.21, lucide-react@^0.331.0, match-sorter@^6.3.4, monaco-editor-core@^0.46.0, pdfjs-dist@^4.0.379, postcss@^8.4.35, react@^18.2.0, react-dom@^18.2.0, react-pdf@^7.7.0, react-router-dom@^6.22.1, sort-by@^1.2.0, tailwind-merge@^2.2.1, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5.2.2, vite@^5.1.0 ### Recent commits (newest first) - h2h - holy fuck that ui - merged - pls - Merge pull request #9 from derekmil/william - Merge branch 'main' into william - fixing my mistakes - Merge pull request #7 from derekmil/shane - Merge branch 'main' into shane - style - Merge branch 'main' into shane - Merge branch 'main' of github.com:derekmil/treehacks-2024 - moreee - Merge branch 'shane' of https://github.com/derekmil/treehacks-2024 into shane - Merge branch 'main' into shane - Merge pull request #8 from derekmil/william - Merge branch 'main' into william - monaco dynamic change - pdf rendering - Merge branch 'main' into shane ## Key source files (fetched from GitHub, selected and truncated for size) ### backend/pyproject.toml ``` [tool.poetry] name = "treehacks-2024" version = "0.1.0" description = "" authors = ["Your Name "] readme = "README.md" [tool.poetry.dependencies] python = "^3.12" fastapi = "^0.109.2" python-dotenv = "^1.0.1" firebase-admin = "^6.4.0" pyrebase4 = "^4.7.1" sqlalchemy = "^2.0.27" psycopg2-binary = "^2.9.9" uvicorn = "^0.27.1" setuptools = "^69.1.0" python-multipart = "^0.0.9" [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api" ``` ### client/package.json ``` { "name": "client", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc && vite build", "lint": "eslint . --ext ts,tsx --report-unused-disable-directives --max-warnings 0", "preview": "vite preview" }, "dependencies": { "@monaco-editor/react": "^4.6.0", "@radix-ui/react-dialog": "^1.0.5", "@radix-ui/react-dropdown-menu": "^2.0.6", "@radix-ui/react-hover-card": "^1.0.7", "@radix-ui/react-icons": "^1.3.0", "@radix-ui/react-navigation-menu": "^1.1.4", "@radix-ui/react-scroll-area": "^1.0.5", "@radix-ui/react-separator": "^1.0.3", "@radix-ui/react-slot": "^1.0.2", "@radix-ui/react-tooltip": "^1.0.7", "@react-pdf/renderer": "^3.3.8", "@tanstack/react-query": "^5.21.4", "class-variance-authority": "^0.7.0", "clsx": "^2.1.0", "cmdk": "^0.2.1", "localforage": "^1.10.0", "lodash": "^4.17.21", "lucide-react": "^0.331.0", "match-sorter": "^6.3.4", "monaco-editor-core": "^0.46.0", "pdfjs-dist": "^4.0.379", "react": "^18.2.0", "react-dom": "^18.2.0", "react-pdf": "^7.7.0", "react-router-dom": "^6.22.1", "sort-by": "^1.2.0", "tailwind-merge": "^2.2.1", "tailwindcss-animate": "^1.0.7" }, "devDependencies": { "@types/lodash": "^4.14.202", "@types/node": "^20.11.19", "@types/react": "^18.2.55", "@types/react-dom": "^18.2.19", "@typescript-eslint/eslint-plugin": "^6.21.0", "@typescript-eslint/parser": "^6.21.0", "@vitejs/plugin-react": "^4.2.1", "autoprefixer": "^10.4.17", "eslint": "^8.56.0", "eslint-plugin-react-hooks": "^4.6.0", "eslint-plugin-react-refresh": "^0.4.5", "postcss": "^8.4.35", "tailwindcss": "^3.4.1", "typescript": "^5.2.2", "vite": "^5.1.0" } } ``` ### backend/main.py ```python from fastapi import FastAPI, HTTPException, Depends, Response, File, UploadFile, Body from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import JSONResponse, FileResponse from fastapi.exceptions import HTTPException from fastapi.requests import Request import backend.sql.models as models from backend.sql.database import SessionLocal, engine from sqlalchemy.orm import Session from pydantic import BaseModel from typing import Annotated from backend.sql.database import get_db import firebase_admin from firebase_admin import credentials, auth from dotenv import load_dotenv import os import pyrebase from backend.api_models import SignUpSchema, LoginSchema, EmailSchema, LatexDocSchema from sqlalchemy.exc import IntegrityError import subprocess import secrets import string load_dotenv(dotenv_path='.env') app = FastAPI() models.Base.metadata.create_all(bind=engine) db_dependency = Annotated[Session, Depends(get_db)] origins = [ "http://localhost:5173", ] app.add_middleware( CORSMiddleware, allow_origins=['*'], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) if not firebase_admin._apps: cred = credentials.Certificate(os.getenv("SERVICE_ACCOUNT_KEY")) firebase_admin.initialize_app(cred) FIREBASE_CONFIG = { "apiKey": os.getenv('FIREBASE_APIKEY'), "authDomain": os.getenv('FIREBASE_AUTHDOMAIN'), "projectId": os.getenv('FIREBASE_PROJECTID'), "storageBucket": os.getenv('FIREBASE_STORAGEBUCKET'), "messagingSenderId": os.getenv('FIREBASE_MESSAGINGSENDERID'), "appId": os.getenv('FIREBASE_APPID'), "databaseURL": os.getenv('FIREBASE_DATABASEURL') } firebase = pyrebase.initialize_app(FIREBASE_CONFIG) ####################################################### # # # FUNCS # # # ####################################################### def generate_random_string(length=8): characters = string.ascii_letters + string.digits return ''.join(secrets.choice(characters) for _ in range(length)) ####################################################### # # # ROUTES # # # ####################################################### @app.post("/api/render-latex/") async def render_latex(request: LatexDocSchema): latex = request.latex unsanitized_latex_content = latex.replace('\\\\', '\\') tag = generate_random_string() root_dir = "/Users/derekmiller/Documents/sideproj/treehacks-2024" render_dir = f"backend/render_dir" os.system(f"rm {render_dir}/*") os.makedirs(f"{root_dir}/{render_dir}", exist_ok=True) file_path = os.path.join(f"{root_dir}/{render_dir}", f"{tag}.tex") with open(file_path, "w") as file_object: file_object.write(unsanitized_latex_content) try: compile_command = [ "docker", "run", "--rm", "-v", f"{root_dir}/{render_dir}:/workdir", "texlive/texlive", "pdflatex", f"{tag}.tex" ] result = subprocess.run(compile_command, cwd=f"{root_dir}/{render_dir}/", capture_output=True, text=True) if result.returncode != 0: raise HTTPException(status_code=501, detail=result.stderr) except Exception as e: print("something", e) raise HTTPException(status_code=500, detail=str(e)) response = FileResponse(f"{root_dir}/{render_dir}/{tag}.pdf") return response @app.post('/createaccount/') async def create_account(udata: SignUpSchema, db: db_dependency): try: if db.query(models.Users).filter(models.Users.email == udata.email).first() is not None: raise IntegrityError user = auth.create_user( email=udata.email, password=udata.password, ) db_user = models.Users( email=udata.email, uid=user.uid, first_name=udata.first_name, last_name=udata.last_name, date_of_birth=udata.date_of_birth, gender=udata.gender ) db.add(db_user) db.commit() return JSONResponse( content={"message" : f"User account created successfuly for user {user.uid}"}, status_code=201 ) except (auth.EmailAlreadyExistsError, IntegrityError): raise HTTPException( status_code=400, detail= f"Account already created for the email {udata.email}" ) @app.post('/login') async def login(user_data: LoginSchema, response:Response): email = user_data.email password = user_data.password try: user = firebase.auth().sign_in_with_email_and_password( email = email, password = password ) token = user['idToken'] print(token) response.set_cookie(key="jwt", value=token, httponly=True, secure=True, samesite='Lax') return JSONResponse( content={ "token":token },status_code=200 ) except: raise HTTPException( status_code=400,detail="Invalid Credentials" ) @app.post('/ping') async def validate_token(request:Request): headers = request.headers jwt = headers.get('authorization') user = auth.verify_id_token(jwt, check_revoked=True) return user["user_id"] @app.post('/checkemail') async def check_email(email_data:EmailSchema, db: db_dependency): if db.query(models.Users).filter(models.Users.email == email_data.email).first() is not None: return JSONResponse( content={ "message":"Email already exists", "exists":True },status_code=200 ) else: return JSONResponse( content={ [truncated — 108 more characters] ``` ### client/src/main.tsx ```typescript import React from 'react' import ReactDOM from 'react-dom/client' import App from './App.tsx' import Docs from './Docs.tsx' import './index.css' import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; const queryClient = new QueryClient({}); import { ThemeProvider } from "@/components/theme-provider"; import { createBrowserRouter, RouterProvider, } from "react-router-dom"; const router = createBrowserRouter([ { path: "/", element: , }, { path: "/docs", element: , }, ]); ReactDOM.createRoot(document.getElementById('root')!).render( , ) ``` ### client/src/App.tsx ```typescript "use client"; import { useEffect, useState } from "react"; import { ModeToggle } from "@/components/mode-toggle"; import MonacoEditor from "@/components/monaco-editor"; import { useQuery } from "@tanstack/react-query"; import PDFViewer from "@/components/pdf"; import {latexTemplate} from '@/components/defaultText'; import { Skeleton } from "@/components/ui/skeleton" import {NavBar} from "@/components/navbar"; import { ScrollBaby } from "@/components/ScrollBaby"; import { useMonaco } from "@monaco-editor/react"; const value = /* set from `myEditor.getModel()`: */ `function hello() { alert('Hello world!'); }`; export default function Home() { // const [pdfPath, setPdfPath] = useState(""); const monaco = useMonaco(); //const [editorText, setEditorText] = useState(''); const [editorText, setEditorText] = useState(""); const { isLoading, data } = useQuery({ queryKey: ['retrieve-latex', editorText], queryFn: async () => { const latexContent = editorText.replace(/\\/g, '\\\\'); const response = await fetch('http://localhost:8000/api/render-latex/', { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify({ latex: latexContent }), }); let data = await response.blob(); const url = URL.createObjectURL(data); return url; }, enabled: !!editorText, // This ensures the query doesn't run until editorText is not empty }); useEffect(() => { const keyword = "/ai"; if (editorText.includes(keyword)) { console.log("found keyword"); // Get the line count const lines = editorText.split("\n"); const lineIndex = lines.findIndex(line => line.includes(keyword)); // Get the content of the last line if (lineIndex !== -1) { // Check if keyword is found // Get the content of the line const lineContent = lines[lineIndex]; console.log("Line containing keyword:", lineContent); const newEditorText = editorText.replace(keyword, "Submitted"); console.log("Editor text:", newEditorText); setEditorText(newEditorText); var currentEditor = monaco?.editor.getModels()[0]; if (currentEditor) { const editOperation = { range: currentEditor.getFullModelRange(), text: newEditorText, forceMoveMarkers: true, }; currentEditor.applyEdits([editOperation]); } } } }, [editorText]); return (
{/* */}
{" "}
{!isLoading ? : }
); } ``` ### client/postcss.config.js ```javascript export default { plugins: { tailwindcss: {}, autoprefixer: {}, }, } ``` ### client/vite.config.ts ```typescript import path from "path" import react from "@vitejs/plugin-react" import { defineConfig } from "vite" export default defineConfig({ plugins: [react()], resolve: { alias: { "@": path.resolve(__dirname, "./src"), }, }, }) ``` ### client/index.html ```html Vite + React + TS
``` ### backend/api_models.py ```python from pydantic import BaseModel from datetime import date class SignUpSchema(BaseModel): email:str first_name:str last_name:str password:str date_of_birth:date gender:str class LoginSchema(BaseModel): email:str password:str class EmailSchema(BaseModel): email:str class LatexDocSchema(BaseModel): latex: str class SearchSchema(BaseModel): query:str ``` ### client/tailwind.config.js ```javascript /** @type {import('tailwindcss').Config} */ module.exports = { darkMode: ["class"], content: [ './pages/**/*.{ts,tsx}', './components/**/*.{ts,tsx}', './app/**/*.{ts,tsx}', './src/**/*.{ts,tsx}', ], prefix: "", theme: { container: { center: true, padding: "2rem", screens: { "2xl": "1400px", }, }, extend: { colors: { border: "hsl(var(--border))", input: "hsl(var(--input))", ring: "hsl(var(--ring))", background: "hsl(var(--background))", foreground: "hsl(var(--foreground))", primary: { DEFAULT: "hsl(var(--primary))", foreground: "hsl(var(--primary-foreground))", }, secondary: { DEFAULT: "hsl(var(--secondary))", foreground: "hsl(var(--secondary-foreground))", }, destructive: { DEFAULT: "hsl(var(--destructive))", foreground: "hsl(var(--destructive-foreground))", }, muted: { DEFAULT: "hsl(var(--muted))", foreground: "hsl(var(--muted-foreground))", }, accent: { DEFAULT: "hsl(var(--accent))", foreground: "hsl(var(--accent-foreground))", }, popover: { DEFAULT: "hsl(var(--popover))", foreground: "hsl(var(--popover-foreground))", }, card: { DEFAULT: "hsl(var(--card))", foreground: "hsl(var(--card-foreground))", }, }, borderRadius: { lg: "var(--radius)", md: "calc(var(--radius) - 2px)", sm: "calc(var(--radius) - 4px)", }, keyframes: { "accordion-down": { from: { height: "0" }, to: { height: "var(--radix-accordion-content-height)" }, }, "accordion-up": { from: { height: "var(--radix-accordion-content-height)" }, to: { height: "0" }, }, }, animation: { "accordion-down": "accordion-down 0.2s ease-out", "accordion-up": "accordion-up 0.2s ease-out", }, }, }, plugins: [require("tailwindcss-animate")], } ``` [30 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]