# Project export: Texify

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: UC Berkeley AI Hackathon 2025
- Tagline: Transform your messy handwritten notes and sketches seamlessly into pristine, verbatim LaTeX documents and PDFs, powered by intelligent LLM-based agents—preserving every detail effortlessly.
- Devpost: https://devpost.com/software/texify-ntedrf
- GitHub: https://github.com/Fuyi-Yang/calhacks25
- Video: https://www.youtube.com/embed/koNmBT3Vfuc?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 3 GitHub contributor(s) — Kevin Sheng (16 commits), Xuanhao Cui (8 commits), Fuyi-Yang (3 commits)

## Devpost submission (written by the team)

### Inspiration

We’ve all been there — spending hours typing LaTeX just to get a clean-looking homework or report. It’s slow, picky, and kind of a pain, especially when some classes require everything in LaTeX. On top of that, turning your scratch paper sketches into polished figures for papers or slides is another time sink. Existing handwriting-to-LaTeX conversion tools focus on processing single equations or short expressions. While useful for small tasks, they fall short when it comes to full derivations, structured proofs, or entire documents. These systems often lack contextual understanding, leading to errors when symbols are ambiguous or when spatial layout carries meaning. Additionally, requiring users to process one equation at a time interrupts workflow and becomes cumbersome. As a result, they provide only marginal assistance and fail to reduce the effort required to write and format LaTeX documents, especially in academic or technical settings. That’s why we built Texify. It takes your handwritten notes and diagrams and turns them into high-quality LaTeX PDFs, automatically. No more retyping math or redrawing figures — just write like you normally would, and Texify does the rest.

### What it does

Texify takes user handwritten notes, figures, problem solutions, or even scratch work, and converts them into PDF format. Texify cleans up and smoothes out the logical flow within the derivation, inserting English explanations for better reader understanding even when the user inputs only handwritten math.

### How we built it

Texify generates a polished LaTeX PDF by running two pipelines in parallel: text processing and figure rendering. Users can upload either individual images or a full PDF; images are first converted into a unified PDF for consistency. The text pipeline uses Gemini 2.5 Pro to extract handwritten content and convert it into structured LaTeX. It supports three processing modes: verbatim transcription, logical rewriting for improved clarity, and summarization for concise output. During this step, placeholders are inserted where figures appear, and the system retries compilation if errors are detected. In parallel, Texify processes figures by detecting visual elements in the input and generating a numbered list of figure descriptions. This numbering provides a unique identifier for each figure, allowing a pool of LLM agents to process them all simultaneously without ambiguity. These descriptions are then used to generate Asymptote code via the LLM agent, which is compiled into high-quality diagrams. Rendering is performed concurrently for efficiency, and failed attempts are retried with appended error diagnostics. Once both text and figures are ready, they are merged into a single .tex file, which can be compiled and returned to the user as a clean, readable PDF, Markdown, or other document formats.

### Challenges we ran into

We ran into many problems with syntactically incorrect Asymptote code, which we fixed by appending the error message and asking the model to revise the Asymptote code. Our underlying multimodal model also struggles to replicate the figure exactly with Asymptote and closely align the generated images with the original figures. Therefore, we will look into using more traditional CV methods to improve image generation and construct higher quality figures.

### Accomplishments we're proud of

We finished this project in just 10 hours and got to spend the rest of our time sleeping and socializing.

### What we learned

We found Gemini 2.5 Pro to be especially effective for rapid prototyping. By first designing a high-level logic flow diagram, we were able to prompt Gemini to generate the initial skeleton of our entire codebase. This significantly accelerated development, allowing us to iterate quickly and focus our efforts on fine-tuning functionality and integrating complex components.

### What's next

In future iterations, we aim to improve the quality and reliability of figure generation by incorporating deep learning models specifically for Asymptote code generation, which would help Texify better understand diagram structure and produce more accurate visual outputs. Specifically, we plan to integrate traditional computer vision techniques to more precisely detect and crop hand-drawn sketches from PDFs. To further enhance visual fidelity, we are exploring generative models such as diffusion models and GANs for style transfer and image sharpening, enabling more aligned and aesthetically refined figure generation.

## README (from the GitHub repository)

Ok so I guess we are making like a HW/document translator for text to latex/markdown.


## Detected evidence (automated analysis)

Indexed codebase: 18 recognized source files, 34 KB.
- CSS (language) — detected in the code
- Next.js (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- FastAPI (technology) — claimed on Devpost, not found in the code
- Google Gemini (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (25 of 25)

```
.gitignore
api.py
document_processor.py
frontend/.gitignore
frontend/components.json
frontend/next.config.ts
frontend/package.json
frontend/postcss.config.mjs
frontend/src/app/globals.css
frontend/src/app/layout.tsx
frontend/src/app/page.tsx
frontend/src/components/ui/button.tsx
frontend/src/components/ui/input.tsx
frontend/src/components/ui/label.tsx
frontend/src/lib/utils.ts
frontend/tsconfig.json
handlers/__init__.py
handlers/figure_processor.py
handlers/llm_handler.py
handlers/pdf_handler.py
main.py
README.md
requirements.txt
utils/__init__.py
utils/latex_renderer.py
```

### Dependencies

- frontend/package.json: @radix-ui/react-label@^2.1.7, @radix-ui/react-slot@^1.2.3, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, class-variance-authority@^0.7.1, clsx@^2.1.1, lucide-react@^0.522.0, next@15.3.4, react@^19.0.0, react-dom@^19.0.0, tailwind-merge@^3.3.1, tailwindcss@^4, tw-animate-css@^1.3.4, typescript@^5
- requirements.txt: aiofiles, fastapi[standard], google-generativeai, Pillow, PyMuPDF, python-dotenv

### Recent commits (newest first)

- disable upload button when processing too
- ok at least the download button works
- why the hell did i have to copy paste a library manually
- startup a stupid api
- remove dead code & update deps
- refined figure generation pipeline
- Merge branch 'main' of https://github.com/Fuyi-Yang/calhacks25
- update
- images should also be allowed
- jesus christ is this what gpt can do
- init frontend
- bro what
- removed starting and ending tags from final latex output
- Merge branch 'main' of https://github.com/Fuyi-Yang/calhacks25
- fix
- god ai is so fake
- clean unused function
- Merge branch 'main' of https://github.com/Fuyi-Yang/calhacks24
- fixes
- make `pdf_handler` actually handle pdf

## Key source files (fetched from GitHub, selected and truncated for size)

### requirements.txt

```
google-generativeai
python-dotenv
PyMuPDF
Pillow
fastapi[standard]
aiofiles

```

### frontend/package.json

```
{
  "name": "texify",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "@radix-ui/react-label": "^2.1.7",
    "@radix-ui/react-slot": "^1.2.3",
    "class-variance-authority": "^0.7.1",
    "clsx": "^2.1.1",
    "lucide-react": "^0.522.0",
    "next": "15.3.4",
    "react": "^19.0.0",
    "react-dom": "^19.0.0",
    "tailwind-merge": "^3.3.1"
  },
  "devDependencies": {
    "@tailwindcss/postcss": "^4",
    "@types/node": "^20",
    "@types/react": "^19",
    "@types/react-dom": "^19",
    "tailwindcss": "^4",
    "tw-animate-css": "^1.3.4",
    "typescript": "^5"
  }
}

```

### main.py

```python
import argparse
from dotenv import load_dotenv
from document_processor import DocumentProcessor


def main():
    # Load environment variables from .env file
    load_dotenv(override=True)

    parser = argparse.ArgumentParser(
        description="Convert an image or PDF file to a LaTeX document with generated figures.",
        formatter_class=argparse.RawTextHelpFormatter,
    )
    parser.add_argument(
        "input_file", type=str, help="Path to the input image or PDF file."
    )
    parser.add_argument(
        "-o",
        "--output_dir",
        type=str,
        default="output",
        help="Directory to save the output .tex file and figures (default: 'output').",
    )
    parser.add_argument(
        "-m",
        "--mode",
        type=str,
        choices=["rewriting", "summarizing", "verbatim"],
        default="verbatim",
        help="The mode for processing text:\n"
        "  rewriting:   Rewrite text for clarity.\n"
        "  summarizing: Summarize the text.\n"
        "  verbatim:    Keep original text, but format it. (default)",
    )

    args = parser.parse_args()

    processor = DocumentProcessor(
        input_path=args.input_file, output_dir=args.output_dir, text_mode=args.mode
    )
    processor.process()


if __name__ == "__main__":
    main()

```

### frontend/src/app/layout.tsx

```typescript
import type { Metadata } from "next";
import { Geist, Geist_Mono } from "next/font/google";
import "./globals.css";

const geistSans = Geist({
  variable: "--font-geist-sans",
  subsets: ["latin"],
});

const geistMono = Geist_Mono({
  variable: "--font-geist-mono",
  subsets: ["latin"],
});

export const metadata: Metadata = {
  title: "Create Next App",
  description: "Generated by create next app",
};

export default function RootLayout({
  children,
}: Readonly<{
  children: React.ReactNode;
}>) {
  return (
    <html lang="en">
      <body
        className={`${geistSans.variable} ${geistMono.variable} antialiased`}
      >
        {children}
      </body>
    </html>
  );
}

```

### frontend/src/app/page.tsx

```typescript
"use client";

import { useEffect, useState } from "react";
import { Button } from "@/components/ui/button";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";

export default function Home() {
  const [file, setFile] = useState<File | null>(null);
  const [message, setMessage] = useState<string | null>(null);
  const [task, setTask] = useState<string | null>(null);
  const [done, setDone] = useState(false);

  useEffect(() => {
    const interval = setInterval(async () => {
      if (!task) {
        return;
      }
      const { status } = await fetch(`/api/status/${task}`).then((r) =>
        r.json()
      );
      setMessage(`status: ${status}`);
      if (status === "done") {
        setDone(true);
      }
    }, 3000);
    return () => clearInterval(interval);
  }, [task]);

  const handleUpload = async () => {
    if (!file) {
      return;
    }

    const formData = new FormData();
    formData.append("file", file);
    const { tid } = await fetch("/api/verbatim", {
      body: formData,
      method: "POST",
    }).then((r) => r.json());

    setDone(false);
    setTask(tid);
    setMessage(`task ${tid} is ready to go!`);
  };

  const dl = async () => {
    fetch(`/api/dl/${task}`)
      .then((response) => {
        if (!response.ok) {
          throw new Error("File not found or failed to fetch");
        }
        return response.blob();
      })
      .then((blob) => {
        const link = document.createElement("a");
        const url = window.URL.createObjectURL(blob);
        const name = file!.name.replace(/\.[^/.]+$/, "");
        link.href = url;
        link.download = `${name}.tex`;
        link.click();
        window.URL.revokeObjectURL(url);
      });
  };

  const handleFileChange = (e: React.ChangeEvent<HTMLInputElement>) => {
    const selected = e.target.files?.[0];
    if (!selected) return;

    const allowedTypes = [
      "application/pdf",
      "image/png",
      "image/jpeg",
      "image/jpg",
    ];

    if (allowedTypes.includes(selected.type)) {
      setFile(selected);
      setMessage(null);
    } else {
      setFile(null);
      setMessage("Only PDF or image files (JPG, PNG) are allowed.");
    }
  };

  return (
    <main className="min-h-screen flex flex-col items-center justify-center p-6 bg-gray-100">
      <div className="bg-white p-8 rounded shadow-md w-full max-w-md">
        <h1 className="text-xl font-bold mb-4">Texify</h1>
        <Label htmlFor="upload">Select a PDF or Image</Label>
        <Input
          id="upload"
          type="file"
          accept=".pdf,image/png,image/jpeg"
          onChange={handleFileChange}
          className="mb-4"
        />
        <div className="flex justify-between">
          <Button onClick={handleUpload} disabled={!file || (file && !!task && !done)}>
            Upload
          </Button>
          <Button onClick={dl} disabled={!done}>
            Download
          </Button>
        </div>
        {message && <p className="mt-4 text-sm text-gray-700">{message}</p>}
      </div>
    </main>
  );
}

```

### api.py

```python
import os
from uuid import UUID, uuid4
import threading

import aiofiles
from fastapi import FastAPI, HTTPException, UploadFile
from fastapi.middleware.cors import CORSMiddleware
from starlette.responses import FileResponse

from document_processor import DocumentProcessor

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

status = {}


@app.post("/verbatim")
async def process_pdf(file: UploadFile):
    task_id = uuid4()
    _, ext = os.path.splitext(file.filename)
    path = f"input/{task_id}{ext}"
    async with aiofiles.open(path, "wb") as out_file:
        await out_file.write(await file.read())
    
    status[task_id] = path

    doc = DocumentProcessor(path, "output", "verbatim")
    thread = threading.Thread(target=doc.process)
    thread.start()

    return {"tid": task_id}


@app.get("/status/{tid}")
async def pdf_status(tid: UUID):
    if tid not in status:
        raise HTTPException(status_code=404, detail="Task not found")

    if os.path.exists(f"output/{tid}.tex"):
        return {"status": "done"}
    else:
        return {"status": "processing"}

@app.get("/dl/{tid}")
async def pdf_dl(tid: UUID):
    path = f"output/{tid}.tex"
    return FileResponse(path, media_type="application/octet-stream")

```

### document_processor.py

```python
# document_processor.py

import os
from handlers.llm_handler import GeminiLLM
from handlers.pdf_handler import PDFHandler
from handlers.figure_processor import FigureProcessor
from utils.latex_renderer import LatexRenderer

class DocumentProcessor:
    """Orchestrates the entire conversion process from input file to .tex output."""

    def __init__(self, input_path: str, output_dir: str, text_mode: str):
        if not os.path.exists(input_path):
            raise FileNotFoundError(f"Input file not found: {input_path}")
        
        self.input_path = input_path
        self.output_dir = output_dir
        self.text_mode = text_mode
        self.pdf_path = ""
        
        os.makedirs(self.output_dir, exist_ok=True)
        
        self.llm = GeminiLLM()
        self.renderer = LatexRenderer()

    def _prepare_pdf(self):
        """Ensures the input is a PDF, converting from image if necessary."""
        filename = os.path.basename(self.input_path)
        name, ext = os.path.splitext(filename)
        
        if ext.lower() in ['.jpg', '.jpeg', '.png']:
            print("Image file detected. Converting to PDF...")
            self.pdf_path = os.path.join(self.output_dir, f"{name}.pdf")
            PDFHandler.convert_image_to_pdf(self.input_path, self.pdf_path)
        elif ext.lower() == '.pdf':
            print("PDF file detected.")
            self.pdf_path = self.input_path
        else:
            raise ValueError(f"Unsupported file type: {ext}")

    def _validate_output(self, s):
        s = s.strip(" \n")
    
        # Check if it starts and ends with ```latex ... ```
        if s.startswith("```latex") and s.endswith("```"):
            # Extract everything between the opening line and the closing ```
            # NOTE:might have issue
            return s[8:-3].strip()
        
        # Otherwise, just return the string as-is
        return s

    def process(self):
        """Executes the full document processing workflow."""
        print("--- Starting Document Processing ---")
        self._prepare_pdf()

        # --- Text Processing Path ---
        # The LLM can process the PDF directly, which is more robust than text extraction.
        latex_template = self.llm.extract_text_to_latex(self.pdf_path, self.text_mode)
        
        # --- Figure Processing Path ---
        print("\n--- Checking for Figures ---")
        pdf_handler = PDFHandler(self.pdf_path)
        pdf_pages_as_images = pdf_handler.get_pages_as_images()
        figure_descriptions = self.llm.get_figure_descriptions(pdf_pages_as_images)
        
        generated_figure_files = []
        if figure_descriptions:
            print(f"\nFound {len(figure_descriptions)} potential figures. Starting parallel processing...")
            fig_processor = FigureProcessor(self.llm, self.renderer, self.output_dir)
            generated_figure_files = fig_processor.process_figures_in_parallel(figure_descriptions)
        else:
            print("No figures found or described. Skipping figure generation.")

        # --- Merging Path ---
        print("\n--- Finalizing LaTeX Document ---")
        final_latex_doc = self.llm.merge_latex_and_figures(latex_template, generated_figure_files)

        # --- validate output
        print("\n--- Validating LaTeX Document ---")
        val_final_latex_doc = self._validate_output(final_latex_doc)

        # --- Output ---
        output_filename = os.path.splitext(os.path.basename(self.pdf_path))[0] + ".tex"
        output_filepath = os.path.join(self.output_dir, output_filename)
        with open(output_filepath, "w", encoding='utf-8') as f:
            f.write(val_final_latex_doc)
            
        print(f"\n✅ Success! Final document saved to: {output_filepath}")

```

### frontend/next.config.ts

```typescript
import type { NextConfig } from "next";

const nextConfig: NextConfig = {
  rewrites: async () => {
    return [
      {
        source: "/api/:path*",
        destination:
          process.env.NODE_ENV === "development"
            ? "http://127.0.0.1:8000/:path*"
            : "/api/",
      },
    ];
  },
};

export default nextConfig;

```

### utils/latex_renderer.py

```python
# utils/latex_renderer.py
import subprocess
import os
import tempfile

class LatexRenderer:
    """A utility to compile Asymptote code."""

    @staticmethod
    def compile_asymptote(asy_code: str, output_dir: str, filename_base: str) -> bool:
        """
        Compiles a string of Asymptote code into a TeX file.
        
        Note: Requires the 'asy' command-line tool to be installed.
        
        Args:
            asy_code: The Asymptote code as a string.
            output_dir: The directory to save the output files.
            filename_base: The base name for the output file (e.g., 'figure1').
            
        Returns:
            True if compilation was successful, False otherwise.
        """
        print(f"Attempting to render {filename_base}.asy...")
        
        # Asymptote works best when run from the target directory
        original_cwd = os.getcwd()
        asy_filepath = os.path.join(output_dir, f"{filename_base}.asy")
        
        try:
            os.makedirs(output_dir, exist_ok=True)
            
            with open(asy_filepath, "w") as f:
                f.write(asy_code)

            os.chdir(output_dir)
            
            # Run asymptote, which will produce a .tex file for inclusion
            result = subprocess.run(
                ['asy', '-f', 'pdf', f'{filename_base}.asy'],
                capture_output=True,
                text=True,
                check=True # Raises CalledProcessError on non-zero exit codes
            )
            print(f"Successfully rendered {filename_base}.pdf.")
            return True
        except FileNotFoundError:
            print("\n---")
            print("ERROR: The 'asy' command was not found.")
            print("Please install Asymptote and ensure it is in your system's PATH.")
            print("---\n")
            return False
        except subprocess.CalledProcessError as e:
            print(f"Failed to compile {filename_base}.asy.")
            print(f"Stderr: {e.stderr}")
            return False
        finally:
            os.chdir(original_cwd) # Always change back to the original directory
```

### handlers/pdf_handler.py

```python
import os

import fitz
import google.generativeai as genai
from PIL import Image

class PDFHandler:
    """Handles PDF operations like text and image extraction."""

    def __init__(self, filepath: str):
        """
        Initializes the handler with a path to a PDF file.

        Args:
            filepath: The path to the PDF file.
        """
        try:
            self.doc = fitz.open(filepath)
        except Exception as e:
            print(f"Error opening PDF {filepath}: {e}")
            raise
        print(f"PDF '{filepath}' loaded successfully with {len(self.doc)} pages.")

        genai.configure(api_key=os.getenv("GOOGLE_API_KEY"))
        self.pdf_model = genai.GenerativeModel("gemini-2.5-pro")
        self.uploaded = genai.upload_file(filepath)
        print("uploaded successfully, my goat")

    def extract_full_text(self) -> str:
        """Extracts concatenated text from all pages of the PDF."""

        full_text = self.pdf_model.generate_content([
            "Extract the text from this PDF of handwritten text"
            "without any markup or additional commentary."
            "If the text looks incoherent, try to fill in the blanks yourself to make"
            "it make sense in the context of a mathematical proof."
            "Again: DO NOT ADD ANY COMMENTARY TO THE PROOFS GIVEN."
            "Simply recite the text as it is on the PDF.",
            self.uploaded
        ])
        return full_text

    def get_pages_as_images(self) -> list[Image.Image]:
        """Converts each page of the PDF into a PIL Image."""
        images = []
        for page_num in range(len(self.doc)):
            page = self.doc.load_page(page_num)
            pix = page.get_pixmap()
            img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)
            images.append(img)
        print("Converted all PDF pages to images.")
        return images

    @staticmethod
    def convert_image_to_pdf(image_path: str, output_pdf_path: str) -> None:
        """
        Converts a single image file to a single-page PDF.

        Args:
            image_path: Path to the input image.
            output_pdf_path: Path to save the output PDF.
        """
        try:
            with Image.open(image_path) as image:
                # Ensure image is in a format that can be saved as PDF
                if image.mode == 'RGBA':
                    image = image.convert('RGB')
                image.save(output_pdf_path, "PDF", resolution=100.0)
                print(f"Image '{image_path}' converted to PDF '{output_pdf_path}'.")
        except Exception as e:
            print(f"Failed to convert image to PDF: {e}")
            raise

```

[7 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]