Project Info

KlarifAi

Inspiration

The idea for KlarifAi came from a problem we experienced ourselves while studying for standardized exams like the SAT. We frequently used ChatGPT to help solve math problems, but we found ourselves spending a lot of time manually describing equations, diagrams, graphs, and problem statements. For visual questions in particular, explaining the context accurately was often more work than solving the problem itself. We started asking a simple question: why should students have to translate what they see into text for an AI to understand it? What if an AI could see the problem directly and explain it the same way a tutor sitting next to you would? That question became the foundation for KlarifAi.

What it does

KlarifAi is a desktop AI assistant that analyzes what's on your screen and provides visual, step-by-step explanations for math problems in real time. Instead of copying and pasting questions or manually describing diagrams, users can simply have the problem visible on their screen. KlarifAi identifies the relevant mathematical content, understands the context, and generates guided explanations that break the solution into manageable steps. The goal is not just to provide answers, but to help students understand the reasoning behind them through visual and interactive guidance.

How we built it

At the core of KlarifAi is a multi-step AI workflow designed to replicate how a human tutor approaches a problem. First, the system captures and analyzes the user's screen, identifying the relevant mathematical content and surrounding context. That information is stored and managed through a memory layer that preserves important details throughout the interaction. Next, KlarifAi classifies the user's intent—whether they need a full solution, a hint, conceptual clarification, or a step-by-step walkthrough. Based on that intent, the system performs targeted research and gathers additional context that may help explain the underlying concepts. Finally, all of this information—the screen analysis, conversation memory, user intent, research results, and visual context—is combined into a single reasoning request. The final AI call synthesizes everything into a structured explanation that is then rendered visually on screen for the user. Rather than relying on a single prompt, KlarifAi uses an orchestrated pipeline that separates observation, memory management, research, reasoning, and presentation into a unified workflow.

Challenges we ran into

One of our biggest challenges was reliably understanding complex mathematical content from screenshots. SAT-style problems often include graphs, geometric diagrams, tables, and formatting that can be difficult for AI systems to interpret correctly. Another challenge was balancing accuracy with usability. We wanted explanations that were detailed enough to teach concepts without overwhelming students with unnecessary information. We also spent significant time optimizing the user experience so that assistance felt immediate and seamless rather than requiring multiple manual steps.

Accomplishments we're proud of

We're proud that we transformed a frustration we personally experienced into a working product that feels natural to use. Instead of forcing students to rewrite questions for an AI, KlarifAi allows them to get help directly from what they already have on screen. We're especially proud of the quality of the visual step-by-step explanations and how quickly users can move from confusion to understanding. Most importantly, we built a tool that focuses on learning and comprehension rather than simply generating answers.

What we learned

Building KlarifAi taught us that the biggest barriers to learning are often not the concepts themselves, but the friction involved in accessing help. We learned how important visual context is for educational AI applications and how much information can be lost when students are forced to convert diagrams and equations into plain text. On the technical side, we gained experience integrating computer vision, multimodal AI systems, desktop application development, and user-centered design into a single product.

What's next

Our vision is to expand KlarifAi into a universal learning companion that can assist students across a wide range of subjects, not just mathematics. Future plans include: Support for science, physics, and engineering problems. More interactive visual explanations and tutoring experiences. Personalized learning paths based on student strengths and weaknesses. Improved real-time screen understanding and context awareness. Collaboration features for study groups and classrooms. Ultimately, we want KlarifAi to make high-quality, personalized tutoring accessible to anyone with a computer, turning every screen into an opportunity to learn.

🎓 ScreenTutor AI

A cursor-native AI tutor that lives near your cursor, captures the screen on demand, and draws overlays directly on top of your windows.

What It Does

Cursor-Native Buddy: Buddy lives near your cursor, follows you around in a compact bubble state, and expands to a full chat panel on request. When explaining concepts, Buddy smoothly animates (flies) to coordinates on your screen to point at items.
Captures Screen on Demand: Trigger screen capture explicitly using Ctrl+Shift+E (Cmd+Shift+E on macOS) or the floating microphone button. Never records in the background or polls your screen silently.
Draws Overlays on Top of Apps: Draws circles, arrows, highlighters, step badges, and text labels directly on a transparent SVG window layered over your active screen content.
Remembers Weak Concepts: Stores learner profiles and session history in Convex (or an in-memory fallback), and builds spaced-repetition study plans with a Fetch.ai agent.
Co-drive: Claude computer use proposes the next click or keystroke; nothing happens on your desktop until you approve it.
Speech Interactivity: Push-to-talk microphone input records audio chunk sessions safely. Text-to-speech engine speaks answers out loud while stripping coordinate tags.

Quick Start

Prerequisites

Node.js 20+
pnpm 9+
Python 3.11+ (optional — only for the Fetch.ai study-planner agent)

Install & Run

# 1. Clone the repository
git clone https://github.com/DMgaming00/ScreenTutor_AI.git
cd ScreenTutor_AI

# 2. Install dependencies
pnpm install

# 3. Create env file
cp .env.example .env

# 4. Start Convex for learner memory (optional — falls back to in-memory)
cd services/orchestrator && npx convex dev   # leave running; copy the URL into CONVEX_URL

# 4b. Start the Fetch.ai study planner (optional — see services/fetch-agent/README.md)

# 5. Start development environment (orchestrator + desktop concurrently)
pnpm dev                       # same command on Windows (PowerShell/cmd) and macOS (Terminal)

Configuration & Setup

ScreenTutor AI works out of the box in Mock Mode using local templates for lessons. To configure real integrations, set the following environment variables in .env:

1. Mock Mode (Default)

Runs entirely locally without external API connections or paid accounts.

AI_PROVIDER=mock
MEMORY_ENABLED=true

2. Convex (Memory)

Learner profiles, weak concepts, session turns and the response cache live in Convex. Functions are in services/orchestrator/convex/.

CONVEX_URL=https://your-deployment.convex.cloud   # printed by `npx convex dev`
SCREENTUTOR_CONVEX_SECRET=any-long-random-string

Set the same secret on the deployment: npx convex env set SCREENTUTOR_CONVEX_SECRET <value>. Without these, the orchestrator uses in-process memory.

Moss (Semantic Cache)

Reuses an answer when a reworded question is asked about the same screenshot.

MOSS_PROJECT_ID=your-project-id
MOSS_PROJECT_KEY=your-project-key

3. Claude (Anthropic AI)

For real-time multi-modal screen capture analysis and chat responses.

ANTHROPIC_API_KEY=sk-ant-your-key-here
CLAUDE_MODEL=claude-opus-5-5   # default
CLAUDE_EFFORT=low              # raise to medium/high for harder diagrams
CLAUDE_SPEED=fast              # default: Opus fast mode (~2.5x faster, premium price); "standard" to save cost

Measured on a geometry screenshot (full answer): Opus 5.5 standard 16.7 s, Opus 5.5 fast 6.7 s, Sonnet 5.5 with CLAUDE_THINKING=off 8.6 s, Haiku 5.5 with thinking off 4.5 s. Responses use structured outputs (JSON schema), the system prompt is prompt-cached, and refusals fall back server-side.

Cartesia (Voice)

Streaming Sonic text-to-speech and Ink speech-to-text. The orchestrator only mints short-lived access tokens; the desktop app streams audio directly to/from Cartesia. Falls back to the system voice and browser speech recognition when unset.

CARTESIA_API_KEY=sk_car_your-key
CARTESIA_VOICE_ID=your-voice-id

Apify (Research Mode)

Searches the web for the concept at the learner's level and summarizes the top pages (RAG Web Browser actor).

APIFY_API_TOKEN=apify_api_your-token

Claude Computer Use (Co-drive)

Uses your ANTHROPIC_API_KEY. Claude proposes up to three steps as approval cards; approved steps run on your desktop (Windows and macOS) only when desktop control is on. On macOS, grant Accessibility permission when prompted.

CODRIVE_ENABLED=true
ENABLE_SAFE_ACTIONS=true   # off by default: approvals are shown but not performed

Fetch.ai / Agentverse (Study Planner Agent)

A uAgent in services/fetch-agent routes questions and builds spaced-repetition study plans; it registers on Agentverse via a mailbox.

FETCH_AGENTVERSE_ENABLED=true
FETCH_AGENT_URL=http://localhost:8001

Elastic (Observability)

Each tutoring step is exported as an OpenTelemetry span (OTLP/HTTP protobuf) with latencies and evaluation scores.

ELASTIC_OTLP_ENDPOINT=https://<id>.ingest.<region>.elastic.cloud:443
ELASTIC_API_KEY=your-api-key

Sentry (Error Tracking)

Captures backend runtime errors and desktop crashes.

SENTRY_ENABLED=true
SENTRY_DSN=https://your-dsn-key@o0.ingest.sentry.io/0

macOS notes

Permissions: Screen Recording (capture), Microphone (voice) and Accessibility (Co-drive) are requested on first use. Status is logged at startup as [Permissions] ....
Build a .dmg: pnpm dist:mac (output in apps/desktop/release/). Unsigned builds need right-click → Open the first time; distributing to others needs an Apple Developer ID certificate and notarization.
Windows installer: pnpm dist:win.

Privacy Model & Redaction

On-Demand Capture Only: Captures the screen only when the hotkey is triggered. No background polling or recordings.
Sensitive Window Denylist: Analysis is immediately blocked if any of the 16 sensitive window title terms (e.g. password, keychain, bank, email, stripe) are active.
Data Redaction Layer: A shared RedactionService automatically strips passwords, API keys, emails, credit cards, and query parameters from all logs, Sentry events, Elastic traces, and UI debug views.
Developer Redaction Override: Available as a toggle in the developer panel, allowing developers to see raw data for debugging. It is turned off by default.
No Screenshot Storage: Screenshots are processed in memory and never written to disk or sent to observability endpoints. Only a secure MD5 hash is stored.

Bundled Demo Lessons

ScreenTutor includes six bundled offline lessons accessible by clicking the Graduate Cap (🎓) icon in the header:

Algebra: distributive property
Algebra: common mistake
Calculus: chain rule
Calculus: u-substitution
Graph interpretation: vertex/slope
Coding error: off-by-one IndexError

Selecting a lesson displays a static lesson preview and allows you to test the hotkey, drawing overlays, and voice synthesis without external network calls.

Troubleshooting

`Failed to reach Convex, falling back to InMemoryService`

Cause: CONVEX_URL is wrong, npx convex dev isn't running, or the secret doesn't match the deployment's SCREENTUTOR_CONVEX_SECRET.
Fix: Memory is optional; the app keeps working in RAM. Re-run npx convex env set SCREENTUTOR_CONVEX_SECRET <value> with the value from .env.

Hotkey `Ctrl+Shift+E` (`Cmd+Shift+E` on macOS) Not Working

macOS: The hold-to-talk hotkey is detected by polling key state (not expected to need extra permission, but untested on every macOS version), while Co-drive clicks and keystrokes need Accessibility in System Settings → Privacy & Security → Accessibility. If push-to-talk never starts, set SCREENTUTOR_HOTKEY_MODE=toggle in .env (press once to start, again to stop).
Windows: Run as Administrator if hotkeys are intercepted by other fullscreen windows.

Screen Capture Black / Empty

macOS: Grant Screen Recording in System Settings → Privacy & Security → Screen Recording, then quit and relaunch the app (macOS only applies it on restart). In pnpm dev the permission belongs to the Electron binary (or your terminal app), not ScreenTutor.

`listen EADDRINUSE :::4317`

Cause: Port 4317 is already in use by another process.
Fix: Change ORCHESTRATOR_PORT in your .env file to a free port.

Analysis

Compare with all teams

View

Metric

Cursor
4
Andre Agle
3

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

AnthropicIn code
CSSIn code
HTMLIn code
JavaScriptIn code
ReactIn code
TypeScriptIn code
RedisClaimed

6 of 7 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

CursorCommits

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

709 KB

Source files

147

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.