HackStack

Project Info

Winner

[Graphite] Most Likely to get Acquired by a Sponsor (OB–4 black bluetooth speaker); [Vercel] Best Use of Vercel/Best Deployed on Vercel (2k cash prize + $2.4k in Pro-level access credits ($100/month for 4 winners, access for 6 months.)

Snappier

Devpost

Inspiration

Most people can't automate their work because automation tools require programming skills. You need to understand APIs, data mapping, and conditional logic. Meanwhile, those same people teach their coworkers how to do things all the time by just showing them. We wanted to make automation work the way humans naturally teach: by demonstration, not configuration.

What it does

Snappier turns screen recordings into automated workflows in 3 steps: Record — Perform your workflow once in the browser. Our Chrome extension captures everything (e.g., scrape LinkedIn → add to Google Sheets → send Slack message) Record — Perform your workflow once in the browser. Our Chrome extension captures everything (e.g., scrape LinkedIn → add to Google Sheets → send Slack message) Generate — AI watches the video and builds the automation. Gemini extracts what you did, Claude figures out the API calls, and we map it to 100+ SaaS tools via Composio Generate — AI watches the video and builds the automation. Gemini extracts what you did, Claude figures out the API calls, and we map it to 100+ SaaS tools via Composio Run — One click executes the entire workflow across your connected apps. Run it again anytime, or schedule it automatically Run — One click executes the entire workflow across your connected apps. Run it again anytime, or schedule it automatically No drag-and-drop builders. No field mapping. Just show us once and we automate it forever.

How we built it

Frontend: Next.js 15 + React 19 + TypeScript + Tailwind CSS + React Flow (interactive workflow graphs) AI Pipeline: Google Gemini (gemini-3-pro-preview) for multimodal video understanding. Analyzes recordings frame-by-frame to extract structured steps with intent, confidence scores, and timestamps Claude Haiku for fast intent summarization and noise filtering to remove accidental clicks Claude Sonnet for workflow execution reasoning with tool-use calling OpenAI Whisper for audio transcription for narrated workflows Vercel AI SDK as unified interface for all LLMs with structured output (Zod schemas) Integrations: Composio provides 100+ pre-built SaaS tool integrations with OAuth and tool-calling APIs Cloudflare R2 for scalable video storage Clerk for user authentication Chrome Extension: Manifest V3 extension that captures screen video (WebM) + granular DOM events (clicks, keyboard, navigation) Architecture: Multi-modal step extraction (AI vision + DOM events + audio) → Intent summarization → Tool detection via URL pattern matching → Workflow generation → Composio execution with full audit logs

Challenges we ran into

Video understanding reliability – Getting Gemini to consistently extract structured steps from noisy screen recordings took extensive prompt engineering and low-temperature sampling (0.2) Video understanding reliability – Getting Gemini to consistently extract structured steps from noisy screen recordings took extensive prompt engineering and low-temperature sampling (0.2) Intent vs. action gap – Users click buttons in the browser, but workflows need to call APIs. Bridging "clicked the blue button" to "send_email via Gmail API" required semantic reasoning with Claude Intent vs. action gap – Users click buttons in the browser, but workflows need to call APIs. Bridging "clicked the blue button" to "send_email via Gmail API" required semantic reasoning with Claude Noise filtering – We didn't realize how many accidental clicks, scrolls, and micro-adjustments users make. Our AI had to learn to ignore 80% of recorded actions Noise filtering – We didn't realize how many accidental clicks, scrolls, and micro-adjustments users make. Our AI had to learn to ignore 80% of recorded actions OAuth at scale – Managing connections across 100+ SaaS tools (each with different auth flows) was only feasible thanks to Composio's abstraction layer OAuth at scale – Managing connections across 100+ SaaS tools (each with different auth flows) was only feasible thanks to Composio's abstraction layer Real-time workflow editing – Keeping the React Flow graph, step list, and video player synchronized while the user edits was a complex state management challenge Real-time workflow editing – Keeping the React Flow graph, step list, and video player synchronized while the user edits was a complex state management challenge

Accomplishments we're proud of

Multi-modal AI fusion – We combined video, DOM events, and audio into a single coherent understanding. Each modality validates the others (if Gemini sees a click AND the DOM recorded a click, high confidence) Multi-modal AI fusion – We combined video, DOM events, and audio into a single coherent understanding. Each modality validates the others (if Gemini sees a click AND the DOM recorded a click, high confidence) End-to-end working demo – From recording to execution, the full pipeline works. You can actually record a workflow and run it against real SaaS APIs End-to-end working demo – From recording to execution, the full pipeline works. You can actually record a workflow and run it against real SaaS APIs Intent-aware automation – We don't just replay mechanical actions. Snappier understands what you were trying to accomplish and finds the best API-level way to do it Intent-aware automation – We don't just replay mechanical actions. Snappier understands what you were trying to accomplish and finds the best API-level way to do it Control flow detection – Our AI detects loops ("for each row in this spreadsheet...") and conditionals, not just linear sequences Control flow detection – Our AI detects loops ("for each row in this spreadsheet...") and conditionals, not just linear sequences Production-ready architecture – Cloudflare R2 storage, Clerk auth, comprehensive test suite (Vitest), type-safe throughout Production-ready architecture – Cloudflare R2 storage, Clerk auth, comprehensive test suite (Vitest), type-safe throughout

What we learned

Multimodal > unimodal – Video analysis alone misses context. DOM events alone can't infer intent. Audio alone is ambiguous. Together, they're powerful. Multimodal > unimodal – Video analysis alone misses context. DOM events alone can't infer intent. Audio alone is ambiguous. Together, they're powerful. Low-temperature LLMs are essential – For structured extraction, we needed temperature=0.2 or lower. Creativity kills reliability here. Low-temperature LLMs are essential – For structured extraction, we needed temperature=0.2 or lower. Creativity kills reliability here. Tool abstractions unlock scale – Building 100+ integrations from scratch would've been impossible. Composio's tool-calling abstraction made it feasible. Tool abstractions unlock scale – Building 100+ integrations from scratch would've been impossible. Composio's tool-calling abstraction made it feasible. Users are noisy – Humans don't realize how many random clicks, back-buttons, and scrolls they do. Filtering signal from noise is 80% of the challenge. Users are noisy – Humans don't realize how many random clicks, back-buttons, and scrolls they do. Filtering signal from noise is 80% of the challenge. Video-first is fundamentally different – Teaching by demonstration feels more natural than form-based builders. It's how we teach humans; why not computers? Video-first is fundamentally different – Teaching by demonstration feels more natural than form-based builders. It's how we teach humans; why not computers?

What's next

Runtime control flow – Execute loops and conditionals, not just detect them Scheduled workflows – Cron-style automation (run every morning, every week, etc.) Team collaboration – Share workflow libraries across organizations Workflow marketplace – Discover and remix workflows from the community Mobile recording – iOS/Android screen recording support Advanced parameter inference – Better extraction of dynamic values (names, emails, dates) from recordings Multi-step editing – Visual graph editor to rearrange, add, or remove steps post-recording

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only
  • CSSIn code
  • Google GeminiIn code
  • Next.jsIn code
  • OpenAIIn code
  • ReactIn code
  • TypeScriptIn code
  • Vercel AI SDKIn code
  • Tailwind CSSClaimed
  • VercelClaimed

7 of 9 appear in the indexed code. 2 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

  • Claude CodeCommits

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

661 KB

Source files

127

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

0 stars