Inspiration

We've all had that moment — you reach for the dish soap mid-wash, and it's empty. You're out of coffee filters on a Monday morning. The essentials we depend on daily always seem to run out at the worst possible time. We realized that the cameras and smart devices already in our homes could do more than just sit there. What if they could see what you're running low on and handle it before you even notice? That's the idea behind Foresight — turning passive home cameras into an intelligent, proactive shopping assistant that never lets you run out of the things that matter. What It Does Foresight is an AI-powered smart home commerce platform that uses your existing cameras to detect household items in use, predict when they're running low, and automatically purchase replacements through Amazon — all with a single tap of approval. Core Features Real-Time AI Vision Detection — A live camera feed powered by OpenAI's GPT-5 nano analyzes your environment in real time, identifying household products and their usage context (e.g., washing dishes → dish soap, cooking → olive oil). Complementary Item Recommendations — The AI doesn't just detect what's visible — it recommends complementary items you might need based on what it sees (see a mouse → recommend a keyboard, see a TV → recommend a streaming device). One-Tap Purchase Approval — Detected items appear as cards on your dashboard. Approve with a single tap, and Foresight's AI agent handles the rest. Autonomous Browser Shopping Agent — Powered by Browserbase and Playwright, an AI agent opens a real browser session, searches Amazon, selects the best product, and adds it to your cart — all autonomously. Live Shopping View — Watch the AI agent shop in real time through an embedded live browser view directly in the dashboard. A browser-style modal shows the agent navigating Amazon, searching, and adding items to cart. Smart Dashboard — An analytics dashboard with weekly spending charts, total savings, items tracked, and a live activity feed of all purchases and detections. Additional Features Items stack without duplicates, limited to 2 items per detection for quality recommendations Confidence scores randomized between 70-100% for realistic detection display Activity feed with timestamped entries for approvals, declines, and purchases Dynamic stats (total saved, items tracked, automation rate) update in real time Responsive design with dark theme optimized for home dashboard displays How We Built It Our tech stack combines cutting-edge AI with modern web technologies: Frontend: React 18 + TypeScript, built with Vite for instant hot module reloading. Tailwind CSS and shadcn/ui provide a sleek, dark-themed dashboard UI. Framer Motion powers smooth animations throughout. AI Vision: OpenAI's GPT-5 nano model processes camera frames via the Chat Completions API with vision capabilities. We chose GPT-5 nano for its speed — real-time detection needs sub-second responses. The model receives base64-encoded frames and returns structured JSON with item names, brands, confidence scores, and reasoning. Prompt Engineering: Carefully crafted prompts instruct the model to recommend complementary items rather than just identifying what's on screen. The AI thinks about what activities are happening and what related products would be useful. Browser Automation: Browserbase SDK creates cloud browser sessions, and Playwright-core connects via Chrome DevTools Protocol (CDP) to automate the full Amazon shopping flow — search, product selection, and add-to-cart. Live View: Browserbase's debug API provides a real-time fullscreen view URL that we embed as an iframe, letting users watch the AI agent shop in real time. Backend: A lightweight Express.js server handles Browserbase session management and Playwright automation. Runs on Node.js (required by Playwright — Bun is not supported). State Management: localStorage bridges the Camera component and Dashboard, with polling for real-time sync of pending purchases. Security: All API keys (OpenAI, Browserbase) stored in .env and loaded via dotenv — never exposed in client-side code or git history. Challenges We Ran Into Playwright + Bun Incompatibility — Discovered mid-build that Playwright doesn't support Bun's runtime. Had to set up a separate Node.js server process alongside the Bun-powered Vite frontend. GPT-5 Token Handling — GPT-5 requires max_completion_tokens instead of the older max_tokens parameter. Initial attempts with 500 tokens resulted in the model using all tokens for reasoning with no visible output — bumped to 2000. Browserbase CDP Connection — Getting Playwright to connect to Browserbase sessions via connectOverCDP required using the SDK's session.connectUrl rather than manually constructed WebSocket URLs. Amazon Selector Stability — Amazon's DOM structure varies across page loads. Went through multiple iterations of selectors, settling on broad patterns (.s-main-slot .s-result-item h2 a) with fallbacks and timeouts. Real-Time UX — Balancing the async nature of browser automation (takes 10-20 seconds) with a responsive UI. Solved by returning the live view URL immediately and running automation asynchronously. False Positive Reduction — Early prompts were too aggressive, detecting items in every frame. Refined to an evidence-based approach requiring clear visual evidence before recommendations. Accomplishments We're Proud Of Built a fully autonomous AI shopping agent that navigates real websites in under 36 hours Achieved real-time vision detection with GPT-5 nano at interactive speeds Created a live browser view that lets users watch AI agents work in real time Designed an intuitive dark-themed dashboard that feels like a premium smart home product Engineered complementary item recommendations that feel genuinely useful, not just pattern matching Seamlessly integrated 4 different APIs (OpenAI, Browserbase, Playwright, Amazon) into one cohesive flow What We Learned Advanced prompt engineering for vision models — the difference between "detect everything" and "recommend intelligently" is massive Real-time browser automation at scale with cloud browser infrastructure The nuances of connecting to remote browsers via CDP and managing session lifecycles Complex state management across independent React components using localStorage as a message bus How to build UX around async processes that take 10+ seconds without losing user engagement Web scraping resilience — Amazon's DOM is a moving target What's Next for Foresight 1. Smarter AI Usage tracking over time to predict depletion dates Multi-room camera support with unified inventory Price comparison across retailers before purchasing Learning user brand preferences from past approvals 2. Full Purchase Automation Saved payment methods for true one-click checkout Subscription detection — auto-set up Subscribe & Save for recurring items Budget limits and spending caps per category 3. Platform Expansion Integration with Walmart, Target, and Costco Smart home platform integration (HomeKit, Google Home, Alexa) Mobile app with push notifications for approval requests Voice approval — "Hey Foresight, approve the dish soap" 4. Privacy & Security On-device vision processing option (no cloud API calls) End-to-end encrypted camera feeds Granular privacy controls — exclude rooms, time windows, item categories Our vision is to make Foresight the invisible hand that keeps your home stocked so you never have to think about running out of anything again.