# Project export: Chop Chop!

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 11.0
- Tagline: Motion-tracking cooking assistant
- Devpost: https://devpost.com/software/chop-chop-42q81w
- GitHub: https://github.com/tthacks24/chopchop
- Team: 3 GitHub contributor(s) — drew-tom1 (19 commits), James Kim (12 commits), kylechu8092 (2 commits)

## Devpost submission (written by the team)

### Overview

Problem Statement Many home cooks struggle with multitasking and managing their time effectively while preparing meals, which often leads to overcooked or undercooked food. College students and individuals with ADHD may also experience difficulties with procrastination and maintaining focus in the kitchen. CHOP CHOP aims to alleviate these challenges by providing a fun and interactive cooking experience that enhances coordination and multitasking abilities. Solution Overview CHOP CHOP is a hands-free cooking assistant that leverages Google Gemini for generative AI and optimized advice, alongside gesture control through the Leap Motion hand tracking sensor. By providing structured cooking tasks and leveraging AI to generate detailed task lists, our project helps home cooks manage their time and streamline the cooking process. Key Features Gesture Control with Leap Motion Users can navigate tasks hands-free by utilizing gestures detected by the Leap Motion sensor: Swipe Gesture: Move to the next task or fast-forward. Tap Gesture: Add more time to the current task. Clap Gesture: Pause or stop the timer. Gesture Control with Leap Motion Users can navigate tasks hands-free by utilizing gestures detected by the Leap Motion sensor: Swipe Gesture: Move to the next task or fast-forward. Tap Gesture: Add more time to the current task. Clap Gesture: Pause or stop the timer. Swipe Gesture: Move to the next task or fast-forward. Tap Gesture: Add more time to the current task. Clap Gesture: Pause or stop the timer. AI-Powered Task Generation Prompt Engineering: Users can input specific cooking prompts (e.g., “I want to cook a medium-rare sirloin steak with unchopped onions and chopped broccoli”). Detailed Task Breakdown: The AI generates a structured list of cooking steps, including time estimates and coordination of overlapping tasks. AI-Powered Task Generation Prompt Engineering: Users can input specific cooking prompts (e.g., “I want to cook a medium-rare sirloin steak with unchopped onions and chopped broccoli”). Detailed Task Breakdown: The AI generates a structured list of cooking steps, including time estimates and coordination of overlapping tasks. Interactive Task List Each task in the list features a "Start" button with a countdown timer that dynamically updates based on user interaction (start, pause, fast-forward). Users can easily visualize their progress through the cooking process. Interactive Task List Each task in the list features a "Start" button with a countdown timer that dynamically updates based on user interaction (start, pause, fast-forward). Users can easily visualize their progress through the cooking process. How It Works User Input: Users provide a cooking prompt, and the AI generates a detailed task list tailored to their recipe. Task Management: Users can interact with the task list, starting timers and navigating tasks with gestures. Technology Stack Frontend: Developed with Next.js to manage routing and user interactions efficiently. Backend: Integrates Google Gemini AI capabilities for prompt engineering and task generation. Gesture Control: Utilizes the Leap Motion SDK for accurate gesture recognition. Future Developments Enhanced AI Integration: Further refinement of AI prompts to improve task generation accuracy and efficiency. Expanded Voice Command Features: Adding more voice functionalities for seamless kitchen assistance. User Feedback Mechanism: Implementing a system for users to provide feedback on tasks and improve the AI's learning process. Conclusion CHOP CHOP is designed to revolutionize the cooking experience for home cooks by combining gesture control, AI-driven task management, and voice commands. By addressing the challenges of multitasking and time management, we aim to enhance the overall quality of cooking and make meal preparation more enjoyable for everyone.

## README (from the GitHub repository)

# CHOP CHOP

## Overview

**CHOP CHOP** is a hands-free, AI-powered cooking assistant designed to help home cooks, students, and those with ADHD manage multiple tasks in the kitchen more efficiently. By utilizing voice commands and gesture controls, CHOP CHOP ensures a smooth cooking experience, eliminating the risk of overcooking or undercooking meals.

The project integrates multiple hardware and software components, including Leap Motion for gesture control, Google Home for voice assistance, and a web-based interface for managing cooking tasks.

## Problem Statement

Many people, especially college students and individuals with ADHD, struggle with multitasking and maintaining focus while cooking. This can lead to improperly cooked food and a less enjoyable cooking experience. CHOP CHOP solves this issue by providing an engaging, hands-free challenge that helps users stay organized and focused in the kitchen.

## Features

- **Voice Commands**: Using Google Home and the Google Assistant SDK, CHOP CHOP allows you to control tasks and timers via voice commands.
- **Gesture Control**: Leap Motion Hand Tracking Sensor lets you move through the task list, pause timers, or add extra time with hand gestures.
- **AI Task Generation**: Using Google Gemini, CHOP CHOP can generate detailed cooking instructions based on user input and optimize task coordination for an efficient cooking experience.
- **Dynamic Timer System**: Each cooking task includes a "Start" button that triggers a countdown timer. Users can fast-forward, pause, or adjust the timer dynamically.

## Components

### Hardware:
- **Leap Motion Hand Tracking Sensor**: Enables hands-free control via gestures.
- **Google Home Mini**: Allows for voice-activated control using the Google Assistant SDK.
- **Raspberry Pi**: Connects the Leap Motion sensor to the web application and controls external signals (e.g., LEDs or sounds).

### Software:
- **Google Assistant SDK**: Provides voice interaction for controlling tasks.
- **Google Gemini**: AI model used to generate structured task lists based on user input.
- **Next.js**: Web framework for building the front-end interface and managing routing.
- **UI Library**: Utilizes NextUI or similar libraries for simplified, responsive styling.

## Prompt Engineering for Task Generation

CHOP CHOP uses prompt engineering to generate detailed, step-by-step cooking instructions. 

### Example Input:
"I want to cook a medium-rare sirloin steak and stir-fry unchopped onions with chopped and washed broccoli."

### Example Output:
1. Heat the pan on medium heat for 2 minutes and chop the onions.
2. Season the steak with salt and pepper, and place it on the pan.
3. Flip the steak after 4 minutes, then stir-fry onions in a separate pan for 3 minutes.
4. Add broccoli to the stir-fry after 2 minutes and continue stirring for 3 more minutes.

The instructions are structured to optimize timing and minimize idle time.

## Gesture Controls

- **Swipe Gesture**: Move on to the next task or fast-forward.
- **Tap Gesture**: Add more time to the current task.
- **Clap Gesture**: Pause or stop the timer.

The Leap Motion SDK communicates with the Raspberry Pi to process these gestures and control the app.

## Next Steps

1. **Leap Motion Gesture Control**: Map gestures to actions in the app using the Leap Motion SDK.
2. **AI Task Generation**: Fine-tune prompts to produce efficient and detailed cooking instructions.
3. **Timer & Task Flow**: Build the task list UI, integrate timers, and add functionality to fast-forward or extend time.
4. **Google Home Integration**: Set up voice commands for starting tasks and controlling the cooking process.
5. **Hardware Integration**: Use Raspberry Pi to manage communication between Leap Motion, external signals, and the app.

## How to Run

1. Clone the repository.
   ```bash
   git clone https://github.com/drew-tom1/calhacks2024.git
   ```
2. Install dependencies.
   ```bash
   npm install
   ```
3. Set development + environment variables.
4. Run development server.
  ```bash
  npm run dev
  ```


## Detected evidence (automated analysis)

Indexed codebase: 15 recognized source files, 17 KB.
- CSS (language) — detected in the code
- Express (technology) — detected in the code
- Google Gemini (technology) — detected in the code
- JavaScript (language) — detected in the code
- Next.js (technology) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- Python (language) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (21 of 21)

```
.eslintrc.json
.gitignore
next.config.mjs
package.json
postcss.config.mjs
README.md
src/app/api/generateTask/route.js
src/app/api/generateTask/taskSchema.js
src/app/api/gesture/route.ts
src/app/api/voiceinput/route.ts
src/app/api/webhook/route.ts
src/app/globals.css
src/app/layout.tsx
src/app/page.tsx
src/components/generation/generateButton.js
src/components/generation/generationwrapper.js
src/components/generation/inputBox.js
src/components/hero/Hero.js
src/components/navbar/Navbar.jsx
tailwind.config.ts
tsconfig.json
```

### Dependencies

- package.json: @clerk/nextjs@^5.7.5, @google/generative-ai@^0.21.0, @nextui-org/react@^2.4.8, @tanstack/react-query@^5.59.15, @types/express@^5.0.0, @types/node@^20, @types/react@^18, @types/react-dom@^18, eslint@^8, eslint-config-next@14.2.15, express@^4.21.1, framer-motion@^11.11.9, next@14.2.15, postcss@^8, react@^18, react-dom@^18, tailwindcss@^3.4.1, ts-node@^10.9.2, typescript@^5.6.3

### Recent commits (newest first)

- Merge branch 'main' of github.com:tthacks24/chopchop
- Fixed structure of backend and added route for gemini call
- create webhook api
- hero
- moved generation for deployment
- readded generation endpoint, starting frontend
- fixed file type
- fix routes
- fix typo
- removed empty page.jsx
- removed ...gestureData for deployment
- deploy to vercel
- backend work, server running with basic gesture routes
- expanded on backend structure, working on sep packages and base
- Merge branch 'main' of https://github.com/drew-tom1/calhacks2024
- fix typo
- Merge pull request #2 from tthacks24/feature/landing-navbar
- Merge branch 'main' into feature/landing-navbar
- landing page commit
- Merge branch 'main' of https://github.com/drew-tom1/calhacks2024

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "name": "chopchop",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "@clerk/nextjs": "^5.7.5",
    "@google/generative-ai": "^0.21.0",
    "@nextui-org/react": "^2.4.8",
    "@tanstack/react-query": "^5.59.15",
    "express": "^4.21.1",
    "framer-motion": "^11.11.9",
    "next": "14.2.15",
    "react": "^18",
    "react-dom": "^18"
  },
  "devDependencies": {
    "@types/express": "^5.0.0",
    "@types/node": "^20",
    "@types/react": "^18",
    "@types/react-dom": "^18",
    "eslint": "^8",
    "eslint-config-next": "14.2.15",
    "postcss": "^8",
    "tailwindcss": "^3.4.1",
    "ts-node": "^10.9.2",
    "typescript": "^5.6.3"
  }
}

```

### src/app/page.tsx

```typescript
import Navbar from "@/components/navbar/Navbar";
import Hero from "@/components/hero/Hero";

export default function Home() {
  return (
    <div>
      <Navbar/>
      <Hero/>
    </div>
  );
}

```

### src/app/layout.tsx

```typescript
import type { Metadata } from "next";
import localFont from "next/font/local";
import "./globals.css";

const geistSans = localFont({
  src: "./fonts/GeistVF.woff",
  variable: "--font-geist-sans",
  weight: "100 900",
});
const geistMono = localFont({
  src: "./fonts/GeistMonoVF.woff",
  variable: "--font-geist-mono",
  weight: "100 900",
});

export const metadata: Metadata = {
  title: "Chop Chop!",
  description: "Made @ CalHacks 2024",
};

export default function RootLayout({
  children,
}: Readonly<{
  children: React.ReactNode;
}>) {
  return (
    <html lang="en">
      <body
        className={`${geistSans.variable} ${geistMono.variable} antialiased`}
      >
        {children}
      </body>
    </html>
  );
}

```

### src/app/api/webhook/route.ts

```typescript
import { NextRequest, NextResponse } from 'next/server';

// API route to handle IFTTT webhook requests
export async function POST(req: NextRequest) {
  try {
    const body = await req.json(); // Parse the JSON body from IFTTT
    const userInput = body.userInput; // Extract the user's input from the request

    // Log the user input for debugging
    console.log('IFTTT Webhook Received Input:', userInput);

    // Return a success response (IFTTT does not expect a response body)
    return NextResponse.json({ message: 'Webhook processed successfully' });
  } catch (error) {
    console.error('Error processing IFTTT webhook request:', error);
    return NextResponse.json(
      { message: 'Error processing request' },
      { status: 500 }
    );
  }
}

```

### src/app/api/voiceinput/route.ts

```typescript
import { NextRequest, NextResponse } from "next/server";

export async function POST(req: NextRequest) {
  try {
    // Parse JSON body using NextRequest's .json() method
    const { queryResult } = await req.json();
    const userInput = queryResult.queryText; // Extract the user's voice command

    // Log the user's input to the server console (terminal)
    console.log("User said:", userInput);

    // Respond to Google Assistant
    return NextResponse.json({
      fulfillmentText: "Got it!",
    });
  } catch (error) {
    console.error("Error processing request:", error);
    return NextResponse.json(
      { message: "Error processing request" },
      { status: 500 }
    );
  }
}

// Handle unsupported methods
export async function GET() {
  return NextResponse.json({ message: "Method Not Allowed" }, { status: 405 });
}

```

### src/app/api/gesture/route.ts

```typescript
import { NextRequest, NextResponse } from 'next/server';

// API route handler for gesture detection
export async function POST(req: NextRequest) {
  const body = await req.json(); // Parse JSON body
  const { gesture } = body;

  console.log('Gesture route hit');

  // Handle gesture types
  switch (gesture) {
    case 'swipe':
        console.log('Swipe gesture detected');
        return NextResponse.json({ message: 'Swipe detected!' });

    case 'open_hand':
        console.log('Open hand gesture detected');
        return NextResponse.json({ message: 'Open hand detected!' });

    case 'fist':
        console.log('Fist detected');
        return NextResponse.json({ message: 'Fist detected!' });

    default:
        console.log('Unknown gesture type');
        return NextResponse.json({ error: 'Unknown gesture type' }, { status: 400 });
  }
}

```

### src/app/api/generateTask/route.js

```javascript
import { GoogleGenerativeAI } from "@google/generative-ai";
import { taskSchema } from "./taskSchema";  // Adjust the path to your schema
console.log(process.env)
const GOOGLE_API_KEY = process.env.GOOGLE_API_KEY || "";

if (!GOOGLE_API_KEY) {
  console.error("Google API Key is missing");  // Log if API key is missing
}

const genAI = new GoogleGenerativeAI(GOOGLE_API_KEY);

console.log(GOOGLE_API_KEY + "!");
// Export the named `POST` function
export async function POST(req) {
  try {
    const { input } = await req.json();  // Extract the input from the request body

    console.log("Received input:", input);  // Log the received input

    if (!input) {
      console.error("No input provided");
      return new Response(JSON.stringify({ message: "Input is required" }), { status: 400 });
    }

    // Log that we're creating the model
    console.log("Creating model...");

    const model = genAI.getGenerativeModel({
      model: "gemini-1.5-flash",
      generationConfig: {
        responseMimeType: "application/json",
        responseSchema: taskSchema,
      },
    });

    console.log("Model created successfully");

    const result = await model.generateContent(input);
    console.log("Generated content result:", result);

    const responseText = await result.response.text();
    console.log("Response Text:", responseText);

    return new Response(JSON.stringify({ content: responseText }), { status: 200 });

  } catch (error) {
    console.error("Error in generating tasks:", error);  // Log the error
    return new Response(JSON.stringify({ message: "Error generating content", error: error.message }), { status: 500 });
  }
}

```

### tailwind.config.ts

```typescript
import type { Config } from "tailwindcss";

const config: Config = {
  content: [
    "./src/pages/**/*.{js,ts,jsx,tsx,mdx}",
    "./src/components/**/*.{js,ts,jsx,tsx,mdx}",
    "./src/app/**/*.{js,ts,jsx,tsx,mdx}",
  ],
  theme: {
    extend: {
      colors: {
        background: "var(--background)",
        foreground: "var(--foreground)",
      },
    },
  },
  plugins: [],
};
export default config;

```

### src/app/globals.css

```css
@tailwind base;
@tailwind components;
@tailwind utilities;

:root {
  --background: #ffffff;
  --foreground: #171717;
}

@media (prefers-color-scheme: dark) {
  :root {
    --background: #0a0a0a;
    --foreground: #ededed;
  }
}

body {
  color: var(--foreground);
  background: var(--background);
  font-family: Arial, Helvetica, sans-serif;
}

@layer utilities {
  .text-balance {
    text-wrap: balance;
  }
}

```

### src/components/generation/inputBox.js

```javascript
'use client';

import React from 'react';

const InputBox = ({ prompt, setPrompt }) => {
  return (
    <input
      type="text"
      className="border border-r-0 border-palette-light rounded-l-lg w-2/3 focus:outline-none focus:ring-1 focus:ring-palette-primary text-black text-center"
      required
      onChange={(e) => setPrompt(e.target.value)}
      placeholder="Type Your Prompt"
      value={prompt}  // Controlled input
    />
  );
};

export default InputBox;

```

[5 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]