# Project export: Remy This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: your wearble ai cooking assistant - Devpost: https://devpost.com/software/remy-qhcw4s - GitHub: https://github.com/fightingj305/remy-treehacks-2026 - Video: https://www.youtube.com/embed/ZlQDRcrbpDI?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Kailee Kaocharoen (18 commits), kelvin (14 commits), fightingj305 (10 commits) ## Devpost submission (written by the team) ### Overview Remy is a wearable AI cooking assistant that gives you real time feedback and instructions on anything you're learning to cook. ### Inspiration At this stage in our lives, some of us have grown up cooking, while others have never even microwaved a meal. Cooking is both a survival skill and a rewarding pastime (one that even the culinary illiterate aspire to master). While it’s unrealistic to have a master chef guiding you in the kitchen, we set out to create an AI sous-chef that does more than suggest personalized recipes. Remy walks you through each step, actively observes your progress, and provides real-time feedback as you cook. The name Remy comes from the Pixar rat who proved that anyone can cook. And with the right AI companion, we believe that’s more true than ever! ### What it does Remy is a multi-device system with three core capabilities: Recipe Discovery — A Next.js web dashboard where you describe what you're in the mood for ("something high-protein with chicken") and Claude AI generates three personalized recipe recommendations filtered by your dietary preferences (nut-free, keto, no spicy, etc.). Real-Time Kitchen Vision — A Raspberry Pi camera streams live video to an NVIDIA Jetson Orin Nano, which runs YOLOv8 object detection to identify ingredients and cooking activity, plus a Vision Language Model (Ollama qwen3-vl:2b) that analyzes the scene every few seconds and describes what it sees in natural language. Smart Cooking Control — When you select a recipe, the dashboard sends a structured task queue over TCP to an ESP32 microcontroller that can coordinate cooking hardware, while a DS18B20 temperature sensor monitors your cooking surface in real time. How We Built It Architecture With four devices, this project relies heavily upon effective communications protocols to function. We have one head-mounted raspberry pi which carries the camera along with an ESP32 combined with an amplifier, speaker, and microphone. This provides us with a flexible, compact package that gives the system a well-rounded sense of it surroundings. The heavy hitter of our hardware, the Jetson Orin Nano, handles all the edge compute required to give us relatively low latency for how much processing is being done. Connected serially to another base station raspberry pi, the Nano and the Pi coordinate the overall state of the cooking, and parse all the incoming communications from the peripheral devices. Tech Stack Recipe AI Pipeline When a user asks "prompts the system should I cook?", the system: Sends the query + dietary preferences to Claude (claude-sonnet-4-20250514) Claude returns 3 structured recommendations, each with a name, description, and an ordered recipeTaskQueue of cooking steps The dashboard fetches food images from the Pixabay API for each dish On selection, the task queue is serialized as JSON and sent over TCP with a 4-byte length header to the ESP32 Scene Analysis Pipeline We pass the video feed into the Jetson Nano every 7 seconds in order to generate a text description of what the user is seeing; we choose to do this on device due to the large latency from streaming actual video feeds, and to test out the edge capabilities of Nvidia's devices. This textual description is then appended to a large history buffer, which is used as scene analysis over time. This also allows us to identify when users have completed tasks in their recipes without them having to manually mark them, creating a more streamlined and enjoyable experience. Challenges We Ran Into Audio Quality. Streaming text-to-speech over a DAC into a small 8 ohm speaker brought a lot of unexpected challenges. Since the line-level DAC produces an output matched to higher impedance output devices, we scrambled to find an amplifier that could produce help the output signal match the speaker. Furthermore, since live audio is always a difficult proposition over unreliable internet, we had to implement ring buffers and anti-jitter logic on both ends of our audio pipeline to keep the sound relatively smooth. Hardware Failure. At 4am on Sunday, our Arducam camera gave out on us and left us with no working video feed and five hours of hacking left to go just as we were beginning to integrate all the components of the project. Luckily, we substituted it with a Logitech Brio Camera that we had on hand, and it worked right out of the box! ### Accomplishments we're proud of We're firstly so happy to have built such a complex hardware hack. It wasn't easy interfacing between 2 Raspberry Pi's, a ESP32, and a Jetson Nano through both wireless and wired means, and we learned a lot through it. We're also proud to have learned so much on Nvidia's edge compute devices, and we've seen first hand just how useful they can be for low-latency use cases like live video analysis. "Anyone can cook." — Auguste Gusteau ### What's next We want to incorporate more sensors in order to provide more context to the model. We also would like to have even more powerful edge compute, since image models are large and slow to run easily. Finally, we'd love to test our devices out to real people who are learning how to cook! ## README (from the GitHub repository) # RPi Video Streamer Stream live video from one Raspberry Pi (with camera) to another RPi (base station) over WiFi/TCP. ## Setup ### Both Pis ```bash sudo apt-get update && sudo apt-get install -y python3-opencv python3-numpy python3-picamera2 ``` > The base station doesn't strictly need `python3-picamera2`, but it won't hurt to install it. ## Usage ### 1. Start the Base Station (receiver Pi) ```bash python3 receiver.py --port 9000 ``` This opens a listening socket and waits for the camera Pi to connect. ### 2. Start the Camera Pi (sender Pi) ```bash python3 sender.py --host --port 9000 ``` Replace `` with the base station's IP address. To find it, run `hostname -I` on the base station Pi. ### 3. Quit Press `q` in the receiver's display window. ## Optional Flags (sender only) | Flag | Default | Description | |------------|---------|-------------------| | `--width` | 640 | Frame width (px) | | `--height` | 480 | Frame height (px) | | `--fps` | 30 | Target framerate | Example at higher resolution: ```bash python3 sender.py --host 192.168.1.50 --port 9000 --width 1280 --height 720 --fps 24 ``` ## Finding IP Addresses On each Pi, run: ```bash hostname -I ``` ## Troubleshooting - **"Connection refused"** — Make sure the receiver is started first. - **No display window** — The receiver needs a display (monitor, VNC, or X forwarding). - **Camera not detected** — Run `rpicam-hello` to verify the camera works. ## Detected evidence (automated analysis) Indexed codebase: 31 recognized source files, 202 KB. - Anthropic (technology) — detected in the code - C (language) — detected in the code - CSS (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Vercel (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (44 of 44) ``` .gitignore apriltag_deskew.py audio_processing/esp32_stt.py audio_processing/tts_to_esp32.py CLAUDE.md dac_test/dac_test.ino dac_test/pcm5102a.h dac_test/stream_to_dac.py dac_test/udp_echo/basic_udp.py dac_test/udp_echo/udp_echo.ino dashboard/.gitignore dashboard/app/api/chat/route.ts dashboard/app/api/send-task/route.ts dashboard/app/globals.css dashboard/app/layout.tsx dashboard/app/page.tsx dashboard/components/CookingInput.tsx dashboard/components/PreferencesModal.tsx dashboard/components/RecommendationCard.tsx dashboard/components/RecommendationsGrid.tsx dashboard/components/Toast.tsx dashboard/eslint.config.mjs dashboard/next.config.ts dashboard/package.json dashboard/postcss.config.mjs dashboard/README.md dashboard/tsconfig.json ds18b20.py esp32_audio_wifi_bt/esp32_audio_wifi_bt.ino esp32_audio_wifi_only/esp32_audio_wifi_only.ino esp32_audio_wifi_only/receive_mic_audio.py jetson_processor.py mic_test/mic_audio.py mic_test/mic_bluetooth/mic_bluetooth.ino mic_test/mic_test.ino README.md receiver_deskew.py receiver_jetson_full.py receiver_jetson.py receiver.py RUN_PIPELINE.md sender.py test_recipe_send.py thermistor_test/thermistor_test.ino ``` ### Dependencies - dashboard/package.json: @anthropic-ai/sdk@^0.74.0, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, eslint@^9, eslint-config-next@16.1.6, next@16.1.6, react@19.2.3, react-dom@19.2.3, tailwindcss@^4, typescript@^5 ### Recent commits (newest first) - Merge branch 'main' of https://github.com/fightingj305/remy-treehacks-2026 - fixed images - final - API rewiring - s added on line 87 - username commented out - Interface update - Merge branch 'main' of https://github.com/fightingj305/remy-treehacks-2026 - New send - ip address update - Merge branch 'main' of https://github.com/fightingj305/remy-treehacks-2026 - removed interleaving - recipe receipt draft - Merge branch 'main' of https://github.com/fightingj305/remy-treehacks-2026 - added support for brio - Merge branch 'main' of https://github.com/fightingj305/remy-treehacks-2026 - adde dreceiver jsetson full - full end to end - minor - error toast when packet doesn't send correctly ## Key source files (fetched from GitHub, selected and truncated for size) ### RUN_PIPELINE.md ```markdown # Running the Full Pipeline This guide walks through starting the complete Camera Pi → Base Station Pi → Jetson Orin Nano pipeline. ## Network Setup All three devices must be on the same network (or have direct routes to each other). You will need to know: - **Camera Pi IP** — the RPi with the camera module - **Base Station Pi IP** — the RPi with the display (runs the GUI) - **Jetson IP** — the Jetson Orin Nano (default `192.168.55.1` if using USB device mode) ### Port summary | Port | From → To | Content | |------|-----------|---------| | 9000 | Camera Pi → Base Station | Raw JPEG frames | | 9001 | Base Station → Jetson | Forwarded JPEG frames | | 9002 | Jetson → Base Station | Annotated JPEG frames | | 9003 | Jetson → Base Station | VLM analysis text | Make sure these ports are not blocked by any firewalls on the devices. --- ## Prerequisites ### Camera Pi ```bash sudo apt-get install -y python3-picamera2 ``` Verify the camera is detected: ```bash libcamera-hello --list-cameras ``` ### Base Station Pi ```bash sudo apt-get install -y python3-opencv python3-numpy python3-pil python3-pil.imagetk python3-tk ``` ### Jetson Orin Nano ```bash # Core dependencies sudo apt-get install -y python3-opencv python3-numpy # Ollama (required for VLM analysis) curl -fsSL https://ollama.com/install.sh | sh ollama pull qwen3-vl:2b # YOLOv8 (only if you plan to use --yolo) pip3 install ultralytics ``` --- ## Step-by-step startup Start the devices in this order: **Jetson → Base Station → Camera Pi**. ### Step 1: Start Ollama on the Jetson SSH into the Jetson and make sure Ollama is running: ```bash # Check if Ollama is already running ollama list # If not running, start it ollama serve & ``` Confirm the VLM model is available: ```bash ollama list # Should show: qwen3-vl:2b ``` ### Step 2: Start the Jetson processor On the Jetson, from the repo directory: ```bash # VLM-only mode (default — no YOLO, just passthrough + VLM analysis) python3 jetson_processor.py # With YOLOv8 object detection enabled python3 jetson_processor.py --yolo # With custom settings python3 jetson_processor.py --yolo --vlm-interval 10 --conf 0.3 ``` You should see: ``` YOLO disabled (passthrough mode). Use --yolo to enable detection. Jetson processor listening on UDP port 9001 ... Will return processed frames on port 9002 VLM analysis log: vlm_logs/vlm_analysis_YYYYMMDD_HHMMSS.log ``` **Jetson processor flags:** | Flag | Default | Description | |------|---------|-------------| | `--port` | 9001 | UDP port to receive frames | | `--return-port` | 9002 | UDP port to send processed frames back | | `--reply-host` | auto-detect | Override reply IP address | | `--jpeg-quality` | 85 | JPEG encode quality for output | | `--yolo` | off | Enable YOLOv8 TensorRT detection | | `--model` | yolov8n.pt | YOLOv8 model path (only with `--yolo`) | | `--conf` | 0.25 | Detection confidence threshold (only with `--yolo`) | | `--vlm-port` | 9003 | UDP port for VLM analysis text out [truncated — 3600 more characters] ``` ### CLAUDE.md ```markdown # CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Project Overview RPi Video Streamer + Jetson AI Pipeline + Voice AI Kitchen Assistant — streams live MJPEG video from a Raspberry Pi camera (`sender.py`) to a base station RPi which forwards frames to a Jetson Orin Nano (`jetson_processor.py`) for AI processing. The Jetson runs optional YOLOv8 object detection and periodic Ollama VLM scene analysis, streaming results back to the base station GUI. An ESP32 with microphone and speaker provides a voice interface: the user asks cooking questions, which are transcribed (ElevenLabs STT), answered by Claude (using VLM scene context), and spoken back (ElevenLabs TTS). Built for TreeHacks 2026. ## Architecture Three devices communicating over UDP: - **sender.py** (Camera Pi): Uses `picamera2` to capture video, encodes as MJPEG, sends each JPEG frame over UDP with a 4-byte big-endian length header. - **receiver_jetson.py** (Base Station Pi): Receives camera frames, displays them, forwards to Jetson, receives processed frames back, and displays both side-by-side in a Tkinter GUI. Also receives VLM analysis text and shows it in a scrollable log widget. (Video only — see `receiver_jetson_full.py` for the unified version with voice AI.) - **receiver_jetson_full.py** (Base Station Pi): Unified version of `receiver_jetson.py` that adds the voice AI pipeline. Receives ESP32 mic audio, runs Silero VAD, transcribes via ElevenLabs STT, queries Claude with VLM context, generates TTS, and streams audio back to ESP32. - **jetson_processor.py** (Jetson Orin Nano): Receives forwarded frames, optionally runs YOLOv8n TensorRT detection, returns annotated frames. Periodically sends frames to local Ollama `qwen3-vl:2b` VLM for scene analysis and streams results back via UDP. ### Port allocation | Port | Direction | Content | |------|-----------|---------| | 9000 | Camera Pi → Base Station | Raw JPEG frames (UDP) | | 9001 | Base Station → Jetson | Forwarded JPEG frames (UDP) | | 9002 | Jetson → Base Station | Annotated JPEG frames (UDP) | | 9003 | Jetson → Base Station | VLM analysis text (plain UTF-8, UDP) | | 12345 | ESP32 → Base Station | Raw mic audio (16-bit mono PCM, 44.1kHz, UDP) | | 12345 | Base Station → ESP32 | TTS playback audio (16-bit stereo PCM, 44.1kHz, UDP) | ### Wire protocol `[4-byte big-endian uint32 frame length][JPEG frame bytes]` per frame on ports 9000/9001/9002. Port 9003 sends plain UTF-8 text datagrams (no length header). ### Threading model (Jetson) ``` Main thread (full frame rate) VLM daemon thread (every ~5s) ───────────────────────────── ──────────────────────────── recv UDP:9001 jpeg = queue.get() optional YOLO detect + annotate result = query_ollama(jpeg) encode + send UDP:9002 log to file + print every 5s: queue.put_nowait(jpeg) sendto UDP:9003 ``` The VLM thread has zero impact on the main loop — `queue. [truncated — 5205 more characters] ``` ### dashboard/package.json ``` { "name": "dashboard", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint" }, "dependencies": { "@anthropic-ai/sdk": "^0.74.0", "next": "16.1.6", "react": "19.2.3", "react-dom": "19.2.3" }, "devDependencies": { "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "16.1.6", "tailwindcss": "^4", "typescript": "^5" } } ``` ### dashboard/app/layout.tsx ```typescript import type { Metadata } from "next"; import { DM_Sans } from "next/font/google"; import "./globals.css"; const dmSans = DM_Sans({ variable: "--font-dm-sans", subsets: ["latin"], weight: ["400", "500", "600", "700"], }); export const metadata: Metadata = { title: "Dashboard", description: "Cooking insights dashboard", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### dashboard/app/page.tsx ```typescript 'use client'; import { useState } from 'react'; import PreferencesModal from '@/components/PreferencesModal'; import CookingInput from '@/components/CookingInput'; import RecommendationsGrid, { Recommendation } from '@/components/RecommendationsGrid'; import Toast from '@/components/Toast'; export default function Dashboard() { const [isPreferencesModalOpen, setIsPreferencesModalOpen] = useState(false); const [userPreferences, setUserPreferences] = useState([ 'Nut Free', 'Keto', 'No Spicy' ]); const [recommendations, setRecommendations] = useState([]); const [isLoading, setIsLoading] = useState(false); const [aiMessage, setAiMessage] = useState(''); const [error, setError] = useState(null); const [isDarkMode, setIsDarkMode] = useState(false); const [playingCardId, setPlayingCardId] = useState(null); const [toastMessage, setToastMessage] = useState(null); const handleSavePreferences = (preferences: string[]) => { setUserPreferences(preferences); console.log('Saved preferences:', preferences); }; const handleSubmit = async (userInput: string) => { setIsLoading(true); setError(null); try { const response = await fetch('/api/chat', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ message: userInput, preferences: userPreferences, }), }); if (!response.ok) { const errorData = await response.json().catch(() => ({})); throw new Error(errorData.error || `Server error: ${response.status}`); } const data = await response.json(); if (data.recommendations && data.recommendations.length > 0) { setRecommendations( data.recommendations.map((rec: any, index: number) => ({ id: index + 1, name: rec.name, imageUrl: rec.imageUrl || '/images/dishes/placeholder.png', description: rec.description, recipeTaskQueue: Array.isArray(rec.recipeTaskQueue) ? rec.recipeTaskQueue : [] })) ); } if (data.message) { setAiMessage(data.message); } } catch (error) { console.error('Error getting recommendations:', error); setError(error instanceof Error ? error.message : 'Failed to get recommendations. Please try again.'); } finally { setIsLoading(false); } }; const handleCook = (item: Recommendation) => { console.log(`Sending recipe to 172.20.10.13:8080 →`, item.recipeTaskQueue); // Construct the payload to match your server's expected schema const payload = { message: `Cooking ${item.name}`, recommendations: [ { name: item.name, imageUrl: item.imageUrl, description: item.description, recipeTaskQueue: item.recipeTaskQueue, // This is what the server extracts } ] }; fetch('/api/send-task', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(payload), }) .then((res) => { if (!res.ok) throw new Error(`HTTP ${res.status}`); return res.json(); }) .then((data) => { setToastMessage(`Sent recipe: ${item.name}`); }) .catch((err) => { console.error('Send error:', err); setToastMessage(`Failed to send recipe: ${err.message}`); }); }; return (
{/* Logo */}
Logo { e.currentTarget.style.filter = 'drop-shadow(0 0 20px rgba(175, 67, 29, 0.8))'; }} onMouseLeave={(e) => { e.currentTarget.style.filter = 'drop-shadow(0 0 0px rgba(175, 67, 29, 0))'; }} onClick={() => setIsDarkMode(!isDarkMode)} />
{/* Greeting */} {/*

HELLO ALLEN

*/}

What do you wish to cook?

setIsPreferencesModalOpen(true)} /> setPlayingCardId(playingCardId === id ? null : id)} onCook={handleCook} />
{toastMessage && ( setToastMessage(null)} /> )} setIsPreferencesModalOpen(false)} onSave={handleSavePreferences} initialPreferences={userPreferences} isDarkMode={i [truncated — 38 more characters] ``` ### dashboard/app/api/send-task/route.ts ```typescript import { NextResponse } from 'next/server'; export async function POST(request: Request) { const body = await request.json(); try { const response = await fetch('http://172.20.10.13:8080/api/chat', { method: 'POST', headers: { 'Content-Type': 'text/plain' }, body: JSON.stringify(body), }); const data = await response.json(); return NextResponse.json(data); } catch (error) { return NextResponse.json({ error: 'Failed to reach API' }, { status: 500 }); } } ``` ### dashboard/app/api/chat/route.ts ```typescript import Anthropic from '@anthropic-ai/sdk'; import { NextRequest, NextResponse } from 'next/server'; const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY, }); const PIXABAY_API_KEY = process.env.PIXABAY_API_KEY; const PLACEHOLDER_IMAGE = '/images/dishes/placeholder.png'; interface PixabayHit { id: number; webformatURL: string; largeImageURL: string; previewURL: string; } interface PixabayResponse { total: number; totalHits: number; hits: PixabayHit[]; } async function fetchFoodImage(mealName: string): Promise { if (!PIXABAY_API_KEY) { console.warn('PIXABAY_API_KEY not set, using placeholder image'); return PLACEHOLDER_IMAGE; } try { const url = new URL('https://pixabay.com/api/'); url.searchParams.set('key', PIXABAY_API_KEY); url.searchParams.set('q', mealName); url.searchParams.set('image_type', 'photo'); url.searchParams.set('category', 'food'); url.searchParams.set('per_page', '3'); const response = await fetch(url.toString()); if (!response.ok) { console.error(`Pixabay API error: ${response.status}`); return PLACEHOLDER_IMAGE; } const data: PixabayResponse = await response.json(); if (data.hits && data.hits.length > 0) { return data.hits[0].webformatURL; } return PLACEHOLDER_IMAGE; } catch (error) { console.error('Error fetching image from Pixabay:', error); return PLACEHOLDER_IMAGE; } } export async function POST(request: NextRequest) { try { const { message, preferences } = await request.json(); if (!message) { return NextResponse.json( { error: 'Message is required' }, { status: 400 } ); } // Build system prompt with user preferences const preferencesText = preferences && preferences.length > 0 ? `The user has the following dietary preferences and restrictions: ${preferences.join(', ')}.` : 'The user has no specific dietary preferences.'; const systemPrompt = `You are a helpful cooking assistant. ${preferencesText} When the user asks what to cook or requests meal recommendations, provide 3 specific meal suggestions that match their preferences and request. Return a JSON object that matches this exact schema: { "recommendations": [ { "name": string, "description": string, "imageUrl": string, "recipeTaskQueue": string[] } ], "message": string } "name" is the meal name, "description" is a brief description of the dish, "imageUrl" will be in the form "/images/dishes/placeholder.png", "recipeTaskQueue" recipeTaskQueue must be an ordered array of short, clear cooking steps. Each step should be a single actionable instruction. "message" is a friendly response to the user's request } Make sure the meal names are specific and appealing. Consider the user's preferences when making recommendations. You must return ONLY valid JSON. Do not include explanations, markdown, or text outside the JSON object. `; const response = await anthropic.messages.create({ model: 'claude-sonnet-4-20250514', max_tokens: 1024, system: systemPrompt, messages: [ { role: 'user', content: message, }, ], }); // Extract the text content from Claude's response const textContent = response.content[0]; if (textContent.type !== 'text') { throw new Error('Unexpected response format from Claude'); } // Parse the JSON response from Claude let parsedResponse; try { // Try to extract JSON from the response const jsonMatch = textContent.text.match(/\{[\s\S]*\}/); if (jsonMatch) { parsedResponse = JSON.parse(jsonMatch[0]); } else { // Fallback if Claude doesn't return JSON parsedResponse = { recommendations: [], message: textContent.text, }; } } catch (parseError) { // If parsing fails, return a default response parsedResponse = { recommendations: [], message: textContent.text, }; } // Fetch images from Pixabay for each recommendation if (parsedResponse.recommendations && parsedResponse.recommendations.length > 0) { const recommendationsWithImages = await Promise.all( parsedResponse.recommendations.map(async (rec: any) => { const imageUrl = await fetchFoodImage(rec.name); return { ...rec, imageUrl, }; }) ); parsedResponse.recommendations = recommendationsWithImages; } return NextResponse.json(parsedResponse); } catch (error) { console.error('Error calling Anthropic API:', error); return NextResponse.json( { error: 'Failed to get recommendations' }, { status: 500 } ); } } ``` ### receiver.py ```python """ receiver.py — Runs on the Base Station RPi. Listens for incoming UDP datagrams from the camera Pi and displays the received MJPEG stream. Usage: python3 receiver.py --port 9000 """ import argparse import socket import struct import cv2 import numpy as np MAX_UDP_RECV = 65535 def start_receiver(port): sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 4 * 1024 * 1024) sock.bind(("0.0.0.0", port)) print(f"Base station listening on UDP port {port} ...") # Set up fullscreen window for DSI display window_name = "Base Station - Live Feed" cv2.namedWindow(window_name, cv2.WINDOW_NORMAL) cv2.setWindowProperty(window_name, cv2.WND_PROP_FULLSCREEN, cv2.WINDOW_FULLSCREEN) connected = False while True: try: data, addr = sock.recvfrom(MAX_UDP_RECV) except OSError: continue if not connected: print(f"Receiving from {addr}") connected = True if len(data) < 4: continue frame_len = struct.unpack(">I", data[:4])[0] jpeg_data = data[4:] if len(jpeg_data) != frame_len: continue frame = cv2.imdecode( np.frombuffer(jpeg_data, dtype=np.uint8), cv2.IMREAD_COLOR ) if frame is not None: cv2.imshow(window_name, frame) if cv2.waitKey(1) & 0xFF == ord("q"): print("Quitting ...") sock.close() cv2.destroyAllWindows() return def main(): parser = argparse.ArgumentParser(description="RPi Base Station (receiver)") parser.add_argument("--port", type=int, default=9000, help="Port (default 9000)") args = parser.parse_args() start_receiver(args.port) if __name__ == "__main__": main() ``` ### ds18b20.py ```python #!/usr/bin/env python3 """DS18B20 temperature sensor reader via 1-Wire kernel interface on BCM GPIO 17 (physical pin 11).""" import glob import os import subprocess import sys import time W1_DEVICES_PATH = "/sys/bus/w1/devices/" DS18B20_PREFIX = "28-" GPIO_PIN = 17 # Physical pin 11 = BCM GPIO 17 def setup_kernel_modules(): """Load w1-gpio and w1-therm kernel modules and enable the dtoverlay.""" try: subprocess.run( ["sudo", "dtoverlay", "w1-gpio", f"gpiopin={GPIO_PIN}"], check=True, ) except subprocess.CalledProcessError: print("Warning: dtoverlay command failed. Ensure w1-gpio overlay is enabled in /boot/config.txt:") print(f" dtoverlay=w1-gpio,gpiopin={GPIO_PIN}") for module in ("w1-gpio", "w1-therm"): subprocess.run(["sudo", "modprobe", module], check=True) # Give the kernel a moment to discover devices time.sleep(1) def find_sensor(): """Find the first DS18B20 device directory.""" devices = glob.glob(os.path.join(W1_DEVICES_PATH, DS18B20_PREFIX + "*")) if not devices: return None return devices[0] def read_temperature(device_path): """Read temperature in Celsius from the sensor's sysfs file. Returns the temperature as a float, or None on read failure. """ slave_file = os.path.join(device_path, "w1_slave") with open(slave_file, "r") as f: lines = f.readlines() # First line ends with YES if the CRC check passed if len(lines) < 2 or "YES" not in lines[0]: return None # Second line contains t= idx = lines[1].find("t=") if idx == -1: return None raw = int(lines[1][idx + 2:]) return raw / 1000.0 def main(): print(f"Setting up 1-Wire on BCM GPIO {GPIO_PIN}...") setup_kernel_modules() sensor = find_sensor() if sensor is None: print("No DS18B20 sensor found. Check wiring and ensure the data pin is on physical pin 11 (BCM 17) with a 4.7kΩ pull-up resistor.") sys.exit(1) sensor_id = os.path.basename(sensor) print(f"Found sensor: {sensor_id}") try: while True: temp = read_temperature(sensor) if temp is not None: print(f"{temp:.1f} °C / {temp * 9 / 5 + 32:.1f} °F") else: print("CRC error, retrying...") time.sleep(1) except KeyboardInterrupt: print("\nStopped.") if __name__ == "__main__": main() ``` ### test_recipe_send.py ```python #!/usr/bin/env python3 """ Test script to send recipe steps to receiver_jetson_full.py Usage: python3 test_recipe_send.py [host] [port] Example: python3 test_recipe_send.py localhost 9005 python3 test_recipe_send.py 100.71.232.77 9005 """ import json import socket import struct import sys def send_recipe(host, port, recipe_steps): """Send recipe steps to the receiver using length-prefixed TCP protocol.""" print(f"Connecting to {host}:{port}...") sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.settimeout(5.0) try: sock.connect((host, port)) print(f"Connected to {host}:{port}") # Encode recipe steps as JSON payload = json.dumps(recipe_steps).encode('utf-8') print(f"Sending {len(recipe_steps)} steps ({len(payload)} bytes)") # Create 4-byte big-endian length header header = struct.pack('>I', len(payload)) # Send header + payload sock.sendall(header + payload) print("Data sent successfully") # Shutdown write half of connection sock.shutdown(socket.SHUT_WR) print("Connection closed") return True except socket.timeout: print("ERROR: Connection timed out") return False except ConnectionRefusedError: print("ERROR: Connection refused. Is the receiver running?") return False except Exception as e: print(f"ERROR: {e}") return False finally: sock.close() def main(): # Parse arguments host = sys.argv[1] if len(sys.argv) > 1 else "localhost" port = int(sys.argv[2]) if len(sys.argv) > 2 else 9005 # Test recipe recipe_steps = [ "Preheat oven to 350°F (175°C)", "Mix 2 cups flour, 1 cup sugar, and 1/2 tsp salt in a bowl", "Add 3 eggs and 1/2 cup milk, mix until smooth", "Pour batter into greased 9x13 pan", "Bake for 25-30 minutes until golden brown", "Let cool for 10 minutes before serving" ] print("=" * 60) print("Recipe TCP Sender Test") print("=" * 60) print(f"Target: {host}:{port}") print(f"Recipe steps to send:") for i, step in enumerate(recipe_steps, 1): print(f" {i}. {step}") print("=" * 60) print() success = send_recipe(host, port, recipe_steps) if success: print("\n✓ Recipe sent successfully!") return 0 else: print("\n✗ Failed to send recipe") return 1 if __name__ == "__main__": sys.exit(main()) ``` [19 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]