# Project export: Orderly This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Orderly is an AI-powered drive-through assistant that lets customers place and manage orders through natural conversation. - Devpost: https://devpost.com/software/orderly - GitHub: https://github.com/JJJasperl/CalHack11.0 - Team: 3 GitHub contributor(s) — Jie (Jasper) Liu (13 commits), YunzheLi (11 commits), Jason Ji (3 commits) ## Devpost submission (written by the team) ### Inspiration Orderly is an AI-powered solution designed to revolutionize the drive-thru experience, addressing the high-demand environment where 60% to 70% of fast food sales occur. By leveraging advanced voice technology and Large Language Model (LLM) agents, Orderly streamlines the ordering process, reducing human error, cutting operational costs, and improving transaction efficiency. It offers smooth, human-like interactions that ease the burden on drive-thru operators while enhancing customer satisfaction. This innovation promises to transform drive-thru operations, creating a faster, more accurate, and user-friendly experience that benefits both restaurant owners and customers. From Orderly, you, no matter you are a customer or an owner, will get: Smooth transaction drive-thru experience Lower employment cost Higher ordering efficiency ### What it does Orderly is an AI-powered voice assistant that allows customers to place, modify, and manage their drive-through orders using natural conversation. It handles everything from adding items to the cart to clearing orders when needed, all through hands-free, real-time voice interaction. The system can process multiple items, handle special requests, and ensure orders are accurate before submission. ### How we built it We built Orderly using a combination of modern web technologies and AI services. The frontend is a React-based interface that uses Socket.IO for real-time communication. On the backend, we used Flask with Socket.IO to handle voice data and order processing. For speech recognition and natural language processing, we integrated third-party APIs like Deepgram to transcribe audio and generate responses. The cart and order management system is powered by a Flask-based API that manages the state of the customer's order. ### Challenges we ran into One of the biggest challenges was ensuring accurate speech recognition, especially in noisy environments like drive-throughs. Handling complex or ambiguous customer requests also required fine-tuning the natural language processing and conversation flow. Additionally, ensuring smooth real-time communication between the frontend and backend, while processing voice data efficiently, was a technical hurdle we had to address. ### Accomplishments we're proud of We’re proud of successfully creating a voice-driven system that can handle real-time order management with high accuracy. The seamless integration between voice input, order processing, and cart management is a major achievement. Additionally, we were able to implement features like order modifications and clearing the cart, which bring real-world value to the drive-through experience. ### What we learned We learned a lot about the challenges of working with voice recognition in noisy environments and how to optimize AI models for better accuracy. We also gained valuable experience in real-time communication technologies, including Socket.IO, and how to build scalable, responsive APIs. Most importantly, we learned the importance of designing intuitive, user-friendly systems for end-users who may not be tech-savvy. Moving forward, we plan to improve Orderly by incorporating more advanced AI features, such as personalized order recommendations based on previous history and dynamic upselling. We also aim to refine the speech recognition system to handle accents and dialects better. Expanding Orderly to integrate with popular POS systems and testing it in real-world drive-through environments are also on our roadmap. ## README (from the GitHub repository) # CalHack11.0 Orderly is an AI-powered solution designed to revolutionize the drive-thru experience, addressing the high-demand environment where 60% to 70% of fast food sales occur. By leveraging advanced voice technology and Large Language Model (LLM) agents, Orderly streamlines the ordering process, reducing human error, cutting operational costs, and improving transaction efficiency. It offers smooth, human-like interactions that ease the burden on drive-thru operators while enhancing customer satisfaction. This innovation promises to transform drive-thru operations, creating a faster, more accurate, and user-friendly experience that benefits both restaurant owners and customers. From Orderly, you, no matter you are a customer or an owner, will get: * Smooth transaction drive-thru experience * Lower employment cost * Higher ordering efficiency [![Watch the video](./demo.png)](./Orderly.mp4) # Drive-Through Voice Ordering System This project is a proof-of-concept for using an AI voice agent to help place food orders in a drive-through store. It consists of two main components: 1. **Front-End**: A React app for the user interface. 2. **Back-End**: A Flask app that handles voice transcription, order processing, and database interaction. --- ## Prerequisites Before setting up the project, ensure you have the following installed: - [Node.js](https://nodejs.org/) (for React front-end) - [Python 3.x](https://www.python.org/) (for Flask back-end) - [Git](https://git-scm.com/) (for version control) --- ## Setup Instructions ### 1. Clone the Repository First, clone the repository to your local machine: ```bash git clone cd ## Detected evidence (automated analysis) Indexed codebase: 26 recognized source files, 50 KB. - CSS (language) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code ## Codebase structure (from repository index) ### Files (36 of 36) ``` .gitignore Assets/Data/menu.csv Assets/Data/selectedColumn.csv drive-through-voice-order/.gitignore drive-through-voice-order/package.json drive-through-voice-order/public/audio-processor.js drive-through-voice-order/public/index.html drive-through-voice-order/public/manifest.json drive-through-voice-order/public/robots.txt drive-through-voice-order/README.md drive-through-voice-order/src/App.css drive-through-voice-order/src/App.js drive-through-voice-order/src/App.test.js drive-through-voice-order/src/components/Cart.css drive-through-voice-order/src/components/Cart.js drive-through-voice-order/src/components/ChatBox.css drive-through-voice-order/src/components/ChatBox.js drive-through-voice-order/src/index.css drive-through-voice-order/src/index.js drive-through-voice-order/src/reportWebVitals.js drive-through-voice-order/src/setupTests.js flask-backend/.gitignore flask-backend/app.py flask-backend/autogen_model/__init__.py flask-backend/autogen_model/menu_query.py flask-backend/autogen_model/model.py flask-backend/autogen_model/prompt.py flask-backend/autogen_model/test_case.txt flask-backend/autogen_model/util.py flask-backend/cart/__init__.py flask-backend/cart/cart_handler.py flask-backend/deepgram/__init__.py flask-backend/deepgram/deepgram_agent.py flask-backend/requirements.txt flask-backend/test.py README.md ``` ### Dependencies - drive-through-voice-order/package.json: @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, axios@^1.7.7, react@^18.3.1, react-dom@^18.3.1, react-scripts@5.0.1, reconnecting-websocket@^4.4.0, socket.io-client@^4.8.0, web-vitals@^2.1.4 - flask-backend/requirements.txt: altgraph@==0.17.4, annotated-types@==0.7.0, anyio@==4.4.0, autogen@==0.3.0, bidict@==0.23.1, blinker@==1.8.2, certifi@==2024.8.30, cffi@==1.17.1, charset-normalizer@==3.3.2, click@==8.1.7, colorama@==0.4.6, cryptography@==43.0.1, Cython@==3.0.11, diskcache@==5.6.3, distro@==1.9.0, docker@==7.1.0, FLAML@==2.2.0, Flask@==3.0.0, Flask-SocketIO@==5.4.1, h11@==0.14.0, httpcore@==1.0.5, httpx@==0.27.2, idna@==3.8, itsdangerous@==2.2.0, Jinja2@==3.1.4, jiter@==0.5.0, joblib@==1.4.2, MarkupSafe@==3.0.2, numpy@==1.26.4, openai@==1.44.1, packaging@==24.1, pandas@==2.2.3, pefile@==2024.8.26, pyarmor@==8.5.11, pyarmor.cli.core@==6.5.3, pycparser@==2.22, pydantic@==2.9.1, pydantic_core@==2.23.3, pyinstaller@==6.10.0, pyinstaller-hooks-contrib@==2024.8, python-dateutil@==2.9.0.post0, python-dotenv@==1.0.1, python-engineio@==4.10.1, python-socketio@==5.11.4, pytz@==2024.2, pywin32@==307, pywin32-ctypes@==0.2.3, regex@==2024.7.24, requests@==2.32.3, scikit-learn@==1.5.2, scipy@==1.14.1, setuptools@==75.1.0, simple-websocket@==1.1.0, six@==1.16.0, sniffio@==1.3.1, termcolor@==2.4.0, threadpoolctl@==3.5.0, tiktoken@==0.7.0, tqdm@==4.66.5, typing_extensions@==4.12.2, tzdata@==2024.1, urllib3@==2.2.2, Werkzeug@==3.0.4, wheel@==0.44.0, wsproto@==1.2.0 ### Recent commits (newest first) - Merge branch 'main' of https://github.com/JJJasperl/CalHack11.0 - Add demo image. - Update video link in README.md - Update README.md with embedded demo video. - Merge branch 'main' of https://github.com/JJJasperl/CalHack11.0 - Upload demo video. - Fixed deepgram bug - Added negative quantitiy feature - Complete add cart. - Update requirements.txt - Make demo run. - Added model name - Merge branch 'main' of https://github.com/JJJasperl/CalHack11.0 - Requirement issue for autogen - added test.py - Merge pull request #2 from JJJasperl/full-stack - Merge pull request #1 from JJJasperl/dataFetch - Refactor autogen into backend. - added frontend conversation ui - Merge AutoGen to main. ## Key source files (fetched from GitHub, selected and truncated for size) ### drive-through-voice-order/package.json ``` { "name": "drive-through-voice-order", "version": "0.1.0", "private": true, "dependencies": { "@testing-library/jest-dom": "^5.17.0", "@testing-library/react": "^13.4.0", "@testing-library/user-event": "^13.5.0", "axios": "^1.7.7", "react": "^18.3.1", "react-dom": "^18.3.1", "react-scripts": "5.0.1", "reconnecting-websocket": "^4.4.0", "socket.io-client": "^4.8.0", "web-vitals": "^2.1.4" }, "scripts": { "start": "react-scripts start", "build": "react-scripts build", "test": "react-scripts test", "eject": "react-scripts eject" }, "eslintConfig": { "extends": [ "react-app", "react-app/jest" ] }, "browserslist": { "production": [ ">0.2%", "not dead", "not op_mini all" ], "development": [ "last 1 chrome version", "last 1 firefox version", "last 1 safari version" ] } } ``` ### flask-backend/requirements.txt ``` altgraph==0.17.4 annotated-types==0.7.0 anyio==4.4.0 autogen==0.3.0 bidict==0.23.1 blinker==1.8.2 certifi==2024.8.30 cffi==1.17.1 charset-normalizer==3.3.2 click==8.1.7 colorama==0.4.6 cryptography==43.0.1 Cython==3.0.11 diskcache==5.6.3 distro==1.9.0 docker==7.1.0 FLAML==2.2.0 Flask==3.0.0 Flask-SocketIO==5.4.1 h11==0.14.0 httpcore==1.0.5 httpx==0.27.2 idna==3.8 itsdangerous==2.2.0 Jinja2==3.1.4 jiter==0.5.0 joblib==1.4.2 MarkupSafe==3.0.2 numpy==1.26.4 openai==1.44.1 packaging==24.1 pandas==2.2.3 pefile==2024.8.26 pyarmor==8.5.11 pyarmor.cli.core==6.5.3 pycparser==2.22 pydantic==2.9.1 pydantic_core==2.23.3 pyinstaller==6.10.0 pyinstaller-hooks-contrib==2024.8 python-dateutil==2.9.0.post0 python-dotenv==1.0.1 python-engineio==4.10.1 python-socketio==5.11.4 pytz==2024.2 pywin32==307 pywin32-ctypes==0.2.3 regex==2024.7.24 requests==2.32.3 scikit-learn==1.5.2 scipy==1.14.1 setuptools==75.1.0 simple-websocket==1.1.0 six==1.16.0 sniffio==1.3.1 termcolor==2.4.0 threadpoolctl==3.5.0 tiktoken==0.7.0 tqdm==4.66.5 typing_extensions==4.12.2 tzdata==2024.1 urllib3==2.2.2 Werkzeug==3.0.4 wheel==0.44.0 wsproto==1.2.0 ``` ### flask-backend/app.py ```python # app.py import os import json from flask import Flask, jsonify, request from flask_socketio import SocketIO, emit from threading import Thread from flask_cors import CORS import requests from autogen_model.model import return_menu_query_information app = Flask(__name__) CORS(app) app.config['SECRET_KEY'] = 'your_secret_key' socketio = SocketIO(app, cors_allowed_origins="*") DEEPGRAM_API_KEY = os.getenv('DEEPGRAM_API_KEY') if not DEEPGRAM_API_KEY: raise ValueError("DEEPGRAM_API_KEY environment variable not set.") # ---- WebSocket Chatbox Functionality ---- @socketio.on('connect') def handle_connect(): print('Client connected:', request.sid) @socketio.on('disconnect') def handle_disconnect(): print('Client disconnected:', request.sid) @socketio.on('audio-stop') def handle_audio_stop(audio_blob): sid = request.sid print(f"Received 'audio-stop' from SID {sid}, blob size: {len(audio_blob)} bytes") # Save the Blob to a file for inspection (optional) # audio_filename = f"received_audio_{sid}.wav" # with open(audio_filename, "wb") as f: # f.write(audio_blob) # print(f"Saved audio data to {audio_filename}") def deepgram_transcribe(): deepgram_url = 'https://api.deepgram.com/v1/listen' headers = { 'Authorization': f'Token {DEEPGRAM_API_KEY}', 'Content-Type': 'audio/wav; rate=16000; channels=1', # Must match frontend's WAV settings } try: response = requests.post(deepgram_url, headers=headers, data=audio_blob) if response.status_code == 200: result = response.json() transcript = result.get('results', {}).get('channels', [{}])[0].get('alternatives', [{}])[0].get('transcript', '') if transcript: print(f"Transcription for SID {sid}: {transcript}") # Emit the transcript back to the client socketio.emit('transcript', {'transcript': transcript}, room=sid) # Process the transcript with AutoGen # Adjust based on AutoGen's method to process text autogen_output = return_menu_query_information(transcript) # print(f"BackEND==========AutoGen Output for SID {sid}: {autogen_output}") # Emit the AutoGen output back to the client socketio.emit('autogen_output', {'autogen_output': autogen_output}, room=sid) else: print(f"No transcript found for SID {sid}") socketio.emit('transcript', {'transcript': 'No transcript available.'}, room=sid) else: print(f"Deepgram API Error for SID {sid}: {response.status_code} - {response.text}") socketio.emit('transcript', {'transcript': 'Error with Deepgram transcription.'}, room=sid) except Exception as e: print(f"Exception during Deepgram transcription for SID {sid}: {e}") socketio.emit('transcript', {'transcript': 'Exception during transcription.'}, room=sid) # Run Deepgram transcription in a separate thread to avoid blocking thread = Thread(target=deepgram_transcribe) thread.start() # ---- Shopping Cart API Functionality ---- # Assuming you have a ShoppingCart class defined somewhere from cart.cart_handler import ShoppingCart cart = ShoppingCart() # Initialize the shopping cart @app.route('/') def index(): return "Welcome to the Voice Ordering System!" # API to add a product to the cart @app.route('/add-to-cart', methods=['POST']) def add_to_cart(): data = request.json product = data.get('product') price = data.get('price') quantity = data.get('quantity', 1) # Default quantity is 1 if not provided additional_info = data.get('additional_info', '') # Add the product to the shopping cart cart.add_item(product, price, quantity, additional_info) return jsonify({"message": "Product added to cart", "cart": cart.get_cart(), "total": cart.get_total()}), 200 # API to get the current cart @app.route('/cart', methods=['GET']) def get_cart(): return jsonify({"cart": cart.get_cart(), "total": cart.get_total()}), 200 # @app.route('/clear-cart', methods=['POST']) # def clear_cart(): # cart = ShoppingCart() # return jsonify({'success': True, 'message': 'Cart has been cleared.'}) if __name__ == '__main__': socketio.run(app, debug=True, host='0.0.0.0', port=5001, allow_unsafe_werkzeug=True) ``` ### drive-through-voice-order/src/index.js ```javascript import React from 'react'; import ReactDOM from 'react-dom/client'; import './index.css'; import App from './App'; import reportWebVitals from './reportWebVitals'; const root = ReactDOM.createRoot(document.getElementById('root')); root.render( ); // If you want to start measuring performance in your app, pass a function // to log results (for example: reportWebVitals(console.log)) // or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals reportWebVitals(); ``` ### drive-through-voice-order/src/App.js ```javascript // src/App.js import React, { useState } from 'react'; import ChatBox from './components/ChatBox'; import Cart from './components/Cart'; import './App.css'; function App() { const [cartItems, setCartItems] = useState([]); const [totalPrice, setTotalPrice] = useState(0); // Function to handle adding items to the cart const addToCart = (item) => { setCartItems((prevItems) => [...prevItems, item]); setTotalPrice((prevPrice) => prevPrice + item.price); }; return (
{/* Header */}

Friendly Restaurant

{/* Main Content */}
{/* Left: ChatBox */}
{/* Right: Cart */}
); } export default App; ``` ### flask-backend/test.py ```python # test_deepgram.py import os import requests DEEPGRAM_API_KEY = os.getenv('DEEPGRAM_API_KEY') if not DEEPGRAM_API_KEY: raise ValueError("DEEPGRAM_API_KEY environment variable not set.") deepgram_url = 'https://api.deepgram.com/v1/listen' headers = { 'Authorization': f'Token {DEEPGRAM_API_KEY}', 'Content-Type': 'audio/l16; rate=16000; channels=1', } # Replace 'path_to_sample.l16' with the path to a valid 16-bit PCM audio file with open('./test.wav', 'rb') as f: audio_data = f.read() response = requests.post(deepgram_url, headers=headers, data=audio_data) if response.status_code == 200: result = response.json() print(result) transcript = result.get('results', {}).get('channels', [{}])[0].get('alternatives', [{}])[0].get('transcript', '') print('Transcript:', transcript) else: print(f"Error: {response.status_code} - {response.text}") ``` ### drive-through-voice-order/src/setupTests.js ```javascript // jest-dom adds custom jest matchers for asserting on DOM nodes. // allows you to do things like: // expect(element).toHaveTextContent(/react/i) // learn more: https://github.com/testing-library/jest-dom import '@testing-library/jest-dom'; ``` ### drive-through-voice-order/src/App.test.js ```javascript import { render, screen } from '@testing-library/react'; import App from './App'; test('renders learn react link', () => { render(); const linkElement = screen.getByText(/learn react/i); expect(linkElement).toBeInTheDocument(); }); ``` ### drive-through-voice-order/src/reportWebVitals.js ```javascript const reportWebVitals = onPerfEntry => { if (onPerfEntry && onPerfEntry instanceof Function) { import('web-vitals').then(({ getCLS, getFID, getFCP, getLCP, getTTFB }) => { getCLS(onPerfEntry); getFID(onPerfEntry); getFCP(onPerfEntry); getLCP(onPerfEntry); getTTFB(onPerfEntry); }); } }; export default reportWebVitals; ``` ### drive-through-voice-order/src/index.css ```css body { margin: 0; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue', sans-serif; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } code { font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New', monospace; } ``` [13 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]