Project Info

Orderly

This project did not submit a demo video on Devpost.

Inspiration

Orderly is an AI-powered solution designed to revolutionize the drive-thru experience, addressing the high-demand environment where 60% to 70% of fast food sales occur. By leveraging advanced voice technology and Large Language Model (LLM) agents, Orderly streamlines the ordering process, reducing human error, cutting operational costs, and improving transaction efficiency. It offers smooth, human-like interactions that ease the burden on drive-thru operators while enhancing customer satisfaction. This innovation promises to transform drive-thru operations, creating a faster, more accurate, and user-friendly experience that benefits both restaurant owners and customers. From Orderly, you, no matter you are a customer or an owner, will get: Smooth transaction drive-thru experience Lower employment cost Higher ordering efficiency

What it does

Orderly is an AI-powered voice assistant that allows customers to place, modify, and manage their drive-through orders using natural conversation. It handles everything from adding items to the cart to clearing orders when needed, all through hands-free, real-time voice interaction. The system can process multiple items, handle special requests, and ensure orders are accurate before submission.

How we built it

We built Orderly using a combination of modern web technologies and AI services. The frontend is a React-based interface that uses Socket.IO for real-time communication. On the backend, we used Flask with Socket.IO to handle voice data and order processing. For speech recognition and natural language processing, we integrated third-party APIs like Deepgram to transcribe audio and generate responses. The cart and order management system is powered by a Flask-based API that manages the state of the customer's order.

Challenges we ran into

One of the biggest challenges was ensuring accurate speech recognition, especially in noisy environments like drive-throughs. Handling complex or ambiguous customer requests also required fine-tuning the natural language processing and conversation flow. Additionally, ensuring smooth real-time communication between the frontend and backend, while processing voice data efficiently, was a technical hurdle we had to address.

Accomplishments we're proud of

We’re proud of successfully creating a voice-driven system that can handle real-time order management with high accuracy. The seamless integration between voice input, order processing, and cart management is a major achievement. Additionally, we were able to implement features like order modifications and clearing the cart, which bring real-world value to the drive-through experience.

What we learned

We learned a lot about the challenges of working with voice recognition in noisy environments and how to optimize AI models for better accuracy. We also gained valuable experience in real-time communication technologies, including Socket.IO, and how to build scalable, responsive APIs. Most importantly, we learned the importance of designing intuitive, user-friendly systems for end-users who may not be tech-savvy. Moving forward, we plan to improve Orderly by incorporating more advanced AI features, such as personalized order recommendations based on previous history and dynamic upselling. We also aim to refine the speech recognition system to handle accents and dialects better. Expanding Orderly to integrate with popular POS systems and testing it in real-world drive-through environments are also on our roadmap.

CalHack11.0

From Orderly, you, no matter you are a customer or an owner, will get:

Smooth transaction drive-thru experience
Lower employment cost
Higher ordering efficiency

Drive-Through Voice Ordering System

This project is a proof-of-concept for using an AI voice agent to help place food orders in a drive-through store. It consists of two main components:

Front-End: A React app for the user interface.
Back-End: A Flask app that handles voice transcription, order processing, and database interaction.

Prerequisites

Before setting up the project, ensure you have the following installed:

Node.js (for React front-end)
Python 3.x (for Flask back-end)
Git (for version control)

Setup Instructions

1. Clone the Repository

First, clone the repository to your local machine:

git clone <repository-url>
cd <repository-folder>

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

CSSIn code
FlaskIn code
HTMLIn code
JavaScriptIn code
OpenAIIn code
PythonIn code
ReactIn code

7 of 7 appear in the indexed code.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

50 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Feature verification

7 Verified3 Code-supported2 Claimed only

Add items to cart / orderVerified
Handles everything from adding items to the cart... can process multiple items
Claimed on readmehigh confidence
- flask-backend/cart/cart_handler.py:6 — ShoppingCart.add_item adds or merges items with quantity and price
- flask-backend/app.py:94 — /add-to-cart endpoint calls cart.add_item, and ChatBox.js posts parsed AutoGen items to this endpoint
Deepgram speech-to-text transcriptionVerified
Integrated third-party APIs like Deepgram to transcribe audio
Claimed on Devposthigh confidence
- flask-backend/app.py:45 — deepgram_transcribe() posts the audio blob to https://api.deepgram.com/v1/listen using DEEPGRAM_API_KEY and parses the transcript from the response
LLM agent (AutoGen) order processingVerified
Leveraging LLM agents to streamline the ordering process; built with autogen
Claimed on Devposthigh confidence
- flask-backend/autogen_model/model.py:21 — Defines Entrypoint_Agent, Check_Agent, and Transcription_Agent as autogen ConversableAgent instances and runs a multi-agent chat via initiate_chats to turn the transcript into structured order info
- flask-backend/requirements.txt:3 — autogen==0.3.0 is a declared dependency
React frontend interfaceVerified
The frontend is a React-based interface
Claimed on Devposthigh confidence
- drive-through-voice-order/src/App.js:1 — React app composing ChatBox and Cart components
Real-time cart display in UIVerified
cart and order management system is powered by a Flask-based API that manages the state of the customer's order
Claimed on Devposthigh confidence
- drive-through-voice-order/src/components/Cart.js:10 — Polls GET /cart every second and renders items/total from the Flask cart API
- flask-backend/app.py:107 — /cart endpoint returns cart.get_cart() and cart.get_total()
Real-time communication with Socket.IOVerified
Frontend uses Socket.IO for real-time communication; backend uses Flask with Socket.IO to handle voice data
Claimed on Devposthigh confidence
- flask-backend/app.py:17 — SocketIO(app, cors_allowed_origins="*") plus @socketio.on handlers for connect/disconnect/audio-stop and emits for transcript/autogen_output
- drive-through-voice-order/src/components/ChatBox.js:21 — Frontend connects with io('http://localhost:5001') and listens for 'transcript' and 'autogen_output' events
Voice-based drive-through ordering via natural conversationVerified
AI-powered voice assistant that lets customers place and manage orders through natural conversation
Claimed on Devposthigh confidence
- drive-through-voice-order/src/components/ChatBox.js:213 — startRecording/stopRecording capture microphone audio, encode as WAV, and emit it to the backend over Socket.IO
- flask-backend/app.py:34 — handle_audio_stop receives the audio blob, sends it to Deepgram for transcription, then routes the transcript to the AutoGen pipeline
Clear / cancel the orderCode-supported
all through hands-free, real-time voice interaction... clearing orders when needed
Claimed on readmemedium confidence
- flask-backend/cart/cart_handler.py:38 — clear_cart() method exists on ShoppingCart
- flask-backend/app.py:111 — The /clear-cart Flask route is commented out, so the endpoint the frontend calls does not exist server-side
- drive-through-voice-order/src/components/ChatBox.js:141 — clearCart() posts to /clear-cart, which is unreachable given the commented-out route, so this path would fail at runtime
Handle special requests / custom item notesCode-supported
The system can process multiple items, handle special requests, and ensure orders are accurate before submission
Claimed on Devpostmedium confidence
- flask-backend/cart/cart_handler.py:10 — add_item accepts an additional_info/comment field per item
- flask-backend/autogen_model/menu_query.py:25 — Prompt asks the LLM to produce a 'Custom Message' column, but there is no code that validates order accuracy before submission
Menu-aware order matching (menu data lookup and pricing)Code-supported
streamlines the ordering process, reducing human error... higher ordering efficiency
Claimed on Devpostmedium confidence
- flask-backend/autogen_model/menu_query.py:6 — read_data/preProcessData load menu CSV data and generatePrompting embeds instructions to match user input against menu data and sizes, but the LLM output is not cross-checked against the menu file programmatically
Order modification (updating items already placed)Claimed only
allows customers to place, modify, and manage their drive-through orders
Claimed on readmemedium confidence
Standalone always-on Deepgram Voice Agent (mic-to-speaker loop)Claimed only
advanced voice technology... smooth, human-like interactions
Claimed on Devposthigh confidence

An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.