# Project export: ShopShadow

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: CruzHacks 2025
- Tagline: A smart cart that follows you, scans items with a camera, and lets you pay through an app—no lines, no hassle. Just tap your phone, shop, and go. Fast, seamless, and contactless.
- Devpost: https://devpost.com/software/shopshadow
- GitHub: https://github.com/themegh1465/ShopShadow
- Demo: https://github.com/parshG/CruzHacks25.git
- Result: winner (Best AI Hack)
- Team: 1 GitHub contributor(s) — themegh1465 (1 commits)

## Devpost submission (written by the team)

### Inspiration

Long checkout lines, crowded stores, and the lack of tech integration in physical retail inspired us to create a smarter, smoother shopping experience—one where the cart does the work for you.

### What it does

ShopShadow is a smart shopping cart that follows the user via phone tap, scans items using a smart camera, and bills them directly through a connected mobile app for a fully contactless experience.

### How we built it

We used a 4-DC motor and ultrasonic sensor setup with an Arduino Uno on a chassis to simulate the cart. Originally planned with GPS, we switched to an HM-10 Bluetooth module and triangulated RSSI signals to track the user’s phone. For item detection, we used a Luxonis 12MP neural processing camera (1.2 TFLOPS), trained a custom YOLOv8 nano PyTorch model on 17,000 grocery images across 200+ classes, achieving 99% return rates at 86% confidence. The camera wirelessly sends data to a Swift + Flask backend app, which identifies the item and bills the user in real time.

### Challenges we ran into

As with any hackathon, we faced a multitude of challenges. The toughest was getting the ultrasonic object avoidance system to work in sync with the Bluetooth-based tracking. Just building a functional robot car—soldering, wiring, and writing stable C++ code for the Arduino—was a challenge on its own. We had to create logic gates for the ultrasonic sensors and develop a triangulation algorithm using Bluetooth RSSI values. Balancing navigation was tricky—we built a weighted bias system to prioritize obstacle avoidance while still tracking the user’s direction when taking detours. On the vision side, formatting raw output from the Luxonis camera proved difficult. It outputs in BLOB format, so we had to convert our PyTorch model to ONNX, then to BLOB, and wrap it inside a YOLOv8 nano structure to optimize the layer handling. Integrating this with the backend app added another layer of complexity due to data formatting and real-time transfer issues.

### Accomplishments we're proud of

We’re incredibly proud that we were able to build a fully functional smart shopping cart prototype within a limited hackathon timeframe. Despite multiple hardware and software roadblocks, we successfully integrated object detection, autonomous following, and mobile payment into one cohesive system. Getting the robot car to move reliably using 4 DC motors and ultrasonic sensors took hours of soldering, wiring, and testing. But seeing it finally respond to logic gates and real-world obstacles was a huge milestone. We’re also proud of our custom YOLOv8 nano model, which we trained on 17,000 images of grocery items across 200+ classes. Achieving 99% return rates at an 86% confidence level—on a model optimized to run efficiently on the Luxonis camera’s 1.2 TFLOPS hardware—was no small feat. Converting the model from PyTorch to ONNX to BLOB, and still getting great performance, proved our attention to optimization really paid off. Finally, we managed to tie everything together using a Swift + Flask backend app, with wireless data streaming from the Luxonis camera to the phone. It was a true full-stack integration of robotics, AI, and user experience—and seeing it all work in harmony was a proud moment for the entire team.

### What we learned

We learned a lot about system integration, especially when juggling low-level robotics and high-level app development. On the hardware side, we deepened our understanding of how to control DC motors and ultrasonic sensors through Arduino C++, and how to build logic gates and bias functions to prioritize obstacle avoidance over target tracking without completely sacrificing directionality. The process of triangulating Bluetooth RSSI signals for user tracking—especially without access to a GPS module—taught us creative problem-solving under pressure. On the AI front, we gained valuable experience in curating and training a large image dataset, optimizing a neural network model, and deploying it to a specialized edge device. Working with the Luxonis camera’s BLOB format and finding ways to convert and wrap our model so that it retained both accuracy and speed pushed our understanding of machine learning deployment in real-world systems. Finally, we learned how to coordinate hardware, AI, and app communication over a wireless network in real time—a challenging but incredibly rewarding experience that made us better engineers and collaborators.

### What's next

Moving forward, we plan to add a weight sensor system to validate the detected grocery items, minimizing false positives and improving billing accuracy. We also want to improve the pathfinding logic by integrating smarter obstacle detour algorithms that still bias toward the direction of the user’s phone. Our current weighted bias system works well, but with more time, we’d like to make it adaptive based on environmental complexity. For the camera, we’re exploring more efficient ways to reduce backend latency—possibly through local edge processing for basic classification, with final billing verification handled in the app. On the user experience side, we aim to expand the app’s capabilities to include store maps, item recommendations, and voice-based commands so the cart can respond more intuitively in-store. Ultimately, we’d love to pilot ShopShadow in a controlled retail environment—whether that’s a small local store or a campus pop-up—to gather real-world feedback and test how our system scales with more users, more items, and real shopping chaos.

## README (from the GitHub repository)

No README available.

## Detected evidence (automated analysis)

Indexed codebase: 6 recognized source files, 15 KB.
- Python (language) — detected in the code
- Flask (technology) — claimed on Devpost, not found in the code
- PyTorch (technology) — claimed on Devpost, not found in the code
- Swift (language) — claimed on Devpost, not found in the code
- TensorFlow (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (13 of 13)

```
args.yaml
best.onnx
best.pt
convert.py
data.yaml
detect-grocery.py
last.pt
README.dataset.txt
README.roboflow.txt
results.csv
setup.py
train_script.py
yolov8n.pt
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- Add files via upload

## Key source files (fetched from GitHub, selected and truncated for size)

### convert.py

```python
from ultralytics import YOLO

model = YOLO("runs/detect/train6/weights/best.pt")
model.export(format="onnx", opset=12)  # Export to best.onnx

```

### train_script.py

```python
from ultralytics import YOLO

# Load base model (you can use yolov8n, yolov8s, yolov8m, etc.)
model = YOLO("grocery-dataset/yolov8n.pt")

# Train on your Roboflow grocery dataset
model.train(data="grocery-dataset/data.yaml", epochs=15, imgsz=500, batch=16)

```

### data.yaml

```yaml
train: C:/Users/meghp/depthai-python/grocery-dataset/train/images
val: C:/Users/meghp/depthai-python/grocery-dataset/valid/images
test: C:/Users/meghp/depthai-python/grocery-dataset/test/images

nc: 58
names: ['-', 'Apple', 'Asparagus', 'Avocado', 'Banana', 'Beans', 'Blackberries', 'Blueberries', 'Book', 'Broccoli', 'Brussel Sprouts', 'Butter', 'Cabbage', 'Cantaloupe', 'Carrots', 'Cauliflower', 'Cerealbox', 'Cheese', 'Clementine', 'Coffee', 'Corn', 'Cucumber', 'Detergent', 'Drinks', 'Egg', 'Eggplant', 'Eggs', 'Galia', 'Grapes', 'Honeydew', 'Juice', 'Lettuce', 'Meat', 'Milk', 'Mushrooms', 'Nectarine', 'Orange', 'Oranges', 'Pineapple', 'Plum', 'Pomegranate', 'Raspberries', 'Salad', 'Sauce', 'Spinach', 'Squash', 'Strawberries', 'Strawberry', 'Tofu', 'Tomatoes', 'Watermelon', 'Yogurt', 'Zucchini', 'beverage', 'food-box', 'fruit', 'utility-box', 'vegetable']

roboflow:
  workspace: identvintern
  project: groceries-9vwuo
  version: 3
  license: CC BY 4.0
  url: https://universe.roboflow.com/identvintern/groceries-9vwuo/dataset/3
```

### detect-grocery.py

```python
import time

import depthai as dai
import cv2
import numpy as np

from examples.Yolo.yolov8_nano import labelMap

# Load your compiled YOLOv8 .blob
BLOB_PATH = "C:/Users/meghp/depthai-python/examples/models/detect-grocery/best.blob"


pipeline = dai.Pipeline()


# Camera node
cam = pipeline.create(dai.node.ColorCamera)
cam.setPreviewSize(512, 512)
cam.setInterleaved(False)
cam.setFps(30)

# Neural network node
nn = pipeline.create(dai.node.NeuralNetwork)
nn.setBlobPath(BLOB_PATH)
cam.preview.link(nn.input)

# Output streams
xout_rgb = pipeline.create(dai.node.XLinkOut)
xout_rgb.setStreamName("rgb")
cam.preview.link(xout_rgb.input)

xout_nn = pipeline.create(dai.node.XLinkOut)
xout_nn.setStreamName("nn")
nn.out.link(xout_nn.input)

# Run the device
with dai.Device(pipeline) as device:
    q_rgb = device.getOutputQueue("rgb", maxSize=4, blocking=False)
    q_nn = device.getOutputQueue("nn", maxSize=4, blocking=False)

    while True:
        frame = q_rgb.get().getCvFrame()
        in_nn = q_nn.tryGet()

        # (Optional) display bounding boxes
        if in_nn is not None:
            for det in in_nn.detections:
                x1 = int(det.xmin * frame.shape[1])
                y1 = int(det.ymin * frame.shape[0])
                x2 = int(det.xmax * frame.shape[1])
                y2 = int(det.ymax * frame.shape[0])
                cv2.rectangle(frame, (x1, y1), (x2, y2), (0,255,0), 2)

        cv2.imshow("Grocery Detection", frame)
        if cv2.waitKey(1) == ord('q'):
            break
```

### args.yaml

```yaml
task: detect
mode: train
model: grocery-dataset/yolov8n.pt
data: grocery-dataset/data.yaml
epochs: 15
time: null
patience: 100
batch: 16
imgsz: 500
save: true
save_period: -1
cache: false
device: null
workers: 8
project: null
name: train6
exist_ok: false
pretrained: true
optimizer: auto
verbose: true
seed: 0
deterministic: true
single_cls: false
rect: false
cos_lr: false
close_mosaic: 10
resume: false
amp: true
fraction: 1.0
profile: false
freeze: null
multi_scale: false
overlap_mask: true
mask_ratio: 4
dropout: 0.0
val: true
split: val
save_json: false
conf: null
iou: 0.7
max_det: 300
half: false
dnn: false
plots: true
source: null
vid_stride: 1
stream_buffer: false
visualize: false
augment: false
agnostic_nms: false
classes: null
retina_masks: false
embed: null
show: false
save_frames: false
save_txt: false
save_conf: false
save_crop: false
show_labels: true
show_conf: true
show_boxes: true
line_width: null
format: torchscript
keras: false
optimize: false
int8: false
dynamic: false
simplify: true
opset: null
workspace: null
nms: false
lr0: 0.01
lrf: 0.01
momentum: 0.937
weight_decay: 0.0005
warmup_epochs: 3.0
warmup_momentum: 0.8
warmup_bias_lr: 0.1
box: 7.5
cls: 0.5
dfl: 1.5
pose: 12.0
kobj: 1.0
nbs: 64
hsv_h: 0.015
hsv_s: 0.7
hsv_v: 0.4
degrees: 0.0
translate: 0.1
scale: 0.5
shear: 0.0
perspective: 0.0
flipud: 0.0
fliplr: 0.5
bgr: 0.0
mosaic: 1.0
mixup: 0.0
copy_paste: 0.0
copy_paste_mode: flip
auto_augment: randaugment
erasing: 0.4
crop_fraction: 1.0
cfg: null
tracker: botsort.yaml
save_dir: C:\Users\meghp\depthai-python\runs\detect\train6

```

### setup.py

```python
import os
import io
import re
import sys
import platform
import subprocess
import find_version
import multiprocessing

from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext
from distutils.version import LooseVersion
from pathlib import Path

### NAME
MODULE_NAME = 'depthai'
DEPTHAI_CLI_MODULE_NAME = 'depthai_cli'

### VERSION
here = os.path.abspath(os.path.dirname(__file__))
version_file = os.path.join(here, "generated", "version.py")
os.makedirs(os.path.join(here, "generated"), exist_ok=True)
if os.environ.get('CI') != None :
    ### If CI build, respect 'BUILD_COMMIT_HASH' to determine final version if set
    final_version = find_version.get_package_version()
    if os.environ.get('BUILD_COMMIT_HASH') != None:
        final_version = find_version.get_package_dev_version(os.environ['BUILD_COMMIT_HASH'])
    with open(version_file, 'w') as vf :
        vf.write("__version__ = '" + final_version + "'")
elif os.path.exists(".git"):
    ### else if .git folder exists, create depthai with commit hash retrieved from git rev-parse HEAD
    commit_hash = 'dev'
    try:
        commit_hash = (
            subprocess.check_output(
                ["git", "rev-parse", "HEAD"], stderr=subprocess.STDOUT
            )
            .splitlines()[0]
            .decode()
        )
    except subprocess.CalledProcessError as e:
        # cannot get commit hash, leave empty
        commit_hash = 'dev'
    final_version = find_version.get_package_dev_version(commit_hash)

    with open(version_file, 'w') as vf :
        vf.write("__version__ = '" + final_version + "'")


# If not generated, generate from find_version
if os.path.isfile(version_file) == False :
    # generate from find_version
    final_version = find_version.get_package_dev_version('dev')
    with open(version_file, 'w') as vf :
        vf.write("__version__ = '" + final_version + "'")

### Get version from version.py (sdist will have this pregenerated)
exec(open(version_file).read())
buildCommitHash = None
if len(__version__.split("+")) > 1 :
    buildCommitHash = __version__.split("+")[1]


## Read description (README.md)
long_description = io.open("README.md", encoding="utf-8").read()

## Early settings
MACOS_ARM64_WHEEL_NAME_OVERRIDE = 'macosx-11.0-arm64'
if sys.platform == 'darwin' and platform.machine() == 'arm64':
    os.environ['_PYTHON_HOST_PLATFORM'] = MACOS_ARM64_WHEEL_NAME_OVERRIDE

class CMakeExtension(Extension):
    def __init__(self, name, sourcedir=''):
        Extension.__init__(self, name, sources=[])
        self.sourcedir = os.path.abspath(sourcedir)


class CMakeBuild(build_ext):

    def run(self):
        try:
            out = subprocess.check_output(['cmake', '--version'])
        except OSError:
            raise RuntimeError("CMake must be installed to build the following extensions: " +
                               ", ".join(e.name for e in self.extensions))

        if platform.system() == "Windows":
            cmake_version = LooseVersion(re.search(r'version\s*([\d.]+)', out.decode()).group(1))
            if cmake_version < '3.2.0':
                raise RuntimeError("CMake >= 3.2.0 is required on Windows")

        for ext in self.extensions:
            self.build_extension(ext)

    def build_extension(self, ext):
        if ext.name == DEPTHAI_CLI_MODULE_NAME:
            # Copy cam_test.py and it's dependencies to depthai_cli/
            cam_test_path = os.path.join(here, "utilities", "cam_test.py")
            cam_test_dest = os.path.join(self.build_lib, DEPTHAI_CLI_MODULE_NAME, "cam_test.py")
            cam_test_gui_path = os.path.join(here, "utilities", "cam_test_gui.py")
            cam_test_gui_dest = os.path.join(self.build_lib, DEPTHAI_CLI_MODULE_NAME, "cam_test_gui.py")
            stress_test_path = os.path.join(here, "utilities", "stress_test.py")
            stress_test_dest = os.path.join(self.build_lib, DEPTHAI_CLI_MODULE_NAME, "stress_test.py")
            files_to_copy = [(cam_test_path, cam_test_dest), (cam_test_gui_path, cam_test_gui_dest), (stress_test_path, stress_test_dest)]
            for src, dst in files_to_copy:
                with open(src, "r") as f:
                    with open(dst, "w") as f2:
                        f2.write(f.read())
            return

        extdir = os.path.abspath(os.path.dirname(self.get_ext_fullpath(ext.name)))
        # required for auto-detection of auxiliary "native" libs
        if not extdir.endswith(os.path.sep):
            extdir += os.path.sep

        # initialize cmake_args and build_args
        cmake_args = []
        build_args = []
        env = os.environ.copy()

        # Specify output directory and python executable
        cmake_args += ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=' + extdir, '-DPYTHON_EXECUTABLE=' + sys.executable]
        # Specify dir of python executable (pybind11)
        if platform.system() == "Windows":
            # Windows - remove case insensitive variants
            env = {key:env[key] for key in env if key.upper() != 'pythonLocation'.upper()}
        env['pythonLocation'] = str(Path(sys.executable).parent.absolute())


        # Pass a commit hash
        if buildCommitHash != None :
            cmake_args += ['-DDEPTHAI_PYTHON_COMMIT_HASH=' + buildCommitHash]

        # Pass a docstring option
        if 'DEPTHAI_PYTHON_DOCSTRINGS_INPUT' in os.environ:
            cmake_args += ['-DDEPTHAI_PYTHON_DOCSTRINGS_INPUT='+os.environ['DEPTHAI_PYTHON_DOCSTRINGS_INPUT']]

        # Pass installation directory
        if 'DEPTHAI_INSTALLATION_DIR' in os.environ:
            cmake_args += ['-DDEPTHAI_PYTHON_USE_FIND_PACKAGE=ON']
            cmake_args += ['-DCMAKE_PREFIX_PATH='+os.environ['DEPTHAI_INSTALLATION_DIR']]

        # Set build type (debug vs release for library as well as dependencies)
        cfg = 'Debug' if self.debug else
[truncated — 4861 more characters]
```