Project Info

Winner

Hyperbolic: AI Inference Application Bounty

CLaiM

Inspiration

After the recent hurricanes in Florida, one of our team members who is from Florida had damage to their house. Normally, the process for filing an insurance claim after disaster is extremely cumbersome, requiring proof of all items and their value that were in the house before the disaster. This is very hard for disaster victims to prove in such a tough situation, which dissuades many people from filing insurance claims that cover the full value of all the items they lost.

What it does

Our app allows homeowners to record a video of their house, while our innovative AI automatically catalogues all items in your house and their approximate value. After a disaster, you an rescan your house and we automatically suggest items that might be lost to add to your insurance claim, with image proof of the before and after.

How we built it

Our frontend is built with next.js. Our backend uses Flask and sqllite. We also use xrpl in order to put items on the blockchain for proof of their existence before the disaster. Here are the steps our AI uses to process videos and pictures into items: The video is stitched together into a large panorama with OpenCV We use YOLOv8 and Meta's segment anything model to generate areas of interest on the image. We use Hyperbolic's inference (Qwen2 VL7B) to analyze each area and decide if it is an object to record and find its estimated monetary value We place all images in ChromaDB to group/count items of the same type and match images before and after disaster. We originally used ResNet for our embedding for the vector database in order to compare items, but later found all-MiniLM-L6-v2 to work better for vector encoding. The ResNet embeddings were too high dimensional. We generate a pdf claim to submit to the insurance company based on items that didn't appear after the disaster and were added by the user.

Challenges we ran into

Segmentation problem: We tried a lot of different things in order to segment images well. We tried the segment everything model, SAM+yolo, and tried asking gemini to bounding boxes (it's the only vision language model that advertises this capability). We found SAM+yolo worked the best by far but only after lots of iteration and tuning. System Design: lots of iteration required and thinking about the customer. For simplicity chose to host a single AWS EC2 instance although seriously considered hosting multiple servers for each general task like segmentation. Video Panorama: How do you do segmentation on a video without getting duplicate items across frames? Our solution was to turn the video into a large image and run our machine learning on this image. CORS struck us once again.

Accomplishments we're proud of

Super fast Segment Everything + Yolo + recognition + grouping pipeline that robustly recognized and catalogued items from video. Well designed and aesthetic front-end.

What we learned

There are no power outlets in SF. We learned how to deploy an EC2 instance, and also how to use vector databases.

What's next

We would like to

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

CSSIn code
Next.jsIn code
PythonIn code
ReactIn code
Tailwind CSSIn code
TypeScriptIn code
AWSClaimed
FlaskClaimed

6 of 8 appear in the indexed code. 2 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

341 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Feature verification

9 Verified4 Code-supported1 Blocked

all-MiniLM-L6-v2 embeddings (replacing ResNet)Verified
Switched from ResNet to all-MiniLM-L6-v2 for vector encoding since ResNet embeddings were too high dimensional
Claimed on Devposthigh confidence
- server/image_embedding.py:29 — uses chromadb DefaultEmbeddingFunction (all-MiniLM-L6-v2) for text embeddings; ResNet50 code is present but commented out, matching the claimed switch
ChromaDB grouping/counting and before/after matchingVerified
Images placed in ChromaDB to group/count items of same type and match before/after disaster
Claimed on Devposthigh confidence
- server/chroma.py:73 — get_item_uuid_of_embedding finds nearest existing embedding within a distance threshold to group repeated items into the same item_id
- server/chroma.py:104 — filter_images_by_metadata supports filtering by before flag, used to separate before/after images
Flask + SQLite backendVerified
Our backend uses Flask and sqllite
Claimed on Devposthigh confidence
- server/server.py:11 — Flask app instantiated with routes
- server/db.py:10 — sqlite3.connect('items_and_images.db') used for the Items table
Hyperbolic Qwen2-VL item recognition and valuationVerified
Hyperbolic's inference (Qwen2 VL7B) analyzes each area to decide if it's an object and estimate monetary value
Claimed on Devposthigh confidence
- server/hyperbolic.py:21 — calls api.hyperbolic.xyz chat/completions with model Qwen/Qwen2-VL-7B-Instruct, prompt asks for name/description/category/price/is_object
Image proof of before/after itemsVerified
We automatically suggest items... with image proof of the before and after
Claimed on Devpostmedium confidence
- server/server.py:82 — /inventory endpoint joins item images (with before flag in metadata) to items, letting the frontend display before/after images
- ch24-client/src/app/uploads/page.tsx:58 — Images column renders up to 3 associated images per item
Next.js frontendVerified
Our frontend is built with next.js
Claimed on Devposthigh confidence
- ch24-client/next.config.mjs — Next.js config file present
- ch24-client/package.json — package.json in ch24-client sets up a Next.js app
Record/upload house video and photosVerified
Homeowners record a video of their house for cataloguing
Claimed on Devposthigh confidence
- server/server.py:33 — /upload_media endpoint accepts image and video files and routes to process_video/process_image
- ch24-client/src/app/upload/page.tsx:118 — frontend upload dropzone accepts video/mp4, video/quicktime and images, posts to /upload_media
Video panorama stitching with OpenCVVerified
The video is stitched together into a large panorama with OpenCV
Claimed on Devposthigh confidence
- server/stitcher.py:8 — create_panorama extracts frames with cv2.VideoCapture and stitches them using the stitching.Stitcher (OpenCV-based) library
YOLOv8 + SAM object detection/segmentationVerified
YOLOv8 and Meta's segment anything model generate areas of interest on the image
Claimed on Devposthigh confidence
- server/ml.py:92 — loads YOLO('yolov8n.pt') for detection
- server/ml.py:94 — loads SamModel.from_pretrained('facebook/sam-vit-huge') and segments detected boxes
AWS EC2 / S3 hostingCode-supported
Hosted on a single AWS EC2 instance, uses S3
Claimed on Devpostmedium confidence
- server/aws.py:1 — aws.py implements open_s3_client and upload_image_to_s3 used by ml.py to store segmented item images in S3
PDF claim generation for missing itemsCode-supported
Generate a pdf claim to submit to the insurance company based on items that didn't appear after the disaster
Claimed on Devpostmedium confidence
- ch24-client/src/app/pdf/page.tsx:74 — renders a PDF document via @react-pdf/renderer from a 'claimedItems' list, but reads from localStorage 'claimedItems' which no other visible code path actually populates (the localStorage.setItem for claimedItems is commented out in uploads/page.tsx), and there is no server-side logic that auto-computes 'items that didn't appear after the disaster'
- ch24-client/src/app/uploads/page.tsx:273 — the code that would set claimedItems in localStorage and open /pdf is commented out, and confirm_matches instead just updates image status server-side with no diffing logic for missing items
Suggest items possibly lost after disaster (before/after diffing)Code-supported
After a disaster, rescan your house and we automatically suggest items that might be lost to add to your insurance claim
Claimed on Devpostlow confidence
- server/db.py:17 — Items table tracks before_count and after_count per item, which could support detecting items missing after a disaster
- server/server.py:139 — confirm_matches endpoint exists to mark matched images, but no endpoint/query computes 'items with before_count > after_count' to auto-suggest missing items
XRPL blockchain proof of existenceCode-supported
We use xrpl in order to put items on the blockchain for proof of their existence before the disaster
Claimed on Devpostlow confidence
- server/BLOCKCHAIN.py:60 — put_on_blockchain mints an XRPL NFT via NFTokenMint for each image URL using xrpl-py against the testnet faucet
- server/chroma.py:6 — chroma.py does 'import blockchain' (lowercase) and calls blockchain.put_on_blockchain, but the only module on disk is BLOCKCHAIN.py (uppercase); on a case-sensitive filesystem this import would fail, so the wiring between image upload and blockchain minting is unconfirmed/likely broken as committed
Winner of Calhacks 2024Blocked
Winner of Calhacks 2024 prize
Claimed on readmelow confidence

An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.