Project Info

Nexus

Inspiration

If you forget a toothbrush on a vacation, it’s annoying. If you pack the wrong gear for a disaster relief mission, it’s dangerous. Right now, logistics coordinators for high-stakes deployments—whether it’s search-and-rescue or remote medical aid—are still relying on spreadsheets and mental math. They have to mentally juggle weight limits, material science (e.g., knowing that "cotton kills" in hypothermia scenarios), and cross-domain utility (a mylar blanket is both shelter and medical gear). We asked a simple question: Why can we search the entire internet in milliseconds, but we still have to rummage through bins to find our own stuff? We built Nexus to digitize the physical world and treat packing like the mathematical optimization problem it actually is.

What it does

Nexus is a search engine and logistics officer for your physical gear. It "Sees" Physics: You don't type in data. You just snap a photo. Nexus analyzes the image to understand not just what an item is, but what it does. It extracting factors such as thermal ratings, materials, waterproofing, and medical utility automatically. It Understands Context: You don't search for "flashlights." You search for "lighting for a power outage in heavy rain." Nexus understands the semantic difference between a keychain light and a tactical floodlight. It Reasons for Safety: A cross-domain synthesis layer actively rejects dangerous item selections (e.g., packing cotton clothing for cold-weather survival) and explains why, flagging critical gaps like "No water purification detected." It Solves the Knapsack Problem: You give it a mission ("72-hour cold-climate medical response") and constraints ("Bag with 15kg Weight Limit"). Nexus uses a constraint solver to mathematically prove the optimal loadout, ensuring you have enough medical gear without blowing your weight limit on heavy batteries.

How we built it

The "Eyes" (Vision & Extraction): We pipe camera feeds from Flutter directly to GPT-5. We prompt-engineered it to act like a materials scientist, extracting structured JSON metadata about an item's capabilities and failure modes via strict schema validation. The "Eyes" (Vision & Extraction): We pipe camera feeds from Flutter directly to GPT-5. We prompt-engineered it to act like a materials scientist, extracting structured JSON metadata about an item's capabilities and failure modes via strict schema validation. The "Brain" (Multimodal Embeddings): Standard text embeddings weren't enough. We used Voyage AI’s voyage-multimodal-3.5 model to generate 1024-dimensional vectors from the interleaved image plus serialized context text. This eliminates the modality gap, ensuring the vector representation includes both visual textures and explicitly extracted metadata. The "Brain" (Multimodal Embeddings): Standard text embeddings weren't enough. We used Voyage AI’s voyage-multimodal-3.5 model to generate 1024-dimensional vectors from the interleaved image plus serialized context text. This eliminates the modality gap, ensuring the vector representation includes both visual textures and explicitly extracted metadata. The "Memory" (Vector DB): We utilized Supabase with the pgvector extension and HNSW indexing. It handles both relational metadata and lightning-fast cosine similarity search in a single instance. The "Memory" (Vector DB): We utilized Supabase with the pgvector extension and HNSW indexing. It handles both relational metadata and lightning-fast cosine similarity search in a single instance. The "Math" (Optimization): We use the Google OR-Tools CP-SAT solver to run a bounded knapsack optimization. It balances diversity (e.g., minimum medical items, tag requirements) against weight limits across multi-container bin packing scenarios. The "Math" (Optimization): We use the Google OR-Tools CP-SAT solver to run a bounded knapsack optimization. It balances diversity (e.g., minimum medical items, tag requirements) against weight limits across multi-container bin packing scenarios.

Challenges we ran into

Dangerous Recommendations: Naive vector search returned items that were semantically similar but contextually dangerous (e.g., cotton for cold weather). We fixed this by explicitly extracting unsuitable_contexts and failure_modes during vision analysis, embedding them as first-class features. Dangerous Recommendations: Naive vector search returned items that were semantically similar but contextually dangerous (e.g., cotton for cold weather). We fixed this by explicitly extracting unsuitable_contexts and failure_modes during vision analysis, embedding them as first-class features. A user might ask for "3 medical items" when they only own 2. A standard solver just crashes and says "Infeasible." We had to write a wrapper that progressively relaxes constraints (e.g., dropping the item count requirement) so the app can say, "I couldn't find 3 items, but here are the 2 you have," instead of just throwing an error. A user might ask for "3 medical items" when they only own 2. A standard solver just crashes and says "Infeasible." We had to write a wrapper that progressively relaxes constraints (e.g., dropping the item count requirement) so the app can say, "I couldn't find 3 items, but here are the 2 you have," instead of just throwing an error. Weight Estimation without Scales: Most users don't know the exact weight of their gear. We built an AI weight estimation system that maps vision-inferred categories to gram estimates, allowing the knapsack solver to function without precise data. Weight Estimation without Scales: Most users don't know the exact weight of their gear. We built an AI weight estimation system that maps vision-inferred categories to gram estimates, allowing the knapsack solver to function without precise data. Accomplishments It actually works in 5 seconds: We managed to get the full pipeline—snap photo → GPT-5 Vision extraction → Multimodal Embedding → Database Write. Watching a raw image turn into a queryable, data-rich inventory item feels like magic every time. It actually works in 5 seconds: We managed to get the full pipeline—snap photo → GPT-5 Vision extraction → Multimodal Embedding → Database Write. Watching a raw image turn into a queryable, data-rich inventory item feels like magic every time. Cross-Domain Reasoning: We didn't hardcode rules. The system learned that a sleeping bag is useful for medical shock treatment purely through semantic similarity. Cross-Domain Reasoning: We didn't hardcode rules. The system learned that a sleeping bag is useful for medical shock treatment purely through semantic similarity. Optimal Packing: We aren't just guessing. When Nexus tells you what to pack, it's mathematically the best possible combination of items for your specific weight limit. Optimal Packing: We aren't just guessing. When Nexus tells you what to pack, it's mathematically the best possible combination of items for your specific weight limit. Visualizing the "Latent Space": We didn't just store vectors; we visualized them. We built a 3D force-directed graph (using a WebView bridge to React Three Fiber) that lets users physically "fly" through their inventory. You can visually see how "Batteries" cluster near "Flashlights" but far from "Socks," turning abstract math into a tangible map. Visualizing the "Latent Space": We didn't just store vectors; we visualized them. We built a 3D force-directed graph (using a WebView bridge to React Three Fiber) that lets users physically "fly" through their inventory. You can visually see how "Batteries" cluster near "Flashlights" but far from "Socks," turning abstract math into a tangible map.

What we learned

Optimization > Search: Semantic search is cool, but it's easy. The real value is in the constraint solver. People don't just want to find their gear; they want mathematical proof of what they should take. Vision Models are Materials Scientists: We were surprised by how much physical intuition GPT-5 Vision has. It correctly identified that a specific fabric looked like ripstop nylon without us telling it.

What's next

Team Mode & Collaborative Inventories: Shared loadouts for search-and-rescue teams or expedition groups with role-based access (e.g., "I have the tent, you bring the stove"). Hardware Integration: Connecting to Bluetooth luggage scales to track weight in real-time as you drop items into the box. Historical Analytics: Letting users rate their loadout post-mission ("The heavy sleeping bag was overkill") so the model learns preferences and improves future packing recommendations.

Nexus

AI-powered packing intelligence for missions, travel, and logistics.

Nexus turns your physical inventory into a searchable, semantically-rich vector database. Photograph your items, describe your mission in plain English, and get an optimized, LLM-explained packing manifest in seconds.

Architecture

flowchart LR
    subgraph frontend [Flutter Frontend]
        Camera[Camera Scan]
        Search[NL Search]
        Grid[Items Grid]
    end

    subgraph backend [FastAPI Backend]
        Ingest[Ingest Route]
        SearchRoute[Search Route]
        PackRoute[Pack Route]
    end

    subgraph ai [AI Pipeline]
        Vision["GPT-5 Vision\n(Context Extraction)"]
        Voyage["Voyage AI\n(Multimodal Embedding)"]
        Synth["GPT-5 LLM\n(Mission Synthesis)"]
        Knapsack["OR-Tools CP-SAT\n(Knapsack Optimizer)"]
    end

    subgraph db [Supabase]
        PG["PostgreSQL + pgvector"]
        Storage["Storage Bucket\n(manifest-assets)"]
    end

    Camera --> Ingest
    Ingest --> Vision --> Voyage --> PG
    Ingest --> Storage
    Search --> SearchRoute --> Voyage
    Voyage --> PG
    PG --> Synth --> Grid
    PG --> Knapsack --> PackRoute

Data Flows

Ingest — Image → GPT-5 Vision extracts structured context (material, thermal rating, medical use, durability, tags) → Voyage AI generates 1024-dim multimodal embedding → Supabase upsert + Storage upload

Search — Natural language query → Voyage AI embeds query → pgvector cosine similarity search → GPT-5 LLM curates results into an explained mission plan

Pack — Search results → OR-Tools CP-SAT bounded knapsack solver (weight limits, category diversity, tag requirements) → Optimized manifest with constraint relaxation reporting

Tech Stack

Layer	Technology
Languages	Python 3.13, Dart, SQL
Frontend	Flutter (Material 3), Dio, Supabase Flutter SDK, Image Picker
Backend	FastAPI, Uvicorn, Pydantic, HTTPX
AI — Vision	OpenAI GPT-5 Vision (structured context extraction)
AI — Embeddings	Voyage AI `voyage-multimodal-3.5` (1024-dim), CLIP ViT-B-32 (offline fallback)
AI — Synthesis	OpenAI GPT-5 (mission plan curation)
Optimization	Google OR-Tools CP-SAT Solver (bounded knapsack)
Database	Supabase PostgreSQL + pgvector
Storage	Supabase Storage (`manifest-assets` bucket)
Utilities	NumPy, Pillow, python-dotenv

Project Structure

nexus/
├── backend/
│   ├── ai_modules/              # Core AI pipeline
│   │   ├── config.py            # API keys, model config, constants
│   │   ├── context_extractor.py # GPT-5 Vision → structured ItemContext
│   │   ├── embedding_engine.py  # Voyage multimodal + CLIP fallback
│   │   ├── knapsack_optimizer.py# OR-Tools CP-SAT bounded knapsack
│   │   ├── mission_synthesizer.py# GPT-5 mission plan generation
│   │   ├── models.py            # Pydantic data models
│   │   ├── pipeline.py          # Top-level orchestrator (ingest/search/pack)
│   │   ├── vector_store.py      # Supabase pgvector integration
│   │   └── test_images/         # Sample images (camping, clothing, medical, tech)
│   ├── server/                  # FastAPI HTTP layer
│   │   ├── main.py              # App entry point, CORS, lifespan
│   │   ├── dependencies.py      # Singleton pipeline injection
│   │   ├── schemas.py           # Request/response Pydantic models
│   │   └── routes/
│   │       ├── ingest.py        # POST /api/v1/ingest, /ingest/upload
│   │       ├── search.py        # POST /api/v1/search/semantic
│   │       ├── pack.py          # POST /api/v1/pack
│   │       ├── items.py         # GET  /api/v1/items
│   │       └── containers.py    # CRUD /api/v1/containers
│   ├── migrations/              # SQL migrations (001–013)
│   ├── requirements.txt         # All Python dependencies
│   ├── seed_test_images.py      # Batch ingest local test images
│   ├── seed_dummyjson.py        # Batch ingest from DummyJSON API
│   └── .env                     # Environment variables (not committed)
├── frontend/
│   ├── lib/
│   │   ├── main.dart            # Flutter app (4-tab UI)
│   │   └── api_service.dart     # Backend API client
│   ├── pubspec.yaml             # Dart dependencies
│   └── .env                     # Frontend environment variables
└── README.md                    # This file

Setup

Prerequisites

Python 3.13+
Flutter SDK 3.2+
A Supabase project with pgvector enabled
API keys: OpenAI, Voyage AI

1. Database Setup

Open your Supabase project dashboard.

Go to SQL Editor and run the migrations in order:

backend/migrations/001_extensions.sql
backend/migrations/002_enums.sql
backend/migrations/003_profiles.sql
backend/migrations/004_manifest_items.sql
backend/migrations/005_missions.sql
backend/migrations/006_mission_items.sql
backend/migrations/007_rls_policies.sql
backend/migrations/008_vector_search.sql
backend/migrations/009_storage_policies.sql
backend/migrations/010_allow_null_user_id.sql
backend/migrations/011_storage_containers.sql
backend/migrations/012_enhanced_context_fields.sql
backend/migrations/013_enhanced_vector_search.sql

Or run 000_run_all_manifest.sql which includes all of the above.

Go to Storage → New bucket → Name: manifest-assets → Public: ON.

2. Backend Setup

cd backend

# Create virtual environment
python -m venv .venv
# Windows:
.\.venv\Scripts\Activate.ps1
# macOS/Linux:
source .venv/bin/activate

# Install all dependencies
pip install -r requirements.txt

# Configure environment
cp .env.example .env   # Then edit with your API keys

backend/.env requires:

Variable	Description
`OPENAI_API_KEY`	OpenAI API key (for GPT-5 Vision + synthesis)
`VOYAGE_API_KEY`	Voyage AI API key (for multimodal embeddings)
`SUPABASE_URL`	Supabase project URL
`SUPABASE_SERVICE_KEY`	Supabase service_role key (not the anon key)
`API_BASE_URL`	Backend URL, default `http://localhost:8000`

Run the server:

python -m uvicorn server.main:app --reload --port 8000

API docs available at http://localhost:8000/docs.

3. Frontend Setup

cd frontend

# Install Dart dependencies
flutter pub get

Create frontend/.env:

SUPABASE_URL=https://your-project.supabase.co
SUPABASE_ANON_KEY=your-anon-key
API_BASE_URL=http://localhost:8000

Run the app:

flutter run

Seed Data

Populate your Supabase database with sample items:

Local test images (34 items across camping, clothing, medical, tech):

cd backend
python seed_test_images.py

DummyJSON products (100 diverse items from a public API):

cd backend
python seed_dummyjson.py

Each seed script uploads images to Supabase Storage, runs GPT-5 Vision context extraction, generates Voyage multimodal embeddings, and upserts everything into the manifest_items table.

API Endpoints

All routes are prefixed with /api/v1.

Method	Endpoint	Description
`POST`	`/ingest`	Ingest item by image URL
`POST`	`/ingest/upload`	Ingest item by file upload
`POST`	`/search/semantic`	Natural language semantic search
`GET`	`/items`	List all items in the database
`POST`	`/pack`	Search + knapsack optimization
`GET/POST`	`/containers`	CRUD for storage containers
`GET`	`/health`	Health check

AI Pipeline Detail

1. Context Extraction (`context_extractor.py`)

GPT-5 Vision analyzes each item photo and returns structured JSON: name, category, material, weight estimate, thermal rating, water resistance, medical application, utility summary, semantic tags, durability, and compressibility.

2. Multimodal Embedding (`embedding_engine.py`)

Voyage AI voyage-multimodal-3.5 processes interleaved image + text through a single transformer backbone, producing a 1024-dimensional vector. Supports Matryoshka flexible dimensions (2048, 1024, 512, 256). Local CLIP ViT-B-32 available as offline fallback.

3. Vector Storage & Search (`vector_store.py`)

Supabase PostgreSQL with pgvector extension. Custom match_manifest_items RPC function performs filtered cosine similarity search with optional user and category scoping.

4. Mission Synthesis (`mission_synthesizer.py`)

GPT-5 curates raw search results into explained mission plans. Filters dangerous items (e.g., cotton in cold weather), flags critical gaps, identifies cross-domain utility, and provides per-item reasoning.

5. Knapsack Optimization (`knapsack_optimizer.py`)

Google OR-Tools CP-SAT solver handles bounded knapsack with category diversity minimums, tag requirements, and inventory constraints. Progressive constraint relaxation when no feasible solution exists. Supports multi-container bin-packing.

License

MIT

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

CIn code
C++In code
DartIn code
FastAPIIn code
HTMLIn code
KotlinIn code
OpenAIIn code
PythonIn code
PyTorchIn code
SQLIn code
SupabaseIn code
SwiftIn code
PostgreSQLClaimed

12 of 13 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

Claude CodeCommits
CursorCommits

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

543 KB

Source files

104

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Project Info

Nexus

Inspiration

What it does

How we built it

Challenges we ran into

What we learned

What's next

Nexus

Architecture

Data Flows

Tech Stack

Project Structure

Setup

Prerequisites

1. Database Setup

2. Backend Setup

3. Frontend Setup

Seed Data

API Endpoints

AI Pipeline Detail

1. Context Extraction (`context_extractor.py`)

2. Multimodal Embedding (`embedding_engine.py`)

3. Vector Storage & Search (`vector_store.py`)

4. Mission Synthesis (`mission_synthesizer.py`)

5. Knapsack Optimization (`knapsack_optimizer.py`)

License

Analysis

Technology

AI coding agents

Codebase size

Structure

Interface

API & routing

Application logic

Data & schema

Languages

Dependencies

Nexus

Inspiration

What it does

How we built it

Challenges we ran into

What we learned

What's next

Nexus

Architecture

Data Flows

Tech Stack

Project Structure

Setup

Prerequisites

1. Database Setup

2. Backend Setup

3. Frontend Setup

Seed Data

API Endpoints

AI Pipeline Detail

1. Context Extraction (context_extractor.py)

2. Multimodal Embedding (embedding_engine.py)

3. Vector Storage & Search (vector_store.py)

4. Mission Synthesis (mission_synthesizer.py)

5. Knapsack Optimization (knapsack_optimizer.py)

License

Technology

AI coding agents

Codebase size

Structure

Interface

API & routing

Application logic

Data & schema

Languages

Dependencies

1. Context Extraction (`context_extractor.py`)

2. Multimodal Embedding (`embedding_engine.py`)

3. Vector Storage & Search (`vector_store.py`)

4. Mission Synthesis (`mission_synthesizer.py`)

5. Knapsack Optimization (`knapsack_optimizer.py`)