HackStack

Project Info

Engram

Devpost

Inspiration

Today's robots are brilliant at perceiving a scene and amnesiac about it a second later. A vision model can tell you "that's a mug," but it can't be taught "this is my mug" without retraining, and it has no memory of where it saw things. We wanted to build the missing piece for physical AI: a memory layer, instant, teachable, and spatial, that sits between a robot's eyes and its actions. The north-star question we kept asking: "Would a real robotics team actually use this?"

What it does

Engram gives a Booster K1 humanoid a cloud brain and instant memory: Perceives with a 72-billion-parameter vision model (Qwen2.5-VL) running on Nebius GPUs. It captions whatever the robot's camera sees. Remembers every object as a vector in Redis, with sub-2 ms nearest-neighbor recall over its whole memory bank. Learns in one shot. Show it a brand-new object it's never seen, teach it once (one vector written to Redis), and it recognizes it instantly. No retraining. Recognizes and attends. It turns its head to look at the object it recalls, tracking the object's position continuously. Has spatial memory. Walk it through a scene and it captures every object and the heading where it saw it. Fetches on command. Say "go to the red soda can" and it first checks whether it even remembers one, recalls where it was, walks toward it (closed-loop on odometry), stops at a safe distance using the robot's depth camera so it doesn't crash into the table, faces it, and waves. You can watch the robot's-eye view and its live decision (recalled object, confidence, Redis search latency, the Nebius model) on a real-time dashboard.

How we built it

Architecture, a thin robot plus a cloud brain: Perception (Nebius Token Factory): OpenAI-compatible API. Qwen/Qwen2.5-VL-72B-Instruct turns a camera frame into a short caption; Qwen/Qwen3-Embedding-8B turns the caption into a 4096-d vector. One VLM call also returns the object's horizontal center for tracking. Memory (Redis Stack): a FLAT cosine vector index over the embeddings, plus numeric fields storing the robot's pose (x, y, θ) where each object was seen. KNN recall runs in ~2 ms over a 124-object bank. A text query ("is the green bottle here?") embeds and searches the same index. Robot (Booster K1): ROS2 Humble + booster_robotics_sdk_python. RGB from /booster_video_stream, metric depth from /StereoNetNode/stereonet_depth (mono16, millimeters), odometry, and high-level motion (RotateHead, Move, WaveHand). Closed-loop locomotion: turns and walks are odometry-feedback controlled (we calibrated a slip factor because the K1's odometry over-reports rotation ~1.4× from foot slip), so "turn 180°" actually turns 180°. Depth-gated safe approach: during the final walk, we sample the nearest obstacle in the depth image and stop the robot a safe distance from the object before it can hit the table. Robust recognition: an object captured from several angles gets several captions, so we match the live view to the target by semantic similarity, not exact strings. Live demo dashboard: the brain serves an HTML/JS page showing the robot's view, the recalled label + confidence, a gaze marker, the Redis search latency, and the Nebius model, the "decision on screen." Laptop ↔ robot connected over Tailscale so the robot can be untethered.

Challenges we ran into

Laptop ↔ robot connected over Tailscale so the robot can be untethered. Challenges we ran into Firmware reality: the robot's SDK build didn't implement the Cartesian arm-control API (returned 501 Not Implemented), so our planned "point at it" pivoted to a head-look + walk-up + wave that the firmware does support. Firmware reality: the robot's SDK build didn't implement the Cartesian arm-control API (returned 501 Not Implemented), so our planned "point at it" pivoted to a head-look + walk-up + wave that the firmware does support. Odometry slip: open-loop turns were wildly inaccurate; we switched to closed-loop odometry feedback and calibrated the rotation scale so turns land on target. Odometry slip: open-loop turns were wildly inaccurate; we switched to closed-loop odometry feedback and calibrated the rotation scale so turns land on target. Depth that reads the wrong thing: a naive center-of-frame depth sample looked over the table at the far wall, so the robot didn't stop. We moved to a nearest-obstacle sample over a wider, lower band. Depth that reads the wrong thing: a naive center-of-frame depth sample looked over the table at the far wall, so the robot didn't stop. We moved to a nearest-obstacle sample over a wider, lower band. One object, many captions: the same object captioned differently from different angles broke exact-label matching during approach, fixed with semantic matching. One object, many captions: the same object captioned differently from different angles broke exact-label matching during approach, fixed with semantic matching. Networking gauntlet: captive portals, a flaky Ethernet link, and hotspot/Tailscale relays. We ended up on Tailscale so the robot could roam. Networking gauntlet: captive portals, a flaky Ethernet link, and hotspot/Tailscale relays. We ended up on Tailscale so the robot could roam. Staging perception: the VLM captions the dominant object, so objects have to fill enough of the frame, a real lesson in demo design.

Accomplishments we're proud of

Staging perception: the VLM captions the dominant object, so objects have to fill enough of the frame, a real lesson in demo design. Accomplishments that we're proud of One-shot live learning that actually works. Teach a brand-new object once and it's known instantly. A vision model alone can't do this. One-shot live learning that actually works. Teach a brand-new object once and it's known instantly. A vision model alone can't do this. A genuinely large model, 72B parameters, running in the live robot loop on Nebius, not a toy API call. A genuinely large model, 72B parameters, running in the live robot loop on Nebius, not a toy API call. Sub-2 ms memory recall in Redis over the whole object bank. Sub-2 ms memory recall in Redis over the whole object bank. A full spatial fetch: check memory, recall heading, walk, depth-safe stop, wave, built and tuned on real hardware in a single hack. A full spatial fetch: check memory, recall heading, walk, depth-safe stop, wave, built and tuned on real hardware in a single hack. Closed-loop humanoid navigation calibrated from scratch in hours.

What we learned

Closed-loop humanoid navigation calibrated from scratch in hours. What we learned Caption → embed → vector store is a remarkably robust, nearly-free perception path; it matches on caption consistency more than accuracy. Caption → embed → vector store is a remarkably robust, nearly-free perception path; it matches on caption consistency more than accuracy. Closed-loop beats open-loop for everything on a real robot. Trust the sensor, not the command. Closed-loop beats open-loop for everything on a real robot. Trust the sensor, not the command. Depth needs careful region sampling. "What's directly ahead of me" is not "the median of the center of the image." Depth needs careful region sampling. "What's directly ahead of me" is not "the median of the center of the image." The memory layer is the differentiator: perception is a commodity; a teachable, instant, spatial memory is what makes a robot feel like it understands its space.

What's next

The memory layer is the differentiator: perception is a commodity; a teachable, instant, spatial memory is what makes a robot feel like it understands its space. What's next for Engram Image embeddings (Path B): CLIP-style image vectors on a Nebius GPU to tell lookalikes apart (two similar cans). Image embeddings (Path B): CLIP-style image vectors on a Nebius GPU to tell lookalikes apart (two similar cans). Brain on the robot: run the whole pipeline on the K1's onboard Jetson for full autonomy, untethered. Brain on the robot: run the whole pipeline on the K1's onboard Jetson for full autonomy, untethered. True spatial navigation / SLAM: persistent maps so it can be walked through a building and sent anywhere. True spatial navigation / SLAM: persistent maps so it can be walked through a building and sent anywhere. Depth for 3D grasping: use the RGBD stream to reach and pick, not just point. Depth for 3D grasping: use the RGBD stream to reach and pick, not just point. Scale the memory: thousands of objects, still sub-ms. Scale the memory: thousands of objects, still sub-ms.

Analysis

Compare with all teams

View

Metric

No commits on this project resolved to a GitHub account.

Technology

Found in codeClaimed only
  • PythonIn code
  • JavaScriptClaimed
  • RedisClaimed

1 of 3 appear in the indexed code. 2 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

  • Claude CodeCommits

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

101 KB

Source files

18

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

0 stars