Project Info

Screen Mosaic

This project did not submit a description on Devpost.

Screen Mosaic

Turn several ordinary displays into one coordinated canvas using a single phone photo. Each screen shows four ArUco markers in its corners; you photograph them all from one spot; the host then warps content per-screen so that — viewed from where the photo was taken — every screen lines up into one continuous image.

Gallery

Run the host

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

python mosiac

This starts the backend and two web apps on :5003 (LAN URLs are printed):

Screen slave — open http://<host-ip>:5003/display on each screen. Each browser auto-claims the next display slot (1st → display_1 with marker IDs 0–3, 2nd → display_2 with 4–7, …) and shows its four corner markers.
Phone — open http://<host-ip>:5003/phone. Take a photo of all screens, then map content onto them.

The phone's live camera (used by live calibration and the hand-tracked visualizations) needs a secure context, so the host also serves HTTPS on :5004 (PORT + 1) with a self-signed cert — open https://<host-ip>:5004/phone and accept the one-time warning. Everything else works over plain HTTP.

Phases (switched from the phone)

Calibration — every screen shows ArUco markers flush in its far corners. Tapping Take Photo on the phone reveals the markers; the photo is detected, each screen's corners are recovered (using the marker corner that touches the real screen corner), and any screen that didn't fully make the photo is highlighted on that screen with a message.
Mapping (default) — each screen renders mapped content, projectively warped by its photographed corners so skewed screens look straight from the camera. Content options:
- UV map — x→red, y→green gradient (default).
- Uploaded image — Fill (stretch to the screens' bounding box) or Fit (preserve aspect ratio).
- Visualization — a live animation rendered server-side (GPU when available) at a resolution matching the screens' bounding-box orientation, streamed (MJPEG) and warped per screen. Built in:
  - Particle Flow — flow-field particles.
  - Smoke — a stable-fluids fire/smoke sim.
  - Charges (plus Charges 1 / Charges 2, independent copies you can tune separately) — magnetic-charge particles that chase a cursor.
  - Fish Boids / Bird Boids — flocking sims (cohesion / alignment / separation + edge avoidance) with a Normal and a Game mode.

The phone's Content dropdown offers UV map / Upload image / Visualization; picking Visualization reveals a second dropdown populated from whatever is registered in the mosiac/visualizations/ package.

Hand-tracked visualizations (red-sticker CV)

Some visualizations are driven by your hand: put a red sticker on it, stream the phone camera, and the host tracks the largest red blob (HSV thresholding in red_tracker.py) and feeds its position to the sim — the fish flee it like a predator, the charge particles chase it as the cursor. These vizzes (Fish Boids, Charges & its copies) set NEEDS_PHONE_CAMERA, so selecting one auto-starts the phone camera stream (use the HTTPS URL).

A ⭕ Hand ring toggle appears on the phone for these vizzes to show/hide the translucent gray ring drawn at the tracked hand position. (An older YOLOv8-pose tracker, hands.py, is still available via HAND_TRACKER = "yolo".)

Live calibration — a camera continuously watches the screens and updates each screen's warp live (default 24 fps). Start it from the phone's 🔴 Live calibration button, then pick the camera source:
- Phone camera — the phone streams its own camera frames to the host.
- Server device camera — the host opens a local camera (cv2.VideoCapture).
Each screen keeps four smaller markers on screen (over the content, no ID labels) so the camera can track them. If a screen isn't fully visible in a frame, its warp holds at the last good value instead of blanking.

Browsers only allow camera access over a secure context, so the host serves HTTPS with a self-signed cert (accept the one-time warning on each device). Toggle with USE_HTTPS in consts.py.

Tunables in mosiac/consts.py: PORT, USE_HTTPS / HTTPS_PORT, MARKER_PX, LIVE_MARKER_PX, LIVE_FPS, LIVE_MAX_WIDTH, CAMERA_INDEX; hand tracking: HAND_FPS, HAND_TRACKER defaults, FISH_HAND_MARKER_FRAC (gray-ring size), plus the HAND_* YOLO options.

Adding a visualization

Drop a new file in mosiac/visualizations/ and import it from that package's __init__.py:

# mosiac/visualizations/rings.py
from . import Visualization, register, torch, _DEVICE

@register("rings", "Rings")
class Rings(Visualization):
    def step(self): ...
    def render(self): return frame   # H x W x 3 uint8 BGR

It appears in the phone dropdown automatically (GET /visualizations) — no server or frontend changes needed. Preview locally with python -m mosiac.visualizations rings.

The UV domain is the bounding box of all detected screen corners (plus a small margin), so the gradient/image/particles span only the region the screens cover.

Layout

Path	Purpose
`mosiac/`	The host. `python mosiac` runs `__main__` → `server.py`.
`mosiac/server.py`	Flask host: both web apps, calibration, mapping, content, hand stream.
`mosiac/detector.py`	ArUco/AprilTag detection → grouped, ordered, normalized.
`mosiac/red_tracker.py`	Red-sticker hand tracker (HSV blob centroid) — drives hand-tracked vizzes.
`mosiac/hands.py`	Alternative YOLOv8-pose hand tracker (`HAND_TRACKER = "yolo"`).
`mosiac/visualizations/`	Visualization package: framework in `__init__.py`, one file per viz (`particleflow.py`, `smokesim.py`, `charges*.py`, `fishboids.py`, `birdboids.py`).
`tools/`	Standalone analysis utilities (`python -m tools.cli IMAGE`, etc.).
`legacy/`	Earlier desktop prototype (`master/`, `slave/`, `shared/`).

Tools

python -m tools.make_test_image            # writes a synthetic test image
python -m tools.cli IMAGE --annotated out.png   # detect + visualize one image
python -m tools.app                        # standalone image-analysis web UI

Notes

Coordinates are always the photo's own space (origin top-left, x right, y down); no real-world depth/scale/pose is estimated. Marker→display grouping lives in detector.DEFAULT_DISPLAY_MAPPING.

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

FlaskIn code
HTMLIn code
PythonIn code
PyTorchIn code
JavaScriptClaimed

4 of 5 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

Claude CodeCommits

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

284 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Feature verification

15 Verified2 Code-supported

Alternative YOLOv8-pose hand tracker (HAND_TRACKER = "yolo")Verified
An older YOLOv8-pose hand tracker (hands.py) is still available via HAND_TRACKER = "yolo"
Claimed on readmehigh confidence
- mosiac/hands.py:33 — loads YOLO('yolov8n-pose.pt'), optional CoreML export, wrist keypoints -> hand positions
- mosiac/server.py:758 — hand_tracker(name) selects between 'red' and default 'yolo' tracker threads
ArUco marker-based multi-screen calibration from a single phone photoVerified
Each screen shows four ArUco markers in its corners; photographing them all from one spot lets the host recover each screen's corners
Claimed on readmehigh confidence
- mosiac/detector.py:78 — detect_markers uses cv2.aruco to find markers, tries multiple dictionaries including DICT_4X4_50
- mosiac/detector.py:149 — build_displays groups markers into per-display corner quads and normalizes them to the photo
- mosiac/server.py:171 — /marker/<id>.png renders ArUco markers for each display's corner slots
Auto-claiming display slots with marker ID ranges (1st -> 0-3, 2nd -> 4-7, ...)Verified
Each browser auto-claims the next display slot with sequential marker ID blocks
Claimed on readmehigh confidence
- mosiac/server.py:138 — _register assigns sequential display_id and marker_ids = base..base+3 per new client_id
Auto-discovered visualization plugin system (drop a file, no server/frontend changes needed)Verified
Adding a visualization is just dropping a file in mosiac/visualizations/ and importing it; it appears in the phone dropdown automatically via GET /visualizations
Claimed on readmehigh confidence
- mosiac/visualizations/__init__.py:53 — register() decorator adds a class to _REGISTRY; available() lists them
- mosiac/server.py:605 — GET /visualizations returns visualizations.available() consumed by the phone dropdown
Built with claimed technologies: aruco, flask, numpy, opencv, python, pytorch, javascript, http, htmlVerified
Built with aruco, flask, html, http, javascript, numpy, opencv, python, pytorch
Claimed on Devposthigh confidence
- requirements.txt:1 — lists opencv-contrib-python, numpy, flask, torch, flask-sock, ultralytics matching the built-with tags
- mosiac/detector.py:23 — cv2.aruco usage confirms ArUco
Calibration phase highlighting screens that didn't fully make the photoVerified
Any screen that didn't fully make the photo is highlighted on that screen with a message
Claimed on readmehigh confidence
- mosiac/server.py:216 — #incomplete banner shown when !s.complete in showCalibration
- mosiac/detector.py:173 — build_displays marks a display complete:false with missing_marker_ids when not all 4 markers detected
Hand-tracked visualizations via red-sticker HSV blob trackingVerified
Put a red sticker on your hand; the host tracks the largest red blob via HSV thresholding and feeds its position to Fish Boids / Charges sims, which chase or flee it
Claimed on readmehigh confidence
- mosiac/red_tracker.py:26 — _find_red_centroid does HSV inRange thresholding + largest contour centroid
- mosiac/server.py:758 — _ensure_hands_thread runs red_tracker.run when hand_tracker(viz_name) == 'red'
HTTPS server with self-signed cert for secure-context camera accessVerified
Host serves HTTPS on :5004 (PORT+1) with a self-signed cert for the phone's live camera
Claimed on readmehigh confidence
- mosiac/server.py:1713 — make_server on consts.HTTPS_PORT with ssl_context="adhoc" (self-signed)
- mosiac/consts.py:9 — USE_HTTPS / HTTPS_PORT = PORT + 1 tunables
Legacy earlier desktop prototype (master/slave/shared)Verified
legacy/ contains an earlier desktop prototype with master/, slave/, shared/ subfolders
Claimed on readmehigh confidence
- legacy/master — directory exists as described
- legacy/slave — directory exists as described
- legacy/shared — directory exists as described
Live calibration: continuous camera feed updating each screen's warp in real time (default 24 fps)Verified
A camera continuously watches the screens and updates each screen's warp live at default 24 fps, via phone camera or server device camera
Claimed on readmehigh confidence
- mosiac/server.py:1568 — @sock.route('/live/frames') and /live/corners/<display_id> websocket routes drive live updates
- mosiac/server.py:1510 — /live/status and /live/source endpoints select phone vs server camera
- mosiac/consts.py:30 — LIVE_FPS = 24 default tunable
Mapping phase content options: UV map, uploaded image (fill/fit), and visualizationsVerified
Mapping phase content options include UV map, uploaded image (Fill/Fit), or a live visualization
Claimed on readmehigh confidence
- mosiac/server.py:649 — /content POST accepts an uploaded image with mode fill/fit
- mosiac/server.py:668 — /content/visualization switches mapped content to a named live viz
- mosiac/server.py:272 — uvDataURL builds the default x-red/y-green UV gradient
Per-screen projective warp so screens line up as one continuous imageVerified
The host warps content per-screen so, viewed from the photo's vantage point, every screen lines up into one continuous image
Claimed on readmehigh confidence
- mosiac/server.py:429 — renderUV computes a homography from photographed corners to the screen rect and applies it via CSS matrix3d
- mosiac/server.py:814 — _warp_field_to_screen uses cv2.getPerspectiveTransform/warpPerspective per display for server-rendered content
Server-rendered streamed visualizations (Particle Flow, Smoke, Charges x3, Fish Boids, Bird Boids)Verified
Built-in visualizations: Particle Flow, Smoke (stable-fluids), Charges/Charges1/Charges2, Fish Boids, Bird Boids with flocking and game modes
Claimed on readmehigh confidence
- mosiac/visualizations/__init__.py:287 — registers particleflow, smokesim, charges, charges1, charges2, fishboids, birdboids
- mosiac/visualizations/gradients.py:1 — gradients module used by the smoke viz's gradient selector
Standalone analysis tools: make_test_image, cli.py, app.pyVerified
tools/ provides python -m tools.make_test_image, python -m tools.cli IMAGE --annotated out.png, and python -m tools.app standalone web UI
Claimed on readmehigh confidence
- tools/make_test_image.py:1 — file exists implementing the synthetic test image generator
- tools/cli.py:1 — file exists implementing the CLI detect+visualize tool
- tools/app.py:1 — file exists implementing the standalone image-analysis web UI
Two web apps served from one Flask backend: /display and /phoneVerified
Starts the backend and two web apps on :5003, screen slave at /display and phone app at /phone
Claimed on readmehigh confidence
- mosiac/server.py:508 — GET /display route renders DISPLAY_PAGE
- mosiac/server.py:1002 — PHONE_PAGE template served for the phone app
Hand ring toggle overlay and NEEDS_PHONE_CAMERA auto-start for hand-driven vizzesCode-supported
A hand ring toggle shows/hides a translucent gray ring at the tracked hand position; selecting a hand-driven viz auto-starts the phone camera stream
Claimed on readmemedium confidence
- mosiac/server.py:1087 — ringToggle button exists in the phone page, referenced by _vizNeedsPhone logic
- mosiac/consts.py:19 — FISH_HAND_MARKER_FRAC tunable for the gray-ring size mentioned in README
Live mode holds last good warp if a screen isn't fully visible in a frameCode-supported
If a screen isn't fully visible in a frame, its warp holds at the last good value instead of blanking
Claimed on readmemedium confidence
- mosiac/server.py:413 — showLive keeps displaying the last renderUV state when not (s.captured && s.complete); comment states corners persist server-side between frames, consistent with the claim

An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.

Project Info

Screen Mosaic

Screen Mosaic

Gallery

Run the host

Phases (switched from the phone)

Hand-tracked visualizations (red-sticker CV)

Adding a visualization

Layout

Tools

Notes

Analysis

Technology

AI coding agents

Codebase size

Structure

Application logic

Languages

Dependencies

Feature verification