# Project export: Neural-Holo This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: Capture. Reconstruct. Project. Our system transforms live camera data into measurable real-time 3D holograms using AI and holography. - Devpost: https://devpost.com/software/neuroholo - GitHub: https://github.com/IshTale/Hologram - Video: https://www.youtube.com/embed/wgDIAD4lBZ8?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 6 GitHub contributor(s) — IshTale (8 commits), Areeb Sheikh (4 commits), Kentstar Harsono (3 commits), richik-p (2 commits), Claude Opus 4.8 (1 commits), Cursor (1 commits) ## Devpost submission (written by the team) ### Inspiration Our inspiration for this project began at the Beckman Laser Institute, where we developed a strong passion for optics, photonics, and the transformative potential of light-based technologies. Surrounded by cutting-edge research, we became fascinated by how photonics can be used to solve complex real-world problems, and we began asking ourselves how artificial intelligence could further enhance these systems. After connecting with two deeply passionate teammates with strong backgrounds in AI and optics, we found a shared vision: integrating intelligent computation with advanced optical systems to push the boundaries of visualization. Together, we were inspired by the challenge of overcoming the limitations of traditional 2D imaging and exploring how AI-powered depth estimation combined with holography could enable real-time 3D visualization. This project represents our collective passion for bridging photonics and AI to create smarter, faster, and more immersive imaging systems for the future. ### What it does NeuroHolo is a real-time AI-powered holographic imaging system that transforms live camera input into dynamic holographic projections. Our system captures visual data through a camera, uses AI-based depth estimation to reconstruct spatial information, and generates holographic phase patterns that are displayed through a spatial light modulator to produce a 3D visualization. By combining artificial intelligence with computational optics, NeuroHolo enables fast and adaptive hologram generation, allowing users to visualize objects with depth and spatial structure in real time. This creates new possibilities for applications in biomedical imaging, interactive visualization, and next-generation display technologies. ### How we built it We built NeuroHolo by integrating AI-driven computer vision with a real-time holographic optical system. First, a live camera captures visual input, which is processed using an AI depth estimation model to generate a depth map and recover spatial information from the scene. This data is then fed into our hologram generation pipeline, where computational algorithms convert the reconstructed 3D information into phase patterns suitable for holographic projection. These phase patterns are displayed on a Spatial Light Modulator, which modulates incoming coherent light to reconstruct the holographic image through a 4f optical system. By combining software, machine learning, and photonic hardware, we created an end-to-end pipeline capable of transforming real-world scenes into dynamic holographic visualizations in real time. ### Challenges we ran into One of the biggest challenges we faced was working with phase light modulators, as holographic projection and phase modulation were new concepts for much of our team. We had to quickly learn how phase patterns interact with coherent light and how small errors in alignment or calibration could significantly impact hologram quality. Another major challenge was limited access to specialized optical equipment, which required us to carefully optimize our setup using only the hardware available, including our camera, laser source, and optical components. On top of the technical challenges, we were also working under tight time constraints, which meant balancing rapid experimentation, debugging both software and hardware issues, and iterating on our AI and optical pipeline within a short development window. Despite these challenges, each obstacle pushed us to learn quickly, adapt creatively, and strengthen our interdisciplinary collaboration. ### Accomplishments we're proud of One of our biggest accomplishments was successfully generating and projecting a hologram without relying on an overly complex or time-intensive setup, making holographic visualization more accessible and practical for real-time applications. We are especially proud of how we creatively improvised with the resources available, including designing makeshift fixtures to precisely align our laser and optical components during early testing. Beyond the hardware, we also iterated through multiple machine learning approaches to improve depth estimation and hologram generation, continuously refining our pipeline to achieve better performance. Most importantly, we are proud of how our team embraced the entire process—from problem-solving and rapid prototyping to learning new concepts together—while keeping the experience fun, collaborative, and driven by curiosity. ### What we learned This project taught us that in optics, precision matters—especially when it comes to optical alignment, where even small misalignments can significantly affect system performance and hologram quality. We also learned that organization is essential when working with both hardware and software, as managing multiple components, iterations, and debugging steps requires clear communication and structure. Beyond the technical lessons, we learned the importance of teamwork and collaboration. By supporting each other, sharing ideas, and even taking breaks to play ping pong and recharge, we were able to maintain momentum and work through challenges together. This experience showed us that meaningful innovation comes not just from technical skill, but from patience, collaboration, and enjoying the process as a team. ### What's next for NeuralHolo Looking ahead, we aim to push NeuroHolo beyond a proof-of-concept into a scalable and impactful platform for real-time holographic imaging. On the technical side, we plan to expand toward multi-laser 3D reconstruction to improve depth fidelity, color accuracy, and overall hologram quality. We also want to further optimize our AI and computational pipeline to enable faster, low-latency phase generation for real-time holographic projection. From a product perspective, our goal is to develop a market-ready MVP that demonstrates practical use cases in fields such as biomedical imaging, scientific visualization, and immersive display technologies. We also plan to engage with potential users, researchers, and industry partners to better understand customer needs, refine our solution, and explore pathways for broader adoption and distribution. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 45 recognized source files, 693 KB. - Python (language) — detected in the code - PyTorch (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (53 of 53) ``` .gitignore 0.67_PLM_Hologram_Example.py benchmark_cnn_vs_plm.py benchmark_output/RESULTS.txt compare_pipelines.py DepthAnything/camera.py DepthAnything/depth_model.py DepthAnything/depth_stack.py DepthAnything/main.py DepthAnything/test.py DepthAnything/visualization.py fast_predict.py FastCGHNet.py HYBRID_ANALYSIS_SUMMARY.md hybrid_optimization_analysis.py hybrid_timing_analysis.py image_to_training_data.py kria_predict.py MultiImageShift.py one_shot_cgh.py PLM.py plmcontroller.py PLMCTRL_LOCAL_SETUP.md realtime_hologram_pipeline.py realtime_hologram_temporal.py requirements-kria.txt requirements.txt setup_kria.sh stereo_3d_mvp/.gitignore stereo_3d_mvp/body_mesh_common.py stereo_3d_mvp/calibrate_stereo.py stereo_3d_mvp/capture_calibration_pairs.py stereo_3d_mvp/fuse_stereo_view.py stereo_3d_mvp/FUSED_VIEW_README.md stereo_3d_mvp/list_cameras.py stereo_3d_mvp/live_face_hand_mesh.py stereo_3d_mvp/live_raw_depth_mask.py stereo_3d_mvp/live_reconstruct.py stereo_3d_mvp/live_stereo_mesh.py stereo_3d_mvp/make_chessboard_svg.py stereo_3d_mvp/README.md stereo_3d_mvp/reconstruct_face_hand.py stereo_3d_mvp/reconstruct_pair.py stereo_3d_mvp/requirements_face_hand.txt stereo_3d_mvp/requirements.txt stereo_3d_mvp/stereo_common.py stereo_3d_mvp/stereo_mesh_common.py TIMING_BREAKDOWN.txt TIPLMMonoMulti.py TIPLMMonoMultiRunner.py TIPLMSuiteHologram.py verify_perspective_shift.py vitis_ai_helper.py ``` ### Dependencies - requirements.txt: numpy@>=1.26, Pillow@>=10.0 - stereo_3d_mvp/requirements.txt: numpy@>=1.24, opencv-python@>=4.8 ### Recent commits (newest first) - hashing shit - Testing for Cnn and new architecture - Adding Hologram Temporal and TIPLMMonoMulti - Add stereo_3d_mvp: two-camera pipeline + stereo->RGB-D fusion mask - Faster CGH - New CHGNet - Creating pipline for realtime_hologram - Add TIPLMMonoMultiRunner for temporal multiplexing - Update gitignore, fast_predict (PyTorch/.pt inference), add compare_pipelines - clean somethings - PLMCTRL - Merge branch 'main' of https://github.com/IshTale/Hologram - Added DepthAnything pipeline - fit the whole image into the order-window box and center it on a black frame - Hologram Shift Test - The Bears and Such - Kria - Add files via upload - Added CNN - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### PLMCTRL_LOCAL_SETUP.md ```markdown # Local PLMCtrl Setup This folder is configured to use the Python wrapper and DLL from: ```text plmctrl-main/wrappers/PLMController.py plmctrl-main/bin/plmctrl.dll plmctrl-main/bin/BitpackHologramsCS.hlsl ``` The PLM video input configuration sent by `plmcontroller.py` uses `Parallel RGB`, `24-bit`, input port data swap `ABC -> ABC`, and HDMI by default. Use `--port-swap bac` only if your hardware setup was specifically verified with `ABC -> BAC`. Normal frame-sequence mode uses a windowed DirectX swapchain by default because that path initializes reliably during testing. Packed-bitplane mode uses the fullscreen/exclusive swapchain path by default for real PLM timing; pass `--windowed` only for debugging. Install Python dependencies: ```powershell python -m pip install -r requirements.txt ``` Verify that Python can load the wrapper and DLL without touching the PLM: ```powershell python .\plmcontroller.py --check-only ``` ## Normal frame sequence Display the first four built-in Bear TMUX frames as normal full frames, with no bitplane packing: ```powershell python .\plmcontroller.py ``` By default, normal mode uses frames `000` through `003`, uploads each image into its own PLMCtrl frame slot, and loops them through the PLMCtrl VSync-paced sequence player. This is video-frame playback, not PLM bitplane playback: expect about `30 fps` over HDMI or `60 fps` over DisplayPort. For normal `60 fps` playback, use DisplayPort: ```powershell python .\plmcontroller.py --connection displayport ``` Include all five built-in frames: ```powershell python .\plmcontroller.py --sequence-image-count 0 ``` For a slow visible check, use manual advance mode: ```powershell python .\plmcontroller.py --advance-mode manual --image-duration 1 ``` ## Packed bitplane playback Display the first four built-in Bear TMUX frames packed into one 24-bitplane RGB frame over HDMI: ```powershell python .\plmcontroller.py --playback-mode bitplane-package ``` Packed mode uses frames `000` through `003`, repeats them evenly across the 24 RGB bitplanes as `0, 1, 2, 3` six times, uploads one packed frame into PLMCtrl slot `0`, and uses HDMI timing: `30 Hz x 24 bitplanes = 720 Hz`. Packed mode expects the input BMPs to already be binary `0/255`, `2716 x 1600`, PLM-ready bitplane images. It does not generate valid phase holograms from grayscale photos or phase maps. If you deliberately want the old thresholding behavior, add `--allow-threshold-packed-images`. If your hardware actually needs `ABC -> BAC`, add the port swap option: ```powershell python .\plmcontroller.py --playback-mode bitplane-package --port-swap bac ``` Packed mode will not print `Displaying image 1/5` style messages because Python is not stepping through separate frames. The PLM reads the packed bitplanes inside one RGB frame. Use grouped bitplanes instead, six copies of each image: ```powershell python .\plmcontroller.py --playback-mode bitplane-package --bitplane-layout grouped ``` If you swit [truncated — 2056 more characters] ``` ### HYBRID_ANALYSIS_SUMMARY.md ```markdown # Hybrid CNN + PLM Optimization Analysis ## Executive Summary The analysis shows **how much time is spent in optimization** when using a hybrid approach (CNN initialization + PLM refinement). Key finding: **Early iterations are highly efficient, with 20 iterations recovering 75% of the quality gap in just 12% of full PLM time.** --- ## Core Results ### Timing Breakdown | Approach | Time | Quality | Speedup | Notes | |----------|------|---------|---------|-------| | **CNN only** | 713 ms | 2.37 dB | **81x** | Very fast, but poor quality | | CNN + 10 iter PLM | 4.7 s | 4.29 dB | **11x** | ~80% quality gain in minimal time | | CNN + 20 iter PLM | 6.7 s | 4.93 dB | **7.6x** | Sweet spot for interactive use | | CNN + 30 iter PLM | 9.1 s | 5.24 dB | **5.6x** | Good balance | | CNN + 50 iter PLM | 14.0 s | 5.68 dB | **3.7x** | High quality, still fast | | CNN + 100 iter PLM | 26.2 s | 6.03 dB | **1.9x** | Near-final quality | | Full PLM (200 iter) | 51.8 s | 6.30 dB | **1x** | Reference/maximum | ### PLM Optimization Time per Iteration - **10 iterations**: 3,943 ms total → **~394 ms per iteration** - **20 iterations**: 5,968 ms total → **~298 ms per iteration** - **50 iterations**: 13,296 ms total → **~266 ms per iteration** - **100 iterations**: 25,496 ms total → **~255 ms per iteration** - **200 iterations**: 51,058 ms total → **~255 ms per iteration** **Observation**: Early iterations are slower (~400ms) because they're doing the most work. By iteration 50+, convergence plateaus and iterations become more efficient (~255ms each). --- ## Efficiency Analysis: Quality Gain Per Time ### Diminishing Returns Curve ``` Quality Gain per Second of Optimization: 10 iterations: +0.49 dB/sec 20 iterations: +0.43 dB/sec 30 iterations: +0.34 dB/sec 50 iterations: +0.25 dB/sec 75 iterations: +0.18 dB/sec 100 iterations: +0.14 dB/sec 150 iterations: +0.10 dB/sec 200 iterations: +0.08 dB/sec ``` **Key insight**: The optimization becomes progressively less efficient. The first 50 iterations account for ~85% of the final quality, but take only 26% of total time. --- ## Time Breakdown for Different Configurations ### Real-time Streaming ``` Configuration: CNN only ├─ CNN inference: 713 ms ├─ PLM optimization: 0 ms └─ Total: 713 ms (81x faster than full PLM) Quality: 2.37 dB (poor, but usable for preview) Trade-off: Maximum speed, sacrifice quality ``` ### Interactive Preview (Quick) ``` Configuration: CNN + 10 PLM iterations ├─ CNN inference: 713 ms ├─ PLM optimization: 3,943 ms (first iteration spike) └─ Total: 4.7 seconds (11x faster than full PLM) Quality: 4.29 dB (+1.92 dB recovery from CNN baseline) Trade-off: Good speed, recovers ~50% of quality gap ``` ### Interactive Use (Balanced) ``` Configuration: CNN + 20 PLM iterations ├─ CNN inference: 713 ms ├─ PLM optimization: 5,968 ms └─ Total: 6.7 seconds (7.7x faster than full PLM) Quality: 4.93 dB (+2.56 dB recovery from CNN baseline) Trade-off: Still fast, recovers ~65 [truncated — 6730 more characters] ``` ### requirements.txt ``` numpy>=1.26 Pillow>=10.0 ``` ### stereo_3d_mvp/requirements.txt ``` numpy>=1.24 opencv-python>=4.8 ``` ### DepthAnything/main.py ```python import cv2 import time import numpy as np from camera import Camera, MultiCamera from depth_model import DepthEstimator from depth_stack import DepthStack from visualization import Visualizer class StereoCalibrator: def __init__(self, pattern_size=(8, 5), square_size=1.0, calib_file="stereo_calibration.npz"): self.pattern_size = pattern_size self.square_size = square_size self.calib_file = calib_file self.objpoints = [] self.imgpoints_left = [] self.imgpoints_right = [] self.maps = None self.image_size = None def _object_point(self, pattern_size=None): if pattern_size is None: pattern_size = self.pattern_size objp = np.zeros((pattern_size[0] * pattern_size[1], 3), np.float32) objp[:, :2] = np.mgrid[0:pattern_size[0], 0:pattern_size[1]].T.reshape(-1, 2) return objp * self.square_size def find_corners(self, gray): # Try several candidate patterns and preprocessing variants to improve robustness. flags = cv2.CALIB_CB_ADAPTIVE_THRESH | cv2.CALIB_CB_NORMALIZE_IMAGE used_pattern = None method = None try: clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8)) gray_clahe = clahe.apply(gray) except Exception: gray_clahe = gray gray_thresh = cv2.adaptiveThreshold( gray_clahe, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2 ) # Candidate patterns: user-specified, user-1 (squares->internal), and swapped candidates = [self.pattern_size, (max(1, self.pattern_size[0] - 1), max(1, self.pattern_size[1] - 1)), (self.pattern_size[1], self.pattern_size[0])] # Try on original, CLAHE, then threshold for pattern in candidates: if used_pattern is not None: break for img in (gray, gray_clahe, gray_thresh): found, corners = cv2.findChessboardCorners(img, pattern, flags) if found: used_pattern = pattern method = 'chessboard' break # Try SB detector as a last resort if used_pattern is None and hasattr(cv2, 'findChessboardCornersSB'): try: found, corners = cv2.findChessboardCornersSB(gray_clahe, self.pattern_size) if found: used_pattern = self.pattern_size method = 'chessboard_sb' except Exception: found = False # If chessboard wasn't found, try circles grid (symmetric) as a fallback if used_pattern is None: for pattern in candidates: if used_pattern is not None: break for img in (gray, gray_clahe, gray_thresh): try: flags_circ = cv2.CALIB_CB_SYMMETRIC_GRID found_c, corners_c = cv2.findCirclesGrid(img, pattern, flags=flags_circ) if found_c: used_pattern = pattern corners = corners_c method = 'circles' found = True break except Exception: found_c = False if used_pattern is not None: break if used_pattern is None: return False, None, None # refine corners criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001) try: corners = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria) except Exception: # cornerSubPix may fail for circle grids; ignore refinement in that case pass print(f"Detection: method={method}, pattern={used_pattern}") return True, corners, used_pattern def add_pair(self, left_gray, right_gray): found_left, corners_left, used_left = self.find_corners(left_gray) found_right, corners_right, used_right = self.find_corners(right_gray) if found_left and found_right and used_left == used_right: self.objpoints.append(self._object_point(pattern_size=used_left)) self.imgpoints_left.append(corners_left) self.imgpoints_right.append(corners_right) return True if not found_left and not found_right: print("Chessboard not found in either image.") elif not found_left: print("Chessboard not found in left image.") elif not found_right: print("Chessboard not found in right image.") else: print(f"Detected different patterns: left={used_left}, right={used_right}. Make sure both cameras see the same pattern.") return False def calibrate(self, image_size): if len(self.objpoints) < 6: raise RuntimeError("Need at least 6 calibration pairs.") self.image_size = image_size ret_left, mtx_left, dist_left, _, _ = cv2.calibrateCamera( self.objpoints, self.imgpoints_left, image_size, None, None ) ret_right, mtx_right, dist_right, _, _ = cv2.calibrateCamera( self.objpoints, self.imgpoints_right, image_size, None, None ) flags = cv2.CALIB_FIX_INTRINSIC criteria = (cv2.TERM_CRITERIA_MAX_ITER + cv2.TERM_CRITERIA_EPS, 30, 1e-6) _, mtx_left, dist_left, mtx_right, dist_right, R, T, _, _ = cv2.stereoCalibrate( self.objpoints, self.imgpoints_left, self.imgpoints_right, mtx_left, dist_left, mtx_right, dist_right, image_size, criteria=criteria, flags=flags, ) R1, R2, P1, P2, _, _, _ = cv2.stereoRectify( mtx_left, [truncated — 13497 more characters] ``` ### 0.67_PLM_Hologram_Example.py ```python from TIPLMSuiteHologram import CGHGenerator, DeviceLibrary import cv2 import numpy as np from pathlib import Path try: import torch except ImportError: torch = None # create instance of TI CGHGenerator and DeviceLibrary G = CGHGenerator() D = DeviceLibrary() # Define input parameters ViewImage1 = "./Bear1.png" # replace with first view image ViewImage2 = "./Bear2.png" # replace with second view image RandomSeed = 123 NumIter = 500 ViewShiftPixels = 330 # Near image_width / 4 for maximum wrapped-order separation on the 1358-wide PLM. PreparedDir = "./angle_mux_prepared" FitMode = "contain" # Preserve each photo's aspect ratio inside its order window. Options: "stretch", "crop", "contain". DitherMethod = "floyd_steinberg" # Options: "floyd_steinberg", "jarvis", "atkinson", "ordered". # Keep the order window narrow enough that the two angular orders do NOT overlap: # OrderWindowWidthFraction < viewShiftPixels / (image_width / 2) = 330 / 679 ~= 0.49. # 0.62 (the old value) violates this, so the two views bleed into each other. # With ConfineTargetsToOrderWindow=True the whole image is fit inside this window, # so a narrow window no longer "cuts content" -- do NOT raise it toward 1.0. OrderWindowWidthFraction = 0.38 OrderWindowHeightFraction = 1.0 OrderWindowFeatherFraction = 0.04 OutsideOrderWeight = 0.25 # Pushes light away from the left/right stray orders. ConfineTargetsToOrderWindow = True # Fit the whole image into the kept order window (full image per view, no overlap). if(RandomSeed is not None): np.random.seed(int(RandomSeed)) if(torch is not None): torch.manual_seed(int(RandomSeed)) # Call create angle-multiplexed hologram. # The two source images are stretched to 1358x800 and saved as photo-halftoned 1-bit BMPs. G.createAngleMultiplexedCGH(DeviceDictionary=D.defineDevice('0.67'), # Device Dictionary filenames=[ViewImage1, ViewImage2], # Two input view images outputPreparedDir=PreparedDir, # Stores 1358x800 1-bit BMP inputs viewShiftPixels=ViewShiftPixels, # Symmetric angular carrier separation numIter=NumIter, learningRate=0.08, threshold=0.5, dither=True, ditherMethod=DitherMethod, fitMode=FitMode, blackPercentile=1.0, whitePercentile=99.0, gamma=0.9, sharpenAmount=0.35, FlipUD=True, # 0.67 EVM requires an image flip. ShiftFOV=True, orderWindowWidthFraction=OrderWindowWidthFraction, orderWindowHeightFraction=OrderWindowHeightFraction, orderWindowFeatherFraction=OrderWindowFeatherFraction, outsideOrderWeight=OutsideOrderWeight, maskRecoveredOrders=True, confineTargetsToOrderWindow=ConfineTargetsToOrderWindow, showImages=False) outputRoot = Path(ViewImage1).stem + "_TO_" + Path(ViewImage2).stem G.writeCGHToFile(outputRoot + "_AngleMux_CGH.bmp") # Write the TI PLM mapped CGH to a file. # Optional: Display both reconstructed angular views. def display_normalize(image, high_percentile=99.5): image = np.asarray(image, dtype=np.float32) lo = float(np.percentile(image, 1.0)) hi = float(np.percentile(image, high_percentile)) if hi <= lo: return cv2.normalize(image, None, 0, 1, cv2.NORM_MINMAX) image = np.clip(image, lo, hi) return ((image - lo) / (hi - lo)).astype(np.float32) cv2.destroyAllWindows() cv2.imshow("Angle View 1", display_normalize(G.angleMuxRecovered_disc[0])) cv2.imshow("Angle View 2", display_normalize(G.angleMuxRecovered_disc[1])) cv2.waitKey(0) ``` ### setup_kria.sh ```shell #!/bin/bash # Quick setup script for KRIA deployment # Run on KRIA board: bash setup_kria.sh set -e echo "╔════════════════════════════════════════════════════════════╗" echo "║ FastCGHNet - AMD KRIA Setup ║" echo "╚════════════════════════════════════════════════════════════╝" # Check architecture ARCH=$(uname -m) echo "Architecture: $ARCH" if [[ ! "$ARCH" =~ armv7|aarch64 ]]; then echo "⚠️ This script is for ARM architecture" echo " Current: $ARCH" exit 1 fi # Detect if running on actual KRIA if [ -f /proc/device-tree/model ]; then BOARD=$(cat /proc/device-tree/model) echo "✓ Detected board: $BOARD" fi echo "" echo "═══════════════════════════════════════════════════════════════" echo "Step 1: Update system packages" echo "═══════════════════════════════════════════════════════════════" apt-get update -qq apt-get install -y python3-pip python3-dev echo "✓ System packages installed" echo "" echo "═══════════════════════════════════════════════════════════════" echo "Step 2: Install Python dependencies" echo "═══════════════════════════════════════════════════════════════" # Use pre-built wheels for ARM when available pip install --upgrade pip setuptools wheel -q echo "Installing ONNX Runtime for ARM..." pip install onnxruntime -q echo "Installing image processing libraries..." pip install opencv-python pillow numpy -q echo "✓ Python dependencies installed" echo "" echo "═══════════════════════════════════════════════════════════════" echo "Step 3: Verify installation" echo "═══════════════════════════════════════════════════════════════" python3 -c "import onnxruntime; print(f'✓ ONNX Runtime version: {onnxruntime.__version__}')" python3 -c "import cv2; print(f'✓ OpenCV version: {cv2.__version__}')" python3 -c "import numpy; print(f'✓ NumPy version: {numpy.__version__}')" echo "" echo "═══════════════════════════════════════════════════════════════" echo "Step 4: Benchmark (optional)" echo "═══════════════════════════════════════════════════════════════" read -p "Run benchmark? (y/n) " -n 1 -r echo if [[ $REPLY =~ ^[Yy]$ ]]; then if [ -f "fastcghnet_lite.onnx" ]; then echo "Benchmarking FastCGHNet..." python3 kria_predict.py --benchmark --model fastcghnet_lite.onnx else echo "⚠️ Model not found. Copy .onnx files first" fi fi echo "" echo "╔════════════════════════════════════════════════════════════╗" echo "║ Setup Complete! ║" echo "╚════════════════════════════════════════════════════════════╝" echo "" echo "Next steps:" echo " 1. Copy model files:" echo " scp models/fastcghnet_lite.onnx root@kria:/root/" echo "" echo " 2. Copy inference script:" echo " scp kria_predict.py root@kria:/root/" echo "" echo " 3. Run inference:" echo " python3 /root/kria_predict.py image.png --output hologram.bmp" echo "" echo "Documentation: See KRIA_DEPLOYMENT.md" ``` ### vitis_ai_helper.py ```python """ Vitis AI quantization and deployment helper for FastCGHNet Requires: Vitis AI SDK installed on KRIA or development machine """ import torch import numpy as np from pathlib import Path import os import sys def quantize_for_vitis_ai( model_path="models/best_model_lite.pt", output_dir="vitis_ai_models", quant_method="int8", calib_data=None, ): """ Prepare model for Vitis AI deployment with quantization Args: model_path: PyTorch model path output_dir: Output directory for quantized models quant_method: 'int8', 'int4', or 'mixed' calib_data: Calibration data for quantization Requires Vitis AI installed: pip install xilinx-vitis-ai """ print("╔════════════════════════════════════════════════════════╗") print("║ Vitis AI Quantization Helper ║") print("╚════════════════════════════════════════════════════════╝") try: from pytorch_quantization import quant_modules from pytorch_quantization import nn as quant_nn except ImportError: print("\n⚠️ Vitis AI quantization tools not installed") print("\nInstall Vitis AI:") print(" pip install xilinx-vitis-ai") print("\nOr use ONNX Runtime for CPU inference (no FPGA)") return None Path(output_dir).mkdir(exist_ok=True) print(f"\n[INFO] Loading model: {model_path}") from FastCGHNet import FastCGHNet device = torch.device("cpu") model = FastCGHNet(lite=True).to(device) checkpoint = torch.load(model_path, map_location=device) model.load_state_dict(checkpoint['model_state_dict']) # Prepare for quantization print(f"[INFO] Preparing quantization ({quant_method})...") quant_modules.initialize() # Disable quantization on first and last layers model.encoder[0].weight.disable_quantizer = True model.decoder[-2].weight.disable_quantizer = True print(f"[INFO] Quantization setup complete") print(f"\nTo deploy on KRIA with Vitis AI:") print(f" 1. Export to ONNX: python FastCGHNet.py export {model_path}") print(f" 2. Quantize: vai_q_pytorch --input_model model.onnx \\") print(f" --quant_mode {quant_method} \\") print(f" --output_dir {output_dir}/") print(f" 3. Compile for board: vai_c_onnx --model model_quantized.onnx \\") print(f" --arch /opt/vitis_ai/compiler/arch/kria/kv260.json \\") print(f" --output_dir compiled/") return model def install_vitis_ai(): """Helper to install Vitis AI on development machine""" print("\n[INFO] Installing Vitis AI...") print("\nOptions:") print(" 1. Docker (Recommended):") print(" docker pull xilinx/vitis-ai:latest") print(" docker run -it xilinx/vitis-ai bash") print("") print(" 2. Native (Ubuntu 20.04+):") print(" pip install xilinx-vitis-ai") print("") print(" 3. On KRIA board:") print(" apt-get install xilinx-vitis-ai-runtime") def build_docker_pipeline(): """Generate docker-compose for Vitis AI pipeline""" dockerfile = """FROM xilinx/vitis-ai:latest WORKDIR /workspace RUN pip install opencv-python pillow numpy onnx onnxruntime COPY FastCGHNet.py . COPY requirements.txt . COPY models/ models/ # Quantize model RUN vai_q_pytorch --input_model models/fastcghnet_lite.onnx \\ --quant_mode int8 \\ --output_dir quantized/ # Compile for KV260 RUN vai_c_onnx --model quantized/fastcghnet_lite_quantized.onnx \\ --arch /opt/vitis_ai/compiler/arch/kria/kv260.json \\ --output_dir compiled/ ENTRYPOINT ["/bin/bash"] """ docker_compose = """version: '3' services: vitis-ai: image: xilinx/vitis-ai:latest volumes: - .:/workspace working_dir: /workspace command: bash stdin_open: true tty: true """ print("\n[INFO] Docker pipeline files:") print("\nDockerfile:") print(dockerfile) print("\ndocker-compose.yml:") print(docker_compose) if __name__ == "__main__": import argparse parser = argparse.ArgumentParser(description="Vitis AI helper for FastCGHNet") parser.add_argument("--quantize", action="store_true", help="Quantize model") parser.add_argument("--model", default="models/best_model_lite.pt", help="Model path") parser.add_argument("--quant-method", default="int8", help="int8, int4, or mixed") parser.add_argument("--output-dir", default="vitis_ai_models", help="Output directory") parser.add_argument("--install", action="store_true", help="Show installation steps") parser.add_argument("--docker", action="store_true", help="Generate Docker pipeline") args = parser.parse_args() if args.install: install_vitis_ai() elif args.docker: build_docker_pipeline() elif args.quantize: quantize_for_vitis_ai(args.model, args.output_dir, args.quant_method) else: parser.print_help() ``` ### compare_pipelines.py ```python """ compare_pipelines.py ==================== Runs Bear1.png through the ONNX (FastCGHNet) pipeline and Bear2.png through the classic PLM.py iterative pipeline, then produces a side-by-side difference report. Usage: conda run -n Hologram python compare_pipelines.py """ import sys, time import numpy as np import cv2 from pathlib import Path sys.path.insert(0, str(Path(__file__).parent)) PLM_W, PLM_H = 1358, 800 OUT = Path("comparison_output") OUT.mkdir(exist_ok=True) # ── helpers ────────────────────────────────────────────────────────────────── def load_gray(path): img = cv2.imread(str(path), cv2.IMREAD_GRAYSCALE) if img is None: raise FileNotFoundError(path) return cv2.resize(img, (PLM_W, PLM_H), interpolation=cv2.INTER_AREA) def norm_u8(arr): a = arr.astype(np.float32) lo, hi = a.min(), a.max() return ((a - lo) / (hi - lo + 1e-8) * 255).astype(np.uint8) def ssim_simple(a, b): a, b = a.astype(np.float32)/255, b.astype(np.float32)/255 mu_a = cv2.GaussianBlur(a,(11,11),1.5) mu_b = cv2.GaussianBlur(b,(11,11),1.5) s2_a = cv2.GaussianBlur(a*a,(11,11),1.5) - mu_a**2 s2_b = cv2.GaussianBlur(b*b,(11,11),1.5) - mu_b**2 s_ab = cv2.GaussianBlur(a*b,(11,11),1.5) - mu_a*mu_b c1,c2 = 0.01**2, 0.03**2 return float(np.mean((2*mu_a*mu_b+c1)*(2*s_ab+c2) / ((mu_a**2+mu_b**2+c1)*(s2_a+s2_b+c2)+1e-10))) # ── Pipeline A: PyTorch (FastCGHNet .pt) ───────────────────────────────────── def run_pt(image_path, pt_path="models/best_model.pt"): import torch from FastCGHNet import FastCGHNet ckpt = torch.load(pt_path, map_location="cpu") model = FastCGHNet(lite=ckpt.get("lite", False)) model.load_state_dict(ckpt["model_state_dict"]) model.eval() print(f" Epoch {ckpt.get('epoch')} loss {ckpt.get('loss', '?'):.4f}") raw = load_gray(image_path).astype(np.float32) / 255.0 raw = (raw - raw.min()) / (raw.max() - raw.min() + 1e-8) x = torch.from_numpy(raw[np.newaxis, np.newaxis]) t0 = time.perf_counter() with torch.no_grad(): phase = model(x).squeeze().numpy() ms = (time.perf_counter() - t0) * 1000 from PLM import DeviceLibrary, CGHGenerator D = DeviceLibrary() G = CGHGenerator() dd = D.defineDevice("0.67") _, state = G.discretePhase(phase, dd["nLevel"], dd["pLevel"]) bmp = D.formatPLM(dd, state) return norm_u8(bmp), phase, ms # ── Pipeline B: Classic iterative (PLM.py / ADAM+GS) ───────────────────────── def run_plm(image_path, num_iter=200): from PLM import DeviceLibrary, CGHGenerator D = DeviceLibrary() G = CGHGenerator() dd = D.defineDevice("0.67") t0 = time.perf_counter() G.createCGH( dd, filename=str(image_path), alg="ADAMWGS", numIter=num_iter, initialPhase="Random", propMethod="Fourier", ShiftFOV=True, FlipUD=True, showImages=False, lossMode="mse", ) ms = (time.perf_counter() - t0) * 1000 bmp = G.CGH_mapped phase = G.CGH_output_cont return norm_u8(bmp), phase, ms # ── Main ────────────────────────────────────────────────────────────────────── print("=" * 62) print("PIPELINE COMPARISON") print("=" * 62) # --- A: PyTorch on Bear1 ----------------------------------------------------- print("\n[A] Bear1.png → FastCGHNet (.pt) …") try: bmp_a1, phase_a1, ms_a1 = run_pt("Bear1.png") cv2.imwrite(str(OUT / "Bear1_PT.bmp"), bmp_a1) print(f" Time : {ms_a1:.0f} ms") print(f" Saved: {OUT}/Bear1_PT.bmp") onnx1_ok = True except Exception as e: print(f" ❌ {e}") onnx1_ok = False # --- A2: PyTorch on Bear2 ---------------------------------------------------- print("\n[A2] Bear2.png → FastCGHNet (.pt) …") try: bmp_a2, phase_a2, ms_a2 = run_pt("Bear2.png") cv2.imwrite(str(OUT / "Bear2_PT.bmp"), bmp_a2) print(f" Time : {ms_a2:.0f} ms") print(f" Saved: {OUT}/Bear2_PT.bmp") onnx2_ok = True except Exception as e: print(f" ❌ {e}") onnx2_ok = False # --- B: PLM iterative on Bear2 ----------------------------------------------- print("\n[B] Bear2.png → PLM.py iterative (200 iter) …") try: bmp_b, phase_b, ms_b = run_plm("Bear2.png", num_iter=200) cv2.imwrite(str(OUT / "Bear2_PLM.bmp"), bmp_b) print(f" Time : {ms_b:.0f} ms") print(f" Saved: {OUT}/Bear2_PLM.bmp") plm_ok = True except Exception as e: print(f" ❌ {e}") plm_ok = False # --- Diff & metrics ---------------------------------------------------------- def compare(name_a, img_a, name_b, img_b): h = min(img_a.shape[0], img_b.shape[0]) w = min(img_a.shape[1], img_b.shape[1]) a, b = img_a[:h, :w], img_b[:h, :w] mse_v = float(np.mean((a.astype(np.float32) - b.astype(np.float32))**2)) psnr_v = 10 * np.log10(255**2 / (mse_v + 1e-10)) ssim_v = ssim_simple(a, b) print(f"\n {name_a} vs {name_b}") print(f" MSE : {mse_v:.1f} | PSNR: {psnr_v:.1f} dB | SSIM: {ssim_v:.4f}") return a, b, mse_v, ssim_v print("\n" + "=" * 62) print("DIFFERENCE METRICS") print("=" * 62) if onnx2_ok and plm_ok: a2, b, _, _ = compare("Bear2 PT", bmp_a2, "Bear2 PLM", bmp_b) diff = np.abs(a2.astype(np.float32) - b.astype(np.float32)) cv2.imwrite(str(OUT / "Bear2_PT_vs_PLM_diff.png"), cv2.applyColorMap(norm_u8(diff), cv2.COLORMAP_INFERNO)) if onnx1_ok and onnx2_ok: compare("Bear1 PT", bmp_a1, "Bear2 PT", bmp_a2) # 4-panel: Bear1-PT | Bear2-PT | Bear2-PLM | diff(PT vs PLM) imgs, labels = [], [] if onnx1_ok: imgs.append(cv2.cvtColor(bmp_a1, cv2.COLOR_GRAY2BGR)); labels.append("Bear1 PT") if onnx2_ok: imgs.append(cv2.cvtColor(bmp_a2, cv2.COLOR_GRAY2BGR)); labels.append("Bear2 PT") if plm_ok: imgs.append(cv2.cvtColor(bmp_ [truncated — 703 more characters] ``` ### DepthAnything/test.py ```python import torch print(torch.cuda.is_available()) print(torch.cuda.device_count()) print(torch.cuda.get_device_name(0)) ``` [33 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]