# Project export: Freestroke This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Freestroke lets users learn Chinese and other character-based languages by naturally drawing in the air, with computer vision detection and haptic feedback for practice, test, and comprehension modes. - Devpost: https://devpost.com/software/freestroke - GitHub: https://github.com/ashiriag/treehacks26 - Team: 3 GitHub contributor(s) — luke qiao (19 commits), ashiriag (9 commits), emadee05 (6 commits) ## Devpost submission (written by the team) ### Inspiration Learning character-based languages like Chinese can feel overwhelming, especially online. Apps often reduce writing to tapping pre-made strokes or tracing on a screen, which doesn’t build true muscle memory. Inspired by the physicality of handwriting practice and the importance of stroke order in languages like Chinese, we wanted to recreate the embodied experience of writing, but without pen and paper. Freestroke was born from the idea that language learning should be active, immersive, and intuitive. We combine motion, haptics, and computer vision, turning the air around you into a canvas. ### What it does Freestroke is an interactive language-learning app that lets users learn Chinese and other character-based languages by drawing characters in the air. Using computer vision, the system detects and interprets the user’s hand movements as character strokes in real time. The haptic feedback helps enhance the sense of physical interaction, making the input feel grounded and intentional. The app also provides accuracy feedback and signals mistakes, helping users internalize stroke order and structure. The app includes three core modes: Together, these modes build muscle memory, accuracy, and understanding. ### How we built it Freestroke is implemented as a real-time multimodal system primarily implemented in Python using OpenCV and MediaPipe. We used the MakeMeAHanzi dataset for the ground-truth Chinese strokes in our character stroke detection algorithm. The system integrates: The architecture is event-driven and frame-synchronous at ~30 FPS. MediaPipe Hand Tracking Freestroke uses MediaPipe’s real-time hand landmark detection pipeline to extract 21 3D keypoints per frame from the camera stream. We primarily track the midpoint of the user's thumb and pointer fingers as the drawing point, and continuously convert from camera frame coordinates to a normalized drawing space for character stroke detection. We reduce jitter via temporal smoothing to improve accuracy. Pinch-Based Gesture Control for Air Drawing Air writing requires a “pen-down” signal so that the user is able to write discrete strokes. Freestroke implements a hysteresis-based pinch detection controller, where a pinch is detected by computing the distance \(d = |P_{index} - P_{thumb}|\), and detecting when it falls below a certain threshold. Hysteresis here means that when the user pinches, the distance threshold increases to make it less likely for the pen to erroneously lift due to the distance jittering above and below the original threshold. This creates stable stroke segmentation boundaries and ensures deterministic start/end stroke events. When the "pen-down" signal is received, it also sends a WiFi signal to the ESP8266 microcontroller controlling the haptic feedback motor. Character Stroke Detection This is the main algorithm that powers Freestroke, allowing it to detect Chinese characters. This section describes the geometric process used for real-time stroke evaluation. The method consists of defining the active drawing region, mapping fingertip pixels into canonical character space, computing reference medians and tangents, estimating the user’s adaptive slope, comparing local directions, and applying final stroke acceptance criteria. All drawing and evaluation occurs within an active square region: \(\mathcal{B} = (x_0, y_0, x_1, y_1)\). By default, this region is computed automatically. Let the camera frame have width w and height h. We define a centered square occupying 85% of the smaller dimension: $$ s = 0.85 \cdot \min(w, h) $$ $$ x_0 = \frac{w - s}{2} $$ $$ y_0 = \frac{h - s}{2} $$ $$ x_1 = x_0 + s $$ $$ y_1 = y_0 + s $$ This region is recomputed each frame unless the user overrides it through calibration. If calibration is enabled, the user selects two opposite corners: \(c_1 = (x^{(1)}, y^{(1)})\) and \(c_2 = (x^{(2)}, y^{(2)})\). An axis-aligned rectangle is formed: $$ \tilde{x}_0 = \min(x^{(1)}, x^{(2)}) $$ $$ \tilde{y}_0 = \min(y^{(1)}, y^{(2)}) $$ $$ \tilde{x}_1 = \max(x^{(1)}, x^{(2)}) $$ $$ \tilde{y}_1 = \max(y^{(1)}, y^{(2)}) $$ To preserve canonical character proportions, the rectangle is converted into a square: $$ s = \min(\tilde{x}_1 - \tilde{x}_0,\; \tilde{y}_1 - \tilde{y}_0) $$ $$ x_0 = \tilde{x}_0 $$ $$ y_0 = \tilde{y}_0 $$ $$ x_1 = x_0 + s $$ $$ y_1 = y_0 + s $$ This calibrated square replaces the default drawing region. Reference stroke medians are defined in canonical display space: \([0,1024] \times [0,1024]\). Given fingertip pixel coordinates: \((x_{\text{px}}, y_{\text{px}})\), let: $$ b_w = x_1 - x_0 $$ $$ b_h = y_1 - y_0 $$ The normalization mapping into canonical coordinates is: $$ x_d = (x_{\text{px}} - x_0)\frac{1024}{b_w} $$ $$ y_d = (y_{\text{px}} - y_0)\frac{1024}{b_h} $$ This ensures that user strokes and reference medians are expressed in the same coordinate system. Each stroke is represented by a median polyline: \(m = {m_0, \dots, m_{K-1}}\). The median is resampled uniformly by arc length to produce a dense representation: \(r = {r_0, \dots, r_{N-1}}\). The reference arc length is: \(L_{\text{ref}} =\sum_{j=0}^{N-2}|r_{j+1} - r_j|\). The expected drawing direction at dense index \(j\) is approximated using central differences \(\tilde{t}_j\), so we can normalize to \(t_j =\frac{\tilde{t}_j}{|\tilde{t}_j|}\). The user stroke in canonical coordinates is: \(p = {p_0, \dots, p_T}\). The accumulated drawn arc length is: \(L_{\text{drawn}} = \sum_{i=0}^{T-1}|p_{i+1} - p_i|\). Given the newest user point \(p_T\), the closest reference point is determined as \(j^* = \arg\min_j |r_j - p_T|\). The expected direction at that location is \(t_{\text{ref}} = t_{j^*}\). To estimate the user’s local drawing direction, a trailing window of size W (typically 3) is used: $$ \tilde{u} = \sum_{i=T-W+1}^{T-1} (p_{i+1} - p_i) $$ $$ u = \frac{\tilde{u}} {|\tilde{u}|} $$ The angular deviation between user direction and reference tangent is: $$ \theta = \cos^{-1}\left( \mathrm{clip}(u \cdot t_{\text{ref}}, -1, 1) \right) $$ $$ \theta_{\deg} = \theta \frac{180}{\pi} $$ A segment is considered directionally correct if \(\theta_{\deg} \le 35^\circ\). Directional accuracy at stroke completion is: \(\text{DirPct} = 100 \cdot\frac{{\theta_{\deg} \le 35^\circ}}{\text{evaluated segments}}\). Length coverage is \(\text{LenPct} =100 \cdot\frac{L_{\text{drawn}}}{L_{\text{ref}}}\). A stroke is accepted if \(\text{DirPct} \ge 60\%\) and \(\text{LenPct} \ge 75\%\). In Free-Draw mode, stroke segmentation via pinch is disabled. The index finger alone controls drawing. A stroke begins automatically when the hand is detected and ends only after more than G_max consecutive frames without detection: $$ g > G_{\max} $$ To reduce jitter, a new point is appended only if: $$ |f_t - f_{t-1}| > \delta $$ where \(\delta\) is a fixed movement threshold (typically 5 pixels). Direction is still computed using the same short-window aggregation shown above, which provides temporal smoothing and stable stroke rendering. WiFi-Based Custom Haptic Feedback System As previously mentioned, when the "pen-down" signal is received, the computer sends a WiFi signal to the ESP8266 microcontroller that is connected to the same network. The microcontroller sets one of its GPIO pins high, turning on a NPN transistor that allows for the required current to be supplied from the power source to power a DC motor and LED. We attached an asymmetrical servo arm to the DC motor to generate the vibrations. The firmware also constantly checks the strength of the WiFi RSSI level (typically around -20dB to -30dB) and continuously monitors the connection. Real-Time UI Overlay The UI overlay in the Zoom camera feed displays buttons for each of the modes that can be selected using the same pinch gesture for writing. We also allow the user to rescale the size of a bounding box that defines the area in which the character is drawn. We also display the stroke feedback and accuracy metrics so that they are easily visible to the user. ### Challenges we ran into One of the biggest challenges was integrating live gesture recognition with Zoom in a way that actually felt seamless for teaching. Our system uses the computer camera to track hand movements and render stroke paths in real time, which we then stream into Zoom using a virtual camera setup. Getting this pipeline stable was non-trivial, and we tried different approaches to ensure the most robust and useable interface. Another major challenge came from the nature of Chinese characters themselves. Unlike simple gesture systems that recognize straight lines or isolated shapes, many characters contain strokes that are long, curved, or change direction mid-stroke. This made stroke tracking and matching significantly harder. We had to design logic that could interpret fluid, continuous motion rather than just discrete segments. On top of that, we wanted the UI and feedback system to feel natural, not overly strict, but still accurate enough to teach proper stroke order and structure. If the matching tolerance was too tight, users would get frustrated because their writing “looked right” but failed recognition. Too loose, and the educational value dropped. Balancing this required iterating on stroke thresholds, visual guides, and feedback cues so that writing felt intuitive while still pedagogically meaningful. ### Accomplishments we're proud of We’re especially proud of building a fully working system that lets users write Chinese characters in the air and see their strokes rendered live on screen. Turning hand motion into structured stroke data, and doing it in real time, was a major milestone for us. Seeing characters form naturally from gesture alone felt like bringing calligraphy into a digital, interactive space. We’re also proud of successfully creating a teaching workflow that works inside live video calls. By routing our rendered stroke feed through a virtual camera, we made it possible for instructors to demonstrate writing live while students follow along. This transforms what is usually a static screen-share experience into something far more dynamic and engaging. Another accomplishment was developing stroke-matching logic that can handle complex, multi-directional characters. Instead of limiting recognition to simple gestures, our system can interpret longer, curved, and compound strokes, which is essential for accurately representing real Chinese writing. Finally, we're proud of integrating hardware into this project in the form of a custom haptic feedback motor and controller that communicates over a WiFi link. This made the user experience feel much more satisfying and interactive. ### What we learned One of the biggest things we learned was how to bridge computer vision systems with real-time user interaction. Hand tracking on its own is a solved problem in many demos, but making it reliable enough for teaching required deeper work in smoothing motion, filtering noise, and interpreting intent from imperfect gestures. We gained a much better understanding of how small variations in tracking data can dramatically affect downstream recognition. We also learned how important latency and visual feedback are in learning tools. Even slight delays between a hand movement and the rendered stroke made the experience feel disconnected. This pushed us to think carefully about rendering pipelines, frame processing, and how to keep the system feeling responsive and “alive” for both instructors and students. Another key learning was around the structure of Chinese writing itself. Implementing stroke tracking forced us to study stroke order, directionality, and how complex characters are composed. We developed a new appreciation for how nuanced character writing is, especially when translating it into computational representations like stroke paths and matching algorithms. Finally, we learned the value of balancing technical accuracy with user experience. A system that is perfectly precise but frustrating to use fails as a teaching tool. Iterating on tolerance levels, guidance overlays, and feedback messaging taught us how to design for learning, not just recognition. This mindset shift, from building a cool demo to building something pedagogically useful, was one of our most important takeaways. ### What's next Extend support for more character-based languages like Korean or Japanese Miniaturize haptic feedback motor circuit with improved hardware for better user experience Personalized learning feedback by tracking stroke smoothness, velocity, and curvature habits Incorporate large character datasets and crowdsourced trajectories Optimize latency with GPU inference and microcontroller-side haptics for <40 ms end-to-end. ## README (from the GitHub repository) # Freestroke ### Inspiration Learning character-based languages like Chinese can feel overwhelming, especially online. Apps often reduce writing to tapping pre-made strokes or tracing on a screen, which doesn’t build true muscle memory. Inspired by the physicality of handwriting practice and the importance of stroke order in languages like Chinese, we wanted to recreate the embodied experience of writing, but without pen and paper. Freestroke was born from the idea that language learning should be active, immersive, and intuitive. We combine motion, haptics, and computer vision, turning the air around you into a canvas. ### What it does Freestroke is an interactive language-learning app that lets users learn Chinese and other character-based languages by drawing characters in the air. Using computer vision, the system detects and interprets the user’s hand movements as character strokes in real time. The haptic feedback helps enhance the sense of physical interaction, making the input feel grounded and intentional. The app also provides accuracy feedback and signals mistakes, helping users internalize stroke order and structure. The app includes three core modes: - Practice Mode: Guided character writing with stroke-by-stroke feedback. - Test Mode: Independent writing with scoring based on accuracy and stroke order. - Comprehension Mode: Reinforces recognition and meaning through reading and contextual exercises. Together, these modes build muscle memory, accuracy, and understanding. ### How we built it Freestroke is implemented as a real-time multimodal system primarily implemented in Python using OpenCV and MediaPipe. We used the MakeMeAHanzi dataset for the ground-truth Chinese strokes in our character stroke detection algorithm. The system integrates: - MediaPipe hand tracking - Pinch-based gesture control for air drawing - Character stroke detection - WiFi-based custom haptic feedback system - Real-time UI overlay The architecture is event-driven and frame-synchronous at ~30 FPS. ## Features ### Teaching Mode - See outline of Chinese character, English definition, pinyin - Real-time arrows showing stroke direction and feedback with character stroke accuracy - Complete one stroke at a time with visual feedback ### Pinyin Recognition Mode - See the pinyin and recall the character - Real-time feedback on stroke accuracy ### English Translation Mode - See English word, write the corresponding character ### WiFi-Based Custom Haptic Feedback System When the "pen-down" signal is received, the computer sends a WiFi signal to the ESP8266 microcontroller that is connected to the same network. The microcontroller sets one of its GPIO pins high, turning on a NPN transistor that allows for the required current to be supplied from the power source to power a DC motor and LED. We attached an asymmetrical servo arm to the DC motor to generate the vibrations. The firmware also constantly checks the strength of the WiFi RSSI level (typically around -20dB to -30dB) and continuously monitors the connection. ### Real-Time UI Overlay The UI overlay in the Zoom camera feed displays buttons for each of the modes that can be selected using the same pinch gesture for writing. We also allow the user to rescale the size of a bounding box that defines the area in which the character is drawn. We also display the stroke feedback and accuracy metrics so that they are easily visible to the user. --- ## Quick Start ### Installation ```bash cd /Users/lukeqiao/Documents/Projects/treehacks_2026 uv venv uv sync uv run launcher.py ``` ### First Run 1. Press **1**, **2**, or **3** to choose a learning mode 2. Start writing characters with your finger in front of the camera 3. Press **SPACE** to submit your work or move to the next character --- ## Learning Modes ### Teaching Mode (Press 1) Learn proper stroke technique with guided instructions. ``` Teaching Mode - 一 (one) Pinyin: yi1 ➜ ➜ ➜ ➜ (animated arrow) ───────────────── (stroke 1) Stroke 1 / 1 ``` **How it works:** - Animated arrow shows stroke direction - Semi-transparent guide shows exact path - Real-time validation after each stroke - Move to next stroke automatically on success ### Pinyin Recognition Mode (Press 2) Practice recalling characters from their sound. ``` Pinyin Mode Write the character that sounds like: shui3 (water) [Your drawing here] Score: 0150 ``` **Scoring:** - Correct character: +150 points - Incorrect: 0 points (try again) - Build your vocabulary systematically ### English Translation Mode (Press 3) The ultimate memory challenge with gamification. ``` Translation Mode Write the character for: WATER [Your drawing here] Completed: 5 Score: 00850 ✓ Correct! 水 ``` --- ## ⌨️ Keyboard Controls | Key | Action | |-----|--------| | **1** | Teaching Mode | | **2** | Pinyin Recognition Mode | | **3** | English Translation Mode | | **SPACE** | Submit drawing / Next character | | **C** | Clear current drawing | | **M** | Return to mode selection | | **Q** | Quit application | --- ## Zoom Integration ### Setup for Virtual Teaching 1. **Start Zoom meeting** 2. **Launch tutor**: `python main_app.py` 3. **Share screen**: Click "Share Screen" → Select tutor window 4. **Everyone sees**: Live character learning with feedback! ### Perfect For: - Virtual Chinese classes - Group tutoring sessions - Hybrid learning (in-person + Zoom) - Student demonstrations - Interactive practice sessions ### Best Practices: - Use **Teaching Mode** for demonstrations - Use **Pinyin/English Modes** for interactive practice - Ask students to draw in their own cameras while you evaluate - Keep window at native resolution for clarity --- ## 🔧 Technical Architecture ### Core Components **`main_app.py`** - Main application - Mode management, state, scoring, event handling **`stroke_engine.py`** - Recognition engine - DTW stroke matching algorithm - Character recognition & validation - Stroke order verification **`ui_renderer.py`** - Visual rendering - Template stroke drawing - Animated arrow generation - UI panels & feedback **`zoom_integration.py`** - Zoom optimization - Screen share compatibility - Setup instructions ### Stroke Matching Algorithm We use **Dynamic Time Warping (DTW)** with angle scoring: ``` Score = DTW_Distance + (Angle_Penalty × 0.3) Match = Score ≤ Threshold ``` **Why DTW?** - ✓ Handles speed variations (fast/slow writing) - ✓ Accounts for individual handwriting styles - ✓ Sensitive to stroke direction mistakes - ✓ Forgiving for minor deviations --- ## Installation & Troubleshooting ### Prerequisites - Python 3.8+ - Webcam with 30+ FPS - Camera permissions enabled ### Troubleshooting **"Camera failed to initialize"** - macOS: Settings → Privacy & Security → Camera → Grant access to Terminal/IDE - Windows: Check camera in Device Manager - Restart the application **"Strokes not detecting"** - Improve lighting in your room - Move finger closer to camera (but keep full hand visible) - Ensure minimum finger movement (MOVE_THRESHOLD = 5 pixels) **"Strokes not matching"** - Stroke must be drawn in the correct direction - Try different handwriting styles - it learns from you - Practice a few times - accuracy improves with familiarity **"Low FPS / Lag"** - Close unnecessary applications - Update camera driver - Lower resolution if needed (edit WINDOW_WIDTH/HEIGHT in main_app.py) --- ## File Structure ``` treehacks_2026/ ├── main_app.py # Main application (run this!) ├── stroke_engine.py # Recognition & validation ├── ui_renderer.py # Visual rendering ├── zoom_integration.py # Zoom setup & helpers ├── characters.json # Character database ├── requirements.txt # Python dependencies └── README.md # This file ``` --- ## Tech Stack - **MediaPipe**: Real-time hand detection - **OpenCV**: Image processing & rendering - **DTW Algorithm**: Dynamic Time Warping for stroke matchin [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 173 recognized source files, 735 KB. - C (language) — detected in the code - C++ (language) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - Python (language) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 203) ``` .gitignore .gitmodules .vscode/settings.json characters.py docs/BACKEND_FLOW.md draw.py experiments/arrow_stroke_error.py experiments/arrow_stroke.py experiments/calibration.py experiments/character_detection.py experiments/config.py experiments/draw_svg.py experiments/draw_windows.py experiments/draw.py experiments/drawing.py experiments/grid.py experiments/main.py experiments/send_wifi_cmd.py experiments/test_1.py experiments/test_2.py experiments/test_21.py experiments/zoom_integration.py haptic_feedback/.DS_Store haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/.piopm haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/ArduinoJson.h haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/examples/JsonConfigFile/JsonConfigFile.ino haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/examples/JsonFilterExample/JsonFilterExample.ino haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/examples/JsonGeneratorExample/JsonGeneratorExample.ino haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/examples/JsonHttpClient/JsonHttpClient.ino haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/examples/JsonParserExample/JsonParserExample.ino haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/examples/JsonServer/JsonServer.ino haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/examples/JsonUdpBeacon/JsonUdpBeacon.ino haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/examples/MsgPackParser/MsgPackParser.ino haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/examples/ProgmemExample/ProgmemExample.ino haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/examples/StringExample/StringExample.ino haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/library.json haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/library.properties haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/LICENSE.txt haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/README.md haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson.h haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Array/ArrayData.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Array/ArrayImpl.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Array/ElementProxy.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Array/JsonArray.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Array/JsonArrayConst.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Array/JsonArrayIterator.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Array/Utilities.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Collection/CollectionData.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Collection/CollectionImpl.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/compatibility.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Configuration.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Deserialization/DeserializationError.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Deserialization/DeserializationOptions.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Deserialization/deserialize.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Deserialization/Filter.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Deserialization/NestingLimit.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Deserialization/Reader.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Deserialization/Readers/ArduinoStreamReader.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Deserialization/Readers/ArduinoStringReader.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Deserialization/Readers/FlashReader.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Deserialization/Readers/IteratorReader.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Deserialization/Readers/RamReader.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Deserialization/Readers/StdStreamReader.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Deserialization/Readers/VariantReader.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Document/JsonDocument.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Json/EscapeSequence.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Json/JsonDeserializer.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Json/JsonSerializer.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Json/Latch.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Json/PrettyJsonSerializer.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Json/TextFormatter.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Json/Utf16.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Json/Utf8.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Memory/Alignment.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Memory/Allocator.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Memory/MemoryPool.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Memory/MemoryPoolList.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Memory/ResourceManager.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Memory/ResourceManagerImpl.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Memory/StringBuffer.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Memory/StringBuilder.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Memory/StringNode.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Memory/StringPool.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Misc/SerializedValue.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/MsgPack/endianness.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/MsgPack/ieee754.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/MsgPack/MsgPackBinary.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/MsgPack/MsgPackDeserializer.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/MsgPack/MsgPackExtension.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/MsgPack/MsgPackSerializer.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Namespace.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Numbers/arithmeticCompare.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Numbers/convertNumber.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Numbers/FloatParts.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Numbers/FloatTraits.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Numbers/JsonFloat.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Numbers/JsonInteger.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Numbers/parseNumber.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Object/JsonObject.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Object/JsonObjectConst.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Object/JsonObjectIterator.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Object/JsonPair.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Object/MemberProxy.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Object/ObjectData.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Object/ObjectImpl.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Polyfills/alias_cast.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Polyfills/assert.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Polyfills/attributes.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Polyfills/ctype.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Polyfills/integer.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Polyfills/limits.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Polyfills/math.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Polyfills/mpl/max.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Polyfills/pgmspace_generic.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Polyfills/pgmspace.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Polyfills/preprocessor.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Polyfills/type_traits.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Polyfills/type_traits/conditional.hpp haptic_feedback/.pio/libdeps/esp12e/ArduinoJson/src/ArduinoJson/Polyfills/type_traits/decay.hpp [83 more files omitted for size] ``` ### Dependencies - pyproject.toml: colour@>=0.1.5, flask-cors@>=6.0.2, flask-socketio@>=5.6.0, mediapipe@>=0.10.32, numpy@>=2.4.2, opencv-python@>=4.13.0.92, pillow@>=12.1.1, python-dotenv@>=1.2.1, requests@>=2.32.5, shapely@>=2.1.2, svg.path@>=7.0 - zoom/requirements.txt: eventlet, Flask@==2.3.2, Flask-CORS@==4.0.0, Flask-SocketIO@==5.3.4, gunicorn, python-dotenv, python-engineio@==4.7.1, python-socketio@==5.9.0, requests@==2.31.0 ### Recent commits (newest first) - moved to experiments - updated wifi network - added esp8266 integration - Fix formatting of 'Inspiration' section header - Update README for Freestroke app details - asejae - select characters - Merge branch 'main' of https://github.com/lkqiao/treehacks26 - wrjw - fix - Merge branch 'main' of https://github.com/lkqiao/treehacks26 - done - buttons - Merge branch 'main' of https://github.com/lkqiao/treehacks26 - fawef - requirements update for render - login - added hint and login maybe - fix bug - fasef ## Key source files (fetched from GitHub, selected and truncated for size) ### docs/BACKEND_FLOW.md ```markdown # Backend flow: who talks to whom ``` ┌─────────────────┐ ┌──────────────────────────────────────────┐ │ Browser │ │ launcher.py (port 5001) │ │ (user) │ │ ┌─────────────────────────────────────┐ │ │ │ HTTP │ │ Flask + SocketIO server │ │ │ Opens │ ──────► │ │ • GET / → "WebSocket ..." │ │ │ http:// │ │ │ • GET /api/* → (see below) │ │ │ localhost:5050 │ │ │ • SocketIO: set_character, │ │ │ │ │ │ set_mode, trigger_action │ │ │ (page served │ │ └──────────────┬──────────────────────┘ │ │ by zoom/app) │ │ │ │ └────────┬────────┘ │ ▼ │ │ │ ┌─────────────────────────────────────┐ │ │ SocketIO │ │ tutor_app_instance = TutorApp() │ │ │ (to zoom:5050) │ │ → set_character_remote(char) │ │ │ │ │ → set_mode_remote(mode) │ │ ▼ │ │ → handle_action_remote(action) │ │ ┌─────────────────┐ │ └─────────────────────────────────────┘ │ │ zoom/app.py │ HTTP │ (main_app.TutorApp — the OpenCV window) │ │ (port 5050) │ ──────► │ │ │ │ POST │ Same process: WebSocket thread + │ │ Serves │ /api/ │ main thread running TutorApp.run() │ │ index.html │ ... └──────────────────────────────────────────┘ └─────────────────┘ ``` ## Which file interacts with which | File | Role | Interacts with | |------|------|-----------------| | **launcher.py** | Entry point. Starts WebSocket server (port 5001) in a thread and runs **main_app.TutorApp** in the main thread. | **main_app.py** (holds `tutor_app_instance` and calls its `set_character_remote`, `set_mode_remote`, `handle_action_remote`). | | **main_app.py** | The actual tutor (camera, drawing, modes). Defines `TutorApp`. | No direct reference to launcher or zoom. It only reacts when launcher calls the three `*_remote` methods. | | **zoom/app.py** | Web UI server (port 5050). Serves the page and tries to control the tutor. | **launcher.py** — by HTTP to `LAUNCHER_URL` (e.g. `http://localhost:5001`) at `/api/character`, `/api/mode`, `/api/action`, and `GET /health`. | So: **launcher.py** is the only thing that talks to **main_app.py**. **zoom/app.py** talks to **launcher.py** (intended via HTTP or later WebSocket). **main_app.py** does not import or call launcher or zoom. ## Event flow (when it’s wired up) 1. User does something in the browser (e.g. picks a character). 2. Frontend sends to zoom/app (e.g. `POST /api/send-character` or SocketIO `send_character`). 3. zoom/app forwards to launcher (e.g. `POST http://localhost:5001/api/character`). 4. Launcher recei [truncated — 593 more characters] ``` ### pyproject.toml ``` [project] name = "treehacks26" version = "0.1.0" description = "Add your description here" readme = "README.md" requires-python = ">=3.11" dependencies = [ "mediapipe>=0.10.32", "numpy>=2.4.2", "opencv-python>=4.13.0.92", "shapely>=2.1.2", "svg.path>=7.0", "requests>=2.32.5", "pillow>=12.1.1", "colour>=0.1.5", "flask-socketio>=5.6.0", "flask-cors>=6.0.2", "python-dotenv>=1.2.1", ] ``` ### zoom/requirements.txt ``` Flask==2.3.2 requests==2.31.0 Flask-CORS==4.0.0 Flask-SocketIO==5.3.4 python-socketio==5.9.0 python-engineio==4.7.1 python-dotenv gunicorn eventlet ``` ### experiments/main.py ```python def main(): print("Hello from treehacks26!") if __name__ == "__main__": main() ``` ### zoom/app.py ```python import os import requests from flask import Flask, render_template, request, redirect, url_for, jsonify from flask_socketio import SocketIO, emit from flask_cors import CORS app = Flask(__name__) app.config['SECRET_KEY'] = 'zoom-tutor-secret' CORS(app) socketio = SocketIO(app, cors_allowed_origins="*") CLIENT_ID = os.getenv("ZOOM_CLIENT_ID") CLIENT_SECRET = os.getenv("ZOOM_CLIENT_SECRET") ACCOUNT_ID = os.getenv("ZOOM_ACCOUNT_ID") # Local launcher endpoint (set via environment variable) # Example: "wss://abcd1234.ngrok.io" or "ws://localhost:5001" for local testing LAUNCHER_WS_URL = os.getenv("LAUNCHER_WS_URL", "ws://localhost:5001") LAUNCHER_URL = os.getenv("LAUNCHER_URL", "http://localhost:5001") CHINESE_CHARACTERS = ["你", "好", "学", "习", "书"] # your 5 demo characters def get_s2s_token(): url = "https://zoom.us/oauth/token" params = {"grant_type": "account_credentials", "account_id": ACCOUNT_ID} auth = (CLIENT_ID, CLIENT_SECRET) r = requests.post(url, params=params, auth=auth) r.raise_for_status() return r.json()["access_token"] def create_meeting(token): url = "https://api.zoom.us/v2/users/me/meetings" headers = {"Authorization": f"Bearer {token}", "Content-Type": "application/json"} payload = {"topic": "Language Learning", "type": 1, "settings": {"join_before_host": True, "approval_type": 0, # automatically approve participants "waiting_room": False }} r = requests.post(url, headers=headers, json=payload) r.raise_for_status() return r.json() @app.route("/", methods=["GET"]) def index(): return render_template("index.html", launcher_ws_url=LAUNCHER_WS_URL) @app.route("/start-meeting", methods=["POST"]) def start_meeting(): try: token = get_s2s_token() meeting = create_meeting(token) join_url = meeting.get("join_url") # show_chars=False: show the Zoom link first return render_template("index.html", join_url=join_url, show_chars=False, launcher_ws_url=LAUNCHER_WS_URL) except Exception as e: return render_template("index.html", error=str(e)) @app.route("/choose-characters", methods=["GET"]) def choose_characters(): # show 5 characters return render_template("index.html", show_chars=True, characters=CHINESE_CHARACTERS, launcher_ws_url=LAUNCHER_WS_URL) @app.route("/select-character", methods=["POST"]) def select_character(): selected = request.form.get("character") return render_template("index.html", selected_char=selected, launcher_ws_url=LAUNCHER_WS_URL) # ===== API ENDPOINTS TO COMMUNICATE WITH LOCAL LAUNCHER ===== @app.route("/api/send-character", methods=["POST"]) def send_character(): """Send a character to the local launcher.""" try: data = request.json character = data.get("character") response = requests.post( f"{LAUNCHER_URL}/api/character", json={"character": character}, timeout=5 ) response.raise_for_status() return jsonify({"status": "ok", "message": f"Sent character: {character}"}) except requests.exceptions.ConnectionError: return jsonify({ "status": "error", "message": "Could not reach launcher. Is it running? Set LAUNCHER_URL env var." }), 500 except Exception as e: return jsonify({"status": "error", "message": str(e)}), 500 @app.route("/api/send-mode", methods=["POST"]) def send_mode(): """Change mode on the local launcher.""" try: data = request.json mode = data.get("mode") # 1, 2, or 3 response = requests.post( f"{LAUNCHER_URL}/api/mode", json={"mode": mode}, timeout=5 ) response.raise_for_status() mode_names = {1: "Teaching", 2: "Pinyin Recognition", 3: "English Translation"} return jsonify({"status": "ok", "message": f"Mode set to: {mode_names.get(mode)}"}) except requests.exceptions.ConnectionError: return jsonify({"status": "error", "message": "Could not reach launcher"}), 500 except Exception as e: return jsonify({"status": "error", "message": str(e)}), 500 @app.route("/api/send-action", methods=["POST"]) def send_action(): """Trigger an action on the local launcher.""" try: data = request.json action = data.get("action") # "submit", "clear", "next", etc response = requests.post( f"{LAUNCHER_URL}/api/action", json={"action": action}, timeout=5 ) response.raise_for_status() return jsonify({"status": "ok", "message": f"Action triggered: {action}"}) except requests.exceptions.ConnectionError: return jsonify({"status": "error", "message": "Could not reach launcher"}), 500 except Exception as e: return jsonify({"status": "error", "message": str(e)}), 500 @app.route("/api/launcher-status", methods=["GET"]) def launcher_status(): """Check if the launcher is running.""" try: response = requests.get(f"{LAUNCHER_URL}/health", timeout=2) response.raise_for_status() return jsonify({"status": "online", "launcher": response.json()}) except: return jsonify({"status": "offline", "launcher_url": LAUNCHER_URL}) # ===== WEBSOCKET ENDPOINTS ===== @socketio.on('connect') def handle_connect(): """Handle WebSocket connection from web client.""" print("[WebSocket] Web client connected") emit('connection_response', {'data': 'Connected to web app'}) @socketio.on('disconnect') def handle_disconnect(): """Handle WebSocket disconnection.""" print("[WebSocket] Web client disconnected") @socketio.on('send_character') def handle_send_character(data): """Forward character change to launcher via WebSocket.""" character = data.get('character') print(f"[WebSocket] Sending character: {character}") # This would connect to launcher's WebSocket and forward the m [truncated — 736 more characters] ``` ### draw.py ```python import cv2 import mediapipe as mp import numpy as np import sys # ------------------------------- # MediaPipe Hand Setup # ------------------------------- mp_hands = mp.solutions.hands hands = mp_hands.Hands( max_num_hands=1, min_detection_confidence=0.7, min_tracking_confidence=0.7 ) mp_draw = mp.solutions.drawing_utils # ------------------------------- # Webcam Setup (macOS built-in camera) # ------------------------------- CAMERA_INDEX = 0 # Adjust if needed cap = cv2.VideoCapture(CAMERA_INDEX, cv2.CAP_AVFOUNDATION) if not cap.isOpened(): print("Error: Camera failed to initialize.") print("Check System Settings → Privacy & Security → Camera") print("Make sure Terminal / IDE has camera access.") sys.exit(1) # ------------------------------- # Drawing Setup # ------------------------------- canvas = None strokes = [] drawing = False prev_point = None MOVE_THRESHOLD = 5 # Minimum movement to add a point Z_THRESHOLD = -0.05 # Max z for “finger close to screen” (adjust experimentally) # ------------------------------- # Main Loop # ------------------------------- while True: ret, frame = cap.read() if not ret: print("Warning: Failed to read frame from camera") break frame = cv2.flip(frame, 1) h, w, _ = frame.shape if canvas is None: canvas = np.zeros_like(frame) rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) results = hands.process(rgb) finger_detected = False drawing_allowed = False # NEW: for UI indicator if results.multi_hand_landmarks: hand_landmarks = results.multi_hand_landmarks[0] # Index fingertip tip = hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP] x, y, z = int(tip.x * w), int(tip.y * h), tip.z # Only draw if finger is “close enough” if z < Z_THRESHOLD: drawing_allowed = True finger_detected = True # Green fingertip indicator cv2.circle(frame, (x, y), 12, (0, 255, 0), -1) if not drawing: drawing = True prev_point = (x, y) strokes.append([(x, y)]) else: if not strokes: strokes.append([(x, y)]) dist = np.linalg.norm(np.array(prev_point) - np.array((x, y))) if dist > MOVE_THRESHOLD: cv2.line(canvas, prev_point, (x, y), (0, 255, 0), 5) strokes[-1].append((x, y)) prev_point = (x, y) else: drawing_allowed = False # Red fingertip indicator (not drawing) cv2.circle(frame, (x, y), 12, (0, 0, 255), -1) # Finger lifted → end stroke if not finger_detected and drawing: drawing = False prev_point = None print(f"Stroke finished. Total strokes: {len(strokes)}") # Draw hand landmarks for feedback if results.multi_hand_landmarks: mp_draw.draw_landmarks(frame, hand_landmarks, mp_hands.HAND_CONNECTIONS) # Drawing status text (put on frame BEFORE blending) if drawing_allowed: cv2.putText(frame, "DRAWING ENABLED", (20, 40), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2) else: cv2.putText(frame, "LIFT FINGER (NOT DRAWING)", (20, 40), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 2) # Overlay canvas on webcam feed combined = cv2.addWeighted(frame, 0.7, canvas, 0.3, 0) cv2.imshow("Finger Drawing MVP", combined) key = cv2.waitKey(1) if key == 27: # ESC → quit break elif key == ord("c"): # Clear canvas canvas = np.zeros_like(frame) strokes = [] drawing = False prev_point = None print("Canvas cleared") cap.release() cv2.destroyAllWindows() ``` ### launcher.py ```python #!/usr/bin/env python3 """ Quick launcher for the Chinese Character Tutor Run this to start the application with one command """ import sys import os import subprocess import threading import json from flask import Flask, request, jsonify from flask_socketio import SocketIO, emit, disconnect from flask_cors import CORS def print_banner(): banner = """ ╔══════════════════════════════════════════════════════════╗ ║ ║ ║ Chinese Character Tutor ║ ║ ║ ║ Learn to write Chinese with real-time feedback ║ ║ Powered by MediaPipe hand detection & DTW matching ║ ║ ║ ╚══════════════════════════════════════════════════════════╝ """ print(banner) def check_dependencies(): """Check if all required packages are installed.""" required = ['cv2', 'mediapipe', 'numpy'] missing = [] for package in required: try: __import__(package) except ImportError: missing.append(package) if missing: print(f"[ERROR] Missing dependencies: {', '.join(missing)}") print("Run: uv sync") return False print("All dependencies installed") return True def check_camera(): """Check if camera is accessible.""" import cv2 cap = cv2.VideoCapture(0) if not cap.isOpened(): print("[ERROR] Camera not accessible") print("macOS: System Preferences > Security & Privacy > Camera > Grant access") print("Windows: Check camera in Device Manager") cap.release() return False cap.release() print("Camera accessible") return True def check_character_data(): """Check if MakeMeAHanzi graphics.txt exists and characters module loads.""" graphics_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'makemeahanzi', 'graphics.txt') if not os.path.exists(graphics_path): print("[ERROR] makemeahanzi/graphics.txt not found") return False try: from characters import CHARACTER_LIST print(f"Character database loaded ({len(CHARACTER_LIST)} characters)") except Exception as e: print(f"[ERROR] Failed to load characters module: {e}") return False return True # Global state for the tutor app tutor_app_instance = None socketio = None ui_state = { "current_character": None, "mode": 1, # 1=Teaching, 2=Pinyin, 3=Translation } def start_websocket_server(): """Start a Flask-SocketIO server for real-time communication.""" global socketio flask_app = Flask(__name__) flask_app.config['SECRET_KEY'] = 'chinese-tutor-secret' CORS(flask_app) socketio = SocketIO(flask_app, cors_allowed_origins="*") @flask_app.route("/", methods=["GET"]) def index(): return "WebSocket Server Running" @socketio.on('connect') def handle_connect(): print("[WebSocket] Client connected") emit('connection_response', {'data': 'Connected to launcher'}) @socketio.on('disconnect') def handle_disconnect(): print("[WebSocket] Client disconnected") @socketio.on('set_character') def handle_set_character(data): """Receive character change from web app.""" global tutor_app_instance character = data.get('character') ui_state['current_character'] = character print(f"[WEB] Character set to: {character}") if tutor_app_instance: tutor_app_instance.set_character_remote(character) emit('character_updated', {'character': character}, broadcast=True) @socketio.on('set_mode') def handle_set_mode(data): """Receive mode change from web app.""" global tutor_app_instance mode = data.get('mode') ui_state['mode'] = mode mode_names = {1: "Teaching", 2: "Pinyin Recognition", 3: "English Translation"} print(f"[WEB] Mode set to: {mode_names.get(mode, 'Unknown')}") if tutor_app_instance: tutor_app_instance.set_mode_remote(mode) emit('mode_updated', {'mode': mode}, broadcast=True) @socketio.on('trigger_action') def handle_trigger_action(data): """Receive action from web app.""" global tutor_app_instance action = data.get('action') print(f"[WEB] Action triggered: {action}") if tutor_app_instance: tutor_app_instance.handle_action_remote(action) emit('action_completed', {'action': action}, broadcast=True) @socketio.on('get_state') def handle_get_state(): """Send current state to client.""" emit('state_update', ui_state) print("\n" + "=" * 60) print("WEBSOCKET SERVER STARTED") print("=" * 60) print("Local WebSocket: ws://localhost:5001") print("\nTo expose to web app:") print(" 1. Install ngrok: https://ngrok.com/download") print(" 2. Run: ngrok http 5001") print(" 3. Use wss://your-ngrok-url in web app settings") print("=" * 60 + "\n") socketio.run(flask_app, host="0.0.0.0", port=5001, debug=False, allow_unsafe_werkzeug=True) def main(): print_banner() print("\nPre-launch checks...") checks = [ ("Dependencies", check_dependencies), ("Camera", check_camera), ("Character DB", check_character_data) ] all_passed = True for name, check_fn in checks: try: if not check_fn(): all_passed = False except Exception as e: print(f"[ERROR] {name}: {str(e)}") all_passed = False if not all_passed: print("\n[WARNING] Some checks failed. Fix issues above and try again.") sys.exit(1) print("\n [truncated — 1001 more characters] ``` ### stroke_engine.py ```python """ Stroke Recognition & Validation Engine - Matches user-drawn strokes to template strokes - Validates stroke order and correctness - Detects character recognition """ import numpy as np import json import os from typing import List, Tuple, Dict, Optional # =============================== # Geometry & Normalization Utils # =============================== def dist(a, b): """Euclidean distance between two points.""" a = np.array(a, dtype=np.float32) b = np.array(b, dtype=np.float32) return float(np.linalg.norm(a - b)) def polyline_length(pts): """Total length of a polyline.""" if len(pts) < 2: return 0.0 s = 0.0 for i in range(1, len(pts)): s += dist(pts[i - 1], pts[i]) return s def resample_polyline(pts, n=64): """Resample polyline to exactly n points spaced by arc-length.""" if len(pts) == 0: return [(0.0, 0.0)] * n if len(pts) == 1: return [pts[0]] * n pts = [(float(x), float(y)) for x, y in pts] L = polyline_length(pts) if L < 1e-6: return [pts[0]] * n dists = [0.0] for i in range(1, len(pts)): dists.append(dists[-1] + dist(pts[i - 1], pts[i])) targets = np.linspace(0.0, dists[-1], n) out = [] j = 0 for t in targets: while j < len(dists) - 2 and dists[j + 1] < t: j += 1 d0, d1 = dists[j], dists[j + 1] p0, p1 = np.array(pts[j]), np.array(pts[j + 1]) if abs(d1 - d0) < 1e-9: out.append(tuple(p0)) else: alpha = (t - d0) / (d1 - d0) p = p0 + alpha * (p1 - p0) out.append((float(p[0]), float(p[1]))) return out def normalize_points(pts): """Normalize stroke: translate to centroid, scale to unit box.""" arr = np.array(pts, dtype=np.float32) if len(arr) == 0: return arr c = np.mean(arr, axis=0) arr = arr - c minxy = np.min(arr, axis=0) maxxy = np.max(arr, axis=0) size = np.max(maxxy - minxy) if size < 1e-6: size = 1.0 arr = arr / size return arr def dtw_distance(a, b): """DTW distance between two sequences of 2D points.""" a = np.array(a, dtype=np.float32) b = np.array(b, dtype=np.float32) n, m = len(a), len(b) if n == 0 or m == 0: return 1e9 dp = np.full((n + 1, m + 1), np.inf, dtype=np.float32) dp[0, 0] = 0.0 for i in range(1, n + 1): for j in range(1, m + 1): cost = np.linalg.norm(a[i - 1] - b[j - 1]) dp[i, j] = cost + min(dp[i - 1, j], dp[i, j - 1], dp[i - 1, j - 1]) return float(dp[n, m] / (n + m)) # =============================== # Stroke Matching # =============================== def calculate_stroke_angle(pts): """Calculate primary direction angle of a stroke (0-360 degrees).""" if len(pts) < 2: return 0.0 pts_arr = np.array(pts, dtype=np.float32) start = pts_arr[0] end = pts_arr[-1] delta = end - start angle = np.arctan2(delta[1], delta[0]) * 180.0 / np.pi # Normalize to 0-360 if angle < 0: angle += 360 return angle def stroke_match_score(user_pts, template_pts, resample_n=64): """ Score how well user stroke matches template. Returns score (0 = perfect, higher = worse). """ if len(user_pts) < 3: return 1e9 # Resample both strokes user_resampled = resample_polyline(user_pts, resample_n) template_resampled = resample_polyline(template_pts, resample_n) # Normalize to unit space user_norm = normalize_points(user_resampled) template_norm = normalize_points(template_resampled) # DTW distance dtw = dtw_distance(user_norm, template_norm) # Angle consistency (prefer same direction as template) user_angle = calculate_stroke_angle(user_pts) template_angle = calculate_stroke_angle(template_pts) angle_diff = abs(user_angle - template_angle) # Normalize angle difference to 0-180 if angle_diff > 180: angle_diff = 360 - angle_diff angle_penalty = angle_diff / 180.0 * 0.3 # 30% weight final_score = dtw + angle_penalty return final_score def match_stroke_to_template( user_strokes: List[List[Tuple[float, float]]], template_strokes: List[List[Tuple[float, float]]], threshold: float = 0.25 ) -> Dict: """ Match user-drawn strokes to template strokes. Returns: { 'matched': bool, 'correct_strokes': int, 'wrong_strokes': [], 'missing_strokes': [], 'accuracy': float (0-1) } """ n_template = len(template_strokes) matched_count = 0 wrong_strokes = [] for i, user_stroke in enumerate(user_strokes): if i >= n_template: # Extra strokes wrong_strokes.append({"stroke_idx": i, "reason": "extra_stroke"}) continue template_stroke = template_strokes[i] score = stroke_match_score(user_stroke, template_stroke) if score <= threshold: matched_count += 1 else: wrong_strokes.append({ "stroke_idx": i, "reason": "incorrect_shape", "score": score }) # Missing strokes missing_count = max(0, n_template - len(user_strokes)) missing_strokes = list(range(len(user_strokes), n_template)) accuracy = matched_count / max(1, n_template) matched = (accuracy >= 0.8) # 80% threshold for "correct" return { "matched": matched, "correct_strokes": matched_count, "total_strokes": n_template, "wrong_strokes": wrong_strokes, "missing_strokes": missing_strokes, "accuracy": accuracy } # =============================== # Character Recognition # =============================== class CharacterDatabase: """Load and manage character templates.""" def __init__(self, json_path: str = "characters.json"): [truncated — 4130 more characters] ``` ### ui_renderer.py ```python """ UI and Rendering Layer - Handles all visual display - Zoom-compatible rendering - Animations and feedback """ import cv2 import numpy as np from typing import List, Tuple, Optional, Dict import math import time class UIRenderer: """Main UI rendering class.""" def __init__(self, width: int = 1280, height: int = 720): self.width = width self.height = height self.frame_time = time.time() self.animation_frame = 0 def draw_template_stroke( self, canvas: np.ndarray, stroke: List[Tuple[float, float]], color: Tuple[int, int, int] = (200, 200, 200), thickness: int = 3, alpha: float = 0.5 ) -> np.ndarray: """Draw a template stroke (normalized 0-1 coordinates).""" h, w = canvas.shape[:2] # Draw area: center with padding padding = 50 drawing_width = w - 2 * padding drawing_height = h - 2 * padding x_offset = padding y_offset = padding # Convert normalized coords to pixel coords points = [] for norm_x, norm_y in stroke: px = x_offset + norm_x * drawing_width py = y_offset + norm_y * drawing_height points.append((int(px), int(py))) if len(points) < 2: return canvas # Draw with transparency overlay = canvas.copy() for i in range(len(points) - 1): cv2.line(overlay, points[i], points[i+1], color, thickness) cv2.addWeighted(overlay, alpha, canvas, 1 - alpha, 0, canvas) return canvas def draw_strokes( self, canvas: np.ndarray, strokes: List[List[Tuple[float, float]]], colors: Optional[List[Tuple[int, int, int]]] = None, thickness: int = 4 ) -> np.ndarray: """Draw multiple strokes on canvas.""" h, w = canvas.shape[:2] padding = 50 drawing_width = w - 2 * padding drawing_height = h - 2 * padding x_offset = padding y_offset = padding if colors is None: colors = [(0, 255, 0)] * len(strokes) for stroke, color in zip(strokes, colors): points = [] for norm_x, norm_y in stroke: px = x_offset + norm_x * drawing_width py = y_offset + norm_y * drawing_height points.append((int(px), int(py))) for i in range(len(points) - 1): cv2.line(canvas, points[i], points[i+1], color, thickness) return canvas def draw_animated_arrow( self, canvas: np.ndarray, stroke: List[Tuple[float, float]], animation_progress: float = 0.5, color: Tuple[int, int, int] = (255, 100, 0) ) -> np.ndarray: """ Draw animated arrow showing stroke direction. animation_progress: 0 = start, 1 = end """ h, w = canvas.shape[:2] padding = 50 drawing_width = w - 2 * padding drawing_height = h - 2 * padding x_offset = padding y_offset = padding if len(stroke) < 2: return canvas # Convert normalized coords points = [] for norm_x, norm_y in stroke: px = x_offset + norm_x * drawing_width py = y_offset + norm_y * drawing_height points.append(np.array([px, py], dtype=np.float32)) # Get segment based on animation progress total_len = sum(np.linalg.norm(points[i+1] - points[i]) for i in range(len(points)-1)) target_len = total_len * animation_progress cumulative = 0.0 arrow_start = None arrow_end = None for i in range(len(points) - 1): seg_len = np.linalg.norm(points[i+1] - points[i]) if cumulative + seg_len >= target_len: # The arrow is in this segment alpha = (target_len - cumulative) / seg_len arrow_start = points[i] + alpha * (points[i+1] - points[i]) # Arrow end (look ahead) look_ahead_len = 30 if i + 1 < len(points) - 1: next_seg_len = np.linalg.norm(points[i+2] - points[i+1]) if seg_len - alpha * seg_len + next_seg_len >= look_ahead_len: alpha2 = look_ahead_len / (seg_len - alpha * seg_len) arrow_end = points[i+1] + alpha2 * (points[i+2] - points[i+1]) else: arrow_end = arrow_start + (points[i+1] - points[i]) / np.linalg.norm(points[i+1] - points[i]) * look_ahead_len else: arrow_end = arrow_start + (points[i+1] - points[i]) / np.linalg.norm(points[i+1] - points[i]) * look_ahead_len break cumulative += seg_len if arrow_start is not None and arrow_end is not None: arrow_start = tuple(map(int, arrow_start)) arrow_end = tuple(map(int, arrow_end)) # Draw arrow line cv2.line(canvas, arrow_start, arrow_end, color, 3) # Draw arrowhead direction = np.array(arrow_end) - np.array(arrow_start) direction = direction / (np.linalg.norm(direction) + 1e-6) # Arrowhead size arrow_size = 15 angle = np.arctan2(direction[1], direction[0]) pt1 = arrow_end - arrow_size * np.array([np.cos(angle - np.pi/6), np.sin(angle - np.pi/6)]) pt2 = arrow_end - arrow_size * np.array([np.cos(angle + np.pi/6), np.sin(angle + np.pi/6)]) cv2.line(canvas, arrow_end, tuple(map(int, pt1)), color, 3) cv2.line(canvas, arrow_end, tuple(map(int, pt2)), color, 3) return canvas [truncated — 5698 more characters] ``` ### experiments/send_wifi_cmd.py ```python # import requests # import time # ESP_IP = "172.20.10.12" # replace with your ESP IP # def set_drawing(state: bool): # url = f"http://{ESP_IP}/drawing" # payload = {"drawing": state} # try: # requests.post(url, json=payload, timeout=0.1) # except: # pass # # Test # while True: # set_drawing(True) # time.sleep(1) # set_drawing(False) # time.sleep(1) import requests import time ESP_IP = "172.20.10.12" URL = f"http://{ESP_IP}/drawing" def set_drawing(state: bool): # Do NOT reuse a Session here (ESP8266 often closes keep-alive) for attempt in range(5): try: r = requests.post(URL, json={"drawing": state}, timeout=2) # optional: print(r.status_code, r.text) return True except requests.exceptions.RequestException as e: print(f"Request failed (attempt {attempt+1}/5): {e}") time.sleep(0.2) return False while True: print("Turning on drawing...") set_drawing(True) time.sleep(2) print("Turning off drawing...") set_drawing(False) time.sleep(2) ``` [161 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]