# Project export: Show and Tell - Capturing Emotion in Sign Language This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Enabling emotion and expression for the hard of hearing. Through innovative wearable technology and LLM integration, Show and Tell tackles the issue of accessibility for the hard of hearing. - Devpost: https://devpost.com/software/show-and-tell-capturing-emotion-in-sign-language - GitHub: https://github.com/raviriley/aidactyl - Result: winner (Most Creative Hack (4x Nintendo Switches); Human Capital: Memorial Partnership Journey (Box @ Warriors Game)) - Team: 2 GitHub contributor(s) — Ravi Riley (19 commits), ethanjags (2 commits) ## Devpost submission (written by the team) ### Inspiration Our team focuses extensively on opportunities to ignite change through innovation. In this pursuit, we began to investigate the impact of sign-language glove technology on the hard of hearing community. In this, we learned that despite previous efforts to enhance accessibility, feedback from deaf advocates highlighted a critical gap: earlier technologies often facilitated communication for hearing individuals with the deaf, rather than empowering the deaf to communicate on their terms. After discussing this problem with a friend of ours who faces disabilities related to her hearing, we realized that this problem impacts many people's daily lives, significantly affecting their ability to engage with those around them. By focusing on human centered design and integrating feedback presented in numerous journals, we solve these problems by developing an accessible, easy to use interface that enables the hard of hearing and mute to converse seamlessly. Through integration of a wearable component and a sophisticated LLM, we completely change the landscape of interpersonal communications for the hard of hearing. ### What it does Our solution consists of two components, a wearable glove and a mobile video call interface. The wearable glove is meant to be utilized by a deaf or hard of hearing individual when conversing with another person. This glove, fitted with numerous flex sensors and an Inertial Measurement Unit (IMU), can discern what gloss (term for a word in ASL) the wearer is signing at any moment in time. From here, the data moves to the second component of the solution - the mobile video call interface. Here, the user's signs are converted into both text and speech during a call. The text is displayed on the screen while the speech is stated, ensuring to include emotional cues as picked up by the integrated computer vision model. This effectively helps users communicate with others, especially loved ones, in a manner that accurately represents their intent and emotions. This experience is one that is currently not offered anywhere else on the market. In tandem, both of these technologies enable us to understand body language, emotion, and signs from a user, and also help vocalize the feelings of a person who is hard of hearing. ### How we built it Two vastly different components call for drastically different approaches. However, we needed to ensure that these two approaches still stayed true to the same intent. We first began by identifying our design strategy, based in our problem statement and objective. From here, we moved forward with set goals and milestones. On the hardware side of things, we spent an extensive amount of time in the on-site lab fabricating our prototype. In order to ensure the validity of our design, we researched circuit diagrams and characteristics, ultimately building our own. We performed a variety of tests on this prototype, including practical use testing by taking it around campus while interacting with others. The glove withstood numerous handshakes and even a bit of rain! On the software side, we also had two problems to face - interfacing with the glove, and creating the mobile application. To interface with the glove, we began with the Arduino IDE for testing. After we ensured that our design was functional and gained test data, we moved to a python implementation that sends sensed words up to an API, which can later be accessed by the mobile application. Moving to the mobile application, we utilized SwiftUI for our design. From there, we used the StreamAPI to build a FaceTime style infrastructure. We prototyped and integrated between Figma designs and our prototype to best understand where we could increase capabilities and improve the user experience. ### Challenges we ran into This project was ambitious, and as such, was also chock full of complications. Initially, we faced extensive challenges on the hardware side. Due to the nature of the design, we have many components that are trying to draw power or ground from the same source. This provided increased complexity in our manufacturing process, as we had to come up with an innovative solution to a sleek design that maintained functionality. Even after we found our first solution, our prototype was inconsistent due to manufacturing flaws. On the last day, 2 hours before the submission deadline, we completely disassembled and rebuilt our prototype using a new methodology. This proved to be successful, minimizing the issues seen previously and resulting in an amazing product. On the software side, we also pursued ambitious desires that didn't distinctly align with our team's expertise. Due to this, we faced great difficulty when troubleshooting the numerous errors we faced in initial implementation. This set us back quite extensively, but we were able to successfully recover. ### Accomplishments we're proud of We are proud of the magnitude of success we were able to show in the short frame of this hackathon. We came in knowing that we had ambitious and lofty goals, but were unsure if we would truly be able to achieve them. Thankfully, we complete this hackathon with a functional, viable MVP that clearly represents our goals and desires for this project. ### What we learned Because of the cross discipline nature of this project. All of our team members got the opportunity to explore new spaces. Through collaboration, we all learned about these fields and technologies from and with each other and how we can integrate them into our systems in the future. We also learned about best practices for manufacturing in general. Additionally, we were able to become more comfortable with SwiftUI and creating our own APIs for our video calling component. These valuable skills shaped our experiences at TreeHacks and will stick with us for many years to come. ### What's next We hope to continue to pursue this idea and bring independence to the hard of hearing population worldwide. In a market that has underserved the deaf population, we see Show and Tell as the optimal solution for accessibility. In the future, we want to flesh out the hardware prototype further by investing in custom PCB's, streamlining the production process and making it much more professional. Additionally, we want to build out functionality within the video calling app, adding in as many helpful features as possible. ## README (from the GitHub repository) # Show and Tell Hardware (smart glove) + Software (emotion-powered AI) to enable ASL speakers to engage in natural language, real time conversations with non-signers ## Detected evidence (automated analysis) Indexed codebase: 838 recognized source files, 2721 KB. - C (language) — detected in the code - Flask (technology) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - Ruby (language) — detected in the code - Swift (language) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 1019) ``` .gitattributes .gitignore backend/backend/__init__.py backend/backend/app.py backend/backend/full.ipynb backend/backend/image_receive.py backend/poetry.lock backend/pyproject.toml backend/README.md backend/requirements.txt backend/scripts/__init__.py backend/scripts/scripts.py backend/tests/__init__.py README.md ShowAndTell/ShowAndTell.xcodeproj/project.pbxproj ShowAndTell/ShowAndTell.xcodeproj/project.xcworkspace/contents.xcworkspacedata ShowAndTell/ShowAndTell.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist ShowAndTell/ShowAndTell.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved ShowAndTell/ShowAndTell.xcodeproj/project.xcworkspace/xcuserdata/raviriley.xcuserdatad/UserInterfaceState.xcuserstate ShowAndTell/ShowAndTell.xcodeproj/xcuserdata/raviriley.xcuserdatad/xcdebugger/Breakpoints_v2.xcbkptlist ShowAndTell/ShowAndTell.xcodeproj/xcuserdata/raviriley.xcuserdatad/xcschemes/xcschememanagement.plist ShowAndTell/ShowAndTell/AppViewModel.swift ShowAndTell/ShowAndTell/ASLViewModel.swift ShowAndTell/ShowAndTell/Assets.xcassets/AccentColor.colorset/Contents.json ShowAndTell/ShowAndTell/Assets.xcassets/AppIcon.appiconset/Contents.json ShowAndTell/ShowAndTell/Assets.xcassets/Contents.json ShowAndTell/ShowAndTell/CameraFeedManager.swift ShowAndTell/ShowAndTell/ContentView.swift ShowAndTell/ShowAndTell/CustomCallControlsView.swift ShowAndTell/ShowAndTell/CustomViewFactory.swift ShowAndTell/ShowAndTell/EmotionViewModel.swift ShowAndTell/ShowAndTell/MainOverlayView.swift ShowAndTell/ShowAndTell/Preview Content/Preview Assets.xcassets/Contents.json ShowAndTell/ShowAndTell/ShowAndTellApp.swift ShowAndTell/ShowAndTell/Snapshots/StreamSnapshotTrigger.swift stream-video-swift-main/.github/actions/bootstrap/action.yml stream-video-swift-main/.github/actions/python-cache/action.yml stream-video-swift-main/.github/actions/ruby-cache/action.yml stream-video-swift-main/.github/actions/xcode-cache/action.yml stream-video-swift-main/.github/CODEOWNERS stream-video-swift-main/.github/ISSUE_TEMPLATE/bug_report.md stream-video-swift-main/.github/ISSUE_TEMPLATE/feature_request.md stream-video-swift-main/.github/PULL_REQUEST_TEMPLATE.md stream-video-swift-main/.github/stale.yml stream-video-swift-main/.github/workflows/cron-checks.yml stream-video-swift-main/.github/workflows/docusaurus.yml stream-video-swift-main/.github/workflows/publish-release.yml stream-video-swift-main/.github/workflows/smoke-checks.yml stream-video-swift-main/.github/workflows/testflight.yml stream-video-swift-main/.github/workflows/update-copyright.yml stream-video-swift-main/.github/workflows/vale-doc-lint.yml stream-video-swift-main/.gitignore stream-video-swift-main/.slather.yml stream-video-swift-main/.spi.yml stream-video-swift-main/.styles/Google/Acronyms.yml stream-video-swift-main/.styles/Google/AMPM.yml stream-video-swift-main/.styles/Google/Colons.yml stream-video-swift-main/.styles/Google/Contractions.yml stream-video-swift-main/.styles/Google/DateFormat.yml stream-video-swift-main/.styles/Google/Ellipses.yml stream-video-swift-main/.styles/Google/EmDash.yml stream-video-swift-main/.styles/Google/EnDash.yml stream-video-swift-main/.styles/Google/Exclamation.yml stream-video-swift-main/.styles/Google/FirstPerson.yml stream-video-swift-main/.styles/Google/Gender.yml stream-video-swift-main/.styles/Google/GenderBias.yml stream-video-swift-main/.styles/Google/HeadingPunctuation.yml stream-video-swift-main/.styles/Google/Headings.yml stream-video-swift-main/.styles/Google/Latin.yml stream-video-swift-main/.styles/Google/LyHyphens.yml stream-video-swift-main/.styles/Google/meta.json stream-video-swift-main/.styles/Google/OptionalPlurals.yml stream-video-swift-main/.styles/Google/Ordinal.yml stream-video-swift-main/.styles/Google/OxfordComma.yml stream-video-swift-main/.styles/Google/Parens.yml stream-video-swift-main/.styles/Google/Passive.yml stream-video-swift-main/.styles/Google/Periods.yml stream-video-swift-main/.styles/Google/Quotes.yml stream-video-swift-main/.styles/Google/Ranges.yml stream-video-swift-main/.styles/Google/Semicolons.yml stream-video-swift-main/.styles/Google/Slang.yml stream-video-swift-main/.styles/Google/Spacing.yml stream-video-swift-main/.styles/Google/Spelling.yml stream-video-swift-main/.styles/Google/Units.yml stream-video-swift-main/.styles/Google/vocab.txt stream-video-swift-main/.styles/Google/We.yml stream-video-swift-main/.styles/Google/Will.yml stream-video-swift-main/.styles/Google/WordList.yml stream-video-swift-main/.styles/Vocab/Base/accept.txt stream-video-swift-main/.styles/Vocab/Base/reject.txt stream-video-swift-main/.swiftformat stream-video-swift-main/.vale.ini stream-video-swift-main/Brewfile stream-video-swift-main/Brewfile.lock.json stream-video-swift-main/CallIntent/Info.plist stream-video-swift-main/CallIntent/IntentHandler.swift stream-video-swift-main/CHANGELOG.md stream-video-swift-main/Dangerfile stream-video-swift-main/DemoApp/Info.plist stream-video-swift-main/DemoApp/Preview Content/Preview Assets.xcassets/Contents.json stream-video-swift-main/DemoApp/Resources/AppIntentVocabulary.plist stream-video-swift-main/DemoApp/Resources/Assets/Assets.xcassets/AccentColor.colorset/Contents.json stream-video-swift-main/DemoApp/Resources/Assets/Assets.xcassets/AppIcon.appiconset/Contents.json stream-video-swift-main/DemoApp/Resources/Assets/Assets.xcassets/capture.imageset/Contents.json stream-video-swift-main/DemoApp/Resources/Assets/Assets.xcassets/Contents.json stream-video-swift-main/DemoApp/Resources/Assets/Assets.xcassets/logo.imageset/Contents.json stream-video-swift-main/DemoApp/Resources/Assets/Assets.xcassets/video.imageset/Contents.json stream-video-swift-main/DemoApp/Resources/Assets/Assets.xcassets/viewfinder.imageset/Contents.json stream-video-swift-main/DemoApp/Resources/Entitlements/DemoApp-Debug.entitlements stream-video-swift-main/DemoApp/Resources/Entitlements/DemoApp.entitlements stream-video-swift-main/DemoApp/Resources/GoogleSignIn.plist stream-video-swift-main/DemoApp/Resources/Launch Screen.storyboard stream-video-swift-main/DemoApp/Sources/AppDelegate.swift stream-video-swift-main/DemoApp/Sources/Components/AppEnvironment.swift stream-video-swift-main/DemoApp/Sources/Components/AppState.swift stream-video-swift-main/DemoApp/Sources/Components/AudioFilter/AudioFilter.swift stream-video-swift-main/DemoApp/Sources/Components/AudioFilter/DemoVoiceProcessor.swift stream-video-swift-main/DemoApp/Sources/Components/AudioFilter/RobotVoiceFilter.swift stream-video-swift-main/DemoApp/Sources/Components/Authentication/AuthenticationProvider.swift stream-video-swift-main/DemoApp/Sources/Components/Chat/DemoChatAdapter.swift [899 more files omitted for size] ``` ### Dependencies - backend/pyproject.toml: flask@^3.0.2, flask-socketio@^5.3.6, hume@^0.4.1, ipykernel@^6.29.2, mypy@^1.8.0, numpy@^1.26.4, openai@^1.12.0, pillow@^10.2.0, pyserial@^3.5, ruff@^0.2.1 - backend/requirements.txt: annotated-types@==0.6.0, anyio@==4.2.0, bidict@==0.23.0, blinker@==1.7.0, certifi@==2024.2.2, charset-normalizer@==3.3.2, click@==8.1.7, colorama@==0.4.6, distro@==1.9.0, exceptiongroup@==1.2.0, flask@==3.0.2, flask-socketio@==5.3.6, h11@==0.14.0, httpcore@==1.0.3, httpx@==0.26.0, hume@==0.4.1, idna@==3.6, itsdangerous@==2.1.2, jinja2@==3.1.3, markupsafe@==2.1.5, numpy@==1.26.4, openai@==1.12.0, pydantic@==2.6.1, pydantic-core@==2.16.2, pyserial@==3.5, python-engineio@==4.9.0, python-socketio@==5.11.1, requests@==2.31.0, simple-websocket@==1.0.0, sniffio@==1.3.0, tqdm@==4.66.2, typing-extensions@==4.9.0, urllib3@==2.2.0, werkzeug@==3.0.1, wsproto@==1.2.0 - stream-video-swift-main/requirements.txt: lizard ### Recent commits (newest first) - latest changes - Update README.md - Update README.md - Update image_receive.py - Update image_receive.py - extract response - working emotion - Rename app and add slow camera feed manager - Add image receive - Update app.py - Update app.py - Add pyserial and ipykernel - Merge branch 'main' of https://github.com/raviriley/aidactyl - Create requirements.txt - fixed full jupyer - added jupyter websocket flask - Add flask-socketio - Add package as local dependency to override - Current progress - Delete example code ## Key source files (fetched from GitHub, selected and truncated for size) ### stream-video-swift-main/SECURITY.md ```markdown # Reporting a Vulnerability At Stream we are committed to the security of our Software. We appreciate your efforts in disclosing vulnerabilities responsibly and we will make every effort to acknowledge your contributions. Report security vulnerabilities at the following email address: ``` [security@getstream.io](mailto:security@getstream.io) ``` Alternatively it is also possible to open a new issue in the affected repository, tagging it with the `security` tag. A team member will acknowledge the vulnerability and will follow-up with more detailed information. A representative of the security team will be in touch if more information is needed. # Information to include in a report While we appreciate any information that you are willing to provide, please make sure to include the following: * Which repository is affected * Which branch, if relevant * Be as descriptive as possible, the team will replicate the vulnerability before working on a fix. ``` ### stream-video-swift-main/CHANGELOG.md ```markdown # StreamVideo iOS SDK CHANGELOG The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). # Upcoming ### ✅ Added - New API that allows adding capturePhotoOutput, videoOutput on the local participant's AVCaptureSession. [#301](https://github.com/GetStream/stream-video-swift/pull/301) - New API that allows zooming the local participant's camera. [#301](https://github.com/GetStream/stream-video-swift/pull/301) # [0.5.0](https://github.com/GetStream/stream-video-swift/releases/tag/0.5.0) _February 15, 2024_ ### 🔄 Changed - The following API changes occurred as part of the redesign. [#269](https://github.com/GetStream/stream-video-swift/pull/269) & [#270](https://github.com/GetStream/stream-video-swift/pull/270) - `OutgoingCallView` now accepts an additional `callTopView` parameter to align with the updated design. - `CallParticipantsInfoView` and the `ViewFactory.makeParticipantsListView` method aren't accept the `availableFrame` anymore. - `ParticipantsGridLayout` `orientation` parameter isn't required anymore. - The `onRotate` ViewModifier has been removed. You can use the `InjectedValues[\.orientationAdapter]` which is an ObservableObject that can provide information regarding device orientation. # [0.4.2](https://github.com/GetStream/stream-video-swift/releases/tag/0.4.2) _December 08, 2023_ ### ⚠️ Important - Nuke dependency is no longer exposed. If you were using this dependency we were exposing, you would need to import it manually. This is due to our newest addition supporting Module Stable XCFramework, see more below in the "Added" section. If you encounter any SPM-related problems, be sure to reset the package caches. ### ✅ Added - Add support for pre-built XCFrameworks - Fast reconnection - New redesigned UI components. [#236](https://github.com/GetStream/stream-video-swift/pull/236) ### 🔄 Changed - You can now focus on a desired point in the local video stream. [#221](https://github.com/GetStream/stream-video-swift/pull/221) - The following API changes occurred as part of the redesign. [#221](https://github.com/GetStream/stream-video-swift/pull/221) - `CornerDragableView` has been renamed to `CornerDraggableView` and initializer changed. - `LocalParticipantViewModifier` & `VideoCallParticipantModifier` now accept a few parameters that allow you to further control their presentation. - `ScreenSharingView` now accepts a `isZoomEnabled` parameter to control if the the view will be zoom-able. - `LocalVideoView` now accepts a `availableFrame` parameter. # [0.4.1](https://github.com/GetStream/stream-video-swift/releases/tag/0.4.1) _October 16, 2023_ ### 🐞 Fixed - Video tracks remain disabled when they become visible [#191](https://github.com/GetStream/stream-video-swift/pull/191) # [0.4.0](https://github.com/GetStream/stream-video-swift/releases/tag/0.4.0) _October 11, 2023_ ### ✅ Added - Picture-in-Picture support - Livestream Player - Call stats report ### 🔄 Changed - [truncated — 1053 more characters] ``` ### stream-video-swift-main/requirements.txt ``` lizard ``` ### stream-video-swift-main/Gemfile ``` # frozen_string_literal: true source 'https://rubygems.org' git_source(:github) { |repo_name| "https://github.com/#{repo_name}" } gem 'danger', group: :danger_dependencies gem 'fastlane', group: :fastlane_dependencies gem 'jazzy' gem 'json' gem 'rubocop', '1.38', group: :rubocop_dependencies gem 'sinatra', group: :sinatra_dependencies gem 'slather' eval_gemfile('fastlane/Pluginfile') group :fastlane_dependencies do gem 'badge' gem 'cocoapods' gem 'fastlane-plugin-lizard' gem 'plist' gem 'xcode-install' gem 'xctest_list' end group :sinatra_dependencies do gem 'puma' gem 'rackup' end group :rubocop_dependencies do gem 'rubocop-performance' gem 'rubocop-require_tools' end group :danger_dependencies do gem 'danger-commit_lint' end ``` ### backend/pyproject.toml ``` [tool.poetry] name = "backend" version = "0.1.0" description = "" authors = ["Ravi Riley "] readme = "README.md" [tool.poetry.dependencies] python = "^3.10" hume = "^0.4.1" flask = "^3.0.2" openai = "^1.12.0" numpy = "^1.26.4" flask-socketio = "^5.3.6" pyserial = "^3.5" pillow = "^10.2.0" [tool.poetry.group.dev.dependencies] ruff = "^0.2.1" mypy = "^1.8.0" ipykernel = "^6.29.2" [tool.poetry.scripts] lint = "scripts.scripts:lint" l = "scripts.scripts:lint" format = "scripts.scripts:format_code" f = "scripts.scripts:format_code" format-lint = "scripts.scripts:format_and_lint" fl = "scripts.scripts:format_and_lint" test = "scripts.scripts:test" t = "scripts.scripts:test" format-lint-test = "scripts.scripts:format_lint_test" flt = "scripts.scripts:format_lint_test" dev = "scripts.scripts:dev" [tool.mypy] python_version = "3.10" disallow_untyped_defs = true check_untyped_defs = true warn_redundant_casts = true warn_unused_ignores = true ignore_missing_imports = true [tool.ruff] line-length = 120 [tool.ruff.lint] [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api" ``` ### backend/requirements.txt ``` annotated-types==0.6.0 ; python_version >= "3.10" and python_version < "4.0" anyio==4.2.0 ; python_version >= "3.10" and python_version < "4.0" bidict==0.23.0 ; python_version >= "3.10" and python_version < "4.0" blinker==1.7.0 ; python_version >= "3.10" and python_version < "4.0" certifi==2024.2.2 ; python_version >= "3.10" and python_version < "4" charset-normalizer==3.3.2 ; python_version >= "3.10" and python_version < "4" click==8.1.7 ; python_version >= "3.10" and python_version < "4.0" colorama==0.4.6 ; python_version >= "3.10" and python_version < "4.0" and platform_system == "Windows" distro==1.9.0 ; python_version >= "3.10" and python_version < "4.0" exceptiongroup==1.2.0 ; python_version >= "3.10" and python_version < "3.11" flask-socketio==5.3.6 ; python_version >= "3.10" and python_version < "4.0" flask==3.0.2 ; python_version >= "3.10" and python_version < "4.0" h11==0.14.0 ; python_version >= "3.10" and python_version < "4.0" httpcore==1.0.3 ; python_version >= "3.10" and python_version < "4.0" httpx==0.26.0 ; python_version >= "3.10" and python_version < "4.0" hume==0.4.1 ; python_version >= "3.10" and python_version < "4" idna==3.6 ; python_version >= "3.10" and python_version < "4" itsdangerous==2.1.2 ; python_version >= "3.10" and python_version < "4.0" jinja2==3.1.3 ; python_version >= "3.10" and python_version < "4.0" markupsafe==2.1.5 ; python_version >= "3.10" and python_version < "4.0" numpy==1.26.4 ; python_version >= "3.10" and python_version < "4.0" openai==1.12.0 ; python_version >= "3.10" and python_version < "4.0" pydantic-core==2.16.2 ; python_version >= "3.10" and python_version < "4.0" pydantic==2.6.1 ; python_version >= "3.10" and python_version < "4.0" pyserial==3.5 ; python_version >= "3.10" and python_version < "4.0" python-engineio==4.9.0 ; python_version >= "3.10" and python_version < "4.0" python-socketio==5.11.1 ; python_version >= "3.10" and python_version < "4.0" requests==2.31.0 ; python_version >= "3.10" and python_version < "4" simple-websocket==1.0.0 ; python_version >= "3.10" and python_version < "4.0" sniffio==1.3.0 ; python_version >= "3.10" and python_version < "4.0" tqdm==4.66.2 ; python_version >= "3.10" and python_version < "4.0" typing-extensions==4.9.0 ; python_version >= "3.10" and python_version < "4" urllib3==2.2.0 ; python_version >= "3.10" and python_version < "4" werkzeug==3.0.1 ; python_version >= "3.10" and python_version < "4.0" wsproto==1.2.0 ; python_version >= "3.10" and python_version < "4.0" ``` ### backend/backend/app.py ```python import base64 import json import os import threading import time import numpy as np import serial # websocket from flask import Flask, jsonify, render_template from flask_socketio import SocketIO, emit from numpy import dot from numpy.linalg import norm from openai import OpenAI # import pyaudio # Flask App app = Flask(__name__) socketio = SocketIO(app) # Dataset of sign language readings sign_language_dataset = { "A": [3916, 3939, 4015, 4004, 3950, 4036], "B": [4034, 4100, 3966, 4034, 4067, 3976], "C": [3966, 3950, 3950, 3899, 3956, 4046], "D": [4035, 4030, 4075, 3956, 3965, 3956], "E": [4059, 3975, 4140, 3967, 4012, 4053], "F": [3976, 3957, 4039, 4041, 4007, 3934], "G": [3956, 3952, 3934, 4031, 4052, 3957], "H": [4049, 4024, 4093, 4016, 3882, 4002], "I": [3952, 3998, 3989, 4026, 4008, 3937], "J": [3994, 4042, 3939, 4015, 4034, 3960], "K": [4082, 3998, 3996, 3998, 3850, 3961], "L": [4032, 3983, 3970, 3981, 4034, 4067], "M": [4029, 3991, 4004, 3908, 4020, 3954], "N": [4058, 4002, 3983, 3985, 4012, 3986], "O": [4039, 4100, 3963, 4020, 4024, 3979], "P": [4051, 4037, 3950, 4004, 4024, 3922], "Q": [3937, 4055, 3977, 4010, 4022, 4005], "R": [3966, 4057, 4024, 4024, 3981, 3935], "S": [3944, 3983, 3847, 4003, 3997, 4105], "T": [4005, 3954, 3963, 4071, 4028, 3981], "U": [4011, 4056, 3957, 4001, 4063, 3926], "V": [4005, 4121, 4006, 4027, 4132, 3951], "W": [3994, 3959, 3983, 3974, 3958, 3926], "X": [3986, 3939, 3963, 3975, 3944, 4008], "Y": [3951, 3955, 3949, 4071, 3942, 4083], "Z": [3961, 3943, 4052, 4006, 4010, 4001], } client = OpenAI(api_key="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") def cosine_similarity(A, B): return dot(A, B) / (norm(A) * norm(B)) def compare_readings(sensorArray, sign_language_dataset): best = 0 best_key = "" for key, value in sign_language_dataset.items(): similarity = cosine_similarity(sensorArray, value) if similarity > best: best = similarity best_key = key return best_key # post request to openAI def callOpenAI(words): try: # Join the words into a single string as prompt for the API prompt = " ".join(words) # Call the OpenAI API with the prompt completion = client.chat.completions.create( model="gpt-3.5-turbo", messages=[ { "role": "system", "content": "I will be passing you glosses of American Sign Language (ASL) words and phrases. Please provide the spoken English translation.", }, {"role": "user", "content": prompt}, ], ) # Extract the text from the response generated_text = completion.choices[0].message.content print("Spoken text:", generated_text) spokenText.append(generated_text) return generated_text except Exception as e: print(f"An error occurred: {str(e)}") # RECIEVING FROM ARDUINO # Replace '/dev/ttyUSB0' with your serial port and adjust baudrate # ser = serial.Serial("/dev/cu.usbmodem2101", 9600) sameWordCounter = 0 spokenText = [] words = [] reading = False def startHandCapture(): global words, sameWordCounter, reading while reading: data = ser.readline() if data: sensorCapture = data.decode("utf-8").strip() print(sensorCapture) # convert sensorCapture to an array of integers based on seperating the string by a delimiter sensorArray = sensorCapture.split(",") # cast every element in the array to an integer sensorArray = [int(i) for i in sensorArray] print(sensorArray) try: word = compare_readings(sensorArray, sign_language_dataset) # Process the data except: # go to next loop continue # check if equal to the last word received if len(words) == 0 or words[-1] != word: words.append(word) # Append the word to the array # else: # sameWordCounter += 1 # if sameWordCounter == 3: # callOpenAI(words) # sameWordCounter = 0 # words = [] print("word: ", word) print("words: ", words) time.sleep(2) # Wait for 2 seconds before processing the next piece of data def stopHandCapture(): global reading reading = False # HUME EMOTION VISION def storeImage(json): image_data = json["image"] image_bytes = base64.b64decode(image_data) # Make sure the 'assets' directory exists assets_directory = "./assets" if not os.path.exists(assets_directory): os.makedirs(assets_directory) # Define the full path where the image will be stored image_path = os.path.join(assets_directory, "image.jpg") with open(image_path, "wb") as image_file: image_file.write(image_bytes) print("Image stored at:", image_path) from hume import HumeBatchClient from hume.models.config import FaceConfig def callHumeAPI(): client = HumeBatchClient("a6nLyAAG6WerOIyqNiaUfdljGEBxPSNpflWtScb0O521e7we") filepaths = ["./assets/image.jpg"] config = FaceConfig() job = client.submit_job(None, [config], files=filepaths) print(job) print("Running...") details = job.await_complete() # job.download_predictions("predictions.json") results = job.get_predictions() # print(type(results)) print(results) first = results[0] emotions = first["results"]["predictions"][0]["models"]["face"]["grouped_predictions"][0]["predictions"][0][ "emotions" ] sorted_audio_emotions = sorted(emotions, key=lambda x: x["score"], reverse=True) print("sorted emotions: ", sorted_audio_emotions) print("top emotion: ", sorted_audio_emotions[0]) return sorted_audi [truncated — 864 more characters] ``` ### stream-video-swift-main/.spi.yml ```yaml version: 1 builder: configs: - platform: ios documentation_targets: [StreamVideoSwiftUI, StreamVideoUIKit, StreamVideo] scheme: StreamChatSwiftUI ``` ### stream-video-swift-main/.slather.yml ```yaml coverage_service: sonarqube_xml xcodeproj: StreamVideo.xcodeproj scheme: StreamVideo configuration: Test build_directory: derived_data/ source_directory: Sources/ output_directory: reports ignore: - "**/*_Vendor.swift" - "**/Generated/" - "**/protobuf/" - "**/OpenApi/" - "Sources/StreamVideoSwiftUI/StreamNuke" ``` ### stream-video-swift-main/Package.swift ```swift // swift-tools-version:5.6 import Foundation import PackageDescription let package = Package( name: "StreamVideo", defaultLocalization: "en", platforms: [ .iOS(.v13), .macOS(.v11) ], products: [ .library( name: "StreamVideo", targets: ["StreamVideo"] ), .library( name: "StreamVideoSwiftUI", targets: ["StreamVideoSwiftUI"] ), .library( name: "StreamVideoUIKit", targets: ["StreamVideoUIKit"] ) ], dependencies: [ .package(url: "https://github.com/apple/swift-protobuf.git", from: "1.18.0"), .package(url: "https://github.com/GetStream/stream-video-swift-webrtc.git", exact: "114.5735.08") ], targets: [ .target( name: "StreamVideo", dependencies: [ .product(name: "SwiftProtobuf", package: "swift-protobuf"), .product(name: "StreamWebRTC", package: "stream-video-swift-webrtc") ] ), .target( name: "StreamVideoSwiftUI", dependencies: ["StreamVideo"] ), .target( name: "StreamVideoUIKit", dependencies: ["StreamVideo", "StreamVideoSwiftUI"] ) ] ) ``` [895 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]