# Project export: BallAR This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Unite AR and electro-stimulation to perfect your beer pong game. - Devpost: https://devpost.com/software/ballar - GitHub: https://github.com/YetSquire/BallAR - Video: https://www.youtube.com/embed/Kqn162iM-P0?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Snap: Best Use of Snap Spectacles) - Team: 1 GitHub contributor(s) — Andy (22 commits) ## Devpost submission (written by the team) ### Inspiration While our hackathon team is less Greek Life and more Geek Life, we recognize the utility, hilarity, and surprising difficulty of the game ‘Beer Pong’. This party game requires precise hand-eye coordination, inter-team collaboration, and strategic planning. It is also frequently played in dim conditions and under various forms of impairment, meaning that participants can use all the help they can get. With that in mind, we developed BallAR, an augmented reality (AR) tool developed on Snap Spectacles paired with electro-stimulation pads so users can practice their aim, receive assistance during matches, and even have our experience flex their beer-pong muscles for them. ### What it does BallAR has a physical and an AR component. The physical component is centered around electro-stimulation sensors that are actuated via an Arduino, relay module, and ESP 32 module. When the user wishes to launch the ball, the electro-simulation sensors hooked up to the user’s arm will actuate certain sensors in a timed fashion through the relay module. The AR component is built for Snap Spectacles. The primary functionality of this experience is to create a spatial UI that draws an optimal ping-pong ball shot from the user’s hand to a single red solo cup. The user’s hand is tracked via the Snap Lens Studio Interaction Manager, which serves as one end of the ideal trajectory. The other end is defined differently based on the two primary modes, single-cup and multi-cup. In single-cup mode, the user can click a spatial button to take a photo for Gemini to analyze. Using the returned XY coordinate and the Z depth coordinate created by the depth mapping during the photo, we can define a 3D coordinate that represents the endpoint of a ping-pong ball’s ideal trajectory. In multi-cup mode, the user is able to place and adjust a predefined triangular object representing the 10-cup arrangement of a Beer Pong game. This object maps onto flat surfaces initially, but after placement can be physically moved to any location in 3D space, depending on the user. The user can then dynamically select a cup to aim for, which will serve as the next target. Finally, the AR component also connects to the electrostimulation component via BLE, and offers a few levels of protection when it comes to the initiation of said component. Because the stimulation has the potential to adversely affect the user, we placed high priority on ensuring that the experience would be difficult to accidentally trigger. Before anything could begin, our electrostimulation UI was locked behind the acknowledgement of a successful connection from both parties (Snap Spectacles and ESP32 Microcontroller). After this, a safety button would be enabled, which needs to be fully pressed in order to enable the electrostimulation fire button, which would in turn, after being pressed, reenable safety and hide the fire button. ### How we built it The snap lenses send a message over bluetooth to the Arduino. This causes the Arduino to activate certain muscles with the electric sensors. To better develop a user’s ability to throw consistently we have a relay module to time the moment we actuate the user’s muscles. This timing ensures that we are able to separate the general throwing motion from the motion we use to release the ball. The user interface and cup detection was developed through Lens Studio in TypeScript. We used the Gemini API to allow us to detect the beer pong cups and use a lightweight trajectory analysis algorithm to derive a realistic trajectory for the user to aim for. The algorithm for this trajectory was inspired by research conducted around the optimal trajectory for basketball free throws. We followed a similar methodology where given the initial positions and target positions, we perform a series of frame translations to arrive at a 2D projection, fit a parabolic arc, and translate it back into the world frame which the glasses operate in. The arc was derived from conducting a series of video analysis to analyze the trajectory of the ball observed in a simulated game of beer pong. ### Challenges we ran into From the very start, BallAR was defined by overcoming challenges and rethinking plans. Our team’s initial concept was a Basketball trainer in AR (we aim to see the Golden State Warriors immediately after this hackathon). However, two factors led to us reconsidering: first, we found an existing Snap Spectacles Basketball trainer experience. Second, after testing with our electrostimulation technology, we found that the amount of electricity required to launch a basketball (via activating the tricep and wrist muscles) would cause irritation and discomfort. However, after brainstorming, we were able to pivot and buy some red solo cups and halloween themed ping-pong balls to pursue a different iteration of BallAR! Furthermore, we, alongside all of CalHacks, quickly came into contact with connectivity problems onsite. A poor initial internet connection led to a mass adoption of hotspot activity, which in turn decreased internet quality. The prisoner’s dilemma was in full force, and our team decided that the best solution would be to purposefully avoid the transmission of large amounts of data over wireless networks within our project. Our systems were built to be as independent as possible, with only an images’ worth of an API call and a small string sent via BLE necessary for a single runthrough of BallAR. This kind of redundancy and system safety was a novel but interesting challenge as we prototyped our experience. ### Accomplishments we're proud of BallAR, despite its funny use case, combines the areas of anatomy, electrical engineering, firmware development, physics, game design, and augmented reality development for a smooth, cohesive experience. This project demonstrates our team’s ability to pull together knowledge from various sources of personal experience and learn new skills quickly, putting together several self-contained but mutually reinforcing components. Our ability to rapidly and successfully integrate our initial independent work (as compared to prior hackathon experiences) indicates not only that we understood one another as we converged and diverged, but also that we understood the project and its overall goals– a critical skill in school and in society. ### What we learned Broadly, this hackathon made our team realize the critical importance of connectivity both in terms of our project scope and our ability to iterate and develop. Unstable internet, while not crippling, in practice greatly limited the API and toolcalling ability of our project, as we wished to offer a live demo during judging. Our team learned to brainstorm around such fundamental challenges, and quickly developed a habit of using AI not within our experience, but instead to augment our research to overcome the problems we were seeing. Beyond internet issues, our team was able to learn tremendous amounts of seemingly disconnected information in order to apply it to our final product. From anatomy (an understanding of musculature was necessary for proper electrostimulation) to physics (the algorithm used for ping-pong ball targeting needed to be grounded in reality) to stateful programming (AR design is much like game design, in that sense). We grew in all these fields, either directly learning or indirectly hearing about difficulties overcome. ### What's next BallAR, at the moment, is a very one-size-fits-all experience. Individuals can customize left-and-right handedness, but we offer no way to calibrate aspects such as average ball velocity or electrostimulation sensitivity within the experience itself. While our team was able to avoid this problem during demos by manually adjusting, in the future we could add further customizability to both the AR and stimulation components of our experience. ## README (from the GitHub repository) # Depth Cache [![SIK](https://img.shields.io/badge/SIK-Light%20Gray?color=D3D3D3)](https://developers.snap.com/spectacles/spectacles-frameworks/spectacles-interaction-kit/features/overview?) [![Remote Service Gateway](https://img.shields.io/badge/Remote%20Service%20Gateway-Light%20Gray?color=D3D3D3)](https://developers.snap.com/spectacles/about-spectacles-features/overview) [![ASR](https://img.shields.io/badge/ASR-Light%20Gray?color=D3D3D3)](https://developers.snap.com/spectacles/about-spectacles-features/apis/asr-module) [![AI](https://img.shields.io/badge/AI-Light%20Gray?color=D3D3D3)](https://developers.snap.com/spectacles/spectacles-frameworks/spectacles-interaction-kit/features/overview?) [![Depth](https://img.shields.io/badge/Depth-Light%20Gray?color=D3D3D3)](https://developers.snap.com/lens-studio/features/ar-tracking/world/world-mesh-and-depth-texture) [![AR Tracking](https://img.shields.io/badge/AR%20Tracking-Light%20Gray?color=D3D3D3)](https://developers.snap.com/spectacles/spectacles-frameworks/spectacles-interaction-kit/features/overview?) [![Object Tracking](https://img.shields.io/badge/Object%20Tracking-Light%20Gray?color=D3D3D3)](https://developers.snap.com/spectacles/spectacles-frameworks/spectacles-interaction-kit/features/overview?) [![ASR](https://img.shields.io/badge/ASR-Light%20Gray?color=D3D3D3)](https://developers.snap.com/spectacles/about-spectacles-features/apis/asr-module) [![AI](https://img.shields.io/badge/AI-Light%20Gray?color=D3D3D3)](https://developers.snap.com/spectacles/about-spectacles-features/compatibility-list) [![Depth](https://img.shields.io/badge/Depth-Light%20Gray?color=D3D3D3)](https://developers.snap.com/lens-studio/features/ar-tracking/world/world-mesh-and-depth-texture) [![AR Tracking](https://img.shields.io/badge/AR%20Tracking-Light%20Gray?color=D3D3D3)](https://developers.snap.com/spectacles/about-spectacles-features/compatibility-list) [![Object Tracking](https://img.shields.io/badge/Object%20Tracking-Light%20Gray?color=D3D3D3)](https://developers.snap.com/lens-studio/features/ar-tracking/world/object-tracking) essentials ## Overview The Depth Module API now allows caching of depth frames, enabling pixel-to-3D projection even after a delay. This is especially useful for cloud-based vision models — once the results are returned, you can map image-space coordinates back into world space using the cached depth data. This example lens demonstrates that workflow using the spatial reasoning capabilities of Gemini 2.5 Pro Preview, but the same approach can be applied to any vision model that outputs pixel or image-space coordinates. > **NOTE:** > This project will only work for the Spectacles platform and Lens Studio. ## Design Guidelines Designing Lenses for Spectacles offers all-new possibilities to rethink user interaction with digital spaces and the physical world. Get started using our [Design Guidelines](https://developers.snap.com/spectacles/best-practices/design-for-spectacles/introduction-to-spatial-design) ## Prerequisites - **Lens Studio**: v5.15.0+ - **Spectacles OS Version**: v5.64+ - **Spectacles App iOS**: v0.64+ - **Spectacles App Android**: v0.64+ To update your Spectacles device and mobile app, please refer to this [guide](https://support.spectacles.com/hc/en-us/articles/30214953982740-Updating). You can download the latest version of Lens Studio from [here](https://ar.snap.com/download?lang=en-US). ## Getting Started To obtain the project folder, clone the repository. > **IMPORTANT:** > This project uses Git Large Files Support (LFS). Downloading a zip file using the green button on GitHub **will not work**. You must clone the project with a version of git that has LFS. > You can download Git LFS [here](https://git-lfs.github.com/). ## Project Overview This example uses ASR to listen for voice input when the microphone button is pinched or tapped. The transcription is sent to Gemini Pro via the Remote Service Gateway package. Before sending an image to Gemini we have to cache the depth data. DepthCache.ts listens to a stream of depth frame data (using the Depth Module API) and also keeps a small history of camera frames, aligning them with timestamps. When we call saveDepthFrame() it returns an ID that we can use to get the depth frame data and associated camera frame from that point in time. When we get a result from Gemini we can call getWorldPositionWithID() from DepthCache.ts to convert the pixel coordinates (center of a bounding box) to 3D world coordinates from that cached information. In GeminiAPI.ts you can see the response format that we are requesting from Gemini. It returns a text response that we display to the user, as well as a data object that contains the bounding boxes and labels. ## Debug Visuals For testing purposes there is DebugVisualizer.ts which will plot points over a camera frame texture. This can be turned on in SceneController.ts by checking showDebugVisuals to true in the hierarchy. Now when you make a request you will see the points Gemini found in the image. ## Testing the Lens ### In Lens Studio Editor 1. Open the Preview panel in Lens Studio. 2. Use Interactive Preview, WASD (+QE for elevation), LMB and RMB to move around scene. 3. Input your key for the Remote Service Gateway in the hierarchy. ### On Spectacles Device 1. Build and deploy the project to your Spectacles device. 2. Follow the [Spectacles guide](https://developers.snap.com/spectacles/get-started/start-building/preview-panel) for device testing. ## Support If you have any questions or need assistance, please don't hesitate to reach out. Our community is here to help, and you can connect with us and ask for support [here](https://www.reddit.com/r/Spectacles/). We look forward to hearing from you and are excited to assist you on your journey! ## Contributing Feel free to provide improvements or suggestions or directly contributing via merge request. By sharing insights, you help everyone else build better Lenses. --- *Built with 👻 by the Spectacles team* ## Detected evidence (automated analysis) Indexed codebase: 18 recognized source files, 52 KB. - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 180) ``` .gitattributes .gitignore Assets/Cylinder.mesh Assets/Cylinder.mesh.meta Assets/Images/Base.png.meta Assets/Images/camera.png.meta Assets/Images/cannon.png.meta Assets/Images/circle.png.meta Assets/Images/ImageMaterial.mat Assets/Images/ImageMaterial.mat.meta Assets/Images/loading.gif.meta Assets/Images/MaterialParams.png.meta Assets/Images/MicIcon.png.meta Assets/Images/multi-cup.png.meta Assets/Images/Normal.png.meta Assets/Images/play-button.png.meta Assets/Images/single-cup.png.meta Assets/Images/warning.png.meta Assets/Images/wifi.png.meta Assets/Materials/cameraIcon.mat Assets/Materials/cameraIcon.mat.meta Assets/Materials/CaptureTexture.mat Assets/Materials/CaptureTexture.mat.meta Assets/Materials/CircleOutline.mat Assets/Materials/CircleOutline.mat.meta Assets/Materials/circleOutline.ss_graph Assets/Materials/circleOutline.ss_graph.meta Assets/Materials/fireIcon.mat Assets/Materials/fireIcon.mat.meta Assets/Materials/Image 2.mat Assets/Materials/Image 2.mat.meta Assets/Materials/image_unlit.ss_graph Assets/Materials/image_unlit.ss_graph.meta Assets/Materials/Image.mat Assets/Materials/Image.mat.meta Assets/Materials/label.ss_graph Assets/Materials/label.ss_graph.meta Assets/Materials/loadingGif.mat Assets/Materials/loadingGif.mat.meta Assets/Materials/loadingGif.ss_graph Assets/Materials/loadingGif.ss_graph.meta Assets/Materials/micIcon.mat Assets/Materials/micIcon.mat.meta Assets/Materials/multiCupIcon.mat Assets/Materials/multiCupIcon.mat.meta Assets/Materials/MyAi_Voice.mat Assets/Materials/MyAi_Voice.mat.meta Assets/Materials/MyAi_Voice.ss_graph Assets/Materials/MyAi_Voice.ss_graph.meta Assets/Materials/PBR 2.mat Assets/Materials/PBR 2.mat.meta Assets/Materials/PBR.mat Assets/Materials/PBR.mat.meta Assets/Materials/pbr.ss_graph Assets/Materials/pbr.ss_graph.meta Assets/Materials/PinchButton.mat Assets/Materials/PinchButton.mat.meta Assets/Materials/playIcon.mat Assets/Materials/playIcon.mat.meta Assets/Materials/pointRedLabeled.mat Assets/Materials/pointRedLabeled.mat.meta Assets/Materials/safeIcon.mat Assets/Materials/safeIcon.mat.meta Assets/Materials/singleCupIcon.mat Assets/Materials/singleCupIcon.mat.meta Assets/Materials/SuperTransparent.mat Assets/Materials/SuperTransparent.mat.meta Assets/Materials/SWIS_flat_color.ss_graph Assets/Materials/SWIS_flat_color.ss_graph.meta Assets/Materials/SWIS_tile.mat Assets/Materials/SWIS_tile.mat.meta Assets/Materials/Transparent.mat Assets/Materials/Transparent.mat.meta Assets/Materials/unlitColor.ss_graph Assets/Materials/unlitColor.ss_graph.meta Assets/Materials/unlitTexture.ss_graph Assets/Materials/unlitTexture.ss_graph.meta Assets/Materials/unlitTextureColor.ss_graph Assets/Materials/unlitTextureColor.ss_graph.meta Assets/Materials/warningIcon.mat Assets/Materials/warningIcon.mat.meta Assets/Materials/White.mat Assets/Materials/White.mat.meta Assets/Materials/WorldLabelFrame.mat Assets/Materials/WorldLabelFrame.mat.meta Assets/Mesh/arrow.mesh Assets/Mesh/arrow.mesh.meta Assets/Mesh/SWIS_tile_solid.mesh Assets/Mesh/SWIS_tile_solid.mesh.meta Assets/Mesh/Unit Box.mesh Assets/Mesh/Unit Box.mesh.meta Assets/Mesh/Unit Plane.mesh Assets/Mesh/Unit Plane.mesh.meta Assets/Mesh/Unit Sphere.mesh Assets/Mesh/Unit Sphere.mesh.meta Assets/Modules/Bluetooth Central Module.bluetoothCentralModule Assets/Modules/Bluetooth Central Module.bluetoothCentralModule.meta Assets/Modules/Camera Module.cameraModule Assets/Modules/Camera Module.cameraModule.meta Assets/PBR 2.mat Assets/PBR 2.mat.meta Assets/PBR 3.mat Assets/PBR 3.mat.meta Assets/PBR 4.mat Assets/PBR 4.mat.meta Assets/PBR 5.mat Assets/PBR 5.mat.meta Assets/PBR 6.mat Assets/PBR 6.mat.meta Assets/PBR.mat Assets/PBR.mat.meta Assets/pbr.ss_graph Assets/pbr.ss_graph.meta Assets/Prefabs/ColliderT.lspkg/Materials/PBR.mat Assets/Prefabs/ColliderT.lspkg/Materials/PBR.mat.meta Assets/Prefabs/ColliderT.lspkg/Materials/Shader Graphs/pbr.ss_graph Assets/Prefabs/ColliderT.lspkg/Materials/Shader Graphs/pbr.ss_graph.meta Assets/Prefabs/ColliderT.lspkg/Meshes/Sphere.mesh Assets/Prefabs/ColliderT.lspkg/Meshes/Sphere.mesh.meta Assets/Prefabs/ColliderT.lspkg/Textures/Base.png.meta [60 more files omitted for size] ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Keysrevoked - Pyramid moving better - Togglehandling - Stateful fixes - added pyramid, some states - Everything functional! - Disabled Tracking location, broken pool - BLE Button Setup and New Pool thing - x2 - Pretty good two modes - SanitySave - Successful bluetooth - Material and balls - Acceptable images - inverted z in projection - Mathy math - Semifunctional pathing - Canvas not good, pathing in progress - New icons - Locations are working ## Key source files (fetched from GitHub, selected and truncated for size) ### Assets/Scripts/SendPosition.ts ```typescript import { WorldQueryHit } from "./WorldQueryHit"; @component export class SendPosition extends BaseScriptComponent { onTap(){ let pos = this.getTransform().getWorldPosition(); WorldQueryHit.WorldQueryHit.setCoords(pos); } } ``` ### Assets/Scripts/FaceCamera.ts ```typescript @component export class FaceCamera extends BaseScriptComponent { @input camera: SceneObject; private camTransform: Transform; private transform: Transform; onAwake() { this.transform = this.getTransform(); this.camTransform = this.camera.getTransform(); this.createEvent("UpdateEvent").bind(this.onUpdate.bind(this)); } onUpdate() { var worldCameraForward = this.camTransform.right.cross(vec3.up()); var lookRot = quat.lookAt(worldCameraForward, vec3.up()); this.transform.setWorldRotation(lookRot); } } ``` ### Assets/Scripts/Labels.ts ```typescript @component export class Labels extends BaseScriptComponent { @input labelPrefab: ObjectPrefab; onAwake() {} loadLables(pointCloud: any, labelString: any) { var labels = JSON.parse(labelString); for (var i = 0; i < labels.objects.length; i++) { var label = labels.objects[i]; var labelLocalCamPos = new vec3(label.pos[0], label.pos[1], label.pos[2]); var labelObj = this.labelPrefab.instantiate(this.getSceneObject()); labelObj .getTransform() .setWorldPosition( pointCloud.camLocalToWorld.multiplyPoint(labelLocalCamPos) ); var labelText = labelObj.getChild(0).getComponent("Component.Text"); labelText.text = label.label; } } } ``` ### Assets/Scripts/Loading.ts ```typescript import animate, { CancelSet } from "SpectaclesInteractionKit.lspkg/Utils/animate"; @component export class Loading extends BaseScriptComponent { private loadingTrans: Transform; private startScale: vec3; onAwake() { this.loadingTrans = this.getSceneObject().getTransform(); this.startScale = this.loadingTrans.getLocalScale(); this.loadingTrans.setLocalScale(vec3.zero()); } activateLoder(activate: boolean) { var currScale = this.loadingTrans.getLocalScale(); var desiredScale = activate ? this.startScale : vec3.zero(); animate({ easing: "ease-out-elastic", duration: 0.5, update: (t) => { this.loadingTrans.setLocalScale(vec3.lerp(currScale, desiredScale, t)); }, ended: null, cancelSet: new CancelSet(), }); } } ``` ### Assets/Scripts/WorldLabel.ts ```typescript @component export class WorldLabel extends BaseScriptComponent { @input frameRend: RenderMeshVisual; @input public textComp: Text; private trans: Transform = null; onAwake() { this.trans = this.getSceneObject().getTransform(); //wait some random time before scaling in this.trans.setLocalScale(vec3.zero()); //delay for random time beween 0 and 2 seconds var delayTime = Math.random() * 1.7; var delayEvent = this.createEvent("DelayedCallbackEvent"); delayEvent.bind(() => { //scale in label this.createEvent("UpdateEvent").bind(this.onUpdate.bind(this)); }); delayEvent.reset(delayTime); } onUpdate() { //scale in label or arrow this.trans.setLocalScale( vec3.lerp(this.trans.getLocalScale(), vec3.one(), getDeltaTime() * 7) ); var textSize = this.textComp.text.length; this.frameRend.enabled = textSize > 0; } } ``` ### Assets/Scripts/APIKeyHint.ts ```typescript import { AvaliableApiTypes, RemoteServiceGatewayCredentials, } from "RemoteServiceGateway.lspkg/RemoteServiceGatewayCredentials"; @component export class APIKeyHint extends BaseScriptComponent { @input text: Text; private static readonly PLACEHOLDER_MESSAGES = { [AvaliableApiTypes.Snap]: "[INSERT SNAP TOKEN HERE]", [AvaliableApiTypes.OpenAI]: "[INSERT OPENAI TOKEN HERE]", [AvaliableApiTypes.Google]: "[INSERT GOOGLE TOKEN HERE]", }; private static readonly HINT_MESSAGE = "Set your API Token in the Remote Service Gateway Credentials component to use the examples"; onAwake() { const apiTypes = [ AvaliableApiTypes.Snap, AvaliableApiTypes.OpenAI, AvaliableApiTypes.Google, ]; const hasInvalidApiKey = apiTypes.some(apiType => { const apiKey = RemoteServiceGatewayCredentials.getApiToken(apiType); const placeholder = APIKeyHint.PLACEHOLDER_MESSAGES[apiType]; return apiKey === placeholder || apiKey === ""; }); if (hasInvalidApiKey) { this.text.text = APIKeyHint.HINT_MESSAGE; } else { this.text.enabled = false; } } } ``` ### Assets/Scripts/ASRController.ts ```typescript import Event from "SpectaclesInteractionKit.lspkg/Utils/Event"; @component export class ASRController extends BaseScriptComponent { private asr: AsrModule = require("LensStudio:AsrModule"); private options = null; //make an event for partial voice event that returns a string onPartialVoiceEvent = new Event(); onFinalVoiceEvent = new Event(); onAwake() { this.options = AsrModule.AsrTranscriptionOptions.create(); this.options.silenceUntilTerminationMs = 500; this.options.mode = AsrModule.AsrMode.Balanced; this.options.onTranscriptionUpdateEvent.add((args) => { if (args.isFinal) { print("Final Transcription: " + args.text); this.onFinalVoiceEvent.invoke(args.text); } else { print("Partial: " + args.text); this.onPartialVoiceEvent.invoke(args.text); } }); this.options.onTranscriptionErrorEvent.add((args) => { print("Error: " + args); }); } startListening() { this.asr.startTranscribing(this.options); this.onPartialVoiceEvent.invoke("Waiting for 'Initialize' keyword-\nlook straight down at the\nRed Solo Cup(s)"); } stopListening() { this.asr.stopTranscribing(); } } ``` ### Assets/Scripts/DebugVisualizer.ts ```typescript import SIK from "SpectaclesInteractionKit.lspkg/SIK"; import { SceneController } from "./SceneController"; import { Interactor, InteractorInputType } from "SpectaclesInteractionKit.lspkg/Core/Interactor/Interactor"; import { InteractionManager } from "SpectaclesInteractionKit.lspkg/Core/InteractionManager/InteractionManager"; import { PathCalculation } from "./PathCalculation"; @component export class DebugVisualizer extends BaseScriptComponent { @input sceneController: SceneController; @input pathCalculator: PathCalculation; handInputData: InteractionManager; handInteractor: Interactor; righthand:boolean; cup_coord:vec3; origin:vec3; public single_cup_coord:vec3; public multi_cup_coord:vec3; updateEvent: UpdateEvent; onAwake() { this.righthand = true; this.cup_coord = this.multi_cup_coord = this.single_cup_coord = this.origin = new vec3(0, 0, 0); this.createEvent("OnStartEvent").bind(this.onStart.bind(this)); } setHandedness(){ this.righthand = !this.righthand; //change the icons } private onStart() { // this.testCamVisualObj.enabled = this.sceneController.showDebugVisuals; this.handInputData = SIK.InteractionManager; this.updateEvent = this.createEvent("UpdateEvent"); this.updateEvent.bind(this.onUpdate); } setCupCoord(vec: vec3){ this.cup_coord = vec; this.pathCalculator.objects[0].getTransform().setWorldPosition(vec); } getCupCoord(){ return this.cup_coord; } private onUpdate = (eventData: any) => { if (this.righthand){ this.handInteractor = this.handInputData.getInteractorsByType( InteractorInputType.RightHand )[0]; } else { this.handInteractor = this.handInputData.getInteractorsByType( InteractorInputType.LeftHand )[0]; } if (this.handInteractor) { if (this.cup_coord !== this.origin){ //sceneController will define cup_coord this.pathCalculator.enablePointObjs(); // print(this.rightHandInteractor.startPoint); // print(this.cup_coord); this.cup_coord = this.pathCalculator.objects[0].getTransform().getWorldPosition(); this.pathCalculator.calculatePath(this.handInteractor.startPoint, this.cup_coord); return; } } this.pathCalculator.disablePointObjs(); } } ``` ### Assets/Scripts/ResponseUI.ts ```typescript import animate, { CancelSet, } from "SpectaclesInteractionKit.lspkg/Utils/animate"; import { WorldLabel } from "./WorldLabel"; import { DebugVisualizer } from "./DebugVisualizer"; import { setTimeout } from "SpectaclesInteractionKit.lspkg/Utils/FunctionTimingUtils"; const MAIN_RESPONSE_CHARACTER_COUNT = 175; @component export class ResponseUI extends BaseScriptComponent { @input responseAIText: Text; @input worldLabelPrefab: ObjectPrefab; @input responseUIObj: SceneObject; @input debugVisualizer: DebugVisualizer; private responseBubbleTrans: Transform; onAwake() { this.responseBubbleTrans = this.responseUIObj.getTransform(); this.responseBubbleTrans.setLocalScale(vec3.zero()); this.closeResponseBubble(); } openResponseBubble(message: string) { //truncate message if too long if (message.length > MAIN_RESPONSE_CHARACTER_COUNT) { message = message.substring(0, MAIN_RESPONSE_CHARACTER_COUNT) + "..."; } this.responseAIText.text = message; this.animateResponseBubble(true); setTimeout(() => { this.closeResponseBubble(); }, 5000); } closeResponseBubble() { this.responseAIText.text = ""; this.animateResponseBubble(false); } loadWorldLabel(label: string, worldPosition: vec3) { //create and position label in world space var prefab = this.worldLabelPrefab; var labelObj = prefab.instantiate(this.getSceneObject()); labelObj.getTransform().setWorldPosition(worldPosition); var worldLabel = labelObj.getComponent(WorldLabel.getTypeName()); // worldLabel.textComp.text = worldPosition.toString(); worldLabel.enabled = false; console.log("SET CUP LOCATION TO " + worldPosition.toString()); } clearLabels() { var points = []; for (var i = 0; i < this.getSceneObject().getChildrenCount(); i++) { var childObj = this.getSceneObject().getChild(i); points.push(childObj); } for (var i = 0; i < points.length; i++) { var child = points[i]; child.destroy(); } } private animateResponseBubble(open: boolean) { var currScale = this.responseBubbleTrans.getLocalScale(); var desiredScale = open ? vec3.one() : vec3.zero(); animate({ easing: "ease-out-elastic", duration: 1, update: (t) => { this.responseBubbleTrans.setLocalScale( vec3.lerp(currScale, desiredScale, t) ); }, ended: null, cancelSet: new CancelSet(), }); } } ``` ### Assets/Scripts/SpeechUI.ts ```typescript import animate, { CancelSet } from "SpectaclesInteractionKit.lspkg/Utils/animate"; import Event from "SpectaclesInteractionKit.lspkg/Utils/Event"; import { ASRController } from "./ASRController"; const UI_CAM_DISTANCE = 50; const UI_CAM_HEIGHT = -9; @component export class SpeechUI extends BaseScriptComponent { @input mainCamObj: SceneObject; @input speecBocAnchor: SceneObject; @input micRend: RenderMeshVisual; @input speechButtonCollider: ColliderComponent; onSpeechReady = new Event(); private speechBubbleTrans: Transform; private trans: Transform; private mainCamTrans: Transform; private currentlyListening = false; onAwake() { this.speechBubbleTrans = this.speecBocAnchor.getTransform(); this.speechBubbleTrans.setLocalScale(vec3.zero()); this.trans = this.getSceneObject().getTransform(); this.mainCamTrans = this.mainCamObj.getTransform(); this.animateSpeechBubble(true); this.createEvent("UpdateEvent").bind(this.onUpdate.bind(this)); } activateSpeechButton(activate: boolean) { this.speechButtonCollider.enabled = activate; } onSpeechButtonUp() { this.animateSpeechIcon(true); this.animateSpeechBubble(false); this.onSpeechReady.invoke("Where is the red cups with the white interior"); // this.currentlyListening = !this.currentlyListening; // if (this.currentlyListening) // { // this.startListening(); // } // else { // this.stopListening(); // } } //DEPRACATED // startListening(){ // print("Starting speech UI"); // this.animateSpeechBubble(false); // // this.asrVoiceController.startListening(); // this.onSpeechReady.invoke("Identify the location of the red solo cup"); // } // stopListening() { // print("Disabling speech UI"); // this.animateSpeechBubble(false); // // this.asrVoiceController.stopListening(); // } private onUpdate() { var camPos = this.mainCamTrans.getWorldPosition(); var desiredPosition = camPos.add( this.mainCamTrans.forward.uniformScale(-UI_CAM_DISTANCE) ); desiredPosition = desiredPosition.add( this.mainCamTrans.up.uniformScale(UI_CAM_HEIGHT) ); this.trans.setWorldPosition( vec3.lerp( this.trans.getWorldPosition(), desiredPosition, getDeltaTime() * 10 ) ); var desiredRotation = quat.lookAt(this.mainCamTrans.forward, vec3.up()); this.trans.setWorldRotation( quat.slerp( this.trans.getWorldRotation(), desiredRotation, getDeltaTime() * 10 ) ); } private animateSpeechIcon(active: boolean) { this.micRend.mainPass.Tweak_N23 = active ? 3 : 0; this.micRend.mainPass.Tweak_N33 = active ? 3 : 0; this.micRend.mainPass.Tweak_N37 = active ? 0.2 : 0; } animateSpeechBubble(open: boolean) { var currScale = this.speechBubbleTrans.getLocalScale(); var desiredScale = open ? vec3.one() : vec3.zero(); animate({ easing: "ease-out-elastic", duration: 1, update: (t) => { this.speechBubbleTrans.setLocalScale( vec3.lerp(currScale, desiredScale, t) ); }, ended: null, cancelSet: new CancelSet(), }); } } ``` [7 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]