# Project export: AutoLeARn: AR + AI-Based Learning for Personal Car Repairs This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: AutoLeARn is an educational platform that uses AR and LLMs to guide car owners through diagnostics, maintenance, and repairs–extending vehicle longevity and fostering hands-on learning. - Devpost: https://devpost.com/software/autolearn-ar-ai-based-learning-for-personal-car-repairs - GitHub: https://github.com/zaid-ahmed1/AutoLeARn - Demo: https://github.com/Guuzzeji/AutoLeARn-backend - Video: https://www.youtube.com/embed/NZYmF-WqMVU?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Zaid (24 commits) ## Devpost submission (written by the team) ### Inspiration Introduction: Ever had a car issue that you have no idea to fix? You take it into the shop, wait for a few hours, and then get hit with an astronomical bill—often costing hundreds, if not thousands, of dollars—to fix superficial issues. The thing is, you're not alone. Thousands of people around the world face the same frustrations, not only with struggling to identify what is wrong with their cars, but also with being overcharged by auto repair shops for services that could easily be handled on their own. The Problem: A recent survey of 1,000 randomly selected U.S. adults reveals alarming insights about car maintenance habits: 46% of drivers admit they’ve paid for a repair that could have been avoided with better upkeep [1]. It’s mainly because people don’t actually know how to care for their cars (whether that be through regular oil changes, brake replacements, or other maintenance work), leading to excessive spending on repairs that, for many, might be financially out of reach. Customer Insights: See Infographic Above Our Solution This report envisions a future where augmented reality (AR) and large language model (LLM) technology becomes integral in our everyday lives (potentially within the next few years). As hardware continues to evolve, AR devices will become lighter and easier to use, spreading throughout homes, workplaces, and even car maintenance. Leading companies like Google and Meta are already exploring how AR and LLM technologies can work together. With this shift, learning and problem-solving will be more accessible, enabling anyone to easily engage with and master a wide range of skills. Our project is aimed at predicting the possibilities that exist in this world and using them to solve the problem at hand. Introducing AutoLeARn, an interactive AR-based learning platform designed to guide and educate car owners through essential maintenance tasks, such as oil changes, brake replacements, and more. AutoLeARn empowers users by helping them learn to maintain their vehicles, prolong their lifespan, and minimize costly repairs by providing clear, comprehensive guidance at every step. The device uses augmented reality to overlay visual instructions directly onto the car, ensuring users can follow along with ease (even while using their tools). ### What it does (key features): Speech-to-Text Integration: Simply specify your car details and the service you want to perform using speech commands. This is further enabled by intuitive hand gestures, allowing users to quickly select and customize tasks. Interactive Step-by-Step Guidance: Take a screenshot using a hand gesture, and AutoLeARn returns a sequence of steps to perform the service. Each step is displayed in a panel, where users can swipe left and right to view different instructions. Every panel includes a step title and detailed description, ensuring the user follows along with ease. Integrated Video Assistance: If more guidance is needed, AutoLeARn provides an option to download and watch a tutorial video. The video is pulled based on the task at hand, and the video player is incorporated into a panel with rotation and translation manipulators for ease of use. Step Contextualization: For added convenience, AutoLeARn can point users to specific instructions based on the current step in the repair process. This feature guides users to the exact step they need (skipping over any parts that AutoLeARn detects the user has already completed). With AutoLeARn, users can educate themselves with hands-on, personalized guidance as they learn the process of maintaining their cars, reducing repair costs, and increasing their confidence in car care. ### How we built it Speech-to-Text Integration: Leveraging the Unity SDK audio system, and then passing that into Groq Text-To-Speech [using Whisper], we extract the car model, make, year, and the issue with the car through the user’s speech prompt. We take all this information, save it as JSON, and send it back to Unity and save that information within the app. We used all three Groq modalities in this project, which was a new experience for us, especially considering this has never been done before in XR. Interactive Step-by-Step Guidance: We are also using Groq's image-to-text understanding to extract context from what the user is seeing in real time. Through a thumbs-up hand pose analyzed by the Meta Interaction SDK Computer Vision, we trigger a request for a screenshot to capture the car the person is working on through their lens. We send that back-end API to be analyzed by a Vision Language Model (VLM). We then take the text description generated by the VLM and send that to Perplexity (which allows us to do deep research to figure out the necessary steps for the user to follow in the car repair process). We then utilize Groq again to take the response from Perplexity and turn it into a structured JSON, which can then be fed back into Unity for the canvas steps panel display. Integrated Video Assistance: Developed an API to take a Perplexity video citation and download a video tutorial, which is then parsed into the step-by-step segments for maintenance/repair. This required parsing of auto-generated subtitles and importing it back into a reasoning model to split video into informative steps based on the instructions. Step Contextualization: Build as an additional feature to the step-by-step guidance. Leverages the image description to identify which step the user is on and, from there, skip to the current state of the repair process. ### Challenges we ran into Due to interface issues between unpredictable LLM output and XR interaction, the feature of having a video ready for any step of the user experience had to be simplified to complete the project within the time frame. Integrating XR with multiple LLM features and video tutorials, due to the unpredictability of LLM response and current material availability, was a challenge. We were faced with limitations of the Meta Quest 3 which did not include full support for features like display image capture. We had to develop workarounds with a special screen capture API. Also, we attempted to implement an AI agent-based system but faced limitations in how we wanted the LLM to use the provided tools. Through many iterations, we realized that incorporating AI agents within our project’s scope and timeline would not be feasible. After reaching this conclusion, we simplified our AI system to a chain of LLMs and VLMs that work together to generate steps for the user to follow. ### Accomplishments we're proud of We are proud of creating a project that fuses the worlds of AR and advanced AI while innovating an impactful product. With a team of hackers with mixed experience we were able to work well together as a team and learn a lot collectively. ### What we learned With a variety of experience going in, we all learned different skills and got to test different tools for the first time. Some learned about API interfacing and testing LLMs such as Groq, Gemini, OpenAI, Deepseek, and Perplexity. Some learned a lot about how development in XR works. We learned a lot about limitations of the technology we work with and implementing innovative workarounds like a special API for image capture on the Meta Quest 3 headset. Looking Forward Impact: AutoLeARn will dramatically simplify car maintenance for users, making it both easy and accessible for anyone to take charge of their vehicle’s care. With the power of AR, users no longer need to stop in the middle of a repair to search for instructions or watch complicated video tutorials. Instead, they can receive real-time, step-by-step guidance right in front of them, without having to put down their tools. The hands-free experience eliminates distractions and interruptions, making repairs quicker, more effective, and less stressful. By educating car users with this level of immediate, intuitive assistance, AutoLeARn not only reduces the need for costly professional help, but also gives users the skills and knowledge they need to maintain their cars independently. Other Applications: 1. Helping Government-Run Centers in Training New Car Mechanics Another application of AutoLeARN is the key role it can play in helping government-run training centers who are responsible for training new car mechanics. As noted by industry experts, a major challenge faced by several government agencies responsible for training car mechanics is that they must constantly update their teaching materials to keep up with the rapid advancements in technology (whether that be with the rise of electric vehicles, hybrid cars, and/or increasingly complex trucks) [5]. AutoLeARn can help these government training programs evolve to cover a wider range of systems and tools, providing a better and more effective education. 2. Performing Safety and Functionality Inspections for Car Dealerships & Used Car Lots AutoLeARn also has great potential for use in safety and functionality inspections at car dealerships and used car lots. The analyses provided by AutoLeARn can help these dealerships and used car lots in providing quick appraisals of cars and checking critical components to make sure that the vehicles meet the necessary standards before being sold. This not only makes the inspection process more efficient for dealerships, but will also improve trust and customer satisfaction as customers can feel confident knowing that the vehicle has been thoroughly checked using the latest standards and technology. References (primarily from the customer insights section for background information) [1] https://www.motor.com/2023/04/ sixty-four-percent-of-drivers-are-putting-off-necessary-car-maintenance/. [2] 2024 survey published by the Auto Care Association using analysis from Hanover Research (https://hedgescompany.com/blog/2024/07/characteristics-automotive-diy-consumers/). [3] Car Parts July 2023 Website Survey (1500 respondents - https://www.carparts.com/blog/ready-to-roll-do-you-truly-know-your-car-repair-proficiency/). [4] 2020 SimpleTire Website Survey (1000 respondents - https://simpletire.com/press/releases/ simpletire-new-survey-car-owners-steering-towards-DIY). [5] https://autosphere.ca/dealerships/2023/05/25/the-challenges-of-automotive-training/. ## README (from the GitHub repository) ## Demo Link: [![Video](https://img.youtube.com/vi/NZYmF-WqMVU&t=3s/0.jpg)](https://www.youtube.com/watch?v=NZYmF-WqMVU&t=3s) # Inspiration ## Introduction Ever had a car issue that you have no idea how to fix? You take it into the shop, wait for a few hours, and then get hit with an astronomical bill—often costing hundreds, if not thousands, of dollars—to fix superficial issues. The thing is, you're not alone. Thousands of people around the world face the same frustrations, not only with struggling to identify what is wrong with their cars, but also with being overcharged by auto repair shops for services that could easily be handled on their own. ## The Problem A recent survey of 1,000 randomly selected U.S. adults reveals alarming insights about car maintenance habits: 46% of drivers admit they’ve paid for a repair that could have been avoided with better upkeep [1]. It’s mainly because people don’t actually know how to care for their cars (whether that be through regular oil changes, brake replacements, or other maintenance work), leading to excessive spending on repairs that, for many, might be financially out of reach. ## Customer Insights See Infographic Above --- # Our Solution This report envisions a future where augmented reality (AR) and large language model (LLM) technology becomes integral in our everyday lives (potentially within the next few years). As hardware continues to evolve, AR devices will become lighter and easier to use, spreading throughout homes, workplaces, and even car maintenance. Leading companies like Google and Meta are already exploring how AR and LLM technologies can work together. With this shift, learning and problem-solving will be more accessible, enabling anyone to easily engage with and master a wide range of skills. Our project is aimed at predicting the possibilities that exist in this world and using them to solve the problem at hand. Introducing **AutoLeARn**, an interactive AR-based learning platform designed to guide and educate car owners through essential maintenance tasks, such as oil changes, brake replacements, and more. AutoLeARn empowers users by helping them learn to maintain their vehicles, prolong their lifespan, and minimize costly repairs by providing clear, comprehensive guidance at every step. The device uses augmented reality to overlay visual instructions directly onto the car, ensuring users can follow along with ease (even while using their tools). ## What it does (key features): - **Speech-to-Text Integration**: Simply specify your car details and the service you want to perform using speech commands. This is further enabled by intuitive hand gestures, allowing users to quickly select and customize tasks. - **Interactive Step-by-Step Guidance**: Take a screenshot using a hand gesture, and AutoLeARn returns a sequence of steps to perform the service. Each step is displayed in a panel, where users can swipe left and right to view different instructions. Every panel includes a step title and detailed description, ensuring the user follows along with ease. - **Integrated Video Assistance**: If more guidance is needed, AutoLeARn provides an option to download and watch a tutorial video. The video is pulled based on the task at hand, and the video player is incorporated into a panel with rotation and translation manipulators for ease of use. - **Step Contextualization**: For added convenience, AutoLeARn can point users to specific instructions based on the current step in the repair process. This feature guides users to the exact step they need (skipping over any parts that AutoLeARn detects the user has already completed). With AutoLeARn, users can educate themselves with hands-on, personalized guidance as they learn the process of maintaining their cars, reducing repair costs, and increasing their confidence in car care. --- # How we built it - **Speech-to-Text Integration**: Leveraging the Unity SDK audio system, and then passing that into Groq Text-To-Speech [using Whisper], we extract the car model, make, year, and the issue with the car through the user’s speech prompt. We take all this information, save it as JSON, and send it back to Unity and save that information within the app. We used all three Groq modalities in this project, which was a new experience for us, especially considering this has never been done before in XR. - **Interactive Step-by-Step Guidance**: We are also using Groq's image-to-text understanding to extract context from what the user is seeing in real time. Through a thumbs-up hand pose analyzed by the Meta Interaction SDK Computer Vision, we trigger a request for a screenshot to capture the car the person is working on through their lens. We send that back-end API to be analyzed by a Vision Language Model (VLM). We then take the text description generated by the VLM and send that to Perplexity (which allows us to do deep research to figure out the necessary steps for the user to follow in the car repair process). We then utilize Groq again to take the response from Perplexity and turn it into a structured JSON, which can then be fed back into Unity for the canvas steps panel display. - **Integrated Video Assistance**: Developed an API to take a Perplexity video citation and download a video tutorial, which is then parsed into the step-by-step segments for maintenance/repair. This required parsing of auto-generated subtitles and importing it back into a reasoning model to split video into informative steps based on the instructions. - **Step Contextualization**: Build as an additional feature to the step-by-step guidance. Leverages the image description to identify which step the user is on and, from there, skip to the current state of the repair process. --- # Challenges we ran into - Due to interface issues between unpredictable LLM output and XR interaction, the feature of having a video ready for any step of the user experience had to be simplified to complete the project within the time frame. - Integrating XR with multiple LLM features and video tutorials, due to the unpredictability of LLM response and current material availability, was a challenge. We were faced with limitations of the Meta Quest 3 which did not include full support for features like display image capture. We had to develop workarounds with a special screen capture API. - Also, we attempted to implement an AI agent-based system but faced limitations in how we wanted the LLM to use the provided tools. Through many iterations, we realized that incorporating AI agents within our project’s scope and timeline would not be feasible. After reaching this conclusion, we simplified our AI system to a chain of LLMs and VLMs that work together to generate steps for the user to follow. --- # Accomplishments that we're proud of We are proud of creating a project that fuses the worlds of AR and advanced AI while innovating an impactful product. With a team of hackers with mixed experience, we were able to work well together as a team and learn a lot collectively. --- # What we learned With a variety of experience going in, we all learned different skills and got to test different tools for the first time. Some learned about API interfacing and testing LLMs such as Groq, Gemini, OpenAI, Deepseek, and Perplexity. Some learned a lot about how development in XR works. We learned a lot about limitations of the technology we work with and implementing innovative workarounds like a special API for image capture on the Meta Quest 3 headset. --- # Looking Forward ## Impact AutoLeARn will dramatically simplify car maintenance for users, making it both easy and accessible for anyone to take charge of their vehicle’s care. With the power of AR, users no longer need to stop in the middle of a repair to search for instructions or watch complicated video tutorials. Instead, they can receive real-time, step-by-step guidance right in front of them, without having to put down their tools. The h [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 12 recognized source files, 44 KB. - C# (language) — detected in the code - Flask (technology) — claimed on Devpost, not found in the code - LangChain (technology) — claimed on Devpost, not found in the code - Python (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 199) ``` .gitignore .idea/.idea.AutoLeARn/.idea/.gitignore .idea/.idea.AutoLeARn/.idea/indexLayout.xml .idea/.idea.AutoLeARn/.idea/vcs.xml Assets/Hand Shapes.meta Assets/Hand Shapes/Screenshot.asset Assets/Hand Shapes/Screenshot.asset.meta Assets/Oculus.meta Assets/Oculus/OculusProjectConfig.asset Assets/Oculus/OculusProjectConfig.asset.meta Assets/Plugins.meta Assets/Plugins/Android.meta Assets/Plugins/Android/AndroidManifest.xml Assets/Plugins/Android/AndroidManifest.xml.meta Assets/Resources.meta Assets/Resources/ImmersiveDebuggerSettings.asset Assets/Resources/ImmersiveDebuggerSettings.asset.meta Assets/Resources/MetaXRAcousticMaterialMapping.asset Assets/Resources/MetaXRAcousticMaterialMapping.asset.meta Assets/Resources/MetaXRAcousticSettings.asset Assets/Resources/MetaXRAcousticSettings.asset.meta Assets/Resources/MetaXRAudioSettings.asset Assets/Resources/MetaXRAudioSettings.asset.meta Assets/Resources/OculusRuntimeSettings.asset Assets/Resources/OculusRuntimeSettings.asset.meta Assets/Resources/OVRPlatformToolSettings.asset Assets/Resources/OVRPlatformToolSettings.asset.meta Assets/Samples.meta Assets/Samples/Meta XR Interaction ​SDK.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/BodyPoseDetectionExamples.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/BodyPoseDetectionExamples.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/ComprehensiveRigExample.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/ComprehensiveRigExample.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/ConcurrentHandsControllersExamples.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/ConcurrentHandsControllersExamples.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/DistanceGrabExamples.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/DistanceGrabExamples.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/GestureExamples.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/GestureExamples.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/HandGrabExamples.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/HandGrabExamples.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/HandGrabUseExamples.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/HandGrabUseExamples.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/HandOnHandInteraction.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/HandOnHandInteraction.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/LocomotionExamples.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/LocomotionExamples.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/PanelWithManipulators.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/PanelWithManipulators.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/PokeExamples.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/PokeExamples.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/PoseExamples.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/PoseExamples.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/RayExamples.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/RayExamples.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/SnapExamples.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/SnapExamples.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/TouchGrabExamples.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/TouchGrabExamples.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/TransformerExamples.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/TransformerExamples.unity.meta Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/UISetExamples.unity Assets/Samples/Meta XR Interaction ​SDK/72.0.0/Example Scenes/UISetExamples.unity.meta Assets/Samples/OpenAI Unity.meta Assets/Samples/OpenAI Unity/0.2.2.meta Assets/Samples/OpenAI Unity/0.2.2/Whisper.meta Assets/Samples/OpenAI Unity/0.2.2/Whisper/fill.png.meta Assets/Samples/OpenAI Unity/0.2.2/Whisper/icons8-microphone-50.png.meta Assets/Samples/OpenAI Unity/0.2.2/Whisper/SaveWav.cs Assets/Samples/OpenAI Unity/0.2.2/Whisper/SaveWav.cs.meta Assets/Samples/OpenAI Unity/0.2.2/Whisper/Whisper Sample.unity Assets/Samples/OpenAI Unity/0.2.2/Whisper/Whisper Sample.unity.meta Assets/Samples/OpenAI Unity/0.2.2/Whisper/Whisper.cs Assets/Samples/OpenAI Unity/0.2.2/Whisper/Whisper.cs.meta Assets/Scenes.meta Assets/Scenes/SampleScene.unity Assets/Scenes/SampleScene.unity.meta Assets/Scripts.meta Assets/Scripts/API.cs Assets/Scripts/API.cs.meta Assets/Scripts/icons8-camera-50.png.meta Assets/Scripts/PoseManager.cs Assets/Scripts/PoseManager.cs.meta Assets/Scripts/SaveWav.cs Assets/Scripts/SaveWav.cs.meta Assets/Scripts/StepManager.cs Assets/Scripts/StepManager.cs.meta Assets/Scripts/STT.cs Assets/Scripts/STT.cs.meta Assets/TextMesh Pro.meta Assets/TextMesh Pro/Documentation.meta Assets/TextMesh Pro/Documentation/TextMesh Pro User Guide 2016.pdf.meta Assets/TextMesh Pro/Fonts.meta Assets/TextMesh Pro/Fonts/LiberationSans - OFL.txt Assets/TextMesh Pro/Fonts/LiberationSans - OFL.txt.meta Assets/TextMesh Pro/Fonts/LiberationSans.ttf.meta Assets/TextMesh Pro/Resources.meta Assets/TextMesh Pro/Resources/Fonts & Materials.meta Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF - Drop Shadow.mat Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF - Drop Shadow.mat.meta Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF - Fallback.asset Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF - Fallback.asset.meta Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF - Outline.mat Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF - Outline.mat.meta Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF.asset Assets/TextMesh Pro/Resources/Fonts & Materials/LiberationSans SDF.asset.meta Assets/TextMesh Pro/Resources/LineBreaking Following Characters.txt Assets/TextMesh Pro/Resources/LineBreaking Following Characters.txt.meta Assets/TextMesh Pro/Resources/LineBreaking Leading Characters.txt Assets/TextMesh Pro/Resources/LineBreaking Leading Characters.txt.meta Assets/TextMesh Pro/Resources/Sprite Assets.meta Assets/TextMesh Pro/Resources/Sprite Assets/EmojiOne.asset Assets/TextMesh Pro/Resources/Sprite Assets/EmojiOne.asset.meta Assets/TextMesh Pro/Resources/Style Sheets.meta Assets/TextMesh Pro/Resources/Style Sheets/Default Style Sheet.asset Assets/TextMesh Pro/Resources/Style Sheets/Default Style Sheet.asset.meta Assets/TextMesh Pro/Resources/TMP Settings.asset Assets/TextMesh Pro/Resources/TMP Settings.asset.meta [79 more files omitted for size] ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Update README.md - Update README.md - Create README.md - steps working - video demo - pause the video with hand - make the canvas interactable - icon change and appear for the panelcanvas - both swipes work - steps working kinda - got this stupid parsing working for the JSON - debugging the http response - debugging the agent - json working - validate output for TTS and make new hand gesture for sc - TTS Sample working - testing STT - import API client - modify the STT - STT File ## Key source files (fetched from GitHub, selected and truncated for size) ### Assets/VideoManager.cs ```c# using System.Collections; using System.Collections.Generic; using UnityEngine; using UnityEngine.Video; public class VideoManager : MonoBehaviour { // Start is called before the first frame update public VideoPlayer videoPlayer; void Start() { } public void TogglePlayPause() { Debug.Log("TogglePlayPause"); if (videoPlayer.isPlaying) { videoPlayer.Pause(); } else { videoPlayer.Play(); } } // Update is called once per frame void Update() { } } ``` ### Assets/Scripts/PoseManager.cs ```c# using System.Collections; using System.Collections.Generic; using UnityEngine; public class PoseManager : MonoBehaviour { // Start is called before the first frame update void Start() { } // Update is called once per frame void Update() { } } ``` ### Assets/Scripts/StepManager.cs ```c# using TMPro; using UnityEngine; using UnityEngine.UI; using UnityEngine.Video; public class StepManager : MonoBehaviour { public TextMeshProUGUI stepDescriptionText; public TextMeshProUGUI stepNumberText; private StepsTutorial stepsTutorial; private int currentStepIndex = 0; public GameObject canvas; public TextMeshProUGUI sttText; public Image stepImage; // Public field for the sprite public Sprite defaultStep; // Public field for the default sprite public VideoPlayer videoPlayer; public APIResponse response; public API api; public void Initialize(APIResponse response) { canvas.SetActive(true); stepsTutorial = response.step_breakdown; // videoPlayer.url = response.video_path; // Debug.Log("Video path: " + response.video_path); currentStepIndex = 0; sttText.text = "Press the record button to start recording your voice"; stepImage.sprite = defaultStep; // Find the current step for (int i = 0; i < stepsTutorial.steps.Length; i++) { if (stepsTutorial.steps[i].is_current_step) { currentStepIndex = i; break; } } Debug.Log("Current step index: " + currentStepIndex); if (response.youtube_link == "") { videoPlayer.enabled = false; } else { videoPlayer.url = response.youtube_link; videoPlayer.Play(); } UpdateUI(); } public void NextStep() { if (stepsTutorial != null && currentStepIndex < stepsTutorial.steps.Length - 1) { stepsTutorial.steps[currentStepIndex].is_current_step = false; currentStepIndex++; stepsTutorial.steps[currentStepIndex].is_current_step = true; UpdateUI(); } } public void PreviousStep() { if (stepsTutorial != null && currentStepIndex > 0) { stepsTutorial.steps[currentStepIndex].is_current_step = false; currentStepIndex--; stepsTutorial.steps[currentStepIndex].is_current_step = true; UpdateUI(); } } private void UpdateUI() { if (stepsTutorial != null && stepsTutorial.steps.Length > 0) { Step currentStep = stepsTutorial.steps[currentStepIndex]; stepDescriptionText.text = currentStep.step_description; stepNumberText.text = $"{currentStep.step_number}/{stepsTutorial.steps.Length}"; } } } ``` ### Assets/Scripts/SaveWav.cs ```c# // Copyright (c) 2012 Calvin Rien // http://the.darktable.com // // This software is provided 'as-is', without any express or implied warranty. In // no event will the authors be held liable for any damages arising from the use // of this software. // // Permission is granted to anyone to use this software for any purpose, // including commercial applications, and to alter it and redistribute it freely, // subject to the following restrictions: // // 1. The origin of this software must not be misrepresented; you must not claim // that you wrote the original software. If you use this software in a product, // an acknowledgment in the product documentation would be appreciated but is not // required. // // 2. Altered source versions must be plainly marked as such, and must not be // misrepresented as being the original software. // // 3. This notice may not be removed or altered from any source distribution. // // ============================================================================= // // derived from Gregorio Zanon's script // http://forum.unity3d.com/threads/119295-Writing-AudioListener.GetOutputData-to-wav-problem?p=806734&viewfull=1#post806734 using System; using System.IO; using UnityEngine; using System.Collections.Generic; public static class SaveWav { private const int HeaderSize = 44; public static byte[] Save(string filename, AudioClip clip) { if (!filename.ToLower().EndsWith(".wav")) { filename += ".wav"; } var filepath = Path.Combine(Application.persistentDataPath, filename); // Make sure directory exists if user is saving to sub dir. Directory.CreateDirectory(Path.GetDirectoryName(filepath) ?? string.Empty); using(var memoryStream = CreateEmpty()) { ConvertAndWrite(memoryStream, clip); WriteHeader(memoryStream, clip); return memoryStream.GetBuffer(); } } public static AudioClip TrimSilence(AudioClip clip, float min) { var samples = new float[clip.samples]; clip.GetData(samples, 0); return TrimSilence(new List(samples), min, clip.channels, clip.frequency); } public static AudioClip TrimSilence(List samples, float min, int channels, int hz, bool stream = false) { int i; for (i = 0; i < samples.Count; i++) { if (Mathf.Abs(samples[i]) > min) { break; } } samples.RemoveRange(0, i); for (i = samples.Count - 1; i > 0; i--) { if (Mathf.Abs(samples[i]) > min) { break; } } samples.RemoveRange(i, samples.Count - i); var clip = AudioClip.Create("TempClip", samples.Count, channels, hz, stream); clip.SetData(samples.ToArray(), 0); return clip; } static MemoryStream CreateEmpty() { var memoryStream = new MemoryStream(); byte emptyByte = new byte(); for (int i = 0; i < HeaderSize; i++) //preparing the header { memoryStream.WriteByte(emptyByte); } return memoryStream; } static void ConvertAndWrite(MemoryStream memoryStream, AudioClip clip) { var samples = new float[clip.samples]; clip.GetData(samples, 0); Int16[] intData = new Int16[samples.Length]; //converting in 2 float[] steps to Int16[], //then Int16[] to Byte[] Byte[] bytesData = new Byte[samples.Length * 2]; //bytesData array is twice the size of //dataSource array because a float converted in Int16 is 2 bytes. int rescaleFactor = 32767; //to convert float to Int16 for (int i = 0; i < samples.Length; i++) { intData[i] = (short)(samples[i] * rescaleFactor); Byte[] byteArr = BitConverter.GetBytes(intData[i]); byteArr.CopyTo(bytesData, i * 2); } memoryStream.Write(bytesData, 0, bytesData.Length); } static void WriteHeader(MemoryStream memoryStream, AudioClip clip) { var hz = clip.frequency; var channels = clip.channels; var samples = clip.samples; memoryStream.Seek(0, SeekOrigin.Begin); Byte[] riff = System.Text.Encoding.UTF8.GetBytes("RIFF"); memoryStream.Write(riff, 0, 4); Byte[] chunkSize = BitConverter.GetBytes(memoryStream.Length - 8); memoryStream.Write(chunkSize, 0, 4); Byte[] wave = System.Text.Encoding.UTF8.GetBytes("WAVE"); memoryStream.Write(wave, 0, 4); Byte[] fmt = System.Text.Encoding.UTF8.GetBytes("fmt "); memoryStream.Write(fmt, 0, 4); Byte[] subChunk1 = BitConverter.GetBytes(16); memoryStream.Write(subChunk1, 0, 4); // UInt16 two = 2; UInt16 one = 1; Byte[] audioFormat = BitConverter.GetBytes(one); memoryStream.Write(audioFormat, 0, 2); Byte[] numChannels = BitConverter.GetBytes(channels); memoryStream.Write(numChannels, 0, 2); Byte[] sampleRate = BitConverter.GetBytes(hz); memoryStream.Write(sampleRate, 0, 4); Byte[] byteRate = BitConverter.GetBytes(hz * channels * 2); // sampleRate * bytesPerSample*number of channels, here 44100*2*2 memoryStream.Write(byteRate, 0, 4); UInt16 blockAlign = (ushort)(channels * 2); memoryStream.Write(BitConverter.GetBytes(blockAlign), 0, 2); UInt16 bps = 16; Byte[] bitsPerSample = BitConverter.GetBytes(bps); memoryStream.Write(bitsPerSample, 0, 2); Byte[] datastring = System.Text.Encoding.UTF8.GetBytes("data"); memoryStream.Write(datastring, 0, 4); Byte[] subChunk2 = BitConverter.GetBytes(samples * channels * 2); memoryStream.Write(subChunk2, 0, 4); // fileStream.Close(); } } ``` ### Assets/Scripts/STT.cs ```c# using System.Collections; using UnityEngine; using UnityEngine.UI; using UnityEngine.Networking; using System.IO; using System.Text; using TMPro; public class STT : MonoBehaviour { [SerializeField] private Button recordButton; [SerializeField] private TextMeshProUGUI message; private readonly string fileName = "output.wav"; private readonly int duration = 5; // Fixed recording duration in seconds private AudioClip clip; private bool isRecording; private string apiUrl = "http://localhost:5000/api/transcribe"; // Replace with actual API URL private string selectedMic = null; private Coroutine recordingCoroutine; private void Start() { #if UNITY_WEBGL && !UNITY_EDITOR message.text = "Microphone not supported on WebGL"; #else var devices = Microphone.devices; if (devices.Length == 0) { message.text = "No microphone detected!"; recordButton.interactable = false; return; } // Print all available microphones Debug.Log("Available Microphones:"); foreach (var device in devices) { Debug.Log(device); } // Select the first microphone selectedMic = devices[0]; // Add listener for the button recordButton.onClick.AddListener(OnRecordButtonPressed); #endif } public void OnRecordButtonPressed() { Debug.Log("Record button pressed"); if (!isRecording) { message.text = "Listening..."; recordingCoroutine = StartCoroutine(RecordForDuration()); } } private IEnumerator RecordForDuration() { StartRecording(); yield return new WaitForSeconds(duration); // Wait for the fixed duration EndRecording(); } private void StartRecording() { if (selectedMic == null) return; isRecording = true; #if !UNITY_WEBGL clip = Microphone.Start(selectedMic, false, duration, 44100); #endif } private void EndRecording() { isRecording = false; message.text = "Transcribing..."; StartCoroutine(ProcessRecording()); } private IEnumerator ProcessRecording() { #if !UNITY_WEBGL Microphone.End(selectedMic); #endif byte[] data = SaveWav.Save(fileName, clip); string filePath = Path.Combine(Application.persistentDataPath, fileName); File.WriteAllBytes(filePath, data); yield return StartCoroutine(UploadAudio(filePath)); } private IEnumerator UploadAudio(string filePath) { byte[] audioData = File.ReadAllBytes(filePath); WWWForm form = new WWWForm(); form.AddBinaryData("audio", audioData, "audio.wav", "audio/wav"); using (UnityWebRequest www = UnityWebRequest.Post(apiUrl, form)) { yield return www.SendWebRequest(); if (www.result == UnityWebRequest.Result.Success) { var responseJson = JsonUtility.FromJson(www.downloadHandler.text); message.text = responseJson.transcript; // Invoke ConvertLangToStruct if transcription is successful string textToConvert = responseJson.transcript; string structType = "CarInfo"; // Change this to the desired struct type StartCoroutine(ConvertLangToStruct(textToConvert, structType)); } else { message.text = "Error: " + www.error; } } } private IEnumerator ConvertLangToStruct(string text, string structType) { TextStructData data = new TextStructData { text = text, type = structType }; string jsonData = JsonUtility.ToJson(data); using (UnityWebRequest request = new UnityWebRequest("http://localhost:5000/api/lang_to_struct", "POST")) { Debug.Log(text); Debug.Log(structType); byte[] bodyRaw = Encoding.UTF8.GetBytes(jsonData); request.uploadHandler = new UploadHandlerRaw(bodyRaw); request.downloadHandler = new DownloadHandlerBuffer(); request.SetRequestHeader("Content-Type", "application/json"); yield return request.SendWebRequest(); if (request.result == UnityWebRequest.Result.Success) { Debug.Log($"Converted struct: {request.downloadHandler.text}"); // Parse the JSON response CarInfo parsedResponse = JsonUtility.FromJson(request.downloadHandler.text); // Check if any field is missing or invalid if (parsedResponse.year == -1 || parsedResponse.issue_with_car == "Unknown" || parsedResponse.make == "Unknown" || parsedResponse.model == "Unknown") { message.text = "Sorry, please ask again - we need the make, model, year, and service."; } else { message.text = $"Issue: {parsedResponse.issue_with_car}\nMake: {parsedResponse.make}\nModel: {parsedResponse.model}\nYear: {parsedResponse.year}"; API.carInfo = parsedResponse; } } else { Debug.LogError($"Failed to convert text to struct: {request.error}"); message.text = "Error processing request."; } } } [System.Serializable] private class TranscriptionResponse { public string transcript; } [System.Serializable] private class TextStructData { public string text; public string type; } } ``` ### Assets/Scripts/API.cs ```c# using System.Collections; using UnityEngine; using UnityEngine.Networking; using System.Text; using System.Diagnostics; using Debug = UnityEngine.Debug; public static class UnityWebRequestExtensions { public static bool IsSuccess(this UnityWebRequest request) { // Check for network errors if (request.isNetworkError || request.isHttpError) { return false; } // Check for successful response codes if (request.responseCode == 0 || request.responseCode == (long)System.Net.HttpStatusCode.OK) { return true; } return false; } } public class API : MonoBehaviour { private const string API_URL = "http://localhost:5000/api"; public APIResponse response; public static CarInfo carInfo; public StepManager stepManager; public void GetWindows() { Debug.Log("Sending request to /windows"); StartCoroutine(SendGetRequest("/windows")); } public void TakeWindowScreenshot(string windowTitle) { Debug.Log("Sending request to /window_screenshot"); StartCoroutine(TakeWindowScreenshotCoroutine(windowTitle)); } private IEnumerator TakeWindowScreenshotCoroutine(string windowTitle) { yield return StartCoroutine( SendPostRequest("/window_screenshot", new ScreenData { window_title = windowTitle })); QueryAgent(carInfo, response.filename); } public void DownloadYouTubeVideo(string url) { Debug.Log("Sending request to /yt_dl"); StartCoroutine(SendPostRequest("/yt_dl", new YouTubeDownloadData { url = url })); } public void ConvertLangToStruct(string text, string type) { Debug.Log("Sending request to /lang_to_struct"); StartCoroutine(SendPostRequest("/lang_to_struct", new TextStructData { text = text, type = type })); } public void QueryAgent(CarInfo carInfo, string imagePath) { Debug.Log("Sending request to /agent"); StartCoroutine(SendPostRequest("/agent", new AgentData { car_info = carInfo, filename = imagePath })); } private IEnumerator SendGetRequest(string endpoint) { using (UnityWebRequest request = UnityWebRequest.Get(API_URL + endpoint)) { yield return request.SendWebRequest(); HandleResponse(request, endpoint); } } private IEnumerator SendPostRequest(string endpoint, T data) { string jsonData = JsonUtility.ToJson(data); using (UnityWebRequest request = new UnityWebRequest(API_URL + endpoint, "POST")) { byte[] bodyRaw = Encoding.UTF8.GetBytes(jsonData); request.uploadHandler = new UploadHandlerRaw(bodyRaw); request.downloadHandler = new DownloadHandlerBuffer(); request.SetRequestHeader("Content-Type", "application/json"); yield return request.SendWebRequest(); HandleResponse(request, endpoint); } } private void HandleResponse(UnityWebRequest request, string endpoint) { if (request.IsSuccess()) { Debug.Log($"Success Response from {endpoint}: " + request.downloadHandler.text); try { // Parse the JSON response into the APIResponse class response = JsonUtility.FromJson(request.downloadHandler.text); Debug.Log("Response object:"); Debug.Log(JsonUtility.ToJson(response, true)); // Pretty print the entire response object if (response != null) { Debug.Log($"Success: {response.success}"); Debug.Log($"Message: {response.message}"); Debug.Log($"Filename: {response.filename}"); Debug.Log($"Original Text: {response.original_text}"); Debug.Log($"Error: {response.error}"); Debug.Log($"YT Link: {response.youtube_link}"); if (response.success) { // Handle success cases based on the endpoint Debug.Log(endpoint); switch (endpoint) { case "/window_screenshot": if (!string.IsNullOrEmpty(response.filename)) { // Save the image name to a variable string savedImageName = response.filename; Debug.Log("Saved Image Name: " + savedImageName); // You can now use the savedImageName variable as needed } break; case "/agent": if (response.step_breakdown != null) { Debug.Log("Step Breakdown: " + JsonUtility.ToJson(response.step_breakdown, true)); Debug.Log("YT Link: " + response.youtube_link); } Debug.Log(JsonUtility.ToJson(response.step_breakdown, true)); stepManager.Initialize(response); break; default: Debug.Log("Response received for endpoint: " + endpoint); break; } } else { Debug.LogError($"API Error: {response.error}"); if (endpoint == "/agent") { Debug.Log("Can't call a function"); retur [truncated — 2389 more characters] ``` ### .idea/.idea.AutoLeARn/.idea/vcs.xml ```xml ``` ### .idea/.idea.AutoLeARn/.idea/indexLayout.xml ```xml ``` ### Assets/Plugins/Android/AndroidManifest.xml ```xml  ``` ### Assets/Samples/OpenAI Unity/0.2.2/Whisper/Whisper.cs ```c# using OpenAI; using UnityEngine; using UnityEngine.UI; namespace Samples.Whisper { public class Whisper : MonoBehaviour { [SerializeField] private Button recordButton; [SerializeField] private Image progressBar; [SerializeField] private Text message; [SerializeField] private Dropdown dropdown; private readonly string fileName = "output.wav"; private readonly int duration = 5; private AudioClip clip; private bool isRecording; private float time; private OpenAIApi openai = new OpenAIApi(); private void Start() { #if UNITY_WEBGL && !UNITY_EDITOR dropdown.options.Add(new Dropdown.OptionData("Microphone not supported on WebGL")); #else foreach (var device in Microphone.devices) { dropdown.options.Add(new Dropdown.OptionData(device)); } recordButton.onClick.AddListener(StartRecording); dropdown.onValueChanged.AddListener(ChangeMicrophone); var index = PlayerPrefs.GetInt("user-mic-device-index"); dropdown.SetValueWithoutNotify(index); #endif } private void ChangeMicrophone(int index) { PlayerPrefs.SetInt("user-mic-device-index", index); } private void StartRecording() { isRecording = true; recordButton.enabled = false; var index = PlayerPrefs.GetInt("user-mic-device-index"); #if !UNITY_WEBGL clip = Microphone.Start(dropdown.options[index].text, false, duration, 44100); #endif } private async void EndRecording() { message.text = "Transcripting..."; #if !UNITY_WEBGL Microphone.End(null); #endif byte[] data = SaveWav.Save(fileName, clip); var req = new CreateAudioTranscriptionsRequest { FileData = new FileData() {Data = data, Name = "audio.wav"}, // File = Application.persistentDataPath + "/" + fileName, Model = "whisper-1", Language = "en" }; var res = await openai.CreateAudioTranscription(req); progressBar.fillAmount = 0; message.text = res.Text; recordButton.enabled = true; } private void Update() { if (isRecording) { time += Time.deltaTime; progressBar.fillAmount = time / duration; if (time >= duration) { time = 0; isRecording = false; EndRecording(); } } } } } ``` [1 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]