Project Info
Inspiration
In rural areas, limited power, connectivity, and resources often restrict access to education. Several Asian countries face a stark divide in urban-rural English literacy. English proficiency, a key to unlocking better jobs and higher earnings, remains out of reach for many due to inadequate facilities and a shortage of qualified teachers. LEP (Limited English proficiency) workers earn 25-40% less than their English proficient counterparts. We wanted to experiment with a way of providing personalized education given the constraint of poor connectivity.
What it does
Key components of our hack include the NVIDIA Jetson Nano, Logitech web-camera and AI models like: YOLO (You Only Look Once) v8 Whisper (Open AI: Speech to Text) Kokoro-82M (TTS model) Llama 3.2 distilled model (Dolphin3.0-Llama3.2-1B-GGUF) For our backend, our models are loaded on Jetson. The inputs to our program is a video live stream that is obtained using the Logitech web-camera. The video live stream is then fed through this new fine-tuned model, and it performed well in recognising our faces. The output of YOLOv8 is the class label for our face (yu xi, bryann or dexter). We have a SQLite database of students and their relevant data like reading, writing, vocabulary and speaking scores out of 100. It also contains the numerical progress for each student for each relevant chapter. When the class label is retrieved from YOLOv8, the relevant data is retrieved from the database with sqlite3. This data is passed to a distilled version of Llama3.2, an auto-regressive AI model. After, the text is passed through Llama3.2 and lesson material is curated for the student. For example, if the student is halfway through Chapter 2, the LLM might generate content to continue where the student left off. Alternatively, if the student’s vocabulary scores are very low relative to their other scores, the Llama could come up with a fun game for the student to learn new words that match their level of vocabulary. The text output of Llama3.2 is converted to speech. After it is done, the speech to text model listens for speech from the student and the process repeats. At the end of every session, the student’s progress is updated in the SQL database.
How we built it
Frontend: We built the front-end of Teacher.AI using HTML, Tailwind CSS, and JavaScript to create a clean, interactive, and responsive interface. Tailwind CSS ensures a modern, visually appealing design, while AOS (Animate on Scroll) adds smooth animations for an engaging user experience. The homepage features a hero section with a welcome message, while the Features page includes interactive elements like a face recognition demo using a webcam and a text-to-speech function. Users can create a profile with their name, school year, subject, and knowledge level, and ask questions in real time. Additionally, we implemented a Jetson connection status checker in JavaScript to continuously ping the Jetson and update its online/offline status. The UI dynamically updates progress, displays system messages when the Jetson is connected, and includes a progress tracker with animated progress bars. While JavaScript plays a crucial role in handling webcam integration, speech synthesis, and real-time updates, making Teacher.AI a more immersive and intelligent assistant for students. Backend: We created an original dataset of our selfies and preprocessed them using OpenCV to crop each picture to 640x640 pixels such that it only contained our faces. We then fine tuned the YOLOv8 model on our faces and it performed pretty well, with F1 scores of 97%, indicating a good balance between precision and recall. We set up interconnected servers and piped outputs between Jetson devices, ensuring every build was optimized for CUDA. This approach led to significant performance improvements—for example, while running whisper.cpp on a CPU takes 17 seconds for speech-to-text conversion, leveraging the CUDA capabilities of the Jetson Orin reduced that time to just 0.3 seconds. Additionally, we seamlessly integrated a 1B parameter LLM model without compromising efficiency, achieving information retrieval in approximately 2 seconds—on par with API call response times from OpenAI. Moreover, by connecting a Logitech camera, microphone, and Google AIY Voice Kit to the Jetson, we enabled two-way communication. The GPU’s abundant VRAM ensures smooth interactions with the AI education agent, even when operating offline.
Challenges we ran into
Integrating everything together at the systems level was quite challenging as there were many moving parts. For example, the video livestream caused many problems downstream due to it operating in the same thread as most of the other main code
Accomplishments we're proud of
1, Personalized to each student’s unique needs (Students in personalized learning environments outperformed their peers by 9 percentile points in reading over a year, according to a study by the Bill & Melinda Gates Foundation) Continually updated based on student’s aptitude Does not require internet access Can be battery operated, does not require access to electricity Ability to store large amounts of memory (integrate more advanced models, possibly models for math and science in the future) Lightweight, less than 2kg, 20x20x10cm Learning everywhere, anywhere
What's next
We want to integrate more AI models to enable students to learn other subjects like mathematics and science as well. We hope to make the hardware components of Teacher.AI more integrated, such that it is easier to carry around. A student’s grasp of English is also a lot more than just four competencies, so we hope to better represent a student’s understanding.
This project did not link a GitHub repository.
Analysis
No indexed repository for this project, so there are no commit stats to show.
Technology
- CSSUnchecked
- HTMLUnchecked
- JavaScriptUnchecked
- OpenAIUnchecked
- PythonUnchecked
- SQLUnchecked
- Tailwind CSSUnchecked
No repository was indexed for this project, so these Devpost claims have not been checked against code.
AI coding agents
No repository was indexed, so agent usage could not be checked.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
No repository was indexed, so there is no codebase to measure.
This project did not link a GitHub repository, so there is nothing to diagram.
This project did not link a GitHub repository, so its feature claims have not been checked against code.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.