# Project export: Polyglot.AI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Polyglot.ai: Revolutionize foreign language learning with AI! Speak and learn with our conversational companion. Dive into an interactive, conversation-based lesson and start your journey to fluency! - Devpost: https://devpost.com/software/polyglot-ai - GitHub: https://github.com/vishaaljamched/TreeHacks2024 - Video: https://www.youtube.com/embed/HFKzAHUQ3h4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Judah Engel (13 commits), Mehul Rao (10 commits), Vishaal Jamched (6 commits), krith-raju (2 commits) ## Devpost submission (written by the team) ### Inspiration As members of immigrant families, we often encounter the significant challenge of language barriers when communicating with family members in our home countries. This issue has become more prevalent since the onset of the COVID-19 pandemic, leading to missed opportunities and lack of meaningful connections with long distance family members. While the straightforward solution might seem to be speaking the language at home and utilizing educational platforms like DuoLingo or Babbel for practice, it's not as simple as it appears. People often lose motivation or focus solely on obtaining the correct answers on these apps. This challenge inspired us to create a more engaging and interactive conversational companion. By offering users the opportunity to engage in real conversations, this tool enhances language practice and improves proficiency through repeated speaking iterations. ### What it does Our web platform offers users the opportunity to learn various languages through a tailored lesson plan upon selecting a language. It includes sub-lessons that involve engagement with articles or short videos on specific topics. This design emulates the interactive experience of a traditional classroom setting. Following the consumption of these materials, users engage with AI Chat Companion for a unique twist on language learning: voice-to-voice conversations. Users speak into the microphone, and their input is processed in real time. The AI companion analyzes the spoken responses and replies with audio feedback. Conversations are seamlessly conducted and can be concluded using a language-specific keyword(eg "terminador" for Spanish). The AI focuses primarily on understanding, with grammar and punctuation as secondary feedback priorities. We had the intent of fostering a conversational environment that enhances speaking skills and provides a comprehensive speaking experience for learners. ### How we built it Our product was developed using a combination of various technologies and frameworks to ensure an ideal interactive user experience. For image conversion, we utilized Covertio, which is a tool used for converting JPEG images into SVG format, which allows us to integrate a high quality vector graphic into the website's UI. The user interface was designed using Reflex, which is a flexible, open-source, full stack python framework which allowed for an efficient deployment of the app. The framework unified front and backend, which allowed us to implement both purely in python. We adopted a sidebar template from Reflex to organize content and ensure an intuitive navigation experience for users. Python was the core of our AI functionality, where the chatbot service and speech-to-text and text-to-speech processing was developed using Python, Open AI APIs, and Speech Text from Google Cloud. The use of Python allowed for access to diverse libraries and frameworks that support AI. The chatbot was integrated using Python enabling real time interactions with users. The component was crucial for simulating the conversation experience. For speech recognition and synthesis, we used Google Cloud's Speech Text APIs, which provided the backbone for our backend implementation for accurate NLP training, and our basis for prompt engineering. The speech detection feature on google cloud allowed for strong capabilities in audio processing. The Open AI API allowed for real time translation within a language during conversations. ### Challenges we ran into The development process involved navigating through a complex landscape of technical challenges. One of our initial struggles was dealing with framework compatibility. The integration of Reflex, which is a significantly different framework compared to Node.js and React, led to a fairly substantial learning curve. Understanding the framework required time to understand the components and best practices to ensure a responsive design. We were able to perform comprehensive research as a team and speak with the sponsors for any issues we had. We performed extensive testing to ensure compatibility. Configuring speech-to-text service to accurately recognize and process spoken language presented difficulties with dialects, accents, and other background noise. Effective prompt engineering that elicited useful responses form the chatbot while ensuring flow of conversation was difficult. We were able to utilize Google Cloud's Speech Text API for its real time speech recognition capabilities. We applied practices in natural language processing and conducted user testing to refine interactions. Deciding on the best platform for speech-to-text development involved in evaluating various options based on latency was another difficulty, along with ensuring the chat bot remained on topic with coherent responses were some of the final backend challenges. ### Accomplishments we're proud of We are proud of figuring out and programming an organized UI platform using an architecture and framework we had no experience with. Working together as a team and problem solving with each through issues in speech to text interpretation, prompt engineering, and training the LLM is something we feel great about. ### What we learned We learned that pivoting to a brand new UI platform with no prior knowledge about is a challenging task, especially regarding the difficulty of having a comprehensive understanding within the duration of the hackathon. We learned how to implement OpenAI APIs and Speech Text APIs from Google Cloud, and the development behind creating speech(from user) to speech(chatbot) communication. We also learned how to divide and conquer, as well as work together on tasks that were difficult. ### What's next Fine tuning the LLM for a more structured response from the chat bot would make the learning experience more ideal. Improving our UIUX process and design it to be more interactive and flow more effectively. Improving the environment of the platform to make the communication purely as speech to speech. ## README (from the GitHub repository) # Welcome to Polygot.AI to run: `pip install -r requirements.txt && reflex run` Make sure to include your OpenAI and/or TogetherAI API key in the `.env` file. ## Inspiration As members of immigrant families, we often encounter the significant challenge of language barriers when communicating with family members in our home countries. This issue has become more prevalent since the onset of the COVID-19 pandemic, leading to missed opportunities and lack of meaningful connections with long distance family members. While the straightforward solution might seem to be speaking the language at home and utilizing educational platforms like DuoLingo or Babbel for practice, it's not as simple as it appears. People often lose motivation or focus solely on obtaining the correct answers on these apps. This challenge inspired us to create a more engaging and interactive conversational companion. By offering users the opportunity to engage in real conversations, this tool enhances language practice and improves proficiency through repeated speaking iterations. ## What it does Our web platform offers users the opportunity to learn various languages through a tailored lesson plan upon selecting a language. It includes sub-lessons that involve engagement with articles or short videos on specific topics. This design emulates the interactive experience of a traditional classroom setting. Following the consumption of these materials, users engage with AI Chat Companion for a unique twist on language learning: voice-to-voice conversations. Users speak into the microphone, and their input is processed in real time. The AI companion analyzes the spoken responses and replies with audio feedback. Conversations are seamlessly conducted and can be concluded using a language-specific keyword(eg "terminador" for Spanish). The AI focuses primarily on understanding, with grammar and punctuation as secondary feedback priorities. We had the intent of fostering a conversational environment that enhances speaking skills and provides a comprehensive speaking experience for learners. ## How we built it Our product was developed using a combination of various technologies and frameworks to ensure an ideal interactive user experience. For image conversion, we utilized Covertio, which is a tool used for converting JPEG images into SVG format, which allows us to integrate a high quality vector graphic into the website's UI. The user interface was designed using Reflex, which is a flexible, open-source, full stack python framework which allowed for an efficient deployment of the app. The framework unified front and backend, which allowed us to implement both purely in python. We adopted a sidebar template from Reflex to organize content and ensure an intuitive navigation experience for users. Python was the core of our AI functionality, where the chatbot service and speech-to-text and text-to-speech processing was developed using Python, Open AI APIs, and Speech Text from Google Cloud. The use of Python allowed for access to diverse libraries and frameworks that support AI. The chatbot was integrated using Python enabling real time interactions with users. The component was crucial for simulating the conversation experience. For speech recognition and synthesis, we used Google Cloud's Speech Text APIs, which provided the backbone for our backend implementation for accurate NLP training, and our basis for prompt engineering. The speech detection feature on google cloud allowed for strong capabilities in audio processing. The Open AI API allowed for real time translation within a language during conversations. ## Challenges we ran into The development process involved navigating through a complex landscape of technical challenges. One of our initial struggles was dealing with framework compatibility. The integration of Reflex, which is a significantly different framework compared to Node.js and React, led to a fairly substantial learning curve. Understanding the framework required time to understand the components and best practices to ensure a responsive design. We were able to perform comprehensive research as a team and speak with the sponsors for any issues we had. We performed extensive testing to ensure compatibility. Configuring speech-to-text service to accurately recognize and process spoken language presented difficulties with dialects, accents, and other background noise. Effective prompt engineering that elicited useful responses form the chatbot while ensuring flow of conversation was difficult. We were able to utilize Google Cloud's Speech Text API for its real time speech recognition capabilities. We applied practices in natural language processing and conducted user testing to refine interactions. Deciding on the best platform for speech-to-text development involved in evaluating various options based on latency was another difficulty, along with ensuring the chat bot remained on topic with coherent responses were some of the final backend challenges. ## Accomplishments that we're proud of We are proud of figuring out and programming an organized UI platform using an architecture and framework we had no experience with. Working together as a team and problem solving with each through issues in speech to text interpretation, prompt engineering, and training the LLM is something we feel great about. ## What we learned We learned that pivoting to a brand new UI platform with no prior knowledge about is a challenging task, especially regarding the difficulty of having a comprehensive understanding within the duration of the hackathon. We learned how to implement OpenAI APIs and Speech Text APIs from Google Cloud, and the development behind creating speech(from user) to speech(chatbot) communication. We also learned how to divide and conquer, as well as work together on tasks that were difficult. ## What's next for Polyglot.AI Fine tuning the LLM for a more structured response from the chat bot would make the learning experience more ideal. Improving our UIUX process and design it to be more interactive and flow more effectively. Improving the environment of the platform to make the communication purely as speech to speech. ## Detected evidence (automated analysis) Indexed codebase: 18 recognized source files, 51 KB. - OpenAI (technology) — detected in the code - Python (language) — detected in the code - Tailwind CSS (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (21 of 21) ``` .gitignore .vscode/settings.json README.md requirements.txt rxconfig.py TreeHacks2024/__init__.py TreeHacks2024/backend/speechToText.py TreeHacks2024/chat_state.py TreeHacks2024/components/__init__.py TreeHacks2024/components/chat.py TreeHacks2024/components/sidebar.py TreeHacks2024/pages/__init__.py TreeHacks2024/pages/index.py TreeHacks2024/pages/leccion1.py TreeHacks2024/pages/leccion2.py TreeHacks2024/pages/leccion3.py TreeHacks2024/prompts.py TreeHacks2024/styles.py TreeHacks2024/templates/__init__.py TreeHacks2024/templates/template.py TreeHacks2024/TreeHacks2024.py ``` ### Dependencies - requirements.txt: google-cloud-speech, openai, python-dotenv, reflex@==0.4.0 ### Recent commits (newest first) - README - fix bottom bar - fix text color - remove text - Done - Just need to add the last prompt - fixed listen button - Merge pull request #1 from vishaaljamched/Transcribe - Merge branch 'main' into Transcribe - functional speech - grammar - testing transcribe - first pass at grammar - add new page - Merge branch 'chat_function' - UI - add styling - added locks - Add files via upload - trying to create interactive ai ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` reflex==0.4.0 openai python-dotenv google-cloud-speech ``` ### TreeHacks2024/pages/index.py ```python """The home page of the app.""" from TreeHacks2024 import styles from TreeHacks2024.templates import template from TreeHacks2024.templates.template import footer import reflex as rx @template(route="/", title="Home", image="/home.svg") def index() -> rx.Component: """The home page. Returns: The UI for the home page. """ header = rx.chakra.container( rx.chakra.hstack( rx.chakra.heading("¡Hola", class_name="text-black-600"), rx.chakra.heading("Vishaal Jamched!", class_name="text-xl"), class_name="justify-center flex-auto" ), rx.chakra.spacer(class_name="p-1"), rx.chakra.divider(), rx.chakra.spacer(class_name="p-2"), rx.chakra.link( rx.chakra.box( rx.chakra.text("Lección 1", class_name="font-bold text-lg no-underline"), rx.chakra.hstack( rx.chakra.text("Progress:", class_name=""), rx.chakra.progress(value=37, width="100%"), rx.chakra.text("37%"), ), class_name="shadow p-5 rounded-md hover:scale-105 hover:cursor-pointer duration-300"), href="/leccion1/intro", class_name="hover:no-underline" ), rx.chakra.box( rx.chakra.text("Lección 2", class_name="font-bold text-lg"), rx.chakra.hstack( rx.chakra.text("Progress:", class_name=""), rx.chakra.progress(value=0, width="100%"), rx.chakra.text("0"), ), class_name="shadow p-5 rounded-md hover:scale-105 duration-300 my-4" ), rx.chakra.box( rx.chakra.text("Lección 3", class_name="font-bold text-lg"), rx.chakra.hstack( rx.chakra.text("Progress:", class_name=""), rx.chakra.progress(value=0, width="100%"), rx.chakra.text("0"), ), class_name="shadow p-5 rounded-md hover:scale-105 duration-300" ), footer() ) return rx.chakra.hstack(header) ``` ### rxconfig.py ```python import reflex as rx config = rx.Config( app_name="TreeHacks2024", tailwind={}, ) ``` ### TreeHacks2024/__init__.py ```python """Base template for Reflex.""" ``` ### TreeHacks2024/TreeHacks2024.py ```python """Welcome to Reflex!.""" from TreeHacks2024 import styles # Import all the pages. from TreeHacks2024.pages import * import reflex as rx class State(rx.State): """Define empty state to allow access to rx.State.router.""" # Create the app. app = rx.App(style=styles.base_style) ``` ### TreeHacks2024/styles.py ```python """Styles for the app.""" import reflex as rx border_radius = "0.375rem" box_shadow = "0px 0px 0px 1px rgba(84, 82, 95, 0.14)" border = "1px solid #F4F3F6" text_color = "black" accent_text_color = "#1A1060" accent_color = "#F5EFFE" hover_accent_color = {"_hover": {"color": accent_color}} hover_accent_bg = {"_hover": {"bg": accent_color}} content_width_vw = "90vw" sidebar_width = "20em" template_page_style = {"padding_top": "2em", "padding_x": ["auto", "2em"], "flex": "1"} template_content_style = { "align_items": "flex-start", "box_shadow": box_shadow, "border_radius": border_radius, "padding": "1em", "margin_bottom": "2em", } link_style = { "color": text_color, "text_decoration": "none", **hover_accent_color, } overlapping_button_style = { "background_color": "white", "border": border, "border_radius": border_radius, } base_style = { rx.chakra.MenuButton: { "width": "3em", "height": "3em", **overlapping_button_style, }, rx.chakra.MenuItem: hover_accent_bg, } ``` ### TreeHacks2024/chat_state.py ```python import reflex as rx from openai import OpenAI import requests import os from dotenv import load_dotenv from TreeHacks2024.backend.speechToText import main from .prompts import DISCUSSION1_PROMPT, DISCUSSION2_PROMPT load_dotenv() TOGETHER_API_KEY = os.environ.get("TOGETHER_API_KEY") OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY") client = OpenAI(api_key=OPENAI_API_KEY) class QA(rx.Base): """A question and answer pair.""" question: str answer: str DEFAULT_CHATS = { "Discussion1": [], "Discussion2": [] } class ChatState(rx.State): """Define empty state to allow access to rx.State.router.""" chats: dict[str, list[QA]] = DEFAULT_CHATS current_chat: str = "Discussion1" question: str processing: bool = False api_type: str prompt: str def set_chat(self, chat_name: str): self.current_chat = chat_name def add_ai_chat(self, ai_chat): prompt = QA(question="", answer=ai_chat) self.chats[self.current_chat].append(prompt) def clear_chat(self, chat_name: str): DEFAULT_CHATS[chat_name] = [] def set_prompt(self, prompt: str): self.prompt = prompt def set_discussion2(self): self.clear_chat("Discussion1") self.clear_chat("Discussion2") self.current_chat = "Discussion2" self.set_prompt(DISCUSSION2_PROMPT) self.add_ai_chat("Antes de resumir el vídeo, da brevemente tu reacción. ¿Qué opinas?") def set_discussion1(self): self.clear_chat("Discussion1") self.clear_chat("Discussion2") self.current_chat = "Discussion1" self.set_prompt(DISCUSSION1_PROMPT) self.add_ai_chat("Antes de resumir el vídeo, da brevemente tu reacción. ¿Qué opinas?") async def transcribe_question(self, form_data: dict[str, str]): self.processing=True yield question = main() self.processing=False if question == "": return question = (" ").join(question.split(" ")[0:-1]) model = self.openai_process_question async for value in model(question): yield value async def process_question(self, form_data: dict[str, str]): question = form_data["question"] if question == "": return model = self.openai_process_question async for value in model(question): yield value async def openai_process_question(self, question: str): """Get the response from the API. Args: form_data: A dict with the current question. """ qa = QA(question=question, answer="") self.chats[self.current_chat].append(qa) messages = [ {"role": "system", "content": self.prompt} ] for qa in self.chats[self.current_chat]: messages.append({"role": "user", "content": qa.question}) messages.append({"role": "assistant", "content": qa.answer}) messages = messages[:-1] print(messages) session = client.chat.completions.create( model="gpt-4-turbo-preview", messages=messages, stream=True, ) for item in session: if hasattr(item.choices[0].delta, "content"): answer_text = item.choices[0].delta.content if answer_text is not None: self.chats[self.current_chat][-1].answer += answer_text yield self.processing = False ``` ### TreeHacks2024/prompts.py ```python DISCUSSION1_PROMPT = """ Eres un profesor de español dando una clase sobre cambio climático. Su estudiante acaba de ver un video y su trabajo es asegurarse de que comprenda firmemente los conceptos cubiertos. Aquí hay un resumen: El cambio climático es el resultado del aumento anormal de la temperatura del planeta, provocado por las actividades humanas. Este fenómeno aumenta e intensifica los desastres naturales como huracanes, sequías e inundaciones, afectando la seguridad alimentaria y los medios de vida de poblaciones enteras. El aumento de los gases de efecto invernadero ha provocado desequilibrios ambientales, resultando en importantes pérdidas económicas; en el caso de Costa Rica, más de 710 millones de dólares. Costa Rica cuenta con áreas y ecosistemas especialmente vulnerables a los efectos del cambio climático, como manglares, arrecifes y zonas montañosas. Los impactos del cambio climático incluyen consecuencias ecológicas, económicas y sociales, como la escasez de agua, problemas en la generación de electricidad debido a las sequías y el aumento de zonas propensas a inundaciones. Finalmente, el video sugiere que, a pesar de ser una amenaza, el cambio climático también representa una oportunidad, posiblemente en alusión a la posibilidad de tomar medidas para mitigar sus efectos y adaptarse a los cambios. El estudiante responderá al mensaje: Resuma el vídeo, por favor. ¿Cuáles son tres puntos principales del video? Trata de usar palabras nuevas. Su trabajo es decidir si el estudiante ha satisfecho la pregunta. Si han proporcionado con precisión tres puntos cubiertos en el video, entonces usted puede decir "¡Genial! Puede pasar a la siguiente discusión. No les dé las respuestas si están fuera de tema; en su lugar, simplemente guíe al estudiante de regreso a El tema." " Siéntase libre de pedirle al estudiante que explique más sobre un punto, pero también sea generoso al decidir si uno de los puntos que menciona coincide con un punto planteado en el video. El estudiante es un hablante nativo de inglés, así que asegúrese de corregir sus errores gramaticales u ortográficos, o sugerirle un mejor vocabulario para que lo use. También anime al estudiante a permanecer en español si cambia al inglés. Ajusta la dificultad de tu vocabulario dependiendo de qué tan cómodo se sienta el estudiante con el español. Antes de profundizar en las preguntas, se preguntará al alumno su reacción ante el vídeo. Siempre que ofrezcan una reacción razonable, puedes continuar respondiendo a sus pensamientos y pedirles que procedan a resumir los puntos principales del video. Siempre que el estudiante cometa un error gramatical, debes corregirlo cortésmente antes de continuar con la conversación. Si el estudiante habla inglés debes animarlo a hablar español. SÓLO DEBES responder en español. """ DISCUSSION2_PROMPT = """ Eres un profesor de español dando una clase sobre cambio climático. Su estudiante acaba de leer un artículo y su trabajo es asegurarse de que comprenda firmemente los conceptos cubiertos. Aquí hay un resumen: El artículo de la NASA analiza los efectos actuales y futuros del cambio climático, destacando fenómenos como la pérdida de hielo marino, el derretimiento de los glaciares, el aumento del nivel del mar, olas de calor más intensas y un aumento de los fenómenos meteorológicos severos. . Los efectos futuros podrían incluir incendios forestales más frecuentes, períodos prolongados de sequía y tormentas tropicales más fuertes. Enfatiza que estos cambios están ocurriendo ahora, son irreversibles en las escalas de tiempo humanas y empeorarán con el tiempo debido a los gases de efecto invernadero producidos por el hombre. El artículo proporciona un resumen detallado de los impactos actuales y futuros del cambio climático, basado en evidencia científica y evaluaciones del Grupo Intergubernamental de Expertos sobre el Cambio Climático (IPCC). Aquí hay un resumen que abarca los puntos clave: Impactos Actuales: Ya estamos experimentando efectos previstos por científicos como la pérdida de hielo marino, derretimiento de glaciares y capas de hielo, aumento del nivel del mar y olas de calor más intensas. Estos efectos son resultado del calentamiento global inducido por el ser humano y son irreversibles en la escala de tiempo de las generaciones actuales. Futuros Efectos Globales: Se anticipa que los efectos del cambio climático se intensificarán, incluyendo incendios forestales más frecuentes, sequías prolongadas en ciertas regiones, y un aumento en la duración e intensidad de tormentas tropicales. Los ecosistemas y la biodiversidad seguirán siendo afectados negativamente, con cambios en los rangos geográficos de diversas especies y alteraciones en los ciclos de crecimiento de plantas. Costos y Daños: Se proyecta que los daños causados por eventos climáticos severos aumentarán, con costos netos significativos que se intensificarán con el tiempo. Cambios en los patrones climáticos, como sequías e incendios forestales, están sucediendo a un ritmo más acelerado de lo que se había evaluado previamente. Aumento de Temperaturas: Las emisiones humanas de gases de efecto invernadero ya han calentado el planeta en casi 2 grados Fahrenheit desde la época preindustrial, con proyecciones que indican un aumento de la temperatura media mundial que alcanzará o superará los 1.5 grados Celsius en las próximas décadas. Efectos en Estados Unidos: Se espera que el nivel del mar aumente entre 1 y 6.6 pies para 2100. Los cambios climáticos, incluyendo un aumento en la intensidad de huracanes, más sequías y olas de calor, así como una temporada de incendios forestales más larga y severa, continuarán afectando a Estados Unidos. Efectos Regionales en EE.UU.: Los impactos del cambio climático variarán significativamente entre regiones, afectando la infraestructura, la economía, los ecosistemas y la salud pública de manera desigual. Desde olas de calor y fuertes aguaceros en el Noreste hasta problemas de suministro de agua en [truncated — 2046 more characters] ``` ### TreeHacks2024/templates/__init__.py ```python from .template import template ``` ### TreeHacks2024/pages/__init__.py ```python from .leccion1 import leccion1 from .leccion2 import leccion2 from .leccion3 import leccion3 from .index import index ``` [7 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]