# Project export: LinguaMedia

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 11.0
- Tagline: Start learning a new language with LinguaMedia! Provide a video link in a foreign language; the app will parse the content and help you practice in a post-learning session and quiz section!
- Devpost: https://devpost.com/software/linguamedia
- GitHub: https://github.com/we-dont-like-javascript/LinguMedia_pinguAI
- Team: 3 GitHub contributor(s) — Honghui-Li-8 (13 commits), Kim Han Nguyen (4 commits), Axel Lorens (1 commits)

## Devpost submission (written by the team)

### Inspiration

Videos are made to help people learn another language by injecting the learning into a popular foreign show.

### What it does

Simply provide a video link in a foreign language. LinguaMedia will parse the content, extract meaningful words, generate subtitles and word bubbles, and help you practice newly learned vocabulary in a post-learning session and quiz section! As the user watches the video, LinguaMedia will parse it in real-time, generating subtitles and popping up keyword chat along the way. Once the video finishes, a review section will display detailed information about each vocabulary word, along with pronunciation audio. This will be followed by a quiz section to reinforce learning.

### How we built it

The app front-end backend is built with Relex.dev The backend logic/task is modulized and chained with fetch.ai and uagent framework. Video transcription is captured using Python script. Transcripts are processed using the GROQ model to extract the meaningful vocabulary. Translation, Description, Pronunciation text, and Quiz content are generated with the Google Gemini model. Pronunciation Audio is generated with Cartesia API

### Challenges we ran into

new framework and technology and documentations to read and learn

### Accomplishments we're proud of

It works functionally, and the app has components.

### What we learned

Modulize a process to break down big problems.

### What's next

Fully link front-end and backend logics for full experience.

## README (from the GitHub repository)

# LinguMedia

## Detected evidence (automated analysis)

Indexed codebase: 13 recognized source files, 57 KB.
- Python (language) — detected in the code
- Google Gemini (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (16 of 16)

```
.gitignore
.vscode/settings.json
agents/cartesiaTextToSpeech.py
agents/flashcardGenerationAgent.py
agents/geminiAccess.py
agents/mainRunner.py
agents/textToAudioAgent.py
examples/cartesiaTextToSpeech_old.py
examples/geminiUsage.py
examples/test.py
examples/translatorAgent.py
LinguMedia_pinguAI/__init__.py
LinguMedia_pinguAI/LinguMedia_pinguAI.py
README.md
requirements.txt
rxconfig.py
```

### Dependencies

- requirements.txt: cartesia, ffmpeg-python, google.generativeai, python-dotenv, reflex@==0.6.3, requests, uagents, websockets@>=12.0

### Recent commits (newest first)

- flashcardGen update
- Merge branch 'main' of https://github.com/we-dont-like-javascript/LinguMedia_pinguAI
- Merge branch 'lesson-generation_HL' of https://github.com/we-dont-like-javascript/LinguMedia_pinguAI
- textToSpeech  update
- flashcardAgent
- communication btw agents
- fixed sizes
- Merge branch 'main' of https://github.com/we-dont-like-javascript/LinguMedia_pinguAI
- updated UI
- text to speech
- gemini example
- Merge branch 'lesson-generation_HL' of https://github.com/we-dont-like-javascript/LinguMedia_pinguAI
- update gitignore
- update on gitignore and requirement
- Merge branch 'main' of https://github.com/we-dont-like-javascript/LinguMedia_pinguAI into lesson-generation_HL
- test files
- reflex init
- Initial commit

## Key source files (fetched from GitHub, selected and truncated for size)

### requirements.txt

```
reflex==0.6.3
requests
uagents
python-dotenv
google.generativeai
cartesia
ffmpeg-python
websockets>=12.0
```

### rxconfig.py

```python
import reflex as rx

config = rx.Config(
    app_name="LinguMedia_pinguAI",
)
```

### examples/translatorAgent.py

```python
from uagents import Agent, Context, Model


class ContextPrompt(Model):
    context: str
    text: str


class Response(Model):
    text: str


agent = Agent()


AI_AGENT_ADDRESS = "agent1q0h70caed8ax769shpemapzkyk65uscw4xwk6dc4t3emvp5jdcvqs9xs32y"


code = "\"飞机\""

prompt = ContextPrompt(
    context="Provide Translation and Explanation of below mandarin word  in english",
    text=code,
)


@agent.on_event("startup")
async def send_message(ctx: Context):
    await ctx.send(AI_AGENT_ADDRESS, prompt)


@agent.on_message(Response)
async def handle_response(ctx: Context, sender: str, msg: Response):
    ctx.logger.info(f"Received response from {sender}: {msg.text}")


if __name__ == "__main__":
    agent.run()

```

### agents/mainRunner.py

```python
import json
from uagents import Agent, Bureau, Context, Model
from textToAudioAgent import textToAudioAgent


class Message(Model):
    jsonStr: str
    
starUpAgent = Agent(
  name="startUp",
  port="8180",
  seed="dasdafs",
  endpoint=["http://127.0.0.1:8180/submit"]
)
@starUpAgent.on_event("startup")
async def test_agents(ctx: Context):
  data = {
    "transcript":"茉莉花",
    "language":"zh"
  }
  print("-- send to textToAudio")
  data_jsonStr = json.dumps(data)
  recipient_address = (
    f"test-agent://{textToAudioAgent.address}"
  )
  response = await ctx.send(recipient_address, Message(jsonStr=data_jsonStr))
  
  print(f"response:{response}")

print(textToAudioAgent.address)
bureau = Bureau()
bureau.add(starUpAgent)
bureau.add(textToAudioAgent)
 
if __name__ == "__main__":
    bureau.run()
```

### examples/geminiUsage.py

```python
import google.generativeai as genai
import os
from dotenv import load_dotenv
import json

# Load environment variables from .env file
load_dotenv()

api_key = os.getenv('GEMINI_API_KEY')
if api_key == None:
  assert("Can't find API key, Check .env file")

genai.configure(api_key=api_key)
generation_config = {
  "temperature": 1,
  "top_p": 0.95,
  "top_k": 64,
  "max_output_tokens": 8192,
  "response_mime_type": "application/json",
}


def getDescription(word):

  model = genai.GenerativeModel(
    model_name="gemini-1.5-flash",
    generation_config=generation_config,
    system_instruction="get the translation and description of given mandarin word into english, provide pronunciation and combine both language for demonstration purpose as you are an language teacher",
    # provide an example output JSON format, and output format to regulate the output
  )

  chat_session = model.start_chat(
    history=[
      {
        "role": "user",
        "parts": [
          word,
        ],
      },
    ]
  )

  response = chat_session.send_message("INSERT_INPUT_HERE")
  
  response_json = json.loads(response.text)
  print(response_json)
  print("------------------------")
  print(response_json["translation"])
  print("------------------------")

  return response.text

print(getDescription("火锅"))
```

### agents/cartesiaTextToSpeech.py

```python
from cartesia import Cartesia
import os
from dotenv import load_dotenv
import requests
import json

load_dotenv()

api_key = os.getenv('CARTESIA_API_KEY')

# transcript = "           火锅"
# language = "zh"
voice_id = "a0e99841-438c-4a64-b679-ae501e7d6091"  # Barbershop Man
model_id = "sonic-multilingual"
voice = {
    "mode": "id",
    "id": voice_id,
    "__experimental_controls": {
      "speed": -1,
      "emotion": ["curiosity:high"]
    }
}
output_format = {
    "container": "wav",
    "encoding": "pcm_f32le",
    "sample_rate": 44100,
}
url = "https://api.cartesia.ai/tts/bytes"
headers = {
    "Cartesia-Version": "2024-06-10",
    "X-API-Key": api_key,
    "Content-Type": "application/json"
}


def cartesiaTextToSpeech(transcript, language, fileName):
  # Data to be sent in the request
  data = {
      "model_id": model_id,
      "transcript": transcript,
      "voice": voice,
      "output_format": output_format,
      "language": language
  }
  # Make the POST request
  response = requests.post(url, headers=headers, data=json.dumps(data))

  # Check the response
  if response.status_code == 200:
      print("Request successful!", transcript)
      try:
        # Handle the response bytes (e.g., saving audio data)
        # with open("./audios/"+str(transcript)+".wav", "wb") as f:
        with open(f"./audios/{fileName}.wav", "wb") as f:
            f.write(response.content)
        
        return f"./audios/{fileName}.wav"
      except Exception as e:
          print(f"error: {e}")
          print(f"Failed to save wav file for \"{str(transcript)}\"")
          return None
  else:
      print(f"Request failed with status code {response.status_code}")
      print(response.text)
      return None


```

### examples/cartesiaTextToSpeech_old.py

```python
import struct
from cartesia import Cartesia
import os
from dotenv import load_dotenv
import wave
import io

load_dotenv()

api_key = os.getenv('CARTESIA_API_KEY')
client = Cartesia(api_key=api_key)

voice_id = "a0e99841-438c-4a64-b679-ae501e7d6091"  # Barbershop Man
model_id = "sonic-english"
transcript = "Hello! Welcome to Cartesia"

output_format = {
    "container": "raw",
    "encoding": "pcm_f32le",
    "sample_rate": 44100,
}

# Set up a WebSocket connection.
ws = client.tts.websocket()

# Variable for raw PCM audio bytes
pcm_bytes = io.BytesIO()

# Generate and stream audio.
for output in ws.send(
    model_id=model_id,
    transcript=transcript,
    voice_id=voice_id,
    stream=True,
    output_format=output_format,
):
    buffer = output["audio"]  # buffer contains raw PCM audio bytes
    pcm_bytes.write(buffer)

# Close the connection to release resources
ws.close()

print("== Done Generation ==")

sample_rate = 44100  # Hz
channels = 1  # Mono
sample_width = 4  # 4 bytes for float32
wav_file = "output.wav"

# read out raw pcm bytes
pcm_bytes.seek(0)
pcm_data = pcm_bytes.read()

# Create a WAV file and set its parameters
with wave.open(wav_file, "wb") as wav_f:
    wav_f.setnchannels(channels)  # Mono
    wav_f.setsampwidth(4)  # 32-bit float = 4 bytes per sample
    wav_f.setframerate(sample_rate)  # 44.1 kHz sample rate

    # Convert the raw PCM float32 data into binary frames
    for i in range(0, len(pcm_data), 4):
        # Read 4 bytes (32 bits, little-endian float) and unpack to float
        float_value = struct.unpack('<f', pcm_data[i:i+4])[0]
        # Pack the float as little-endian 32-bit and write to the WAV file
        wav_f.writeframes(struct.pack('<f', float_value))

print(f"WAV file '{wav_file}' created successfully.")

print("WAV file created")

```

### agents/textToAudioAgent.py

```python
import json
from cartesiaTextToSpeech import cartesiaTextToSpeech
from uagents import Agent, Context, Model

class Message(Model):
    jsonStr: str

class Response(Model):
    jsonStr: str
    errorStr: str

textToAudioPort = 8083
textToAudioAgent = Agent(
  name="textToAudio",
  port=textToAudioPort,
  seed="callhack-11.0-wdljs-textToAudio",
  endpoint=[f"http://127.0.0.1:{str(textToAudioPort)}/submit"]
)


def parseJson(jsonStr):
    try:
        # Attempt to parse the JSON string
        parsed_data = json.loads(jsonStr)
        return parsed_data
    except json.JSONDecodeError as e:
        # Handle parsing error, such as malformed JSON
        print(f"Failed to parse JSON: {e}")
        return None

@textToAudioAgent.on_event("startup")
async def starUp_printing(ctx: Context):
    ctx.logger.info(f"textToAudio agent running, address:{textToAudioAgent.address}")
  
@textToAudioAgent.on_message(model=Message)
async def textToAudio_message_handler(ctx: Context, sender: str, msg: Message):
    data = parseJson(msg.jsonStr)
    
    # input validation
    errorInput = False
    if data == None:
        errorInput = True
    elif "transcript" not in data:
        errorInput = True
    elif "translation" not in data:
        errorInput = True
    elif "language" not in data:
        errorInput = True
    elif not isinstance(data["transcript"], str):
        errorInput = True
    elif not isinstance(data["translation"], str):
        errorInput = True
    elif not isinstance(data["language"], str):
        errorInput = True
      
    if errorInput:
        ctx.logger.error(f"textToAudio received invalid input:{msg.jsonStr}")
        await ctx.send(sender, Response(jsonStr="", errorStr="Error Input"))
        return
    
    # correct input
    audioPath = cartesiaTextToSpeech(data["transcript"], data["language"], data["translation"])
    await ctx.send(sender, Response(jsonStr=audioPath, errorStr=""))
    

```

### agents/flashcardGenerationAgent.py

```python
import json
from uagents import Agent, Context, Model
from textToAudioAgent import textToAudioAgent
from geminiAccess import getDescriptionUseGemini, generateQuizUseGemini


def parseJson(jsonStr):
    try:
        # Attempt to parse the JSON string
        parsed_data = json.loads(jsonStr)
        return parsed_data
    except json.JSONDecodeError as e:
        # Handle parsing error, such as malformed JSON
        print(f"Failed to parse JSON: {e}")
        return None
      
def isDuplicate(word):
  return False # pending implementation

def getRecipientAddress(agent):
    recipient_address = (
      f"test-agent://{agent.address}"
    )
    return recipient_address
  
class Message(Model):
    jsonStr: str

class Response(Model):
    jsonStr: str
    errorStr: str

flashCardGenPort = 8082
flashCardGenAgent = Agent(
    name="flashCardGen",
    port=flashCardGenPort,
    seed="callhack-11.0-wdljs-flashCardGen",
    endpoint=["http://127.0.0.1:{flashCardGenPort}/submit"]
)


@flashCardGenAgent.on_event("startup")
async def starUp_printing(ctx: Context):
    ctx.logger.info(f"flashCardGen agent running, address:{flashCardGenAgent.address}")


@flashCardGenAgent.on_message(model=Message)
async def flashCardGen_message_handler(ctx: Context, sender: str, msg: Message):
    data = parseJson(msg.jsonStr)
    
    # input validation
    errorInput = False
    if data == None:
        errorInput = True
    elif "time_start" not in data:
        errorInput = True
    elif "time_end" not in data:
        errorInput = True
    elif "keywords" not in data:
        errorInput = True
        
    if errorInput:
        ctx.logger.error(f"flashCardGen received invalid input:{msg.jsonStr}")
        await ctx.send(sender, Response(jsonStr="", errorStr="Error Input"))
        return

    # correct input
    try:
      time_start = data["time_start"]
      time_end = data["time_end"]
      
      count = 0
      for keyword in data["keywords"]:
        if count >= 2:
          return # limit the rate of descriptions
        if isDuplicate(keyword["text"]):
          continue 
        
        data = {
          "transcript":keyword["text"],
          "language":data["language"]
        }
        
        # get description and quiz
        description_data = getDescriptionUseGemini(data["transcript"], data["language"])
        quiz_data = generateQuizUseGemini(data["transcript"], data["language"])
        
        # generate audio
        recipient_address = getRecipientAddress(textToAudioAgent)
        data_jsonStr = json.dumps(data)
        response = await ctx.send(recipient_address, Message(jsonStr=data_jsonStr))
        if response.errorStr != "":
          # there is an error
          continue # skip this key
        audioPath = response.jsonStr
        
        # construct final data
        # save to database
        
        # response
        
    except Exception as e:
      print(e)
    await ctx.send(sender, Message(message="Hello there alice."))

```

### agents/geminiAccess.py

```python
import google.generativeai as genai
import os
from dotenv import load_dotenv
import json

# Load environment variables from .env file
load_dotenv()

api_key = os.getenv('GEMINI_API_KEY')
if api_key == None:
  assert("Can't find API key, Check .env file")

genai.configure(api_key=api_key)
generation_config = {
  "temperature": 1,
  "top_p": 0.95,
  "top_k": 64,
  "max_output_tokens": 8192,
  "response_mime_type": "application/json",
}

def languageSelector(language):
  if language == "zh":
    return "mandarin"
  # more languages
  
  return language
  

def getDescriptionUseGemini(word, language):
  language = languageSelector(language)
  model = genai.GenerativeModel(
    model_name="gemini-1.5-flash",
    generation_config=generation_config,
    system_instruction=f"you are a language instructor. Provide the translation and description of given {language} word into english in json format with fields [word], [pronunciation], [translation], [description], combine both language in description for demonstration purpose",
    # provide an example output JSON format, and output format to regulate the output
  )

  chat_session = model.start_chat(
    history=[
      {
        "role": "user",
        "parts": [
          word,
        ],
      },
    ]
  )

  response = chat_session.send_message("INSERT_INPUT_HERE")
  
  try:
    response_json = json.loads(response.text)
    # validate json format
    errorFlag = False
    if "word" not in response_json:
      return None
    elif "pronunciation" not in response_json:
      return None
    elif "translation" not in response_json:
      return None
    elif "description" not in response_json:
      return None
      
    return response_json
  except:
    return None

def generateQuizUseGemini(word, language):
  language = languageSelector(language)
  model = genai.GenerativeModel(
    model_name="gemini-1.5-flash",
    generation_config=generation_config,
    system_instruction=f"you are a language instructor. Provide 1 Synonyms and 1 Antonyms of given english term and 1 random word with description of them in json format with fields [synonyms], [synonymsDescription], [antonyms], [antonymsDescription], [random], [randomDescription]",
    # provide an example output JSON format, and output format to regulate the output
  )

  chat_session = model.start_chat(
    history=[
      {
        "role": "user",
        "parts": [
          word,
        ],
      },
    ]
  )

  response = chat_session.send_message("INSERT_INPUT_HERE")
  
  try:
    response_json = json.loads(response.text)
    # validate json format
    errorFlag = False
    #[synonyms], [synonymsDescription], [antonyms], [antonymsDescription], [random], [randomDescription]
    if "synonyms" not in response_json or "synonymsDescription" not in response_json:
      return None
    elif "antonyms" not in response_json or "antonymsDescription" not in response_json:
      return None
    elif "random" not in response_json or "randomDescription" not in response_json:
      return None
      
    return response_json
  except:
    return None
# print(getDescriptionUseGemini("火锅", "zh"))
# print(generateQuizUseGemini("hot pot", "zh"))
```

[2 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]