# Project export: SpeakEasy: AI Language Companion
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: Cal Hacks 11.0
- Tagline: Visiting another country but don't want to sound like a robot? Want to learn a new language but can't get your intonation to sound like other people's? SpeakEasy can make you sound like, well, you!
- Devpost: https://devpost.com/software/speakeasy-ai-language-companion
- GitHub: https://github.com/Boomaa23/speak-easy
- Result: winner (Cartesia: Best AI Voice Project)
- Team: 4 GitHub contributor(s) — Zayd Ali (33 commits), Smit Malde (23 commits), Boomaa23 (13 commits), Sidd Shashi (10 commits)
## Devpost submission (written by the team)
### Overview
Overview SpeakEasy: AI Language Companion Visiting another country but don't want to sound like a robot? Want to learn a new language but can't get your intonation to sound like other people's? SpeakEasy can make you sound like, well, you! Features SpeakEasy is an AI language companion which centers around localizing your own voice into other languages. If, for example, you wanted to visit another country but didn't want to sound like a robot or Google Translate, you could still talk in your native language. SpeakEasy can then automatically repeat each statement in the target language in exactly the intonation you would have if you spoke that language. Say you wanted to learn a new language but couldn't quite get your intonation to sound like the source material you were learning from. SpeakEasy is able to provide you phrases in your own voice so you know exactly how your intonation should sound. Background SpeakEasy is the product of a group of four UC Berkeley students. For all of us, this is our first submission to a hackathon and the result of several years of wanting to get together to create something cool together. We are excited to present every part of SpeakEasy; from the remarkably accurate AI speech to just how much we've all learned about rapidly developed software projects.
### Inspiration
Our group started by thinking of ways we could make an impact. We then expanded our search to include using and demonstrating technologies developed by CalHacks' generous sponsors, as we felt this would be a good way to demonstrate how modern technology can be used to help everyday people. In the end, we decided on SpeakEasy and used Cartesia to realize many of the AI-powered functions of the application. This enabled us to make something which addresses a specific real-world problem (robotic-sounding translations) many of us have either encountered or are attempting to avoid. Challenges Our group has varying levels of software development experience, and especially given our limited hackathon experience (read: none), there were many challenging steps. For example: deciding on project scope, designing high-level architecture, implementing major features, and especially debugging. What was never a challenge, however, was collaboration. We worked quite well as a team and had a good time doing it. Accomplishments / Learning We are proud to say that despite the many challenges we accomplished a great deal with this project. We have a fully functional Flask backend with React frontend (see "Technical Details") which uses multiple different APIs. This project successfully ties together audio processing, asynchonrous communication, artificial intelligence, UI/UX design, database management, and so much more. What's more is that many of our group members learned this from base fundamentals. Technical Details As mentioned in an earlier section, SpeakEasy is designed with a Flask (Python) backend and React (JavaScript) frontent. This is a very standard setup that is used often at hackathons due to its easy implementation and relatively limited required setup. Flask only requires two lines of code to make an entirely new endpoint, while React can make a full audio-playing page with callbacks that looks absolutely beautiful in less than an hour. For storing data, we use SQLAlchemy (backed by SQLite). When a user opens SpeakEasy, they are first sent to a landing page. After pressing any key, they are taken to a training screen. Here they will record a 15-20 second message (ideally the one shown on screen) which will be used to create an embedding. This is accomplished with the Cartesia "Clone Voice from Clip" endpoint. A Cartesia Voice (abbreviated as "Voice") is created from the returned embedding (using the "Create Voice" endpoint) which contains a Voice ID. This Voice ID is used to uniquely identify each voice, which itself is in a specific language. The database then stores this voice and creates a new user which this voice is associated with. When the recording is complete and the user clicks "Next", they will be taken to a split screen where they can choose between the two main program functions of SpeakEasy. If the user clicks on the vocal translation route, they will be brought to another recording screen. Here, they record a sound in English which is then sent to the backend. The backend encodes this MP3 data into PCM, sends it to a speech-to-text API, and then transfers it into a text translation API. Separately, the backend trains a new Voice (using the Cartesia Localize Voice endpoint, wrapped by get/create Voice since Localize requires an embedding instead of a Voice ID) with the intended target language and uses the Voice ID it returns. The backend then sends the translated text to the Cartesia "Text to Speech (Bytes)" endpoint using this new Voice ID. This is then played back to the user as a response to the original backend request. All created Voices are stored in the database and associated with the current user. This is done so returning users do not have to retrain their voices in any language. If the user clicks on the language learning route, they will be brought to a page which displays a randomly selected phrase in a certain language. It will then query the Cartesia API to pronounce that phrase in that language, using the preexisting Voice ID if available (or prompting to record a new phrase if not). A request is made to the backend to input some microphone input, which is then compared to Cartesia's estimation of your speech in a target language. The backend then returns a set of feedback using the difference between the two pronounciations, and displays that to the user on the frontend. After each route is selected, the user may choose to go back and select either route (the same route again or the other route). Cartesia Issues We were very impressed with Cartesia and its abilities, but noted a few issues which would improve the development experience. Clone Voice From Clip endpoint documentation The documentation for the endpoint in question details a Response which includes a variety of fields: id, name, language, and more. However, the endpoint only returns the embedding in a dictonary. It is then required to send the embedding into the "Create Voice" endpoint to create an id (and other fields), which are required for some further endpoints. The documentation for the endpoint in question details a Response which includes a variety of fields: id, name, language, and more. However, the endpoint only returns the embedding in a dictonary. It is then required to send the embedding into the "Create Voice" endpoint to create an id (and other fields), which are required for some further endpoints. Clone Voice From Clip endpoint length requirements The clip supplied to the endpoint in question appears to require a duration of greater than a second or two. Se "Error reporting" for further details. The clip supplied to the endpoint in question appears to require a duration of greater than a second or two. Se "Error reporting" for further details. Text to Speech (Bytes) endpoint output format The TTS endpoint requires an output format be specified. This JSON object notably lacks an encoding field in the MP3 configuration which is present for the other formats (raw and WAV). The solution to this is to send an encoding field with the value for one of the other two formats, despite this functionally doing nothing. The TTS endpoint requires an output format be specified. This JSON object notably lacks an encoding field in the MP3 configuration which is present for the other formats (raw and WAV). The solution to this is to send an encoding field with the value for one of the other two formats, despite this functionally doing nothing. Embedding format The embedding is specified as a list of 192 numbers, some of which may be negative. Python's JSON parser does not like the dash symbol and frequently encounters issues with this. If possible, it would be good to either allow this encoding to be base64 encoded, hashed, or something else to prevent negatives. Optimally embeddings do not have negatives, though this seems difficult to realize. The embedding is specified as a list of 192 numbers, some of which may be negative. Python's JSON parser does not like the dash symbol and frequently encounters issues with this. If possible, it would be good to either allow this encoding to be base64 encoded, hashed, or something else to prevent negatives. Optimally embeddings do not have negatives, though this seems difficult to realize. Response code mismatches Some response codes returned from endpoints do not match their listed function. For example, a response code of 405 should not be returned when there is a formatting error in the request. Similarly, 400 is returned before 404 when using invalid endpoints, making it difficult to debug. There are several other instances of this but we did not collate a list. Some response codes returned from endpoints do not match their listed function. For example, a response code of 405 should not be returned when there is a formatting error in the request. Similarly, 400 is returned before 404 when using invalid endpoints, making it difficult to debug. There are several other instances of this but we did not collate a list. Error reporting If (most) endpoints return in JSON format, errors should also be turned in JSON format. This prevents many parsing issues and would simplify design. In addition, error messages are too vague to glean any useful information. For example, 500 is always "Bad request" regardless of the underlying error cause. This is the same thing as the error name. If (most) endpoints return in JSON format, errors should also be turned in JSON format. This prevents many parsing issues and would simplify design. In addition, error messages are too vague to glean any useful information. For example, 500 is always "Bad request" regardless of the underlying error cause. This is the same thing as the error name. Future Improvements In the future, it would be interesting to investigate the following: Proper authentication Cloud-based database storage (with redundancy) Increased error checking Unit and integration test coverage, with CI/CD Automatic recording quality analysis Audio streaming (instead of buffering) using WebSockets Mobile device compatibility Reducing audio processing overhead
## README (from the GitHub repository)
# SpeakEasy
## Overview
SpeakEasy: AI Language Companion
Visiting another country but don't want to sound like a robot? Want to learn a new language but can't get your intonation to sound like other people's? SpeakEasy can make you sound like, well, you!
## Authors
- Sidd Shashi (sshashi@berkeley.edu)
- Zayd Ali (mzali@berkeley.edu)
- Smit Malde (smit334@berkeley.edu)
- boomaa23 (cmo93003@yahoo.com)
## Features
SpeakEasy is an AI language companion which centers around localizing your own voice into other languages.
If, for example, you wanted to visit another country but didn't want to sound like a robot or Google Translate, you could still talk in your native language. SpeakEasy can then automatically repeat each statement in the target language in exactly the intonation you would have if you spoke that language.
Say you wanted to learn a new language but couldn't quite get your intonation to sound like the source material you were learning from. SpeakEasy is able to provide you phrases in your own voice so you know exactly how your intonation should sound.
## Background
SpeakEasy is the product of a group of four UC Berkeley students. For all of us, this is our first submission to a hackathon and the result of several years of wanting to get together to create something cool together. We are excited to present every part of SpeakEasy; from the remarkably accurate AI speech to just how much we've all learned about rapidly developed software projects.
### Inspiration
Our group started by thinking of ways we could make an impact. We then expanded our search to include using and demonstrating technologies developed by CalHacks' generous sponsors, as we felt this would be a good way to demonstrate how modern technology can be used to help everyday people.
In the end, we decided on SpeakEasy and used Cartesia to realize many of the AI-powered functions of the application. This enabled us to make something which addresses a specific real-world problem (robotic-sounding translations) many of us have either encountered or are attempting to avoid.
### Challenges
Our group has varying levels of software development experience, and especially given our limited hackathon experience (read: none), there were many challenging steps. For example: deciding on project scope, designing high-level architecture, implementing major features, and especially debugging.
What was never a challenge, however, was collaboration. We worked quite well as a team and had a good time doing it.
### Accomplishments / Learning
We are proud to say that despite the many challenges we accomplished a great deal with this project. We have a fully functional Flask backend with React frontend (see "Technical Details") which uses multiple different APIs. This project successfully ties together audio processing, asynchonrous communication, artificial intelligence, UI/UX design, database management, and so much more. What's more is that many of our group members learned this from base fundamentals.
## Technical Details
As mentioned in an earlier section, SpeakEasy is designed with a Flask (Python) backend and React (JavaScript) frontent. This is a very standard setup that is used often at hackathons due to its easy implementation and relatively limited required setup. Flask only requires two lines of code to make an entirely new endpoint, while React can make a full audio-playing page with callbacks that looks absolutely beautiful in less than an hour. For storing data, we use SQLAlchemy (backed by SQLite).
1. When a user opens SpeakEasy, they are first sent to a landing page.
2. After pressing any key, they are taken to a training screen. Here they will record a 15-20 second message (ideally the one shown on screen) which will be used to create an embedding. This is accomplished with the Cartesia "Clone Voice from Clip" endpoint. A Cartesia Voice (abbreviated as "Voice") is created from the returned embedding (using the "Create Voice" endpoint) which contains a Voice ID. This Voice ID is used to uniquely identify each voice, which itself is in a specific language. The database then stores this voice and creates a new user which this voice is associated with.
3. When the recording is complete and the user clicks "Next", they will be taken to a split screen where they can choose between the two main program functions of SpeakEasy.
4. If the user clicks on the vocal translation route, they will be brought to another recording screen. Here, they record a sound in English which is then sent to the backend. The backend encodes this MP3 data into PCM, sends it to a speech-to-text API, and then transfers it into a text translation API. Separately, the backend trains a new Voice (using the Cartesia Localize Voice endpoint, wrapped by get/create Voice since Localize requires an embedding instead of a Voice ID) with the intended target language and uses the Voice ID it returns. The backend then sends the translated text to the Cartesia "Text to Speech (Bytes)" endpoint using this new Voice ID. This is then played back to the user as a response to the original backend request. All created Voices are stored in the database and associated with the current user. This is done so returning users do not have to retrain their voices in any language.
5. If the user clicks on the language learning route, they will be brought to a page which displays a randomly selected phrase in a certain language. It will then query the Cartesia API to pronounce that phrase in that language, using the preexisting Voice ID if available (or prompting to record a new phrase if not). A request is made to the backend to input some microphone input, which is then compared to Cartesia's estimation of your speech in a target language. The backend then returns a set of feedback using the difference between the two pronounciations, and displays that to the user on the frontend.
6. After each route is selected, the user may choose to go back and select either route (the same route again or the other route).
## Cartesia Issues
We were very impressed with Cartesia and its abilities, but noted a few issues which would improve the development experience.
- Clone Voice From Clip endpoint documentation
- The documentation for the endpoint in question details a `Response` which includes a variety of fields: `id`, `name`, `language`, and more. However, the endpoint only returns the embedding in a dictonary. It is then required to send the embedding into the "Create Voice" endpoint to create an `id` (and other fields), which are required for some further endpoints.
- Clone Voice From Clip endpoint length requirements
- The clip supplied to the endpoint in question appears to require a duration of greater than a second or two. Se "Error reporting" for further details.
- Text to Speech (Bytes) endpoint output format
- The TTS endpoint requires an output format be specified. This JSON object notably lacks an `encoding` field in the MP3 configuration which is present for the other formats (raw and WAV). The solution to this is to send an `encoding` field with the value for one of the other two formats, despite this functionally doing nothing.
- Embedding format
- The embedding is specified as a list of 192 numbers, some of which may be negative. Python's JSON parser does not like the dash symbol and frequently encounters issues with this. If possible, it would be good to either allow this encoding to be base64 encoded, hashed, or something else to prevent negatives. Optimally embeddings do not have negatives, though this seems difficult to realize.
- Response code mismatches
- Some response codes returned from endpoints do not match their listed function. For example, a response code of 405 should not be returned when there is a formatting error in the request. Similarly, 400 is returned before 404 when using invalid endpoints, making it difficult to debug. There are several other instances of this but
[README truncated for size]
## Detected evidence (automated analysis)
Indexed codebase: 27 recognized source files, 79 KB.
- CSS (language) — detected in the code
- Flask (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
## Codebase structure (from repository index)
### Files (34 of 34)
```
client/.gitignore
client/package.json
client/public/index.html
client/public/manifest.json
client/public/robots.txt
client/README.md
client/src/App.css
client/src/App.js
client/src/components/HomeButton.js
client/src/cookieUtils.js
client/src/index.js
client/src/pages/ChooseToolPage.js
client/src/pages/CommunicatePage.js
client/src/pages/pages.css
client/src/pages/PracticeLangPage.js
client/src/pages/training.css
client/src/pages/TrainingInputPage.js
client/src/pages/WelcomePage.js
client/src/reportWebVitals.js
client/src/Routes.js
client/src/setupTests.js
README.md
server/.env.template
server/.gitignore
server/app.py
server/cartesia.py
server/feedback.py
server/README.md
server/requirements.txt
server/routes.py
server/storage_test.py
server/storage.py
server/words.py
test/sandbox.py
```
### Dependencies
- client/package.json: @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, react@^18.3.1, react-dom@^18.3.1, react-icons@^5.3.0, react-router-dom@^6.27.0, react-scripts@^5.0.1, web-vitals@^2.1.4
- server/requirements.txt: flask@==3.0.3, flask-cors@==5.0.0, Flask-SQLAlchemy@==3.0.4, googletrans@==4.0.0-rc1, pydub@==0.25.1, python-dotenv@==1.0.0, requests@==2.32.3, SpeechRecognition@==3.10.4, werkzeug@==3.0.3
### Recent commits (newest first)
- Pushing fixes made after the hackathon, and addition of temporary clear cookies button
- Fixed some styling
- fix playbutton
- improved feedback
- finalizing
- Working state 6AM
- working!
- feedback working
- loading screen
- added traing stuff FE
- disappear lang screen after selection
- added pages and fixed communicate
- fixed issue with loading first example phrase
- learning almost working
- traininginput FE done
- Add README.md
- wrote api/speak and api/nextphrase
- pushing css and trainingpage
- Merge branch 'master' of https://github.com/Boomaa23/speak-easy
- working demo!
## Key source files (fetched from GitHub, selected and truncated for size)
### server/requirements.txt
```
flask==3.0.3
flask-cors==5.0.0
Flask-SQLAlchemy==3.0.4
googletrans==4.0.0-rc1
pydub==0.25.1
python-dotenv==1.0.0
requests==2.32.3
SpeechRecognition==3.10.4
werkzeug==3.0.3
```
### client/package.json
```
{
"name": "client",
"version": "0.1.0",
"private": true,
"dependencies": {
"@testing-library/jest-dom": "^5.17.0",
"@testing-library/react": "^13.4.0",
"@testing-library/user-event": "^13.5.0",
"react": "^18.3.1",
"react-dom": "^18.3.1",
"react-icons": "^5.3.0",
"react-router-dom": "^6.27.0",
"react-scripts": "^5.0.1",
"web-vitals": "^2.1.4"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
"eslintConfig": {
"extends": [
"react-app",
"react-app/jest"
]
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
}
}
```
### server/app.py
```python
import os
import dotenv
dotenv.load_dotenv()
import flask
from flask_cors import CORS
from werkzeug.exceptions import HTTPException
from routes import api_blueprint
from storage import db
app = flask.Flask(__name__)
CORS(app)
app.config['SQLALCHEMY_DATABASE_URI'] = f'sqlite:///{os.getcwd()}/db.db' # In-memory DB
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
db.init_app(app)
# Create all database tables within the app context
with app.app_context():
db.create_all()
app.register_blueprint(api_blueprint)
@app.teardown_appcontext
def close_connection(exception):
# TODO documentation
db = getattr(flask.g, '_database', None)
if db is not None:
db.close()
@app.errorhandler(HTTPException)
def handle_exception(e):
"""Return JSON instead of HTML for HTTP errors."""
# Start with the correct headers and status code from the error
response = e.get_response()
# Replace the body with JSON
response.data = flask.json.dumps(
flask.jsonify(
e.code,
{
'name': e.name,
'description': e.description,
},
),
indent=4,
)
response.content_type = 'application/json'
return response
#TODO: ADD DB!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
```
### client/src/App.js
```javascript
import React from 'react';
import { BrowserRouter as Router } from 'react-router-dom';
import RoutesComponent from './Routes'; // Import the Routes component
const App = () => {
return (
);
};
export default App;
```
### client/src/index.js
```javascript
import React from 'react';
import ReactDOM from 'react-dom/client';
import App from './App';
import reportWebVitals from './reportWebVitals';
const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
);
// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();
```
### test/sandbox.py
```python
import sys
import os
import requests
# Get the current directory of the script (sandbox.py)
current_dir = os.path.dirname(os.path.abspath(__file__))
# Add the server directory to sys.path
server_dir = os.path.abspath(os.path.join(current_dir, '..', 'server'))
sys.path.append(server_dir)
from feedback import transcribe_audio, compare_transcriptions, generate_suggestions
CARTESIA_API_KEY_2 = "a1b1697a-174b-4afc-8720-b48feca19d51"
#client = Cartesia(api_key=os.environ.get("CARTESIA_API_KEY"))
def test_clone():
# Clone a voice using filepath
url = "https://api.cartesia.ai/voices/clone/clip"
# Open the audio file in binary read mode
with open('clone_voice.mp3', 'rb') as file:
files = { "clip": file }
payload = { "enhance": "true" }
headers = {
"Cartesia-Version": "2024-06-10",
"X-API-Key": CARTESIA_API_KEY_2
}
response = requests.post(url, data=payload, files=files, headers=headers)
print(response.status_code) # To print the status code
print(response.json())
def test_feedback():
# test feedback on improving pronounciation
audio_kentucky = r"C:\Users\mzayd\Downloads\School\CalHacks11\speak-easy\test\kentucky_testing.wav"
audio_hindi = r"C:\Users\mzayd\Downloads\School\CalHacks11\speak-easy\test\hindi_testing.wav"
kentucky_transcribe = transcribe_audio(audio_kentucky)
hindi_transcribe = transcribe_audio(audio_hindi)
diff = compare_transcriptions(kentucky_transcribe, hindi_transcribe)
suggestions = generate_suggestions(diff)
print(suggestions)
#test_feedback()
test_clone()
```
### server/words.py
```python
# Retrieve practice words for different languages
# English words
english_words = [
"House", "Water", "Friend", "Book", "Food",
"Love", "Family", "Dog", "Sun", "Happy"
]
# English phrases
english_phrases = [
"What is your name?",
"How are you today?",
"Where do you live?",
"I would like some coffee, please.",
"Can you help me?",
"What time is it?",
"I am learning English.",
"It’s nice to meet you.",
"I need some water.",
"What do you do for work?"
]
# Spanish words
spanish_words = [
"Casa", "Agua", "Amigo", "Libro", "Comida",
"Amor", "Familia", "Perro", "Sol", "Feliz"
]
# Spanish phrases
spanish_phrases = [
"¿Cómo te llamas?", # What is your name?
"¿Cómo estás hoy?", # How are you today?
"¿Dónde vives?", # Where do you live?
"Me gustaría un café, por favor.", # I would like some coffee, please.
"¿Puedes ayudarme?", # Can you help me?
"¿Qué hora es?", # What time is it?
"Estoy aprendiendo español.", # I am learning Spanish.
"Es un placer conocerte.", # It’s nice to meet you.
"Necesito un poco de agua.", # I need some water.
"¿A qué te dedicas?" # What do you do for work?
]
# French words
french_words = [
"Maison", "Eau", "Ami", "Livre", "Nourriture",
"Amour", "Famille", "Chien", "Soleil", "Heureux"
]
# French phrases
french_phrases = [
"Comment t'appelles-tu ?", # What is your name?
"Comment ça va aujourd'hui ?", # How are you today?
"Où habites-tu ?", # Where do you live?
"Je voudrais un café, s'il vous plaît.", # I would like some coffee, please.
"Peux-tu m'aider ?", # Can you help me?
"Quelle heure est-il ?", # What time is it?
"J'apprends le français.", # I am learning French.
"Enchanté de te rencontrer.", # It’s nice to meet you.
"J'ai besoin d'eau.", # I need some water.
"Que fais-tu dans la vie ?" # What do you do for work?
]
practice_words = { "en": {"words": english_words,
"phrases": english_phrases},
"es": {"words": spanish_words,
"phrases": spanish_phrases},
"fr": {"words": french_words,
"phrases": french_phrases},
}
def practice_word(index=0, language='en'):
return practice_words.get(language).get('words')[index]
def practice_phrase(index=0, language='en'):
return practice_words.get(language).get('phrases')[index]
```
### server/cartesia.py
```python
import os
import json
import dotenv
import requests
dotenv.load_dotenv()
_REQUEST_HEADERS = {
'Cartesia-Version': '2024-10-19',
'X-API-Key': os.getenv('CARTESIA_API_KEY'),
}
def text_to_speech(text, voice_id, language):
return _cartesia_request(
'/tts/bytes',
data=json.dumps({
'model_id': ('sonic-english' if language == 'en' else 'sonic-multilingual'),
'transcript': text,
'voice': {
'mode': 'id',
'id': voice_id,
},
'output_format': {
'container': 'mp3',
'bit_rate': 128000,
'sample_rate': 44100,
'encoding': 'pcm_f32le'
},
'language': language
}),
method='POST',
headers = {
'Cartesia-Version': '2024-10-19',
'X-API-Key': os.getenv('CARTESIA_API_KEY'),
"Content-Type": "application/json"
}
)
def clone_voice(audio_bytes):
return _cartesia_request(
'/voices/clone/clip',
files={'clip': audio_bytes},
data={'enhance': "true"},
method='POST',
headers = _REQUEST_HEADERS
)
def localize_voice(voice_id, target_language):
original_voice = _cartesia_request(
f'/voices/{voice_id}',
method='GET',
headers = _REQUEST_HEADERS
)
og_voice = json.loads(original_voice.content)
localized_embedding = _cartesia_request(
'/voices/localize',
data=json.dumps({
'embedding': og_voice.get('embedding'),
'language': target_language,
'original_speaker_gender': 'male', # TODO auto-select for female speakers
#'dialect': 'us'
}),
method='POST',
headers = {
'Cartesia-Version': '2024-10-19',
'X-API-Key': os.getenv('CARTESIA_API_KEY'),
"Content-Type": "application/json"
}
).json()
localized_voice = create_voice(og_voice.get('name'), target_language, localized_embedding['embedding'])
return localized_voice
def create_voice(name, language, embedding):
return _cartesia_request(
'/voices',
data= json.dumps({
'name': name,
'description': f'{name} ({language})',
'embedding': embedding,
'language': language,
}),
method='POST',
headers = {
'Cartesia-Version': '2024-10-19',
'X-API-Key': os.getenv('CARTESIA_API_KEY'),
"Content-Type": "application/json"
}
)
def _cartesia_request(endpoint, method='GET', **kwargs):
return requests.request(
method=method,
url=f'https://api.cartesia.ai{endpoint}',
**kwargs
)
```
### server/feedback.py
```python
import difflib
import string
import unicodedata
import speech_recognition as sr
from pydub import AudioSegment
def transcribe_audio(file_path):
"""Transcribe the given audio file to text using SpeechRecognition."""
recognizer = sr.Recognizer()
audio = AudioSegment.from_file(file_path)
# Convert audio to a format supported by SpeechRecognition
temp_file = "temp.wav"
audio.export(temp_file, format="wav")
with sr.AudioFile(temp_file) as source:
audio_data = recognizer.record(source) # Read the entire audio file
try:
# Use the Google Web Speech API for transcription
transcription = recognizer.recognize_google(audio_data)
return transcription
except sr.UnknownValueError:
return "Could not understand audio."
except sr.RequestError as e:
return f"Could not request results from Google Speech Recognition service; {e}"
def strip_accents(text):
"""Remove accents from a given text."""
# Normalize the text and remove combining diacritical marks
return ''.join(
char for char in unicodedata.normalize('NFD', text)
if unicodedata.category(char) != 'Mn'
)
def clean_text(text):
"""Remove punctuation and accents, and convert to lowercase."""
# Strip accents
text_no_accents = strip_accents(text)
# Remove punctuation
text_no_punctuation = ''.join(char for char in text_no_accents if char not in string.punctuation)
# Convert to lowercase for case-insensitive comparison
return text_no_punctuation.lower()
def compare_transcriptions(correct_transcription, user_transcription):
"""Compare correct and user transcriptions and return the differences."""
# TODO: give phonetic feedback from comparison of audio files (pitch, freq, etc.)
d = difflib.Differ()
correct_cleaned = clean_text(correct_transcription)
user_cleaned = clean_text(user_transcription)
diff = list(d.compare(correct_cleaned.split(), user_cleaned.split()))
return '\n'.join(diff)
def generate_suggestions(differences):
"""Generate suggestions based on the differences between the transcriptions."""
missing_words = []
extra_words = []
feedback = []
for line in differences.splitlines():
if line.startswith('- '): # Missing in user transcription
word = line[2:]
missing_words.append(word)
#feedback.append(f"Consider practicing the pronunciation of the word: '{word}'.")
elif line.startswith('+ '): # Extra in user transcription
word = line[2:]
extra_words.append(word)
#feedback.append(f"The word '{word}' was not in the expected transcription. Please check if it was pronounced correctly.")
# Add feedback about missing words
if missing_words:
feedback += f"You missed the following word{'s' if len(missing_words) > 1 else ''} from the phrase: {', '.join(missing_words)}. Make sure to practice pronouncing {'these words' if len(missing_words) > 1 else 'this word'}.\n"
else:
feedback += "Great job! You pronounced all the expected words.\n"
# Add feedback about extra words
if extra_words:
feedback += f"You said the following extra word{'s' if len(extra_words) > 1 else ''}: {', '.join(extra_words)}. Try to focus on the exact phrase next time.\n"
suggestions = {
"missing_words": missing_words,
"extra_words": extra_words,
"feedback": feedback
}
return suggestions
```
### server/storage.py
```python
from flask_sqlalchemy import SQLAlchemy
from datetime import datetime
db = SQLAlchemy()
# -------------------------
# Models
# -------------------------
class User(db.Model):
"""
Represents a user in the system.
Each user can have multiple voices associated with them.
"""
user_id = db.Column(db.String, primary_key=True) # Cartesia user_id
created_at = db.Column(db.DateTime, default=datetime.now())
# One-to-Many relationship: A user can have multiple voices
voices = db.relationship('VoiceModel', backref='user', lazy=True)
def __repr__(self):
return f""
# Check if the user speaks a specific language
def speaks_lang(self, lang):
"""Check if the user has a voice model in the given language."""
return any(voice.language == lang for voice in self.voices)
# Retrieve all voice models associated with this user
def get_voices(self):
"""Return all voice models associated with this user."""
return self.voices
# Delete the user and their associated voice models
def delete(self):
"""Delete the user and all their voice models."""
for voice in self.voices:
db.session.delete(voice) # Delete related voices
db.session.delete(self) # Delete the user
db.session.commit()
# Get voice ID from language
def get_voiceid_from_lang(self, lang):
"""Return the ID of the voice with the given language, if it exists."""
if self.speaks_lang(lang):
for voice in self.voices:
if voice.language == lang:
return voice.voice_id
return None
class VoiceModel(db.Model):
"""
Represents a voice associated with a user.
Includes metadata like language and privacy settings.
"""
voice_id = db.Column(db.String, primary_key=True) # Cartesia voice_id
user_id = db.Column(db.String, db.ForeignKey('user.user_id'), nullable=False) # Links to User
is_public = db.Column(db.Boolean, default=False) # Voice privacy setting
description = db.Column(db.Text, nullable=True) # Voice description
language = db.Column(db.String(2), nullable=False) # Language code (e.g., 'en', 'fr')
created_at = db.Column(db.DateTime, default=datetime.now())
def __repr__(self):
return f""
# Get the user associated with this voice
def belongs_to_user(self):
"""Return the user associated with this voice."""
return self.user
# Delete this voice model
def delete(self):
"""Delete this voice model and return success status."""
db.session.delete(self)
db.session.commit()
# Verify if the object was deleted
deleted = VoiceModel.query.get(self.voice_id)
return deleted is None # Return True if deletion succeeded
# -------------------------
# Repository Functions
# -------------------------
def create_user(data):
"""Create a new user from a JSON object."""
user_id = data.get("user_id")
created_at = data.get("created_at", datetime.now())
new_user = User(user_id=user_id, created_at=created_at)
db.session.add(new_user)
db.session.commit()
return new_user
def create_voice(data):
"""Create a new voice model from a JSON object."""
voice_id = data.get("voice_id")
user_id = data.get("user_id") # User ID must exist in the database
language = data.get("language")
is_public = data.get("is_public", False) # Default to False if not provided
description = data.get("description")
created_at = data.get("created_at", datetime.now()) # Use provided date or current date
new_voice = VoiceModel(
voice_id=voice_id,
user_id=user_id,
language=language,
is_public=is_public,
description=description,
created_at=created_at
)
db.session.add(new_voice)
db.session.commit()
return new_voice
def get_user_by_id(user_id):
"""Retrieve a user by their user_id."""
return User.query.get(user_id)
def get_voices_by_user_id(user_id):
"""Retrieve all voice models associated with a user."""
user = get_user_by_id(user_id)
if user:
return user.get_voices()
return []
def user_exists(user_id):
"""Check if a user with the given ID exists in the database."""
return User.query.get(user_id) is not None
```
[16 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]