# Project export: Loqui This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 10.0 - Tagline: Speak like the best. - Devpost: https://devpost.com/software/loqui - GitHub: https://github.com/anshgupta1234/CalHacks_Backend - Demo: https://github.com/anshgupta1234/CalHacks-Frontend - Team: 2 GitHub contributor(s) — skbhujangari (13 commits), Ansh Gupta (7 commits) ## Devpost submission (written by the team) ### Inspiration Our inspiration was from being actively involved in student advocacy and change such as Genup. That often involved working with people who had many years of experience although we were also well informed we didn't have the same years of experience under our belt. This is why we're trying to help people to garner their public speaking skills by creating Loqui. ### What it does Provides you with multiple videos of speakers all around the world who are well known for their public speaking skills. This website allows you to submit a video of yourself and get immediate feedback. ### How we built it We used ReactJS for the frontend and Hume's and Chat gpt's API. We also used single store to store vectors. ### Challenges we ran into Some challenges we ran into was integration and the API we used. The large transcript made it hard to fit into the small API. ### Accomplishments we're proud of We're proud of our similarity search with any amount of data we receive. Given the we weren't given lots of time this is something that is impressive for our group. ### What we learned We learned a lot about integration and working with the Hume API. In addition to that we learned how to work with large transcripts. ### What's next In the future we hope to expand this website and adding features to help people with speech impediments and cater to their needs. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 8 recognized source files, 23 KB. - FastAPI (technology) — detected in the code - LangChain (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - Flask (technology) — claimed on Devpost, not found in the code - React (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (14 of 14) ``` .DS_Store .gitignore api.py db.py example.json extract.py hume_calls.py pipeline.py rand.py rand.txt requirements.txt speech_utils.py transcript.txt utils.py ``` ### Dependencies - requirements.txt: aiohttp@==3.8.6, aiosignal@==1.3.1, annotated-types@==0.6.0, anyio@==3.7.1, async-timeout@==4.0.3, attrs@==23.1.0, backoff@==2.2.1, bcrypt@==4.0.1, build@==1.0.3, cachetools@==5.3.2, certifi@==2023.7.22, charset-normalizer@==3.3.1, chroma-hnswlib@==0.7.3, chromadb@==0.4.15, click@==8.1.7, coloredlogs@==15.0.1, dataclasses-json@==0.6.1, Deprecated@==1.2.14, exceptiongroup@==1.1.3, fastapi@==0.104.0, filelock@==3.13.0, flatbuffers@==23.5.26, frozenlist@==1.4.0, fsspec@==2023.10.0, google-auth@==2.23.3, googleapis-common-protos@==1.61.0, greenlet@==3.0.1, grpcio@==1.59.0, h11@==0.14.0, httptools@==0.6.1, huggingface-hub@==0.17.3, humanfriendly@==10.0, idna@==3.4, importlib-metadata@==6.8.0, importlib-resources@==6.1.0, jsonpatch@==1.33, jsonpointer@==2.4, kubernetes@==28.1.0, langchain@==0.0.325, langsmith@==0.0.53, marshmallow@==3.20.1, monotonic@==1.6, mpmath@==1.3.0, multidict@==6.0.4, mypy-extensions@==1.0.0, numpy@==1.26.1, oauthlib@==3.2.2, onnxruntime@==1.16.1, openai@==0.28.1, opentelemetry-api@==1.20.0, opentelemetry-exporter-otlp-proto-common@==1.20.0, opentelemetry-exporter-otlp-proto-grpc@==1.20.0, opentelemetry-proto@==1.20.0, opentelemetry-sdk@==1.20.0, opentelemetry-semantic-conventions@==0.41b0, overrides@==7.4.0, packaging@==23.2, parsimonious@==0.10.0, posthog@==3.0.2, protobuf@==4.24.4, pulsar-client@==3.3.0, pyasn1@==0.5.0, pyasn1-modules@==0.3.0, pydantic@==2.4.2, pydantic_core@==2.10.1, PyJWT@==2.8.0, PyPika@==0.48.9, pyproject_hooks@==1.0.0, python-dateutil@==2.8.2, python-dotenv@==1.0.0, PyYAML@==6.0.1, regex@==2023.10.3, requests@==2.31.0, requests-oauthlib@==1.3.1, rsa@==4.9, singlestoredb@==0.9.3, six@==1.16.0, sniffio@==1.3.0, SQLAlchemy@==2.0.22, sqlparams@==5.1.0, starlette@==0.27.0, sympy@==1.12, tenacity@==8.2.3, tiktoken@==0.5.1, tokenizers@==0.14.1, tomli@==2.0.1, tqdm@==4.66.1, typer@==0.9.0, typing_extensions@==4.8.0, typing-inspect@==0.9.0, urllib3@==1.26.18, uvicorn@==0.23.2, uvloop@==0.19.0, watchfiles@==0.21.0, websocket-client@==1.6.4, websockets@==12.0, wrapt@==1.15.0, yarl@==1.9.2, zipp@==3.17.0 ### Recent commits (newest first) - update to parse properly - update func get_diff - api and ml method to allow for summarizing a json into a human readable form and humanistic output - all changes - DB search function updated to return an entire row of data as a dictionary given the matching keypoint - fix robustly the add_vector and then play around with updates for search - open and close connections to fit with db - add_vector changes? - changes - yolo - updated db functions with proper formatting for the add vector and to account for the new table and emotion row and defined search sql - yeehawbama - some random commits to boost pipeline and get method - refactor the pipeline, consolidate functions, rewrite code logic and rework system design to holistically make use of existing modules and reduce redundancy. Design new system procedures and implement to address the video/audio pairing mismatch to generate transcripts for keypoint ChatGPT processing upon video upload, a GET function for database procedures to get the emotional data associated with a model embedding, and rework api and system collaboration - db functions for singlestore vector management and ml chatgpt functions for extraction of keypoints + reqs file for source control - i hate vids - sarry sarry - compare - Removed hume key - finished most functions ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` aiohttp==3.8.6 aiosignal==1.3.1 annotated-types==0.6.0 anyio==3.7.1 async-timeout==4.0.3 attrs==23.1.0 backoff==2.2.1 bcrypt==4.0.1 build==1.0.3 cachetools==5.3.2 certifi==2023.7.22 charset-normalizer==3.3.1 chroma-hnswlib==0.7.3 chromadb==0.4.15 click==8.1.7 coloredlogs==15.0.1 dataclasses-json==0.6.1 Deprecated==1.2.14 exceptiongroup==1.1.3 fastapi==0.104.0 filelock==3.13.0 flatbuffers==23.5.26 frozenlist==1.4.0 fsspec==2023.10.0 google-auth==2.23.3 googleapis-common-protos==1.61.0 greenlet==3.0.1 grpcio==1.59.0 h11==0.14.0 httptools==0.6.1 huggingface-hub==0.17.3 humanfriendly==10.0 idna==3.4 importlib-metadata==6.8.0 importlib-resources==6.1.0 jsonpatch==1.33 jsonpointer==2.4 kubernetes==28.1.0 langchain==0.0.325 langsmith==0.0.53 marshmallow==3.20.1 monotonic==1.6 mpmath==1.3.0 multidict==6.0.4 mypy-extensions==1.0.0 numpy==1.26.1 oauthlib==3.2.2 onnxruntime==1.16.1 openai==0.28.1 opentelemetry-api==1.20.0 opentelemetry-exporter-otlp-proto-common==1.20.0 opentelemetry-exporter-otlp-proto-grpc==1.20.0 opentelemetry-proto==1.20.0 opentelemetry-sdk==1.20.0 opentelemetry-semantic-conventions==0.41b0 overrides==7.4.0 packaging==23.2 parsimonious==0.10.0 posthog==3.0.2 protobuf==4.24.4 pulsar-client==3.3.0 pyasn1==0.5.0 pyasn1-modules==0.3.0 pydantic==2.4.2 pydantic_core==2.10.1 PyJWT==2.8.0 PyPika==0.48.9 pyproject_hooks==1.0.0 python-dateutil==2.8.2 python-dotenv==1.0.0 PyYAML==6.0.1 regex==2023.10.3 requests==2.31.0 requests-oauthlib==1.3.1 rsa==4.9 singlestoredb==0.9.3 six==1.16.0 sniffio==1.3.0 SQLAlchemy==2.0.22 sqlparams==5.1.0 starlette==0.27.0 sympy==1.12 tenacity==8.2.3 tiktoken==0.5.1 tokenizers==0.14.1 tomli==2.0.1 tqdm==4.66.1 typer==0.9.0 typing-inspect==0.9.0 typing_extensions==4.8.0 urllib3==1.26.18 uvicorn==0.23.2 uvloop==0.19.0 watchfiles==0.21.0 websocket-client==1.6.4 websockets==12.0 wrapt==1.15.0 yarl==1.9.2 zipp==3.17.0 ``` ### rand.py ```python def get_emotion_for_keypoint(keypoint): keypoint_data[keypoint]["expected"] = expected_speaker_emotions time_info = get_sentence_time_segment(keypoint, speech_info) clip_path = get_clip("yo.mov", time_info) job = get_emotion_data(clip_path) ``` ### hume_calls.py ```python from hume import HumeBatchClient from hume.models.config import FaceConfig from hume.models.config import ProsodyConfig from hume.models.config import BurstConfig def get_emotion_data(video_path): client = HumeBatchClient("gQfbmvuND79DFtBCSRyYki2VPrQY89K2Ijcwxf2FMNP59nPI") urls = [] files = [video_path] configs = [FaceConfig(fps_pred=60), ProsodyConfig(granularity="conversational_turn")] job = client.submit_job(urls, configs, files=files) return job # print(job) # print("Running...") # job.await_complete() # return job.get_predictions() ``` ### speech_utils.py ```python import whisper_timestamped as whisper from io import BytesIO import subprocess import numpy as np from difflib import SequenceMatcher def load_audio_from_video(video_file, sr=16000): """ Extract audio from a video file and read it as a mono waveform, resampling as necessary. Parameters ---------- video_file: file object The video file object received from the API. sr: int The sample rate to resample the audio if necessary. Returns ------- A NumPy array containing the audio waveform, in float32 dtype. """ # Create a BytesIO object to read the video file video_data = video_file.read() video_data_io = BytesIO(video_data) print(video_data_io.getvalue()) # This launches a subprocess to extract audio while down-mixing # and resampling as necessary. Requires the ffmpeg CLI in PATH. cmd = [ "ffmpeg", "-nostdin", "-i", "pipe:0", # Read input from pipe "-f", "s16le", "-ac", "1", "-acodec", "pcm_s16le", "-ar", str(sr), "-" ] out = subprocess.run(cmd, input=video_data_io.read(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True) audio_data = np.frombuffer(out.stdout, np.int16).flatten().astype(np.float32) / 32768.0 return audio_data def get_speech_info(video_path): audio = whisper.load_audio(video_path) model = whisper.load_model("tiny", device="cpu") result = whisper.transcribe(model, audio, language="en") return result def get_sentence_time_segment(sentence, speech_info): for segment in speech_info["segments"]: if SequenceMatcher(None, sentence, segment["text"]).ratio() > 0.45: return (segment["start"], segment["end"]) return (0, 0) ``` ### db.py ```python import singlestoredb as s2 #connection instructions conn_link = 'https://admin:Calhacks123@host:port/database?local_infile=True' host = 'svc-237c8144-15b8-45fa-b7f9-0c2f105c6881-dml.aws-virginia-6.svc.singlestore.com' port = '3306' user = 'admin' password = 'Calhacks123' #add a vector embedding to our database #add a speech/transcript keypoint to our database--this is called per keypoint per speech per speaker def add_vector(speaker_id, transcript_id, keypoint, emotion, vector): conn = s2.connect(host=host, port=port, user=user, password=password, database='openaidb', results_type='tuple') ADD_VECTOR_SQL = f''' INSERT INTO speechvectorspace (speaker_id, transcript_id, keypoint, emotion, vector) VALUES ("{speaker_id}", "{transcript_id}", "{keypoint}", '{emotion}', JSON_ARRAY_PACK("{vector}")); ''' with conn: with conn.cursor() as cur: cur.execute(ADD_VECTOR_SQL) cur.close() #given a speaker id and a model embedding, we'd like to return the emotional data #correlating with the model speaker's embedding def get(speaker_id, embedding): conn = s2.connect(host=host, port=port, user=user, password=password, database='openaidb', results_type='tuple') GET_SQL = f'' with conn: with conn.cursor() as cur: cur.execute(GET_SQL) cur.close() #get the similar embeddings to the given vector embedding #we will be matching a given new set of keypoints with the existing keypoints #After finding the k most smilar keypoints in the model speaker's speech, we'll #return those similar keypoints. We simply are matching the linguistic scenarios #here so we can pass those into hume later for the emotional cues and expressions that we expect def search(embedding): SEARCH_SQL = f'select keypoint, dot_product(vector, JSON_ARRAY_PACK("${embedding}")) as score from speechvectorspace order by score desc limit 1;' conn = s2.connect(host=host, port=port, user=user, password=password, database='openaidb', results_type='tuple') model_closest_keypoint = '' with conn: with conn.cursor() as cur: cur.execute(SEARCH_SQL) row = cur.fetchone() model_closest_keypoint, n = (row) cur.close() print('M: ', model_closest_keypoint) GET_SQL = f'SELECT * from speechvectorspace WHERE keypoint = "{model_closest_keypoint}"' conn = s2.connect(host=host, port=port, user=user, password=password, database='openaidb', results_type='dict') keypoint_dictionary = {} with conn: with conn.cursor() as cur: cur.execute(GET_SQL) row = cur.fetchone() keypoint_dictionary = row cur.close() return keypoint_dictionary ``` ### api.py ```python from flask import Flask, request, jsonify from speech_utils import * from utils import * from extract import * from flask_cors import CORS, cross_origin from hume_calls import get_emotion_data app = Flask(__name__) cors = CORS(app) app.config['CORS_HEADERS'] = 'Content-Type' @app.route('/process_video', methods=['POST']) @cross_origin() def process_video(): # Check if the request contains a file and a speaker string if 'video' not in request.files or 'speaker' not in request.form: return jsonify({'error': 'Video file and speaker string are required'}), 400 video_file = request.files['video'] filename = video_file.filename speaker = request.form['speaker'] video_file.save(filename) # Check if the video file has an allowed extension (e.g., .mp4) allowed_extensions = ['mp4', 'mov'] if '.' in video_file.filename and video_file.filename.rsplit('.', 1)[1].lower() not in allowed_extensions: return jsonify({'error': 'Invalid file format. Allowed formats: .mp4, .mov'}), 400 speech_info = get_speech_info(filename) transcript = speech_info["text"] text_file = open("./temp/transcript_api.txt", "w+") text_file.write(transcript) text_file.close() keypoint_data = {} learner_keypoints = keypoints("./temp/transcript_api.txt") for keypoint in learner_keypoints: # get model kp data keypoint_data[keypoint] = {} # get user kp data time_info = get_sentence_time_segment(keypoint, speech_info) if sum(time_info) > 0: clip_path = get_clip(filename, time_info) embd = str(get_embedding(keypoint)) model_kp_details = search(embd) keypoint_data[keypoint]["expected"] = model_kp_details["emotion"] job = get_emotion_data(clip_path) keypoint_data[keypoint]["job"] = job else: keypoint_data[keypoint]["job"] = None for keypoint in learner_keypoints: if keypoint_data[keypoint]["job"] != None: keypoint_data[keypoint]["job"].await_complete() raw_job_data = keypoint_data[keypoint]["job"].get_predictions() avg_data = post_process(raw_job_data) keypoint_data[keypoint]["actual"] = avg_data suggestions = [] threshold = 0.2 face_mistakes, voice_mistakes, summary = compare(keypoint_data, learner_keypoints, threshold) # Process the video file and speaker string (you can add your custom logic here) # For demonstration purposes, we'll just return the received data return jsonify({ 'face_mistakes': face_mistakes, 'voice_mistakes': voice_mistakes, 'summary': summary, 'filename': video_file.filename, 'response': get_difference_summary(summary) }) if __name__ == '__main__': app.run(debug=True, port=5005) ``` ### pipeline.py ```python from extract import * from speech_utils import * from utils import * from hume_calls import * import json #generate text transcripts for all uploaded videos def read_transcript(path): with open(path) as f: lines = f.readlines() return def process(fileselector, speaker_id): #process all the given files in a directory model_keypoints = [] for fpath in glob.glob(fileselector): kp_associated_data = { 'speaker_id': speaker_id, 'transcript_id': fpath[0:fpath.index('.mov')] + '.txt', 'video_id': fpath, 'speech_info': get_speech_info(fpath), 'keypoints': keypoints(fpath) } model_keypoints.append(kp_associated_data) #for each keypoint #generate embeddings and emotion data to store in Singlestore for keypoint in model_keypoints: #find where the keypoint lingual statement maps to audio with Whisper time_info = get_sentence_time_segment(keypoint, keypoint['speech_info']) clip_path = get_clip(keypoint['video_id'], time_info) emotion = get_emotion_data(clip_path) embd = get_embedding(keypoint) keypoint['model_emotion'] = emotion keypoint['vector_embedding'] = embd return model_keypoints def update_db(data): #at this point, we've populated our model_keypoints with all the necessary data #Let's store this somewhere for use -- namely Singlestore for keypoint in data: add_vector(speaker_id=keypoint['speaker_id'], transcript_id=keypoint['transcript_id'], video_id=keypoint['video_id'], emotion=keypoint['emotion'], vector=keypoint['vector_embedding']) #end of populating the data for one speaker given a set of their videos to analyze #given a new recording, predict the model speaker's intonations and emotions that they'd use in this scenario def predict(user_recording_filepath, transcript_filepath, model_speaker_id): #find new keypoints in this transcript #essentially the same steps as those for the speaker so user_keypoints = aggregate_keypoints(transcript_filepath, model_speaker_id) print(user_keypoints) speech_info = get_speech_info(user_recording_filepath) print(speech_info) #get emotional and speaking cues + the vector embeddings for search purposes for speech in user_keypoints: for keypoint in speech["keypoints"]: #find where the keypoint lingual statement maps to audio with Whisper time_info = get_sentence_time_segment(keypoint, speech_info) if sum(time_info) > 0: print(time_info) clip_path = get_clip(user_recording_filepath, time_info) job = get_emotion_data(clip_path) job.await_complete() user_emotion = job.get_predictions() if user_emotion != None: processed_emotion_info = json.dumps(post_process(user_emotion)) user_embd = str(get_embedding(keypoint)) add_vector(model_speaker_id, transcript_filepath, keypoint, processed_emotion_info, user_embd) print(predict("video2.mp4", "testing.txt", "Joebama")) ``` ### utils.py ```python import subprocess import json def get_clip(video_path, time_info): start_time, end_time = time_info output_path = f"clip_{start_time}_{end_time}.mp4" # Specify the output file name # Run FFmpeg to extract the clip cmd = [ "ffmpeg", "-i", video_path, "-ss", str(start_time), "-to", str(end_time), "-c:v", "libx264", # Specify the video codec (libx264) "-c:a", "aac", # Specify the audio codec (AAC) "-strict", "experimental", # Use experimental AAC encoding output_path ] try: subprocess.run(cmd, check=True) return output_path except subprocess.CalledProcessError as e: print("Error:", e) return None def post_process(job_data): processed_data = { "face_data": {}, "voice_data": {} } predictions = job_data[0]["results"]["predictions"][0] if (len(predictions["models"]["face"]["grouped_predictions"]) > 0): raw_face_data = predictions["models"]["face"]["grouped_predictions"][0]["predictions"] num_face_preds = len(raw_face_data) total_face_data = {} for prediction in raw_face_data: for emotion in prediction["emotions"]: try: total_face_data[emotion["name"]] += emotion["score"] except: total_face_data[emotion["name"]] = emotion["score"] processed_data["face_data"] = { k: v / num_face_preds for k, v in total_face_data.items() } voice_emotions = predictions["models"]["prosody"]["grouped_predictions"][0]["predictions"][0]["emotions"] for emotion in voice_emotions: processed_data["voice_data"][emotion["name"]] = emotion["score"] return processed_data else: return None def compare(keypoint_data, keypoints_list, threshold): face_data_comparisons = [] voice_data_comparisons = [] summary = {} for keypoint in keypoints_list: expected_face = keypoint_data[keypoint]["expected"]["face_data"] expected_voice = keypoint_data[keypoint]["expected"]["voice_data"] actual_face = keypoint_data[keypoint]["actual"]["face_data"] actual_voice = keypoint_data[keypoint]["actual"]["voice_data"] summary[keypoint] = [] for emotion in expected_face.keys(): diff = expected_face[emotion] - actual_face[emotion] if diff > threshold: face_data_comparisons.append({ "keypoint": keypoint, "emotion": emotion, "difference": diff }) summary[keypoint].append({ "emotion": emotion, "difference": diff, "type": "face" }) for emotion in expected_voice.keys(): diff = expected_voice[emotion] - actual_voice[emotion] if abs(diff) > threshold: voice_data_comparisons.append({ "keypoint": keypoint, "emotion": emotion, "difference": diff }) summary[keypoint].append({ "emotion": emotion, "difference": diff, "type": "voice" }) return face_data_comparisons, voice_data_comparisons, summary ``` ### extract.py ```python from langchain.document_loaders import TextLoader from langchain.llms import OpenAI import json from langchain.embeddings.openai import OpenAIEmbeddings from langchain.text_splitter import CharacterTextSplitter from langchain.vectorstores import Chroma from langchain.prompts import PromptTemplate from langchain.chains import RetrievalQA import openai import os import singlestoredb as s2 import glob from db import add_vector, search import json from dotenv import load_dotenv load_dotenv() os.environ["OPENAI_API_KEY"] = os.getenv('OPENAI_API_KEY') openai.api_key = os.environ["OPENAI_API_KEY"] #identify the key points in a transcript #We describe a key point as a major speech point that the speaker makes in an effort #to emphasize a certain concept or emotion. We use ChatGPT which does a good job of #pattern matching and classification to essentially to find keypoints in a speech. Given #the large amount of media, it's processed--it should be adept with speeches. def identify_keypoints(filepath): loader = TextLoader(filepath) docs = loader.load() text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0) texts = text_splitter.split_documents(docs) print("@identify_keypoints") print("\n\n What are the texts? \n\n") print(texts) print("\n\n") embeddings = OpenAIEmbeddings() docsearch = Chroma.from_documents(texts, embeddings) prompt_template = """Use the following pieces of context to answer the question at the end. {context} Question: {question} """ PROMPT = PromptTemplate( template=prompt_template, input_variables=["context", "question"] ) chain_type_kwargs = {"prompt": PROMPT} qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=docsearch.as_retriever()) #Add a confidence level field alongside each field you extract representing how confident you are in your result. query = ''' You are given a speech's transcript here. Your goal here is to identify the key points in this speech. For the purposes of this task, you can define a key point as something that makes a new significant argument in the speech. Follow these guidelines for a keypoint. 1. It is a direct quote from the speech word for word. 2. The maximum size of the keypoint you can choose is one sentence. Do not choose a keypoint consisting of multiple sentences. 3. All keypoints are meant to be unique and shouldn't overlap with other keypoints. 4. It should be relevant and hold high significance to the argument. You MUST express your list where the elements are the keypoints with the opening and ending brace as such: ["keypoint1", "keypoint2", ...] Choose at most 5 keypoints. You MUST give an answer--even if it is an empty list. Do not say "I don't know", provide your best guess. ''' result = qa.run(query) return result def keypoints(filepath): result = identify_keypoints(filepath) if result == "I don't know.": return [] #convert the text output into a list #print("Result of chat is: ", result) begin = result.index("[") end = result.index("]") list_string = result[begin: end + 1] print(list_string) list = json.loads(list_string) return list #now that we have our keypoints, we address the entire labeling problem--primary system #Let's run this method over all the media that we're given to develop a holistic view #aggregating the necessary keypoints for us to run similarity search and determine labels #returns a list of dictionary elements that contain an id with a list of keypoints for the file of filename of id def aggregate_keypoints(fileselector, speaker_id): ungrouped_kps = [] for fpath in glob.glob(fileselector): kp_associated_data = { 'speaker_id': speaker_id, 'transcript_id': fpath, 'keypoints': keypoints(fpath) } ungrouped_kps.append(kp_associated_data) for ungrouped_kp in ungrouped_kps: print(ungrouped_kp['keypoints']) print('\n\n...\n\n') return ungrouped_kps #speakers = aggregate_keypoints('./transcripts/*.txt') def get_embedding(text): return openai.Embedding.create(input = text, model="text-embedding-ada-002")['data'][0]['embedding'] def group_keypoints(spkr_id, trscpt_id, speakers): for speaker in speakers: for kp in speaker['keypoints']: embd = get_embedding(kp) add_vector(spkr_id, trscpt_id, kp, embd) #Difference function to evaluate the differences in speech and emotional cues of #two specific keypoint analyses from Hume.AIg def difference(model_speech_emos, human_speech_emos): return def score(transcript, speaker_id): #identif #return json() #keypoints from your transcript #{kp:[ # your_keypoints, # kp1: { # expected_emotion_data # } # # } speakers = aggregate_keypoints('./data/model/*.txt') #update the database by extracting embedding for each keypoint for speech for each speaker group_keypoints(speaker_id, transcript + uuid(), speakers) #access hume functions customer_kps = aggregate_keypoints('./data/customer/*.txt') differences = [{}] for customer_kp in customer_kps: for customer_keypoint in customer_kp['keypoints']: customer_embedding = get_embedding(customer_keypoint) most_similar_model_keypoint, transcript_id = search(customer_embedding) customer_emotion_at_keypoint = get_human_emotion(customer_keypoint) differences.append({ 'source_customer_keypoint': customer_keypoint, 'nearest_model_keypoint': most_similar_model_keypoint, 'customer_emotion_data': customer_emotion_at_keypoint, 'most_similar_model_data': hume_emotion_from_clip(clip_path(get_timestep_for_keypoint(transcript_id, most_similar_model_keypoint))), 'em [truncated — 2814 more characters] ```