# Project export: ShaScam This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: AI-enabled security layer for real-time cell phone conversations - Devpost: https://devpost.com/software/shascam - GitHub: https://github.com/dariuskia/Shascam - Video: https://www.youtube.com/embed/IJ1XevZ5PIM?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Darius (20 commits), Nitya Arora (11 commits) ## Devpost submission (written by the team) ### Inspiration We decided to build this project after noticing the increasing number of phishing and spam calls that the elderly and other individuals experience. Many people are unassuming to the tactics used by spam callers, and easily give away information like credit card numbers, social security numbers and personal details that put them at risk of being subject to financial fraud and even physical danger. Furthermore, most advice regarding scam calls and dealing with them are preventative—they focus on learning to avoid callers in large databases of identified numbers, or educating individuals about template tips to follow that change rapidly over time and are far outrun by the pace at which scammers change strategies and implement new scams. As such, our goal was to build a tool that allows individuals to actively control the processor of engaging with a scam call as opposed to regretting them and being susceptible to new and innovative measures that many scammers employ to catch their next victims. Harnessing the power of LLM inference and real-time data allowed us to do this instantly enough for users to take action against scam callers. ### What it does Our application allows a user to safeguard themselves from being scammed through introducing an intelligent security layer that determines the likelihood that a call is from a scammer as it is occurring. Our interactions are rooted completely in user consent - individuals have full autonomy over their information and are receptive of other's opinions, and can choose to give their Twilio phone number out to individuals not as close to them or marketers and other individuals they may not trust. All you have to do as a user is install our app and create a Twilio phone number for usage as your proxy, so that ShaScam can lay over any potential unwanted spam calls and warn you before you slip into a scammer's trap with well-founded analysis and steps to engage with the other person on the line. ### How we built it We made use of Twilio's API to create proxy phone numbers for users of our application to give away to individuals not close to them. With user consent, we sat over a call initiated by a potential scammer, rerouting it to the Twilio proxy number owned by the user and transcribing real time audio data into text using Google Cloud's speech to text integration with audio packets streamed from Twilio's API. Next, we progressively passed cohesive 12-20 word chunks of transcribed output as it was streamed to a 13B parameter Llama 2 model for inference about the likelihood of whether a caller over the phone was a scammer or not. Finally, we built out an intuitive, minimalistic interface for users to create a Twilio number to use and receive push notifications from to be alerted about a spam call should they receive it and obtain analysis about why the call is suspicious, as well as how they should respond to maximize their safety and ensure the person on the other end of the line is not malicious. ### Challenges we ran into We wanted to ensure that the accuracy of our model in detecting spam calls would not bias itself towards finding every interaction to be with a scammer. This is why we set out to build or find a dataset of spam calls to use. We found that several researchers had approached this problem, but there was no final solution for where to source datasets for this topic, as they were either hidden, or not up-to-par of the density of data we needed to make inferences and fine-tune an existing LLM to inform those inference. Initially, we also set out to try to make the experience of using our product as user-friendly as possible, and tried call-forwarding through iOS/Android phones, which seemed like an excellent way to sit over, transcribe and analyze a call. We soon realized, however, that this process reduced autonomy of the users of our product in controlling when their calls were monitored and introduced latency and platform challenges on Android and iOS. We quickly pivoted to creating proxy numbers that users could control use, monitoring and distribution of, thus giving them control of their own security while enabling scam call prevention for those that wish to safeguard themselves more strictly. We even included voice prompts that adhered to regulation regarding user consent of data collection and utilization, using Text-to-Speech through Twilio to confirm user comfort with being recorded over call. Furthermore, we wanted to finetune the model we were using for LLM inference to reduce generalization errors we noticed with the models we were using being particularly selective and overfitting to the definition and examples we provided of scam calls. We set out to find datasets of normal phone conversations and those recorded from scam calls, and hit a significant obstacle with finding the latter as such data is challenging to find due to individuals not typically being able to record scam calls they are on, and phone calls being difficult to record due to user privacy concerns. While we finally sourced a dataset of chatbot recorded scam calls—the Lenny dataset—and transcribed them, we hit many issues trying to finetune the model with Together API and Monster.API, instead pivoting to using a larger-parameter model that we were able to experiment with and prompt engineer to yield better results than our zero-shot approach with Llama-2. ### Accomplishments we're proud of We are most proud of the progress we've made with analysis of scam calls using LLMs—no existing solution, per our research, is able to parse real-time audio data into text and monitor calls as they happen, before the danger and intentions of a scammer have already affected a user. ### What we learned The problem we are solving with this project is one that has existed for a long time, but has had suboptimal solutions that leave people unsafe. A key learning we had while ideating for this hackathon was that problems that are long standing and as simple to describe as scam call detection have layers of underlying detail and complexity that can be unpacked to lead to higher levels of security. Streaming real-time data into an LLM and communicating its continuously generated results with a clean, frictionless interface for our target demographic—senior citizens, who are primarily the most susceptible to these telephone attacks—was a huge lesson and experience in dealing with threading, parallel programming and integrating inference models with the power of real-time, dynamic data as opposed to the static forms each of us have typically used in projects outside of this hackathon. In particular, dealing with the platform restrictions of calling on iOS vs Android due to tight handles by carriers led us to innovate tremendously with our UI/UX flow and the control flow of our core program. Taking small pivots when we hit roadblocks with instances like call forwarding on iOS and chunking data intelligently to prevent overflowing our inference model with an overwhelming number of requests from streamed, noisy data taught us a lot about dealing with volatile data in-flow and finding technical innovations that move us forward towards our solution quickly. ### What's next for Shascam Alongside providing reasoning for why an attack may potentially be from a scammer, making it possible to categorize different scams experienced by users with respect to recency of certain scripts being experienced or tricks by scammers on call is a goal we have to help slowly educate users about the types of trends they are experiencing and are most susceptible to, introducing a form of personalization of what risks they may be putting themselves in during such conversations. Another area we want to continue in is finetuning. Specifically, we want to finetune an LLM on clean and diverse data which, due to the time constraints and resource constraints, we weren’t able to create and find during the hackathon. ## README (from the GitHub repository) # ShaScam: Live, AI-Enabled Call Security ## Overview ShaScam leverages cutting-edge technologies to protect users from phishing and spam calls in real-time. By integrating Twilio's API for call management and Google Cloud's speech-to-text for live call transcription, alongside advanced LLM inference with a Mixtral-8x7B model, ShaScam provides an innovative defense mechanism against the rapidly evolving threat landscape of scam calls. ## Features - [x] **Real-time Scam Analysis:** Utilizes Twilio for call routing and Google Cloud for transcription, analyzing conversations in real-time to detect scam likelihood. - [x] **Intelligent Call Filtering:** Employs a Mixtral-8x7B model for dynamic analysis of call content, offering immediate risk assessment and response recommendations. - [x] **User-Centric Design:** Offers a simple, intuitive interface for managing Twilio proxy numbers and receiving instant alerts on potential scam calls. ## Technical Challenges and Solutions - **Data Acquisition:** Faced with the scarcity of quality spam call datasets, we navigated through various sources to compile a viable dataset for model training and validation. - **Model Accuracy:** Addressed challenges in model generalization and overfitting by opting for a higher-parameter Mixtral-8x7B model and refining our dataset for enhanced precision in scam detection. - **User Privacy and Autonomy:** Implemented Twilio proxy numbers to maintain user privacy and control over call monitoring, ensuring compliance with data collection and privacy standards. ## Future Directions - [ ] **Enhanced Scam Trend Analysis:** Developing capabilities to categorize and analyze scam trends for personalized user alerts. - [ ] **Model Fine-Tuning:** Continuous improvement of the LLM's accuracy through the acquisition and incorporation of diverse and high-quality data sets. ShaScam represents a significant step forward in the use of AI and real-time data processing for scam call detection and prevention, offering users a proactive tool against potential security threats. Built with ❤️ by Nitya Arora, Tanush Chopra, Sahil Gupta, and Darius Kianersi. ## Detected evidence (automated analysis) Indexed codebase: 4190 recognized source files, 59435 KB. - C (language) — detected in the code - C++ (language) — detected in the code - CSS (language) — detected in the code - Firebase (technology) — detected in the code - Flask (technology) — detected in the code - Hugging Face (technology) — detected in the code - Java (language) — detected in the code - JavaScript (language) — detected in the code - Kotlin (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Swift (language) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 7684) ``` .DS_Store .gitignore legacy/shascam/.expo-shared/assets.json legacy/shascam/.gitignore legacy/shascam/.svn/.watchman-cookie-MacBook-Pro.local-625-2750 legacy/shascam/.svn/entries legacy/shascam/.svn/format legacy/shascam/.svn/pristine/00/003458a46525e8114e89111a9dadf6129e2748dc.svn-base legacy/shascam/.svn/pristine/00/009f5ad9d4f02d56f92359bb4ce9696a4175b202.svn-base legacy/shascam/.svn/pristine/01/01808937dcc9e74e08c485d2dfd1c1042474baf4.svn-base legacy/shascam/.svn/pristine/01/01ea802d382c0386442b9e5cc74f41ea2b17af84.svn-base legacy/shascam/.svn/pristine/02/0256b00fa2272b2c21e643a05c9615e400b7997b.svn-base legacy/shascam/.svn/pristine/02/0286e49e719883da5544a9257640a0055b3b503c.svn-base legacy/shascam/.svn/pristine/02/028c0591f58611282ac5f3b6e831d1936a8c56e6.svn-base legacy/shascam/.svn/pristine/02/02b7db99ebcd70fd5ce6933a6ac270904bdc0530.svn-base legacy/shascam/.svn/pristine/02/02cce7fd1e6125158c23a951b8ccfd4e1de7f2f7.svn-base legacy/shascam/.svn/pristine/03/030fb414e5b72eb35dcd770f880330df19ae108c.svn-base legacy/shascam/.svn/pristine/03/033c6aebe67bb583f7ac883f08e00262a44e9144.svn-base legacy/shascam/.svn/pristine/03/0370d4acd23cc3cc7b78bac21cfbe7cabee18fbb.svn-base legacy/shascam/.svn/pristine/03/03889908b23659b36153ea09b56fa62c4f07ab46.svn-base legacy/shascam/.svn/pristine/03/03e3d6753d07a8a1b3ad1066b2cd07988c497641.svn-base legacy/shascam/.svn/pristine/04/0405b3553699b4536835da3c35ace479a4d079f5.svn-base legacy/shascam/.svn/pristine/04/0406fe75d9a4b4280582109fd9e60e38130d9137.svn-base legacy/shascam/.svn/pristine/04/0460d6d9a527efdcdf47986321611e1a7f22c5ac.svn-base legacy/shascam/.svn/pristine/04/04645f7ee9d9a75dac2209622c94368dd2d943df.svn-base legacy/shascam/.svn/pristine/04/04692d94d441c18bb1be77d48a8c1ae78d8cb262.svn-base legacy/shascam/.svn/pristine/04/04704e857a962a5cf718489ea31d3632203c2735.svn-base legacy/shascam/.svn/pristine/04/04989b817c3dbbd02bc2446cfd78098800e899b6.svn-base legacy/shascam/.svn/pristine/04/04b351c67fcc622e61da02cf6a2e029fcb914810.svn-base legacy/shascam/.svn/pristine/04/04bb4323460f64fb4c2f97c8c24f9cc005d61af4.svn-base legacy/shascam/.svn/pristine/04/04e0056c9de2d9a141844779db0bdbf833e1beaa.svn-base legacy/shascam/.svn/pristine/05/0580eed387dafa88bef5f08fed76a6e52062c959.svn-base legacy/shascam/.svn/pristine/05/05b3146d99a5b967f9e6d23e8b8750c4206d1532.svn-base legacy/shascam/.svn/pristine/05/05e81c775d3e8127dcf93d8ee334949971e4fe5a.svn-base legacy/shascam/.svn/pristine/06/0663260671099bdddcf1cb5eb2b193d3e1fb50f1.svn-base legacy/shascam/.svn/pristine/06/06dfa6cc6d02a9a0d3149bfeea63f8a74a9d644f.svn-base legacy/shascam/.svn/pristine/07/070e02c8e79ba3be6174f78b1fd82ce8da2db204.svn-base legacy/shascam/.svn/pristine/07/07b5217ab7d87c99a6ba6f8b0cb7e05c8a015cdb.svn-base legacy/shascam/.svn/pristine/07/07d477ba708acbf4711956aa4148ad355203e7e3.svn-base legacy/shascam/.svn/pristine/08/084f00fbeb17d9d28dc455d0570d6173fcc59fac.svn-base legacy/shascam/.svn/pristine/08/08d512c9ad8b652c8ae89f6242c5ad2824577912.svn-base legacy/shascam/.svn/pristine/08/08e60a5773f85bf063f5265c0fa00e51b5db8697.svn-base legacy/shascam/.svn/pristine/08/08e902e08916f42dbf666db4c84b22ec6bb69c0d.svn-base legacy/shascam/.svn/pristine/08/08f548f825df23c75e47a786d3d278cd1c7b307f.svn-base legacy/shascam/.svn/pristine/08/08f66ce68c7c5c50cbec18ac16df14da2d895d29.svn-base legacy/shascam/.svn/pristine/08/08f7777ec168b0effaa28b57afc50ff9a44ce84f.svn-base legacy/shascam/.svn/pristine/09/0919c7d09886e90105021c98047266bf7a6efbd5.svn-base legacy/shascam/.svn/pristine/09/0920736426a8577efb3a6f6d264f9764b3ee8578.svn-base legacy/shascam/.svn/pristine/09/092ed6716c0cc536c4f777cb613dac5d5f1f5c15.svn-base legacy/shascam/.svn/pristine/09/098a6f75179c7a7f0cf752dcc238a0703f632774.svn-base legacy/shascam/.svn/pristine/09/09c7218ba299900c08282c961c37355bce82110c.svn-base legacy/shascam/.svn/pristine/0a/0a4b11fb264cbaa3c46c21f8c2fada3588808c27.svn-base legacy/shascam/.svn/pristine/0a/0a60f1e18870eeb5486a752e8ad8564a44426851.svn-base legacy/shascam/.svn/pristine/0a/0a7598d021cd1fba28fb4148647fe90171d2162e.svn-base legacy/shascam/.svn/pristine/0a/0aa2eca868c637aa2a50c180aaf8fef231c379e8.svn-base legacy/shascam/.svn/pristine/0a/0accb5ee5b94a0b7c6c2004186d5e76f1b2cf789.svn-base legacy/shascam/.svn/pristine/0a/0ae1a2e1493de205731e68494de838f2f1574816.svn-base legacy/shascam/.svn/pristine/0a/0aeff593d02191c68582c690224a846461614587.svn-base legacy/shascam/.svn/pristine/0b/0b2911d6c58d1a989f275698a8fa0d82c4d0f00f.svn-base legacy/shascam/.svn/pristine/0b/0b2c429eb5bcc9aea8065b182fcf6b7a29b174a3.svn-base legacy/shascam/.svn/pristine/0b/0b42b3e3c179a5fb8ca50017cd3d3bb4dbcb25bd.svn-base legacy/shascam/.svn/pristine/0b/0b506b5882fdb3ea8d1a9ae70b937dce227c352d.svn-base legacy/shascam/.svn/pristine/0b/0b60e7eb5c9ae38136211b29788271ddf2711d45.svn-base legacy/shascam/.svn/pristine/0b/0b6798a1d0fbd56ffb7549769935e524c34517ce.svn-base legacy/shascam/.svn/pristine/0b/0b69f81551e3f72470983482e03f7a7ffbb92567.svn-base legacy/shascam/.svn/pristine/0b/0bc336d06e36a9e2f01ecf32a6773da50044c6b8.svn-base legacy/shascam/.svn/pristine/0b/0bcc4e4bad14242bfee5589861f18306a42b120c.svn-base legacy/shascam/.svn/pristine/0b/0bda3da74b16379c4a8d19298483f143525054ff.svn-base legacy/shascam/.svn/pristine/0c/0c05a005c3b28c8918337a725eebe418700492ac.svn-base legacy/shascam/.svn/pristine/0c/0c0cb6f1197d9c6ed0060372fe2e9521611fc3fd.svn-base legacy/shascam/.svn/pristine/0c/0c89eb623f61ff7bab6a5d98ff09d82038f5b3f3.svn-base legacy/shascam/.svn/pristine/0c/0cce458f03ea99d8559b481dbca31778e529e09d.svn-base legacy/shascam/.svn/pristine/0c/0ce5f260b99cce4b9ceb62daa45ba1cbfa6155ba.svn-base legacy/shascam/.svn/pristine/0c/0ce9df50d25c2efdbad149523334f3d738acbdfe.svn-base legacy/shascam/.svn/pristine/0d/0d6ddfcea3b84fc174db7a7d606a09819c0424fc.svn-base legacy/shascam/.svn/pristine/0e/0e073a17cc39a55b59016f477fb8d6088ce6296c.svn-base legacy/shascam/.svn/pristine/0e/0e134831ae2b1534702bc59b9afb7bf1c82abb05.svn-base legacy/shascam/.svn/pristine/0e/0e32006ce5946e7890f4dedd24b98b8c540f6b8d.svn-base legacy/shascam/.svn/pristine/0e/0e4808feb3ede80462b45bbe34b65d47c578cd01.svn-base legacy/shascam/.svn/pristine/0e/0e620eec29de03a781df0452061b2bb474692b15.svn-base legacy/shascam/.svn/pristine/0e/0ea3ea88a8f1ba084e0ba19aac29b5719915fe0c.svn-base legacy/shascam/.svn/pristine/0e/0efa74b988e85cfba4e963d8e1de578921b00532.svn-base legacy/shascam/.svn/pristine/0f/0f2c71bb1f3c25176ef616a60adc10741fe786ce.svn-base legacy/shascam/.svn/pristine/0f/0f3486c2683161e90cc52f8dbeb85a43447b887d.svn-base legacy/shascam/.svn/pristine/0f/0f4e2a3a11aaa4f6289c1ff13b06b3bff13ad42f.svn-base legacy/shascam/.svn/pristine/0f/0f6450086da05e365256b1bf1cc89980a624f8c0.svn-base legacy/shascam/.svn/pristine/0f/0fba62e0b8f6b515ad9e4360446e0315043d0b87.svn-base legacy/shascam/.svn/pristine/0f/0fe3620118716f61b36b375df07eb70684540b36.svn-base legacy/shascam/.svn/pristine/10/100453c3a04544568a2fa1b662a48bad2b29ad9d.svn-base legacy/shascam/.svn/pristine/10/10b271fffc66bb566db06e463d7d3ab0460517c1.svn-base legacy/shascam/.svn/pristine/10/10dd497c90936c2b9d1d24994580619451df846e.svn-base legacy/shascam/.svn/pristine/11/11ebaa9a8d58725fabbb3e87b943d996d88eb81a.svn-base legacy/shascam/.svn/pristine/12/120be635872413c1b03c0bfa13a2f7a34d72c2a3.svn-base legacy/shascam/.svn/pristine/12/1252249aafba84c1275efb47bd2a08251f98dfd1.svn-base legacy/shascam/.svn/pristine/12/126a9b8e75bb6bebd37fd67726297c1c900ed154.svn-base legacy/shascam/.svn/pristine/12/12d1f18a65da17008f53b85a741c483b548f015a.svn-base legacy/shascam/.svn/pristine/12/12dac0b68e420b97bf239f3eb6185d41dc9fcab0.svn-base legacy/shascam/.svn/pristine/12/12ee68bb5b69a0333bb64b4d5935a2f05dc9a862.svn-base legacy/shascam/.svn/pristine/13/13590e8688cba03e8c11ed55bf8a3f5f16db188a.svn-base legacy/shascam/.svn/pristine/13/13d91f6111c67b5431f170051a6e8ac07a18f0fb.svn-base legacy/shascam/.svn/pristine/13/13dd39030aea096db95a6dc743656c99fc31f4a0.svn-base legacy/shascam/.svn/pristine/13/13f2877292e0d4c487d4d18b00331f7ff1b46af9.svn-base legacy/shascam/.svn/pristine/14/140169347af74253e9c610f730272e2508126724.svn-base legacy/shascam/.svn/pristine/14/1411e96978fe3d9b4847ce2ddb0b75cb29a47f00.svn-base legacy/shascam/.svn/pristine/14/143aa51f7f06cb647b544d5193fa8d7d1e0d7081.svn-base legacy/shascam/.svn/pristine/14/14a3c3fd35adcf7ee99c558ead3f023cd16d4444.svn-base legacy/shascam/.svn/pristine/14/14d61432b50b447a6d6b02a1bec4717033431fcc.svn-base legacy/shascam/.svn/pristine/14/14f91d4faec1c5c215b98b0ade22b7714e738a89.svn-base legacy/shascam/.svn/pristine/15/15379aee297001f69318bf5e6a349e3db70d217b.svn-base legacy/shascam/.svn/pristine/15/153c88d69d3c487985457e57110f14c79bf47f1a.svn-base legacy/shascam/.svn/pristine/15/15736431147438f50a329ca45b92a42f45a7948a.svn-base legacy/shascam/.svn/pristine/16/160710bc65df3e78c6f63328d5da74faa66c4960.svn-base legacy/shascam/.svn/pristine/16/1673538ed9fb696b0b2b38ea48bb2ac319f4cea3.svn-base legacy/shascam/.svn/pristine/16/16a64708a95d73f3d94919e1aba17faacc9a3a75.svn-base legacy/shascam/.svn/pristine/16/16a80634845081f6796586de898ebdd0a42693b5.svn-base legacy/shascam/.svn/pristine/16/16dfab47762b5c15e57510ba9e619e2ec37b9426.svn-base legacy/shascam/.svn/pristine/16/16f3085fefb5193f60466ce50a853ad7c7cedbce.svn-base legacy/shascam/.svn/pristine/17/1753f0564ce18115f9abf941d7289b8bdc3c35d5.svn-base legacy/shascam/.svn/pristine/17/177f41629fd93ca77b6bcb7d2639b59e38da90d8.svn-base legacy/shascam/.svn/pristine/17/178978c4d98dce15ca2330f4d6f8a62e725aa740.svn-base [7564 more files omitted for size] ``` ### Dependencies - legacy/shascam/package.json: @aws-sdk/client-cognito-identity@^3.131.0, @aws-sdk/client-s3@^3.131.0, @aws-sdk/credential-provider-cognito-identity@^3.131.0, @babel/core@^7.20.0, @freakycoder/react-native-bounceable@^0.2.5, @jesster2k10/react-native-range-slider@^0.3.1, @miblanchard/react-native-slider@^2.1.0, @mxenabled/react-native-widget-sdk@^1.1.1, @ptomasroos/react-native-multi-slider@^2.2.2, @react-native-async-storage/async-storage@1.18.2, @react-native-community/checkbox@^0.5.12, @react-native-community/datetimepicker@7.2.0, @react-native-community/geolocation@^3.0.2, @react-native-community/slider@4.4.2, @react-native-firebase/app@^15.7.0, @react-native-firebase/messaging@^15.7.0, @react-native-picker/picker@2.4.10, @react-navigation/bottom-tabs@^6.2.0, @react-navigation/native@^6.0.8, @react-navigation/native-stack@^6.5.0, @regulaforensics/react-native-face-api-beta@^0.0.64, @types/react-native@^0.69.1, axios@^0.26.0, cross-env@^7.0.3, expo@^49.0.10, expo-checkbox@~2.4.0, expo-document-picker@~11.5.4, expo-font@~11.4.0, expo-image-manipulator@~11.3.0, expo-image-picker@~14.3.2, expo-linear-gradient@~12.3.0, expo-linking@~5.0.2, expo-local-authentication@~13.4.1, expo-media-library@~15.4.1, expo-permissions@~14.2.1, expo-splash-screen@~0.20.5, expo-status-bar@~1.6.0, fraction-unicode@^1.0.1, lottie-react-native@^6.3.1, merge-base64@^1.0.4, merge-images@^2.0.0, merge-img@^2.1.3, moment@^2.29.4, moment-timezone@^0.5.43, patch-package@^8.0.0, pod-install@^0.1.37, radio-buttons-react-native-expo@^1.1.0, react@18.2.0, react-circular-progressbar@^2.1.0, react-dom@18.2.0, react-native@0.72.4, react-native-animatable@^1.3.3, react-native-animated-circular-progress@^1.0.6, react-native-animated-pagination-dot@^0.4.0, react-native-animated-spinkit@^1.5.2, react-native-background-downloader@^2.3.4, react-native-check-box@^2.1.7, react-native-circular-progress@^1.3.7, react-native-communications@^2.2.1, react-native-dialog@^9.2.1, react-native-document-picker@^8.1.1, react-native-document-scanner-plugin@^0.1.0, react-native-dynamic-search-bar@^2.0.2, react-native-element-dropdown@^2.0.0, react-native-file-viewer@^2.1.5, react-native-fs@^2.20.0, react-native-gesture-handler@~2.12.0, react-native-get-random-values@~1.9.0, react-native-image-base64@^0.1.4, react-native-indicators@^0.17.0, react-native-ionicons@^4.6.5, react-native-localize@^3.0.2, react-native-paper@^4.11.2, react-native-progress@^5.0.0, react-native-range-slider@^0.1.2, react-native-range-slider-expo@^1.4.2, react-native-reanimated@~3.3.0, react-native-safe-area-context@4.6.3, react-native-screens@~3.22.0, react-native-select-dropdown@^1.8.0, react-native-sliders@^1.1.1, react-native-snap-carousel@^3.9.1, react-native-spinkit@^1.5.1, react-native-svg@13.9.0, react-native-url-polyfill@^1.3.0, react-native-use-plaid@^0.1.0, react-native-web@~0.19.6, react-native-webview@^11.15.0, react-navigation@^4.4.4, react-two-thumb-input-range@^1.0.7, rn-fetch-blob@^0.12.0, rn-nodeify@^10.3.0, rn-range-slider@^2.1.1, stream@^0.0.2 - pyproject.toml: ffprobe@^0.5, Flask@^3.0.2, flask-sock@^0.7.0, gevent@^24.2.1, gevent-websocket@^0.10.1, google-cloud-speech@^2.24.1, googletrans@3.1.0a0, ipykernel@^6.29.2, pydub@^0.25.1, speechrecognition@^3.10.1, transformers@^4.37.2, yt-dlp@^2023.12.30 - server/requirements.txt: aiohttp@==3.9.3, aiohttp-retry@==2.8.3, aiosignal@==1.3.1, annotated-types@==0.6.0, anyio@==4.2.0, async-timeout@==4.0.3, attrs@==23.2.0, blinker@==1.7.0, CacheControl@==0.14.0, cachetools@==5.3.2, certifi@==2024.2.2, cffi@==1.16.0, charset-normalizer@==3.3.2, click@==8.1.7, cryptography@==42.0.3, dataclasses-json@==0.6.4, exceptiongroup@==1.2.0, firebase@==4.0.1, firebase-admin@==6.4.0, Flask@==3.0.2, flask-sock@==0.7.0, Flask-SQLAlchemy@==3.1.1, frozenlist@==1.4.1, gevent@==24.2.1, gevent-websocket@==0.10.1, google-api-core@==2.17.1, google-api-python-client@==2.118.0, google-auth@==2.28.0, google-auth-httplib2@==0.2.0, google-cloud-core@==2.4.1, google-cloud-firestore@==2.14.0, google-cloud-speech@==2.24.1, google-cloud-storage@==2.14.0, google-crc32c@==1.5.0, google-resumable-media@==2.7.0, googleapis-common-protos@==1.62.0, greenlet@==3.0.3, grpcio@==1.60.1, grpcio-status@==1.60.1, h11@==0.14.0, httplib2@==0.22.0, idna@==3.6, itsdangerous@==2.1.2, Jinja2@==3.1.3, jsonpatch@==1.33, jsonpointer@==2.4, langchain-core@==0.1.23, langsmith@==0.0.87, MarkupSafe@==2.1.5, marshmallow@==3.20.2, msgpack@==1.0.7, multidict@==6.0.5, mypy-extensions@==1.0.0, numpy@==1.26.4, packaging@==23.2, proto-plus@==1.23.0, protobuf@==4.25.3, pyasn1@==0.5.1, pyasn1-modules@==0.3.0, pycparser@==2.21, pydantic@==2.6.1, pydantic_core@==2.16.2, PyJWT@==2.8.0, pyparsing@==3.1.1, PyYAML@==6.0.1, regex@==2023.12.25, requests@==2.31.0, rsa@==4.9, simple-websocket@==1.0.0, sniffio@==1.3.0, SQLAlchemy@==2.0.27, tenacity@==8.2.3, twilio@==8.13.0, typing_extensions@==4.9.0, typing-inspect@==0.9.0, uritemplate@==4.1.1, urllib3@==2.2.0, Werkzeug@==3.0.1, wsproto@==1.2.0, yarl@==1.9.4, zope.event@==5.0, zope.interface@==6.2 - shascam/package.json: @babel/core@^7.20.0, @babel/preset-env@^7.20.0, @babel/runtime@^7.20.0, @react-native-firebase/app@^19.0.0, @react-native-firebase/messaging@^19.0.0, @react-native/babel-preset@0.73.21, @react-native/eslint-config@0.73.2, @react-native/metro-config@0.73.5, @react-native/typescript-config@0.73.1, @react-navigation/native@^6.1.14, @react-navigation/native-stack@^6.9.22, @types/react@^18.2.6, @types/react-test-renderer@^18.0.0, babel-jest@^29.6.3, eslint@^8.19.0, jest@^29.6.3, prettier@2.8.8, react@18.2.0, react-native@0.73.5, react-native-safe-area-context@^4.9.0, react-native-screens@^3.29.0, react-test-renderer@18.2.0, typescript@5.0.4 ### Recent commits (newest first) - add sound to push notif - major fixes; streaming works again and websockets - push notifs working - new firebase credentials - Merge branch 'main' of github.com:dariuskia/Shascam - new xcode proj - New commit - New Commit - Merge branch 'main' of github.com:dariuskia/Shascam - add new react native app - Update README.md - update parameter number in readme - update model in readme - update readme - update readme - Merge branch 'main' of github.com:dariuskia/Shascam - update readme - feat: added data - update reqs' - update request and prompt ## Key source files (fetched from GitHub, selected and truncated for size) ### server/env/lib/python3.10/site-packages/twilio-8.13.0.dist-info/AUTHORS.md ```markdown # Authors We'd like to thank the following people who have contributed to the `twilio-python` repository. - Adam Ballai - Alex Brinsmead - Alex Chan - Andrew Benton - Chad Selph - Comrade DOS - Dan Yang - Dennis Pilarinos - Doug Black - Evan Fossier - Fabian Topfstedt - Florian Le Goff - Frank Tobia - Frederik De Bleser - Guillaume BINET - Hunter Blanks - Joël Franusic - Justin Van Koten - Kenneth Reitz - Kevin Burke - Kyle Conroy - Michael Parker - Moses Palmér - Ryan Horn - Sam Kimbrel - Skylar Saveland - Tiberiu Ana - Zachary Voase - aes - dnathe4th - isbo - negeorge - Evan Cooke - tysonholub - Brodan - Kyle Jones ``` ### server/env/lib/python3.10/site-packages/idna-3.6.dist-info/LICENSE.md ```markdown BSD 3-Clause License Copyright (c) 2013-2023, Kim Davies and contributors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ``` ### pyproject.toml ``` [tool.poetry] name = "shascam" version = "0.1.0" description = "" authors = ["DumbassNamedTuna <50528857+Bluefin-Tuna@users.noreply.github.com>"] readme = "README.md" [tool.poetry.dependencies] python = "^3.9" Flask = "^3.0.2" flask-sock = "^0.7.0" google-cloud-speech = "^2.24.1" gevent = "^24.2.1" gevent-websocket = "^0.10.1" ipykernel = "^6.29.2" pydub = "^0.25.1" yt-dlp = "^2023.12.30" ffprobe = "^0.5" googletrans = "3.1.0a0" transformers = "^4.37.2" speechrecognition = "^3.10.1" [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api" ``` ### shascam/Gemfile ``` source 'https://rubygems.org' # You may use http://rbenv.org/ or https://rvm.io/ to install and use this version ruby ">= 2.6.10" # Cocoapods 1.15 introduced a bug which break the build. We will remove the upper # bound in the template on Cocoapods with next React Native release. gem 'cocoapods', '>= 1.13', '< 1.15' gem 'activesupport', '>= 6.1.7.5', '< 7.1.0' ``` ### shascam/package.json ``` { "name": "shascam", "version": "0.0.1", "private": true, "scripts": { "android": "react-native run-android", "ios": "react-native run-ios", "lint": "eslint .", "start": "react-native start", "test": "jest" }, "dependencies": { "@react-native-firebase/app": "^19.0.0", "@react-native-firebase/messaging": "^19.0.0", "@react-navigation/native": "^6.1.14", "@react-navigation/native-stack": "^6.9.22", "react": "18.2.0", "react-native": "0.73.5", "react-native-safe-area-context": "^4.9.0", "react-native-screens": "^3.29.0" }, "devDependencies": { "@babel/core": "^7.20.0", "@babel/preset-env": "^7.20.0", "@babel/runtime": "^7.20.0", "@react-native/babel-preset": "0.73.21", "@react-native/eslint-config": "0.73.2", "@react-native/metro-config": "0.73.5", "@react-native/typescript-config": "0.73.1", "@types/react": "^18.2.6", "@types/react-test-renderer": "^18.0.0", "babel-jest": "^29.6.3", "eslint": "^8.19.0", "jest": "^29.6.3", "prettier": "2.8.8", "react-test-renderer": "18.2.0", "typescript": "5.0.4" }, "engines": { "node": ">=18" } } ``` ### server/requirements.txt ``` aiohttp==3.9.3 aiohttp-retry==2.8.3 aiosignal==1.3.1 annotated-types==0.6.0 anyio==4.2.0 async-timeout==4.0.3 attrs==23.2.0 blinker==1.7.0 CacheControl==0.14.0 cachetools==5.3.2 certifi==2024.2.2 cffi==1.16.0 charset-normalizer==3.3.2 click==8.1.7 cryptography==42.0.3 dataclasses-json==0.6.4 exceptiongroup==1.2.0 firebase==4.0.1 firebase-admin==6.4.0 Flask==3.0.2 flask-sock==0.7.0 Flask-SQLAlchemy==3.1.1 frozenlist==1.4.1 gevent==24.2.1 gevent-websocket==0.10.1 google-api-core==2.17.1 google-api-python-client==2.118.0 google-auth==2.28.0 google-auth-httplib2==0.2.0 google-cloud-core==2.4.1 google-cloud-firestore==2.14.0 google-cloud-speech==2.24.1 google-cloud-storage==2.14.0 google-crc32c==1.5.0 google-resumable-media==2.7.0 googleapis-common-protos==1.62.0 greenlet==3.0.3 grpcio==1.60.1 grpcio-status==1.60.1 h11==0.14.0 httplib2==0.22.0 idna==3.6 itsdangerous==2.1.2 Jinja2==3.1.3 jsonpatch==1.33 jsonpointer==2.4 langchain-core==0.1.23 langsmith==0.0.87 MarkupSafe==2.1.5 marshmallow==3.20.2 msgpack==1.0.7 multidict==6.0.5 mypy-extensions==1.0.0 numpy==1.26.4 packaging==23.2 proto-plus==1.23.0 protobuf==4.25.3 pyasn1==0.5.1 pyasn1-modules==0.3.0 pycparser==2.21 pydantic==2.6.1 pydantic_core==2.16.2 PyJWT==2.8.0 pyparsing==3.1.1 PyYAML==6.0.1 regex==2023.12.25 requests==2.31.0 rsa==4.9 simple-websocket==1.0.0 sniffio==1.3.0 SQLAlchemy==2.0.27 tenacity==8.2.3 twilio==8.13.0 typing-inspect==0.9.0 typing_extensions==4.9.0 uritemplate==4.1.1 urllib3==2.2.0 Werkzeug==3.0.1 wsproto==1.2.0 yarl==1.9.4 zope.event==5.0 zope.interface==6.2 ``` ### legacy/shascam/package.json ``` { "name": "identitywalletnew", "version": "1.0.0", "appVersion": "1.5.9", "scripts": { "start": "react-native start", "android": "react-native run-android", "ios": "cross-env NODE_OPTIONS='--openssl-legacy-provider' && react-native run-ios", "web": "expo start --web", "eject": "expo eject" }, "dependencies": { "@aws-sdk/client-cognito-identity": "^3.131.0", "@aws-sdk/client-s3": "^3.131.0", "@aws-sdk/credential-provider-cognito-identity": "^3.131.0", "@freakycoder/react-native-bounceable": "^0.2.5", "@jesster2k10/react-native-range-slider": "^0.3.1", "@miblanchard/react-native-slider": "^2.1.0", "@mxenabled/react-native-widget-sdk": "^1.1.1", "@ptomasroos/react-native-multi-slider": "^2.2.2", "@react-native-async-storage/async-storage": "1.18.2", "@react-native-community/checkbox": "^0.5.12", "@react-native-community/datetimepicker": "7.2.0", "@react-native-community/geolocation": "^3.0.2", "@react-native-community/slider": "4.4.2", "@react-native-firebase/app": "^15.7.0", "@react-native-firebase/messaging": "^15.7.0", "@react-native-picker/picker": "2.4.10", "@react-navigation/bottom-tabs": "^6.2.0", "@react-navigation/native": "^6.0.8", "@react-navigation/native-stack": "^6.5.0", "@regulaforensics/react-native-face-api-beta": "^0.0.64", "@types/react-native": "^0.69.1", "axios": "^0.26.0", "cross-env": "^7.0.3", "expo": "^49.0.10", "expo-checkbox": "~2.4.0", "expo-document-picker": "~11.5.4", "expo-font": "~11.4.0", "expo-image-manipulator": "~11.3.0", "expo-image-picker": "~14.3.2", "expo-linear-gradient": "~12.3.0", "expo-linking": "~5.0.2", "expo-local-authentication": "~13.4.1", "expo-media-library": "~15.4.1", "expo-permissions": "~14.2.1", "expo-splash-screen": "~0.20.5", "expo-status-bar": "~1.6.0", "fraction-unicode": "^1.0.1", "lottie-react-native": "^6.3.1", "merge-base64": "^1.0.4", "merge-images": "^2.0.0", "merge-img": "^2.1.3", "moment": "^2.29.4", "moment-timezone": "^0.5.43", "patch-package": "^8.0.0", "pod-install": "^0.1.37", "radio-buttons-react-native-expo": "^1.1.0", "react": "18.2.0", "react-circular-progressbar": "^2.1.0", "react-dom": "18.2.0", "react-native": "0.72.4", "react-native-animatable": "^1.3.3", "react-native-animated-circular-progress": "^1.0.6", "react-native-animated-pagination-dot": "^0.4.0", "react-native-animated-spinkit": "^1.5.2", "react-native-background-downloader": "^2.3.4", "react-native-check-box": "^2.1.7", "react-native-circular-progress": "^1.3.7", "react-native-communications": "^2.2.1", "react-native-dialog": "^9.2.1", "react-native-document-picker": "^8.1.1", "react-native-document-scanner-plugin": "^0.1.0", "react-native-dynamic-search-bar": "^2.0.2", "react-native-element-dropdown": "^2.0.0", "react-native-file-viewer": "^2.1.5", "react-native-fs": "^2.20.0", "react-native-gesture-handler": "~2.12.0", "react-native-get-random-values": "~1.9.0", "react-native-image-base64": "^0.1.4", "react-native-indicators": "^0.17.0", "react-native-ionicons": "^4.6.5", "react-native-localize": "^3.0.2", "react-native-paper": "^4.11.2", "react-native-progress": "^5.0.0", "react-native-range-slider": "^0.1.2", "react-native-range-slider-expo": "^1.4.2", "react-native-reanimated": "~3.3.0", "react-native-safe-area-context": "4.6.3", "react-native-screens": "~3.22.0", "react-native-select-dropdown": "^1.8.0", "react-native-sliders": "^1.1.1", "react-native-snap-carousel": "^3.9.1", "react-native-spinkit": "^1.5.1", "react-native-svg": "13.9.0", "react-native-url-polyfill": "^1.3.0", "react-native-use-plaid": "^0.1.0", "react-native-web": "~0.19.6", "react-native-webview": "^11.15.0", "react-navigation": "^4.4.4", "react-two-thumb-input-range": "^1.0.7", "rn-fetch-blob": "^0.12.0", "rn-nodeify": "^10.3.0", "rn-range-slider": "^2.1.1", "stream": "^0.0.2" }, "devDependencies": { "@babel/core": "^7.20.0" }, "private": true } ``` ### server/env/lib/python3.10/site-packages/langsmith/cli/docker-compose.yaml ```yaml version: "4" services: langchain-playground: image: langchain/${_LANGSMITH_IMAGE_PREFIX-}langchainplus-playground:${_LANGSMITH_IMAGE_VERSION:-latest} ports: - 3001:3001 langchain-frontend: image: langchain/${_LANGSMITH_IMAGE_PREFIX-}langchainplus-frontend-dynamic:${_LANGSMITH_IMAGE_VERSION:-latest} ports: - 80:80 depends_on: - langchain-backend - langchain-playground - langchain-hub langchain-backend: image: langchain/${_LANGSMITH_IMAGE_PREFIX-}langchainplus-backend:${_LANGSMITH_IMAGE_VERSION:-latest} environment: - PORT=1984 - LANGCHAIN_ENV=local_docker - LOG_LEVEL=warning - OPENAI_API_KEY=${OPENAI_API_KEY} - LANGSMITH_LICENSE_KEY=${LANGSMITH_LICENSE_KEY} ports: - 1984:1984 depends_on: langchain-db: condition: service_healthy langchain-redis: condition: service_healthy clickhouse-setup: condition: service_completed_successfully restart: always langchain-queue: image: langchain/${_LANGSMITH_IMAGE_PREFIX-}langchainplus-backend:${_LANGSMITH_IMAGE_VERSION:-latest} environment: - LANGCHAIN_ENV=local_docker - LOG_LEVEL=warning - LANGSMITH_LICENSE_KEY=${LANGSMITH_LICENSE_KEY} entrypoint: "rq worker --with-scheduler -u redis://langchain-redis:6379 --serializer lc_database.queue.serializer.ORJSONSerializer --worker-class lc_database.queue.worker.Worker --connection-class lc_database.queue.connection.RedisRetry --job-class lc_database.queue.job.AsyncJob" depends_on: langchain-db: condition: service_healthy langchain-redis: condition: service_healthy clickhouse-setup: condition: service_completed_successfully restart: always langchain-hub: image: langchain/${_LANGSMITH_IMAGE_PREFIX-}langchainhub-backend:${_LANGSMITH_IMAGE_VERSION:-latest} environment: - PORT=1985 - LANGCHAIN_ENV=local_docker - LOG_LEVEL=warning - LANGSMITH_LICENSE_KEY=${LANGSMITH_LICENSE_KEY} ports: - 1985:1985 depends_on: - langchain-db langchain-db: image: postgres:14.7 command: [ "postgres", "-c", "log_min_messages=WARNING", "-c", "client_min_messages=WARNING", ] environment: - POSTGRES_PASSWORD=postgres - POSTGRES_USER=postgres - POSTGRES_DB=postgres volumes: - langchain-db-data:/var/lib/postgresql/data ports: - 5433:5432 healthcheck: test: ["CMD", "pg_isready", "-U", "postgres"] interval: 2s timeout: 2s retries: 30 langchain-redis: image: redis:7 ports: - 63791:6379 volumes: - langchain-redis-data:/data healthcheck: test: ["CMD", "redis-cli", "ping"] interval: 2s timeout: 2s retries: 30 langchain-clickhouse: image: clickhouse/clickhouse-server:23.9 user: "101:101" restart: always environment: - CLICKHOUSE_DB=default - CLICKHOUSE_USER=default - CLICKHOUSE_PASSWORD=password volumes: - langchain-clickhouse-data:/var/lib/clickhouse - ./users.xml:/etc/clickhouse-server/users.d/users.xml ports: - 8124:8123 - 9001:9000 healthcheck: test: ["CMD", "clickhouse-client", "--query", "SELECT 1"] interval: 2s timeout: 2s retries: 30 clickhouse-setup: image: langchain/${_LANGSMITH_IMAGE_PREFIX-}langchainplus-backend:${_LANGSMITH_IMAGE_VERSION:-latest} depends_on: langchain-clickhouse: condition: service_healthy restart: "no" entrypoint: [ "bash", "-c", "migrate -source file://clickhouse/migrations -database 'clickhouse://langchain-clickhouse:9000?username=default&password=password&database=default&x-multi-statement=true&x-migrations-table-engine=MergeTree' up", ] volumes: langchain-db-data: langchain-redis-data: langchain-clickhouse-data: ``` ### shascam/index.js ```javascript /** * @format */ import {AppRegistry} from 'react-native'; import App from './App'; import {name as appName} from './app.json'; AppRegistry.registerComponent(appName, () => App); ``` ### scraper/main.py ```python # import scrapetube # from youtube_transcript_api import YouTubeTranscriptApi # videos = scrapetube.get_channel(channel_url="https://www.youtube.com/@ToaoDotNet") # videos = [video for video in videos] from langchain.text_splitter import CharacterTextSplitter from langchain_together import Together from textwrap import dedent import os llm = Together( model="NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO", temperature=0.1, max_tokens=8192, top_k=1, together_api_key="8d95b7be50fb95563ac06056642c0c160a9c46424755c2ae9e5813f597dcc0d5", ) PROMPT_TEMPLATE = dedent("""\ ```system You are an AI assistant tasked with classifying cell phone conversations as scam calls. I will provide you the general structure for scam calls, examples of scam call topics and patterns, and finally the transcript of the call in question. I want you to analyze the transcript and decide whether the transcript describes a scam call or a normal call. Respond only with 1 of the 4 following categories: "Very Likely Scam", "Likely Scam", "Unlikely Scam", "Very Unlikely Scam." On a new line add a 2-3 sentence justification for each decision. Give the following extraneous information on a new line depending on the decided category: Very Likely: Provide an action for the user to do. For example, "Hang up immediately.", or "Do NOT give any personal information." Likely: Provide clarifying questions for the user to ask. For example, "Why do you need this information?" Unlikely: Do NOT provide any actions or questions for the user to do. Very Unlikely: Do NOT provide any actions or questions for the user to do. ``` ```structure Greeting (e.g., 'Hello') Self identification (Name of the call agent) Company identification (Name of the business) Warm up talk (e.g., 'How are you today?') Statement of the reason of the call Callee identity check (callee's name and attribute) ``` ```examples Illegitimate/fake company names ('Windows service center' or 'US Grants and Treasury Department') Giving 2 options (no option to decline): ex. Appointment for home improvement technician: spammer asks if the customers prefers 2:30pm or 4pm. Make promises throughout the call (ex. free estimate with no obligation, easy cancellation, a lifetime warranty) Introducing a threatening scenario such as “your computer is getting infected” or “your air duct system is badly contaminated” Convincing the customer to make a payment (ex. by giving credit card information or home address for the bill) ``` ```transcript {curr_text} ``` ```assistant\ """) import pickle from tqdm import tqdm text_splitter = CharacterTextSplitter.from_tiktoken_encoder(chunk_size=4096, chunk_overlap=0) transcript_fns = os.listdir("./transcripts")[0:150] outputs = { "prompts": [], "outputs": [] } for transcript_fn in tqdm(transcript_fns): transcript_txt = open(f"./transcripts/{transcript_fn}", "r").read() if not transcript_txt: continue text_splitter = CharacterTextSplitter.from_tiktoken_encoder(chunk_size=4096, chunk_overlap=0) transcript_txt = text_splitter.split_text(transcript_txt) prompt = PROMPT_TEMPLATE.format(curr_text=transcript_txt) output = llm.invoke(prompt, stop=["```"]) outputs["prompts"].append(prompt) outputs["outputs"].append(output) pickle.dump(outputs, open("outputs.pkl", "wb"), protocol=pickle.HIGHEST_PROTOCOL) pickle.dump(outputs, open("outputs.pkl", "wb"), protocol=pickle.HIGHEST_PROTOCOL) ``` [3879 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]