Project Info

2nd Chance

Inspiration

We've all wished that we would've talked more to people in person when we had the chance - whether you need them to answer an urgent question, your schedules are too busy for frequent phone calls, or your loved one has passed on, 2nd Chance was built to let you talk to a loved one in real time. Created with the next generation of senior citizens in mind, we wanted to take advantage of the abundance of video and text records people will have of each other while implementing senior-friendly UI. We were excited to use new APIs and learn vector embedding techniques to accomplish our goal of simulating an individual based on their records.

What it does

2nd Chance first requires a user to submit audio and/or text records of their loved one to get a sense of how they talk. Then, taking either a written or spoken input from the user, the software processes the input and compares it to the records on hand with vector embedding. If the user's input is spoken, it uses OpenAI's Whisper API to transcribe the message into written text. Using ChatGPT to synthesize the existing responses in the records, the software predicts a response to the user's entry. If the user's entry was spoken, we reproduce their loved one's voice using ElevenLab's API so they can speak their response back to the user.

How we built it

The frontend and backend of our web application was built with Next.js and Convex, respectively. We specifically took advantage of Convex’s Vector Search, Server Functions, and Database features. Our AI simulation was powered by OpenAI’s Chat-GPT model and Vector Embeddings (for Vector Search) were provided by OpenAI’s embedding model. We use Gradio for our speech-to-speech interface, Whisper for audio transcription, and ElevenLabs for voice cloning. Finally, we used Tailwind CSS and Flowbite for overall styling.

Challenges we ran into

Our decision to use Next.js and Convex caused some difficulty when we attempted to integrate our speech-to-speech software. We were required to implement that software in separate Python files, but then we lost access to our Vector Search software (through Convex). In the end, both speech-to-speech and text-to-text software were integrated onto the web application with Vector Search.

Accomplishments we're proud of

We're proud of being able to produce an audio response in the voice of the user's loved one by using ElevenLab's API. It was exciting to get to implement a variety of APIs we hadn't used before to improve our functionality. Another accomplishment was getting the vector embedding to work to help generate responses specific to the user. We’re primarily proud of getting to help people from multiple backgrounds in their day to day lives.

What we learned

We learned about implementing APIs using their documentations, as well as seeing how many APIs are out there and the value of open source code. We also learned that we should plan out all the technologies we want to use so that we can ensure they can be integrated properly. Lastly, we saw how much we could code we could learn and put together in the span of 36 hours.

What's next

We want to continue to improve the integration of the AI speech-to-speech technology with the front-end design of our website to increase usability and design. We also want to ensure that our user’s text logs and/or audio recordings that they submit as reference points are well encrypted to protect their privacy. It would be nice to also increase functionality for non-English speakers to make our technology more accessible for diverse populations.

This is a Next.js project bootstrapped with create-next-app.

Getting Started

First, run the development server:

npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev

Open http://localhost:3000 with your browser to see the result.

You can start editing the page by modifying app/page.tsx. The page auto-updates as you edit the file.

This project uses next/font to automatically optimize and load Inter, a custom Google Font.

Learn More

To learn more about Next.js, take a look at the following resources:

Next.js Documentation - learn about Next.js features and API.
Learn Next.js - an interactive Next.js tutorial.

You can check out the Next.js GitHub repository - your feedback and contributions are welcome!

Deploy on Vercel

The easiest way to deploy your Next.js app is to use the Vercel Platform from the creators of Next.js.

Check out our Next.js deployment documentation for more details.

Analysis

Compare with all teams

View

Metric

Alastair Deng
26
mrTSB
8

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

CSSIn code
JavaScriptIn code
Next.jsIn code
OpenAIIn code
PythonIn code
ReactIn code
Tailwind CSSIn code
TypeScriptIn code

8 of 8 appear in the indexed code.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

86 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.