# Project export: Memento
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: Cal Hacks 11.0
- Tagline: Memento helps elderly people connect with their families by reflecting on and sharing cherished memories, sparking meaningful conversations that strengthen bonds across generations.
- Devpost: https://devpost.com/software/memento-1p0jel
- GitHub: https://github.com/AlexLuu1/Memento
- Result: winner (Ripple: Best App by Ripple; Reflex: Best app built using reflex.dev)
- Team: 3 GitHub contributor(s) — Alex Luu (26 commits), Vrushank Prakash (11 commits), GilbertHarijanto (5 commits)
## Devpost submission (written by the team)
### Inspiration
Our inspiration came from the importance of connecting with family and cherishing proud personal stories. We recognized that many elderly people in nursing homes feel isolated from their families, despite the wealth of memories they carry. These memories hold so much value about family history, wisdom, and identity. By creating a platform that enables them to reflect on and share these moments, we aimed to bridge generational gaps and strengthen family bonds. Through storytelling, we want to foster a tight family bond, ensuring that cherished memories are passed down and that the elderly feel heard, valued, and connected. We wanted to emphasize the story aspect of these memories. When people want to share their memories with their family, especially virtually, they aren't able to fully relive or cherish that memory- a simple text message can't fully do justice to a fond memory. Thus, we wanted to bring life into these memories that shared within families online and especially provide elderly people who might not meet their families often to have an immersive experience with their family's memories.
### What it does
Memento allows families to document fond memories that they have, and share them to the user. Families can upload memories that contain a date, description, and image. We target this product to the elderly in nursing homes who are usually alone and can benefit from having someone like family to talk to. The elderly user can then speak to the application and can have a conversation about the details of any memory. The application will also display the most relevant image to the conversation to help improve the experience. This enables the elderly user to feel like they are talking to a family member or someone they know well. It allows them to stay connected with their loved ones without the continuous presence of them.
### How we built it
We designed Memento to be simple and accessible for both elderly users and their families. For this reason, we used Reflex to implement an elegant UI, and implemented a Chroma database to store the memories and their embeddings for search. We also integrated Whisper, a speech-to-text model through Groq’s fast inference API to decode what the elderly person is saying. Using this input, we query our database, and feed this information through Gemini, an LLM developed by Google, to give a coherent response that incorporates information from the families’ inputs. Finally, we used Deepgram’s text-to-speech model to convert the LLM’s outputs back to an audio format that we could speak back to the elderly user.
### Challenges we ran into
Integration: It was difficult to integrate all of the sponsor’s softwares into the final application; we had to pore over documentation while becoming familiar with each API, which led to many hours of debugging. Non-determinism: Our models were non-deterministic; errors caused by specific outputs from the LLM were hard to replicate. Due to the background noise, we also could not efficiently test our speech-to-text model’s accuracy. Inference speed: Throughout this application, we make many API calls to large models, such as Whisper, Gemini, and the Aura TTS model. Because of this, we had to find clever optimizations to speed up the inference time to quickly speak back to the elderly user, especially since the WiFi was unusable most of the time.
### Accomplishments we're proud of
Design and User Experience: We are proud of our design since it encompasses the mood we were aiming for – a warm, welcoming environment, focusing on the good things that happen in life. Large Language Model and Vector Search: We are especially proud of how the LLM turned out and how well the RAG model worked. We spent lots of time prompting the different components to create the warm, empathetic, and welcoming environment the LLM provides. TTS and STT: Although we struggled a bit with this part, we are really proud about how it turned out. We feel we did a great job encompassing the ideals of the product by allowing users to reflect on past memories and connect closer with family.
### What we learned
Working with STT and TTS models: many members of our group had never worked with speech-to-text or text-to-speech models, so this was a learning experience for all of us. We learned about the impressive accuracy that the state-of-the-art models are able to achieve but also encountered some of the drawbacks of these models, since many of them don’t work as well with moderate levels of background noise. How to make a great UI:
### What's next
Because of time constraints, there were many features and improvements we wanted to implement but could not. Continuous LLM Conversation: We wanted to be able to talk to the LLM continuously without having to press a microphone button. Due to time constraints, we were not able to implement this feature User Personalization and Customization: We aimed to personalize the website to users by adding custom themes, colors, and fonts, but we ran out of time to do so.
## README (from the GitHub repository)
# Memento
### Cal Hacks 11.0 (2024) Winner: Best app built using [reflex.dev](https://reflex.dev/)
### Devpost: https://devpost.com/software/memento-1p0jel
## Tech Stack
- Python
- Reflex
- Groq
- Chroma
- Google Gemini
- Deepgram
## Detected evidence (automated analysis)
Indexed codebase: 7 recognized source files, 45 KB.
- Python (language) — detected in the code
- Google Gemini (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (11 of 11)
```
.gitignore
db_path/.gitkeep
Memento/__init__.py
Memento/components.py
Memento/family.py
Memento/Memento.py
Memento/user.py
README.md
requirements.txt
rxconfig.py
uploaded_files/.gitkeep
```
### Dependencies
- requirements.txt: chromadb@==0.5.15, deepgram-sdk@==3.7.4, google.generativeai@==0.8.3, groq@==0.11.0, pillow@==11.0.0, reflex@==0.6.3, reflex_audio_capture@==0.0.4
### Recent commits (newest first)
- update README.md
- added README.md, added versioning to requirements.txt
- updated design and text
- folders
- cleanup and updates
- fixed talking issue
- STYLESSSSSSS
- testtt
- family
- merge
- finished tts
- updated llm prompt
- fixed issues
- pull components out
- prompt
- frontpage dedign
- Merge branch 'main' of github.com-AlexLuu1:AlexLuu1/Memento
- image size
- FE
- cleanup
## Key source files (fetched from GitHub, selected and truncated for size)
### requirements.txt
```
reflex==0.6.3
reflex_audio_capture==0.0.4
pillow==11.0.0
chromadb==0.5.15
google.generativeai==0.8.3
deepgram-sdk==3.7.4
groq==0.11.0
```
### rxconfig.py
```python
import reflex as rx
config = rx.Config(
app_name="Memento",
)
```
### Memento/components.py
```python
import reflex as rx
def create_navigation_link(link_text, path):
"""Create a navigation link with hover effects and specific padding."""
return rx.el.a(
link_text,
href=path,
font_size="1.25rem",
_hover={"color": "#ffffff"},
padding_left="1.25rem",
padding_right="1.25rem",
padding_top="0.5rem",
padding_bottom="0.5rem",
color="#ffffff",
)
def create_header():
"""Create the main header with logo and navigation links."""
return rx.box( # Use box to ensure full width
rx.flex(
rx.flex(
rx.image(
src="/logo.png",
alt="logo",
height="4.5rem",
margin_right="0.75rem",
width="4.5rem",
),
rx.text.span(
"Memento",
font_weight="600",
color="#ffffff",
font_size="2rem",
line_height="1.75rem",
),
display="flex",
align_items="center",
),
rx.flex(
create_navigation_link(link_text="Home", path="/"),
create_navigation_link(link_text="Talk To Me!", path="/user"),
create_navigation_link(link_text="Family", path="/family"),
create_navigation_link(link_text="About", path="/"),
display="flex",
align_items="center",
),
width="100%",
justify_content="space-between",
align_items="center",
),
width="100vw", # Full viewport width
padding_left="7rem",
padding_right="7rem",
padding_top="0.75rem",
padding_bottom="0.75rem",
background_color="#74452f",
)
def create_footer():
"""Create the footer with simple content."""
return rx.box(
rx.text("© 2024 Memento. All rights reserved."),
text_align="center",
font_size="0.875rem",
line_height="1.25rem",
padding="1rem",
border_top="1px solid #ccc",
background_color="#ffffff",
)
```
### Memento/family.py
```python
import reflex as rx
from rxconfig import config
from datetime import datetime
import os
from PIL import Image
import io
import requests
import chromadb
from chromadb import Documents, EmbeddingFunction, Embeddings
import google.generativeai as genai
import uuid
from typing import List, TypedDict
from .components import *
# Custom Heading Component
def create_custom_heading(
heading_type: str,
font_size: str,
margin_bottom: str,
heading_text: str
) -> rx.Component:
return rx.heading(
heading_text,
font_weight="600",
margin_bottom=margin_bottom,
font_size=font_size,
line_height="1.75rem",
as_=heading_type,
)
# Themed Page Wrapper
def create_themed_page(content: rx.Component) -> rx.Component:
"""Wrap the given content with header and footer, applying the theme."""
return rx.box(
create_header(),
content,
create_footer(),
background_color="#F3F4F6",
font_family=(
'system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, '
'"Helvetica Neue", Arial, "Noto Sans", sans-serif'
),
)
# Configure Google Generative AI
# Replace with your actual API key securely
# **Important**: Use environment variables in production
genai.configure(api_key=os.environ.get('GOOGLE_API_KEY'))
text_to_img_model = genai.GenerativeModel("gemini-1.5-flash")
# Custom Embedding Function for ChromaDB
class GeminiEmbeddingFunction(EmbeddingFunction):
def __call__(self, input: Documents) -> Embeddings:
model = 'models/embedding-001'
title = "Custom query"
return genai.embed_content(
model=model,
content=input,
task_type="retrieval_document",
title=title
)["embedding"]
# State Management for Family Memories
class FamilyState(rx.State):
data: List[dict[str, str]] = []
def get_data(self) -> None:
self.data = []
collection = chromadb.HttpClient(
host='localhost', port=8001
).get_or_create_collection(
name="vectordb",
embedding_function=GeminiEmbeddingFunction()
)
results = collection.get()
documents = results.get("documents", [])
metadatas = results.get("metadatas", [])
for doc, metadata in zip(documents, metadatas):
try:
date_str, description, image_summary = doc.split('|', 2)
date_obj = datetime.strptime(date_str, "%Y-%m-%d")
date_formatted = date_obj.strftime("%B %d, %Y")
image_filename = f"{str(metadata['filename'])}.jpg"
self.data.append({
"date": date_formatted,
"description": description,
"image_filename": image_filename,
"og_date": date_str,
})
except ValueError as e:
print(f"Error parsing document: {e}")
continue
# Sort the data by date in descending order (newest first)
self.data.sort(key=lambda x: x["og_date"], reverse=True)
print(self.data)
# Timeline Event Component
# Timeline Event Component
def TimelineEvent(event_data: dict[str, str]) -> rx.Component:
date_formatted = event_data["date"]
filename = event_data["image_filename"]
# Generate the image URL and set a placeholder in case of error
image_url = rx.get_upload_url(filename)
placeholder_url = "/static/placeholder.jpg"
return rx.box(
rx.hstack(
# Left side: Date (hidden on mobile)
rx.box(
rx.text(
date_formatted,
font_weight="600",
font_size="1rem",
color="#333",
text_align="right",
),
width="175px",
padding_right="1rem",
display={
"base": "none",
"md": "block"
},
),
# Middle: Connector Line and Marker
rx.box(
rx.box(
width="10px",
height="10px",
border_radius="50%",
background_color="#74452f",
margin="0 auto",
z_index="1",
),
rx.box(
width="2px",
flex="1",
background_color="#ccc",
margin="0 auto",
),
display="flex",
flex_direction="column",
align_items="center",
position="relative",
width="20px", # Set a fixed width for the marker column
),
# Right side: Content
rx.box(
rx.image(
src=image_url,
fallback=placeholder_url,
width="100%",
height="auto",
border_radius="1rem",
margin_bottom="0.5rem",
object_fit="cover",
),
rx.text(
event_data["description"],
padding="0.5rem 1rem",
color="#555",
),
flex="1", # Allow content to take up remaining space
),
align_items="flex-start",
width="100%",
),
flex_direction={
"base": "column",
"md": "row"
},
margin_bottom="2rem",
position="relative",
width="100%",
)
# Main Family Memories Page with Vertical Timeline
@rx.page(on_load=FamilyState.get_data, route="/family")
def family_index():
content = rx.box(
rx.box(
rx.heading(
"Family Memories",
font_size="2rem",
[truncated — 8188 more characters]
```
### Memento/user.py
```python
from urllib.request import urlopen
import reflex as rx
from reflex_audio_capture import AudioRecorderPolyfill, get_codec, strip_codec_part
import os
from groq import Groq
from deepgram import (
DeepgramClient,
SpeakOptions,
)
import chromadb
from chromadb import Documents, EmbeddingFunction, Embeddings
import google.generativeai as genai
import re
from .components import *
client = Groq(api_key=os.environ['GROQ_API_KEY'])
deepgram = DeepgramClient(os.environ['DEEPGRAM_API_KEY'])
genai.configure(api_key=os.environ['GOOGLE_API_KEY'])
REF = "myaudio"
# Add the create_themed_page function
def create_custom_heading(heading_type, font_size, margin_bottom, heading_text):
return rx.heading(
heading_text,
font_weight="600",
margin_bottom=margin_bottom,
font_size=font_size,
line_height="1.75rem",
as_=heading_type,
)
def create_themed_page(content):
"""Wrap the given content with header and footer, applying the theme."""
return rx.box(
create_header(),
content,
create_footer(),
background_color="#F3F4F6", # Consistent background
font_family='system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, "Noto Sans", sans-serif',
width="100%"
)
class GeminiEmbeddingFunction(EmbeddingFunction):
def __call__(self, input: Documents) -> Embeddings:
model = 'models/embedding-001'
title = "Custom query"
return genai.embed_content(model=model,
content=input,
task_type="retrieval_document",
title=title)["embedding"]
class UserState(rx.State):
"""The app state."""
has_error: bool = False
processing: bool = False
transcript: list[str] = []
device_id: str = ""
use_mp3: bool = True
tts_output_file: str = ""
is_talking: bool = False
img_to_display: str = ""
text_output: str = ""
filenum: int = 0
history: list[tuple[str, str]] = []
def get_data(self):
self.transcript = []
self.text_output = ""
self.img_to_display = ""
self.history = []
async def on_data_available(self, chunk: str):
mime_type, _, codec = get_codec(chunk).partition(";")
audio_type = mime_type.partition("/")[2]
if audio_type == "mpeg":
audio_type = "mp3"
with urlopen(strip_codec_part(chunk)) as audio_data:
try:
self.processing = True
yield
transcription = client.audio.transcriptions.create(
# Required audio file
file=(("temp." + audio_type, audio_data.read(), mime_type)),
model="whisper-large-v3-turbo", # Required model to use for transcription
prompt="Specify context or spelling", # Optional
response_format="json", # Optional
language="en", # Optional
temperature=0.0 # Optional
)
except Exception as e:
self.has_error = True
yield capture.stop()
raise
finally:
self.processing = False
self.transcript.append(transcription.text)
# Get Documents
collection = chromadb.HttpClient(host='localhost', port=8001).get_or_create_collection(
name="vectordb", embedding_function=GeminiEmbeddingFunction())
results = collection.query(
query_texts=[" ".join(self.transcript)],
n_results=10,
)
print(results)
# Prompt LLM
documents = results["documents"][0]
metadatas = results["metadatas"][0]
output = ""
# Iterate through the
for i, (doc, metadata) in enumerate(zip(documents, metadatas)):
# Split the document into date and description
date, description, image_summary = doc.split('|', 2)
filename = metadata["filename"]
# Add the formatted data to the result string
output += f"""
Date: {date}
Description: {description}
ImageName: {filename}
Image Summary: {image_summary}
"""
output = output.strip()
system = f"""
You are Memento, a memory storage AI designed for elderly individuals in nursing homes.
Your role is to help elderly users recall cherished memories by using voice recognition technology.
When an elderly user speaks about a memory, you retrieve and display relevant images that are stored in the system.
When an elderly user speaks about a memory, you don't just passively listen, you actively engage with their narrative.
You retrieve and display relevant images, creating a rich, multi-sensory experience that brings their memories to life.
Your responses should be carefully crafted based on the specific memory data provided, ensuring a personalized and accurate reflection of each individual's unique life experiences.
You must always use the `fileNameGrabber` tool call.
**Your responses should be based on the data provided about the memories**
# Persona
- Be Empathetic and Warm
- Have a Clear Communication
- Be Nostalgic and Personalized
- Be Patient and Non-Rushed
- Keep Your Tone Gentle, Slow-Paced, Comforting
# Interaction Guidelines
- Always greet the user with a warm welcome.
- When a memory is mentioned, respond with excitement and genuine interest.
- Do NOT mention the Image Summary directly, understand the Image Summary paired with the Description to understand the context.
- Your response should con
[truncated — 8364 more characters]
```
### Memento/Memento.py
```python
"""Welcome to Reflex! This file outlines the steps to create a basic app."""
import reflex as rx
from rxconfig import config
from .user import user_index
from .family import family_index, add_new_memory
from .components import *
class State(rx.State):
"""The app state."""
...
def create_custom_heading(
heading_type, font_size, margin_bottom, heading_text
):
"""Create a custom heading with specified font weight, margin, font size, and line height."""
return rx.heading(
heading_text,
font_weight="600",
margin_bottom=margin_bottom,
font_size=font_size,
line_height="1.75rem",
as_=heading_type,
)
def create_centered_heading(heading_text, color):
"""Create a centered heading with specific styling for section titles."""
return rx.heading(
heading_text,
font_weight="600",
margin_bottom="2rem",
font_size="1.875rem",
line_height="2.25rem",
text_align="center",
as_="h2",
color=color
)
def create_icon(alt_text, icon_tag):
"""Create an icon with specified dimensions and margins."""
return rx.icon(
alt=alt_text,
tag=icon_tag,
height="4rem",
margin_bottom="1rem",
margin_left="auto",
margin_right="auto",
width="4rem",
)
def create_gray_text(text_content):
"""Create a text element with a specific gray color."""
return rx.text(text_content, color="#4b5563")
def create_feature_box(
icon_alt, icon_tag, title, description
):
"""Create a feature box with an icon, title, and description."""
return rx.box(
create_icon(alt_text=icon_alt, icon_tag=icon_tag),
create_custom_heading(
heading_type="h3",
font_size="1.25rem",
margin_bottom="0.5rem",
heading_text=title,
),
create_gray_text(text_content=description),
text_align="center",
)
def create_description_text(description):
"""Create a description text with specific margin and color."""
return rx.text(
description, margin_bottom="1rem", color="#4B5563"
)
def create_learn_more_link():
"""Create a 'Learn More' link with hover effects."""
return rx.el.a(
"Learn More",
href="#",
font_weight="600",
_hover={"color": "#74452f"},
color="#74452f",
)
def create_feature_card(title, description):
"""Create a feature card with a title, description, and 'Learn More' link."""
return rx.box(
create_custom_heading(
heading_type="h3",
font_size="1.25rem",
margin_bottom="1rem",
heading_text=title,
),
create_description_text(description=description),
create_learn_more_link(),
background_color="#f9fafb",
padding="1.5rem",
border_radius="0.5rem",
box_shadow="0 4px 6px -1px rgba(0, 0, 0, 0.1), 0 2px 4px -1px rgba(0, 0, 0, 0.06)",
)
def create_social_icon(alt_text, icon_tag):
"""Create a social media icon with specified dimensions."""
return rx.icon(
alt=alt_text,
tag=icon_tag,
height="1.5rem",
width="1.5rem",
)
def create_social_link(icon_alt, icon_tag):
"""Create a social media link with an icon and hover effects."""
return rx.el.a(
create_social_icon(
alt_text=icon_alt, icon_tag=icon_tag
),
href="#",
_hover={"color": "#D1D5DB"},
)
# def create_header():
# """Create the main header with logo and navigation links."""
# return rx.flex(
# rx.flex(
# rx.image(
# src="/logo.png",
# alt="logo",
# height="4.5rem",
# margin_right="0.75rem",
# width="4.5rem",
# ),
# rx.text.span(
# "Memento",
# font_weight="600",
# color="#ffffff",
# font_size="2rem",
# line_height="1.75rem",
# ),
# display="flex",
# align_items="center",
# ),
# rx.flex(
# create_navigation_link(link_text="Home", path="/"),
# create_navigation_link(link_text="User", path="/user"),
# create_navigation_link(link_text="Family", path="/family"),
# create_navigation_link(link_text="About", path="/"),
# display="flex",
# align_items="center",
# ),
# width="100%",
# style=rx.breakpoints(
# {
# "640px": {"max-width": "640px"},
# "768px": {"max-width": "768px"},
# "1024px": {"max-width": "1024px"},
# "1280px": {"max-width": "1280px"},
# "1536px": {"max-width": "1536px"},
# }
# ),
# display="flex",
# align_items="center",
# justify_content="space-between",
# margin_left="auto",
# margin_right="auto",
# padding_left="1.5rem",
# padding_right="1.5rem",
# padding_top="0.75rem",
# padding_bottom="0.75rem",
# )
def create_get_started_button():
"""Create a 'Get Started' button with hover effects."""
return rx.el.a(
"Get Started",
href="/family/new-memory",
background_color="#74452f",
transition_duration="300ms",
font_weight="600",
_hover={"background-color": "#DBEAFE"},
padding_left="2rem",
padding_right="2rem",
padding_top="0.75rem",
padding_bottom="0.75rem",
border_radius="9999px",
color="#ffffff",
transition_property="background-color, border-color, color, fill, stroke, opacity, box-shadow, transform",
transition_timing_function="cubic-bezier(0.4, 0, 0.2, 1)",
)
def create_hero_text_section():
"""Create the hero section text content w
[truncated — 8588 more characters]
```