StoryLiving - visionOS App (TreeHacks 2024)

Do you wish you could feel more connected to your loved ones? What if you could share and relive your memories with them? Now, with our app, you can view a spatial video or photo with your loved ones in the exact 3D environment in which the memory happened. You aren't just story-telling. You are now story-living.

• Devpost: https://devpost.com/software/storyliving
• YouTube: https://youtu.be/jHisBQFxLAs?si=hgV63IyufbpRisNO

Team Contributions:

• Steven Le (Stanford) - Project Manager
• Jiahui Chen (Stanford) - Frontend visionOS Dev
• Rylan Pozniak (USC) - Backend visionOS Dev
• Shawn Park (CMU) - Video Editor + Narrative

Inspiration

Social isolation among the elderly is a global problem. Around the world, 1 in 4 individuals above 60 experience social isolation. And 40% of Japan’s population aged above 60 experiences social isolation. The need for richer, more meaningful connections is critical.

The inception of StoryLiving is deeply personal for all of us. After one of our team members lost their grandfather, they were struck by the realization of their missed connections and the stories left untold.

We are all XR developers and enthusiasts, who have interned at several XR companies like Apple, Meta, Snapchat and others. Devices and apps with large, legible text, interactive screens, and voice commands are more accessible to older users. By using the most intuitive headset currently on the market, the Apple Vision Pro, our product enables a more social, narrative-driven way to relive memories that doesn’t require any tutorials.

The advent of the Apple Vision Pro opened a doorway to an innovative solution—spatial memory. This technology allows users to not only reminisce about past stories but to truly relive and share those moments with loved ones in an immersive and social way. We set out to capitalize on this untapped vision.

What it does

StoryLiving is an application that transcends traditional storytelling by leveraging spatial video and photogrammetry to immerse users in 3D environments where memories were made. It enables grandparents to connect with their families in a shared virtual space, experiencing memories together as if they were physically present. Users can select and explore memories together using the Apple ecosystem’s SharePlay with FaceTime, breaking down the barriers of distance and time.

How we built it

We built StoryLiving with the bleeding-edge Apple Vision Pro spatial computing device, focusing on user accessibility and intuitive design. Our development process involved designing an easy-to-navigate UI, ensuring that even users with no technical background could engage seamlessly. By incorporating spatial computing and photogrammetry, we crafted a platform where users can upload and navigate spatial memories, fostering a shared experience regardless of physical distance barriers.

Challenges we ran into

One of the main challenges was designing an interface that was both powerful and accessible to an older, less tech-savvy demographic. Achieving a balance between technological sophistication and ease-of-use required extensive user testing and iteration. We talked to many mentors and elders that provided advice on building an accessible, innovative new technology. Innovating on the brand new Apple Vision Pro platform was an exciting challenge.

Accomplishments that we're proud of

We are particularly proud of creating a platform that genuinely bridges the technology gap for elderly populations, making cutting-edge AR/VR technology accessible and meaningful. We tackled leveraging a combination of AR and VR visualizations, spatial audio, eye interaction, social multiuser presence, and more. A unique technical aspect of using the Vision Pro to build our solution was incorporating depth-based 3D photos and videos to showcase spatial memories.

Our user experience was designed to be easy-to-use without requiring a technical AR/VR background. It’s as simple as tapping a button and watching your family member’s spatial memories unfold in 3D. We were able to continually develop the idea and narrative, giving a compelling use case for elders in XR.

What we learned

Throughout this project, we learned the importance of empathetic design in technology. Crafting experiences that cater to the elderly taught us to prioritize simplicity and accessibility. We had to make thoughtful design decisions for human-centric XR interfaces, and by asking the right people for help, we were able to approach the problem of how to introduce the older generation to spatial computing. We also explored the potential of AR/VR to solve real-world problems such as social isolation by creating shared experiences that are vivid and emotionally resonant.

What's next for StoryLiving

Looking forward, we aim to expand StoryLiving's capabilities to include other sensory experiences, such as haptic feedback, elevating the immersive interactions in the experience. We plan to explore partnerships and integrations that can bring StoryLiving to more users, focusing on making it a tool for social impact. We are excited to contribute to this revolution in accessibility for elders in XR and spatial technology. By continuously refining the technology and user experience, we envision StoryLiving as a leading solution in combating social isolation among the elderly, transforming storytelling into story-living.