# Project export: EchoTales

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 11.0
- Tagline: Interactive Storytelling for Children
- Devpost: https://devpost.com/software/echotales
- GitHub: https://github.com/CanhuiH/calhack11
- Video: https://www.youtube.com/embed/M-800y8GHU4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 2 GitHub contributor(s) — Canhui Huang (3 commits), Sneha Muppalla (1 commits)

## Devpost submission (written by the team)

### Inspiration

The inspiration behind EchoTales came from a desire to create a truly engaging and nurturing reading experience for children. We wanted to combine the magic of storytelling with the power of AI to make stories not only more interactive but also emotionally supportive. We aim to create a comforting, personalized environment that adapts to each child’s unique reactions, fostering a love for reading and emotional growth.

### What it does

EchoTales allows users to not only hear the story, but also interact with it aswell, through talking to the characters, creating voices for their favorite characters, and even jump to their favorite parts of a story. Dynamic Character Voices EchoTales uses AI to simulate character voices that change in tone, pitch, and emotion based on the story's plot and context. As the narrative progresses, characters express a range of emotions, making the story more vivid and engaging for children. This feature allows the app to deliver an immersive experience, bringing characters to life and capturing the child’s imagination. Dynamic Character Voices EchoTales uses AI to simulate character voices that change in tone, pitch, and emotion based on the story's plot and context. As the narrative progresses, characters express a range of emotions, making the story more vivid and engaging for children. This feature allows the app to deliver an immersive experience, bringing characters to life and capturing the child’s imagination. Voice Cloning EchoTales includes a voice cloning option, enabling parents or friends to narrate stories in familiar voices. This feature uses LMNT’s technology to create realistic voice replicas, making bedtime stories more comforting and personal. By hearing the voices of loved ones, children can feel more connected to the story, which helps create a sense of security and warmth during storytelling sessions. Voice Cloning EchoTales includes a voice cloning option, enabling parents or friends to narrate stories in familiar voices. This feature uses LMNT’s technology to create realistic voice replicas, making bedtime stories more comforting and personal. By hearing the voices of loved ones, children can feel more connected to the story, which helps create a sense of security and warmth during storytelling sessions. Interaction with Characters, Change Pace of Story Users are allowed to interact with the characters in the story, whether its figuring out the current events of the story, understanding the characters thought's behind their actions, also talking to the narrator aswell, and asking the narrator to jump to a certain section of the text. Interaction with Characters, Change Pace of Story Users are allowed to interact with the characters in the story, whether its figuring out the current events of the story, understanding the characters thought's behind their actions, also talking to the narrator aswell, and asking the narrator to jump to a certain section of the text.

### How we built it

We developed EchoTales using TypeScript and React Native for the front end, ensuring a responsive and intuitive user interface for young users. The backend integrates LMNT for voice cloning and Hume AI for voice to voice interaction and narration, enabling dynamic voice adjustments and personalized audio interactions. This combination of technologies allows for a seamless, adaptive audiobook experience tailored to each child's emotional response.

### Challenges we ran into

One challenge was figuring out how to allow multiple voices to be applied. For instance, if Character A is saying a dialogue, Character B would now have a dialogue after, but would require a different voice model to use. This was fixed through determining the total number of characters in a story so the creation of base voices will be dont at the beginning.

### Accomplishments we're proud of

We're very proud of being able to apply applications that we are new to into our idea! And of course, being able to work as a team and planning the overall workflow in an effective manner.

### What we learned

We learned a lot of new tools throughout this project and exploring how to integrate various API's such as Hume AI and LMNT.

### What's next

We plan on adding an emotion sense technology that can determine the reactions of the reader throughout the story. The idea would be to integrate Hume AI’s emotion-sensing technology to detect children’s facial expressions during storytelling. If the system detects confusion, disinterest, or drowsiness, EchoTales can adjust the pacing, tone, or level of detail in real time. For example, if a child looks confused, the app pauses and offers a simpler explanation, while signs of drowsiness prompt a slower, more soothing narration. This adaptive feature ensures that the child remains engaged and receives appropriate emotional support throughout the story.

## README (from the GitHub repository)

# calhack11

## Detected evidence (automated analysis)

Indexed codebase: 23 recognized source files, 32 KB.
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- TypeScript (language) — detected in the code

## Codebase structure (from repository index)

### Files (31 of 31)

```
.DS_Store
backend/main.py
backend/requirements.txt
code.py
echotales/.gitignore
echotales/app.json
echotales/App.tsx
echotales/app/_layout.tsx
echotales/app/index.tsx
echotales/app/playing.tsx
echotales/babel.config.js
echotales/components/__tests__/__snapshots__/ThemedText-test.tsx.snap
echotales/components/__tests__/ThemedText-test.tsx
echotales/components/Collapsible.tsx
echotales/components/ExternalLink.tsx
echotales/components/HelloWave.tsx
echotales/components/navigation/TabBarIcon.tsx
echotales/components/ParallaxScrollView.tsx
echotales/components/ThemedText.tsx
echotales/components/ThemedView.tsx
echotales/constants/Colors.ts
echotales/eas.json
echotales/expo-env.d.ts
echotales/hooks/useColorScheme.ts
echotales/hooks/useColorScheme.web.ts
echotales/hooks/useThemeColor.ts
echotales/package.json
echotales/README.md
echotales/scripts/reset-project.js
echotales/tsconfig.json
README.md
```

### Dependencies

- echotales/package.json: @babel/core@^7.20.0, @expo/vector-icons@^14.0.2, @react-native-community/slider@^4.5.4, @react-navigation/bottom-tabs@^6.6.1, @react-navigation/native@^6.1.18, @react-navigation/stack@^6.4.1, @types/jest@^29.5.12, @types/react@~18.2.45, @types/react-test-renderer@^18.0.7, axios@^1.7.7, base64-js@^1.5.1, expo@~51.0.28, expo-av@~14.0.7, expo-constants@~16.0.2, expo-dev-client@~4.0.28, expo-file-system@~17.0.1, expo-font@~12.0.9, expo-linking@~6.3.1, expo-router@~3.5.23, expo-splash-screen@~0.27.5, expo-status-bar@~1.12.1, expo-system-ui@~3.0.7, expo-web-browser@~13.0.3, jest@^29.2.1, jest-expo@~51.0.3, lmnt-node@^1.2.3, react@18.2.0, react-dom@18.2.0, react-native@0.74.5, react-native-gesture-handler@~2.16.1, react-native-reanimated@~3.10.1, react-native-safe-area-context@^4.11.1, react-native-screens@^3.34.0, react-native-star-rating@^1.1.0, react-native-vector-icons@^10.2.0, react-native-web@~0.19.10, react-test-renderer@18.2.0, typescript@~5.3.3

### Recent commits (newest first)

- added echotales
- added backend
- Add files via upload
- Delete audios.zip
- add audios
- add audios
- add images
- add app folder
- uploaded partial codes
- backend
- Add files via upload
- update config
- basic app
- Initial commit

## Key source files (fetched from GitHub, selected and truncated for size)

### echotales/package.json

```
{
  "name": "echotales",
  "main": "expo-router/entry",
  "version": "1.0.0",
  "scripts": {
    "start": "expo start",
    "reset-project": "node ./scripts/reset-project.js",
    "android": "expo start --android",
    "ios": "expo start --ios",
    "web": "expo start --web",
    "test": "jest --watchAll",
    "lint": "expo lint"
  },
  "jest": {
    "preset": "jest-expo"
  },
  "dependencies": {
    "@expo/vector-icons": "^14.0.2",
    "@react-native-community/slider": "^4.5.4",
    "@react-navigation/bottom-tabs": "^6.6.1",
    "@react-navigation/native": "^6.1.18",
    "@react-navigation/stack": "^6.4.1",
    "axios": "^1.7.7",
    "base64-js": "^1.5.1",
    "expo": "~51.0.28",
    "expo-av": "~14.0.7",
    "expo-constants": "~16.0.2",
    "expo-dev-client": "~4.0.28",
    "expo-file-system": "~17.0.1",
    "expo-font": "~12.0.9",
    "expo-linking": "~6.3.1",
    "expo-router": "~3.5.23",
    "expo-splash-screen": "~0.27.5",
    "expo-status-bar": "~1.12.1",
    "expo-system-ui": "~3.0.7",
    "expo-web-browser": "~13.0.3",
    "lmnt-node": "^1.2.3",
    "react": "18.2.0",
    "react-dom": "18.2.0",
    "react-native": "0.74.5",
    "react-native-gesture-handler": "~2.16.1",
    "react-native-reanimated": "~3.10.1",
    "react-native-safe-area-context": "^4.11.1",
    "react-native-screens": "^3.34.0",
    "react-native-star-rating": "^1.1.0",
    "react-native-vector-icons": "^10.2.0",
    "react-native-web": "~0.19.10"
  },
  "devDependencies": {
    "@babel/core": "^7.20.0",
    "@types/jest": "^29.5.12",
    "@types/react": "~18.2.45",
    "@types/react-test-renderer": "^18.0.7",
    "jest": "^29.2.1",
    "jest-expo": "~51.0.3",
    "react-test-renderer": "18.2.0",
    "typescript": "~5.3.3"
  },
  "private": true
}

```

### backend/main.py

```python
from fastapi import FastAPI
from fastapi.responses import FileResponse
from lmnt.api import Speech
import os
import asyncio

app = FastAPI()

@app.get("/synthesize/")
async def synthesize(text: str, voice: str = "lily"):
    async with Speech() as speech:
        synthesis = await speech.synthesize(text, voice)
        with open("output.mp3", "wb") as f:
            f.write(synthesis['audio'])
    return FileResponse("output.mp3", media_type='audio/mpeg')

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)
```

### echotales/App.tsx

```typescript
import React from 'react';
import { NavigationContainer } from '@react-navigation/native';
import { createStackNavigator } from '@react-navigation/stack';
import LibraryPage from './src/LibraryPage'; // Adjust the path to where LibraryPage is located
import PlayingPage from './src/PlayingPage'; // Adjust the path to where PlayingPage is located

const Stack = createStackNavigator();

const App = () => {
  return (
    <NavigationContainer>
      <Stack.Navigator initialRouteName="Library">
        <Stack.Screen name="Library" component={LibraryPage} />
        <Stack.Screen name="Playing" component={PlayingPage} />
      </Stack.Navigator>
    </NavigationContainer>
  );
};

export default App;
```

### echotales/app/_layout.tsx

```typescript
// /app/_layout.tsx
import { Tabs } from 'expo-router';
import { FontAwesome } from '@expo/vector-icons';

export default function Layout() {
  return (
    <Tabs
      screenOptions={({ route }) => ({
        tabBarIcon: ({ color, size }) => {
          let iconName;
          if (route.name === 'index') {
            iconName = 'home';
          } else if (route.name === 'playing') {
            iconName = 'play-circle';
          }
          return <FontAwesome name={iconName} size={size} color={color} />;
        },
        tabBarActiveTintColor: '#A259FF',
        tabBarInactiveTintColor: 'gray',
      })}
    >
      <Tabs.Screen
        name="index"
        options={{
          title: 'Library',
          tabBarLabel: 'home',
        }}
      />
      <Tabs.Screen
        name="[storyId]"
        options={{
          title: 'Now Playing',
          tabBarLabel: '',
        }}
      />
    </Tabs>
  );
}
```

### echotales/app/index.tsx

```typescript
import React from 'react';
import { View, Text, Image, TouchableOpacity, StyleSheet, ScrollView } from 'react-native';
import { useNavigation } from '@react-navigation/native'; // Import the navigation hook

interface Story {
  id: string;
  title: string;
  image: any; // Use local images or external URLs
}

const stories: Story[] = [
  {
    id: '1',
    title: "Goldilocks and the Three Bears",
    image: require('../assets/images/bears.jpg'),
  },
  {
    id: '2',
    title: 'The Ugly Duckling',
    image: require('../assets/images/duck.jpg'),
  },
  {
    id: '3',
    title: 'The Three Little Pigs',
    image: require('../assets/images/pig.jpg'),
  },
  {
    id: '4',
    title: 'Little Red Riding Hood',
    image: require('../assets/images/red.jpg'),
  },
];

const LibraryPage: React.FC = () => {
  const navigation = useNavigation(); // Hook to navigate between screens

  const handleCardPress = () => {
    navigation.navigate('playing'); // Navigate to the PlayingPage without any specific story data
  };

  return (
    <ScrollView contentContainerStyle={styles.container}>
      <Text style={styles.header}>Echo Tales</Text>
      <View style={styles.grid}>
        {stories.map((story) => (
          <TouchableOpacity
            key={story.id}
            style={styles.card}
            onPress={handleCardPress} // Navigate to PlayingPage when a card is pressed
          >
            <Image source={story.image} style={styles.storyImage} />
            <Text style={styles.storyTitle}>{story.title}</Text>
          </TouchableOpacity>
        ))}
      </View>
    </ScrollView>
  );
};

const styles = StyleSheet.create({
  container: {
    padding: 20,
    alignItems: 'center',
    backgroundColor: '#F4F3F9',
  },
  header: {
    fontSize: 24,
    fontWeight: 'bold',
    color: '#A259FF',
    marginBottom: 20,
    textAlign: 'center',
  },
  grid: {
    flexDirection: 'row',
    flexWrap: 'wrap',
    justifyContent: 'space-between', // Space between the cards
  },
  card: {
    width: '48%', // Adjust width to show two cards per row
    backgroundColor: '#FFF',
    borderRadius: 15,
    padding: 15,
    marginBottom: 20,
    alignItems: 'center',
    shadowColor: '#000',
    shadowOffset: { width: 0, height: 2 },
    shadowOpacity: 0.1,
    shadowRadius: 5,
    height: 220, // Fixed height for consistent layout
  },
  storyImage: {
    width: '100%', // Make the image fill the card's width
    height: 120,
    borderRadius: 10,
    marginBottom: 10,
  },
  storyTitle: {
    fontSize: 16,
    fontWeight: 'bold',
    color: '#A259FF',
    textAlign: 'center',
  },
});

export default LibraryPage;
```

### code.py

```python
import asyncio
import base64
import datetime
import os
from dotenv import load_dotenv
from hume.client import AsyncHumeClient
from hume.empathic_voice.chat.socket_client import ChatConnectOptions, ChatWebsocketConnection
from hume.empathic_voice.chat.types import SubscribeEvent
from hume.empathic_voice.types import UserInput
from hume.core.api_error import ApiError
from hume import MicrophoneInterface, Stream

class WebSocketInterface:
    """Interface for containing the EVI WebSocket and associated socket handling behavior."""

    def __init__(self):
        """Construct the WebSocketInterface, initially assigning the socket to None and the byte stream to a new Stream object."""
        self.socket = None
        self.byte_strs = Stream.new()

    def set_socket(self, socket: ChatWebsocketConnection):
        """Set the socket.
        
        This method assigns the provided asynchronous WebSocket connection
        to the instance variable `self.socket`. It is invoked after successfully
        establishing a connection using the client's connect method.

        Args:
            socket (ChatWebsocketConnection): EVI asynchronous WebSocket returned by the client's connect method.
        """
        self.socket = socket

    async def on_open(self):
        """Logic invoked when the WebSocket connection is opened."""
        print("WebSocket connection opened.")

    async def on_message(self, message: SubscribeEvent):
        """Callback function to handle a WebSocket message event.
        
        This asynchronous method decodes the message, determines its type, and 
        handles it accordingly. Depending on the type of message, it 
        might log metadata, handle user or assistant messages, process
        audio data, raise an error if the message type is "error", and more.

        This method interacts with the following message types to demonstrate logging output to the terminal:
        - [chat_metadata](https://dev.hume.ai/reference/empathic-voice-interface-evi/chat/chat#receive.Chat%20Metadata.type)
        - [user_message](https://dev.hume.ai/reference/empathic-voice-interface-evi/chat/chat#receive.User%20Message.type)
        - [assistant_message](https://dev.hume.ai/reference/empathic-voice-interface-evi/chat/chat#receive.Assistant%20Message.type)
        - [audio_output](https://dev.hume.ai/reference/empathic-voice-interface-evi/chat/chat#receive.Audio%20Output.type)

        Args:
            data (SubscribeEvent): This represents any type of message that is received through the EVI WebSocket, formatted in JSON. See the full list of messages in the API Reference [here](https://dev.hume.ai/reference/empathic-voice-interface-evi/chat/chat#receive).
        """

        # Create an empty dictionary to store expression inference scores
        scores = {}

        if message.type == "chat_metadata":
            message_type = message.type.upper()
            chat_id = message.chat_id
            chat_group_id = message.chat_group_id
            text = f"<{message_type}> Chat ID: {chat_id}, Chat Group ID: {chat_group_id}"
        elif message.type in ["user_message", "assistant_message"]:
            role = message.message.role.upper()
            message_text = message.message.content
            text = f"{role}: {message_text}"
            if message.from_text is False:
                scores = dict(message.models.prosody.scores)
        elif message.type == "audio_output":
            message_str: str = message.data
            message_bytes = base64.b64decode(message_str.encode("utf-8"))
            await self.byte_strs.put(message_bytes)
            return
        elif message.type == "error":
            error_message: str = message.message
            error_code: str = message.code
            raise ApiError(f"Error ({error_code}): {error_message}")
        else:
            message_type = message.type.upper()
            text = f"<{message_type}>"
        
        # Print the formatted message
        self._print_prompt(text)

        # Extract and print the top 3 emotions inferred from user and assistant expressions
        if len(scores) > 0:
            top_3_emotions = self._extract_top_n_emotions(scores, 3)
            self._print_emotion_scores(top_3_emotions)
            print("")
        else:
            print("")
        
    async def on_close(self):
        """Logic invoked when the WebSocket connection is closed."""
        print("WebSocket connection closed.")

    async def on_error(self, error):
        """Logic invoked when an error occurs in the WebSocket connection.
        
        See the full list of errors [here](https://dev.hume.ai/docs/resources/errors).

        Args:
            error (Exception): The error that occurred during the WebSocket communication.
        """
        print(f"Error: {error}")

    def _print_prompt(self, text: str) -> None:
        """Print a formatted message with a timestamp.

        Args:
            text (str): The message text to be printed.
        """
        now = datetime.datetime.now(tz=datetime.timezone.utc)
        now_str = now.strftime("%H:%M:%S")
        print(f"[{now_str}] {text}")

    def _extract_top_n_emotions(self, emotion_scores: dict, n: int) -> dict:
        """
        Extract the top N emotions based on confidence scores.

        Args:
            emotion_scores (dict): A dictionary of emotions and their corresponding confidence scores.
            n (int): The number of top emotions to extract.

        Returns:
            dict: A dictionary containing the top N emotions as keys and their raw scores as values.
        """
        # Convert the dictionary into a list of tuples and sort by the score in descending order
        sorted_emotions = sorted(emotion_scores.items(), key=lambda item: item[1], reverse=True)

        # Extract the top N emotions
        top_n_emotions = {emotion: score for emotion, score in sorted_emotions[:n]}

        return top_n_emotions

    def _prin
[truncated — 3457 more characters]
```

### echotales/babel.config.js

```javascript
module.exports = function (api) {
  api.cache(true);
  return {
    presets: ['babel-preset-expo'],
  };
};

```

### echotales/expo-env.d.ts

```typescript
/// <reference types="expo/types" />

// NOTE: This file should not be edited and should be in your git ignore
```

### echotales/hooks/useColorScheme.ts

```typescript
export { useColorScheme } from 'react-native';

```

### echotales/components/ThemedView.tsx

```typescript
import { View, type ViewProps } from 'react-native';

import { useThemeColor } from '@/hooks/useThemeColor';

export type ThemedViewProps = ViewProps & {
  lightColor?: string;
  darkColor?: string;
};

export function ThemedView({ style, lightColor, darkColor, ...otherProps }: ThemedViewProps) {
  const backgroundColor = useThemeColor({ light: lightColor, dark: darkColor }, 'background');

  return <View style={[{ backgroundColor }, style]} {...otherProps} />;
}

```

[12 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]