# Project export: StorySpark

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 11.0
- Tagline: Our project is a kid-friendly website that creates stories from voice or text prompts, generates images, and narrates them with a slideshow. Tech used( Fetch.ai, Gemini, LMNT, and Deepgram)
- Devpost: https://devpost.com/software/storyspark-hy5a34
- GitHub: https://github.com/JayJoshi4520/callhacks11.0
- Team: 0 GitHub contributor(s) — 

## Devpost submission (written by the team)

### Inspiration

We were inspired by the power of storytelling in children's development. We wanted to create an interactive platform that fuels kids' imaginations, allowing them to craft their own stories through simple voice or text inputs, bringing their creativity to life with visuals and narration.

### What it does

StorySpark is a kid-friendly platform that generates personalized stories from voice or text prompts, creates matching visuals, and narrates the story with a slideshow. It’s designed to be engaging and educational, sparking creativity while being easy to use. We also offer a terminal-friendly version for flexibility.

### How we built it

We used the Gemini API for generating stories, LMNT for converting text to speech, and Deepgram for speech-to-text functionality. The Fetch.ai uAgents framework enables the terminal-friendly version, ensuring accessibility for all users. The core technologies work seamlessly to deliver a fun, interactive experience.

### Challenges we ran into

One challenge was ensuring the seamless integration of the various APIs, especially managing real-time story generation, image creation, and audio synchronization. We also worked on optimizing the terminal version for users who prefer a non-browser experience.

### Accomplishments we're proud of

We’re proud of building an engaging platform that brings together story generation, visuals, and audio for a unique storytelling experience. Successfully implementing the terminal version using Fetch.ai uAgents was another achievement that makes our platform versatile.

### What we learned

We learned how to integrate multiple APIs effectively, ensuring smooth real-time interaction between story, visuals, and audio. Additionally, working with Fetch.ai uAgents gave us deeper insights into creating user-friendly terminal applications.

### What's next

Next, we plan to enhance the AI capabilities to allow for more complex storylines, add more customization options for visuals, and introduce multilingual support. We’ll also continue refining the user experience and expanding the platform’s reach to engage more children globally.

## README (from the GitHub repository)

No README available.

## Detected evidence (automated analysis)

Indexed codebase: 17 recognized source files, 29 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Google Gemini (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (23 of 23)

```
.DS_Store
.gitattributes
Frontend/.DS_Store
Frontend/package.json
Frontend/public/index.html
Frontend/public/manifest.json
Frontend/public/robots.txt
Frontend/README.md
Frontend/src/App.css
Frontend/src/App.js
Frontend/src/App.test.js
Frontend/src/components/ImagePlaceholder.js
Frontend/src/components/SidePanel.js
Frontend/src/css/homePage.css
Frontend/src/index.css
Frontend/src/index.js
Frontend/src/reportWebVitals.js
Frontend/src/setupProxy.js
Frontend/src/setupTests.js
Server/deepgram_agent.py
Server/gemini_agent.py
Server/lmnt_agent.py
Server/server.py
```

### Dependencies

- Frontend/package.json: @iconscout/unicons@^4.0.8, @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, axios@^1.7.7, http-proxy-middleware@^3.0.3, react@^18.3.1, react-dom@^18.3.1, react-scripts@5.0.1, web-vitals@^2.1.4

### Recent commits (newest first)

- Add Pack file
- Final Commit
- Final Commit
- Minar changes
- first commit

## Key source files (fetched from GitHub, selected and truncated for size)

### Frontend/package.json

```
{
  "name": "story-frontend",
  "version": "0.1.0",
  "proxy": "http://localhost:8000",
  "private": true,
  "dependencies": {
    "@iconscout/unicons": "^4.0.8",
    "@testing-library/jest-dom": "^5.17.0",
    "@testing-library/react": "^13.4.0",
    "@testing-library/user-event": "^13.5.0",
    "axios": "^1.7.7",
    "http-proxy-middleware": "^3.0.3",
    "react": "^18.3.1",
    "react-dom": "^18.3.1",
    "react-scripts": "5.0.1",
    "web-vitals": "^2.1.4"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": [
      "react-app",
      "react-app/jest"
    ]
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  }
}

```

### Server/server.py

```python
# Importing necessary libraries
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from deepgram_agent import speech_to_text, Message as DeepGramMessage
from gemini_agent import handle_message, Message as GeminiMessage
from lmnt_agent import handle_text_to_speech, Message as LMNTMessage



app = FastAPI()
origins = [
    "http://localhost:3000",
    # Add your frontend origins here
]

app.add_middleware(
    CORSMiddleware,
    allow_origins=origins,
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)



@app.post('/deepgram')
async def run_deepGram(msg: DeepGramMessage):
    data = await speech_to_text(msg)
    return data

@app.post('/gemini')
async def run_gemini(msg: GeminiMessage):
    data = await handle_message(msg)
    return data

@app.post('/lmnt')
async def run_LMNT(msg: LMNTMessage):
    data = await handle_text_to_speech(msg)
    return data

if __name__ == "__main__":
    import uvicorn
    print("......DeepGram Server")
    uvicorn.run(app, host="0.0.0.0", port=8000)
```

### Frontend/src/index.js

```javascript
import React from 'react';
import ReactDOM from 'react-dom';
import './App.css';  // Importing global CSS
import App from './App';  // Importing the main App component

ReactDOM.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>,
  document.getElementById('root')  // Rendering App inside 'root' div in public/index.html
);

```

### Frontend/src/App.js

```javascript
import React, { useState, useRef, useEffect } from "react";
import axios from 'axios';
import './App.css';
import './css/homePage.css'
import { isVisible } from "@testing-library/user-event/dist/utils";


const App = () => {
  const [inputText, setInputText] = useState("");
  const [currentImage, setCurrentImage] = useState(0);
  const [storyText, setStoryText] = useState([]);
  const [isAnimating, setIsAnimating] = useState(false);
  const [audioSrc, setAudioSrc] = useState([]); // Store decoded audio URL
  const [isRecording, setIsRecording] = useState(false); // Track if the mic is recording
  const mediaRecorderRef = useRef(null);
  const chunks = useRef([]); // To store audio data
  const [readyState, setreadyState] = useState(false)
  const [isMainVisible, setisMainVisible] = useState(true)


  const images = [
    '/images/7.png',
    '/images/8.png',
    '/images/9.png',
  ];

  useEffect(() => {
    // Set up the interval to switch the image every 5 seconds (5000 ms)
    const intervalId = setInterval(() => {
      setCurrentImage((prevImage) => (prevImage + 1) % images.length);
    }, 5000);

    // Clear the interval when the component unmounts
    return () => clearInterval(intervalId);
  }, [images.length]);

  const BASE_API = "http://localhost:8000"



  // const handleNext = () => {
  //   if (!isAnimating) {
  //     setIsAnimating(true);
  //     setTimeout(() => {
  //       setCurrentImage((prev) => (prev === images.length - 1 ? 0 : prev + 1));
  //       setIsAnimating(false);
  //     }, 500); // Duration of the animation
  //   }
  // };

  // const handlePrevious = () => {
  //   if (!isAnimating) {
  //     setIsAnimating(true);
  //     setTimeout(() => {
  //       setCurrentImage((prev) => (prev === 0 ? images.length - 1 : prev - 1));
  //       setIsAnimating(false);
  //     }, 500); // Duration of the animation
  //   }
  // };

  const handleRecord = () => {
    if (isRecording) {
      mediaRecorderRef.current.stop(); // Stop recording
      setIsRecording(false);
    } else {
      // Request audio access from the user
      navigator.mediaDevices.getUserMedia({ audio: true }).then((stream) => {
        const mediaRecorder = new MediaRecorder(stream);
        mediaRecorderRef.current = mediaRecorder;
        chunks.current = [];

        // Store the data in chunks
        mediaRecorder.ondataavailable = (e) => {
          chunks.current.push(e.data);
        };

        // When the recording stops
        mediaRecorder.onstop = () => {
          const blob = new Blob(chunks.current, { type: "audio/wav" });
          sendAudioToAPI(blob);
          setIsRecording(false)
          mediaRecorder.stop()
        };

        mediaRecorder.start();
        setIsRecording(true); // Update state to show that we are recording
      });
    }
  };

  const convertBlobToBase64 = (blob) => {
    return new Promise((resolve, reject) => {
      var reader = new FileReader();
      reader.onloadend = () => {
        const base64String = reader.result.split(',')[1]; // Extract Base64 string without MIME type
        resolve(base64String);
      };
      reader.onerror = (error) => reject(error);
      reader.readAsDataURL(blob);
    });
  };

  const decodeBase64Audio = (base64String) => {
    if (base64String) {
      const binaryString = atob(base64String);
      const binaryLen = binaryString.length;
      const bytes = new Uint8Array(binaryLen);

      for (let i = 0; i < binaryLen; i++) {
        bytes[i] = binaryString.charCodeAt(i);
      }


      const blob = new Blob([bytes], { type: 'audio/wav' });


      const url = URL.createObjectURL(blob);
      setAudioSrc(url);
    }
  };

  const handleStory = () => {
    setisMainVisible(false)
    const audio = new Audio(audioSrc)
    audio.play()
  }

  const submitRequest = () => {
    if (inputText) {
      axios.post(BASE_API + "/gemini", { "message": String(inputText) }).then((resGemi) => {
        setStoryText(resGemi.data);
        axios.post(BASE_API + "/lmnt", { "message": resGemi.data }).then((resLMNT) => {
          for (let index = 0; index < resLMNT.data.length; index++) {
            decodeBase64Audio(resLMNT.data[index]);
          }

          setreadyState(true)
          handleStory()

        }).catch(e => alert(e));

      });
    }

  };

  // Function to send the recorded audio blob to an API endpoint
  const sendAudioToAPI = (audioBlob) => {
    convertBlobToBase64(audioBlob).then((base64) => {
      axios.post(BASE_API + "/deepgram", {
        "message": base64
      }).then((res) => {
        axios.post(BASE_API + "/gemini", { "message": res.data }).then((resGemi) => {
          setStoryText(resGemi.data);
          axios.post(BASE_API + "/lmnt", { "message": resGemi.data }).then((resLMNT) => {
            for (let index = 0; index < resLMNT.data.length; index++) {
              decodeBase64Audio(resLMNT.data[index]);
            }
            setreadyState(true)
            handleStory()
            alert("Story Generated");
          }).catch(e => alert(e));
        });
      });
    });
  };
  const [text, setText] = useState('');
  const fullText = 'Hi, Excited for Story'; // The full text to be typed
  const typingSpeed = 100; // Speed in milliseconds

  const [currentImageIndex, setCurrentImageIndex] = useState(0);
  const [inputValue, setInputValue] = useState('');

  useEffect(() => {
    const intervalId = setInterval(() => {
      setCurrentImageIndex((prevIndex) => (prevIndex + 1) % images.length);
    }, 5000); // Change image every 5 seconds

    return () => clearInterval(intervalId); // Clean up the interval on unmount
  }, [images.length]);

  useEffect(() => {
    let index = 0;
    const intervalId = setInterval(() => {
      if (index < fullText.length) {
        setText((prevText) => prevText + fullText.charAt(index)); // Add one character at a time
        index++;
      } else {
        clearInterval(intervalId); // Clear interval when done
      }
    }, []);

    retu
[truncated — 2896 more characters]
```

### Server/lmnt_agent.py

```python
from lmnt.api import Speech
import base64
from pydub import AudioSegment
import playsound
from pydub.playback import play
import ast
from fastapi import FastAPI
from pydantic import BaseModel
import os

LMNT_API_KEY = '09dabaa99b474797a6df69afaa3721d7'  


class Message(BaseModel):
    message: list[str]



def play_output(subfile):
    song = AudioSegment.from_wav(subfile)
    play(song)

def encode_wav_to_base64(wav_file_path):
    with open(wav_file_path, "rb") as wav_file:
        wav_data = wav_file.read()
        return base64.b64encode(wav_data).decode('utf-8')


app = FastAPI()



@app.post('/lmnt')
async def handle_text_to_speech(msg: Message):
    count = 0
    baseList = []
    async with Speech('09dabaa99b474797a6df69afaa3721d7') as speech:
        for sentences in msg.message:
            synthesis = await speech.synthesize(sentences, voice='lily', format='wav')
            with open(f'output{count}.wav', 'wb') as f:
                f.write(synthesis['audio'])
                baseList.append(encode_wav_to_base64(f'output{count}.wav'))
                f.close()
                os.remove(f'output{count}.wav')
                count += 1
    return baseList
        




if __name__ == "__main__":
    import uvicorn
    print("......LMNT Server")
    uvicorn.run(app, host="0.0.0.0", port=3001)

```

### Server/deepgram_agent.py

```python

import ast
from dotenv import load_dotenv
import logging
from deepgram.utils import verboselogs
from datetime import datetime, timedelta
from io import BufferedReader
from deepgram import DeepgramClientOptions
import logging

from deepgram import (
    DeepgramClient,
    DeepgramClientOptions,
    StreamSource,
    PrerecordedOptions,
)
from fastapi.middleware.cors import CORSMiddleware
import base64
from pydub import AudioSegment
from pydub.playback import play
from fastapi import FastAPI
from pydantic import BaseModel



class Message(BaseModel):
    message: str




def decode_blob_to_wav(blob_data, output_file):
    base64_audio = blob_data

    # Decode the base64 string
    audio_data = base64.b64decode(base64_audio)

    # Save the decoded data to a .wav file

    with open(output_file, "wb") as f:
        f.write(audio_data)
        f.close()

    print(f"Audio saved to {output_file}")

    


# Example usage:


def get_data():
    try:
        config = DeepgramClientOptions(
            verbose=verboselogs.SPAM,
        )
        deepgram = DeepgramClient("455f87ad3614a2faf17b24d07b892654e3e9f03b", config)


        with open("output.wav", "rb") as stream:
            payload: StreamSource = {
                "stream": stream,
            }
            options = PrerecordedOptions(
                model="nova-2",
            )
            response = deepgram.listen.rest.v("1").transcribe_file(payload, options)


            print("Response received:")
            
            data = ast.literal_eval(response.to_json(indent=4))

            
            if isinstance(data, dict):
                transcript = data.get('results', {}).get('channels', [{}])[0].get('alternatives', [{}])[0].get('transcript', '')
                if transcript:
                    return str(transcript) 
                else:
                    print("No transcription available.")
            else:
                print("Unexpected response format.")

    except Exception as e:
        return -1

app = FastAPI()

origins = [
    "http://localhost:3003",
    # Add your frontend origins here
]

app.add_middleware(
    CORSMiddleware,
    allow_origins=origins,
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)










async def speech_to_text(msg: Message):
    blobObj = msg.message
    print(blobObj)
    decode_blob_to_wav(blobObj, "output.wav")
    data = get_data()
    
    return data
    

if __name__ == "__main__":
    import uvicorn
    print("......DeepGram Server")
    uvicorn.run(app, host="0.0.0.0", port=3002)

    
    



    
    








```

### Server/gemini_agent.py

```python
# Importing necessary libraries
from fastapi import FastAPI
from pydantic import BaseModel

import google.generativeai as genai

import time

# gemini = agent.Agent(
#     name="Gemini",
#     endpoint="localhost",
#     port="8000"
# )

class Message(BaseModel):
    message: str



 

app = FastAPI()
genai.configure(api_key="AIzaSyBLvM9A5jxeU3q1UXBuM_MVE1eV2E73l-g") 


model = genai.GenerativeModel("gemini-pro")


chat = model.start_chat(history=[])

inappropriate_words = [
    "violence", "kill", "murder", "blood", "gore", "death", 
    "drugs", "alcohol", "intoxicated", "gambling", 
    "sex", "nude", "porn", "prostitute", "abuse", "slavery", 
    "bomb", "terrorist", "gun", "knife", "suicide",
    "racism", "hate", "discrimination", "slur", 
    "curse", "swear", "damn", "hell", 
    "f***", "s***", "b****", "a**", "c***",
    "homophobic", "transphobic", "xenophobia", "misogyny"
]

print("Chat session has started. Type 'quit' to exit.")

def filter_inappropriate(text, inappropriate_words):
    words = text.split()
    for word in words:
        if word in inappropriate_words:
            return True



@app.post('/gemini')
async def handle_message(message: Message):
    while True:

        user_message = message.message


        if user_message.lower() == "quit":
            return -1


        response = chat.send_message(user_message, stream=True)


        full_response_text = []


        for chunk in response:
            full_response_text.append(str(chunk.text))

        return full_response_text



if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=3000)

# @Gemini_agent.on_event("startup")
# async def address(ctx: Context):

#     ctx.logger.info(Gemini_agent.address)



# @Gemini_agent.on_message(model=Message)
# async def handle_query_response(ctx: Context, sender: str, msg: Message):

#     # message = await handle_message(msg.message)
#     print(ctx)
#     print(msg.message)
#     print("Bellow is type in gemini")
#     # print(type(message))
#     # ctx.logger.info(message)
#     # await ctx.send(LMNT_API_KEY, Message(message=str(message)))

# # @Gemini_agent.on_query(model=Message, replies={Response})
# # async def handle_query_api_response(ctx: Context, sender: str, msg: Message):

# #     try: 
# #         response = await handle_message(msg.message)
# #         print("Bellow is type in gemini")
# #         print(type(response))
# #         ctx.logger.info(response)
# #         await ctx.send(sender, Response(text=str(response)))
# #     except Exception:
# #         await ctx.send(sender, Response(text="fail"))


# @Gemini_agent.on_rest_post("/rest/post", Message, Response)
# async def handle_post(ctx: Context, req: Message) -> Response:
#     ctx.logger.info("Received POST request")
#     return Response(
#         text=f"Received: {req.message}",
#         agent_address=ctx.Gemini_agent.address,
#         timestamp=int(time.time()),
#     )

```

### Frontend/src/setupTests.js

```javascript
// jest-dom adds custom jest matchers for asserting on DOM nodes.
// allows you to do things like:
// expect(element).toHaveTextContent(/react/i)
// learn more: https://github.com/testing-library/jest-dom
import '@testing-library/jest-dom';

```

### Frontend/src/App.test.js

```javascript
import { render, screen } from '@testing-library/react';
import App from './App';

test('renders learn react link', () => {
  render(<App />);
  const linkElement = screen.getByText(/learn react/i);
  expect(linkElement).toBeInTheDocument();
});

```

### Frontend/src/index.css

```css
@import url('https://fonts.googleapis.com/css2?family=Roboto+Mono:wght@400&display=swap');

body {
  font-family: 'Roboto Mono', monospace;
  margin: 0;
  padding: 0;
  box-sizing: border-box;
}

code {
  font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New',
    monospace;
}

```

[7 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]