# Project export: FlowSpeak This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: A customizable app that helps non-verbal individuals, including those with autism, communicate using icons, text-to-speech, and predictive AI, offering emotional insights and support through LLMs. - Devpost: https://devpost.com/software/flowspeak - GitHub: https://github.com/UCMercedACM/flowspeak - Team: 3 GitHub contributor(s) — No767 (16 commits), Javier (4 commits), adhya2005 (2 commits) ## Devpost submission (written by the team) ### Inspiration Inspired by the pressing issues of the lack of tools to aid with non-verbal people, Flowspeak was born. Pulled from the ideas and struggles of non-verbal communication, Flowspeak is designed to solve most of the issues in terms of accessibility and ease of use. ### What it does Flowspeak utilizes Picture Exchange Communication System (PECS), an evidence-based and proven method of communication for those who are non-verbal. In this context, non-verbal refers to the inability of communicating using verbal and social cues, and mainly affects those who live with autism, cerebral palsy, down syndrome, and many others. With these cards, an online collection can be created, cataloged, and utilized by the end user. Out of a selection of the collection, possible sentences are formed and processed by Google Gemini, and returned as a set of possible expressions. For example, if a user selects the cards “Mother”, “Park”, and “Ice cream”, a possible sentence would be generated as “Mother, I want to go to the park and eat ice cream”. As a result, the text-to-speech engine takes the sentence and describes it out loud. ### How we built it Our tech stack mainly consists of a traditional full-stack application: React Native for the frontend, FastAPI and SQLite for the backend. Google Gemini powers the possible prompts, which allows for a much faster, and widely accessible platform. The backend also contains most of the data processing, as images are stored internally as their original bytes + hashed w/ SHA-512 to create a unique identifier. This would allow for thousands of images to be stored, and migrations to PostgreSQL can be done very easily through SQLModel. ### Challenges we ran into We found our final team member late, which set us behind schedule. Our planning took too long, and software limitations added further delays. We also struggled with React Native issues, forcing late-night fixes. Despite these setbacks, we learned the importance of careful planning and communication. ### Accomplishments we're proud of What we are most proud of is how we developed much faster processes for development. Utilizing templates, we were able to create our applications much faster and develop standards on the protocol. For example, the backend is designed with a philosophy of rapid iteration rather than spending time learning new technologies. With this, we were able to spearhead operations and develop something that we are ultimately proud of creating in the first place. ### What we learned We learned quite a bit. Ranging from working with React Native and getting emulators properly set up, to faster implementations for backend code and AI integration. This was really a time to shine despite all of the flaws that had been made. We also learned that careful planning really does matter. Without the planning of every single tool used, the designs, and so much more, this would result in finishing the project much faster and not spending so long spending those countless precious hours on it. ### What's next FlowSpeak has major potential for a proper deployment. With the right amount of internal funding, and support, we can entirely deploy this into production. The backend server is entirely designed to be operated with tools such as Docker, and our React Native codebase would need to be polished up in order to be deployed. The next step would be to deploy this to the real world and to get some test results! ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 65 recognized source files, 96 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (105 of 105) ``` app/.gitignore app/api.js app/App.js app/app.json app/assets/avatar.js app/assets/fonts/bold.otf app/assets/fonts/medium.otf app/assets/index.css app/assets/index.js app/assets/public/app.css app/assets/public/index.html app/assets/public/manifest.json app/babel.config.js app/components/announcer.js app/components/announcerButton.js app/components/card.js app/components/child.js app/components/settings/add.js app/components/settings/comment.js app/components/settings/pack.js app/components/settings/setting.js app/components/settings/user.js app/components/setup/setup-five.js app/components/setup/setup-four.js app/components/setup/setup-one.js app/components/setup/setup-three.js app/components/setup/setup-two.js app/components/setup/setup-zero.js app/components/speaking.js app/components/swipeable-bottom/assets/images/index.js app/components/swipeable-bottom/components/SwipeIcon/index.js app/components/swipeable-bottom/index.js app/components/tabbar.js app/components/touchable-scale/index.d.ts app/components/touchable-scale/index.js app/components/touchable-scale/index.js.map app/data/images/README.md app/data/languages/card_de.json app/data/languages/card_el.json app/data/languages/card_en.json app/data/languages/card_es.json app/data/languages/card_fr.json app/data/languages/card_pa.json app/data/languages/card_pl.json app/data/languages/card_ro.json app/data/languages/card_tr.json app/data/languages/unusedlanguages/card_es.json app/data/languages/unusedlanguages/card_he.json app/data/languages/unusedlanguages/card_hi.json app/data/languages/unusedlanguages/card_id.json app/data/languages/unusedlanguages/card_it.json app/data/languages/unusedlanguages/card_nb_NO.json app/data/languages/unusedlanguages/card_nl.json app/data/languages/unusedlanguages/card_pt_BR.json app/data/languages/unusedlanguages/card_pt_PT.json app/data/languages/unusedlanguages/card_ru.json app/data/languages/unusedlanguages/card_sk.json app/data/languages/unusedlanguages/card_th.json app/data/languages/unusedlanguages/card_ua.json app/data/languages/unusedlanguages/card_vi_VN.json app/data/languages/unusedlanguages/card_zh_TW.json app/data/languages/unusedlanguages/card_zh.json app/data/text.json app/fonts/OFL.txt app/index.css app/index.js app/js/assets.js app/js/event.js app/js/language.js app/js/makeid.js app/layouts/cards.js app/layouts/groups.js app/layouts/layout.js app/layouts/search.js app/layouts/setting.js app/layouts/setup.js app/package.json app/public/app.css app/public/index.html app/public/manifest.json app/README.md app/staticAssetRequire.js lefthook.yml server/.gitignore server/config-example.yml server/pyproject.toml server/requirements-dev.txt server/requirements.txt server/src/config.yml server/src/core.py server/src/db/__init__.py server/src/db/id.py server/src/db/models.py server/src/launcher.py server/src/ml/main.py server/src/routes/__init__.py server/src/routes/cards.py server/src/routes/collection.py server/src/routes/me.py server/src/routes/users.py server/src/utils/config.py server/src/utils/request.py server/src/utils/sessions.py server/taskfile.yml server/tox.ini ``` ### Dependencies - app/package.json: @babel/core@^7.20.0, @react-native-async-storage/async-storage@^2.0.0, expo@~51.0.28, expo-localization@~15.0.3, expo-speech@~12.0.2, expo-status-bar@~1.12.1, react@18.2.0, react-native@0.74.5, react-native-storage@^1.0.1 - server/requirements.txt: aiosqlite@>=0.20.0,<1, fastapi@>=0.111.0,<1, google-generativeai@>=0.8.3, orjson@>=3.10.8,<4, PyYAML@>=6.0.2,<7, sonyflake-py@>=1.3.0,<2, SQLAlchemy@>=2.0.36,<3, sqlmodel@>=0.0.22,<1, uvicorn[standard]@>=0.12.0,<1 ### Recent commits (newest first) - Bump cross-spawn in /app (#14) - renuke old app - Implement attempted migration for AAC (#13) - Merge pull request #12 from UCMercedACM/ML - Finished ml model implementation needed - Revert "rm" - deleted old expo app - rm - add blank app - Run lefthook for code fmt (#11) - Implement basic authentication code (#10) - add auth code - install typescript dependencies and got expo running - add voice app - Merge pull request #2 from UCMercedACM/backend - add core database func + testing models - start working on database impl - surpress errors - Merge pull request #1 from UCMercedACM/backend - add changeS ## Key source files (fetched from GitHub, selected and truncated for size) ### server/requirements.txt ``` fastapi>=0.111.0,<1 uvicorn[standard]>=0.12.0,<1 PyYAML>=6.0.2,<7 orjson>=3.10.8,<4 sqlmodel>=0.0.22,<1 sonyflake-py>=1.3.0,<2 SQLAlchemy>=2.0.36,<3 aiosqlite>=0.20.0,<1 google-generativeai>=0.8.3 ``` ### app/package.json ``` { "name": "app", "version": "1.0.0", "main": "expo/AppEntry.js", "scripts": { "start": "expo start --tunnel", "android": "expo start --android --tunnel", "ios": "expo start --ios", "web": "expo start --web" }, "dependencies": { "@react-native-async-storage/async-storage": "^2.0.0", "expo": "~51.0.28", "expo-status-bar": "~1.12.1", "react": "18.2.0", "react-native": "0.74.5", "react-native-storage": "^1.0.1", "expo-speech": "~12.0.2", "expo-localization": "~15.0.3" }, "devDependencies": { "@babel/core": "^7.20.0" }, "private": true } ``` ### server/pyproject.toml ``` [project] name = "kanae" version = "0.1.0" description = "Internal backend server for ACM @ UC Merced" readme = "README.md" requires-python = ">=3.9,<4.0" [tool.pyright] include = [ "utils/**", "core.py", "launcher.py" ] exclude = [ "**/__pycache__", "**/.mypy_cache", "**/.dmpypy.json", ] reportMissingImports = true typeCheckingMode = "basic" reportUnnecessaryTypeIgnoreComment = "warning" [tool.bandit] skips = ["B311", "B101"] [tool.ruff] line-length = 80 extend-include = [ "core.py", "launcher.py", "utils/**" ] extend-exclude = ["**/__pycache__"] [tool.ruff.lint] ignore = ["E501", "N999", "E402", "S101"] select = ["E", "F", "N", "ASYNC", "S", "ERA", I] [tool.ruff.lint.isort] combine-as-imports = true force-wrap-aliases = true ``` ### app/index.js ```javascript import React from 'react'; import ReactDOM from 'react-dom'; import './index.css'; import App from './App'; ReactDOM.render(, document.getElementById('root')); ``` ### app/App.js ```javascript // Differentiate between setup and layout components import React from "react"; import { Text, View, StatusBar } from "react-native"; import Layout from "./layouts/layout"; import Setup from "./layouts/setup"; import * as Font from "expo-font"; import API from "./api"; export default class App extends React.Component { constructor(props) { super(props); this.state = { setup: false, font: false, }; } componentWillMount() { API.getData("setup").then((setupStatus) => { console.log(setupStatus); if (setupStatus == "start") { this.setState({ setup: "start" }); } else { this.setState({ setup: "done" }); } }); } componentDidMount() { Font.loadAsync({ rubik: require("./fonts/Rubik-Regular.ttf"), "rubik-bold": require("./fonts/Rubik-Bold.ttf"), }).then(() => { this.setState({ font: true }); }); this.checkTimeout(); } checkTimeout() { setTimeout(() => { if (!this.state.setup) { API.getData("setup").then((setupStatus) => { if (setupStatus == "start") { this.setState({ setup: "start" }); } else { this.setState({ setup: "done" }); } }); this.checkTimeout(); } }, 300); } setupFinished() { this.setState({ setup: "done" }); API.setData("setup", "done"); } renderMainComponent() { if (this.state.setup == "start") { return ; } else if (this.state.setup == "done") { return ; } else { return null; } } render() { if (this.state.font) { return ( ); } else { return null; } } } ``` ### app/assets/index.js ```javascript import React from 'react'; import ReactDOM from 'react-dom'; import './index.css'; import App from './App'; ReactDOM.render(, document.getElementById('root')); ``` ### app/layouts/layout.js ```javascript import React from 'react'; import { View, Dimensions } from 'react-native'; let { height, width } = Dimensions.get('window'); import SwipeUpDown from '../components/swipeable-bottom'; import Search from './search'; import API from '../api'; import CardArrayLanguage from '../js/language'; import Cards from './cards'; import Groups from './groups'; import Announcer from '../components/announcer'; export default class App extends React.Component { constructor(props){ super(props); API.gridSize = [3, 5]; this.state = { currentGroup: "general", layout: (height > width)?"portrait":"landscape", localizedCardData: this.getLocalizedCardData(API.currentLang) }; Dimensions.addEventListener("change", () => { let { height, width } = Dimensions.get('window'); this.setState({ layout: (height > width)?"portrait":"landscape" }); console.log(this.state.layout); }); } componentWillUnmount(){ Dimensions.addEventListener("change", () => {}); } getLocalizedCardData(lang){ return CardArrayLanguage[lang.split("-")[0]]; } onGroupChanged(newGroupString){ console.log(newGroupString); this.setState({currentGroup: newGroupString}); } componentDidMount(){ API.segment.screen("start"); } render() { return ( <> } // Pass props component when collapsed onShowMini={() => console.log('mini')} onShowFull={() => console.log('full')} onMoveDown={() => console.log('down')} onMoveUp={() => console.log('up')} animation={"spring"} swipeHeight={80} layout={this.state.layout} disablePressToShow={false} // Press item mini to show full style={{ backgroundColor: '#fff', shadowOpacity: 0.3, shadowRadius: 3, shadowOffset: { height: 0, width: 0 }, //android elevation: 1 }} // style for swipe /> ); } } ``` ### server/src/ml/main.py ```python API_KEY='AIzaSyDBecZvfdYJQP2oLX4kcRjPrbZK2_znXZk' import google.generativeai as genai import os genai.configure(api_key=API_KEY) import PIL.Image from pathlib import Path import os from PIL import Image import IPython.display as display import ipywidgets as widgets from pathlib import Path # Initialize the model (already defined) model = genai.GenerativeModel("gemini-1.5-flash") chat = model.start_chat( history=[ {"role": "user", "parts": "Hello"}, {"role": "model", "parts": "Answer the prompt"}, ] ) # Directory containing your PNG files image_dir = Path("/content") # Create a list of all .png images in the directory image_files = list(image_dir.glob("*.png")) # List to store the names (stems) of the selected images selected_images = [] # Function to handle image selection when a button is clicked def on_button_click(b): if b.description == "done": # Generate content using the selected images and pass it to the model generate_response() else: # Add the selected image's stem (filename without extension) to the list image_path = image_dir / b.description selected_images.append(image_path.stem) print(f"Selected images: {selected_images}") # Function to generate content when "done" is pressed def generate_response(): # Concatenate the selected image names into a single string user_input = ", ".join(selected_images) print(user_input) # Send the message to the model with the selected images in a sentence response = chat.send_message(f"Assume the user is a non verbal person. Generate a sentence explaining what I am trying to say using the text. No more information is provided. Give me 3-5 options. Give the answer in first person. DO NOT UNDER ANY CIRCUMSTANCES MENTION THE USER IS NON VERBAL: {user_input}") # Print the model's response (alternatively, display it in Colab) print(response.text) # Create buttons for each image file buttons = [] for img_path in image_files: # Create a button with the image file's name as the label button = widgets.Button(description=img_path.name) button.on_click(on_button_click) buttons.append(button) # Add a "done" button to trigger the content generation done_button = widgets.Button(description="done") done_button.on_click(on_button_click) buttons.append(done_button) # Display the buttons in the Colab interface widgets.VBox(buttons) ``` ### app/components/touchable-scale/index.js ```javascript "use strict"; var __extends = (this && this.__extends) || (function () { var extendStatics = Object.setPrototypeOf || ({ __proto__: [] } instanceof Array && function (d, b) { d.__proto__ = b; }) || function (d, b) { for (var p in b) if (b.hasOwnProperty(p)) d[p] = b[p]; }; return function (d, b) { extendStatics(d, b); function __() { this.constructor = d; } d.prototype = b === null ? Object.create(b) : (__.prototype = b.prototype, new __()); }; })(); var __assign = (this && this.__assign) || Object.assign || function(t) { for (var s, i = 1, n = arguments.length; i < n; i++) { s = arguments[i]; for (var p in s) if (Object.prototype.hasOwnProperty.call(s, p)) t[p] = s[p]; } return t; }; Object.defineProperty(exports, "__esModule", { value: true }); var React = require("react"); var react_native_1 = require("react-native"); var styles = react_native_1.StyleSheet.create({ root: __assign({}, react_native_1.Platform.select({ ios: {}, android: {} })) }); var Touchable = /** @class */ (function (_super) { __extends(Touchable, _super); function Touchable() { var _this = _super !== null && _super.apply(this, arguments) || this; _this.state = { anim: new react_native_1.Animated.Value(1) }; _this.handlePressIn = function () { react_native_1.Animated.timing(_this.state.anim, { toValue: 0.9, duration: 150, easing: react_native_1.Easing.out(react_native_1.Easing.quad), useNativeDriver: true }).start(); }; _this.handlePresOut = function () { react_native_1.Animated.timing(_this.state.anim, { toValue: 1, duration: 300, easing: react_native_1.Easing.out(react_native_1.Easing.quad), useNativeDriver: true }).start(); }; return _this; } Touchable.prototype.render = function () { return ( {this.props.children} ); }; return Touchable; }(React.Component)); exports.default = Touchable; //# sourceMappingURL=index.js.map ``` ### app/components/swipeable-bottom/index.js ```javascript import React, { Component } from 'react'; import {useCallback, useRef} from 'react'; import { Platform, StyleSheet, Text, View, PanResponder, Dimensions, LayoutAnimation, TouchableOpacity, TouchableWithoutFeedback } from 'react-native'; import SwipeIcon from './components/SwipeIcon'; import images from './assets/images'; import Event from '../../js/event'; const MARGIN_TOP = Platform.OS === 'ios' ? 50 : 30; const DEVICE_HEIGHT = Dimensions.get('window').height - MARGIN_TOP; type Props = { hasRef?: () => void, swipeHeight?: number, item?: object, disablePressToShow?: boolean, style?: object, onShowMini?: () => void, onShowFull?: () => void, animation?: 'linear' | 'spring' | 'easeInEaseOut' | 'none' }; export default class SwipeUpDown extends Component { static defaultProps = { disablePressToShow: false, }; constructor(props) { super(props); this.state = { collapsed: true }; this.disablePressToShow = props.disablePressToShow; this.SWIPE_HEIGHT = props.swipeHeight || 60; this._panResponder = null; this.top = Dimensions.get('window').height - MARGIN_TOP - this.SWIPE_HEIGHT; this.height = Dimensions.get('window').height * 2; this.customStyle = { style: { top: this.top, height: this.height } }; this.checkCollapsed = true; this.showFull = this.showFull.bind(this); // Event.on("showFull", () => { // this.showFull(); // }); //// // Event.on("showMini", () => { // this.showMini(); // }); // Event.addListener("showFull", () => {this.showFull()}); Event.addListener("showMini", () => {this.showMini()}); } componentWillMount() { this._panResponder = PanResponder.create({ onMoveShouldSetPanResponder: (event, gestureState) => true, onPanResponderMove: this._onPanResponderMove.bind(this), onPanResponderRelease: this._onPanResponderRelease.bind(this) }); } componentDidMount() { this.props.hasRef && this.props.hasRef(this); } componentWillReceiveProps(newProps){ if(newProps.layout != this.props.layout){ console.log("Layout Changed"); this.forceUpdate(); } } updateNativeProps() { var CustomLayoutSpring = { duration: 400, create: { type: LayoutAnimation.Types.spring, property: LayoutAnimation.Properties.opacity, springDamping: 0.7, }, update: { type: LayoutAnimation.Types.spring, springDamping: 0.7, }, }; switch (this.props.animation) { case 'linear': LayoutAnimation.linear(); break; case 'spring': LayoutAnimation.configureNext(CustomLayoutSpring); break; case 'easeInEaseOut': LayoutAnimation.easeInEaseOut(); break; case 'none': default: break; } const viewRef = useRef(null); viewRef.setNativeProps(this.customStyle); // useCallback(() => {this.viewRef.current?.setNativeProps(this.customStyle)}); // this.updateNatives(); } _onPanResponderMove(event, gestureState) { let DEVICE_HEIGHT = Dimensions.get('window').height - MARGIN_TOP; if (gestureState.dy > 0 && !this.checkCollapsed) { // SWIPE DOWN this.customStyle.style.top = this.top + gestureState.dy; this.swipeIconRef && this.swipeIconRef.setState({ icon: images.minus }); !this.state.collapsed && this.setState({ collapsed: true }); this.updateNativeProps(); } else if (this.checkCollapsed && gestureState.dy < -80) { // SWIPE UP this.top = 0; this.customStyle.style.top = DEVICE_HEIGHT + gestureState.dy; this.swipeIconRef && this.swipeIconRef.setState({ icon: images.minus, showIcon: true }); if (this.customStyle.style.top <= DEVICE_HEIGHT / 2) { this.swipeIconRef && this.swipeIconRef.setState({ icon: images.arrow_down, showIcon: true }); } this.updateNativeProps(); this.state.collapsed && this.setState({ collapsed: false }); } } _onPanResponderRelease(event, gestureState) { if (gestureState.dy < -50 || gestureState.dy < 50) { this.showFull(); } else { this.showMini(); } } showFull() { const { onShowFull } = this.props; this.customStyle.style.top = 0; this.swipeIconRef && this.swipeIconRef.setState({ icon: images.arrow_down, showIcon: true }); this.updateNativeProps(); this.state.collapsed && this.setState({ collapsed: false }); this.checkCollapsed = false; onShowFull && onShowFull(); Event.emit("showKeyboard"); } showMini() { const { onShowMini, item } = this.props; this.customStyle.style.top = Dimensions.get('window').height - MARGIN_TOP - this.SWIPE_HEIGHT; this.swipeIconRef && this.swipeIconRef.setState({ showIcon: false }); this.updateNativeProps(); !this.state.collapsed && this.setState({ collapsed: true }); this.checkCollapsed = true; onShowMini && onShowMini(); Event.emit("dismissKeyboard"); } render() { const { item, style } = this.props; const { collapsed } = this.state; return ( (this.viewRef = ref)} {...this._panResponder.panHandlers} style={[ styles.wrapSwipe, { top: Dimensions.get('window').height - MARGIN_TOP - 80, marginTop: MARGIN_TOP, height: Dimensions.get('window').height * 2 }, !item && collapsed && { marginBottom: -200 }, style ]} > this.showMini()} hasRef={ref => (this.swipeIconRef = ref)} /> {collapsed && this.showFull()}> [truncated — 392 more characters] ``` [56 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]