# Project export: PulseWalk This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Our cutting-edge shoe enhances awareness for the visually impaired by combining haptic, pressurebased feedback, and infrared tech, detecting terrain changes and obstacles for safer, smoother mobility. - Devpost: https://devpost.com/software/pulsewalk - GitHub: https://github.com/lucasreljic/Cal-Hacks-shoe-thing - Result: winner (Cal Hacks: Best Hardware Hack) - Team: 3 GitHub contributor(s) — Lucas Reljic (14 commits), darkdestroyeriii (7 commits), EnnyBae-Chilled (2 commits) ## Devpost submission (written by the team) ### Overview What inspired you? The inspiration behind this project came from my grandmother, who has struggled with poor vision for years. Growing up, I witnessed firsthand how her limited sight made daily tasks and walking in unfamiliar environments increasingly difficult for her. I remember one specific instance when she tripped on an uneven curb outside the grocery store. Though she wasn’t hurt, the fall shook her confidence, and she became hesitant to go on walks or run errands by herself. This incident helped spark the idea of creating something that could help people like her feel safer and more secure while navigating their surroundings. I wanted to develop a solution that would give visually impaired individuals not just mobility but confidence. The goal became clear: to create a product that could intuitively guide users through their environment, detecting obstacles like curbs, steps, and uneven terrain, and providing feedback they could easily understand. By incorporating haptic feedback, pressure-based sensors, and infrared technology, this system is designed to give users more control and awareness over their movements, helping them move through the world with greater independence and assurance. My hope is that this technology can empower people like my grandmother to reclaim their confidence and enjoy everyday activities without fear. ### What it does This project is a smart shoe system designed to help visually impaired individuals safely navigate their surroundings by detecting obstacles and terrain changes. It uses infrared sensors located on both the front and bottom of the shoe to detect the distance to obstacles like curbs and stairs. When the user approaches an obstacle, the system provides real-time feedback through 5 servos. 3 servos are responsible for haptic feedback related to the distance from the ground, and distance in front of them, while the remaining servos are related to help guiding the user through navigation. The infrared sensors detect how far the foot is off the ground, and the servos respond accordingly. The vibrational motors, labeled 1a and 2a, are used when the distance exceeds 6 inches, delivering pulsating signals to inform the user of upcoming terrain changes. This real-time feedback ensures users can sense potential dangers and adjust their steps to prevent falls or missteps. Additionally the user would connect to the shoe based off of bluetooth. The shoe system operates using three key zones of detection: the walking range (0-6 inches), the far walking range (6-12 inches), and the danger zone (12+ inches). In the walking range, the haptic feedback is minimal but precise, giving users gentle vibrations when the shoe detects small changes, such as a flat surface or minor elevation shifts. As the shoe moves into the far walking range (6-12 inches), where curbs or stairs may appear, the intensity of the feedback increases, and the vibrational motors start to pulse more frequently. This alert serves as a warning that the user is approaching a significant elevation change. When the distance exceeds 12 inches—the danger zone—the vibrational motors deliver intense, rapid feedback to indicate a drop-off or large obstacle, ensuring the user knows to take caution and adjust their step. These zones are carefully mapped to provide a seamless understanding of the terrain without overwhelming the user. The system also integrates seamlessly with a mobile app, offering GPS-based navigation via four directional haptic feedback sensors that guide the user forward, backward, left, or right. Users can set their route through voice commands, unfortunately we had trouble integrating Deepgram AI, which would assist by understanding speech patterns, accents, and multiple languages, making it accessible to people who are impaired lingually. Additionally we had trouble integrating Skylo, which the idea would be to serve areas where Wi-Fi is unavailable, or connection unstable, the system automatically switches to Skylo via their Type1SC circuit board and antenna, a satellite backup technology, to ensure constant connectivity. Skylo sends out GPS updates every 1-2 minutes, preventing the shoe from losing its route data. If the user strays off course, Skylo triggers immediate rerouting instructions through google map’s api in the app which we did set up in our app, ensuring that they are safely guided back on track. This combination of sensor-driven feedback, haptic alerts, and robust satellite connectivity guarantees that visually impaired users can move confidently through diverse environments. ### How we built it We built this project using a combination of hardware components and software integrations. To start, we used infrared sensors placed at the front and bottom of the shoe to detect distance and obstacles. We incorporated five servos into the design: three for haptic feedback based on distance sensing 3 and two for GPS-related feedback. Additionally, we used vibrational motors (1a and 2a) to provide intense feedback when larger drops or obstacles were detected. The app we developed integrates the Google Maps API for route setting and navigation. To ensure connectivity in areas with limited Wi-Fi, we integrated Skylo’s Type 1SC satellite hardware, allowing for constant GPS data transmission even in remote areas. For the physical prototype, we constructed a 3D model of a shoe out of cardboard. Attached to this model are two 5000 milliamp-hour batteries, providing a total of 10000 mAh to power the system. We used an ESP32 microcontroller to manage the various inputs and outputs, along with a power distribution board to efficiently allocate power to the servos, sensors, and vibrational motors. All components were securely attached to the cardboard shoe prototype to create a functional model for testing. ### Challenges we ran into One of the main challenges we encountered was working with the Skylo Type 1SC hardware. While the technology itself was impressive, the PDF documentation and schematics were quite advanced, requiring us to dive deeper into understanding the technical details. We successfully established communication between the Arduino and the Type 1SC circuit but faced difficulties in receiving a response back from the modem, which required further troubleshooting. Additionally, distinguishing between the different components on the circuit, such as data pins and shorting components, proved challenging, as the labeling was intricate and required careful attention. These hurdles allowed us to refine our skills in circuit analysis and deepen our knowledge of satellite communication systems. On the software side, we had to address several technical challenges. Matching the correct Java version for our app development was more complex than expected, as version discrepancies affected performance. We also encountered difficulties creating a Bluetooth hotspot that could seamlessly integrate with the Android UI for smooth user interaction. On the hardware end, ensuring reliable connections was another challenge; we found that some of our solder joints for the pins weren’t as stable as needed, leading to occasional issues with connectivity. Through persistent testing and adjusting our approaches, we were able to resolve most of these challenges while gaining valuable experience in both hardware and software integration. ### Accomplishments we're proud of One of the accomplishments we’re most proud of is successfully setting up Skylo services and establishing satellite connectivity, allowing the system to access LTE data in areas with low or no Wi-Fi. This was a key component of the project, and getting the hardware to communicate with satellites smoothly was a significant milestone. Despite the initial challenges with understanding the complex schematics, we were able to wire the Arduino to the Type 1SC board correctly, ensuring that the system could relay GPS data and maintain consistent communication. The experience gave us a deeper appreciation for satellite technology and its role in enhancing connectivity for projects like ours. Additionally, we’re proud of how well the array of sensors was set up and how all the hardware components functioned together. Each sensor, whether for terrain detection or obstacle awareness, worked seamlessly with the servos and haptic feedback system, resulting in better-than-expected performance. The responsiveness of the hardware components was more precise and reliable than we had originally anticipated, which demonstrated the strength of our design and implementation. This level of integration and functionality validates our approach and gives us confidence in the potential impact this project can have for the visually impaired community. ### What we learned Throughout this project, we gained a wide range of new skills that helped bring the system to life. One of our team members learned to solder, which was essential for securing the hardware components and making reliable connections. We also expanded our knowledge in React system programming, which allowed us to create the necessary interactions and controls for the app. Additionally, learning to use Flutter enabled us to develop a smooth and responsive mobile interface that integrates with the hardware components. On the hardware side, we became much more familiar with the ESP32 microcontroller, particularly its Bluetooth connectivity functions, which were crucial for communication between the shoe and the mobile app. We also had the opportunity to dive deep into working with the Type 1SC board, becoming comfortable with its functionality and satellite communication features. These new skills not only helped us solve the challenges within the project but also gave us valuable experience for future work in both hardware and software integration. ### What's next Next for PulseWalk, we plan to enhance the system's capabilities by refining the software to provide even more precise feedback and improve user experience. We aim to integrate additional features, such as obstacle detection for more complex terrains and improved GPS accuracy with real-time rerouting. Expanding the app’s functionality to include more languages and customization options using Deepgram AI will ensure greater accessibility for a diverse range of users. Additionally, we’re looking into optimizing battery efficiency and exploring more durable materials for the shoe design, moving beyond the cardboard prototype. Ultimately, we envision PulseWalk evolving into a fully commercialized product that offers a seamless, dependable mobility aid for the visually impaired which partners with shoe brands to bring a minimalist approach to the brand and make it look less like a medical device and more like an everyday product. ## README (from the GitHub repository) # PulseWalk Project also on Devpost: https://devpost.com/software/pulsewalk ## Detected evidence (automated analysis) Indexed codebase: 23 recognized source files, 31 KB. - C++ (language) — detected in the code - JavaScript (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - Dart (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (35 of 35) ``` .DS_Store .gitignore CalHacks/.gitignore CalHacks/android/.gitignore CalHacks/app.json CalHacks/app/_layout.tsx CalHacks/app/(tabs)/_layout.tsx CalHacks/app/(tabs)/explore.tsx CalHacks/app/(tabs)/index.tsx CalHacks/app/+html.tsx CalHacks/app/+not-found.tsx CalHacks/babel.config.js CalHacks/components/__tests__/__snapshots__/ThemedText-test.tsx.snap CalHacks/components/__tests__/ThemedText-test.tsx CalHacks/components/Collapsible.tsx CalHacks/components/ExternalLink.tsx CalHacks/components/HelloWave.tsx CalHacks/components/navigation/TabBarIcon.tsx CalHacks/components/ParallaxScrollView.tsx CalHacks/components/ThemedText.tsx CalHacks/components/ThemedView.tsx CalHacks/constants/Colors.ts CalHacks/eas.json CalHacks/hooks/useColorScheme.ts CalHacks/hooks/useColorScheme.web.ts CalHacks/hooks/useThemeColor.ts CalHacks/ios/.gitignore CalHacks/package.json CalHacks/README.md CalHacks/scripts/reset-project.js CalHacks/tsconfig.json ESP32-Calhacks/ESP32-Calhacks.ino ESP32-Calhacks/sensorServo.cpp LICENSE README.md ``` ### Dependencies - CalHacks/package.json: @babel/core@^7.20.0, @expo/vector-icons@^14.0.2, @react-navigation/native@^6.0.2, @types/jest@^29.5.12, @types/react@~18.2.45, @types/react-test-renderer@^18.0.7, axios@^1.7.7, expo@~51.0.28, expo-constants@~16.0.2, expo-font@~12.0.9, expo-linking@~6.3.1, expo-location@~17.0.1, expo-router@~3.5.23, expo-splash-screen@~0.27.5, expo-status-bar@~1.12.1, expo-system-ui@~3.0.7, expo-web-browser@~13.0.3, jest@^29.2.1, jest-expo@~51.0.3, react@18.2.0, react-dom@18.2.0, react-native@0.74.5, react-native-gesture-handler@~2.16.1, react-native-maps@1.14.0, react-native-maps-directions@^1.9.0, react-native-reanimated@~3.10.1, react-native-safe-area-context@4.10.5, react-native-screens@3.31.1, react-native-web@~0.19.10, react-test-renderer@18.2.0, typescript@~5.3.3 ### Recent commits (newest first) - Update README.md - final modifications - fixed 9g servos - Merge branch 'main' of https://github.com/lucasreljic/Cal-Hacks-shoe-thing - A mess - gitignore - small clean up - Merge branch 'main' of github.com:lucasreljic/Cal-Hacks-shoe-thing - added direction parsing - Merge branch 'main' of https://github.com/lucasreljic/Cal-Hacks-shoe-thing - Cleaned up the file a little and some other stuff. - i - Updated arduino code - Merge branches 'main' and 'main' of https://github.com/lucasreljic/Cal-Hacks-shoe-thing - Added interaction via bluetooth - removed key - added gitignore - added gitignore - added react native app - removed folder ## Key source files (fetched from GitHub, selected and truncated for size) ### CalHacks/package.json ``` { "name": "calhacks", "main": "expo-router/entry", "version": "1.0.0", "scripts": { "start": "expo start", "reset-project": "node ./scripts/reset-project.js", "android": "expo start --android", "ios": "expo start --ios", "web": "expo start --web", "test": "jest --watchAll", "lint": "expo lint" }, "jest": { "preset": "jest-expo" }, "dependencies": { "@expo/vector-icons": "^14.0.2", "@react-navigation/native": "^6.0.2", "axios": "^1.7.7", "expo": "~51.0.28", "expo-constants": "~16.0.2", "expo-font": "~12.0.9", "expo-linking": "~6.3.1", "expo-location": "~17.0.1", "expo-router": "~3.5.23", "expo-splash-screen": "~0.27.5", "expo-status-bar": "~1.12.1", "expo-system-ui": "~3.0.7", "expo-web-browser": "~13.0.3", "react": "18.2.0", "react-dom": "18.2.0", "react-native": "0.74.5", "react-native-gesture-handler": "~2.16.1", "react-native-maps": "1.14.0", "react-native-maps-directions": "^1.9.0", "react-native-reanimated": "~3.10.1", "react-native-safe-area-context": "4.10.5", "react-native-screens": "3.31.1", "react-native-web": "~0.19.10" }, "devDependencies": { "@babel/core": "^7.20.0", "@types/jest": "^29.5.12", "@types/react": "~18.2.45", "@types/react-test-renderer": "^18.0.7", "jest": "^29.2.1", "jest-expo": "~51.0.3", "react-test-renderer": "18.2.0", "typescript": "~5.3.3" }, "private": true } ``` ### CalHacks/app/_layout.tsx ```typescript import { DarkTheme, DefaultTheme, ThemeProvider } from '@react-navigation/native'; import { useFonts } from 'expo-font'; import { Stack } from 'expo-router'; import * as SplashScreen from 'expo-splash-screen'; import { useEffect } from 'react'; import 'react-native-reanimated'; import { useColorScheme } from '@/hooks/useColorScheme'; // Prevent the splash screen from auto-hiding before asset loading is complete. SplashScreen.preventAutoHideAsync(); export default function RootLayout() { const colorScheme = useColorScheme(); const [loaded] = useFonts({ SpaceMono: require('../assets/fonts/SpaceMono-Regular.ttf'), }); useEffect(() => { if (loaded) { SplashScreen.hideAsync(); } }, [loaded]); if (!loaded) { return null; } return ( ); } ``` ### CalHacks/app/(tabs)/_layout.tsx ```typescript import { Tabs } from 'expo-router'; import React from 'react'; import { TabBarIcon } from '@/components/navigation/TabBarIcon'; import { Colors } from '@/constants/Colors'; import { useColorScheme } from '@/hooks/useColorScheme'; export default function TabLayout() { const colorScheme = useColorScheme(); return ( ( ), }} /> ( ), }} /> ); } ``` ### CalHacks/app/(tabs)/index.tsx ```typescript import React, { useState, useEffect } from 'react'; import { StyleSheet, View, TextInput, Button, Text, ScrollView, Alert } from 'react-native'; import MapView, { Marker } from 'react-native-maps'; import MapViewDirections from 'react-native-maps-directions'; import * as Location from 'expo-location'; import axios from 'axios'; // Use Axios for sending data import { ThemedView } from '@/components/ThemedView'; const App = () => { const [location, setLocation] = useState(null); const [destination, setDestination] = useState(''); const [destinationCoords, setDestinationCoords] = useState(null); const [steps, setSteps] = useState([]); const [currentStepIndex, setCurrentStepIndex] = useState(0); const [region, setRegion] = useState({ latitude: 37.78825, longitude: -122.4324, latitudeDelta: 0.0922, longitudeDelta: 0.0421, }); useEffect(() => { (async () => { let { status } = await Location.requestForegroundPermissionsAsync(); if (status !== 'granted') { console.log('Permission to access location was denied'); return; } let location = await Location.getCurrentPositionAsync({}); setLocation(location.coords); setRegion({ ...region, latitude: location.coords.latitude, longitude: location.coords.longitude, }); })(); }, []); const handleDestinationChange = async () => { try { const response = await fetch( `https://maps.googleapis.com/maps/api/geocode/json?address=${destination}&key=${GOOGLE_MAPS_APIKEY}` ); const data = await response.json(); const location = data.results[0].geometry.location; setDestinationCoords({ latitude: location.lat, longitude: location.lng, }); const directionsResponse = await fetch( `https://maps.googleapis.com/maps/api/directions/json?origin=${region.latitude},${region.longitude}&destination=${location.lat},${location.lng}&mode=walking&key=${GOOGLE_MAPS_APIKEY}` ); const directionsData = await directionsResponse.json(); const routeSteps = directionsData.routes[0].legs[0].steps; setSteps(routeSteps); setCurrentStepIndex(0); // Reset to the first step sendToDevice(routeSteps[0]); // Send the first step to the external device } catch (error) { console.log('Error fetching directions:', error); } }; // Function to send the current step to an external device (simulated) const sendToDevice = async (step) => { try { // Simulate sending the step data (you can replace this with actual hardware logic) const stepData = { instructions: step.html_instructions.replace(/<[^>]+>/g, ''), // Remove HTML tags distance: step.distance.text, }; // Simulate sending data to an external device or server await axios.post('https://your-server.com/device-endpoint', stepData); Alert.alert('Step Sent', `Sent: ${stepData.instructions}`); } catch (error) { console.error('Error sending to device:', error); } }; useEffect(() => { const locationInterval = setInterval(async () => { if (steps.length > 0 && location) { const userLocation = await Location.getCurrentPositionAsync({}); const userCoords = { latitude: userLocation.coords.latitude, longitude: userLocation.coords.longitude, }; if (currentStepIndex < steps.length) { const nextStep = steps[currentStepIndex]; const nextStepLocation = { latitude: nextStep.end_location.lat, longitude: nextStep.end_location.lng, }; const distance = getDistance(userCoords, nextStepLocation); if (distance < 50) { const newIndex = currentStepIndex + 1; setCurrentStepIndex(newIndex); sendToDevice(steps[newIndex]); // Send the next step to the external device } } } }, 2000); return () => clearInterval(locationInterval); }, [location, steps, currentStepIndex]); const getDistance = (coords1, coords2) => { const toRad = (value) => (value * Math.PI) / 180; const R = 6371; const dLat = toRad(coords2.latitude - coords1.latitude); const dLon = toRad(coords2.longitude - coords1.longitude); const a = Math.sin(dLat / 2) * Math.sin(dLat / 2) + Math.cos(toRad(coords1.latitude)) * Math.cos(toRad(coords2.latitude)) * Math.sin(dLon / 2) * Math.sin(dLon / 2); const c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1 - a)); return R * c * 1000; // Distance in meters }; return ( {location && ( {destinationCoords && ( <> )} )}