# Project export: Meshworks - NLP LoRa Mesh Network for Emergency Response

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2024
- Tagline: Imagine an earthquake disabling all telecommunication. We combine mesh-based radio technology with cutting-edge AI processing to build a resilient information network for emergency responders.
- Devpost: https://devpost.com/software/meshworks-nlp-lora-mesh-network-for-emergency-response
- GitHub: https://github.com/AWESDUDEtheCOOL/Treehacks2024
- Video: https://www.youtube.com/embed/2GH_426YqO0?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Intel: Best Use of Intel Developer Cloud ($10k Credits + 4x Lenovo AI PC [1st] & $4k Credits [2nd] & $2k Credits [3rd]))
- Team: 3 GitHub contributor(s) — Alice Heiman (18 commits), AWESDUDEtheCOOL (10 commits), Hechen (Meimei) Liu (1 commits)

## Devpost submission (written by the team)

### Inspiration

In times of disaster, the capacity of rigid networks like cell service and internet dramatically decreases at the same time demand increases as people try to get information and contact loved ones. This can lead to crippled telecom services which can significantly impact first responders in disaster struck areas, especially in dense urban environments where traditional radios don't work well. We wanted to test newer radio and AI/ML technologies to see if we could make a better solution to this problem, which led to this project.

### What it does

Device nodes in the field network to each other and to the command node through LoRa to send messages, which helps increase the range and resiliency as more device nodes join. The command & control center is provided with summaries of reports coming from the field, which are visualized on the map.

### How we built it

We built the local devices using Wio Terminals and LoRa modules provided by Seeed Studio; we also integrated magnetometers into the devices to provide a basic sense of direction. Whisper was used for speech-to-text with Prediction Guard for summarization, keyword extraction, and command extraction, and trained a neural network on Intel Developer Cloud to perform binary image classification to distinguish damaged and undamaged buildings.

### Challenges we ran into

The limited RAM and storage of microcontrollers made it more difficult to record audio and run TinyML as we intended. Many modules, especially the LoRa and magnetometer, did not have existing libraries so these needed to be coded as well which added to the complexity of the project.

### Accomplishments we're proud of

: We wrote a library so that LoRa modules can communicate with each other across long distances We integrated Intel's optimization of AI models to make efficient, effective AI models We worked together to create something that works

### What we learned

: How to prompt AI models How to write drivers and libraries from scratch by reading datasheets How to use the Wio Terminal and the LoRa module

### What's next

We will improve the audio quality captured by the Wio Terminal and move edge-processing of the speech-to-text to increase the transmission speed and reduce bandwidth use. We will add a high-speed LoRa network to allow for faster communication between first responders in a localized area We will integrate the microcontroller and the LoRa modules onto a single board with GPS in order to improve ease of transportation and reliability

## README (from the GitHub repository)

No README available.

## Detected evidence (automated analysis)

Indexed codebase: 29 recognized source files, 56 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- Flask (technology) — claimed on Devpost, not found in the code
- PyTorch (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (77 of 77)

```
.DS_Store
.fseventsd/no_log
.gitignore
.vscode/arduino.json
.vscode/c_cpp_properties.json
.vscode/settings.json
AudioLevel/audio.ino
audiotofile/boot.py
audiotofile/code.py
audiotofile/test2.py
code.py
command.py
Command/.fseventsd/no_log
Command/.metadata_never_index
Command/.Trashes
Command/boot_out.txt
Command/boot.py
Command/code.py
Command/settings.toml
Device 1/.fseventsd/no_log
Device 1/.metadata_never_index
Device 1/.Trashes
Device 1/boot_out.txt
Device 1/boot.py
Device 1/code.py
Device 1/lib/circuitpython_adapter/__init__.py
Device 1/settings.toml
Device 2/.fseventsd/no_log
Device 2/.metadata_never_index
Device 2/.Trashes
Device 2/boot_out.txt
Device 2/boot.py
Device 2/code.py
Device 2/lib/circuitpython_adapter/__init__.py
Device 2/settings.toml
LORA.py
SensorsTest/blink.py
SensorsTest/code.py
SensorsTest/poem.txt
SpeechToText/audio.py
SpeechToText/audiooutput.txt
TreeHacks PortCase v2.step
voltagetoaudio/script.py
WebInterface/.DS_Store
WebInterface/app.py
WebInterface/audioclips/ER1.m4a
WebInterface/audioclips/ER1.txt
WebInterface/audioclips/ER2.m4a
WebInterface/audioclips/ER2.txt
WebInterface/audioclips/ER3.m4a
WebInterface/audioclips/ER3.txt
WebInterface/llmdemo.py
WebInterface/logs/demo.txt
WebInterface/logs/keywords.txt
WebInterface/logs/requests.txt
WebInterface/logs/summaries.txt
WebInterface/logs/summary.txt
WebInterface/models/.DS_Store
WebInterface/models/satclassifier.ipynb
WebInterface/models/weights.h5
WebInterface/predictiontoken.txt
WebInterface/prompts.py
WebInterface/sat.py
WebInterface/satimages/.DS_Store
WebInterface/satimages/nodamage/.DS_Store
WebInterface/scenario.py
WebInterface/static/.DS_Store
WebInterface/static/ER1.m4a
WebInterface/static/ER2.m4a
WebInterface/static/ER3.m4a
WebInterface/static/networkalert.txt
WebInterface/static/scenario.js
WebInterface/static/styles.css
WebInterface/templates/index.html
WebInterface/transcriber.py
WioAudio/code.py
WioAudio/voltage2audio.py
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- images
- Merge branch 'main' of https://github.com/AWESDUDEtheCOOL/Treehacks2024
- backup of command node firmware
- add code for microphone
- fix receiver 2 bug
- final tweaks
- Merge branch 'main' of https://github.com/AWESDUDEtheCOOL/Treehacks2024
- further embedding
- case cad model
- interface to hardware device
- Merge branch 'main' of https://github.com/AWESDUDEtheCOOL/Treehacks2024
- clear files
- Merge branch 'main' of https://github.com/AWESDUDEtheCOOL/Treehacks2024
- working command and node
- model training intel developer cloud
- add keyword and request features
- transcription and summary with preditionguard
- Merge branch 'main' of https://github.com/AWESDUDEtheCOOL/Treehacks2024
- work on response simulation
- lora tx/rx framework

## Key source files (fetched from GitHub, selected and truncated for size)

### WebInterface/app.py

```python
from flask import Flask, render_template, Response, request, send_file
import time
import os
import json
from transcriber import *
from prompts import *
from sat import *

app = Flask(__name__)

ER_LIST = [None, None, None]
ER_AUDIO_LIST = [None, None, None]


def generate_data():
    while True:
        for i in range(len(ER_LIST)):
            if ER_LIST[i]:
                data = ER_LIST[i]
                ER_LIST[i] = None
                yield "data: {}\n\n".format(data)

        time.sleep(0.1)


@app.route("/")
def index():
    return render_template("index.html")


@app.route("/gps", methods=["POST"])
def get_data():
    line = request.data.decode("utf-8")  # Decode byte string to Unicode
    print("Received from client: {}".format(line))
    if line.startswith("ER1"):
        ER_LIST[0] = line
    if line.startswith("ER2"):
        ER_LIST[1] = line
    if line.startswith("ER3"):
        ER_LIST[2] = line

    return "Data received successfully!"


@app.route("/sat")
def get_sat():
    # send dispatch message
    with open("static/networkalert.txt", "w") as f:
        f.write("TRUE")

    print("making inference...")
    confidence = make_inference("static/gps2.jpg")
    print(confidence)

    return {"result": confidence}


@app.route("/msg", methods=["POST"])
def get_msg():
    line = request.data.decode("utf-8")
    print(f"[!!!] INCOMING MESSAGE: {line}")

    return "Got message"


@app.route("/ER1", methods=["POST"])
def trigger_ER1():
    print("Setting ER1...")
    ER_AUDIO_LIST[0] = True
    return "true"


@app.route("/ER2", methods=["POST"])
def trigger_ER2():
    print("Setting ER2...")
    ER_AUDIO_LIST[1] = True
    return "true"


@app.route("/ER2")
def get_ER2_status():
    status = ER_AUDIO_LIST[1]
    ER_AUDIO_LIST[1] = None
    return {"status": status}


@app.route("/ER1")
def get_ER1_status():
    status = ER_AUDIO_LIST[0]
    ER_AUDIO_LIST[0] = None
    return {"status": status}


@app.route("/audio", methods=["POST"])
def get_audio():
    print("[!!!] INCOMING MESSAGE")
    audio_request = request.data.decode("utf-8")
    audio_request = json.loads(audio_request)

    filename = audio_request["audioclip"]
    timestamp = audio_request["timestamp"]
    responder = audio_request["responder"]
    print("Getting transcription...")
    transcription = transcribe("static/" + filename, "logs/demo.txt")

    print("Getting summary...")

    prompt = SUMMARY_PROMPT(transcription)
    result = pg.Completion.create(model="Neural-Chat-7B", prompt=prompt)
    summary = result["choices"][0]["text"]
    summary = summary.replace("\n", "")
    summary = summary.strip()

    with open("logs/summaries.txt", "a") as f:
        f.write(f"{timestamp};{responder};{summary}\n")
    print("Saved summary to logs.")

    prompt = REQUEST_PROMPT(transcription)
    result = pg.Completion.create(model="Neural-Chat-7B", prompt=prompt)
    needs = result["choices"][0]["text"]
    needs = needs.replace("\n", "")
    needs = needs.strip()
    print(f"Request: {needs}")

    if needs != "NONE":
        with open("logs/requests.txt", "a") as f:
            f.write(f"{timestamp};{responder};{needs}\n")

    prompt = KEYWORD_PROMPT(transcription)
    result = pg.Completion.create(model="Neural-Chat-7B", prompt=prompt)
    keyword = result["choices"][0]["text"]
    keyword = keyword.replace("\n", "")
    keyword = keyword.strip()
    keyword = keyword[:18]
    print(f"Keyword: {keyword}")

    with open("logs/keywords.txt", "a") as f:
        f.write(f"{timestamp};{responder};{keyword}\n")

    return "Got message"


# Route for SSE
@app.route("/stream")
def stream():
    return Response(generate_data(), mimetype="text/event-stream")


@app.route("/timeline")
def get_timeline():
    return send_file("logs/summaries.txt")


@app.route("/keywords")
def get_keywords():
    return send_file("logs/keywords.txt")


@app.route("/requests")
def get_requests():
    return send_file("logs/requests.txt")


if __name__ == "__main__":
    app.run(debug=True)

```

### command.py

```python
import serial
import time

# serial_port = 'COM19'
serial_port = "/dev/cu.usbmodem11201"
baud_rate = 115200

ser = serial.Serial(serial_port, baud_rate, timeout=1)


def send_command(command):
    ser.write((command + "\r\n").encode())


def receive_data():
    data = ser.readline().decode().strip()
    return data


while True:
    command = input("Mode (tx/rx): ").strip().lower()
    if command == "tx":
        msg = input("msg: ").strip()
        send_command(msg)
    elif command == "rx":
        print("Press Enter to stop")
        while True:
            response = receive_data()
            if response:
                print(response)
            elif input() == "":
                break
    else:
        print("Invalid command")

```

### LORA.py

```python
import board
import busio
import digitalio
import time
import supervisor

# Set UART Pin to the pin where Grove Lora E5 is connected
uart = busio.UART(board.TX, board.RX, baudrate=9600)
get_input = True
message_started = False
message_print = []
allstring = ""
printshow = False

while True:
    if supervisor.runtime.serial_bytes_available:
        allstring=""
        # wait for user input
        userinput = input().strip() #input command
        # convert to byte
        b = bytes(userinput, 'utf-8')
        # write out the byte
        uart.write(b)
        continue
    byte_read = uart.readline()# read one line
    if byte_read != None:
        allstring += byte_read.decode()
        printshow = True
    else:
        if printshow == True:
            if allstring != "":
                print(allstring)
            allstring=""
            printshow ==False
```

### code.py

```python
import time
import board
import busio
import binascii
import sys
import re
from digitalio import DigitalInOut, Direction, Pull
import pwmio


ID = "2"  # Change this to a unique ID for each device
OFF = 0
ON = 2**15

def send_command(command):
    uart0.write(command)
    time.sleep(1)  # Adjust delay as needed
    response = uart0.readline()
    if response:
        print(response.decode().strip())

def transmit_lora(packet):
    packet_hex = binascii.hexlify((f'{ID}:'+packet).encode()).decode()
    command = f'AT+TEST=TXLRPKT,"3C{packet_hex}3E"'
    send_command(bytes(command, 'utf-8'))

tr = DigitalInOut(board.BUTTON_1)
tr.direction = Direction.INPUT
tr.pull = Pull.UP

buzzer = pwmio.PWMOut(board.BUZZER, variable_frequency=True)
buzzer.frequency = 440  # set a note, like Middle C
buzzer.duty_cycle = ON  # enable the buzzer to play the note
buzzer.duty_cycle = OFF # turn off the note

uart0 = busio.UART(board.TX, board.RX, baudrate=9600)

cmds = [
        "AT+TEST=RFCFG,866,SF7,500,15,15,22,ON,OFF,OFF",
        "AT+MODE=TEST"]
for cmd in cmds:
    send_command(bytes(cmd + '\r\n', 'utf-8'))
print("LORA Configured")

allstring = ""
printshow = False

while True:
    buzzer.duty_cycle = OFF
    if not tr.value:
        buzzer.duty_cycle = ON  # enable the buzzer to play the note
        print("Transmit Mode")
        msgnum = 1

        while not tr.value:
            transmit_lora(f"Hello World #{msgnum}")
            msgnum += 1

    else:
        send_command(bytes("AT+TEST=RXLRPKT\r\n", 'utf-8'))
        print("Receive Mode")
        
        
        regex_pattern = re.compile(r'"3C(.*?)(3E)"')

        while tr.value:

            byte_read = uart0.readline()  # read up to 32 bytes
            if byte_read:
                allstring += byte_read.decode()
                printshow = True
            else:
                if printshow:
                    if allstring:
                        match = regex_pattern.search(allstring)
                        if match:
                            data_read = match.group(1)
                            clear_string = binascii.unhexlify(data_read).decode("utf8")
                            print(clear_string)

                    allstring = ""
                    printshow = False
```

### Device 1/boot.py

```python
import storage

storage.remount("/", readonly=False)

m = storage.getmount("/")
m.label = "D1"

storage.remount("/", readonly=True)

storage.enable_usb_drive()
```

### Device 2/boot.py

```python
import storage

storage.remount("/", readonly=False)

m = storage.getmount("/")
m.label = "D2"

storage.remount("/", readonly=True)

storage.enable_usb_drive()
```

### Command/boot.py

```python
import storage

storage.remount("/", readonly=False)

m = storage.getmount("/")
m.label = "COMMAND"

storage.remount("/", readonly=True)

storage.enable_usb_drive()
```

### voltagetoaudio/script.py

```python
from __future__ import print_function
import scipy.io.wavfile as wavf
import numpy as np

if __name__ == "__main__":

    samples = np.random.randn(44100)
    fs = 44100
    out_f = "out.wav"

    wavf.write(out_f, fs, samples)

```

### SensorsTest/blink.py

```python
import time
import board
from analogio import AnalogIn

analog_in = AnalogIn(board.LIGHT)  # Light Sensor pin on Wio Terminal


def get_voltage(pin):
    return (pin.value * 3.3) / 65536


while True:
    print("Light Sensor Voltage: ", get_voltage(analog_in))
    time.sleep(0.1)

```

### audiotofile/boot.py

```python
import board
import digitalio
import storage

print("hi there")
switch = digitalio.DigitalInOut(board.D4)
switch.direction = digitalio.Direction.INPUT
# switch.pull = digitalio.Pull.UP
print(switch.value) 

# If the D0 is connected to ground with a wire
# CircuitPython can write to the drive
storage.remount("/", readonly=switch.value)

# for some reason this is reversed: readonly=True means not readonly
```

[19 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]