# Project export: Jeff: the LLM-powered Drone Personal Assistant

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2024
- Tagline: Meet Jeff: the drone friend you needed, didn't deserve, but got anyway. Jeff is a versatile personal assistant drone powered by an LLM mobile-app, hovering near you to brighten your day.
- Devpost: https://devpost.com/software/jeff-the-llm-powered-drone-personal-assistant
- GitHub: https://github.com/emmethalm/treehacks24
- Demo: https://github.com/emmethalm/treehacks24/tree/master
- Video: https://www.youtube.com/embed/oR7R8wydCEQ?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 2 GitHub contributor(s) — Emmet Halm (19 commits), Derek Askaryar (3 commits)

## Devpost submission (written by the team)

### Inspiration

Drones shouldn't be impersonal, scary, and industrial. We envisioned Jeff as a friendly and creative alternative to most enterprise and defense drone applications. We believe in a future where everyone has a personal assistant and friend on call to make their lives a little easier. So why not use a drone? We believe that drone personal assistants could serve an important role for the elderly, who are often alone and at risk of falling. With Jeff hovering nearby, they can take photos hands-free, have someone to talk to, and get emergency assistance if they fall.

### What it does

Jeff is a personal assistant drone that hovers around its companion, listens, and performs tasks on command. With Jeff, you always have a camera-man at the ready to take shots from a variety of heights. Jeff can record a 3rd-person view to give content creators a novel "video-game" camera angle. Just open the app and say, "Hey Jeff, take a video of me from above." Jeff can also keep you entertained through witty comments, music, and impromptu spins.

### How we built it

We used the Parrot ANAFI AI drone and corresponding Olympe SDK to write Jeff's core functions in a Python Flask server. Then, we designed and wrote an iOS app in React Native to act as the voice-powered LLM interface to Jeff's functions. We used Convex to store and retrieve data and OpenAI function calling to interact with our Flask server. We also used OpenCV's open-source computer vision models and its built-in HOG + SVM person detection model to implement the person tracking feature.

### Challenges we ran into

As this was our first time working with drones and the Olympe SDK, we faced numerous challenges setting up our development environment, trying to properly understand and use the API for the drone, and integrating all the features on our roadmap. Additionally, deep into Saturday night, the drone mysteriously disappeared only to reappear miraculously a few hours later. Jeff is very much a proof of concept.

### Accomplishments we're proud of

We're proud that we learned a new tech stack, as none us our team had worked with drones before. We spent lots of time working with the Olympe SKD, debugging, and testing drone functions on the Parrot simulator. We're proud that we executed on our vision to ship a unique demo.

### What we learned

We learned about the challenges of writing software for drones, faced interesting network / systems issues, and how to integrate LLMs into hardware applications.

### What's next

We'd love Jeff to have a swarm of drone friends that can provide more dynamic filming, display shapes in the sky, or carry small packages for you. We'd also like to see a team expand the LLM-agent to function calling connections on physical devices.

## README (from the GitHub repository)

# Jeff: the LLM-powered Drone Personal Assistant

https://youtu.be/oR7R8wydCEQ

## Inspiration

Drones shouldn't be impersonal, scary, and industrial. We envisioned Jeff as a friendly and creative alternative to most enterprise and defense drone applications. We believe in a future where everyone has a personal assistant and friend on call to make their lives a little easier. So why not use a drone?

## What it does

Jeff is a personal assistant drone that hovers around its companion, listens, and performs tasks on command. With Jeff, you always have a camera-man at the ready to take shots from a variety of heights. Jeff can record a 3rd-person view to give content creators a novel "video-game" camera angle. Just open the app and say, "Hey Jeff, take a video of me from above."

Jeff can also keep you entertained through witty comments, music, and impromptu spins.

## How we built it

We used the Parrot ANAFI AI drone and corresponding Olympe SDK to write Jeff's core functions in a Python Flask server. Then, we designed and wrote an iOS app in React Native to act as the voice-powered LLM interface to Jeff's functions. We used Convex to store and retrieve data and OpenAI function calling to interact with our Flask server.

## Challenges we ran into

As this was our first time working with drones and the Olympe SDK, we faced numerous challenges setting up our development environment, correctly writing functions for the drone, and integrating all the features on our roadmap. Deep into Saturday night, the drone mysteriously disappeared only to reappear miraculously a few hours later. 

## Accomplishments that we're proud of

We're proud that we learned a new tech stack, as none us our team had worked with drones before. We spent lots of time working with the Olympe SKD, debugging, and testing drone functions on the Parrot simulator. We're proud that we executed on our vision to ship a unique demo.

## What we learned

We learned about the challenges of writing software for drones, faced interesting network / systems issues, and how to integrate LLMs into hardware applications.

## What's next for Jeff: the LLM-powered Drone Personal Assistant

We'd love Jeff to have a swarm of drone friends that can provide more dynamic filming, display shapes in the sky, or carry small packages for you.


## Detected evidence (automated analysis)

Indexed codebase: 25 recognized source files, 73 KB.
- C++ (language) — detected in the code
- Python (language) — detected in the code
- OpenAI (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (31 of 31)

```
controller_basics.py
controller.py
Dockerfile
features.txt
follow.py
helloDrone.py
landing.py
maxtiltget.py
mission_control.py
missions/track_mission.py
moveby.py
README.md
server.py
test_webcam_cv.py
tracking_cv.py
tracking/.clang-format
tracking/.gitignore
tracking/doc/.$state_machine.drawio.bkp
tracking/doc/.$state_machine.drawio.dtmp
tracking/mission.yaml
tracking/README.md
tracking/services/tracking_example/main.cpp
tracking/services/tracking_example/mbox_message.hpp
tracking/services/tracking_example/state_machine.cpp
tracking/services/tracking_example/state_machine.hpp
tracking/services/tracking_example/tracking_msghub.cpp
tracking/services/tracking_example/tracking_msghub.hpp
tracking/services/tracking_example/tracking_object_class.hpp
tracking/services/tracking_example/tracking_telemetry.cpp
tracking/services/tracking_example/tracking_telemetry.hpp
tracking/tracking_hud.py
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- Update README.md
- Create README.md
- /box
- radians
- movement patterns
- fix heights
- fixed ip
- cookin'
- init drone global
- low / high photos
- /track
- /takephoto endpoint
- start / stop script
- add tradkcing exmaple
- mission_controller
- add take off and landing
- formatting
- Finished draft of followMe script
- start to tracking
- Moved all functions to controller.py

## Key source files (fetched from GitHub, selected and truncated for size)

### Dockerfile

```
# Use the official Ubuntu base image
FROM ubuntu:latest

# Set environment variables (optional)
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update
# pyenv dependencies
RUN apt-get install -y make build-essential libssl-dev zlib1g-dev \
         libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev \
         libncursesw5-dev libncurses5 xz-utils tk-dev libffi-dev liblzma-dev \
         python3-openssl git libgdbm-dev libgdbm-compat-dev uuid-dev python3-gdbm \
         gawk

 # python alchemy/dragon build system dependency
RUN apt-get install -y python3
RUN apt-get install -y python3-pip

 # pdraw dependencies
RUN apt-get install -y build-essential yasm cmake libtool libc6 libc6-dev \
   unzip freeglut3-dev libglfw3 libglfw3-dev libjson-c-dev libcurl4-gnutls-dev \
   libgles2-mesa-dev

 # ffmpeg alchemy module build dependencies
RUN apt-get install -y rsync

 # Olympe / PySDL2 / pdraw renderer dependencies
RUN apt-get install -y libsdl2-dev libsdl2-2.0-0 libjpeg-dev libwebp-dev \
  libtiff5-dev libsdl2-image-dev libsdl2-image-2.0-0 libfreetype6-dev \
  libsdl2-ttf-dev libsdl2-ttf-2.0-0 libsdl2-gfx-dev

RUN apt-get install -y nano

RUN apt-get install -y iputils-ping

# Update the package lists and install any necessary packages
RUN pip3 install --no-cache-dir parrot-olympe \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Optionally, you can add additional commands to customize the container further

# Specify the default command to run when the container starts
CMD ["bash"]

```

### server.py

```python
from flask import Flask
import olympe
import os
from olympe.messages.ardrone3.Piloting import TakeOff, moveBy, Landing
from olympe.messages.ardrone3.PilotingState import FlyingStateChanged, AttitudeChanged, AltitudeAboveGroundChanged
from olympe.messages.camera2 import Command, Config, Event
from olympe.media import download_media, indexing_state, media_created
from logging import getLogger
import re
import tempfile
import xml.etree.ElementTree as ET

app = Flask(__name__)

olympe.log.update_config({
    "loggers": {
        "olympe": {"level": "INFO"},
        "photo_example": {
            "level": "INFO",
            "handlers": ["console"],
        },
    }
})

logger = getLogger("photo_example")

DRONE_IP = os.environ.get("DRONE_IP", "10.202.0.1")
DRONE_MEDIA_PORT = os.environ.get("DRONE_MEDIA_PORT", "80")

XMP_TAGS_OF_INTEREST = (
    "CameraRollDegree",
    "CameraPitchDegree",
    "CameraYawDegree",
    "CaptureTsUs",
    # NOTE: GPS metadata is only present if the drone has a GPS fix
    # (i.e. they won't be present indoor)
    "GPSLatitude",
    "GPSLongitude",
    "GPSAltitude",
)


drone = olympe.Drone(DRONE_IP)
drone.connect()

assert drone.media(
    indexing_state(state="indexed")
).wait(_timeout=60).success()

@app.route('/')
def home():
    pass

def setup_photo_burst_mode():
    # For the file_format: jpeg is the only available option
    # dng is not supported in burst mode
    drone(
        Command.Configure(
            camera_id=0,
            config=Config(
                camera_mode="photo",
                photo_mode="burst",
                photo_file_format="jpeg",
                photo_burst_value="10_over_1s",
                photo_dynamic_range="standard",
                photo_resolution="12_mega_pixels",
                photo_format=("rectilinear"),
                photo_signature="none"
            ),
            _timeout=3.0,
        )
    ).wait()

    return "Welcome to the Drone Server!"

setup_photo_burst_mode()

@app.route('/takeoff')
def takeoff():
    assert drone(
        TakeOff()
        >> FlyingStateChanged(state="hovering", _timeout=10)
        ).wait().success()
    
@app.route('/landing')
def landing():
        assert drone(Landing()).wait().success()
        drone.disconnect()
        return "The eagle has landed."

@app.route('/takephoto')
def take_photo_burst():
    media_state = drone(media_created(_timeout=3.0))
    photo_capture = drone(
        Event.Photo(
            type="stop",
            stop_reason="capture_done",
            _timeout=3.0,
            _policy="wait",
        )
        & Command.StartPhoto(camera_id=0)
    )
    if not photo_capture.wait(_timeout=30).success():
        assert False, "take_photo timedout"
    media_state.wait()
    assert media_state, media_state.explain()
    media_info = media_state.received_events().last()
    media_id = media_info.media_id
    photo_count = len(drone.media.resource_info(media_id=media_id))
    # download the photos associated with this media id to a local 'photos directory'
    photos_dir = os.path.join(os.getcwd(), 'photos')
    os.makedirs(photos_dir, exist_ok=True)
    drone.media.download_dir = photos_dir
    logger.info(
        "Download photo burst resources for media_id: {} in {}".format(
            media_id,
            drone.media.download_dir,
        )
    )
    media_download = drone(download_media(media_id, integrity_check=True))
    # Iterate over the downloaded media on the fly
    resources = media_download.as_completed(expected_count=photo_count, timeout=60)
    resource_count = 0
    for resource in resources:
        logger.info(f"Resource: {resource.resource_id}")
        if not resource.success():
            logger.error(f"Failed to download {resource.resource_id}")
            continue
        resource_count += 1
        # parse the xmp metadata
        with open(resource.download_path, "rb") as image_file:
            image_data = image_file.read()
            image_xmp_start = image_data.find(b"<x:xmpmeta")
            image_xmp_end = image_data.find(b"</x:xmpmeta")
            if image_xmp_start < 0 or image_xmp_end < 0:
                logger.error(
                    f"Failed to find XMP photo metadata {resource.resource_id}"
                )
                continue
            image_xmp = ET.fromstring(image_data[image_xmp_start: image_xmp_end + 12])
            for image_meta in image_xmp[0][0]:
                xmp_tag = re.sub(r"{[^}]*}", "", image_meta.tag)
                xmp_value = image_meta.text
                # only print the XMP tags we are interested in
                if xmp_tag in XMP_TAGS_OF_INTEREST:
                    logger.info(f"{resource.resource_id} {xmp_tag} {xmp_value}")
    logger.info(f"{resource_count} media resource downloaded")
    assert resource_count == 10, f"resource count == {resource_count} != 10"
    assert media_download.wait(1.).success(), "Photo burst media download"

@app.route('/takephoto/high')
def takephoto_high():
    assert drone(
        moveBy(0, 0, 15, 0)
        >> FlyingStateChanged(state="hovering", _timeout=10)
    ).wait().success()
    take_photo_burst()
    

@app.route('/takephoto/low')
def takephoto_low():
    assert drone(
        moveBy(0, 0, 1, 0)
        >> FlyingStateChanged(state="hovering", _timeout=10)
    ).wait().success()
    take_photo_burst()

@app.route('/spin')
def spin():
    assert drone(
        moveBy(0, 0, 0, float(3.14))
        >> FlyingStateChanged(state="hovering", _timeout=10)
    ).wait().success()
    assert drone(
        moveBy(0, 0, 0, float(3.14))
        >> FlyingStateChanged(state="hovering", _timeout=10)
    ).wait().success()

@app.route('/box')
def box():
        assert drone(
            moveBy(0, 0, 2, 0)
            >> FlyingStateChanged(state="hovering", _timeout=10)
            ).wait().success()
        assert drone(
            moveBy(0, 1, 0, 0)
            >> FlyingStateChanged(state="hovering", _timeout=10)
            ).wait().success
[truncated — 665 more characters]
```

### landing.py

```python
import olympe
import os
import time
from olympe.messages.ardrone3.Piloting import TakeOff, Landing

DRONE_IP = os.environ.get("DRONE_IP", "10.202.0.1")


def landing():
    drone = olympe.Drone(DRONE_IP)
    drone.connect()
    
    assert drone(Landing()).wait().success()
    drone.disconnect()


if __name__ == "__main__":
    landing()
```

### maxtiltget.py

```python
import olympe
import os
from olympe.messages.ardrone3.PilotingSettingsState import MaxTiltChanged

DRONE_IP = os.environ.get("DRONE_IP", "10.202.0.1")


def test_maxtiltget():
    drone = olympe.Drone(DRONE_IP)
    drone.connect()
    print("Drone MaxTilt = ", drone.get_state(MaxTiltChanged)["current"])
    drone.disconnect()


if __name__ == "__main__":
    test_maxtiltget()
```

### controller_basics.py

```python
import olympe
import os
import time
from olympe.messages.ardrone3.Piloting import TakeOff, Landing
from olympe.messages.ardrone3.PilotingState import FlyingStateChanged

DRONE_IP = os.environ.get("DRONE_IP", "10.202.0.1")

def connect_drone():
    drone = olympe.Drone(DRONE_IP)
    drone.connect()
    return drone

def takeoff(drone):
    assert drone(
        TakeOff()
        >> FlyingStateChanged(state="hovering", _timeout=10)
    ).wait().success()

def landing(drone):
    assert drone(Landing()).wait().success()
    drone.disconnect()

if __name__ == "__main__":
    drone = connect_drone()
    takeoff()
    landing()
```

### controller.py

```python
import olympe
import os
import time
from olympe.messages.ardrone3.Piloting import TakeOff, Landing
from olympe.messages.ardrone3.PilotingState import FlyingStateChanged

DRONE_IP = os.environ.get("DRONE_IP", "10.202.0.1")

def connect_drone():
    drone = olympe.Drone(DRONE_IP)
    drone.connect()
    return drone

def takeoff(drone):
    assert drone(
        TakeOff()
        >> FlyingStateChanged(state="hovering", _timeout=10)
    ).wait().success()

def landing(drone):
    assert drone(Landing()).wait().success()
    drone.disconnect()

if __name__ == "__main__":
    drone = connect_drone()
    takeoff()
    landing()
```

### moveby.py

```python
import olympe
import os
from olympe.messages.ardrone3.Piloting import TakeOff, moveBy, Landing
from olympe.messages.ardrone3.PilotingState import FlyingStateChanged

DRONE_IP = os.environ.get("DRONE_IP", "10.202.0.1")

def test_moveby2():
    drone = olympe.Drone(DRONE_IP)
    drone.connect()
    assert drone(
        TakeOff()
        >> FlyingStateChanged(state="hovering", _timeout=10)
    ).wait().success()

    # 
    
    assert drone(
        moveBy(10, 0, 0, 0)
        >> FlyingStateChanged(state="hovering", _timeout=10)
    ).wait().success()
    assert drone(Landing()).wait().success()
    drone.disconnect()


if __name__ == "__main__":
    test_moveby2()
```

### helloDrone.py

```python
import olympe
import os
from olympe.messages.ardrone3.Piloting import TakeOff, moveBy, Landing
from olympe.messages.ardrone3.PilotingState import FlyingStateChanged

DRONE_IP = os.environ.get("DRONE_IP", "10.202.0.1")

def test():
    drone = olympe.Drone(DRONE_IP)
    drone.connect()
    assert drone(
        TakeOff()
        >> FlyingStateChanged(state="hovering", _timeout=10)
    ).wait().success()
    assert drone(
        moveBy(10, 0, 0, 0)
        >> FlyingStateChanged(state="hovering", _timeout=10)
    ).wait().success()
    assert drone(
        moveBy(0, 10, 0, 0)
        >> FlyingStateChanged(state="hovering", _timeout=10)
    ).wait().success()
    assert drone(Landing()).wait().success()
    drone.disconnect()

if __name__ == "__main__":
    test()
```

### mission_control.py

```python
import olympe
from olympe.messages.mission import activate

def create_and_connect_drone(ip_address):
    drone = olympe.Drone(ip_address)
    drone.connect()
    return drone

def upload_mission(drone, mission_uid):
    assert drone(
        olympe.messages.mission.load(mission_uid, _timeout=10, _no_expect=False, _float_tol=(1e-07, 1e-09))
    ).wait().success()
    drone.disconnect()

def upload_missions(drone, mission_uids):
    for mission_uid in mission_uids:
        upload_mission(drone, mission_uid)

def start_mission_by_uid(drone, mission_uid):
    start_mission(drone, mission_uid)

def start_mission(drone, mission_uid):
    assert drone(
        activate(mission_uid, _timeout=10, _no_expect=False, _float_tol=(1e-07, 1e-09))
    ).wait().success()
    drone.disconnect()

if __name__ == "__main__":
    DRONE_IP = os.environ.get("DRONE_IP", "10.202.0.1")
    drone = create_and_connect_drone(DRONE_IP)
    mission_uids = ["mission_uid1", 
                    "mission_uid2",
                    "mission_uid3"] # Get these from our AirSDK
    upload_missions(drone, mission_uids)
    
    # Ex. Start a mission by its UID
    start_mission_by_uid(drone, "mission_uid1")

```

### test_webcam_cv.py

```python
# import the necessary packages
import numpy as np
import cv2
 
# initialize the HOG descriptor/person detector
hog = cv2.HOGDescriptor()
hog.setSVMDetector(cv2.HOGDescriptor_getDefaultPeopleDetector())

cv2.startWindowThread()

# open webcam video stream
cap = cv2.VideoCapture(0)

# the output will be written to output.avi
out = cv2.VideoWriter(
    'output.avi',
    cv2.VideoWriter_fourcc(*'MJPG'),
    15.,
    (640,480))

while(True):
    # Capture frame-by-frame
    ret, frame = cap.read()

    # resizing for faster detection
    frame = cv2.resize(frame, (640, 480))
    # using a greyscale picture, also for faster detection
    gray = cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY)

    # detect people in the image
    # returns the bounding boxes for the detected objects
    boxes, weights = hog.detectMultiScale(frame, winStride=(8,8) )

    boxes = np.array([[x, y, x + w, y + h] for (x, y, w, h) in boxes])

    for (xA, yA, xB, yB) in boxes:
        # display the detected boxes in the colour picture
        cv2.rectangle(frame, (xA, yA), (xB, yB),
                          (0, 255, 0), 2)
    
    # Write the output video 
    out.write(frame.astype('uint8'))
    # Display the resulting frame
    cv2.imshow('frame',frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# When everything done, release the capture
cap.release()
# and release the output
out.release()
# finally, close the window
cv2.destroyAllWindows()
cv2.waitKey(1)
```

[13 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]