HackStack

Project Info

Jeff: the LLM-powered Drone Personal Assistant

Devpost

Inspiration

Drones shouldn't be impersonal, scary, and industrial. We envisioned Jeff as a friendly and creative alternative to most enterprise and defense drone applications. We believe in a future where everyone has a personal assistant and friend on call to make their lives a little easier. So why not use a drone? We believe that drone personal assistants could serve an important role for the elderly, who are often alone and at risk of falling. With Jeff hovering nearby, they can take photos hands-free, have someone to talk to, and get emergency assistance if they fall.

What it does

Jeff is a personal assistant drone that hovers around its companion, listens, and performs tasks on command. With Jeff, you always have a camera-man at the ready to take shots from a variety of heights. Jeff can record a 3rd-person view to give content creators a novel "video-game" camera angle. Just open the app and say, "Hey Jeff, take a video of me from above." Jeff can also keep you entertained through witty comments, music, and impromptu spins.

How we built it

We used the Parrot ANAFI AI drone and corresponding Olympe SDK to write Jeff's core functions in a Python Flask server. Then, we designed and wrote an iOS app in React Native to act as the voice-powered LLM interface to Jeff's functions. We used Convex to store and retrieve data and OpenAI function calling to interact with our Flask server. We also used OpenCV's open-source computer vision models and its built-in HOG + SVM person detection model to implement the person tracking feature.

Challenges we ran into

As this was our first time working with drones and the Olympe SDK, we faced numerous challenges setting up our development environment, trying to properly understand and use the API for the drone, and integrating all the features on our roadmap. Additionally, deep into Saturday night, the drone mysteriously disappeared only to reappear miraculously a few hours later. Jeff is very much a proof of concept.

Accomplishments we're proud of

We're proud that we learned a new tech stack, as none us our team had worked with drones before. We spent lots of time working with the Olympe SKD, debugging, and testing drone functions on the Parrot simulator. We're proud that we executed on our vision to ship a unique demo.

What we learned

We learned about the challenges of writing software for drones, faced interesting network / systems issues, and how to integrate LLMs into hardware applications.

What's next

We'd love Jeff to have a swarm of drone friends that can provide more dynamic filming, display shapes in the sky, or carry small packages for you. We'd also like to see a team expand the LLM-agent to function calling connections on physical devices.

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only
  • C++In code
  • PythonIn code
  • OpenAIClaimed

2 of 3 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

73 KB

Source files

25

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

0 stars