Project Info

QNX Device Doctor

This project did not submit a demo video on Devpost.

Inspiration

Embedded devices are everywhere: cars, robots, factories, labs, medical systems, and remote infrastructure. But when something goes wrong, debugging is still very manual. Engineers often have to SSH into the device, inspect processes, read logs, diagnose failures, restart services, verify recovery, and write an incident report. We wanted to build an AI system that feels like an embedded systems engineer on call: not just a chatbot, but an agent that can observe, diagnose, plan, ask for approval, act safely, verify the fix, and report what happened. ##

What it does

QNX Device Doctor is an agentic AI SRE for QNX embedded devices. It runs against a Raspberry Pi running QNX 8.0 Non-Commercial. The QNX-side daemon starts real managed demo service processes with real PIDs and heartbeat files. The dashboard monitors those services, hardware-facing adapters, logs, diagnostics, and timeline events. The demo includes: Managed QNX demo services for sensor, telemetry, control, network, and watchdog behavior Real process-level failure detection using PIDs and heartbeats Hardware-adapter diagnostics for a sensor array and status light A natural-language "Ask Device Doctor" flow Human approval before repairs Safe allowlisted repair actions Recovery verification Incident report generation Optional GPT-4o reasoning layer for repair planning Unsafe action blocking for dangerous operations like deleting logs, rebooting, or running arbitrary shell commands One demo flow is: the user reports "my lightbulb is not working." Device Doctor inspects the control service, process health, status light adapter, desired state, actual state, power, driver responsiveness, and config validity. It diagnoses an invalid actuator configuration, proposes a safe repair, waits for approval, applies the fix, verifies that the status light is healthy again, and generates a report. ##

How we built it

We built three main components: A QNX device daemon written in Python standard library only. A plain HTML/CSS/JavaScript dashboard running on the laptop. A Device Doctor agent that performs the workflow: Observe -> Diagnose -> Plan -> Ask Approval -> Act -> Verify -> Report. The QNX daemon exposes HTTP endpoints for device state, service status, logs, timeline, diagnostics, hardware diagnostics, failure injection, repair actions, and unsafe-action blocking. The dashboard connects to the QNX daemon over HTTP and provides a polished live demo interface. The CLI agent can also run the workflow from the terminal. We also added an optional GPT-4o brain. GPT-4o can propose a structured repair plan from device evidence, but it does not execute actions directly. A deterministic safety policy validates every proposed action before anything touches the device. ## Challenges we faced We were first-time QNX users, so we intentionally avoided fragile kernel-level or hardware-specific work. We also avoided Linux-only assumptions like systemd, journalctl, Docker, and arbitrary shell execution. A major challenge was making the demo feel real while staying safe and reliable. We solved this by running real managed demo processes on QNX with real PIDs and heartbeats, while modeling hardware through safe adapters that can later be replaced with real GPIO, I2C, sensor, or actuator integrations. Another challenge was AI safety. We did not want an LLM freely executing commands on an embedded device. Our solution separates reasoning from execution: the AI can propose a plan, but only allowlisted safe tools can act. ##

What we learned

We learned how different embedded operations are from normal web/server operations. QNX does not behave like a typical Linux server environment, and embedded recovery needs to be careful, auditable, and constrained. We also learned that the best AI agent design is not "let the model do everything." It is better to give the model structured tools, safe boundaries, approval gates, and verification steps. ##

What's next

Next, we would replace the demo hardware adapters with real QNX integrations for GPIO, I2C, sensors, actuators, watchdogs, and production service logs. We would also add multi-device fleet monitoring, persistent incident history, stronger authentication, and deeper GPT-4o report generation.

QNX Device Doctor: Agentic AI SRE for Embedded Devices

An agentic AI system that diagnoses, repairs, verifies, and reports failures on QNX embedded devices.

Problem

Embedded devices are hard to debug. Engineers must manually SSH in, inspect logs, diagnose services, restart components, and verify recovery. This is slow and difficult during live incidents.

Solution

Device Doctor provides an agentic workflow:

Observe -> Diagnose -> Plan -> Act -> Verify -> Report

It monitors a QNX Raspberry Pi running managed demo service processes, injects realistic failures, safely repairs them, verifies recovery, and generates an incident report.

The dashboard also includes an "Ask Device Doctor" box for judge-friendly prompts such as "my lightbulb is not working" or "my sensor has no data." Those prompts are mapped to QNX service checks, hardware-adapter diagnostics, safe repair actions, and reports.

Why QNX

QNX is the embedded target device. The project demonstrates how an AI agent can operate against a real QNX system, inspect device health, and perform controlled recovery actions without relying on Linux-specific tools.

Architecture

Laptop Dashboard
|
v
Device Doctor Agent
|
v
QNX Device Daemon on Raspberry Pi
|
v
Managed Demo Services + Hardware Adapters
(sensor, telemetry, control, network, watchdog, sensor array, status light)

File Structure

README.md
qnx-device/
  device_daemon.py
dashboard/
  index.html
agent/
  device_doctor.py
scripts/
  deploy_to_qnx.sh
  start_remote_daemon.sh
  test_qnx.sh
  run_dashboard.sh
  run_agent.sh
docs/
  demo_script.md
  architecture.md
  pitch.md
reports/
  .gitkeep

Setup

chmod +x scripts/*.sh

Deploy to QNX:

./scripts/deploy_to_qnx.sh

Start daemon on QNX:

./scripts/start_remote_daemon.sh

Or manually:

ssh qnxuser@qnxpi21.local
cd ~/device-doctor
python3 device_daemon.py

Run dashboard:

./scripts/run_dashboard.sh

Open:

http://localhost:3000

Run tests:

./scripts/test_qnx.sh

Run agent:

python3 agent/device_doctor.py --base-url http://qnxpi21.local:8080

Optional GPT-4o planner:

OPENAI_API_KEY=sk-... python3 agent/device_doctor.py \
  --base-url http://qnxpi21.local:8080 \
  --llm-plan \
  --issue "my lightbulb is not working"

Without OPENAI_API_KEY, the GPT-4o brain prints that it is disabled and the deterministic agent remains the working fallback.

Run unsafe demo:

python3 agent/device_doctor.py --base-url http://qnxpi21.local:8080 --unsafe-demo

Use backup hotspot IP if mDNS is flaky:

BASE_URL=http://172.20.10.2:8080 ./scripts/test_qnx.sh
python3 agent/device_doctor.py --base-url http://172.20.10.2:8080

Demo Steps

Start QNX daemon.
Open dashboard.
Show all services healthy.
Click Inject Sensor Crash.
Show device becomes FAILED.
Click Run Device Doctor Agent.
Agent observes, diagnoses, plans, restarts service, verifies recovery.
Show incident report.
Type "my lightbulb is not working" into Ask Device Doctor.
Show the agent inspecting hardware-adapter evidence, fixing actuator config, and verifying recovery.
Point to the GPT-4o Brain panel and explain that GPT-4o is wired as an optional reasoning layer, while deterministic policy still validates actions.
Run Unsafe Action Demo to show dangerous action blocked.

What Actually Runs

The QNX daemon starts five real managed Python child processes on the QNX device. Each process has a real PID and emits a heartbeat file. Device Doctor watches those PIDs and heartbeats, so a service crash, stale heartbeat, and restart are real process-level events inside the demo environment.

The daemon also exposes safe hardware-adapter diagnostics for a sensor array and status light. These model evidence an embedded engineer would inspect: stale readings, actuator desired-vs-actual state, driver responsiveness, bus responsiveness, power presence, and config validity.

Some domain data is still demo-mode by design: physical sensor values, CPU pressure, disk pressure, memory pressure, and unknown-process alerts are simulated for hackathon reliability. The MVP does not control physical sensors or production QNX system services.

Optional GPT-4o Brain

The CLI includes an optional GPT-4o planner in agent/openai_brain.py. It is not required for the main demo.

When enabled with --llm-plan and OPENAI_API_KEY, GPT-4o reads the structured state and diagnostics, proposes a JSON repair plan, and the deterministic policy layer validates that plan against the action allowlist before execution.

If GPT-4o is unavailable, disabled, or proposes an invalid action, the agent falls back to deterministic planning.

Future Work

Replace simulated services with real QNX processes.
Add real sensors or GPIO.
Add actual log parsing.
Add VNC support.
Add SSH command tools.
Add integration with Gemini/OpenAI API for natural language reasoning.
Add multi-device fleet monitoring.

Analysis

Compare with all teams

View

Metric

jacobkadari-sys
1

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

HTMLIn code
PythonIn code
CSSClaimed
JavaScriptClaimed
OpenAIClaimed

2 of 5 appear in the indexed code. 3 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

120 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Feature verification

17 Verified1 Code-supported1 Blocked

Agentic Observe -> Diagnose -> Plan -> Ask Approval -> Act -> Verify -> Report workflowVerified
An agent that can observe, diagnose, plan, ask for approval, act safely, verify the fix, and report what happened
Claimed on Devposthigh confidence
- agent/device_doctor.py:251 — run_once() implements observe (GET /state), diagnose (GET /diagnostics), plan (plan_action), act (POST repair endpoint), verify (GET /state and /diagnostics again), and report (build_report/save_report).
- dashboard/index.html:697 — runDeviceDoctorAgent() implements the same observe/diagnose/plan/act/verify/report sequence in the browser, with showRepairPlan()/approveRepair() as the approval gate.
CLI agent can run the workflow from the terminalVerified
The CLI agent can also run the workflow from the terminal
Claimed on Devposthigh confidence
- agent/device_doctor.py:340 — main() with argparse (--base-url, --no-repair, --unsafe-demo, --llm-plan, --watch) runs run_once()/run loop directly from CLI.
- scripts/run_agent.sh — Referenced launcher script exists in scripts/ for running the agent.
Dashboard connects to QNX daemon over HTTP with polished live demo interfaceVerified
The dashboard connects to the QNX daemon over HTTP and provides a polished live demo interface
Claimed on Devposthigh confidence
- dashboard/index.html:260 — Dashboard is a single HTML/CSS/JS file with panels for services, hardware, ask box, GPT-4o brain, failure injection, and agent controls, calling the daemon's HTTP API via api()/injectFailure()/runDeviceDoctorAgent().
Hardware-adapter diagnostics for a sensor array and status lightVerified
Hardware-adapter diagnostics for a sensor array and status light
Claimed on Devposthigh confidence
- qnx-device/device_daemon.py:183 — baseline_hardware() models sensor_array and status_light with fields like bus_responsive, driver_responsive, power_present, config_valid.
- qnx-device/device_daemon.py:421 — hardware_diagnostics() derives problems (stale sensor, unresponsive bus, actuator mismatch) from that hardware state.
Human approval before repairsVerified
Human approval before repairs
Claimed on Devposthigh confidence
- dashboard/index.html:733 — runDeviceDoctorAgent() with requireApproval calls showRepairPlan() and stops; executeRepair() only runs after approveRepair() is clicked.
Incident report generationVerified
Incident report generation
Claimed on Devposthigh confidence
- agent/device_doctor.py:167 — build_report() composes a markdown incident report with Summary/Impact/Evidence/Root Cause/Action/Verification sections, saved via save_report() to reports/.
- reports/incident_20260621_075919.md — Multiple generated report files exist in reports/, showing the flow was actually exercised, not just coded.
Lightbulb demo flow (invalid actuator config diagnosis and repair)Verified
Device Doctor inspects the control service, process health, status light adapter, desired state, actual state, power, driver responsiveness, and config validity; diagnoses an invalid actuator configuration, proposes a safe repair, waits for approval, applies the fix, verifies recovery, and generates a report
Claimed on Devposthigh confidence
- qnx-device/device_daemon.py:437 — hardware_diagnostics() checks status_light desired_state vs status, power_present, driver_responsive, config_valid and maps to fix_hardware_config/human_hardware_check.
- dashboard/index.html:552 — askDeviceDoctor() routes 'lightbulb' text to injecting/inspecting the status_light adapter and then runs the approval-gated repair flow.
Managed QNX demo services for sensor, telemetry, control, network, watchdogVerified
Managed QNX demo services for sensor, telemetry, control, network, and watchdog behavior
Claimed on Devposthigh confidence
- qnx-device/device_daemon.py:34 — SERVICE_DEFS defines exactly these five services (sensor_service, telemetry_service, control_service, network_uplink, watchdog_service).
Natural-language 'Ask Device Doctor' flow mapping prompts to service/hardware checksVerified
A natural-language 'Ask Device Doctor' flow; prompts like 'my lightbulb is not working' are mapped to QNX service checks, hardware-adapter diagnostics, safe repair actions, and reports
Claimed on readmehigh confidence
- dashboard/index.html:547 — askDeviceDoctor() does simple keyword matching on the input text (light/bulb, sensor/temperature, network, cpu) and routes to the corresponding failure/diagnostic path and then runs the agent workflow.
Optional GPT-4o reasoning layer for repair planningVerified
Optional GPT-4o reasoning layer for repair planning
Claimed on Devposthigh confidence
- agent/openai_brain.py:73 — propose_plan() calls the OpenAI chat completions API with a system prompt constraining GPT-4o to propose one allowlisted action as JSON.
- agent/device_doctor.py:100 — maybe_llm_plan() only invokes the brain when --llm-plan is passed, and problem_from_llm_plan() validates the returned action against ALLOWED_LLM_ACTIONS before use.
QNX daemon exposes HTTP endpoints for state, services, logs, timeline, diagnostics, hardware diagnostics, failure injection, repair actions, unsafe-action blockingVerified
The QNX daemon exposes HTTP endpoints for device state, service status, logs, timeline, diagnostics, hardware diagnostics, failure injection, repair actions, and unsafe-action blocking
Claimed on Devposthigh confidence
- qnx-device/device_daemon.py:776 — do_GET handles /state, /services, /logs, /timeline, /diagnostics, /hardware, /hardware_diagnostics, /real_diagnostics.
- qnx-device/device_daemon.py:851 — do_POST handles /inject_failure, /chaos, /repair/*, /reset, /unsafe_action.
QNX-side daemon starts real managed demo service processes with real PIDs and heartbeat filesVerified
The QNX-side daemon starts real managed demo service processes with real PIDs and heartbeat files
Claimed on Devposthigh confidence
- qnx-device/device_daemon.py:80 — start_worker() spawns a real subprocess.Popen child process per service and tracks its PID in SERVICE_PROCS.
- qnx-device/device_daemon.py:48 — worker_main() is the child process entry point that writes a heartbeat file every second.
Real process-level failure detection using PIDs and heartbeatsVerified
Real process-level failure detection using PIDs and heartbeats
Claimed on Devposthigh confidence
- qnx-device/device_daemon.py:126 — sync_service_from_process() checks proc.poll() and heartbeat_age() to detect FAILED status and stale heartbeats.
Recovery verification after repairVerified
Verify the fix / recovery verification
Claimed on Devposthigh confidence
- agent/device_doctor.py:320 — After acting, the CLI agent re-fetches /state and /diagnostics and reports whether the device returned to HEALTHY.
- dashboard/index.html:680 — executeRepair() compares before/after diagnostics counts to decide RECOVERY VERIFIED vs RECOVERY FAILED.
Safe allowlisted repair actionsVerified
Safe allowlisted repair actions
Claimed on Devposthigh confidence
- qnx-device/device_daemon.py:574 — Discrete repair functions (restart_service, fix_config, reset_network, reduce_load, clear_pressure, reset_sensor_pipeline, fix_hardware_config, reset_control_service, human_hardware_check) are the only mutation endpoints exposed under /repair/*.
- agent/device_doctor.py:17 — ALLOWED_LLM_ACTIONS is a fixed allowlist checked before any LLM-proposed action is executed.
Some domain data is simulated by design (sensor values, CPU/disk/memory pressure, unknown-process alerts)Verified
Some domain data is still demo-mode by design: physical sensor values, CPU pressure, disk pressure, memory pressure, and unknown-process alerts are simulated for hackathon reliability
Claimed on readmehigh confidence
- qnx-device/device_daemon.py:519 — disk_pressure/memory_pressure/unknown_process failure injections directly set hardcoded metric values or log a fake security event rather than reading real system telemetry, matching the claimed simulated behavior.
Unsafe action blocking for dangerous operations (delete logs, reboot, arbitrary shell)Verified
Unsafe action blocking for dangerous operations like deleting logs, rebooting, or running arbitrary shell commands
Claimed on Devposthigh confidence
- qnx-device/device_daemon.py:913 — POST /unsafe_action always returns allowed=False/blocked=True and logs a security event; it never executes the requested action.
- agent/device_doctor.py:241 — run_unsafe_demo() calls /unsafe_action with delete_logs and prints that it was blocked.
Deploy-to-QNX and remote start scriptsCode-supported
Deploy to QNX via ./scripts/deploy_to_qnx.sh and start daemon on QNX via ./scripts/start_remote_daemon.sh, connecting to qnxpi21.local
Claimed on readmemedium confidence
- scripts/deploy_to_qnx.sh — Script file exists in the repo file listing alongside start_remote_daemon.sh, but I did not fully inspect their contents for correctness of the SSH/rsync deployment logic (only run_dashboard.sh was read in detail).
Running against a real Raspberry Pi with QNX 8.0 Non-CommercialBlocked
It runs against a Raspberry Pi running QNX 8.0 Non-Commercial
Claimed on Devpostlow confidence

An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.