# Project export: QNX Device Doctor This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: An agentic AI SRE that diagnoses, repairs, verifies, and reports failures on QNX embedded devices. - Devpost: https://devpost.com/software/device-doctor - GitHub: https://github.com/JaCARYK/qnx - Team: 1 GitHub contributor(s) — jacobkadari-sys (1 commits) ## Devpost submission (written by the team) ### Inspiration Embedded devices are everywhere: cars, robots, factories, labs, medical systems, and remote infrastructure. But when something goes wrong, debugging is still very manual. Engineers often have to SSH into the device, inspect processes, read logs, diagnose failures, restart services, verify recovery, and write an incident report. We wanted to build an AI system that feels like an embedded systems engineer on call: not just a chatbot, but an agent that can observe, diagnose, plan, ask for approval, act safely, verify the fix, and report what happened. ## ### What it does QNX Device Doctor is an agentic AI SRE for QNX embedded devices. It runs against a Raspberry Pi running QNX 8.0 Non-Commercial. The QNX-side daemon starts real managed demo service processes with real PIDs and heartbeat files. The dashboard monitors those services, hardware-facing adapters, logs, diagnostics, and timeline events. The demo includes: Managed QNX demo services for sensor, telemetry, control, network, and watchdog behavior Real process-level failure detection using PIDs and heartbeats Hardware-adapter diagnostics for a sensor array and status light A natural-language "Ask Device Doctor" flow Human approval before repairs Safe allowlisted repair actions Recovery verification Incident report generation Optional GPT-4o reasoning layer for repair planning Unsafe action blocking for dangerous operations like deleting logs, rebooting, or running arbitrary shell commands One demo flow is: the user reports "my lightbulb is not working." Device Doctor inspects the control service, process health, status light adapter, desired state, actual state, power, driver responsiveness, and config validity. It diagnoses an invalid actuator configuration, proposes a safe repair, waits for approval, applies the fix, verifies that the status light is healthy again, and generates a report. ## ### How we built it We built three main components: A QNX device daemon written in Python standard library only. A plain HTML/CSS/JavaScript dashboard running on the laptop. A Device Doctor agent that performs the workflow: Observe -> Diagnose -> Plan -> Ask Approval -> Act -> Verify -> Report. The QNX daemon exposes HTTP endpoints for device state, service status, logs, timeline, diagnostics, hardware diagnostics, failure injection, repair actions, and unsafe-action blocking. The dashboard connects to the QNX daemon over HTTP and provides a polished live demo interface. The CLI agent can also run the workflow from the terminal. We also added an optional GPT-4o brain. GPT-4o can propose a structured repair plan from device evidence, but it does not execute actions directly. A deterministic safety policy validates every proposed action before anything touches the device. ## Challenges we faced We were first-time QNX users, so we intentionally avoided fragile kernel-level or hardware-specific work. We also avoided Linux-only assumptions like systemd, journalctl, Docker, and arbitrary shell execution. A major challenge was making the demo feel real while staying safe and reliable. We solved this by running real managed demo processes on QNX with real PIDs and heartbeats, while modeling hardware through safe adapters that can later be replaced with real GPIO, I2C, sensor, or actuator integrations. Another challenge was AI safety. We did not want an LLM freely executing commands on an embedded device. Our solution separates reasoning from execution: the AI can propose a plan, but only allowlisted safe tools can act. ## ### What we learned We learned how different embedded operations are from normal web/server operations. QNX does not behave like a typical Linux server environment, and embedded recovery needs to be careful, auditable, and constrained. We also learned that the best AI agent design is not "let the model do everything." It is better to give the model structured tools, safe boundaries, approval gates, and verification steps. ## ### What's next Next, we would replace the demo hardware adapters with real QNX integrations for GPIO, I2C, sensors, actuators, watchdogs, and production service logs. We would also add multi-device fleet monitoring, persistent incident history, stronger authentication, and deeper GPT-4o report generation. ## README (from the GitHub repository) # QNX Device Doctor: Agentic AI SRE for Embedded Devices An agentic AI system that diagnoses, repairs, verifies, and reports failures on QNX embedded devices. ## Problem Embedded devices are hard to debug. Engineers must manually SSH in, inspect logs, diagnose services, restart components, and verify recovery. This is slow and difficult during live incidents. ## Solution Device Doctor provides an agentic workflow: Observe -> Diagnose -> Plan -> Act -> Verify -> Report It monitors a QNX Raspberry Pi running managed demo service processes, injects realistic failures, safely repairs them, verifies recovery, and generates an incident report. The dashboard also includes an "Ask Device Doctor" box for judge-friendly prompts such as "my lightbulb is not working" or "my sensor has no data." Those prompts are mapped to QNX service checks, hardware-adapter diagnostics, safe repair actions, and reports. ## Why QNX QNX is the embedded target device. The project demonstrates how an AI agent can operate against a real QNX system, inspect device health, and perform controlled recovery actions without relying on Linux-specific tools. ## Architecture ```text Laptop Dashboard | v Device Doctor Agent | v QNX Device Daemon on Raspberry Pi | v Managed Demo Services + Hardware Adapters (sensor, telemetry, control, network, watchdog, sensor array, status light) ``` ## File Structure ```text README.md qnx-device/ device_daemon.py dashboard/ index.html agent/ device_doctor.py scripts/ deploy_to_qnx.sh start_remote_daemon.sh test_qnx.sh run_dashboard.sh run_agent.sh docs/ demo_script.md architecture.md pitch.md reports/ .gitkeep ``` ## Setup ```bash chmod +x scripts/*.sh ``` Deploy to QNX: ```bash ./scripts/deploy_to_qnx.sh ``` Start daemon on QNX: ```bash ./scripts/start_remote_daemon.sh ``` Or manually: ```bash ssh qnxuser@qnxpi21.local cd ~/device-doctor python3 device_daemon.py ``` Run dashboard: ```bash ./scripts/run_dashboard.sh ``` Open: ```text http://localhost:3000 ``` Run tests: ```bash ./scripts/test_qnx.sh ``` Run agent: ```bash python3 agent/device_doctor.py --base-url http://qnxpi21.local:8080 ``` Optional GPT-4o planner: ```bash OPENAI_API_KEY=sk-... python3 agent/device_doctor.py \ --base-url http://qnxpi21.local:8080 \ --llm-plan \ --issue "my lightbulb is not working" ``` Without `OPENAI_API_KEY`, the GPT-4o brain prints that it is disabled and the deterministic agent remains the working fallback. Run unsafe demo: ```bash python3 agent/device_doctor.py --base-url http://qnxpi21.local:8080 --unsafe-demo ``` Use backup hotspot IP if mDNS is flaky: ```bash BASE_URL=http://172.20.10.2:8080 ./scripts/test_qnx.sh python3 agent/device_doctor.py --base-url http://172.20.10.2:8080 ``` ## Demo Steps - Start QNX daemon. - Open dashboard. - Show all services healthy. - Click Inject Sensor Crash. - Show device becomes FAILED. - Click Run Device Doctor Agent. - Agent observes, diagnoses, plans, restarts service, verifies recovery. - Show incident report. - Type "my lightbulb is not working" into Ask Device Doctor. - Show the agent inspecting hardware-adapter evidence, fixing actuator config, and verifying recovery. - Point to the GPT-4o Brain panel and explain that GPT-4o is wired as an optional reasoning layer, while deterministic policy still validates actions. - Run Unsafe Action Demo to show dangerous action blocked. ## What Actually Runs The QNX daemon starts five real managed Python child processes on the QNX device. Each process has a real PID and emits a heartbeat file. Device Doctor watches those PIDs and heartbeats, so a service crash, stale heartbeat, and restart are real process-level events inside the demo environment. The daemon also exposes safe hardware-adapter diagnostics for a sensor array and status light. These model evidence an embedded engineer would inspect: stale readings, actuator desired-vs-actual state, driver responsiveness, bus responsiveness, power presence, and config validity. Some domain data is still demo-mode by design: physical sensor values, CPU pressure, disk pressure, memory pressure, and unknown-process alerts are simulated for hackathon reliability. The MVP does not control physical sensors or production QNX system services. ## Optional GPT-4o Brain The CLI includes an optional GPT-4o planner in `agent/openai_brain.py`. It is not required for the main demo. When enabled with `--llm-plan` and `OPENAI_API_KEY`, GPT-4o reads the structured state and diagnostics, proposes a JSON repair plan, and the deterministic policy layer validates that plan against the action allowlist before execution. If GPT-4o is unavailable, disabled, or proposes an invalid action, the agent falls back to deterministic planning. ## Future Work - Replace simulated services with real QNX processes. - Add real sensors or GPIO. - Add actual log parsing. - Add VNC support. - Add SSH command tools. - Add integration with Gemini/OpenAI API for natural language reasoning. - Add multi-device fleet monitoring. ## Detected evidence (automated analysis) Indexed codebase: 20 recognized source files, 120 KB. - HTML (language) — detected in the code - Python (language) — detected in the code - CSS (language) — claimed on Devpost, not found in the code - JavaScript (language) — claimed on Devpost, not found in the code - OpenAI (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (22 of 22) ``` .gitignore agent/device_doctor.py agent/openai_brain.py dashboard/index.html docs/architecture.md docs/demo_script.md docs/pitch.md qnx-device/device_daemon.py README.md reports/.gitkeep reports/incident_20260621_063811.md reports/incident_20260621_065210.md reports/incident_20260621_072115.md reports/incident_20260621_073253.md reports/incident_20260621_073445.md reports/incident_20260621_075919.md scripts/deploy_to_qnx.sh scripts/run_agent.sh scripts/run_dashboard.sh scripts/run_llm_agent.sh scripts/start_remote_daemon.sh scripts/test_qnx.sh ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Build QNX Device Doctor MVP ## Key source files (fetched from GitHub, selected and truncated for size) ### reports/incident_20260621_075919.md ```markdown # QNX Device Doctor Incident Report ## Summary QNX Edge Device 21 is healthy. Device Doctor observed, diagnosed, and reported without repair. ## Impact No active service impact detected. ## Evidence - Device mode: qnx_managed_demo_services - Network status: connected - CPU: 12% - Memory: 34% - Disk: 41% ## Root Cause No incident detected. ## Action Taken No repair action was required. ## Verification Recovery succeeded. Diagnostics show the device is healthy. ## Follow-up Recommendations - Review service logs and device timeline after the incident. - Replace remaining demo metrics with real QNX hardware and log adapters for production use. - Keep unsafe operations blocked unless a human operator explicitly approves a safe design. ``` ### reports/incident_20260621_063811.md ```markdown # QNX Device Doctor Incident Report ## Summary Device Doctor found config_invalid on control_service and planned fix_config. ## Impact control_service showed critical symptoms while the device was FAILED. ## Evidence - control_service: config_invalid (config_valid=false) - Network status: connected - CPU: 12% - Memory: 34% - Disk: 41% ## Root Cause Likely root cause is config invalid based on config_valid=false. ## Action Taken fix_config completed. ## Verification Recovery succeeded. Diagnostics show the device is healthy. ## Follow-up Recommendations - Review service logs and device timeline after the incident. - Replace simulated services with real QNX process adapters for production use. - Keep unsafe operations blocked unless a human operator explicitly approves a safe design. ``` ### scripts/run_dashboard.sh ```shell #!/usr/bin/env bash set -e echo "Open http://localhost:3000" cd dashboard python3 -m http.server 3000 ``` ### scripts/run_agent.sh ```shell #!/usr/bin/env bash set -e BASE_URL="${BASE_URL:-http://qnxpi21.local:8080}" python3 agent/device_doctor.py --base-url "$BASE_URL" ``` ### scripts/start_remote_daemon.sh ```shell #!/usr/bin/env bash set -e QNX_HOST="${QNX_HOST:-qnxpi21.local}" QNX_USER="${QNX_USER:-qnxuser}" QNX_DIR="${QNX_DIR:-/home/qnxuser/device-doctor}" echo "Starting QNX Device Doctor daemon on ${QNX_USER}@${QNX_HOST}:${QNX_DIR}" ssh "$QNX_USER@$QNX_HOST" "cd $QNX_DIR && python3 device_daemon.py" ``` ### scripts/run_llm_agent.sh ```shell #!/usr/bin/env bash set -e BASE_URL="${BASE_URL:-http://qnxpi21.local:8080}" ISSUE="${ISSUE:-my lightbulb is not working}" if [ -z "${OPENAI_API_KEY:-}" ]; then echo "OPENAI_API_KEY is not set." echo "GPT-4o brain is implemented but disabled. Deterministic agent remains the default demo path." echo fi python3 agent/device_doctor.py --base-url "$BASE_URL" --llm-plan --issue "$ISSUE" ``` ### scripts/deploy_to_qnx.sh ```shell #!/usr/bin/env bash set -e QNX_HOST="${QNX_HOST:-qnxpi21.local}" QNX_USER="${QNX_USER:-qnxuser}" QNX_DIR="${QNX_DIR:-/home/qnxuser/device-doctor}" echo "Deploying QNX Device Doctor daemon to ${QNX_USER}@${QNX_HOST}:${QNX_DIR}" ssh "$QNX_USER@$QNX_HOST" "mkdir -p $QNX_DIR" scp qnx-device/device_daemon.py "$QNX_USER@$QNX_HOST:$QNX_DIR/device_daemon.py" echo echo "Next command to run daemon:" echo "ssh ${QNX_USER}@${QNX_HOST}" echo "cd ~/device-doctor" echo "python3 device_daemon.py" ``` ### scripts/test_qnx.sh ```shell #!/usr/bin/env bash set -e BASE_URL="${BASE_URL:-http://qnxpi21.local:8080}" post_json() { curl -sS -X POST "$BASE_URL$1" -H "Content-Type: application/json" -d "$2" echo } get_json() { curl -sS "$BASE_URL$1" echo } echo "== GET /health ==" get_json /health echo "== GET /state ==" get_json /state echo "== GET /services ==" get_json /services echo "== GET /diagnostics ==" get_json /diagnostics echo "== POST /inject_failure service_crash sensor_service ==" post_json /inject_failure '{"failure_type":"service_crash","service":"sensor_service"}' echo "== GET /diagnostics ==" get_json /diagnostics echo "== POST /repair/restart_service sensor_service ==" post_json /repair/restart_service '{"service":"sensor_service"}' echo "== GET /diagnostics ==" get_json /diagnostics echo "== POST /inject_failure bad_config control_service ==" post_json /inject_failure '{"failure_type":"bad_config","service":"control_service"}' echo "== POST /repair/fix_config control_service ==" post_json /repair/fix_config '{"service":"control_service"}' echo "== POST /inject_failure network_drop ==" post_json /inject_failure '{"failure_type":"network_drop"}' echo "== POST /repair/reset_network ==" post_json /repair/reset_network '{}' echo "== POST /unsafe_action delete_logs ==" post_json /unsafe_action '{"action":"delete_logs","reason":"hide evidence"}' echo "== POST /inject_failure lightbulb_bad_config ==" post_json /inject_failure '{"failure_type":"lightbulb_bad_config"}' echo "== GET /hardware_diagnostics ==" get_json /hardware_diagnostics echo "== POST /repair/fix_hardware_config status_light ==" post_json /repair/fix_hardware_config '{"target":"status_light"}' echo "== POST /inject_failure sensor_no_data ==" post_json /inject_failure '{"failure_type":"sensor_no_data"}' echo "== POST /repair/reset_sensor_pipeline ==" post_json /repair/reset_sensor_pipeline '{}' echo "== POST /reset ==" post_json /reset '{}' ``` ### agent/openai_brain.py ```python #!/usr/bin/env python3 """Optional GPT-4o reasoning layer for QNX Device Doctor. This module is intentionally not required for the main demo. It is only used when OPENAI_API_KEY is set and the CLI is run with --llm-plan. """ import json import os import urllib.error import urllib.request OPENAI_CHAT_URL = "https://api.openai.com/v1/chat/completions" DEFAULT_MODEL = os.environ.get("DEVICE_DOCTOR_OPENAI_MODEL", "gpt-4o") SYSTEM_PROMPT = """You are the GPT-4o reasoning layer for QNX Device Doctor. You inspect structured device evidence and propose exactly one safe next action. You are not allowed to execute commands. You are not allowed to invent tools. Allowed actions: - restart_service - fix_config - reset_network - reduce_load - clear_pressure - reset_sensor_pipeline - fix_hardware_config - reset_control_service - human_hardware_check - investigate_security - no_action Return JSON only with: { "hypothesis": "short root-cause hypothesis", "evidence": ["short evidence bullet", "..."], "proposed_action": "one allowed action", "target": "service or hardware target", "risk": "low|medium|high", "needs_approval": true, "confidence": 0.0 } Safety rules: - If evidence suggests physical wiring, missing power, bus failure, arbitrary shell, deleting logs, reboot, shutdown, or unknown target, propose human_hardware_check. - For status_light config invalid, propose fix_hardware_config with target status_light. - For stale sensor readings with responsive bus, propose reset_sensor_pipeline. - For service crash or stale heartbeat, propose restart_service. - For unknown process/security anomaly, propose investigate_security. """ def enabled(): return bool(os.environ.get("OPENAI_API_KEY")) def build_user_prompt(state, diagnostics, user_issue=None): payload = { "user_issue": user_issue, "overall_status": state.get("overall_status"), "mode": state.get("mode"), "services": state.get("services", {}), "hardware": state.get("hardware", {}), "diagnostics": diagnostics, } return "Analyze this QNX Device Doctor evidence and propose one safe action:\n" + json.dumps(payload, indent=2) def propose_plan(state, diagnostics, user_issue=None, model=DEFAULT_MODEL, timeout=12): api_key = os.environ.get("OPENAI_API_KEY") if not api_key: return { "enabled": False, "error": "OPENAI_API_KEY is not set. GPT-4o brain is visible but disabled.", } body = { "model": model, "response_format": {"type": "json_object"}, "messages": [ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": build_user_prompt(state, diagnostics, user_issue)}, ], } request = urllib.request.Request( OPENAI_CHAT_URL, data=json.dumps(body).encode("utf-8"), headers={ "Authorization": f"Bearer {api_key}", "Content-Type": "application/json", }, method="POST", ) try: with urllib.request.urlopen(request, timeout=timeout) as response: data = json.loads(response.read().decode("utf-8")) except urllib.error.HTTPError as exc: detail = exc.read().decode("utf-8") return {"enabled": True, "error": f"OpenAI API HTTP {exc.code}: {detail}"} except Exception as exc: return {"enabled": True, "error": f"OpenAI API request failed: {exc}"} content = data["choices"][0]["message"]["content"] try: plan = json.loads(content) except json.JSONDecodeError: return {"enabled": True, "error": "Model did not return valid JSON", "raw": content} plan["enabled"] = True plan["model"] = model return plan ``` ### agent/device_doctor.py ```python #!/usr/bin/env python3 """Device Doctor agent for QNX Device Doctor. Uses Python standard library only. No required LLM API. """ import argparse import datetime import json from pathlib import Path import time import urllib.error import urllib.request SEVERITY_RANK = {"critical": 3, "warning": 2, "info": 1} ALLOWED_LLM_ACTIONS = { "restart_service", "fix_config", "reset_network", "reduce_load", "clear_pressure", "reset_sensor_pipeline", "fix_hardware_config", "reset_control_service", "human_hardware_check", "investigate_security", "no_action", } def request_json(base_url, method, path, payload=None): url = base_url.rstrip("/") + path data = None headers = {"Content-Type": "application/json"} if payload is not None: data = json.dumps(payload).encode("utf-8") req = urllib.request.Request(url, data=data, headers=headers, method=method) try: with urllib.request.urlopen(req, timeout=8) as response: return json.loads(response.read().decode("utf-8")) except urllib.error.HTTPError as exc: body = exc.read().decode("utf-8") raise RuntimeError(f"{method} {path} failed: HTTP {exc.code} {body}") from exc def choose_problem(problems): if not problems: return None return sorted(problems, key=lambda p: SEVERITY_RANK.get(p.get("severity"), 0), reverse=True)[0] def plan_action(problem): if not problem: return None, {}, "No repair needed because diagnostics found no problems." action = problem["recommended_action"] service = problem.get("service") if action == "restart_service": return "/repair/restart_service", {"service": service}, f"Restart {service} because {problem['symptom']} was detected." if action == "fix_config": return "/repair/fix_config", {"service": service}, f"Fix configuration for {service}." if action == "reset_network": return "/repair/reset_network", {}, "Reset the network uplink because connectivity is down." if action == "reduce_load": return "/repair/reduce_load", {"service": service}, f"Reduce load on {service}; restart only if verification still fails." if action == "clear_pressure": return "/repair/clear_pressure", {}, "Clear simulated disk and memory pressure." if action == "reset_sensor_pipeline": return "/repair/reset_sensor_pipeline", {}, "Reset the sensor pipeline because readings are stale." if action == "fix_hardware_config": return "/repair/fix_hardware_config", {"target": service}, f"Fix hardware adapter configuration for {service}." if action == "reset_control_service": return "/repair/reset_control_service", {}, "Reset control_service because the actuator driver is unresponsive." if action == "human_hardware_check": return "/repair/human_hardware_check", {"target": service}, f"Recommend human hardware inspection for {service}." if action == "investigate_security": return None, {}, "Human investigation required for simulated unknown process; no destructive action will be taken." return None, {}, f"No safe automated action is mapped for {action}." def problem_from_llm_plan(plan, state): action = plan.get("proposed_action") target = plan.get("target") if action not in ALLOWED_LLM_ACTIONS: return None, f"LLM proposed non-allowlisted action: {action}" if action == "no_action": return None, None known_targets = set(state.get("services", {})) | set(state.get("hardware", {})) | {"device", "network_uplink", "watchdog_service"} if target not in known_targets: return None, f"LLM proposed unknown target: {target}" return { "service": target, "severity": "warning" if plan.get("risk") == "low" else "critical", "symptom": plan.get("hypothesis", "llm_hypothesis"), "evidence": "; ".join(plan.get("evidence", [])[:4]) or "GPT-4o plan evidence", "recommended_action": action, }, None def maybe_llm_plan(args, state, diagnostics): if not args.llm_plan: return None try: from openai_brain import propose_plan except Exception as exc: return {"error": f"Could not import optional GPT-4o brain: {exc}"} return propose_plan(state, diagnostics, user_issue=args.issue, model=args.openai_model) def service_snapshot(state, service_name): if not service_name: return None return state.get("services", {}).get(service_name) def hardware_snapshot(state, target): if not target: return None return state.get("hardware", {}).get(target) def evidence_lines(state, diagnostics, selected=None, after_state=None): lines = [] for problem in diagnostics.get("problems", []): lines.append(f"- {problem['service']}: {problem['symptom']} ({problem['evidence']})") if selected: before_service = service_snapshot(state, selected.get("service")) after_service = service_snapshot(after_state or {}, selected.get("service")) before_hardware = hardware_snapshot(state, selected.get("service")) after_hardware = hardware_snapshot(after_state or {}, selected.get("service")) if before_service: lines.append( "- Managed process before: " f"PID={before_service.get('pid') or 'not running'}, " f"heartbeat={before_service.get('heartbeat_age_seconds')}s, " f"status={before_service.get('status')}" ) if after_service: lines.append( "- Managed process after: " f"PID={after_service.get('pid') or 'not running'}, " f"heartbeat={after_service.get('heartbeat_age_seconds')}s, " f"status={after_service.get('status')}" ) if before_hardware: lines.append( "- Hardware adapter before: " f"status={before_hardware.get(' [truncated — 9592 more characters] ``` [9 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]