# Project export: Stable-RL This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: Offline RL is historically bad out of distribution. We created an OSS to fix this using conformal prediction. - Devpost: https://devpost.com/software/stable-rl - GitHub: https://github.com/khushgx/cql - Demo: https://v0-stable-rl-spncmb.vercel.app/ - Team: 1 GitHub contributor(s) — Khush Gupta (5 commits) ## Devpost submission (written by the team) ### Inspiration Reinforcement Learning (RL) has achieved significant success in decision-making tasks but struggles with real-world applications where interaction with the environment is expensive or dangerous. Offline RL attempts to address this by learning from fixed datasets, but suffers from Q-value overestimation on out-of-distribution (OOD) state-action pairs. The Conformal Q-Learning approach seeks to mitigate these issues by integrating conformal prediction into RL. This method provides distribution-free uncertainty quantification with finite-sample guarantees, helping to stabilize policy learning and prevent overestimation errors. The approach is particularly relevant for safety-critical applications where robust decision-making is necessary. ### What it does Conformal Q-Learning enhances standard RL algorithms by introducing statistical confidence intervals around Q-value estimates. It ensures that: Learned Q-values remain within prediction intervals with high probability. Uncertainty quantification helps mitigate overestimation and unsafe policy decisions. Policies become more stable and robust to OOD state-action pairs. It improves conservatism and optimism balance, performing better than traditional Conservative Q-Learning (CQL) approaches. ### How we built it The Conformal Q-Learning framework was developed by integrating conformal prediction into an actor-critic RL setup, specifically for offline RL. The methodology includes: Q-Network Training: A deep Q-network is trained to estimate Q-values based on historical data. Conformal Interval Calibration: During training, nonconformity scores are computed using a calibration set to construct prediction intervals. Actor-Critic Framework: The actor network (policy) is updated to maximize Q-values, incorporating the uncertainty information from conformal intervals. Empirical evaluations were conducted on CartPole-v1 using offline RL datasets, validating the effectiveness of Conformal Q-Learning. ### Challenges we ran into Handling OOD Actions – Offline RL suffers from extrapolation errors, and ensuring robust Q-value estimates in unseen states was non-trivial. Balancing Conservatism & Optimism – Unlike CQL, which applies a fixed penalty, conformal prediction needed fine-tuned quantile selection. Computational Constraints – Ensuring that conformal interval calibration remains computationally feasible without excessive overhead. Stability in Training – Traditional RL algorithms can suffer from instability, particularly when using confidence intervals in decision-making. ### Accomplishments we're proud of Empirical Success: Demonstrated improved policy stability, robustness to OOD data, and enhanced performance compared to CQL and standard DQN. First-of-its-kind integration of conformal prediction into Q-learning for uncertainty quantification in RL. ### What we learned Uncertainty estimation is critical in RL, particularly for offline settings where exploration is limited. Accelerating RL training with PyTorch optimizations: Leveraging techniques such as torch.jit for just-in-time (JIT) compilation, CUDA acceleration, and efficient tensor operations can significantly speed up Q-network training and conformal interval calculations, making the approach more scalable for real-world applications. ### What's next We are looking to expand the amount of algorithms that we can integrate this in. Currently, we had time to implement this idea with Soft-Actor Critic, but integrating this uncertainty quantification into more popular algorithms is our next goal. Additionally testing its robustness on edge-based scenarios, like autonomous driving, is a plausible next step as our motivation was centered around improving robustness in these areas ## README (from the GitHub repository) ## Inspiration Reinforcement Learning (RL) has achieved significant success in decision-making tasks but struggles with real-world applications where interaction with the environment is expensive or dangerous. Offline RL attempts to address this by learning from fixed datasets, but suffers from Q-value overestimation on out-of-distribution (OOD) state-action pairs. The Conformal Q-Learning approach seeks to mitigate these issues by integrating conformal prediction into RL. This method provides distribution-free uncertainty quantification with finite-sample guarantees, helping to stabilize policy learning and prevent overestimation errors. The approach is particularly relevant for safety-critical applications where robust decision-making is necessary. ## What it does Conformal Q-Learning enhances standard RL algorithms by introducing statistical confidence intervals around Q-value estimates. It ensures that: - Learned Q-values remain within prediction intervals with high probability. - Uncertainty quantification helps mitigate overestimation and unsafe policy decisions. - Policies become more stable and robust to OOD state-action pairs. - It improves conservatism and optimism balance, performing better than traditional Conservative Q-Learning (CQL) approaches. ## How we built it The Conformal Q-Learning framework was developed by integrating conformal prediction into an actor-critic RL setup, specifically for offline RL. The methodology includes: - Q-Network Training: A deep Q-network is trained to estimate Q-values based on historical data. - Conformal Interval Calibration: During training, nonconformity scores are computed using a calibration set to construct prediction intervals. - Actor-Critic Framework: The actor network (policy) is updated to maximize Q-values, incorporating the uncertainty information from conformal intervals. - Empirical evaluations were conducted on CartPole-v1 using offline RL datasets, validating the effectiveness of Conformal Q-Learning. ## Challenges we ran into 1. Handling OOD Actions – Offline RL suffers from extrapolation errors, and ensuring robust Q-value estimates in unseen states was non-trivial. 2. Balancing Conservatism & Optimism – Unlike CQL, which applies a fixed penalty, conformal prediction needed fine-tuned quantile selection. 3. Computational Constraints – Ensuring that conformal interval calibration remains computationally feasible without excessive overhead. 4. Stability in Training – Traditional RL algorithms can suffer from instability, particularly when using confidence intervals in decision-making. ## Accomplishments that we're proud of - Empirical Success: Demonstrated improved policy stability, robustness to OOD data, and enhanced performance compared to CQL and standard DQN. - First-of-its-kind integration of conformal prediction into Q-learning for uncertainty quantification in RL. ## What we learned - Uncertainty estimation is critical in RL, particularly for offline settings where exploration is limited. - Accelerating RL training with PyTorch optimizations: Leveraging techniques such as torch.jit for just-in-time (JIT) compilation, CUDA acceleration, and efficient tensor operations can significantly speed up Q-network training and conformal interval calculations, making the approach more scalable for real-world applications. ## What's next for Stable-RL We are looking to expand the amount of algorithms that we can integrate this in. Currently, we had time to implement this idea with Soft-Actor Critic, but integrating this uncertainty quantification into more popular algorithms is our next goal. Additionally testing its robustness on edge-based scenarios, like autonomous driving, is a plausible next step as our motivation was centered around improving robustness in these areas ## Detected evidence (automated analysis) Indexed codebase: 79 recognized source files, 296 KB. - CSS (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - PyTorch (technology) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (86 of 86) ``` .DS_Store conformal_agent/agent_wrapper.py conformal_agent/agents.py frontend/.gitignore frontend/app/docs/api/configuration/page.tsx frontend/app/docs/api/conformal-q-learning/page.tsx frontend/app/docs/api/utilities/page.tsx frontend/app/docs/conformal-prediction/page.tsx frontend/app/docs/conformal-q-learning/page.tsx frontend/app/docs/globals.css frontend/app/docs/installation/page.tsx frontend/app/docs/layout.tsx frontend/app/docs/offline-learning/page.tsx frontend/app/docs/overview/page.tsx frontend/app/docs/page.tsx frontend/app/docs/quickstart/page.tsx frontend/app/docs/sacagent/page.tsx frontend/app/globals.css frontend/app/layout.tsx frontend/app/page.tsx frontend/components.json frontend/components/doc-layout.tsx frontend/components/kokonutui/background-paths.tsx frontend/components/theme-provider.tsx frontend/components/ui/accordion.tsx frontend/components/ui/alert-dialog.tsx frontend/components/ui/alert.tsx frontend/components/ui/aspect-ratio.tsx frontend/components/ui/avatar.tsx frontend/components/ui/badge.tsx frontend/components/ui/breadcrumb.tsx frontend/components/ui/button.tsx frontend/components/ui/calendar.tsx frontend/components/ui/card.tsx frontend/components/ui/carousel.tsx frontend/components/ui/chart.tsx frontend/components/ui/checkbox.tsx frontend/components/ui/collapsible.tsx frontend/components/ui/command.tsx frontend/components/ui/context-menu.tsx frontend/components/ui/dialog.tsx frontend/components/ui/drawer.tsx frontend/components/ui/dropdown-menu.tsx frontend/components/ui/form.tsx frontend/components/ui/hover-card.tsx frontend/components/ui/input-otp.tsx frontend/components/ui/input.tsx frontend/components/ui/label.tsx frontend/components/ui/menubar.tsx frontend/components/ui/navigation-menu.tsx frontend/components/ui/pagination.tsx frontend/components/ui/popover.tsx frontend/components/ui/progress.tsx frontend/components/ui/radio-group.tsx frontend/components/ui/resizable.tsx frontend/components/ui/scroll-area.tsx frontend/components/ui/select.tsx frontend/components/ui/separator.tsx frontend/components/ui/sheet.tsx frontend/components/ui/sidebar.tsx frontend/components/ui/skeleton.tsx frontend/components/ui/slider.tsx frontend/components/ui/sonner.tsx frontend/components/ui/switch.tsx frontend/components/ui/table.tsx frontend/components/ui/tabs.tsx frontend/components/ui/textarea.tsx frontend/components/ui/toast.tsx frontend/components/ui/toaster.tsx frontend/components/ui/toggle-group.tsx frontend/components/ui/toggle.tsx frontend/components/ui/tooltip.tsx frontend/components/ui/use-mobile.tsx frontend/components/ui/use-toast.ts frontend/hooks/use-mobile.tsx frontend/hooks/use-toast.ts frontend/lib/utils.ts frontend/next.config.mjs frontend/package.json frontend/postcss.config.mjs frontend/styles/globals.css frontend/tailwind.config.ts frontend/tsconfig.json README.md regularized_q.py test_agents.py ``` ### Dependencies - frontend/package.json: @emotion/is-prop-valid@latest, @hookform/resolvers@^3.9.1, @radix-ui/react-accordion@^1.2.2, @radix-ui/react-alert-dialog@^1.1.4, @radix-ui/react-aspect-ratio@^1.1.1, @radix-ui/react-avatar@^1.1.2, @radix-ui/react-checkbox@^1.1.3, @radix-ui/react-collapsible@^1.1.2, @radix-ui/react-context-menu@^2.2.4, @radix-ui/react-dialog@^1.1.4, @radix-ui/react-dropdown-menu@^2.1.4, @radix-ui/react-hover-card@^1.1.4, @radix-ui/react-label@^2.1.1, @radix-ui/react-menubar@^1.1.4, @radix-ui/react-navigation-menu@^1.2.3, @radix-ui/react-popover@^1.1.4, @radix-ui/react-progress@^1.1.1, @radix-ui/react-radio-group@^1.2.2, @radix-ui/react-scroll-area@^1.2.2, @radix-ui/react-select@^2.1.4, @radix-ui/react-separator@^1.1.1, @radix-ui/react-slider@^1.2.2, @radix-ui/react-slot@^1.1.1, @radix-ui/react-switch@^1.1.2, @radix-ui/react-tabs@^1.1.2, @radix-ui/react-toast@^1.2.4, @radix-ui/react-toggle@^1.1.1, @radix-ui/react-toggle-group@^1.1.1, @radix-ui/react-tooltip@^1.1.6, @types/node@^22, @types/react@^18, @types/react-dom@^18, autoprefixer@^10.4.20, class-variance-authority@^0.7.1, clsx@^2.1.1, cmdk@1.0.4, date-fns@4.1.0, embla-carousel-react@8.5.1, framer-motion@latest, input-otp@1.4.1, lucide-react@^0.454.0, next@14.2.16, next-themes@^0.4.4, postcss@^8, react@^18, react-day-picker@8.10.1, react-dom@^18, react-hook-form@^7.54.1, react-resizable-panels@^2.1.7, recharts@2.15.0, sonner@^1.7.1, tailwind-merge@^2.5.5, tailwindcss@^3.4.17, tailwindcss-animate@^1.0.7, typescript@^5, vaul@^0.9.6, zod@^3.24.1 ### Recent commits (newest first) - Update README.md - added frontend - library version - base conformal + sac, testing if works - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### frontend/package.json ``` { "name": "my-v0-project", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@hookform/resolvers": "^3.9.1", "@radix-ui/react-accordion": "^1.2.2", "@radix-ui/react-alert-dialog": "^1.1.4", "@radix-ui/react-aspect-ratio": "^1.1.1", "@radix-ui/react-avatar": "^1.1.2", "@radix-ui/react-checkbox": "^1.1.3", "@radix-ui/react-collapsible": "^1.1.2", "@radix-ui/react-context-menu": "^2.2.4", "@radix-ui/react-dialog": "^1.1.4", "@radix-ui/react-dropdown-menu": "^2.1.4", "@radix-ui/react-hover-card": "^1.1.4", "@radix-ui/react-label": "^2.1.1", "@radix-ui/react-menubar": "^1.1.4", "@radix-ui/react-navigation-menu": "^1.2.3", "@radix-ui/react-popover": "^1.1.4", "@radix-ui/react-progress": "^1.1.1", "@radix-ui/react-radio-group": "^1.2.2", "@radix-ui/react-scroll-area": "^1.2.2", "@radix-ui/react-select": "^2.1.4", "@radix-ui/react-separator": "^1.1.1", "@radix-ui/react-slider": "^1.2.2", "@radix-ui/react-slot": "^1.1.1", "@radix-ui/react-switch": "^1.1.2", "@radix-ui/react-tabs": "^1.1.2", "@radix-ui/react-toast": "^1.2.4", "@radix-ui/react-toggle": "^1.1.1", "@radix-ui/react-toggle-group": "^1.1.1", "@radix-ui/react-tooltip": "^1.1.6", "autoprefixer": "^10.4.20", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "cmdk": "1.0.4", "date-fns": "4.1.0", "embla-carousel-react": "8.5.1", "input-otp": "1.4.1", "lucide-react": "^0.454.0", "next": "14.2.16", "next-themes": "^0.4.4", "react": "^18", "react-day-picker": "8.10.1", "react-dom": "^18", "react-hook-form": "^7.54.1", "react-resizable-panels": "^2.1.7", "recharts": "2.15.0", "sonner": "^1.7.1", "tailwind-merge": "^2.5.5", "tailwindcss-animate": "^1.0.7", "vaul": "^0.9.6", "zod": "^3.24.1", "@emotion/is-prop-valid": "latest", "framer-motion": "latest" }, "devDependencies": { "@types/node": "^22", "@types/react": "^18", "@types/react-dom": "^18", "postcss": "^8", "tailwindcss": "^3.4.17", "typescript": "^5" } } ``` ### frontend/app/layout.tsx ```typescript import type { Metadata } from 'next' import './globals.css' export const metadata: Metadata = { title: 'v0 App', description: 'Created with v0', generator: 'v0.dev', } export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode }>) { return ( {children} ) } ``` ### frontend/app/page.tsx ```typescript "use client" import { useState, useEffect } from "react" import { motion, AnimatePresence } from "framer-motion" import { Button } from "@/components/ui/button" import { Github } from "lucide-react" import Link from "next/link" function StarryBackground() { const [hoveredStar, setHoveredStar] = useState(null) const [stars, setStars] = useState>([]) useEffect(() => { // Create 200 stars with random positions, sizes, and animation delays const newStars = Array.from({ length: 200 }, (_, i) => ({ id: i, x: Math.random() * 100, y: Math.random() * 100, size: Math.random() * 0.15 + 0.05, // Sizes between 0.05 and 0.2 delay: Math.random() * 3, // Random delay for twinkling effect })) setStars(newStars) }, []) return (
{stars.map((star) => ( setHoveredStar(star.id)} onMouseLeave={() => setHoveredStar(null)} /> {hoveredStar === star.id && ( )} ))}
) } export default function LandingPage() { const title = "RL-CP Fusion" const words = title.split(" ") return (

{words.map((word, wordIndex) => ( {word.split("").map((letter, letterIndex) => ( {letter} ))} ))}

Offline Reinforcement Learning with Conformal Prediction Tackle real-world decision-making challenges with our innovative approach that combines the power of Offline Reinforcement Learning and the reliability of Conformal Prediction.
[truncated — 2587 more characters] ``` ### frontend/app/docs/layout.tsx ```typescript import type { Metadata } from "next" import { Inter } from "next/font/google" import "./globals.css" import type React from "react" // Added import for React const inter = Inter({ subsets: ["latin"] }) export const metadata: Metadata = { title: "RL-CP Fusion Documentation", description: "Documentation for the RL-CP Fusion library", } export default function RootLayout({ children, }: { children: React.ReactNode }) { return ( {children} ) } ``` ### frontend/app/docs/page.tsx ```typescript "use client" import { useState, useEffect } from "react" import { motion } from "framer-motion" import Link from "next/link" import { Search, ChevronLeft, Copy } from "lucide-react" import { Button } from "@/components/ui/button" import { Input } from "@/components/ui/input" function StarryBackground() { const [stars, setStars] = useState>([]) useEffect(() => { const newStars = Array.from({ length: 100 }, (_, i) => ({ id: i, x: Math.random() * 100, y: Math.random() * 100, size: Math.random() * 0.1 + 0.05, delay: Math.random() * 3, })) setStars(newStars) }, []) return (
{stars.map((star) => ( ))}
) } const sections = { "GET STARTED": [ { title: "Overview", href: "/docs/overview" }, { title: "Quickstart", href: "/docs/quickstart" }, { title: "Installation", href: "/docs/installation" }, ], "CORE CONCEPTS": [ { title: "SACAgent", href: "/docs/sacagent" }, { title: "Conformal Prediction", href: "/docs/conformal-prediction" }, { title: "Offline Learning", href: "/docs/offline-learning" }, ], "API REFERENCE": [ { title: "SACAgent Class", href: "/docs/api/sacagent" }, { title: "Configuration", href: "/docs/api/configuration" }, { title: "Utilities", href: "/docs/api/utilities" }, ], } export default function DocsPage() { const [searchQuery, setSearchQuery] = useState("") return (
{/* Left Sidebar */}
setSearchQuery(e.target.value)} />
{/* Main Content */}

SACAgent Class

The main interface for working with the Conformal SAC agent.

The SACAgent class provides a high-level wrapper for training and evaluating agents using the Conformal Soft Actor-Critic algorithm with offline datasets.

Initialization

Create a new SACAgent instance with the following parameters:

                {`from conformal_sac.agent_wrapper import SACAgent

agent = SACAgent(
    env_name="halfcheetah-medium-expert",
    offline=True,
    iteration=100000,
    seed=42,
    learning_rate=3e-4,
    gamma=0.99,
    tau=0.005,
    batch_size=256,
    log_interval=2000,
    alpha_q=100,
    q_alpha_update_freq=50
)`}

Parameters