# Project export: Onboardly

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 12.0
- Tagline: automate onboarding and training + realtime coaching
- Devpost: https://devpost.com/software/onboardly-3bp9gt
- GitHub: https://github.com/Preet37/Onboardly
- Video: https://www.youtube.com/embed/ufnTHSQetfM?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Conversion: Best Use of Conversion)
- Team: 3 GitHub contributor(s) — danielachacon (15 commits), Preet Karia (4 commits), Claude (1 commits)

## Devpost submission (written by the team)

### Inspiration

We were inspired by the massive efficiency gap in technical onboarding. Companies often spend days manually provisioning accounts, setting up security, and explaining the same basic cloud concepts repeatedly. This process is intimidating, prone to human error, and lacks real-time guidance for new hires. Our goal was to eliminate this friction entirely, creating an AI that doesn't just create tickets, but actively guarantees a smooth, secure, and educational Day 1 experience by automating both the company-side security setup and the user's hands-on learning process.

### What it does

Onboardly is a Full-Stack, AI-Powered Onboarding Orchestrator that seamlessly bridges IT provisioning and user coaching. For the Company (Automation): From a single form trigger, Onboardly uses the Groq AI to dynamically generate an 8-step, role-specific curriculum (e.g., GCloud training for an SWE Intern). It then uses Jira's API to instantly create a master Epic and all sub-tasks, and SendGrid to deliver a personalized welcome email with a downloadable calendar invite, all assigned directly to the new hire. For the Intern (AI Coach Extension): Once the intern clicks the Jira link and navigates to the Google Cloud Console, our custom Chrome Extension injects an AI Coach panel. This panel uses Gemini Vision AI to capture and analyze the screen, providing real-time, step-by-step guidance on how to complete each task—like a senior developer looking over their shoulder. The entire workflow is tracked, and when the final task is complete, the system automatically transitions the main Jira Epic to DONE.

### How we built it

We built Onboardly using a clean, three-part architecture. 1. The Provisioning Engine (Node.js/Express): This handles the pre-onboarding setup, orchestrating API calls to Groq for curriculum generation, and using the Jira API to create and assign the entire task hierarchy. 2. The AI Brain (Python/Flask with Gemini): This server exposes the vision endpoints. It utilizes Gemini 2.0 Flash for low-latency visual analysis of the screenshot and Gemini 2.5 Pro for sophisticated reasoning and generating clear coaching instructions. 3. The AI Coach Frontend (Chrome Extension): This utilizes a Background Service Worker to securely capture the visible tab screenshot, which the Content Script then sends to the Python backend. The Content Script then renders the real-time coaching UI directly onto the Google Cloud Console interface, providing the interactive guidance needed to complete the Jira tasks.

### Challenges we ran into

The primary challenges involved navigating complex and often fragile enterprise APIs. Jira Provisioning was the biggest obstacle: We faced persistent issues finding the correct internal issue type IDs (10001, 10004) and dealing with the obscure "Epic Name" field ID, which required removal for our simple project. Furthermore, the Jira API user invite process consistently failed on the free tier. We bypassed this by implementing a feature to automatically assign the Epic to the manager's account, ensuring the demo user had instant, authorized access to the tasks. Secondly, AI Key Quotas blocked our progress with an insufficient_quota error, which we resolved by performing a real-time migration to Groq AI's compatible API, maintaining our dynamic curriculum feature.

### Accomplishments we're proud of

We are most proud of achieving true, end-to-end automation of a complex business process within a short hackathon window. This includes: 1. Zero-Touch Provisioning: Successfully creating a Jira Epic, 8 sub-tasks, and sending a personalized welcome email with a downloadable calendar invite—all from a single Node.js trigger. 2. Dynamic Curriculum: Using Groq AI to generate a highly detailed, accurate 8-step GCloud curriculum in under one second. 3. The Wow Factor: Seamlessly integrating real-time Gemini Vision coaching that actually understands what the user is seeing on a complex external site (GCP Console) and guiding them to complete the automated tasks.

### What we learned

We learned three crucial lessons: 1. AI Compatibility is Key: Utilizing the OpenAI-compatible API structure (as provided by Groq) is vital for rapid prototyping and maintaining provider flexibility when quotas are an issue. 2. API Workarounds are Essential: Complex enterprise APIs (like Jira's) often require deep inspection of error messages and unconventional workarounds (like removing required fields or using specific transition IDs) to integrate successfully in a fast-paced environment. 3. The "Isolation Problem" Requires a Stack: A helpful AI coach cannot be built in one script; it requires a stack of communication (Content Script $\leftrightarrow$ Background Script $\leftrightarrow$ Flask Backend) to securely and effectively capture the user's screen and leverage powerful vision models.

### What's next

for onboardly We plan to implement three key features: 1. Jira Webhook Integration: We will eliminate the final "Mark Complete" button by having Jira trigger a webhook back to our server when all 8 sub-tasks are manually marked "Done," achieving truly touchless final Epic completion. 2. Advanced Security Checks: Integrate the GitHub API (which we built a stub for) to check if the intern has enabled required branch protection rules on their new starter repository before marking that task as complete. 3. Gemini Pro Reasoning: We will leverage the powerful reasoning capabilities of Gemini 2.5 Pro to provide non-visual feedback, such as analyzing the security logs in the next step and providing a summary of the threats found directly to the intern.

## README (from the GitHub repository)

# Onboardly - 🏆 Cal Hacks 12.0 Best Workflow App

AI-powered onboarding platform that uses vision analysis to guide users through complex workflows in real-time.

## Overview

Onboardly combines AI vision analysis with intelligent coaching to create interactive onboarding experiences. It monitors user screens, understands context, and provides step-by-step guidance for tasks like GCP setup, Jira workflows, and custom onboarding processes.

### Key Features

- **AI Vision Coaching** - Real-time screenshot analysis using Gemini Vision AI
- **Multi-Platform** - Chrome extension + browser-agnostic embeddable widget
- **Workflow Visualization** - Beautiful 3D visualizations and interactive DAGs
- **Intelligent Guidance** - Context-aware, step-by-step instructions
- **Automated Tracking** - Progress monitoring and completion verification
- **Integration Ready** - Works with GCP, Jira, SendGrid, and custom apps

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                         Frontend                             │
│         React + Vite + Three.js + Tailwind CSS              │
│   (Workflow Visualization, Onboarding Dashboard)             │
└──────────────────────┬──────────────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────────────┐
│                      Backend API                             │
│           Node.js + Express + AI Services                    │
│   • Gemini Vision AI (screenshot analysis)                   │
│   • Groq/OpenAI (workflow generation)                        │
│   • Jira, SendGrid, GCP integrations                        │
└──────────────────────┬──────────────────────────────────────┘
                       │
        ┌──────────────┴──────────────┐
        │                             │
┌───────▼────────┐          ┌─────────▼──────────┐
│ Chrome Extension│          │  Embeddable Widget │
│  (GCP Console)  │          │   (Any Browser)    │
│  • Screenshot   │          │   • Screen Capture │
│  • Auto-coach   │          │   • Drop-in JS     │
└─────────────────┘          └────────────────────┘
```

## Components

### 1. Backend (`/backend`)

Node.js/Express API server with AI integrations.

**Tech Stack:**
- Express.js
- Gemini Vision AI (Google)
- Groq/OpenAI (workflow generation)
- SendGrid (email notifications)
- Jira API integration

**Key Endpoints:**
- `POST /generate-workflow-dag` - Generate workflow from natural language
- `POST /analyze-screenshot` - AI vision analysis of user screens
- `POST /track-event` - Track onboarding events
- `GET /workflow-progress` - Get real-time progress updates (SSE)

**Setup:**
```bash
cd backend
npm install
cp .env.example .env  # Add your API keys
node index.js
```

Required environment variables:
```
GEMINI_API_KEY=your_gemini_key
GROQ_API_KEY=your_groq_key
SENDGRID_API_KEY=your_sendgrid_key
JIRA_BASE_URL=https://your-domain.atlassian.net
JIRA_USER_EMAIL=your_email
JIRA_API_TOKEN=your_jira_token
```

### 2. Frontend (`/frontend`)

React visualization dashboard with 3D workflow representations.

**Tech Stack:**
- React 18
- Vite
- Three.js + React Three Fiber
- Tailwind CSS
- D3.js + Recharts
- Framer Motion

**Features:**
- 3D neural network visualization
- Interactive workflow DAG
- Real-time onboarding progress
- Live activity monitoring
- AI insights panel

**Setup:**
```bash
cd frontend
npm install
npm run dev  # Starts on http://localhost:5173
```

### 3. Chrome Extension (`/extension`)

Chrome extension for AI-powered coaching on GCP Console.

**Features:**
- Auto-capture screenshots every 5 seconds
- Real-time AI guidance overlay
- Step-by-step task validation
- Automatic progress tracking

**Setup:**
1. Open Chrome → `chrome://extensions/`
2. Enable **Developer mode**
3. Click **Load unpacked**
4. Select the `/extension` folder

See [extension/README.md](extension/README.md) for detailed instructions.

### 4. Embeddable Widget (`/webapp`)

Browser-agnostic JavaScript widget for any web application.

**Features:**
- Pure JavaScript (no framework dependencies)
- Screen Capture API for cross-browser support
- Drop-in integration (`<script>` tag)
- Configurable positioning and behavior

**Quick Start:**
```html
<script src="ai-coach-widget.js"></script>
<script>
  AICoach.init({
    apiUrl: 'http://localhost:3000',
    taskType: 'gcp_storage',
    captureInterval: 5000
  });
</script>
```

See [webapp/README.md](webapp/README.md) for integration guide.

## Quick Start

### 1. Clone and Install

```bash
git clone <repository-url>
cd Onboardly

# Install backend dependencies
cd backend
npm install

# Install frontend dependencies
cd ../frontend
npm install
```

### 2. Configure Environment

Create `backend/.env`:
```bash
GEMINI_API_KEY=your_gemini_api_key
GROQ_API_KEY=your_groq_api_key
SENDGRID_API_KEY=your_sendgrid_key
PORT=3000
```

### 3. Start Services

Terminal 1 - Backend:
```bash
cd backend
node index.js
# Backend runs on http://localhost:3000
```

Terminal 2 - Frontend:
```bash
cd frontend
npm run dev
# Frontend runs on http://localhost:5173
```

### 4. Choose Your Integration

**Option A: Chrome Extension** (for GCP Console)
- Load extension in Chrome
- Navigate to GCP Console
- Click extension icon to activate

**Option B: Embeddable Widget** (for any webapp)
- Serve webapp files
- Open `http://localhost:8080/demo.html`
- Click "Start Monitoring"

## Use Cases

### GCP Onboarding
Guide users through Google Cloud Platform setup:
- Create Storage Buckets
- Deploy Cloud Functions
- Configure IAM permissions

### Jira Workflow Training
Teach team members Jira workflows:
- Create and assign issues
- Use custom fields
- Follow approval processes

### Custom Workflows
Create any onboarding experience:
- SaaS product tours
- Internal tool training
- Compliance procedures

## Development

### Project Structure

```
Onboardly/
├── backend/              # Node.js API server
│   ├── index.js         # Main server file
│   ├── package.json
│   └── .env             # API keys (create from .env.example)
│
├── frontend/            # React visualization app
│   ├── src/
│   │   ├── OnboardingApp.jsx    # Onboarding flow UI
│   │   ├── OnboardlyApp.jsx     # Main dashboard
│   │   └── components/          # React components
│   └── package.json
│
├── extension/           # Chrome extension
│   ├── manifest.json
│   ├── background.js    # Screenshot capture
│   └── inject-panel.js  # Content script + UI
│
└── webapp/              # Embeddable widget
    ├── ai-coach-widget.js   # Widget implementation
    ├── ai-coach-widget.css  # Widget styles
    └── demo.html            # Demo page
```

### Adding New Tasks

Define new onboarding tasks in the backend workflow generator:

```javascript
// backend/index.js - Add to task definitions
const taskDefinitions = {
  'your_task': {
    name: 'Your Custom Task',
    steps: [
      { instruction: 'Step 1', type: 'action' },
      { instruction: 'Step 2', type: 'verification' }
    ]
  }
};
```

### API Documentation

#### Generate Workflow
```http
POST /generate-workflow-dag
Content-Type: application/json

{
  "prompt": "Create a workflow for deploying a Docker container"
}
```

#### Analyze Screenshot
```http
POST /analyze-screenshot
Content-Type: application/json

{
  "screenshot": "data:image/png;base64,...",
  "taskType": "gcp_storage",
  "currentStep": 1,
  "email": "user@example.com"
}
```

#### Track Event
```http
POST /track-event
Content-Type: application/json

{
  "email": "user@example.com",
  "eventType": "extension_activated",
  "taskType": "gcp_storage"
}
```

## Browser Compatibility

| Component | Chrome | Firefox | Safari | Edge |
|-----------|--------|---------|--------|------|
| Frontend | ✅ | ✅ | ✅ | ✅ |
| Extension | ✅ | ❌ | ❌ | ✅* |
| Widget | ✅ | ✅ | ✅ | ✅ |

*Edge supports Chrome extensions with minor modifications

## Deployment

### Backend Deployment

```bash
# Build and deploy to your preferred platform
# (Heroku, AWS,

[README truncated for size]

## Detected evidence (automated analysis)

Indexed codebase: 35 recognized source files, 265 KB.
- CSS (language) — detected in the code
- Express (technology) — detected in the code
- Flask (technology) — detected in the code
- Google Gemini (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- Node.js (technology) — claimed on Devpost, not found in the code
- AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers

## Codebase structure (from repository index)

### Files (42 of 42)

```
.claude/settings.local.json
.gitignore
backend/index.js
backend/list-models.js
backend/package.json
backend/requirements.txt
extension/activate.html
extension/background.js
extension/inject-panel.js
extension/manifest.json
extension/README.md
frontend/index.html
frontend/package.json
frontend/postcss.config.js
frontend/public/demo.json
frontend/src/components/ActivitySummaryPanel.jsx
frontend/src/components/InsightsPanel.jsx
frontend/src/components/LiveActivityPanel.jsx
frontend/src/components/NarrationPanel.jsx
frontend/src/components/NeuralCosmos.jsx
frontend/src/components/PipelineFlow.jsx
frontend/src/components/PromptPanel.jsx
frontend/src/components/RunSummaryCard.jsx
frontend/src/index.css
frontend/src/main.jsx
frontend/src/OnboardingApp.jsx
frontend/src/OnboardlyApp.jsx
frontend/src/services/insightGenerator.js
frontend/src/services/mockPipelineStream.js
frontend/src/services/onboardingService.js
frontend/src/services/pipelineStream.js
frontend/src/utils/eventMessages.js
frontend/src/utils/statsCalculator.js
frontend/src/utils/terminology.js
frontend/tailwind.config.js
frontend/vite.config.js
README.md
webapp/ai-coach-widget.css
webapp/ai-coach-widget.js
webapp/demo.html
webapp/proxy.py
webapp/README.md
```

### Dependencies

- backend/package.json: @google/generative-ai@^0.24.1, @sendgrid/mail@^8.1.6, @slack/web-api@^7.12.0, axios@^1.12.2, cors@^2.8.5, dotenv@^17.2.3, express@^5.1.0, googleapis@^164.1.0, ics@^3.8.1, openai@^6.7.0
- backend/requirements.txt: flask@==3.0.0, flask-cors@==4.0.0, google-generativeai@==0.3.2, Pillow@==10.1.0, python-dotenv@==1.0.0, requests@==2.31.0
- frontend/package.json: @react-three/drei@^9.92.0, @react-three/fiber@^8.15.0, @types/react@^18.3.1, @types/react-dom@^18.3.0, @vitejs/plugin-react@^4.2.1, autoprefixer@^10.4.19, d3@^7.9.0, framer-motion@^11.0.0, postcss@^8.4.38, react@^18.3.1, react-dom@^18.3.1, recharts@^2.12.0, tailwindcss@^3.4.3, three@^0.160.0, vite@^5.2.10

### Recent commits (newest first)

- Revise project title and description in README
- Add comprehensive README documentation
- Add leniency for example-based steps (e.g.) - accept any similar resource/item
- Make AI lenient for exploration steps - visiting any mentioned section counts as completion
- AI now receives current sub-instruction context for precise step tracking and evaluation
- Final improvements: observation step support, removed action requirement for completion, onboarded node stays pending until completion
- Major improvements: AI-driven onboarding with proof-based completion, rate limit fixes, and robust error handling
- feat: Add AI-powered Chrome extension with Gemini vision analysis
- fix: Add CORS support for Google Cloud Console origin
- fix: Update UI to properly reflect extension activation and training progress
- fix: Set intern info before form submission
- fix: Add comprehensive logging and fix SSE session persistence
- feat: Implement state flip tracking (0→1→0) for training lifecycle
- feat: Add extension usage tracking and dynamic website configuration
- Merge main into feature/ai-coach-chrome-extension - resolve gitignore conflict
- Add Node.js to gitignore
- Add AI Onboarding Coach Chrome Extension with GCP Cloud Storage support
- Backend up and running
- Backend done
- sfdkjasd

## Key source files (fetched from GitHub, selected and truncated for size)

### backend/requirements.txt

```
flask==3.0.0
flask-cors==4.0.0
google-generativeai==0.3.2
python-dotenv==1.0.0
Pillow==10.1.0
requests==2.31.0

```

### backend/package.json

```
{
  "dependencies": {
    "@google/generative-ai": "^0.24.1",
    "@sendgrid/mail": "^8.1.6",
    "@slack/web-api": "^7.12.0",
    "axios": "^1.12.2",
    "cors": "^2.8.5",
    "dotenv": "^17.2.3",
    "express": "^5.1.0",
    "googleapis": "^164.1.0",
    "ics": "^3.8.1",
    "openai": "^6.7.0"
  }
}

```

### frontend/package.json

```
{
  "name": "onboardly-viz",
  "private": true,
  "version": "0.1.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "vite build",
    "preview": "vite preview"
  },
  "dependencies": {
    "@react-three/drei": "^9.92.0",
    "@react-three/fiber": "^8.15.0",
    "d3": "^7.9.0",
    "framer-motion": "^11.0.0",
    "react": "^18.3.1",
    "react-dom": "^18.3.1",
    "recharts": "^2.12.0",
    "three": "^0.160.0"
  },
  "devDependencies": {
    "@types/react": "^18.3.1",
    "@types/react-dom": "^18.3.0",
    "@vitejs/plugin-react": "^4.2.1",
    "autoprefixer": "^10.4.19",
    "postcss": "^8.4.38",
    "tailwindcss": "^3.4.3",
    "vite": "^5.2.10"
  }
}

```

### frontend/src/main.jsx

```javascript
import React from 'react'
import ReactDOM from 'react-dom/client'
import OnboardingApp from './OnboardingApp'
import './index.css'

ReactDOM.createRoot(document.getElementById('root')).render(
  <React.StrictMode>
    <OnboardingApp />
  </React.StrictMode>,
)

```

### frontend/postcss.config.js

```javascript
export default {
  plugins: {
    tailwindcss: {},
    autoprefixer: {},
  },
}

```

### frontend/vite.config.js

```javascript
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'

export default defineConfig({
  plugins: [react()],
  server: {
    port: 5173,
    open: true
  }
})

```

### frontend/index.html

```html
<!doctype html>
<html lang="en" class="dark">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" type="image/svg+xml" href="/vite.svg" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Onboardly - AI Onboarding Platform</title>
  </head>
  <body class="dark">
    <div id="root"></div>
    <script type="module" src="/src/main.jsx"></script>
  </body>
</html>

```

### backend/list-models.js

```javascript
const { GoogleGenerativeAI } = require('@google/generative-ai');
require('dotenv').config();

const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY);

async function listModels() {
  try {
    const models = await genAI.listModels();

    console.log('Available Gemini models with vision support:\n');

    for await (const model of models) {
      const methods = model.supportedGenerationMethods || [];
      if (methods.includes('generateContent')) {
        console.log(`✓ ${model.name}`);
        console.log(`  Description: ${model.description || 'N/A'}`);
        console.log(`  Methods: ${methods.join(', ')}`);
        console.log('');
      }
    }
  } catch (error) {
    console.error('Error listing models:', error.message);
  }
}

listModels();

```

### extension/background.js

```javascript
/**
 * AI Onboarding Coach - Background Service Worker
 * Handles screenshot capture
 */

// Listen for messages from content script
chrome.runtime.onMessage.addListener((request, sender, sendResponse) => {
  if (request.action === 'captureTab') {
    const tabId = sender.tab?.id;
    const windowId = sender.tab?.windowId;

    if (!tabId) {
      sendResponse({ success: false, error: 'No tab ID available' });
      return true;
    }

    if (!windowId) {
      sendResponse({ success: false, error: 'No window ID available' });
      return true;
    }

    // Capture the visible tab with JPEG compression to reduce payload size
    // Using quality 50 for better compression (service workers can't use canvas/Image for resizing)
    chrome.tabs.captureVisibleTab(windowId, { format: 'jpeg', quality: 50 }, (dataUrl) => {
      if (chrome.runtime.lastError) {
        sendResponse({ success: false, error: chrome.runtime.lastError.message });
      } else if (!dataUrl) {
        sendResponse({ success: false, error: 'No screenshot data' });
      } else {
        sendResponse({ success: true, screenshot: dataUrl });
      }
    });

    return true;
  }

  return false;
});

// Handle extension icon click
chrome.action.onClicked.addListener((tab) => {
  if (!tab.url || !tab.url.includes('console.cloud.google.com')) {
    if (chrome.notifications && chrome.notifications.create) {
      chrome.notifications.create({
        type: 'basic',
        iconUrl: 'icon48.png',
        title: 'AI Onboarding Coach',
        message: 'Navigate to console.cloud.google.com to start coaching'
      });
    }
  }
});

```

### frontend/tailwind.config.js

```javascript
/** @type {import('tailwindcss').Config} */
export default {
  content: [
    "./index.html",
    "./src/**/*.{js,ts,jsx,tsx}",
  ],
  theme: {
    extend: {
      colors: {
        border: "hsl(var(--border))",
        background: "hsl(var(--background))",
        foreground: "hsl(var(--foreground))",
        primary: {
          DEFAULT: "hsl(var(--primary))",
          foreground: "hsl(var(--primary-foreground))",
        },
        secondary: {
          DEFAULT: "hsl(var(--secondary))",
          foreground: "hsl(var(--secondary-foreground))",
        },
        muted: {
          DEFAULT: "hsl(var(--muted))",
          foreground: "hsl(var(--muted-foreground))",
        },
        accent: {
          DEFAULT: "hsl(var(--accent))",
          foreground: "hsl(var(--accent-foreground))",
        },
      },
      fontFamily: {
        sans: ['Lexend', 'system-ui', '-apple-system', 'sans-serif'],
        mono: ['JetBrains Mono', 'Menlo', 'Monaco', 'Courier New', 'monospace'],
      },
      fontWeight: {
        light: '300',
        normal: '300',
        medium: '400',
        semibold: '500',
        bold: '600',
      },
      animation: {
        'fade-in': 'fadeIn 0.5s ease-in-out',
        'slide-up': 'slideUp 0.3s ease-out',
      },
      keyframes: {
        fadeIn: {
          '0%': { opacity: '0' },
          '100%': { opacity: '1' },
        },
        slideUp: {
          '0%': { transform: 'translateY(10px)', opacity: '0' },
          '100%': { transform: 'translateY(0)', opacity: '1' },
        }
      },
      borderRadius: {
        lg: 'var(--radius)',
        md: 'calc(var(--radius) - 2px)',
        sm: 'calc(var(--radius) - 4px)',
      },
    },
  },
  plugins: [],
}

```

[24 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]