# Project export: Lexicon

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: OpenAI Build Week
- Tagline: Your product already knows what to do. Lexicon gives it a voice grounded in the actions, schemas, and rules you trust.
- Devpost: https://devpost.com/software/lexicon-kex50t
- GitHub: https://github.com/Jalkarna/lexicon
- Demo: https://lexicon-wheat.vercel.app/
- Video: https://www.youtube.com/embed/-QVpkpSkF4o?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — Jalkarna (6 commits)

## Devpost submission (written by the team)

### Inspiration

Lexicon started from the problems we saw with browser agents. They inspect a screen or DOM, guess what to click, and repeat this for every step. It looks impressive, but it can be slow and fragile. A changed button, loading state, or hidden permission can break the workflow—or cause the wrong action. We wanted an assistant that works with the product’s real actions instead of operating it from the outside. What we built Lexicon is a TypeScript SDK that exposes product actions as typed capabilities. Each capability defines its inputs, route, handler, and confirmation policy. Gemini understands what the user means. Lexicon validates the request, collects missing fields, applies the product’s rules, and runs the real handler. Sensitive actions require confirmation in a separate user turn, so the model cannot approve its own request. We built Rillwork, a full Next.js product, to demonstrate it. It includes 32 capabilities across 9 routes covering analytics, customers, orders, invoices, approvals, settings, navigation, and exports. Challenges The hardest part was making confirmation actually trustworthy while keeping the conversation natural. We also worked through incomplete user requests, live-audio latency, microphone playback, and keeping the reusable SDK separate from demo-specific business logic.

### What we learned

Browser agents are useful when no integration exists, but they have to infer how a product works from its interface. A capability-based agent gets a direct contract with the application. Our main lesson was simple: let the model understand intent, but let the product keep authority.

## README (from the GitHub repository)

# Lexicon

**Give software a voice without giving the model unrestricted control.**

[Live demo](https://lexicon-wheat.vercel.app/) · [Project story](./ABOUT_PROJECT.md) · Built for OpenAI Build Week

Lexicon is a framework-agnostic TypeScript SDK that connects natural-language and live-voice interfaces to developer-owned product actions. Instead of asking a browser agent to inspect a screen and guess what to click, Lexicon gives the model an explicit map of routes, typed inputs, handlers, and confirmation policies.

The model interprets intent. Lexicon validates the request and enforces policy. The application keeps authority over execution.

## Why Lexicon

Browser agents are useful when an application has no integration surface, but they operate from the outside. They repeatedly inspect pixels or DOM state, infer the next interaction, and hope the interface has not changed. This introduces latency and makes important workflows sensitive to layout changes, loading states, hidden permissions, and incorrect clicks.

Lexicon works from inside the product. A request such as “Create a customer named Northstar Labs on the Foundry plan” maps to a registered `create-customer` capability with typed fields and a real application handler. If the action is guarded, Lexicon prepares a preview and requires explicit confirmation in a later user turn before anything runs.

```text
User request
    ↓
Gemini understands the intent
    ↓
Lexicon matches a registered capability
    ↓
Validate and collect typed inputs
    ↓
Apply direct or confirmation-required policy
    ↓
Run the application-owned handler
```

## What is included

The repository contains two independent projects:

```text
lexicon/
├── sdk/    # @lexicon/sdk — reusable capability runtime
└── demo/   # Rillwork — Next.js product using the SDK
```

### `@lexicon/sdk`

- Serializable capability manifests
- Gemini function declarations
- Input coercion and required-field collection
- Guarded previews and later-turn confirmation
- Runtime handler registration and execution
- Typed Gemini and Live adapters
- Browser microphone PCM capture and gapless audio playback
- Deterministic local resolution for testing and fallback behavior

The SDK does not import React, Next.js, or the demo application. Authentication, data access, navigation, and mutations remain owned by the host product.

### Rillwork demo

Rillwork is a complete Next.js application showing Lexicon across normal product workflows rather than a standalone chat window. The demo registers **32 capabilities across 9 routes**, including:

- Analytics and navigation
- Customers, orders, and invoices
- Approval-required plan changes
- Settings, profile, and notification actions
- Filters, report exports, and live voice
- A capability console for inspecting routes, fields, schemas, and policies

Manual UI interactions and assistant interactions use the same application handlers, so voice does not create a second copy of the product’s business logic.

## Safety model

Confirmation belongs to the Lexicon runtime, not the model. A guarded tool call creates pending state tied to the exact capability and normalized input. The model cannot approve its own request by returning a field such as `confirmed: true`; execution requires a separate user turn that matches the pending action.

Routine reads and navigation can remain fast, while sensitive writes receive a visible preview and explicit consent.

## How Codex and GPT-5.6 were used

We used **Codex with GPT-5.6** throughout Build Week as an engineering collaborator, not as part of Lexicon’s production runtime.

Codex helped us:

- Explore the capability-based architecture and pressure-test it against browser-agent failure modes
- Implement and refactor the framework-independent TypeScript SDK
- Build the Next.js demo, capability console, live-voice surface, and guarded-action UI
- Trace confirmation authority across multiple conversation turns
- Generate focused tests for validation, missing fields, cancellation, handler failures, and model self-confirmation attempts
- Review the package boundary so the SDK stayed independent from React and the demo
- Debug integration issues, improve accessibility and interaction polish, and prepare the project documentation

GPT-5.6 was especially useful for reasoning across the full system—model tools, runtime state, application handlers, and UI feedback—while Codex provided the repository-aware workflow for inspecting code, making changes, running tests, and validating the result. Product direction, safety boundaries, and final implementation decisions remained human-owned.

Lexicon itself uses Gemini for typed intent resolution and live voice. Codex and GPT-5.6 were the development tools used to build and evaluate the project.

## Built with

TypeScript, Next.js, React, Node.js, Gemini API, Gemini Live API, Codex, GPT-5.6, Web Audio API, Tailwind CSS, and Vercel.

## Run locally

Node.js 20 or newer is required.

Build the SDK:

```bash
cd sdk
npm install
npm run build
```

Run the demo:

```bash
cd demo
npm install
cp .env.example .env.local
npm run dev
```

Set `LEXICON_GEMINI_API_KEY` in `demo/.env.local`, then open [http://localhost:3000](http://localhost:3000).

On Windows, run:

```powershell
powershell -ExecutionPolicy Bypass -File .\setup-windows.ps1
```

## Validation

```bash
cd sdk
npm test
npm run typecheck

cd ../demo
npm run typecheck
npm run build
```

The SDK test suite covers manifest sanitization, tool generation, local resolution, missing-field collection, confirmation authority, cancellation, invalid inputs, handler failures, and serializable model responses.


## Detected evidence (automated analysis)

Indexed codebase: 52 recognized source files, 650 KB.
- CSS (language) — detected in the code
- Next.js (technology) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- Vercel (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (72 of 72)

```
.gitignore
ABOUT_PROJECT.md
demo/.env.example
demo/.gitignore
demo/.prettierignore
demo/.prettierrc
demo/app/api/agent/route.ts
demo/app/api/live/session/route.ts
demo/app/api/transcribe/route.ts
demo/app/console/analytics/page.tsx
demo/app/console/customers/page.tsx
demo/app/console/invoices/page.tsx
demo/app/console/layout.tsx
demo/app/console/notifications/page.tsx
demo/app/console/orders/page.tsx
demo/app/console/overview/page.tsx
demo/app/console/page.tsx
demo/app/console/profile/page.tsx
demo/app/console/settings/page.tsx
demo/app/globals.css
demo/app/layout.tsx
demo/app/login/page.tsx
demo/app/page.tsx
demo/app/signup/page.tsx
demo/components.json
demo/components/.gitkeep
demo/components/agents-ui/agent-audio-visualizer-bar.tsx
demo/components/agents-ui/agent-audio-visualizer-radial.tsx
demo/components/theme-provider.tsx
demo/components/ui/animated-number.tsx
demo/components/ui/badge.tsx
demo/components/ui/button.tsx
demo/components/ui/canvas-reveal-effect.tsx
demo/components/ui/circuit-board.tsx
demo/components/ui/direction-aware-tabs.tsx
demo/components/ui/hero-dithering.tsx
demo/components/ui/svg-mask-effect.tsx
demo/components/ui/svg-shapes-animated.tsx
demo/components/ui/terminal-animation.tsx
demo/components/ui/text-animate.tsx
demo/components/ui/texture-overlay.tsx
demo/components/ui/toolbar-expandable.tsx
demo/eslint.config.mjs
demo/hooks/.gitkeep
demo/lib/.gitkeep
demo/lib/utils.ts
demo/next.config.ts
demo/package.json
demo/postcss.config.mjs
demo/PRODUCT.md
demo/public/.gitkeep
demo/README.md
demo/rillwork/action-map.ts
demo/rillwork/capabilities.ts
demo/rillwork/server.ts
demo/rillwork/ui.tsx
demo/tsconfig.json
PRODUCT.md
README.md
sdk/.gitignore
sdk/package.json
sdk/README.md
sdk/src/core.ts
sdk/src/gemini.ts
sdk/src/index.ts
sdk/src/live-audio.ts
sdk/src/runtime.ts
sdk/tests/core.test.mjs
sdk/tests/runtime.test.mjs
sdk/tsconfig.json
setup-windows.ps1
WINDOWS_SETUP.md
```

### Dependencies

- demo/package.json: @google/genai@^2.11.0, @lexicon/sdk@file:../sdk, @paper-design/shaders-react@^0.0.77, @playwright/test@^1.61.1, @radix-ui/react-scroll-area@^1.2.14, @radix-ui/react-slot@^1.3.0, @radix-ui/react-use-controllable-state@^1.2.3, @react-three/fiber@^9.6.1, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, @types/three@^0.185.1, class-variance-authority@^0.7.1, clsx@^2.1.1, eslint@^9, eslint-config-next@16.2.6, framer-motion@^12.42.2, lucide-react@^1.24.0, motion@^12.42.2, next@16.2.6, next-themes@^0.4.6, prettier@^3.8.3, prettier-plugin-tailwindcss@^0.8.0, radix-ui@^1.6.2, react@19.2.4, react-dom@19.2.4, react-use-measure@^2.1.7, shadcn@^4.13.0, tailwind-merge@^3.6.0, tailwindcss@^4, three@^0.185.1, tw-animate-css@^1.4.0, typescript@^5
- sdk/package.json: @google/genai@>=2.11.0, @google/genai@^2.11.0, typescript@^5

### Recent commits (newest first)

- Prepare Build Week submission documentation
- Handle malformed audio submissions
- Ignore Vercel project metadata
- Split SDK and demo applications
- Refine voice interface experience
- Initial commit: Set up Lexicon dashboard and API integration

## Key source files (fetched from GitHub, selected and truncated for size)

### WINDOWS_SETUP.md

```markdown
# Run Lexicon on Windows

## Requirements

- Windows 10 or 11
- Node.js 20 or newer from https://nodejs.org/
- npm (included with Node.js)
- A Gemini API key for the assistant and live-voice features

## Quick setup

Open PowerShell in the extracted `lexicon` folder and run:

```powershell
powershell -ExecutionPolicy Bypass -File .\setup-windows.ps1
```

Then open `demo\.env.local` and set:

```text
LEXICON_GEMINI_API_KEY=your_key_here
```

Start the application:

```powershell
cd .\demo
npm run dev
```

Open http://localhost:3000 in Chrome or Brave. If port 3000 is busy, use the URL printed by Next.js.

## Manual setup

```powershell
cd .\sdk
npm install
npm run build
cd ..\demo
Copy-Item .env.example .env.local
npm install
npm run dev
```

Keep browser zoom at 100%. For recording, also keep Windows display scaling at 100% where possible.

```

### ABOUT_PROJECT.md

```markdown
# About the project

## Inspiration

Lexicon started from the problems we saw with browser agents. They inspect a screen or DOM, guess what to click, and repeat this for every step. It looks impressive, but it can be slow and fragile. A changed button, loading state, or hidden permission can break the workflow—or cause the wrong action.

We wanted an assistant that works with the product’s real actions instead of operating it from the outside.

## What we built

Lexicon is a TypeScript SDK that exposes product actions as typed capabilities. Each capability defines its inputs, route, handler, and confirmation policy.

Gemini understands what the user means. Lexicon validates the request, collects missing fields, applies the product’s rules, and runs the real handler. Sensitive actions require confirmation in a separate user turn, so the model cannot approve its own request.

We built Rillwork, a full Next.js product, to demonstrate it. It includes 32 capabilities across 9 routes covering analytics, customers, orders, invoices, approvals, settings, navigation, and exports.

## Challenges

The hardest part was making confirmation actually trustworthy while keeping the conversation natural. We also worked through incomplete user requests, live-audio latency, microphone playback, and keeping the reusable SDK separate from demo-specific business logic.

## What we learned

Browser agents are useful when no integration exists, but they have to infer how a product works from its interface. A capability-based agent gets a direct contract with the application.

Our main lesson was simple: let the model understand intent, but let the product keep authority.

```

### sdk/package.json

```
{
  "name": "@lexicon/sdk",
  "version": "0.1.0",
  "description": "Framework-agnostic capability runtime for safe voice and typed product actions.",
  "type": "module",
  "sideEffects": false,
  "engines": {
    "node": ">=20"
  },
  "main": "./dist/index.js",
  "types": "./dist/index.d.ts",
  "exports": {
    ".": {
      "types": "./dist/index.d.ts",
      "import": "./dist/index.js"
    },
    "./gemini": {
      "types": "./dist/gemini.d.ts",
      "import": "./dist/gemini.js"
    },
    "./live": {
      "types": "./dist/live-audio.d.ts",
      "import": "./dist/live-audio.js"
    }
  },
  "files": [
    "dist",
    "README.md"
  ],
  "scripts": {
    "build": "tsc -p tsconfig.json",
    "prepare": "npm run build",
    "typecheck": "tsc -p tsconfig.json --noEmit",
    "lint": "tsc -p tsconfig.json --noEmit",
    "test": "npm run build && node --test tests/*.test.mjs"
  },
  "peerDependencies": {
    "@google/genai": ">=2.11.0"
  },
  "peerDependenciesMeta": {
    "@google/genai": {
      "optional": true
    }
  },
  "devDependencies": {
    "@google/genai": "^2.11.0",
    "typescript": "^5"
  }
}

```

### demo/package.json

```
{
  "name": "@lexicon/demo",
  "version": "0.1.0",
  "private": true,
  "type": "module",
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "eslint",
    "format": "prettier --write \"**/*.{ts,tsx}\"",
    "typecheck": "tsc --noEmit"
  },
  "dependencies": {
    "@google/genai": "^2.11.0",
    "@lexicon/sdk": "file:../sdk",
    "@paper-design/shaders-react": "^0.0.77",
    "@radix-ui/react-scroll-area": "^1.2.14",
    "@radix-ui/react-slot": "^1.3.0",
    "@radix-ui/react-use-controllable-state": "^1.2.3",
    "@react-three/fiber": "^9.6.1",
    "class-variance-authority": "^0.7.1",
    "clsx": "^2.1.1",
    "framer-motion": "^12.42.2",
    "lucide-react": "^1.24.0",
    "motion": "^12.42.2",
    "next": "16.2.6",
    "next-themes": "^0.4.6",
    "radix-ui": "^1.6.2",
    "react": "19.2.4",
    "react-dom": "19.2.4",
    "react-use-measure": "^2.1.7",
    "tailwind-merge": "^3.6.0",
    "three": "^0.185.1",
    "tw-animate-css": "^1.4.0"
  },
  "devDependencies": {
    "@playwright/test": "^1.61.1",
    "@tailwindcss/postcss": "^4",
    "@types/node": "^20",
    "@types/react": "^19",
    "@types/react-dom": "^19",
    "@types/three": "^0.185.1",
    "eslint": "^9",
    "eslint-config-next": "16.2.6",
    "prettier": "^3.8.3",
    "prettier-plugin-tailwindcss": "^0.8.0",
    "shadcn": "^4.13.0",
    "tailwindcss": "^4",
    "typescript": "^5"
  },
  "overrides": {
    "postcss": "^8.5.10"
  }
}

```

### sdk/src/index.ts

```typescript
export * from "./core.js"
export * from "./runtime.js"

```

### demo/app/page.tsx

```typescript
import type { Metadata } from "next"

import { LexiconLandingPage } from "@/rillwork/ui"

export const metadata: Metadata = {
  title: "Lexicon — Voice actions for software",
  description:
    "A capability SDK for dependable voice and typed actions inside web products.",
}

export default function HomePage() {
  return <LexiconLandingPage />
}

```

### demo/rillwork/server.ts

```typescript
export function geminiApiKey() {
  return (
    process.env.LEXICON_GEMINI_API_KEY ??
    process.env.GOOGLE_API_KEY ??
    process.env.GEMINI_API_KEY
  )
}

export function liveModelId() {
  return (process.env.GEMINI_LIVE_MODEL ?? "gemini-3.1-flash-live-preview").replace(
    /^models\//,
    ""
  )
}

export function agentModelId() {
  return (process.env.GEMINI_AGENT_MODEL ?? "gemini-3.5-flash").replace(
    /^models\//,
    ""
  )
}

```

### demo/app/layout.tsx

```typescript
import { Geist, Geist_Mono } from "next/font/google"

import "./globals.css"
import { ThemeProvider } from "@/components/theme-provider"
import { cn } from "@/lib/utils";

const geist = Geist({subsets:['latin'],variable:'--font-sans'})

const fontMono = Geist_Mono({
  subsets: ["latin"],
  variable: "--font-mono",
})

export default function RootLayout({
  children,
}: Readonly<{
  children: React.ReactNode
}>) {
  return (
    <html
      lang="en"
      suppressHydrationWarning
      className={cn("antialiased", fontMono.variable, "font-sans", geist.variable)}
    >
      <body>
        <ThemeProvider defaultTheme="dark" enableSystem={false}>
          {children}
        </ThemeProvider>
      </body>
    </html>
  )
}

```

### demo/app/login/page.tsx

```typescript
import type { Metadata } from "next"

import { LexiconAuthPage } from "@/rillwork/ui"

export const metadata: Metadata = {
  title: "Sign in — Lexicon",
}

export default function LoginPage() {
  return <LexiconAuthPage mode="signin" />
}

```

### demo/app/signup/page.tsx

```typescript
import type { Metadata } from "next"

import { LexiconAuthPage } from "@/rillwork/ui"

export const metadata: Metadata = {
  title: "Create a workspace — Lexicon",
}

export default function SignupPage() {
  return <LexiconAuthPage mode="signup" />
}

```

[38 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]