SlopGuard

A fascinating challenge highlighted in a 2025 USENIX Security study. Researchers evaluated 16 code-generation models across 576,000 Python and JavaScript code samples and discovered 205,474 unique hallucinated package names—packages that simply didn't exist. It's called slopsquatting. These fake package names being exploited by attackers as a weapon, register them on npm or PyPI, embed malware, and wait for developers to unknowingly install them.

We saw an opportunity to contribute by building an intelligent review layer that automatically validates AI-generated dependencies before code is merged or deployed, helping developers catch hallucinated or malicious packages early. Our view is, if AI is accelerating software development, it should also be accompanied by smarter safeguards that make the ecosystem safer for everyone i.e. SlopGuard. Read the paper and results.

What it reviews

1. Reads the PR description and changed-file patches through GitHub's pull request API.
2. Extracts package imports from added JavaScript/TypeScript import and require statements, and Python import / from … import statements.
3. Queries npm or PyPI concurrently for each unique package. Both registries provide a registration age; packages registered in the last 30 days are marked for provenance review. For those new packages, SlopGuard also uses release count and the presence of source/repository and maintainer metadata as soft review cues.
4. Applies two transparent heuristics: a description-to-files-changed ratio and removed auth/permission lines with no same-category replacement in that file.
5. Sends structured signals only—never the raw diff—to GPT-5.6 using Structured Outputs. The model returns a 0–100 score, level, explanation, flags, and packages to verify.
6. Posts one advisory issue comment on the PR. If GPT-5.6 is unavailable or returns malformed data, SlopGuard posts a deterministic fallback assessment with a visible disclosure instead of hiding the observed signals.
7. Persists names confirmed missing from a registry in a bounded repository Actions variable. On a later PR, it identifies a repeat offender and cites the previous PR number.

An unavailable npm or PyPI request is always reported as couldn't verify; it is never treated as evidence that a package does not exist.

Install

This repository already contains the workflow at .github/workflows/slopguard.yml. Add an OPENAI_API_KEY repository secret, then commit the repository to the project you want to protect. The workflow runs for opened, synchronize, and reopened pull_request events and asks for contents: read, pull-requests: write, and actions: write permissions. The last permission only enables bounded repeat-offender memory; the review itself still works if a repository policy denies it.

To call the action from another workflow:

name: SlopGuard

on:
  pull_request:
    types: [opened, synchronize, reopened]

permissions:
  actions: write
  contents: read
  pull-requests: write

concurrency:
  group: slopguard-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  dependency-review:
    # This complements SlopGuard's import heuristics with GitHub's manifest
    # and lockfile dependency diff. Keep it advisory to match SlopGuard.
    if: ${{ github.event.pull_request.head.repo.fork == false }}
    permissions:
      contents: read
    runs-on: ubuntu-latest
    steps:
      - name: Check out pull request
        uses: actions/checkout@v4
      - name: Dependency Review
        uses: actions/dependency-review-action@v4
        continue-on-error: true
        with:
          deny-licenses: ${{ vars.SLOPGUARD_DENY_LICENSES }}
  review:
    if: ${{ github.event.pull_request.head.repo.fork == false }}
    runs-on: ubuntu-latest
    steps:
      - name: Run SlopGuard
        uses: vishal1610kamal/slopcheck@main
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          pr-number: ${{ github.event.pull_request.number }}
          repo-owner: ${{ github.repository_owner }}
          repo-name: ${{ github.event.repository.name }}
          deny-licenses: ${{ vars.SLOPGUARD_DENY_LICENSES }}

The action defaults to gpt-5.6; set its optional model input or the OPENAI_MODEL environment variable to override it. GPT-5.6 and the Responses API's JSON-schema Structured Outputs are documented by OpenAI.

Optional policy configuration

Set the repository Actions variable SLOPGUARD_DENY_LICENSES to a comma- separated list of SPDX identifiers, such as GPL-3.0-only,AGPL-3.0-only. SlopGuard then flags a newly imported package whose registry license field matches one of those identifiers. This is an advisory policy match, not a license-compliance engine or legal determination.

SlopGuard stores confirmed missing package names in the non-secret repository Actions variable SLOPGUARD_REPEAT_OFFENDERS. It is capped at 100 package names and 20 PR numbers per name. Delete that variable in repository settings to reset the memory. The workflow needs actions: write to update it; if an organization policy denies that permission, the review still runs but the memory feature is disabled for that run.

Dependency-manifest review

The bundled workflow also runs GitHub's Dependency Review action alongside SlopGuard. It covers dependency-manifest and lockfile changes, including GitHub's vulnerability and license data, instead of duplicating that analysis in SlopGuard. Its step uses continue-on-error: true, so it remains advisory and does not become a merge gate. Copy this separate job when installing SlopGuard in another repository. See GitHub's Dependency Review documentation for the managed dependency-diff checks it supplies:

dependency-review:
  if: ${{ github.event.pull_request.head.repo.fork == false }}
  permissions:
    contents: read
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - uses: actions/dependency-review-action@v4
      continue-on-error: true
      with:
        deny-licenses: ${{ vars.SLOPGUARD_DENY_LICENSES }}

Run and test locally

Node.js 20 or newer is required. Create a token with permission to read PRs and write PR comments, then run:

npm ci
npm test

GITHUB_TOKEN=... \
OPENAI_API_KEY=... \
PR_NUMBER=123 \
REPO_OWNER=your-org \
REPO_NAME=your-repository \
node slopguard.js

The test suite uses Node's built-in test runner and mocks network and model calls. It covers import extraction, registry outcomes (including failure), parallel checks, scope creep, auth removal, structured synthesis, and comment wording.

Security and limitations

• The bundled workflow deliberately uses pull_request, checks out the base commit's action code, and skips forked PRs. GitHub withholds repository secrets and makes the token read-only for forked PRs; using pull_request_target merely to expose an OpenAI key would be an unsafe trade-off. A trusted, separately designed fork-review flow is outside v1.
• It only examines patches returned by GitHub. Binary files and very large diffs can lack a patch, so SlopGuard does not infer their contents.
• Dynamic imports, re-exports, dependency manifests, and ecosystems other than npm/PyPI are out of scope. Python import names do not always equal PyPI distribution names, so SlopGuard maps a few common cases (for example, yaml → PyYAML) but cannot reliably distinguish every local module from a third-party dependency.
• Registry existence and registry metadata—license, age, release count, source URL, and maintainer presence—do not prove a package is safe or legally acceptable. A newly registered package, thin metadata, or denied-license match is a review cue, not a maliciousness or legal claim.
• Scope and auth findings are regex/size heuristics, not data-flow or control-flow proofs. The PR comment states this explicitly.
• This v1 always comments; it never blocks, requests changes, or installs a package.

GitHub's REST API provides the changed-file endpoint and uses issue comments for whole-PR comments; the implementation follows those documented interfaces. GitHub changed files and PR comments are the relevant endpoints. Package existence comes from npm's public registry and PyPI's GET /pypi/<project>/json endpoint. PyPI's JSON API documentation describes the release upload timestamps and its metadata limitations.

Prior art and differentiation

slopcheck already provides broad, multi-ecosystem package checks, typosquat matching, allowlists, and auto-fix. Open Code Review is a broader multi-layer code-review action with optional LLM review. SlopGuard is deliberately narrower: npm and PyPI only, plus scope and auth heuristics, with GPT-5.6 synthesizing the independent signals into one human-oriented PR comment. It does not claim that simple registry checking is novel.

How it was built

SlopGuard built using Node.js 20, the GitHub Octokit SDK, and the OpenAI SDK, keeping the architecture intentionally simple. Before involving AI, SlopGuard performs its own deterministic security checks—such as validating package names against npm and PyPI, detecting unusual scope changes, and identifying removed authentication or authorization checks. Only these structured findings are sent to GPT for analysis, never the raw Pull Request code or diff. GPT then combines the evidence into a single risk score with a clear explanation and actionable recommendations for reviewers.

One of our biggest design decisions was security. Although GitHub provides workflows that can access repository secrets on Pull Requests from forks, we intentionally avoided that approach because it could expose sensitive credentials to untrusted code. Instead, SlopGuard runs only in a secure workflow, uses trusted repository code, and skips forked Pull Requests in the first version. We also leveraged AI tools like Codex to accelerate development by generating the initial scaffolding, test cases, and registry integrations, while the security architecture, review logic, and AI decision-making workflow were carefully designed and implemented by us.

What we learned

One of our biggest takeaways was that AI works best as a decision-support tool, not as a replacement for security checks. SlopGuard first performs reliable rule-based validations, and then GPT brings those findings together into a simple, easy-to-understand risk summary that helps reviewers make faster and better decisions instead of interpreting multiple security signals on their own.

We also learned how much AI can accelerate development. Codex helped us quickly build the initial code, registry integrations, and test cases, allowing us to focus our time on the parts that mattered most-designing a secure architecture, deciding what should be checked, and ensuring the tool remained transparent and trustworthy for developers.

Deliberately left for later

This is just the first version of SlopGuard, and we already have a clear roadmap for improving it. In future releases, we plan to suggest the most likely legitimate package when a hallucinated package is detected, use AI-powered tool calling to make package validation faster and more efficient, and leverage GPT with web search to verify packages that exist but appear suspicious or lack enough credibility. We also plan to update the same Pull Request comment instead of creating a new one after every code change, making reviews cleaner and less noisy.

We intentionally left these features out of the first release so we could focus on building a solution that is simple, reliable, and trustworthy. Our priority was to get the core security checks right before adding more advanced capabilities.