Oorty - Testing for AI-Generated Apps

How It Works

When you run npx oorty plan (Pro), Oorty analyzes your codebase and writes a test plan (Markdown and JSON under .oorty/plans/ by default) that maps what should be tested and why.

Static Analysis

Scans your codebase

Analyzes components, utilities, hooks, and API routes to understand your application structure.

Priority Ranking

Focuses on what matters

Prioritizes tests based on business criticality, complexity, and change frequency.

Coverage Gaps

Identifies missing tests

Compares existing tests against your codebase to find untested critical paths.

Smart Suggestions

Recommends test types

Suggests whether code needs unit, component, or E2E tests based on its purpose.

Generated Test Plan

By default the CLI writes .oorty/plans/test-plan.md and .oorty/plans/test-plan.json. The JSON is what oorty generate reads. Shape is similar to:

test-plan.json (illustrative)

{
  "version": "1.0.0",
  "generatedAt": "2024-01-15T10:30:00Z",
  "summary": {
    "totalFiles": 47,
    "analyzedFiles": 42,
    "existingTests": 12,
    "recommendedTests": 35,
    "coverageGap": "65%"
  },
  "priorities": {
    "critical": [
      {
        "file": "lib/auth/session.ts",
        "reason": "Handles user authentication - security critical",
        "suggestedTests": ["unit"],
        "scenarios": [
          "Valid session token validation",
          "Expired session handling",
          "Invalid token rejection",
          "Session refresh flow"
        ]
      },
      {
        "file": "app/api/checkout/route.ts",
        "reason": "Payment processing - business critical",
        "suggestedTests": ["unit", "e2e"],
        "scenarios": [
          "Successful payment processing",
          "Payment failure handling",
          "Inventory validation",
          "Order confirmation"
        ]
      }
    ],
    "high": [
      {
        "file": "components/cart/cart-provider.tsx",
        "reason": "Core state management for shopping cart",
        "suggestedTests": ["component"],
        "scenarios": [
          "Add item to cart",
          "Remove item from cart",
          "Update item quantity",
          "Cart persistence",
          "Cart total calculation"
        ]
      }
    ],
    "medium": [
      {
        "file": "components/ui/product-card.tsx",
        "reason": "Reusable product display component",
        "suggestedTests": ["component"],
        "scenarios": [
          "Renders product details",
          "Shows sale price",
          "Handles out of stock state",
          "Add to cart button interaction"
        ]
      }
    ],
    "low": [
      {
        "file": "lib/utils/format-currency.ts",
        "reason": "Pure utility function",
        "suggestedTests": ["unit"],
        "scenarios": [
          "Formats USD",
          "Handles negative values",
          "Handles zero"
        ]
      }
    ]
  },
  "e2eFlows": [
    {
      "name": "Complete Purchase Flow",
      "priority": "critical",
      "steps": [
        "User browses products",
        "User adds item to cart",
        "User proceeds to checkout",
        "User enters shipping info",
        "User completes payment",
        "User receives confirmation"
      ]
    },
    {
      "name": "User Authentication",
      "priority": "critical",
      "steps": [
        "User signs up",
        "User verifies email",
        "User logs in",
        "User updates profile",
        "User logs out"
      ]
    }
  ],
  "recommendations": [
    "Add tests for auth session handling - currently untested",
    "Payment flow has no E2E coverage - high risk area",
    "Cart provider tests would prevent regression in core feature",
    "Consider adding visual regression tests for product cards"
  ]
}

Using the Test Plan

The test plan serves as a roadmap for your testing efforts. You can use it to:

Generate tests from the plan

After oorty plan, run oorty generate to create test files from .oorty/plans/test-plan.json (override with -p / --plan).

npx oorty generate
npx oorty generate --priority 1
npx oorty generate --dry-run

Refresh the plan

When your code changes, run oorty plan again to update the Markdown and JSON, then re-run oorty generate as needed.

npx oorty plan
npx oorty coverage

Add one-off tests

For a single template without the plan pipeline, use oorty add:

npx oorty add unit --name format-utils
npx oorty add component --name header
npx oorty add e2e --name smoke

Priority Levels

Critical

Security-sensitive code, payment processing, authentication. Must have comprehensive test coverage.

High

Core business logic, frequently used components, data mutations. Should have good coverage.

Medium

UI components, display logic, helpers. Good to have tests, lower risk of business impact.

Low

Pure utilities, simple formatters, configuration. Quick to test, provides confidence.

Test Plan Generation