Skip to content

Claude Code Test Runner

An intelligent end-to-end testing agent that executes web application tests using natural language descriptions instead of rigid code-based test scripts.

Overview

The Claude Code Test Runner bridges the gap between traditional automated testing and manual sanity checks by using AI to execute tests with human-like intuition. Tests describe what to test rather than how to test it, making them more resilient to UI changes and easier to maintain.

Key Features

Natural Language Testing

  • Tests written as sequential natural language steps
  • No need to specify exact selectors or interaction methods
  • AI adapts to UI changes and finds elements contextually

Intelligent Adaptation

  • Visual understanding of current UI state
  • Context-based element selection
  • Handles unexpected changes during test execution
  • Resilient to network issues and transient problems

Production Ready

  • CLI tool for local and CI/CD execution
  • Docker container for GitHub Actions integration
  • Comprehensive logging and error reporting
  • OAuth and API key authentication options

Architecture

The system consists of three main components:

Test Runner CLI

  • Bun-based orchestrator that manages test execution
  • Handles authentication and configuration
  • Coordinates between MCP servers and Claude Code

MCP Servers

  • Playwright MCP: Browser automation and interaction
  • Test State MCP: Tracks test execution state and results

Claude Code SDK

  • Executes individual test steps using AI reasoning
  • Interprets natural language instructions
  • Makes adaptive decisions about UI interactions

Benefits

  • Low Maintenance: Tests remain valid even when UI elements change
  • High Reliability: AI adapts to loading states and handles edge cases
  • Faster Development: Tests written in plain English by any team member

For detailed usage examples, setup instructions, and implementation guides, see the GitHub repository.

Demo & Discussion

Resources & References