Maestro Helpcenter
Maestro Helpcenter
Breadcrumbs

Authoring tests with AI

How to author Maestro tests with an AI coding assistant — describe what to test in plain language, let the assistant draft the test as YAML, review the diff, and run it. For engineers setting up AI-assisted authoring and for anyone evaluating how the workflow works in practice.

How it works

Maestro test definitions are readable YAML stored in your own Git repository, and the runner code behind each step is plain Python or .NET. Because the format is open and text-based, an AI coding assistant can read and write it directly — there is no binary sequence file and no proprietary editor in the way.

The assistant connects to a station through the Maestro MCP server, which exposes the station's API as a set of tools the assistant can call (validate a definition, list available tests, start a run, search results). The assistant drafts and edits the YAML in your repository; you review every change as a normal Git diff before it runs. The AI does the typing; you keep the approval.

This is the same model whether you use GitHub Copilot, Claude, or Cursor — the only difference is the client configuration.

Before you start

You need:

  • A test package (a Git repository with your YAML definitions and runner code). See Using Maestro → Creating a Package.

  • The MCP server running on the station — it is part of the standard station stack and listens on port 7004.

  • An AI client configured to reach that MCP server.

Configure the client once. For VS Code / GitHub Copilot, add .mcp.json at the repository root:

json

{
  "mcpServers": {
    "maestro-station": {
      "type": "http",
      "url": "http://localhost:7004/mcp"
    }
  }
}

For Claude or Cursor, add the same URL under the client's MCP settings. To author against a remote station, replace localhost with the station IP and make sure port 7004 is reachable. See Integrations & Interfaces → MCP Server for multi-station setup and network requirements.

The authoring workflow

1. Describe the intent

Tell the assistant what to measure and the limits, in plain language — for example: "Add a step that measures the 3V3 rail and passes if it's between 3.2 V and 3.4 V." State the measurement, the limits, and any preconditions. You don't need to write the YAML structure yourself.

2. The assistant drafts the test

The assistant writes the step into your YAML definition, following the Maestro schema — one method per step, named measurements with low_limit/high_limit, step control fields where relevant. Where a step needs new runner logic, it adds the corresponding Python or .NET method. The output is readable YAML and code in your repository, not a generated black box.

3. Validate

Before anything runs, validate the definition. The assistant can call the validate_yaml MCP tool, or you can call it directly:

POST /api/yaml/validate
Content-Type: application/json

{ "yamlContent": "<yaml string>" }

An empty errors array means the definition is valid. Validation errors return with line numbers so they can be fixed before commit.

4. Review the diff

Review the change as a normal Git diff. Because the test is text, the diff shows exactly what changed — the step, the limits, the runner method. Nothing reaches a station until you approve and commit. This review step is where you confirm the limits and logic are what you intended.

5. Run

Run the test from the Test Monitor page in the UI, or have the assistant start it through the start_test MCP tool. Every measurement is stored as a structured, queryable row traced to the exact version of the test that produced it, so you can confirm the new step behaves as expected and review the result immediately.

What the assistant can and can't do

The MCP server gives the assistant a defined set of tools — across system, config, packages, execution, results, and YAML validation. Within that surface it can draft and validate test definitions, list and run tests, and search historical results.

It does not set your pass/fail limits for you — you state them, and they live in version control as part of the YAML. It does not bypass review: changes are diffs you approve. And it does not replace engineering judgment for steps involving visual inspection or physical adjustment. Treat the assistant as a fast, schema-aware author working under your sign-off, not an autonomous test designer.

Tips for good results

  • Give it your conventions. Point the assistant at an existing test in the package so it matches your naming, structure, and runner patterns.

  • Keep one method per step. It keeps each measurement independently named, searchable, and retryable — and it gives the assistant a clean unit to write against.

  • Always state limits explicitly. The engine evaluates limits from the YAML, not the runner, so the limit is only as good as what you specify.

  • Validate before you commit. Catch schema errors with validate_yaml before the diff, not at run time.

  • Run interactively once before enabling a new test in unattended or CI use.

  • Using Maestro — creating packages, running tests, viewing results.

  • How Maestro Works (Concepts) — the YAML execution model, runner model, and step control fields.

  • Integrations & Interfaces → MCP Server — full MCP tool list, multi-station configuration, and network requirements.