Using Maestro

Day-to-day task reference for operators, engineers, and anyone running, reviewing, or automating tests in Maestro.

Projects & Configuration

In Maestro, what other tools call a "project" is a test package — a versioned bundle of YAML test definitions and the runner code that executes them. Configuration is managed separately through Station Config, a two-tier key-value store that keeps hardware-specific values out of YAML files.

Creating a Package (Project)

1. Open the Maestro UI and navigate to Packages (/packages)

2. Click + New Package

3. Click Create. Maestro scaffolds the package directory with a package.json manifest, YAML test file stubs, and a validation script.

If you provided a GitLab URL, the package is immediately committed and pushed to that repository. It will appear in the package browser once the registry catalog is updated.

Editing Package Settings

Package metadata (name, version, description, lifecycle status) is defined in package.json at the root of the package repository. Edit the file directly in your editor and push the change. To update the station's copy:

1. Go to Packages → find the package

2. Click ↻ Refresh Registry to pull catalog changes

3. Click ↓ Download to fetch the updated version

To change a package's lifecycle status without editing the source repository — for example to mark a version obsolete on one station only — use the lifecycle override:

PATCH /api/packages/{name}/lifecycle
Content-Type: application/json

{ "status": "Obsolete" }

Valid values: NotReleased, Evaluation, Released, Obsolete.

Managing Station Configuration

Station Config holds hardware addresses, port assignments, fixture IDs, and any other values that differ between stations. YAML files reference these via {{cfg.KEY_NAME}} so the same test definition runs unchanged on every station.

View and edit configuration in the UI:

1. Navigate to <http://localhost:7001/station-config>

2. Configuration is shown in two sections:

   • Global — defaults shared by all stations

   • Station-local — overrides specific to this station

Add or update a value via the API:

# Add a station-local instrument address
$body = @{
    stationId   = "ST-01"
    key         = "DMM_VISA"
    value       = "TCPIP::192.168.1.100::INSTR"
    category    = "Instruments"
    description = "Digital multimeter address"
    updatedBy   = "admin"
} | ConvertTo-Json

Invoke-RestMethod -Uri "http://localhost:7000/api/config" `
    -Method PUT -Body $body -ContentType "application/json"

View the merged config for a station (what a running test actually sees):

Invoke-RestMethod -Uri "http://localhost:7000/api/config/merged/ST-01"

Configuration changes require an API restart before they take effect at runtime.

Importing and Exporting Configurations

Maestro does not have a dedicated import/export UI button, but every config value is accessible through the REST API, making it straightforward to script a full export and import.

Export all config for a station:

$config = Invoke-RestMethod -Uri "http://localhost:7000/api/config/merged/ST-01"
$config | ConvertTo-Json -Depth 5 | Out-File "station-config-ST-01.json"

Import config from a file:

$entries = Get-Content "station-config-ST-01.json" | ConvertFrom-Json
foreach ($entry in $entries) {
    $body = $entry | ConvertTo-Json
    Invoke-RestMethod -Uri "http://localhost:7000/api/config" `
        -Method PUT -Body $body -ContentType "application/json"
}

This approach is also suitable for provisioning a new station from a known-good configuration baseline, or for promoting configuration from a development station to production.

Running Tests

Running a Test Manually

1. Navigate to Test Monitor (/test-monitor)

2. The Test File Path field validates in real time:

   • Green border = file exists on the server, ready to run

   • Red border = file not found — check the path and re-enter

3. Click Start Test. The button is only enabled when the SignalR connection is active (green dot in the top bar) and the file path is valid.

While the test runs, the page shows three live panels:

Expand any step row to see individual measurement points: actual value, lower limit, upper limit, and pass/fail for each.

Operator Prompts

Some test steps pause and display a prompt asking the operator to perform a physical action, read an instrument, or confirm an observation. When a prompt appears:

• Read the message and complete the described action

• Enter a value if an input field is shown (the field type — numeric, text, boolean, or list — is determined by the test definition)

• Click the appropriate response button to continue

Entered values are recorded as measurements and are available to later steps. A numeric prompt validates the entered value against its configured limits before continuing.

Scheduling Test Runs

Maestro does not include a built-in scheduler. Scheduled or recurring runs are triggered externally using the REST API:

Trigger a run via PowerShell (Windows Task Scheduler)

$body = @{
    yamlFilePath = "/data/tat-packages/board-functional-test/tests/main.yaml"
    serialNumber = "NIGHTLY-$(Get-Date -Format 'yyyyMMdd')"
    operatorId   = "scheduler"
    stationId    = "ST-01"
} | ConvertTo-Json

Invoke-RestMethod -Uri "http://localhost:7000/api/testexecution/run" `
    -Method POST -Body $body -ContentType "application/json"

Trigger a run from a bash cron job (Linux)

#!/bin/bash
curl -s -X POST http://localhost:7000/api/testexecution/run \
  -H "Content-Type: application/json" \
  -d "{
    \"yamlFilePath\": \"/data/tat-packages/board-test/tests/main.yaml\",
    \"serialNumber\": \"NIGHTLY-$(date +%Y%m%d)\",
    \"operatorId\": \"scheduler\",
    \"stationId\": \"ST-01\"
  }"

Use GET /api/packages/test-files to discover the exact server-side YAML path for a given package before building your scheduled trigger.

Running Tests Headless / Unattended

For automated and CI use cases where no operator is present, pass unattendedMode: true in the execution request. This prevents prompt steps from blocking indefinitely.

$body = @{
    yamlFilePath  = "/data/tat-packages/board-test/tests/main.yaml"
    serialNumber  = "CI-001"
    operatorId    = "ci-pipeline"
    unattendedMode = $true
} | ConvertTo-Json

Invoke-RestMethod -Uri "http://localhost:7000/api/testexecution/run" `
    -Method POST -Body $body -ContentType "application/json"

How unattended mode handles prompt steps:

A warning log entry is written for every auto-responded prompt:

[UNATTENDED] Auto-responding to prompt 'Inspect solder joints' with button 'OK' (action: Continue)

Making a test unattended-safe:

• Every button-only prompt should have at least one button with action: Continue or action: Pass

• Every value-input prompt should declare input.default

• Validate with the YAML Validator page before deploying to an automated pipeline

• Run the test interactively at least once before enabling unattended mode in production

Unattended mode is not appropriate for tests involving visual inspection or physical adjustments that require human judgment. Auto-clicking through those steps may allow defective units to pass.

Stopping or Cancelling a Run

From the UI:

Click Abort in the execution status banner on the Test Monitor page. The current step is allowed to complete before execution halts. All results up to that point are saved.

From the API:

# Abort a specific execution
Invoke-RestMethod -Uri "http://localhost:7000/api/testexecution/{executionId}/abort" `
    -Method POST

The execution record is saved with verdict UNDETERMINED and all completed step results are preserved.

Results & Analysis

Viewing Test Results

Navigate to Test Results (/test-results). All historical executions are searchable.

Search filters:

Leave all fields blank and click Search to return the most recent executions.

Opening a detailed report:

Click View on any row. The report shows:

• Header: test name, serial number, operator, station, start/end time, duration, overall verdict

• Step Results table: every step with status, duration, verdict, and expandable measurements

• Inline Images: thumbnails of any PNG/JPEG/SVG artifacts emitted by steps (click to view full size)

• Interactive Charts: Plotly figures rendered with full zoom, pan, and hover — useful for waveform and spectrum data

• Nested Executions: sub-sequences shown as collapsible sections

Each report has a permanent URL (/test-results/{id}) that can be bookmarked or shared.

Understanding Pass / Fail Logic

Verdicts are determined at three levels:

Measurement level — a single numeric data point:

• PASS if low_limit ≤ value ≤ high_limit

• FAIL if the value falls outside the limits

• UNDETERMINED if the measurement could not be taken

Step level — the worst verdict across all measurements in that step:

• If any measurement fails → step is FAIL

• If all measurements pass → step is PASS

• If no measurements were taken → UNDETERMINED

Execution level — the worst verdict across all steps:

• If any step fails → execution is FAIL

• If all steps pass → execution is PASS

• If any step is undetermined and none fail → UNDETERMINED

Step control behaviour — how the test proceeds after a failure is defined in YAML per step:

Skipped steps (excluded by a precondition that evaluated to false) do not affect the execution verdict.

Exporting Results

Print or save as PDF:

Open any report and use your browser's Print function (Ctrl+P). The report layout hides navigation chrome when printing and is formatted for A4/Letter. Interactive charts are hidden during print — use the Plotly toolbar's PNG export button to include a chart snapshot before printing.

Export via API:

# Get the full execution report as JSON
Invoke-RestMethod -Uri "http://localhost:7000/api/testexecution/{executionId}/report"

# Get all measurements for an execution
Invoke-RestMethod -Uri "http://localhost:7000/api/testexecution/{executionId}/measurements"

# Get execution logs
Invoke-RestMethod -Uri "http://localhost:7000/api/testexecution/{executionId}/logs"

Bulk export with filtering:

# All FAIL results in a date range
$params = "verdict=FAIL&from=2026-01-01T00:00:00Z&to=2026-01-31T23:59:59Z"
Invoke-RestMethod -Uri "http://localhost:7000/api/testresults?$params"

Database queries (for bulk analysis and SPC):

Because measurements are stored as structured relational rows — not log strings — they are directly queryable with SQL:

-- Yield rate for a specific test over the last 30 days
SELECT
    COUNT(*) FILTER (WHERE verdict = 'PASS') AS pass_count,
    COUNT(*) AS total_count,
    ROUND(100.0 * COUNT(*) FILTER (WHERE verdict = 'PASS') / COUNT(*), 1) AS yield_pct
FROM test_executions
WHERE test_name = 'board-functional-test'
  AND started_at >= NOW() - INTERVAL '30 days';

-- Measurement drift for a specific limit over time
SELECT started_at, actual_value, low_limit, high_limit
FROM measurements
WHERE measurement_name = 'VOUT_3V3'
  AND station_id = 'ST-01'
ORDER BY started_at DESC
LIMIT 500;

Comparing Runs

The Maestro UI shows one execution at a time. For side-by-side comparison, use the API or database directly.

Compare two specific executions:

$run1 = Invoke-RestMethod -Uri "http://localhost:7000/api/testexecution/101/report"
$run2 = Invoke-RestMethod -Uri "http://localhost:7000/api/testexecution/102/report"

# Compare step verdicts
$run1.steps | Select-Object name, verdict | Format-Table
$run2.steps | Select-Object name, verdict | Format-Table

Compare measurement values for the same serial number across runs:

SELECT
    e.started_at,
    e.station_id,
    m.measurement_name,
    m.actual_value,
    m.low_limit,
    m.high_limit,
    m.verdict
FROM measurements m
JOIN test_executions e ON m.execution_id = e.id
WHERE e.serial_number = 'UNIT-042'
  AND m.measurement_name = 'VOUT_3V3'
ORDER BY e.started_at DESC;

Automation

Creating Automated Workflows

The Maestro REST API exposes the full execution model. Any system that can make an HTTP request can trigger, monitor, and retrieve test results.

Minimal trigger-and-poll workflow:

# 1. Start the test
$body = @{
    yamlFilePath   = "/data/tat-packages/board-test/tests/main.yaml"
    serialNumber   = "UNIT-042"
    operatorId     = "automation"
    unattendedMode = $true
} | ConvertTo-Json

$result = Invoke-RestMethod `
    -Uri "http://localhost:7000/api/testexecution/run" `
    -Method POST -Body $body -ContentType "application/json"

$executionId = $result.executionId

# 2. Poll until terminal state
do {
    Start-Sleep -Seconds 2
    $status = Invoke-RestMethod `
        -Uri "http://localhost:7000/api/testexecution/$executionId/status"
} while ($status.state -notin @("Completed", "Failed", "Aborted"))

# 3. Retrieve the report
$report = Invoke-RestMethod `
    -Uri "http://localhost:7000/api/testexecution/$executionId/report"

Write-Host "Verdict: $($report.verdict)"
exit $(if ($report.verdict -eq 'PASS') { 0 } else { 1 })

Real-time monitoring with SignalR (for low-latency pipelines):

For production automation where polling latency is unacceptable, connect to the SignalR hub and subscribe to execution events. This is the same channel the Maestro UI uses and delivers step updates, measurement values, and completion events as they happen.

// JavaScript example — e.g., in a Node.js CI bridge
const { HubConnectionBuilder } = require("@microsoft/signalr");

const connection = new HubConnectionBuilder()
    .withUrl("http://localhost:7000/hubs/testexecution")
    .build();

connection.on("StepCompleted", (step) => {
    console.log(`[${step.verdict}] ${step.name}`);
});

connection.on("ExecutionCompleted", (execution) => {
    console.log(`Final verdict: ${execution.verdict}`);
    process.exit(execution.verdict === "PASS" ? 0 : 1);
});

await connection.start();

Triggering Maestro from External Systems

From a GitLab CI pipeline:

# .gitlab-ci.yml
hardware-test:
  stage: test
  script:
    - |
      EXECUTION=$(curl -sf -X POST http://$MAESTRO_HOST:7000/api/testexecution/run \
        -H "Content-Type: application/json" \
        -d "{
          \"yamlFilePath\": \"$TEST_FILE_PATH\",
          \"serialNumber\": \"CI-$CI_PIPELINE_ID\",
          \"operatorId\": \"gitlab-ci\",
          \"unattendedMode\": true
        }")
      EXECUTION_ID=$(echo $EXECUTION | jq -r .executionId)
    - |
      while true; do
        STATUS=$(curl -sf http://$MAESTRO_HOST:7000/api/testexecution/$EXECUTION_ID/status)
        STATE=$(echo $STATUS | jq -r .state)
        [ "$STATE" = "Completed" ] || [ "$STATE" = "Failed" ] || [ "$STATE" = "Aborted" ] && break
        sleep 3
      done
    - |
      VERDICT=$(curl -sf http://$MAESTRO_HOST:7000/api/testexecution/$EXECUTION_ID/report | jq -r .verdict)
      [ "$VERDICT" = "PASS" ] || exit 1
  variables:
    MAESTRO_HOST: "maestro-station-01.internal"
    TEST_FILE_PATH: "/data/tat-packages/board-test/tests/main.yaml"

From a GitHub Actions workflow:

- name: Run Maestro test
  run: |
    RESULT=$(curl -sf -X POST http://$MAESTRO_HOST:7000/api/testexecution/run \
      -H "Content-Type: application/json" \
      -d '{
        "yamlFilePath": "${{ env.TEST_FILE_PATH }}",
        "serialNumber": "GH-${{ github.run_id }}",
        "operatorId": "github-actions",
        "unattendedMode": true
      }')
    EXECUTION_ID=$(echo $RESULT | jq -r .executionId)
    echo "EXECUTION_ID=$EXECUTION_ID" >> $GITHUB_ENV

- name: Wait for result
  run: |
    while true; do
      STATE=$(curl -sf http://$MAESTRO_HOST:7000/api/testexecution/$EXECUTION_ID/status | jq -r .state)
      [[ "$STATE" =~ ^(Completed|Failed|Aborted)$ ]] && break
      sleep 3
    done
    VERDICT=$(curl -sf http://$MAESTRO_HOST:7000/api/testexecution/$EXECUTION_ID/report | jq -r .verdict)
    [[ "$VERDICT" == "PASS" ]] || (echo "Test failed with verdict: $VERDICT" && exit 1)

From a MES (Manufacturing Execution System):

Maestro exposes a POST /api/testexecution/run endpoint that accepts a serial number and returns an execution ID. The MES can:

1. Submit a run when a unit arrives at the test station (pass the production serial number)

2. Poll GET /api/testexecution/{id}/status for completion

3. Retrieve the verdict from GET /api/testexecution/{id}/report

4. Write the verdict and measurement data back to the MES

For advanced MES integration, Maestro includes a pluggable IMesService interface with lifecycle hooks for routing validation (can the unit be tested on this station?), pre-run validation (is the unit eligible?), and result reporting (push verdict back to MES). Contact the Maestro team for implementation details.

Best Practices for Stable Automation

Discover test file paths before hardcoding them. The server-side path for a YAML file depends on where the package is installed and is not predictable from the package name alone. Always call GET /api/packages/test-files to retrieve the exact path, and store that value in your pipeline configuration.

# List all available test files with their server-side paths
Invoke-RestMethod -Uri "http://localhost:7000/api/packages/test-files"

Always enable unattendedMode in automated pipelines. Without it, any prompt step will block indefinitely and your pipeline will hang. Review every test that runs in automation for prompt safety before enabling.

Wait for runner warm-up after a fresh package install. The first execution after installing a new package version may fail with runner unavailable while the runner loads the new assembly. Wait 5–10 seconds after install before triggering the first run, or poll GET /api/health until all runners report healthy.

Use execution IDs, not serial numbers, for polling. The execution ID returned from POST /api/testexecution/run is the stable handle for that specific run. Serial numbers are not unique across time.

Build in retry logic for transient network faults. The REST API is synchronous and stateless — if a trigger request fails due to a network blip, there is no partial state to clean up. Wrap your trigger call in a retry loop with a short backoff (2–3 attempts, 5-second intervals).

Set timeouts in your pipeline. A stalled runner will never transition to a terminal state on its own. Set a pipeline timeout appropriate for the expected test duration, and call the abort endpoint if the timeout is reached.

$timeoutSeconds = 300
$elapsed = 0

do {
    Start-Sleep -Seconds 5
    $elapsed += 5
    $status = Invoke-RestMethod -Uri "http://localhost:7000/api/testexecution/$executionId/status"

    if ($elapsed -ge $timeoutSeconds) {
        Invoke-RestMethod -Uri "http://localhost:7000/api/testexecution/$executionId/abort" -Method POST
        throw "Test execution timed out after $timeoutSeconds seconds"
    }
} while ($status.state -notin @("Completed", "Failed", "Aborted"))

Capture the config snapshot for traceability. Every execution record stores the exact cfg.* values that were active at the time. Retrieve this with GET /api/testexecution/{id}/report and archive it alongside your CI artifacts so you can reproduce the exact conditions of any historical run.