> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.rubrichq.io/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.rubrichq.io/_mcp/server. # GitHub Actions CI/CD Run your RubricHQ agent test suites automatically on every push or pull request and **block deploys when agents regress**. The integration works via a published GitHub Action (`RubricHQ/agent-test-action`) or through the raw REST API — use the API path for GitLab CI, Jenkins, CircleCI, or any other CI platform. --- ## Prerequisites * An agent with at least one scenario in RubricHQ. * A RubricHQ API key — **Settings → API Keys → Create key**. * In your GitHub repo: * **Secret** `RUBRICHQ_API_KEY` — your API key (Settings → Secrets and variables → Actions → New repository secret). * **Variable** `RUBRICHQ_AGENT_ID` — the numeric ID of the agent to test (Settings → Secrets and variables → Actions → Variables tab → New repository variable). --- ## Using the GitHub Action ### Quick-start workflow Add this file to `.github/workflows/agent-tests.yml` in your repository: ```yaml name: Agent Tests on: push: branches: [main] workflow_dispatch: jobs: agent-tests: runs-on: ubuntu-latest steps: - name: Run RubricHQ agent tests uses: RubricHQ/agent-test-action@v1 with: api_key: ${{ secrets.RUBRICHQ_API_KEY }} agent_id: ${{ vars.RUBRICHQ_AGENT_ID }} scenario_ids: "12,15,22" ``` The step exits `0` when the verdict is `passed` and `1` when it is `failed` or the timeout is reached — so the job fails exactly when your agents regress. ### Inputs | Input | Required | Default | Description | | ------------------- | -------- | ------------------------- | ------------------------------------------------------------------- | | `api_key` | yes | — | RubricHQ API key. Use a secret: `${{ secrets.RUBRICHQ_API_KEY }}`. | | `agent_id` | yes | — | Numeric ID of the agent to test. | | `scenario_ids` | **yes** | — | Comma-separated list of scenario IDs to run (e.g. `12,15,22`). | | `tags` | no | — | Optional. Tags to *also* include, on top of `scenario_ids` (union). | | `frequency` | no | `1` | How many times to run each scenario. Accepts `1`–`5`. | | `success_threshold` | no | `100` | Minimum pass-rate (0–100) required for the verdict to be `passed`. | | `timeout` | no | `3600` | Seconds to wait for the run to complete before failing the step. | | `poll_interval` | no | `15` | Seconds between status-poll requests. | | `api_url` | no | `https://api.rubrichq.io` | Override for self-hosted or staging deployments. | `scenario_ids` is required. `tags` is optional — when provided, matching scenarios are added on top (union). ### Outputs | Output | Description | | ------------- | ------------------------------------------------------ | | `test_run_id` | Numeric ID of the test run created. | | `verdict` | `passed` or `failed`. | | `pass_rate` | Percentage of runs that passed (e.g. `80.0`). | | `report_url` | Link to the full run report in the RubricHQ dashboard. | ### Testing over web, phone, or text The `channel` input controls how each scenario is run: * `web` — a browser/WebSocket voice call. * `phone` — a real phone call placed over Twilio (the agent must have a phone number configured). * `text` — a text-only conversation. When `channel` is omitted, it defaults to `phone` if the agent has a phone number, otherwise `web`. Run a single channel by setting `channel` on the step: ```yaml - uses: RubricHQ/agent-test-action@v1 with: api_key: ${{ secrets.RUBRICHQ_API_KEY }} agent_id: ${{ vars.RUBRICHQ_AGENT_ID }} scenario_ids: "12,15,22" channel: web # or: phone, text ``` To exercise both web and phone on every push, use two jobs — each reports its own verdict, and either failing blocks the deploy: ```yaml jobs: web-tests: runs-on: ubuntu-latest steps: - uses: RubricHQ/agent-test-action@v1 with: api_key: ${{ secrets.RUBRICHQ_API_KEY }} agent_id: ${{ vars.RUBRICHQ_AGENT_ID }} scenario_ids: "12,15,22" channel: web phone-tests: runs-on: ubuntu-latest steps: - uses: RubricHQ/agent-test-action@v1 with: api_key: ${{ secrets.RUBRICHQ_API_KEY }} agent_id: ${{ vars.RUBRICHQ_AGENT_ID }} scenario_ids: "12,15,22" channel: phone ``` > **Note** > > The two jobs run in parallel. To stop a flaky phone job from blocking the deploy while you stabilize it, add `continue-on-error: true` to the phone job's step — it still reports a verdict but won't fail the workflow. ### Gating deploys Add a `deploy` job that only runs after `agent-tests` passes: ```yaml jobs: agent-tests: runs-on: ubuntu-latest steps: - name: Run RubricHQ agent tests uses: RubricHQ/agent-test-action@v1 with: api_key: ${{ secrets.RUBRICHQ_API_KEY }} agent_id: ${{ vars.RUBRICHQ_AGENT_ID }} scenario_ids: "12,15,22" deploy: needs: agent-tests runs-on: ubuntu-latest steps: - name: Deploy run: ./scripts/deploy.sh ``` If `agent-tests` fails, `deploy` is skipped automatically. ### Staging → production pipeline Use two sequential jobs with different secrets to promote only builds that pass in staging first: ```yaml jobs: test-staging: runs-on: ubuntu-latest steps: - name: Run agent tests against staging uses: RubricHQ/agent-test-action@v1 with: api_key: ${{ secrets.RUBRICHQ_API_KEY_STAGING }} agent_id: ${{ vars.RUBRICHQ_AGENT_ID_STAGING }} scenario_ids: "12,15,22" success_threshold: 90 test-production: needs: test-staging runs-on: ubuntu-latest steps: - name: Run agent tests against production uses: RubricHQ/agent-test-action@v1 with: api_key: ${{ secrets.RUBRICHQ_API_KEY_PROD }} agent_id: ${{ vars.RUBRICHQ_AGENT_ID_PROD }} scenario_ids: "12,15,22" success_threshold: 100 ``` --- ## Using the API directly For GitLab CI, Jenkins, CircleCI, or any CI system that can run shell commands, call the REST API directly. ### Trigger a test run ```bash curl -sS -X POST https://api.rubrichq.io/api/public/v1/test_runs \ -H "Authorization: Bearer $RUBRICHQ_API_KEY" \ -H "Content-Type: application/json" \ -d '{"agent_id": 1, "scenario_ids": [12, 15, 22], "success_threshold": 90}' ``` The API returns `202 Accepted` immediately — the run is queued, not yet complete: ```json { "test_run_id": 42, "status": "pending", "run_count": 5, "scenario_count": 5, "frequency": 1, "success_threshold": 90, "status_url": "https://api.rubrichq.io/api/public/v1/test_runs/42", "report_url": "https://app.rubrichq.io/batch-run/42" } ``` ### Request fields | Field | Type | Required | Description | | ------------------- | -------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `agent_id` | integer | yes | The agent to test. | | `scenario_ids` | int array or comma-separated string | **yes** | Scenarios to run. | | `tags` | string array or comma-separated string | no | Optional. Also include scenarios matching these tags (union). | | `frequency` | integer | no | Runs per scenario (`1`–`5`, default `1`). | | `success_threshold` | integer | no | Minimum pass-rate to pass (`0`–`100`, default `100`). | | `testing_mode` | string | no | `voice` or `text` (default `voice`). | | `channel` | string | no | `phone`, `web`, or `text`. Defaults to `phone` when the agent has a phone number, otherwise `web`. | | `name` | string | no | Human-readable label for the run (shows in the dashboard). | | `ci_metadata` | object | no | Arbitrary JSON stored with the run and echoed back in status responses. Put commit SHA, branch name, and CI run URL here for traceability. | `scenario_ids` is required; `tags` is an optional additional filter. ### Poll for the verdict Voice scenarios take several minutes each. Poll the `status_url` until `verdict` is no longer `pending`: ```bash TEST_RUN_ID=42 while true; do verdict=$(curl -sS "https://api.rubrichq.io/api/public/v1/test_runs/$TEST_RUN_ID" \ -H "Authorization: Bearer $RUBRICHQ_API_KEY" | python3 -c 'import json,sys; print(json.load(sys.stdin)["verdict"])') [ "$verdict" != "pending" ] && break sleep 15 done [ "$verdict" = "passed" ] || exit 1 ``` ### Status response ```json { "id": 42, "status": "completed", "verdict": "failed", "pass_rate": 80.0, "success_threshold": 90, "runs": { "total": 5, "completed": 5, "running": 0, "pending": 0, "failed": 0, "passed": 4 }, "failed_runs": [ { "run_id": 207, "scenario_name": "Angry refund caller", "status": "completed", "reason": "critical metric failed: Greeting Check" } ], "ci_metadata": { "sha": "abc123", "branch": "main" }, "report_url": "https://app.rubrichq.io/batch-run/42", "created_at": "2026-06-11T10:00:00Z", "updated_at": "2026-06-11T10:12:31Z" } ``` --- ## How the verdict works A **run** passes when all of its critical metrics pass (runs with no critical metrics always pass on that dimension). Calls that error out count as failed runs. The **pass rate** is `passed_runs / total_runs × 100`. The **verdict** is `passed` when `pass_rate >= success_threshold`, and `failed` otherwise. The verdict stays `"pending"` until every call in the batch has completed **and** all metric evaluations for those calls have finished. Poll until the verdict is no longer `"pending"` — don't rely on `status == "completed"` alone, because metric evaluation runs asynchronously after call completion. --- ## Troubleshooting | Symptom | Likely cause | Fix | | ------------------------------------------------------------- | -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | Step times out before run completes | Large suite with long voice scenarios (each can take several minutes). | Raise the `timeout` input (Action) or extend your CI job's timeout. A 20-scenario suite at `frequency: 2` can easily run for 40+ minutes. | | `"No scenarios matched tags"` error | Tags are exact-match strings — case and whitespace matter. | Check the scenario tags in the RubricHQ app under **Scenarios** and make sure they match exactly what you're passing. | | HTTP `402 Payment Required` | The workspace has run out of credits. | Top up credits in **Settings → Billing**, then re-run. | | HTTP `422 Unprocessable Entity` listing specific scenario IDs | Those scenario IDs are not attached to the agent, or the scenarios are archived. | Verify the scenario IDs under **Agent → Scenarios**; archived scenarios must be restored before they can be run. | > **Tip** > > Use `ci_metadata` to attach `{"sha": "$COMMIT_SHA", "branch": "$BRANCH", "ci_run": "$CI_RUN_URL"}` to every test run. It's stored on the run and echoed back in status responses, so you can trace any dashboard report back to the exact commit that triggered it.