Test your checkout on every deployment with GitHub Actions

The full workflow, ready to copy

Here is the whole file. The deploy job ships your site to production. The tunnel-paiement job then asks Pathly to replay your Flow in a real browser, from the product page to the payment page. If a step breaks, the job turns red.

The needs: deploy line sets the order. The tunnel-paiement job only starts once the deployment has succeeded. The Flow therefore tests the version that just reached production, not the previous one.

timeout_sec gives the Action time to wait for the verdict. Its default is 120 seconds. A checkout path spans several pages, so the workflow allows five minutes. Past that delay, the Action stops and the job fails.

Three things are left to prepare: the Flow, the API key and the two repository settings. The next sections cover them in that order.

• GitHub pathly-action

# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # Replace with your own deployment command
      - run: ./scripts/deploy.sh

  tunnel-paiement:
    name: Checkout path
    needs: deploy
    runs-on: ubuntu-latest
    steps:
      - name: Pathly Flow
        uses: pathlyhq/pathly-action@v1
        with:
          api_token: ${{ secrets.PATHLY_API_TOKEN }}
          operation: run-scenario
          scenario_id: ${{ vars.PATHLY_CHECKOUT_SCENARIO_ID }}
          timeout_sec: "300"

Why test after the release

Your pre-merge tests check the code. They do not see production: its environment variables, its cache, its payment provider, its third-party scripts. A release can break checkout while every test is green.

The typical case: Ping is green, Flow is red. The site answers 200 and the home page renders, but the Order button no longer does anything. An uptime check cannot see that. A browser that clicks can.

This check does not replace your pre-merge tests. It comes after them. The tests say the code is correct. The Flow says a customer can still reach the payment page, on the real site, a few minutes after the deployment.

Create the checkout Flow

The Flow is created in the console, with the recorder, without writing code. You open your shop in the recorder and walk the path of a customer. Pathly keeps each click as a step.

Typical steps: open a product page, add the product to the cart, open the cart, proceed to the payment page. Add a check at the end, for instance a text that only the payment page shows.

The Flow stops at the payment page. It pays nothing and only types test values. Pick a product that is always available, dedicated to this check if you can.

Once the Flow is saved, note its identifier. It is a UUID, visible in the address of the Flow page in the console. You will store it in a repository variable in the next step.

An API key with two scopes

Create an API key in the console, under Settings → API keys. It starts with sp_… and its secret is shown only once. Give it two scopes only: runs:trigger to start a run, runs:read to read its result.

Create one key per repository. If a repository leaks, you revoke its key without touching the others. With these two scopes, a stolen key can neither create, change nor delete a Flow.

In GitHub, open Settings → Secrets and variables → Actions. Store the key in the PATHLY_API_TOKEN secret. Store the Flow identifier in the PATHLY_CHECKOUT_SCENARIO_ID variable. The identifier is not a secret: a variable is enough, and it stays readable in the logs.

Before wiring the Flow, run a first check with operation: ping. This operation only verifies that the API recognises the key. It starts no run.

- name: Pathly ping
  uses: pathlyhq/pathly-action@v1
  with:
    api_token: ${{ secrets.PATHLY_API_TOKEN }}
    operation: ping

run-scenario or assert-scenario

The Action offers two operations that look alike. After a deployment, run-scenario is the one you need. It starts a new run with POST /v1/scenarios/{id}/run, then polls GET /v1/runs/{id} until a final status. The result therefore describes the version you just shipped.

assert-scenario starts nothing. It reads the last known result of the Flow, with the scenarios:read scope. That result may date from before the deployment: the job would be green while the new version was never tested.

assert-scenario is still useful elsewhere. It consumes no run and answers at once. It fits a check before a risky step, when the Flow already runs on a regular interval. It does not fit right after a release.

- name: Pathly assert
  uses: pathlyhq/pathly-action@v1
  with:
    api_token: ${{ secrets.PATHLY_API_TOKEN }}
    operation: assert-scenario
    scenario_id: ${{ vars.PATHLY_CHECKOUT_SCENARIO_ID }}

Reading a failure

The Action declares three outputs: run_id, run_status and ok. run_status is ok, fail or error. When the run is not ok, the job fails. Its log then shows the run identifier, its status and its message: start from there.

fail: a step or a check of the Flow fails. The site answers, but the customer path is broken. This is the case this gate is meant to catch: a button that no longer reacts, an empty cart, a payment page never reached.

error: the run did not finish because of a network or browser error. Site unreachable, connection timeout, crashed browser. A site that no longer answers at all after the release therefore gives error, not fail.

For the details, call GET https://api.pathlyhq.com/v1/runs/{id} with the same key. The response returns status, message and, when a screenshot exists, screenshotUrl. It is a signed link to the screenshot of the failing step, valid for 15 minutes. screenshotExpiresAt gives the time it expires. Call the route again to get a fresh link.

# RUN_ID: the identifier shown in the job log
curl -sS https://api.pathlyhq.com/v1/runs/$RUN_ID \
  -H "Authorization: Bearer $PATHLY_API_TOKEN" \
  | jq '{status, message, screenshotUrl, screenshotExpiresAt}'

curl -sS "https://api.pathlyhq.com/v1/runs?scenarioId=$PATHLY_CHECKOUT_SCENARIO_ID&limit=1" \
  -H "Authorization: Bearer $PATHLY_API_TOKEN" \
  | jq '.items[0] | {id, status, message}'

React: rollback and alert

A red job is useless if nobody looks at it. Add a rollback job with needs: tunnel-paiement and if: failure(). It only runs when the gate has failed, and it puts the previous version back in production.

failure() is true as soon as an upstream job fails, the deployment included. The condition needs.tunnel-paiement.result == 'failure' restricts the rollback to a failure of the Pathly gate alone.

The job is red on fail and on error alike, but the response differs. On fail, the customer path is broken: roll back. On error, rerun the job once. If the error comes back, roll back as well.

Warn the team too. A webhook subscribed to the run.failed event pushes the failure to your alerting tool. Create it in the console, or through the API with a key that carries alerting:write, separate from the CI key. Third-party channels (Slack, Teams, webhooks) are part of the paid plans.

  rollback:
    needs: tunnel-paiement
    if: failure() && needs.tunnel-paiement.result == 'failure'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # Replace with your own rollback command
      - run: ./scripts/rollback.sh

# PATHLY_ALERTING_TOKEN: a key with alerting:write, not the CI key
curl -sS -X POST https://api.pathlyhq.com/v1/webhooks \
  -H "Authorization: Bearer $PATHLY_ALERTING_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.example.com/hooks/pathly",
    "events": ["run.failed"]
  }'

Going further

Several shops: create one Flow per shop and loop over their identifiers with strategy.matrix. Each shop gets its own job and its own verdict. With fail-fast: false, a failure on one shop does not cancel the check of the others.

Maintenance window: if the deployment takes the site down for a moment, open a window before deploying. It silences the alerts of regular monitoring during the slot. The Action has no dedicated operation: call the /v1/maintenance-windows API with another key, one that carries maintenance:write.

ensure-scenario creates the Ping of the site if it does not exist. If it already exists, it leaves it as it is, without changing it. It finds it by name: rerunning the workflow creates no duplicate. This operation needs the scenarios:write scope, hence a key separate from the gate key.

Two deployments in a row: without a guard, two checks overlap and the second may test a half-shipped version. A concurrency block makes the second run of the workflow wait until the first has finished, gate included.

  tunnel-paiement:
    needs: deploy
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        scenario_id:
          - ${{ vars.PATHLY_CHECKOUT_FR_SCENARIO_ID }}
          - ${{ vars.PATHLY_CHECKOUT_DE_SCENARIO_ID }}
    steps:
      - name: Pathly Flow
        uses: pathlyhq/pathly-action@v1
        with:
          api_token: ${{ secrets.PATHLY_API_TOKEN }}
          operation: run-scenario
          scenario_id: ${{ matrix.scenario_id }}
          timeout_sec: "300"

START=$(date -u +%Y-%m-%dT%H:%M:%SZ)
END=$(date -u -d "+15 minutes" +%Y-%m-%dT%H:%M:%SZ)

curl -sS -X POST https://api.pathlyhq.com/v1/maintenance-windows \
  -H "Authorization: Bearer $PATHLY_MAINTENANCE_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"startsAt\": \"$START\", \"endsAt\": \"$END\"}"

- name: Pathly Ping
  uses: pathlyhq/pathly-action@v1
  with:
    api_token: ${{ secrets.PATHLY_SETUP_TOKEN }}
    operation: ensure-scenario
    scenario_json: |
      {
        "name": "Home",
        "url": "https://www.example.com/",
        "intervalSec": 300
      }

concurrency:
  group: deploy-production
  cancel-in-progress: false

Common pitfalls

The CDN still serves the old version. The Flow then tests the previous site and the job is green for the wrong reason. Purge the cache in the deploy job and wait for the purge to finish before handing over to the gate.

A bot protection or a CAPTCHA blocks the Pathly browser. The Flow fails while your customers get through. A CAPTCHA stops this browser like any other robot: keep it off the pages the Flow crosses, or plan a rule for this test traffic in your application firewall.

The test product is out of stock. The add-to-cart button disappears and the Flow turns red although the code has not changed. Keep a product that is always available for this test.

The quota. A run started by CI counts against the run quota of your plan, like any other. With the quota reached and no run pack, the run ends in fail with a quota message and the job turns red. Read the message before rolling back: here the site is not at fault.