CI/CD Integration

Integrate CallMeter into your CI/CD pipeline for automated VoIP quality gates, regression testing, and deployment validation.

CallMeter integrates with your CI/CD pipeline to automatically validate SIP infrastructure quality before, during, or after deployments. By triggering tests via the API and enforcing quality gates on metrics like MOS, packet loss, and jitter, you can catch VoIP regressions before they reach production.

Use Cases

Pre-Deployment Gate

Run a baseline SIP test before deploying changes to your PBX, SIP proxy, or session border controller. If the test fails or quality metrics fall below thresholds, the deployment is blocked.

Post-Deployment Validation

After deploying, trigger a test to verify that call quality has not degraded. This catches issues that pre-deployment testing might miss, such as configuration errors or network path changes.

Nightly Regression Testing

Schedule a comprehensive test suite to run every night. Detect gradual degradation (increasing jitter, declining MOS) before it affects users.

Canary Testing

Run a quick test against a canary environment before routing production traffic. If the canary passes quality gates, proceed with the full rollout.

Pull Request Validation

Trigger tests on PR branches that modify SIP server configurations. Ensure that configuration changes do not break call quality before merging.

API Workflow

The CI/CD integration follows a three-step pattern:

Trigger -- Start a test run via POST request
Poll -- Check the test run status until completion
Evaluate -- Query metrics and enforce quality thresholds

Step 1: Trigger a Test Run

TEST_RUN_ID=$(curl -s -X POST \
  -H "Authorization: Bearer $CALLMETER_API_KEY" \
  https://callmeter.io/api/v1/projects/$PROJECT_ID/tests/$TEST_ID/run \
  | jq -r '.data.testRunId')

echo "Started test run: $TEST_RUN_ID"

The TEST_ID refers to a pre-configured test in your CallMeter project. Create the test once through the web interface with your desired endpoints, codecs, and duration, then reference it by ID in your pipeline.

Step 2: Poll for Completion

echo "Waiting for test to complete..."
while true; do
  RESPONSE=$(curl -s \
    -H "Authorization: Bearer $CALLMETER_API_KEY" \
    https://callmeter.io/api/v1/projects/$PROJECT_ID/test-runs/$TEST_RUN_ID)

  STATUS=$(echo $RESPONSE | jq -r '.data.status')

  case $STATUS in
    COMPLETED)
      echo "Test completed successfully"
      break
      ;;
    FAILED|CANCELLED|CANNOT_RUN_FOR_NOW)
      # These three are TERMINAL. Leaving them to the catch-all below is the trap: the run will
      # never progress, so the loop polls it to the timeout and the pipeline reports a stall
      # rather than the cancellation that actually happened.
      echo "Test ended with status: $STATUS"
      exit 1
      ;;
    *)
      echo "Status: $STATUS - waiting..."
      sleep 15
      ;;
  esac
done

Four statuses are terminal, not three

A test run ends in one of COMPLETED, FAILED, CANCELLED or CANNOT_RUN_FOR_NOW. Only COMPLETED is worth evaluating; the other three mean the run produced no result.

The trap is treating CANCELLED or CANNOT_RUN_FOR_NOW as "still waiting". A run in either state will never progress, so the loop polls it until the timeout and the pipeline reports a stall — which reads as a CallMeter outage rather than as the cancellation that actually happened. The case statement above names all three failure states explicitly for that reason.

There is no ERROR status. If you have copied a poll loop that branches on one, it has a dead arm.

Polling interval recommendation

Use a 10-15 second polling interval. Shorter intervals waste API quota; longer intervals delay your pipeline unnecessarily. For long-running tests, start with 30-second intervals.

Step 3: Evaluate Quality Metrics

After the test completes, query the test run summary and enforce your quality thresholds:

# Fail fast if a dependency is missing. Without this, an absent comparison tool
# makes every threshold check evaluate to nothing and the gate reports PASS on a
# run it should have blocked -- a gate that always passes is worse than no gate.
for BIN in curl jq awk; do
  command -v "$BIN" >/dev/null || { echo "FAIL: $BIN is required but not installed"; exit 1; }
done

SUMMARY=$(curl -s \
  -H "Authorization: Bearer $CALLMETER_API_KEY" \
  https://callmeter.io/api/v1/projects/$PROJECT_ID/test-runs/$TEST_RUN_ID)

# Float comparison via awk. awk is POSIX and present in every base image; `bc` is
# not -- alpine, debian-slim and the official node images all omit it, and the
# GitLab example below runs on alpine.
lt() { awk -v a="$1" -v b="$2" 'BEGIN { exit !(a <  b) }'; }
gt() { awk -v a="$1" -v b="$2" 'BEGIN { exit !(a >  b) }'; }

# Guard first: a run that placed no calls has nothing to judge. ASR would read 0,
# which is indistinguishable from "every call was rejected" unless you check this.
TOTAL_CALLS=$(echo $SUMMARY | jq -r '.data.callTiming.totalCalls')
if [ "$TOTAL_CALLS" -eq 0 ]; then
  echo "FAIL: the run placed no calls - check registration and endpoint configuration"
  exit 1
fi

# Extract key metrics from the test run summary
MOS=$(echo $SUMMARY | jq -r '.data.summary.avgMos')
LOSS_FRACTION=$(echo $SUMMARY | jq -r '.data.summary.avgPacketLossFraction')
JITTER=$(echo $SUMMARY | jq -r '.data.summary.avgJitter')

# Call establishment, from the sibling callTiming object.
ASR=$(echo $SUMMARY | jq -r '.data.callTiming.asr')
PDD_P95=$(echo $SUMMARY | jq -r '.data.callTiming.pdd.p95')

# avgPacketLossFraction is a fraction, not a percentage: 0.012 means 1.2% loss.
# Convert once here so the thresholds below read as percentages.
PACKET_LOSS=$(awk -v f="$LOSS_FRACTION" 'BEGIN { printf "%.4f", f * 100 }')

echo "MOS: $MOS | Loss: $PACKET_LOSS% | Jitter: ${JITTER}ms | ASR: $ASR% | PDD p95: ${PDD_P95}ms"

# Enforce quality gates
FAILED=0

if lt "$MOS" 4.0; then
  echo "FAIL: MOS $MOS is below threshold 4.0"
  FAILED=1
fi

if gt "$PACKET_LOSS" 1.0; then
  echo "FAIL: Packet loss $PACKET_LOSS% exceeds threshold 1.0%"
  FAILED=1
fi

if gt "$JITTER" 20.0; then
  echo "FAIL: Jitter ${JITTER}ms exceeds threshold 20ms"
  FAILED=1
fi

if lt "$ASR" 95.0; then
  echo "FAIL: ASR $ASR% is below threshold 95%"
  FAILED=1
fi

# Percentiles are null when the run produced no samples for that dimension.
if [ "$PDD_P95" != "null" ] && gt "$PDD_P95" 3000; then
  echo "FAIL: PDD p95 ${PDD_P95}ms exceeds threshold 3000ms"
  FAILED=1
fi

# Share of calls that ended in a 5xx, computed against a denominator you choose.
# sipResponses[].percentage uses its own denominator - calls that produced a final
# status code - which is smaller than totalCalls on a run with timeouts.
SERVER_ERRORS=$(echo $SUMMARY | jq -r '[.data.sipResponses[] | select(.statusCode >= 500) | .count] | add // 0')
ERROR_RATE=$(awk -v e="$SERVER_ERRORS" -v t="$TOTAL_CALLS" 'BEGIN { printf "%.4f", e * 100 / t }')

if gt "$ERROR_RATE" 2.0; then
  echo "FAIL: 5xx rate $ERROR_RATE% exceeds threshold 2.0%"
  FAILED=1
fi

if [ $FAILED -eq 1 ]; then
  echo "Quality gate FAILED - blocking deployment"
  exit 1
fi

echo "All quality gates PASSED"

Check totalCalls before gating on ASR

asr is 0 on a run that placed no calls, which reads identically to a run where every call failed. They are very different problems — a misconfigured registrar versus a broken route — and the ratio alone cannot tell them apart. Read callTiming.totalCalls first, as the example above does. The same applies to ner.

Quality Gate Thresholds

Recommended thresholds for different environments:

Metric	Production	Staging	Minimum Viable
MOS (Mean Opinion Score)	4.0 or higher	3.8 or higher	3.5 or higher
Packet Loss	Below 1%	Below 2%	Below 5%
Jitter	Below 20ms	Below 30ms	Below 50ms
RTT (Round-Trip Time)	Below 150ms	Below 200ms	Below 300ms
R-factor (ITU-T G.107)	80 or higher	70 or higher	60 or higher
ASR (Answer Seizure Ratio)	95% or higher	90% or higher	85% or higher
PDD p95 (Post-Dial Delay)	Below 3000ms	Below 5000ms	Below 8000ms

Gate on any metric the run returns

The examples above gate on MOS, packet loss, jitter, ASR and post-dial delay because those are the most common, but nothing is special about them. The test-runs/{id} response returns the full picture for the run in three blocks:

summary — media quality: MOS and R-factor (min and average), jitter and RTT (average and peak), packet loss and total packets lost, RTCP feedback counts (NACK, PLI, FIR), packet-loss-concealment events, signal and noise levels, and stream counts.
callTiming — call establishment: ASR, NER, per-result call counts, and a p50/p95/p99/avg/min/max distribution for post-dial delay, time-to-trying, time-to-ringing, time-to-first-media, call setup, call duration and REGISTER round-trip.
sipResponses — the distribution of SIP final status codes across the run.

Any of them can drive a gate with the same comparison pattern shown above. For per-route detail, test-runs/{id}/endpoints returns the same timing fields per endpoint. See the metrics reference for what each one measures.

Adjust these thresholds based on your specific SLA commitments and infrastructure characteristics. Stricter thresholds catch regressions earlier but may produce false positives in high-variance environments.

Set realistic thresholds

Overly strict thresholds cause false pipeline failures. Start with lenient thresholds and tighten them gradually as you establish your baseline quality. Use CallMeter dashboards to understand your typical metric ranges before defining CI/CD gates.

GitHub Actions Example

name: VoIP Quality Gate

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  voip-quality-check:
    runs-on: ubuntu-latest
    steps:
      - name: Trigger CallMeter Test
        id: trigger
        run: |
          RESPONSE=$(curl -s -X POST \
            -H "Authorization: Bearer ${{ secrets.CALLMETER_API_KEY }}" \
            https://callmeter.io/api/v1/projects/${{ vars.CALLMETER_PROJECT_ID }}/tests/${{ vars.CALLMETER_TEST_ID }}/run)
          echo "run_id=$(echo $RESPONSE | jq -r '.data.testRunId')" >> $GITHUB_OUTPUT

      - name: Wait for Completion
        id: wait
        run: |
          RUN_ID="${{ steps.trigger.outputs.run_id }}"
          echo "Waiting for test run $RUN_ID..."

          for i in $(seq 1 60); do
            STATUS=$(curl -s \
              -H "Authorization: Bearer ${{ secrets.CALLMETER_API_KEY }}" \
              https://callmeter.io/api/v1/projects/${{ vars.CALLMETER_PROJECT_ID }}/test-runs/$RUN_ID \
              | jq -r '.data.status')

            echo "Attempt $i: Status = $STATUS"

            if [ "$STATUS" = "COMPLETED" ]; then
              echo "status=COMPLETED" >> $GITHUB_OUTPUT
              exit 0
            fi

            case "$STATUS" in
              FAILED|CANCELLED|CANNOT_RUN_FOR_NOW)
                echo "Test ended with status: $STATUS"
                exit 1
                ;;
            esac

            sleep 15
          done

          echo "Timeout waiting for test completion"
          exit 1

      - name: Evaluate Quality Gates
        if: steps.wait.outputs.status == 'COMPLETED'
        run: |
          RUN_ID="${{ steps.trigger.outputs.run_id }}"

          SUMMARY=$(curl -s \
            -H "Authorization: Bearer ${{ secrets.CALLMETER_API_KEY }}" \
            https://callmeter.io/api/v1/projects/${{ vars.CALLMETER_PROJECT_ID }}/test-runs/$RUN_ID)

          # No calls placed means nothing to judge - fail before comparing ratios.
          TOTAL_CALLS=$(echo $SUMMARY | jq -r '.data.callTiming.totalCalls')
          if [ "$TOTAL_CALLS" -eq 0 ]; then
            echo "The run placed no calls - check registration" >> $GITHUB_STEP_SUMMARY
            exit 1
          fi

          MOS=$(echo $SUMMARY | jq -r '.data.summary.avgMos')
          LOSS_FRACTION=$(echo $SUMMARY | jq -r '.data.summary.avgPacketLossFraction')
          JITTER=$(echo $SUMMARY | jq -r '.data.summary.avgJitter')
          ASR=$(echo $SUMMARY | jq -r '.data.callTiming.asr')

          # awk, not bc: bc is absent from many CI images and a missing comparison
          # tool makes every check evaluate to nothing, reporting PASS on a bad run.
          lt() { awk -v a="$1" -v b="$2" 'BEGIN { exit !(a < b) }'; }
          gt() { awk -v a="$1" -v b="$2" 'BEGIN { exit !(a > b) }'; }

          # avgPacketLossFraction is a fraction (0.012 = 1.2%), so convert to a percentage
          LOSS=$(awk -v f="$LOSS_FRACTION" 'BEGIN { printf "%.4f", f * 100 }')

          echo "## VoIP Quality Report" >> $GITHUB_STEP_SUMMARY
          echo "| Metric | Value | Threshold | Status |" >> $GITHUB_STEP_SUMMARY
          echo "|--------|-------|-----------|--------|" >> $GITHUB_STEP_SUMMARY

          PASS=true

          if lt "$MOS" 4.0; then
            echo "| MOS | $MOS | >= 4.0 | FAIL |" >> $GITHUB_STEP_SUMMARY
            PASS=false
          else
            echo "| MOS | $MOS | >= 4.0 | PASS |" >> $GITHUB_STEP_SUMMARY
          fi

          if gt "$LOSS" 1.0; then
            echo "| Packet Loss | $LOSS% | < 1% | FAIL |" >> $GITHUB_STEP_SUMMARY
            PASS=false
          else
            echo "| Packet Loss | $LOSS% | < 1% | PASS |" >> $GITHUB_STEP_SUMMARY
          fi

          if gt "$JITTER" 20.0; then
            echo "| Jitter | ${JITTER}ms | < 20ms | FAIL |" >> $GITHUB_STEP_SUMMARY
            PASS=false
          else
            echo "| Jitter | ${JITTER}ms | < 20ms | PASS |" >> $GITHUB_STEP_SUMMARY
          fi

          if lt "$ASR" 95.0; then
            echo "| ASR | $ASR% | >= 95% | FAIL |" >> $GITHUB_STEP_SUMMARY
            PASS=false
          else
            echo "| ASR | $ASR% | >= 95% | PASS |" >> $GITHUB_STEP_SUMMARY
          fi

          if [ "$PASS" = false ]; then
            echo "Quality gate failed"
            exit 1
          fi

          echo "All quality gates passed"

GitLab CI Example

voip-quality-gate:
  stage: test
  image: alpine:latest
  before_script:
    # awk is in alpine's busybox already; curl and jq are not.
    - apk add --no-cache curl jq
  variables:
    CALLMETER_PROJECT_ID: "your-project-id"
    CALLMETER_TEST_ID: "your-test-id"
  script:
    # Trigger test
    - |
      RUN_ID=$(curl -s -X POST \
        -H "Authorization: Bearer $CALLMETER_API_KEY" \
        https://callmeter.io/api/v1/projects/$CALLMETER_PROJECT_ID/tests/$CALLMETER_TEST_ID/run \
        | jq -r '.data.testRunId')
      echo "Test run ID: $RUN_ID"

    # Poll for completion
    - |
      for i in $(seq 1 60); do
        STATUS=$(curl -s \
          -H "Authorization: Bearer $CALLMETER_API_KEY" \
          https://callmeter.io/api/v1/projects/$CALLMETER_PROJECT_ID/test-runs/$RUN_ID \
          | jq -r '.data.status')
        echo "Poll $i: $STATUS"
        if [ "$STATUS" = "COMPLETED" ]; then break; fi
        case "$STATUS" in
          FAILED|CANCELLED|CANNOT_RUN_FOR_NOW) echo "Test ended with status: $STATUS"; exit 1 ;;
        esac
        sleep 15
      done

    # Evaluate metrics
    - |
      SUMMARY=$(curl -s \
        -H "Authorization: Bearer $CALLMETER_API_KEY" \
        https://callmeter.io/api/v1/projects/$CALLMETER_PROJECT_ID/test-runs/$RUN_ID)
      TOTAL_CALLS=$(echo $SUMMARY | jq -r '.data.callTiming.totalCalls')
      if [ "$TOTAL_CALLS" -eq 0 ]; then
        echo "The run placed no calls - check registration"; exit 1
      fi
      MOS=$(echo $SUMMARY | jq -r '.data.summary.avgMos')
      LOSS_FRACTION=$(echo $SUMMARY | jq -r '.data.summary.avgPacketLossFraction')
      ASR=$(echo $SUMMARY | jq -r '.data.callTiming.asr')
      # awk, not bc — alpine ships neither bc nor jq, hence the apk add above.
      lt() { awk -v a="$1" -v b="$2" 'BEGIN { exit !(a < b) }'; }
      gt() { awk -v a="$1" -v b="$2" 'BEGIN { exit !(a > b) }'; }
      # avgPacketLossFraction is a fraction (0.012 = 1.2%), so convert to a percentage
      LOSS=$(awk -v f="$LOSS_FRACTION" 'BEGIN { printf "%.4f", f * 100 }')
      echo "MOS: $MOS | Loss: $LOSS% | ASR: $ASR%"
      if lt "$MOS" 4.0; then
        echo "MOS below threshold"; exit 1
      fi
      if gt "$LOSS" 1.0; then
        echo "Packet loss above threshold"; exit 1
      fi
      if lt "$ASR" 95.0; then
        echo "ASR below threshold"; exit 1
      fi
      echo "Quality gates passed"
  only:
    - main
    - merge_requests

#!/bin/bash
set -euo pipefail

API_KEY="${CALLMETER_API_KEY:?Set CALLMETER_API_KEY environment variable}"
PROJECT_ID="${1:?Usage: $0 <project-id> <test-id> [mos-threshold] [asr-threshold]}"
TEST_ID="${2:?Usage: $0 <project-id> <test-id> [mos-threshold] [asr-threshold]}"
MOS_THRESHOLD="${3:-4.0}"
ASR_THRESHOLD="${4:-95.0}"
API_BASE="https://callmeter.io/api/v1"

# Trigger
echo "Triggering test $TEST_ID..."
RUN_ID=$(curl -sf -X POST \
  -H "Authorization: Bearer $API_KEY" \
  "$API_BASE/projects/$PROJECT_ID/tests/$TEST_ID/run" | jq -r '.data.testRunId')

echo "Run ID: $RUN_ID"

# Poll
echo "Waiting for completion..."
while true; do
  STATUS=$(curl -sf \
    -H "Authorization: Bearer $API_KEY" \
    "$API_BASE/projects/$PROJECT_ID/test-runs/$RUN_ID" | jq -r '.data.status')

  case $STATUS in
    COMPLETED) break ;;
    FAILED|CANCELLED|CANNOT_RUN_FOR_NOW) echo "FAIL: $STATUS"; exit 1 ;;
    *) sleep 15 ;;
  esac
done

# Evaluate
SUMMARY=$(curl -sf \
  -H "Authorization: Bearer $API_KEY" \
  "$API_BASE/projects/$PROJECT_ID/test-runs/$RUN_ID")

TOTAL_CALLS=$(echo $SUMMARY | jq -r '.data.callTiming.totalCalls')
if [ "$TOTAL_CALLS" -eq 0 ]; then
  echo "QUALITY GATE FAILED: the run placed no calls"
  exit 1
fi

lt() { awk -v a="$1" -v b="$2" 'BEGIN { exit !(a < b) }'; }

MOS=$(echo $SUMMARY | jq -r '.data.summary.avgMos')
ASR=$(echo $SUMMARY | jq -r '.data.callTiming.asr')
echo "MOS: $MOS (threshold: $MOS_THRESHOLD) | ASR: $ASR% (threshold: $ASR_THRESHOLD%)"

if lt "$MOS" "$MOS_THRESHOLD"; then
  echo "QUALITY GATE FAILED"
  exit 1
fi

if lt "$ASR" "$ASR_THRESHOLD"; then
  echo "QUALITY GATE FAILED"
  exit 1
fi

echo "QUALITY GATE PASSED"

Usage:

# Run with default thresholds (MOS 4.0, ASR 95%)
./callmeter-test.sh your-project-id your-test-id

# Run with custom MOS threshold
./callmeter-test.sh your-project-id your-test-id 3.8

# Run with custom MOS and ASR thresholds
./callmeter-test.sh your-project-id your-test-id 3.8 90.0

Best Practices

Test Configuration

Create dedicated CI/CD tests -- Do not reuse manual testing configurations. Create purpose-built tests with appropriate duration and endpoint counts for pipeline execution.
Keep tests short -- CI/CD tests should complete in 1-3 minutes. Use longer tests for nightly regressions.
Test one thing at a time -- Separate tests for audio quality, video quality, call setup reliability, and load capacity.

Pipeline Design

Fail fast -- Check test run status before evaluating metrics. A FAILED status means the test itself could not complete, which is a deployment blocker.
Timeout handling -- Always include a maximum wait time. If a test does not complete within a reasonable window, fail the pipeline rather than blocking indefinitely.
Artifact results -- Store metric outputs as pipeline artifacts for later analysis and trending.

Environment Management

Store API keys in secrets -- Never hardcode keys in pipeline files
Use environment variables -- Reference CALLMETER_API_KEY from your CI/CD secret store
Separate test IDs per environment -- Use different test configurations for staging vs. production validation

Threshold Tuning

Start lenient, tighten over time -- Begin with thresholds that rarely fail, then gradually tighten as you establish baselines
Track trends -- A single test run may have variance; look at trends over multiple runs
Different thresholds per environment -- Production gates should be stricter than staging

Next Steps

API Authentication -- Create and manage API keys
API Endpoints -- Complete API reference
Analyzing Results -- Understand the metrics returned by the API