sdk-lab

PingOps SDK Conformance Lab — deterministic, cross-language suite for the PingOps SDK.

It verifies that language SDKs instrument outgoing HTTP correctly: CLIENT spans, OTLP JSON export, redaction, transforms, domain filters, body capture, startTrace, and related SPEC behaviors.

Spec under test: SPEC.md (reference implementation @pingops/sdk 0.4.x).

Why this exists

SDK instrumentation drifts across languages and releases. This lab provides:

A single scenario source of truth (scenarios.json)
A shared assertion engine that encodes SPEC expectations
Language clients that drive real SDKs against deterministic fixtures
An in-process OTLP collector so every run is inspectable as JSON

Use it to gate SDK releases, catch regressions, and onboard new language implementations against the same contract.

Features

49 scenarios covering methods, errors, redirects, headers, transforms, bodies, compression, streaming, transport, and context
SPEC-aligned config — domainAllowList / domainDenyList, transforms, capture*Body, exportMode, suppressInstrumentation
Node (@pingops/sdk 0.4.x) and Ruby client drivers
Isolated Node workers per scenario (clean SDK init/shutdown between cases)
Deterministic mock API + lightweight OTLP HTTP JSON collector
Docker Compose one-shot full suite
CI for typecheck + scenario schema validation

Requirements

Dependency	Version
Node.js	≥ 20
npm	≥ 10 (lockfile v3)
Ruby + Bundler	≥ 3.1 (optional; Ruby scenarios only)
Docker Compose	optional; full-stack runs

Quick start

# 1. Install
npm install

# 2. Start fixture servers (mock API :8080 + collector :4318)
npm run dev

# 3. In another terminal — run the full suite (Node SDK)
npm run runner

Expected: 49/49 passed with a JSON report under collector/output/.

One-shot with Docker:

docker compose up --build --abort-on-container-exit --exit-code-from runner

Architecture

┌──────────────────────────────────────────┐
│  Test Runner  (runner/)                  │
│  Load scenarios → run client → assert    │
└──────────────────┬───────────────────────┘
                   │
     ┌─────────────┴─────────────┐
     ▼                           ▼
┌─────────────────┐     ┌────────────────────┐
│ Language client │     │ OTLP Collector     │
│ (clients/*)     │────▶│ POST /v1/traces    │
│ PingOps SDK     │     │ GET  /spans        │
└────────┬────────┘     └────────────────────┘
         │ HTTP
         ▼
┌─────────────────┐
│ Mock API        │
│ Deterministic   │
│ fixtures :8080  │
└─────────────────┘

Component	Role
Runner	Orchestrates scenarios, waits for services, asserts spans
Mock API	Deterministic HTTP responses (redirects, compression, streams, …)
Client	Initializes the language SDK and performs the scenario request
Collector	Receives OTLP HTTP JSON; stores spans for assertion

Node scenarios run in a child worker process so each case gets a fresh SDK lifecycle (the SDK’s re-init in-process is not reliable for multi-config suites).

Scenario matrix

Group	IDs	Coverage
A Basic semantics	01–08	GET/POST/PUT/DELETE/HEAD/OPTIONS, 204, invalid JSON
B Error codes	09–14	400, 401, 403, 404, 500, 503
C Redirects	15–18	301, 307, loop (`maxRedirects`), relative `Location`
D Headers / transforms	19–23	Default redaction, cookies, remove/redact transforms
E Body capture	24–27	Global off, domain on, startTrace, size attributes
F Compression	28–31	gzip, brotli, deflate, mismatched encoding
G Streaming	32–35	Chunked, slow, mid-stream abort, SSE
H Transport	36–38	Timeout, cancellation, connection refused
I Context	39–40	`startTrace` propagation, concurrent same-trace
J SPEC extras	41–49	Domain deny/allow/suffix, suppress, `sdk.version`, query/mask/replace/jsonPath transforms

Validate scenario structure anytime:

npm run validate:scenarios

Usage

All scenarios (Node)

npm run runner

All scenarios (Ruby)

bundle install --gemfile clients/ruby/Gemfile
SDK_LANGUAGE=ruby npm run runner
# or
npm run runner -- --language ruby

Filter scenarios

# Exact ID or substring match
npm run runner:scenario -- mvp-01-get-200-json
npm run runner:scenario -- redirect
npm run runner -- --scenario transform

Verbose logs

npm run runner -- --verbose

Exit codes

Code	Meaning
`0`	All selected scenarios passed
`1`	One or more failures, or runner/services error

Configuration

Copy .env.example to .env for local overrides (optional).

Variable	Default	Description
`MOCK_API_URL`	`http://localhost:8080`	Mock API base URL
`MOCK_API_PORT`	`8080`	Mock API listen port
`COLLECTOR_URL`	`http://localhost:4318`	Collector origin (SDK appends `/v1/traces`)
`COLLECTOR_PORT`	`4318`	Collector listen port
`SCENARIOS_FILE`	`./scenarios.json`	Scenario definitions
`OUTPUT_DIR`	`./collector/output`	Reports and session dumps
`SDK_LANGUAGE`	`node`	`node` or `ruby`
`NODE_SDK_VERSION`	`0.4.0`	Version string in reports (Node)
`RUBY_SDK_VERSION`	`0.0.1`	Version string in reports (Ruby)
`RUBY_CLIENT_CMD`	—	Override Ruby launch command

Domain placeholder: scenario domain rules may use $host (or .$host for suffix form). The client rewrites these to the mock API hostname (localhost, Docker DNS name, etc.).

Important: pass the collector origin to the SDK as baseUrl, never …/v1/traces. The SDK always appends /v1/traces (see SPEC §14).

Docker

# Full suite; container exit code follows the runner
docker compose up --build --abort-on-container-exit --exit-code-from runner

# Ruby matrix (if gem is available in the image)
SDK_LANGUAGE=ruby docker compose up --build --abort-on-container-exit --exit-code-from runner

Services: mock-api (8080), collector (4318), runner (depends on healthy fixtures).

Project layout

sdk-lab/
├── SPEC.md                 # PingOps SDK language-agnostic specification
├── scenarios.json          # Canonical scenario matrix (source of truth)
├── runner/                 # Orchestrator, schema checks, assertion engine
├── mock-api/               # Deterministic HTTP fixtures
├── collector/              # OTLP HTTP JSON receiver + output/
├── clients/
│   ├── node/               # Worker-isolated @pingops/sdk driver
│   └── ruby/               # pingops gem driver
├── shared/                 # Types + SDK config mapping helpers
├── scripts/                # CI helpers (e.g. validate-scenarios)
├── docker-compose.yml
└── Dockerfile

How assertions work

For each scenario the runner:

Clears collector spans
Executes the language client (SDK init → HTTP request → flush)
Polls the collector until the expected span count is stable (or timeout)
Runs the assertion engine

Check	Failure type
Exact span count (including 0)	`span_count_mismatch`
Span kind is `CLIENT`	`span_kind_mismatch`
Error status matches expectation	`error_status_mismatch`
Required / forbidden attributes	`missing_attribute` / `forbidden_attribute_present`
Header present / absent / `[REDACTED]`	`header_expectation_failed`
Body capture on/off (+ optional size attrs)	`body_capture_mismatch`
Sensitive values never appear raw	`redacted_value_leaked`
Shared / fixed `traceId`, unique `spanId`	`trace_correlation_failed`

Definition of done

A scenario passes when:

Span count matches exactly
Required attributes are present; forbidden ones are absent
Sensitive header/query/body secrets never appear in raw form
Redaction/transform expectations hold (e.g. default [REDACTED])
Body capture matches the scenario’s configured policy
The client process exits cleanly

Report format

Reports are written to collector/output/report-*.json (gitignored):

{
  "timestamp": "2026-07-29T12:00:00.000Z",
  "language": "node",
  "sdkVersion": "0.4.0",
  "totalScenarios": 49,
  "passed": 49,
  "failed": 0,
  "skipped": 0,
  "results": []
}

Extending the lab

New scenarios

Add an entry to scenarios.json (stable ID: mvp-NN-short-kebab-name)
Ensure mock-api/ can serve any new path/behavior
Run npm run validate:scenarios
Run the scenario: npm run runner:scenario -- mvp-NN-…

New language client

Add clients/<lang>/ implementing the ConformanceClient contract (clients/types.ts)
Map lab SDKConfig → the language SDK’s real options (see shared/sdk-config.ts / Node worker)
Register the language in runner/run.ts (resolveClient / resolveSdkVersion)
Document install/run steps in this README

Prefer a process-isolated worker (like Node/Ruby) so multi-config suites stay reliable.

Development

npm install
npm run typecheck            # TypeScript
npm run validate:scenarios   # Scenario schema + invariants
npm run test                 # Currently validates scenarios
npm run build                # Emit dist/
npm run clean                # Remove dist/ + generated JSON reports
npm run dev                  # mock-api + collector
npm run runner               # Full conformance suite

CI (.github/workflows/ci.yml) runs typecheck and validate:scenarios on push/PR.

Spec alignment notes

Topic	Lab behavior
Export	OTLP HTTP JSON to `{baseUrl}/v1/traces`
Config	SPEC fields: domains, transforms, capture flags, `exportMode`
Headers	Default sensitive redaction → `[REDACTED]`; custom via `transforms`
`startTrace`	Fixed IDs via `attributes.traceId` (not SHA-256 `seed`)
Domain rules	`$host` / `.$host` rewritten to the mock API hostname
`serviceName`	Still sent — required by `@pingops/sdk` 0.4.0 types (SPEC marks it removed)

The authoritative contract is SPEC.md. When the lab and SPEC disagree, prefer updating the lab to match the SPEC (or document a known SDK gap in scenario notes).

Troubleshooting

Symptom	What to check
Services never ready	`curl localhost:8080/health` and `curl localhost:4318/health`; free ports or set `*_PORT`
Zero spans	Collector `baseUrl` is origin only; SDK must export JSON OTLP; inspect `GET /spans`
Domain scenarios flaky	Use `$host` in rules; confirm mock host is `localhost` vs Docker service name
Install fails on `@pingops/sdk`	Ensure registry access if the package is private; pin matches `package.json`
Ruby failures	`bundle install --gemfile clients/ruby/Gemfile`; gem version may lag Node SPEC coverage

More detail lives in scenario expectations.notes fields inside scenarios.json.

License

This project is licensed under the MIT License.

Navigation Menu

Uh oh!

Repository files navigation

sdk-lab

Table of contents

Why this exists

Features

Requirements

Quick start

Architecture

Scenario matrix

Usage

All scenarios (Node)

All scenarios (Ruby)

Filter scenarios

Verbose logs

Exit codes

Configuration

Docker

Project layout

How assertions work

Definition of done

Report format

Extending the lab

New scenarios

New language client

Development

Spec alignment notes

Troubleshooting

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
clients		clients
collector		collector
mock-api		mock-api
runner		runner
scripts		scripts
shared		shared
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
SPEC.md		SPEC.md
docker-compose.yml		docker-compose.yml
package-lock.json		package-lock.json
package.json		package.json
scenarios.json		scenarios.json
tsconfig.json		tsconfig.json

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

sdk-lab

Table of contents

Why this exists

Features

Requirements

Quick start

Architecture

Scenario matrix

Usage

All scenarios (Node)

All scenarios (Ruby)

Filter scenarios

Verbose logs

Exit codes

Configuration

Docker

Project layout

How assertions work

Definition of done

Report format

Extending the lab

New scenarios

New language client

Development

Spec alignment notes

Troubleshooting

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages