Idea: Black-box IdP conformance tests for OpenTDF

[eugenioenko]

Motivation

Today CI and most integration tests stand up Keycloak via docker-compose. That gives us strong coverage of the Keycloak path, but the platform advertises generic OIDC support (plus Claims and Multi-Strategy ERS modes), and in practice users are pointing it at Okta, Auth0, Entra ID, Google, AWS Cognito, Ping, and others. Whenever a real integration breaks, we find out from a downstream issue rather than from CI.

I'd like to propose a black-box IdP conformance suite: a thin, provider-agnostic test pack that exercises the auth surface of the platform (token acquisition, JWKS/discovery, DPoP, claim shapes, audience handling, ERS resolution, KAS rewrap end-to-end) against any OIDC provider behind a config file.

Goals

1. Catch regressions in the platform's auth and ERS code against the providers users actually deploy with.
2. Give us a reproducible way to answer "does OpenTDF work with X?" for any OIDC provider.
3. Lower the cost for the community to contribute provider coverage: drop in a config, run the suite, file the gaps.
4. Make the ERS Multi-Strategy mode a first-class tested surface, not just a documented one.

Non-goals

• Replacing the Keycloak-based integration tests. Those stay as the default path on every PR.
• Testing provider-specific admin APIs (user provisioning, group sync, etc.). The suite treats the IdP as opaque.
• Shipping secrets for paid providers in the public repo.

Shape of the proposal

One suite, run against real tenants, in two triggers:

On PR, path-filtered. When a PR touches auth, ERS, or KAS code, run the suite against the real-provider matrix. Credentials come from GitHub Actions secrets. This means auth-touching PRs pay the slower feedback loop; everything else stays on the fast Keycloak path.

Scheduled, nightly or weekly. Run the full matrix against all providers regardless of what changed. Failures open an issue rather than blocking anything, because the failure mode is often "the provider changed something on their end" and we don't want to wedge main on that.

Initial matrix candidates: Okta (developer tier), Auth0 (free tier), Entra ID, AWS Cognito, Google. We can grow it as contributors donate tenants.

What the suite actually tests

Provider-agnostic checks, driven by a single config file per provider:

• Discovery document is reachable and well-formed
• JWKS fetch and key rotation behavior
• Client credentials flow produces a token the platform accepts
• Audience and issuer validation match what the platform expects
• DPoP proof-of-possession round-trip
• ERS resolves an entity from the token to a representation the AuthZ service can consume
• End-to-end: encrypt a TDF with a policy, decrypt it with a token from this IdP, confirm the rewrap succeeds
• Negative cases: wrong audience, expired token, revoked key, unknown subject

Each check is a pass/fail with a short reason, so the output doubles as a compatibility matrix we can publish.

Open questions

1. Where does the per-provider config live? A test/idp-conformance/providers/*.yaml directory, or inline in the workflow?
2. Do we want to surface the output as a rendered compatibility page on docs.opentdf.io, or keep it internal for now?
3. Multi-Strategy ERS has its own config surface. Do we test it as a separate dimension of the matrix, or fold it into each provider's config?
4. How do we handle providers that require human interaction for initial setup (Entra ID app registration, Okta API token)? Probably a one-time setup script checked in, plus a runbook.
5. What's the right path filter for the PR trigger? At minimum service/pkg/auth/**, service/entityresolution/**, service/kas/**, but there will be cross-cutting changes that don't match any of those.
6. Who owns tenant credentials and their rotation? We'll need an owner per provider who can re-up the secret when it expires.

Why post this as a discussion first

The suite itself is a few days of work; the expensive part is agreeing on the shape (real providers only, path-filtered PR trigger, nightly full matrix) and working out the tenant ownership story before anyone writes it. Feedback welcome on whether this belongs in service/integration, a new top-level test/idp-conformance, or a separate repo entirely.

[arkavo-com]

Like the idea. We will try to add a PoC to https://github.com/arkavo-org/opentdf-tests

[damonbanks]

@eugenioenko FYI, similar conversations happening over here: https://github.com/virtru-corp/data-security-platform/pull/3423
Where we "Say" we are ODIC IdP agnostic, but in truth we have more requirements then that: There are required claims, Dpop, etc.
I think this is in support of these doc updates as well.

[arkavo-com]

Implemented and merged in arkavo-org/opentdf-tests#6: the black-box IdP conformance suite proposed here — 11 provider-agnostic checks (discovery, JWKS, client credentials, aud/iss, DPoP, ERS, negatives, TDF rewrap e2e), per-provider YAML configs, PR path-filtered + nightly strict triggers, Pages compatibility matrix — validated end-to-end against a live Auth0 tenant, which already surfaced a real finding (go SDK RS256 DPoP proof rejected by Auth0: arkavo-org/platform#2).