Skip to content

[2.9]: Improve CoCo provisioning and mixed TDX/SNP workflow - #5362

Open
IsaacYangSLA wants to merge 2 commits into
NVIDIA:2.9from
IsaacYangSLA:improve_CoCo_workflow
Open

IsaacYangSLA wants to merge 2 commits into
NVIDIA:2.9from
IsaacYangSLA:improve_CoCo_workflow

Conversation

@IsaacYangSLA

@IsaacYangSLA IsaacYangSLA commented Oct 4, 2026 •

Copy link
Copy Markdown
Collaborator

Mixed confidential deployments need an ordinary trusted NVFlare server to require a TDX CPU-only client's proof and an SNP+GPU client's CPU/GPU proof without depending on a verifier-only observer. This change adds a complete provisioning example and hardens startup handling so missing registration has a bounded initial grace period, while invalid proofs fail immediately and jobs remain blocked until every required participant validates. An interruptible lock-contention backoff prevents a busy loop after the startup deadline.

  • Add per-site gpu_required workload constraints, rejecting GPU-required peers that present CPU-only evidence.
  • Add the ordinary-server/two-client provisioning example, separate encrypted-image and private KBS handoffs, direct node communication, and a finite validation job whose reviewed code is baked into the application images.
  • Add optional acceptance tooling for A/B topology preparation, sanitized evidence, proof/channel audits, negative fixtures, and JSON-only job validation. An observer is optional test instrumentation and is absent from the normal mixed example. Its component allow-list requires explicit class paths and rejects wildcards and package prefixes.
  • Allow an explicitly selected Intel collateral channel (early by default or standard) in the pinned TDX verifier, preserving strict UpToDate, collateral-expiration, debug-disabled, and event-log requirements. Bound diagnostic output to verified status labels.
  • Update deployment and security documentation with the scope of the successful real Kata mixed run and the effect of independent image keys and default-deny KBS policies.

Validation:

  • Authorizer, CCManager, and provisioning regression suites: 1,385 passed.
  • Mixed example and acceptance suites: 150 passed.
  • Current CCManager, acceptance preparation, negative-fixture, and mixed-workflow suites: 178 passed, including policy rejection, idempotence, deadline contention, and shutdown interruption regressions.
  • Broader CoCo provisioning and CVM profile checks on ARM macOS: 1,010 passed and 273 skipped; 13 Linux-specific failures were resolved by running the three affected suites on x86_64 Linux (57 passed).
  • ./runtest.sh -s --skip-install: passed (Black, isort, flake8, agent skill checks).
  • Public source package and assembled role-kit checks: passed, including syntax, links, and publication scans.
  • Isolated x86_64 Linux TDX verifier build: cargo check --locked --tests, cargo test --locked, and release compilation passed; all seven Rust tests also passed independently in the resulting container.

Real hardware validation completed on 2026-10-04: an ordinary trusted server authenticated one TDX CPU-only and one SNP+NVIDIA GPU Kata client, enforced the required proofs, observed periodic validation, and completed a nonce-bound CPU arithmetic job with values 3 and 7 (aggregate 10). Private infrastructure details, credentials, raw proofs, deployment artifacts, and the results manifest are excluded.

This is functional validation, not full security qualification. Renewal acceptance, confidentiality enforcement, and mandatory hardware denial cases remain outstanding; the arithmetic job does not claim GPU training coverage. The exact effective image-digest comparison remains an assurance limitation, not a demonstrated bypass of independent per-image encryption keys or KBS release authorization.

Copilot AI balanced review requested due to automatic review settings October 4, 2026 20:13
@IsaacYangSLA IsaacYangSLA added this to the 2.9 milestone Oct 4, 2026 — with ChatGPT Codex Connector
@IsaacYangSLA IsaacYangSLA modified the milestones: 2.9, 2.9.1 Oct 4, 2026 — with ChatGPT Codex Connector
@IsaacYangSLA IsaacYangSLA changed the title [2.9] Improve CoCo provisioning and mixed TDX/SNP workflow [2.9]: Improve CoCo provisioning and mixed TDX/SNP workflow Oct 4, 2026
@greptile-apps

greptile-apps Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium risk] Adds acceptance test harness and mixed TDX/SNP deployment examples.

The PR appears safe to merge based on the reviewed changes and current finding state.

Summary

The PR adds a mixed TDX CPU-only/SNP+GPU provisioning example, per-site GPU proof requirements, acceptance tooling, and bounded startup registration handling. The changes since the previous review tighten the observer’s component allow-list and make validation retries wait during lock contention. No new actionable issue was identified.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  TDX["TDX CPU-only client"] --> S["Ordinary NVFlare server"]
  SNP["SNP + GPU client"] --> S
  S --> V["Validate both required proofs"]
  V -->|pass| J["Schedule finite validation job"]
  V -->|missing registration during startup| W["Wait within bounded grace"]
  V -->|invalid proof or expired grace| B["Block job / shut down"]
Loading

Reviews (2) · Last reviewed commit: "[2.9]: Fix CoCo acceptance allow-list an..."

Comment thread examples/devops/coco/acceptance/observer_builder.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Startup lock contention can cause a busy loop, and the observer builder permits package-wide component authorization.

Review effort: Balanced
Findings: 1 High severity · 1 Medium severity · 1 Low severity

Open (3)
What changed in this PR

Improves mixed TDX/SNP confidential deployments with per-site GPU requirements, bounded startup validation, deployment examples, and acceptance tooling.

Changes:

  • Enforces CPU-only versus CPU+GPU proof constraints.
  • Adds bounded participant-registration grace and fail-closed job gating.
  • Adds mixed-deployment examples, audits, validation jobs, tests, and documentation.
File Description
tests/​unit_test/​lighter/​cc_provision/​impl/​test_coco.py Tests constraint propagation.
tests/​unit_test/​lighter/​cc_provision/​impl/​tdx_trusted_proof_audit_test.py Tests proof auditing.
tests/​unit_test/​lighter/​cc_provision/​impl/​tdx_trusted_channel_audit_test.py Tests channel auditing.
tests/​unit_test/​lighter/​cc_provision/​impl/​tdx_negative_fixtures_test.py Tests denial fixtures.
tests/​unit_test/​lighter/​cc_provision/​impl/​tdx_acceptance_prepare_test.py Tests acceptance preparation.
tests/​unit_test/​lighter/​cc_provision/​impl/​tdx_acceptance_job_test.py Tests finite validation job.
tests/​unit_test/​lighter/​cc_provision/​impl/​tdx_acceptance_federation_test.py Tests federation verification.
tests/​unit_test/​lighter/​cc_provision/​impl/​tdx_acceptance_evidence_test.py Tests evidence ledger.
tests/​unit_test/​lighter/​cc_provision/​impl/​mixed_coco_workflow_test.py Tests mixed provisioning.
tests/​unit_test/​app_opt/​confidential_computing/​coco_authorizer_test.py Tests GPU constraints.
tests/​unit_test/​app_opt/​confidential_computing/​cc_manager_test.py Tests startup and validation behavior.
nvflare/​app_opt/​confidential_computing/​coco_authorizer.py Enforces per-site GPU evidence.
nvflare/​app_opt/​confidential_computing/​cc_manager.py Adds bounded registration grace.
examples/​devops/​coco/​trusted_system/​tdx-verifier/​src/​main.rs Adds collateral-channel selection.
examples/​devops/​coco/​trusted_system/​tdx-verifier/​run-verifier.sh Passes validated channel configuration.
examples/​devops/​coco/​trusted_system/​tdx-verifier/​Cargo.toml Adds Trustee storage dependency.
examples/​devops/​coco/​trusted_system/​tdx-verifier/​Cargo.lock Locks the new dependency.
examples/​devops/​coco/​trusted_system/​TDX-LAUNCH-PROFILE.md Documents collateral channels.
examples/​devops/​coco/​service/​SECURITY.md Clarifies image-binding limitations.
examples/​devops/​coco/​RUNTIME-VARIANTS.md Records mixed hardware validation.
examples/​devops/​coco/​RUNTIME-PORTING.md Documents GPU constraints.
examples/​devops/​coco/​README.md Links mixed and acceptance workflows.
examples/​devops/​coco/​provision/​README.md Documents mixed provisioning.
examples/​devops/​coco/​provision/​mixed-tdx-snp/​validation_builder.py Authorizes baked validation classes.
examples/​devops/​coco/​provision/​mixed-tdx-snp/​site-2/​Dockerfile Defines SNP+GPU application image.
examples/​devops/​coco/​provision/​mixed-tdx-snp/​site-1/​Dockerfile Defines TDX application image.
examples/​devops/​coco/​provision/​mixed-tdx-snp/​README.md Provides mixed deployment guide.
examples/​devops/​coco/​provision/​mixed-tdx-snp/​project.yaml Defines mixed federation project.
examples/​devops/​coco/​provision/​mixed-tdx-snp/​cc_site-2.yml Configures SNP+GPU client.
examples/​devops/​coco/​provision/​mixed-tdx-snp/​cc_site-1.yml Configures TDX CPU client.
examples/​devops/​coco/​provision/​CCMANAGER.md Documents validation semantics.
examples/​devops/​coco/​FL-DEPLOYMENT.md Updates deployment workflow.
examples/​devops/​coco/​acceptance/​verify_federation.py Verifies federation and results.
examples/​devops/​coco/​acceptance/​trusted_proof_audit.py Captures sanitized proof metadata.
examples/​devops/​coco/​acceptance/​trusted_channel_audit.py Captures sanitized channel metadata.
examples/​devops/​coco/​acceptance/​README.md Documents acceptance procedures.
examples/​devops/​coco/​acceptance/​prepare.py Generates private A/B inputs.
examples/​devops/​coco/​acceptance/​observer_builder.py Builds verifier-only observer configuration.
examples/​devops/​coco/​acceptance/​negative_fixtures.py Generates isolated denial fixtures.
examples/​devops/​coco/​acceptance/​evidence.py Records private evidence metadata.
examples/​devops/​coco/​acceptance/​application/​tdx_acceptance.py Implements finite validation workload.
examples/​devops/​coco/​acceptance/​application/​generate_job.py Generates JSON-only validation jobs.
docs/​user_guide/​confidential_computing/​coco_security_architecture.rst Updates security architecture guidance.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread examples/devops/coco/acceptance/observer_builder.py
Comment thread nvflare/app_opt/confidential_computing/cc_manager.py
Comment thread examples/devops/coco/acceptance/negative_fixtures.py Outdated

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants