Repository navigation
Stop trusting compromised eMudhra roots in CI and smoke-test images - #9527
eliottness wants to merge 3 commits into
Conversation
Re-pin or bump CI/test images that still trusted emSign ECC Root CA - G3 / AAA Certificate Services (IR-62343): aspnet smoke-test runtime re-pins, an sdk-patched stage in the Linux smoke Dockerfiles that strips both roots from the SDK's NuGet trust lists (codesignctl.pem / timestampctl.pem), root removal in the EOL build images, newer integration-test services in docker-compose.yml, and the alpine 3.24 installer-test runtime. Reviewer notes: the sdk-patched stage changes NuGet package-signature trust (restore of author-signed packages chaining to AAA Certificate Services could fail); postgres 10.5->18.6, rabbitmq 3->4 and redis 4->8 are large jumps for integration-test services. Jira: APPSEC-71282
…anged Restore docker-compose.yml to master's service versions. The previous commit moved the integration-test servers across several major versions (postgres 10.5 -> 18.6, rabbitmq 3 -> 4, redis 4 -> 8). That changes the server versions the integration matrix tests against (oldest clients: Npgsql 4.0.0, RabbitMQ.Client 3.6.9, StackExchange.Redis 1.0.187, ServiceStack.Redis 4.0.48), and that cannot be verified on a fork PR, where the Azure pipelines do not run. These three services become follow-ups for the owning team to pick supported server versions. Jira: APPSEC-71282 (parent APPSEC-71273)
Split per review (andrewlock): keep this PR to the changes CI pulls fresh on every run (smoke-test Dockerfiles, smoke-test runtime images, dd-dotnet failure-test runtime image). The build-toolchain Dockerfiles (alpine.build, alpine.build.arm64, centos7.build, debian) need image republishing / VM rebuilds and move to a separate PR. Jira: APPSEC-71282 (parent APPSEC-71273)
BenchmarksBenchmark execution time: 2026-10-26 15:59:59 Comparing candidate commit f690218 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 72 metrics, 0 unstable metrics, 7 known flaky benchmarks, 119 flaky benchmarks without significant changes.
|
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (9527) and master. ✅ No regressions detected |
|
/azp run |
|
Closing and reopening to re-trigger Azure CI: build 212455 failed only in unit_tests_macos, where the build-macos-working-directory artifact download timed out at 100% (infra), and /azp run did not start a new build. |
|
CI status for this PR after two full runs, each failing on a different infra issue:
For comparison, master's Smoke coverage:
Next step: queue a build with |
Summary of changes
Stops this repository's CI, build and smoke-test images from trusting emSign ECC Root CA - G3 and AAA Certificate Services. The APPSEC-71273 ledger lists 152 impacted image references here: 20 fixed in this PR, 3 fixed at their in-repo image source in #9528, 129 follow-ups.
Reason for change
Incident IR-62343: the compromised eMudhra intermediates chain to these two roots, and every CI image whose CA stores still trust them keeps the exposure. Jira: APPSEC-71282 (parent APPSEC-71273).
Implementation details
tracer/build/_build/SmokeTests/smoke-test-images.docker-compose.yml: re-pins 19mcr.microsoft.com/dotnet/aspnetruntime tags (8.0, 9.0, 10.0, 11.0; bookworm-slim, jammy, noble, resolute, alpine 3.24, chiseled and composite variants) to their current MCR digests.tracer/build/_build/Build.VariableGenerations.cs: the dd-dotnet failure-test matrix runtime image moves from EOLaspnet:9.0-alpine3.20toaspnet:9.0-alpine3.24. That matrix is a small representative set (one image per package format), so the Alpine minor is not what it tests.smoke.dockerfile,smoke.chiseled,smoke.dd-dotnet,smoke.dotnet-tool,smoke.dotnet-tool.nuget,smoke.dotnet-tool.self-instrument,smoke.nuget,smoke.nuget.dd-dotnet,smoke.trimming): no published SDK tag is clean, so a newsdk-patchedstage (FROM mcr.microsoft.com/dotnet/sdk:$DOTNETSDK_VERSION) removes both roots from the SDK's NuGet trust lists (sdk/*/trustedroots/*.pem), and each builder stage buildsFROM sdk-patched.alpine.build,alpine.build.arm64,centos7.build,debian) moved to Stop trusting compromised eMudhra roots in the build-toolchain images #9528, because they need image republishing and VM rebuilds (andrewlock's review on Stop trusting compromised eMudhra roots in CI and smoke-test images #9517).Test coverage
Every new image reference was pulled and every changed Dockerfile was built, then scanned with the eMudhra scanner (it checks every file in the image for the two roots' public keys); all returned empty hits. Images that CI never builds or pulls are left alone.
smoke.dockerfile --target builder(includingdotnet publishfor net10.0, which restores its packages) built with the stripped trust lists, so the packages it restores do not depend on signatures chaining to these roots.Other details
Follow-ups (not in this PR):
public.ecr.aws/datadog/agent,registry.ddbuild.io/images/ci/images:1.2.6,registry.ddbuild.io/ci/serverless-tools:1,registry.ddbuild.io/images/benchmarking-platform-tools-ubuntu:newest,ghcr.io/datadog/dd-apm-test-agent/ddapm-test-agent:latest.docker-compose.yml, kept at their tested versions (3):postgres:10.5-alpine,rabbitmq:3-management,redis:4-alpine. Every newer image in their major lines still trusts the roots, and moving them to postgres 18 / rabbitmq 4 / redis 8 changes the server versions the integration matrix tests against (oldest clients: Npgsql 4.0.0, RabbitMQ.Client 3.6.9, StackExchange.Redis 1.0.187, ServiceStack.Redis 4.0.48). The owning team should pick supported server versions and run the integration tests.localstack/localstack:4.14.0,andrewlock/ok-api:latest,elasticsearch:7.10.1,mongo:5.0.31,mysql/mysql-server:8.0,mcr.microsoft.com/azure-sql-edge:latest,mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview,bentonam/couchbase-docker:community-5.0.1,docker.elastic.co/elasticsearch/elasticsearch:7.14.1,mcr.microsoft.com/mssql/server:latest,mcr.microsoft.com/azure-messaging/servicebus-emulator:1.1.2,mcr.microsoft.com/azure-storage/azurite:3.37.0,mcr.microsoft.com/azure-messaging/eventhubs-emulator:latest,workleap/eventgridemulator:0.6.37,confluentinc/cp-zookeeper:6.1.1,confluentinc/cp-server:6.1.1,mcr.microsoft.com/dotnet/aspnet:10.0-alpine3.22,mcr.microsoft.com/dotnet/aspnet:10.0-alpine3.22-composite,mcr.microsoft.com/dotnet/sdk:8.0-jammy,mcr.microsoft.com/dotnet/sdk:9.0-bookworm-slim,mcr.microsoft.com/dotnet/sdk:9.0-noble,mcr.microsoft.com/dotnet/sdk:10.0-alpine3.22,mcr.microsoft.com/dotnet/sdk:10.0-noble,mcr.microsoft.com/dotnet/sdk:10.0-resolute,mcr.microsoft.com/dotnet/sdk:11.0-alpine3.24,mcr.microsoft.com/dotnet/sdk:11.0-resolute,andrewlock/dotnet-centos-stream:8-3.1,andrewlock/dotnet-centos-stream:8-5.0,andrewlock/dotnet-centos-stream:8-6.0,andrewlock/dotnet-centos-stream:9-6.0,andrewlock/dotnet-centos-stream:9-9.0,andrewlock/dotnet-debian:trixie-10.0,andrewlock/dotnet-debian:trixie-8.0,andrewlock/dotnet-debian:trixie-9.0,andrewlock/dotnet-fedora:29-2.1,andrewlock/dotnet-fedora:29-3.1,andrewlock/dotnet-fedora:33-3.1,andrewlock/dotnet-fedora:33-5.0,andrewlock/dotnet-fedora:34-3.1,andrewlock/dotnet-fedora:34-5.0,andrewlock/dotnet-fedora:34-6.0,andrewlock/dotnet-fedora:35-3.1,andrewlock/dotnet-fedora:35-5.0,andrewlock/dotnet-fedora:35-7.0,andrewlock/dotnet-fedora:40-9.0,andrewlock/dotnet-fedora-arm64:34-6.0,andrewlock/dotnet-fedora-arm64:35-5.0,andrewlock/dotnet-fedora-arm64:35-7.0,andrewlock/dotnet-fedora-arm64:40-9.0,andrewlock/dotnet-opensuse:15-2.1,andrewlock/dotnet-opensuse:15-3.1,andrewlock/dotnet-opensuse:15-5.0,andrewlock/dotnet-opensuse:15-6.0,andrewlock/dotnet-opensuse:15-7.0,andrewlock/dotnet-opensuse:15-9.0,andrewlock/dotnet-opensuse:15-10.0,andrewlock/dotnet-rhel:8-3.1,andrewlock/dotnet-rhel:8-5.0,andrewlock/dotnet-rhel:8-6.0,andrewlock/dotnet-rhel:8-7.0,andrewlock/dotnet-rhel:8-9.0,andrewlock/dotnet-rhel:9-9.0,public.ecr.aws/lambda/dotnet:8,public.ecr.aws/lambda/dotnet:9,public.ecr.aws/lambda/dotnet:10,mcr.microsoft.com/dotnet/sdk.docker.elastic.co/elasticsearch/elasticsearch:6.4.2,docker.elastic.co/elasticsearch/elasticsearch:5.6.16,mysql/mysql-server:5.7,mcr.microsoft.com/dotnet/aspnet:2.1-alpine3.12,mcr.microsoft.com/dotnet/aspnet:2.1-stretch-slim,mcr.microsoft.com/dotnet/aspnet:3.1-alpine3.14,mcr.microsoft.com/dotnet/aspnet:3.1-bionic,mcr.microsoft.com/dotnet/aspnet:3.1-bullseye-slim,mcr.microsoft.com/dotnet/aspnet:5.0-alpine3.13,mcr.microsoft.com/dotnet/aspnet:5.0-alpine3.14,mcr.microsoft.com/dotnet/aspnet:5.0-buster-slim,mcr.microsoft.com/dotnet/aspnet:5.0-focal,mcr.microsoft.com/dotnet/aspnet:6.0-alpine3.14,mcr.microsoft.com/dotnet/aspnet:6.0-alpine3.18,mcr.microsoft.com/dotnet/aspnet:6.0-bullseye-slim,mcr.microsoft.com/dotnet/aspnet:7.0-alpine3.16,mcr.microsoft.com/dotnet/aspnet:7.0-alpine3.18,mcr.microsoft.com/dotnet/aspnet:7.0-bullseye-slim,mcr.microsoft.com/dotnet/aspnet:8.0-alpine3.18,mcr.microsoft.com/dotnet/aspnet:8.0-alpine3.18-composite,mcr.microsoft.com/dotnet/aspnet:8.0-alpine3.19,mcr.microsoft.com/dotnet/aspnet:8.0-alpine3.19-composite,mcr.microsoft.com/dotnet/aspnet:8.0-alpine3.20,mcr.microsoft.com/dotnet/aspnet:9.0-alpine3.20,mcr.microsoft.com/dotnet/aspnet:9.0-alpine3.20-composite,mcr.microsoft.com/dotnet/sdk:3.1-alpine3.15,mcr.microsoft.com/dotnet/sdk:3.1-bullseye,mcr.microsoft.com/dotnet/sdk:6.0-alpine3.16,mcr.microsoft.com/dotnet/sdk:6.0-bullseye-slim,mcr.microsoft.com/dotnet/sdk:7.0-alpine3.16,mcr.microsoft.com/dotnet/sdk:7.0-bullseye-slim,mcr.microsoft.com/dotnet/sdk:8.0-alpine3.18,mcr.microsoft.com/dotnet/sdk:9.0-alpine3.20,andrewlock/dotnet-centos:7-2.1,andrewlock/dotnet-centos:7-3.1,andrewlock/dotnet-centos:7-5.0,andrewlock/dotnet-centos:7-6.0,andrewlock/dotnet-centos:7-7.0,public.ecr.aws/lambda/dotnet:6,public.ecr.aws/lambda/dotnet:7.486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/benchmarking-platform:dotnet-throughput-9.