[eliottness]

Summary of changes

Stops this repository's CI, build and smoke-test images from trusting emSign ECC Root CA - G3 and AAA Certificate Services. The APPSEC-71273 ledger lists 152 impacted image references here: 20 fixed in this PR, 3 fixed at their in-repo image source in #9523, 129 follow-ups.

Reason for change

Incident IR-62343: the compromised eMudhra intermediates chain to these two roots, and every CI image whose CA stores still trust them keeps the exposure. Jira: APPSEC-71282 (parent APPSEC-71273).

Implementation details

• tracer/build/_build/SmokeTests/smoke-test-images.docker-compose.yml: re-pins 19 mcr.microsoft.com/dotnet/aspnet runtime tags (8.0, 9.0, 10.0, 11.0; bookworm-slim, jammy, noble, resolute, alpine 3.24, chiseled and composite variants) to their current MCR digests.
• tracer/build/_build/Build.VariableGenerations.cs: the dd-dotnet failure-test matrix runtime image moves from EOL aspnet:9.0-alpine3.20 to aspnet:9.0-alpine3.24. That matrix is a small representative set (one image per package format), so the Alpine minor is not what it tests.
• Smoke Dockerfiles (smoke.dockerfile, smoke.chiseled, smoke.dd-dotnet, smoke.dotnet-tool, smoke.dotnet-tool.nuget, smoke.dotnet-tool.self-instrument, smoke.nuget, smoke.nuget.dd-dotnet, smoke.trimming): no published SDK tag is clean, so a new sdk-patched stage (FROM mcr.microsoft.com/dotnet/sdk:$DOTNETSDK_VERSION) removes both roots from the SDK's NuGet trust lists (sdk/*/trustedroots/*.pem), and each builder stage builds FROM sdk-patched.
• The build-toolchain Dockerfiles (alpine.build, alpine.build.arm64, centos7.build, debian) moved to Stop trusting compromised eMudhra roots in the build-toolchain images #9523, because they need image republishing and VM rebuilds (see the review below).

Test coverage

Every new image reference was pulled and every changed Dockerfile was built, then scanned with the eMudhra scanner (it checks every file in the image for the two roots' public keys); all returned empty hits. Images that CI never builds or pulls are left alone.

• NuGet trust: smoke.dockerfile --target builder (including dotnet publish for net10.0, which restores its packages) built with the stripped trust lists, so the packages it restores do not depend on signatures chaining to these roots.
• This is a fork PR, so the Azure DevOps pipelines (the real build, smoke and integration tests) have not run. A maintainer needs to comment /azp run or push the branch to DataDog/dd-trace-dotnet.
• add-labels fails because the PR needs a label, and the author cannot add one from a fork (area:builds fits). check_for_code_freeze fails only because the status API returns 403 for fork PRs; its log says "No code freeze is in place".

Other details

Follow-ups (not in this PR):

• Needs a rebuild at its source (Datadog-built or mirrored image) (5): public.ecr.aws/datadog/agent, registry.ddbuild.io/images/ci/images:1.2.6, registry.ddbuild.io/ci/serverless-tools:1, registry.ddbuild.io/images/benchmarking-platform-tools-ubuntu:newest, ghcr.io/datadog/dd-apm-test-agent/ddapm-test-agent:latest.
• Integration-test servers in docker-compose.yml, kept at their tested versions (3): postgres:10.5-alpine, rabbitmq:3-management, redis:4-alpine. Every newer image in their major lines still trusts the roots, and moving them to postgres 18 / rabbitmq 4 / redis 8 changes the server versions the integration matrix tests against (oldest clients: Npgsql 4.0.0, RabbitMQ.Client 3.6.9, StackExchange.Redis 1.0.187, ServiceStack.Redis 4.0.48). The owning team should pick supported server versions and run the integration tests.
• No clean upstream version yet (66): localstack/localstack:4.14.0, andrewlock/ok-api:latest, elasticsearch:7.10.1, mongo:5.0.31, mysql/mysql-server:8.0, mcr.microsoft.com/azure-sql-edge:latest, mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview, bentonam/couchbase-docker:community-5.0.1, docker.elastic.co/elasticsearch/elasticsearch:7.14.1, mcr.microsoft.com/mssql/server:latest, mcr.microsoft.com/azure-messaging/servicebus-emulator:1.1.2, mcr.microsoft.com/azure-storage/azurite:3.37.0, mcr.microsoft.com/azure-messaging/eventhubs-emulator:latest, workleap/eventgridemulator:0.6.37, confluentinc/cp-zookeeper:6.1.1, confluentinc/cp-server:6.1.1, mcr.microsoft.com/dotnet/aspnet:10.0-alpine3.22, mcr.microsoft.com/dotnet/aspnet:10.0-alpine3.22-composite, mcr.microsoft.com/dotnet/sdk:8.0-jammy, mcr.microsoft.com/dotnet/sdk:9.0-bookworm-slim, mcr.microsoft.com/dotnet/sdk:9.0-noble, mcr.microsoft.com/dotnet/sdk:10.0-alpine3.22, mcr.microsoft.com/dotnet/sdk:10.0-noble, mcr.microsoft.com/dotnet/sdk:10.0-resolute, mcr.microsoft.com/dotnet/sdk:11.0-alpine3.24, mcr.microsoft.com/dotnet/sdk:11.0-resolute, andrewlock/dotnet-centos-stream:8-3.1, andrewlock/dotnet-centos-stream:8-5.0, andrewlock/dotnet-centos-stream:8-6.0, andrewlock/dotnet-centos-stream:9-6.0, andrewlock/dotnet-centos-stream:9-9.0, andrewlock/dotnet-debian:trixie-10.0, andrewlock/dotnet-debian:trixie-8.0, andrewlock/dotnet-debian:trixie-9.0, andrewlock/dotnet-fedora:29-2.1, andrewlock/dotnet-fedora:29-3.1, andrewlock/dotnet-fedora:33-3.1, andrewlock/dotnet-fedora:33-5.0, andrewlock/dotnet-fedora:34-3.1, andrewlock/dotnet-fedora:34-5.0, andrewlock/dotnet-fedora:34-6.0, andrewlock/dotnet-fedora:35-3.1, andrewlock/dotnet-fedora:35-5.0, andrewlock/dotnet-fedora:35-7.0, andrewlock/dotnet-fedora:40-9.0, andrewlock/dotnet-fedora-arm64:34-6.0, andrewlock/dotnet-fedora-arm64:35-5.0, andrewlock/dotnet-fedora-arm64:35-7.0, andrewlock/dotnet-fedora-arm64:40-9.0, andrewlock/dotnet-opensuse:15-2.1, andrewlock/dotnet-opensuse:15-3.1, andrewlock/dotnet-opensuse:15-5.0, andrewlock/dotnet-opensuse:15-6.0, andrewlock/dotnet-opensuse:15-7.0, andrewlock/dotnet-opensuse:15-9.0, andrewlock/dotnet-opensuse:15-10.0, andrewlock/dotnet-rhel:8-3.1, andrewlock/dotnet-rhel:8-5.0, andrewlock/dotnet-rhel:8-6.0, andrewlock/dotnet-rhel:8-7.0, andrewlock/dotnet-rhel:8-9.0, andrewlock/dotnet-rhel:9-9.0, public.ecr.aws/lambda/dotnet:8, public.ecr.aws/lambda/dotnet:9, public.ecr.aws/lambda/dotnet:10, mcr.microsoft.com/dotnet/sdk.
• EOL test target kept on purpose (moving the tag would change the tested version) (40): docker.elastic.co/elasticsearch/elasticsearch:6.4.2, docker.elastic.co/elasticsearch/elasticsearch:5.6.16, mysql/mysql-server:5.7, mcr.microsoft.com/dotnet/aspnet:2.1-alpine3.12, mcr.microsoft.com/dotnet/aspnet:2.1-stretch-slim, mcr.microsoft.com/dotnet/aspnet:3.1-alpine3.14, mcr.microsoft.com/dotnet/aspnet:3.1-bionic, mcr.microsoft.com/dotnet/aspnet:3.1-bullseye-slim, mcr.microsoft.com/dotnet/aspnet:5.0-alpine3.13, mcr.microsoft.com/dotnet/aspnet:5.0-alpine3.14, mcr.microsoft.com/dotnet/aspnet:5.0-buster-slim, mcr.microsoft.com/dotnet/aspnet:5.0-focal, mcr.microsoft.com/dotnet/aspnet:6.0-alpine3.14, mcr.microsoft.com/dotnet/aspnet:6.0-alpine3.18, mcr.microsoft.com/dotnet/aspnet:6.0-bullseye-slim, mcr.microsoft.com/dotnet/aspnet:7.0-alpine3.16, mcr.microsoft.com/dotnet/aspnet:7.0-alpine3.18, mcr.microsoft.com/dotnet/aspnet:7.0-bullseye-slim, mcr.microsoft.com/dotnet/aspnet:8.0-alpine3.18, mcr.microsoft.com/dotnet/aspnet:8.0-alpine3.18-composite, mcr.microsoft.com/dotnet/aspnet:8.0-alpine3.19, mcr.microsoft.com/dotnet/aspnet:8.0-alpine3.19-composite, mcr.microsoft.com/dotnet/aspnet:8.0-alpine3.20, mcr.microsoft.com/dotnet/aspnet:9.0-alpine3.20, mcr.microsoft.com/dotnet/aspnet:9.0-alpine3.20-composite, mcr.microsoft.com/dotnet/sdk:3.1-alpine3.15, mcr.microsoft.com/dotnet/sdk:3.1-bullseye, mcr.microsoft.com/dotnet/sdk:6.0-alpine3.16, mcr.microsoft.com/dotnet/sdk:6.0-bullseye-slim, mcr.microsoft.com/dotnet/sdk:7.0-alpine3.16, mcr.microsoft.com/dotnet/sdk:7.0-bullseye-slim, mcr.microsoft.com/dotnet/sdk:8.0-alpine3.18, mcr.microsoft.com/dotnet/sdk:9.0-alpine3.20, andrewlock/dotnet-centos:7-2.1, andrewlock/dotnet-centos:7-3.1, andrewlock/dotnet-centos:7-5.0, andrewlock/dotnet-centos:7-6.0, andrewlock/dotnet-centos:7-7.0, public.ecr.aws/lambda/dotnet:6, public.ecr.aws/lambda/dotnet:7.
• Unscannable from outside the registry (pull access denied) (1): 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/benchmarking-platform:dotnet-throughput-9.
• Not used by any CI job, left untouched: 13 reference(s).

[eliottness]

/azp run

[andrewlock]

@eliottness you will need to open up this PR against the repo directly, we don't run CI against fork.

It's also likely going to cause a lot of failures, because we will need to rebuild our VMs to re-pull all the images in our docker-compose etc. I would suggest splitting this into "safe" changes (the docker smoke tests, which we pull fresh each time) and ones which will need additional work to merge 🙂

[eliottness]

Thanks @andrewlock, that makes sense. Split done:

• This PR now only contains changes that CI pulls fresh on every run:
  • the sdk-patched stage in the smoke Dockerfiles;
  • the aspnet runtime re-pins in smoke-test-images.docker-compose.yml;
  • the dd-dotnet failure-test runtime image in Build.VariableGenerations.cs. Tell me if that one also depends on the VM images.
• The build-toolchain Dockerfiles (alpine.build, alpine.build.arm64, centos7.build, debian) moved to Stop trusting compromised eMudhra roots in the build-toolchain images #9523.
• The docker-compose.yml service bumps were already reverted; they are listed as follow-ups.

On running CI: I don't have write access to DataDog/dd-trace-dotnet, so I can't push the branch to the repo myself. Could someone on apm-dotnet push eliottness/emudhra-ca-roots there, or would you rather I request write access?

[andrewlock]

@eliottness You can request access through Mosaic 🙂

[eliottness]

Superseded by #9527: same commit, now opened from a branch in DataDog/dd-trace-dotnet so CI runs on it. @andrewlock, as you suggested.