Skip to content

Revert "Stop trusting compromised eMudhra roots in the build-toolchain images (#9528)" #9540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
eliottness wants to merge 1 commit into from
Closed

Revert "Stop trusting compromised eMudhra roots in the build-toolchain images (#9528)" #9540

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 0 additions & 8 deletions tracer/build/_build/docker/alpine.build.arm64.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,6 @@
# See alpine.build.dockerfile for more information on how to build this image
FROM alpine:3.18 as base

# alpine 3.18 is the musl baseline for linux-musl-arm64 (tracer/build/artifacts/requirements.json) and is EOL,
# so no fixed ca-certificates package will ship: drop the distrusted eMudhra roots
# "emSign ECC Root CA - G3" / "AAA Certificate Services" (IR-62343) from the system trust store.
RUN apk add --no-cache --upgrade ca-certificates ca-certificates-bundle \
&& if [ -f /etc/ca-certificates.conf ]; then sed -i -e 's|^mozilla/emSign_ECC_Root_CA_-_G3.crt|!&|' -e 's|^mozilla/Comodo_AAA_Services_root.crt|!&|' /etc/ca-certificates.conf; fi \
&& rm -f /usr/share/ca-certificates/mozilla/emSign_ECC_Root_CA_-_G3.crt /usr/share/ca-certificates/mozilla/Comodo_AAA_Services_root.crt \
&& update-ca-certificates

RUN apk update \
&& apk upgrade \
&& apk add --no-cache \
Expand Down
8 changes: 0 additions & 8 deletions tracer/build/_build/docker/alpine.build.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,6 @@
# docker manifest push andrewlockdd/alpine-clang:dotnet10
FROM alpine:3.17 as base

# alpine 3.17 is the pinned musl toolchain baseline for linux-musl-x64 (see tracer/build/artifacts/requirements.json
# and NativeValidation/Symbols.cs) and is EOL, so no fixed ca-certificates package will ship: drop the distrusted
# eMudhra roots "emSign ECC Root CA - G3" / "AAA Certificate Services" (IR-62343) from the system trust store.
RUN apk add --no-cache --upgrade ca-certificates ca-certificates-bundle \
&& if [ -f /etc/ca-certificates.conf ]; then sed -i -e 's|^mozilla/emSign_ECC_Root_CA_-_G3.crt|!&|' -e 's|^mozilla/Comodo_AAA_Services_root.crt|!&|' /etc/ca-certificates.conf; fi \
&& rm -f /usr/share/ca-certificates/mozilla/emSign_ECC_Root_CA_-_G3.crt /usr/share/ca-certificates/mozilla/Comodo_AAA_Services_root.crt \
&& update-ca-certificates

RUN apk update \
&& apk upgrade \
&& apk add --no-cache \
Expand Down
19 changes: 0 additions & 19 deletions tracer/build/_build/docker/centos7.build.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,6 @@ RUN sed -i s/mirror.centos.org/vault.centos.org/g /etc/yum.repos.d/*.repo \
&& sed -i s/^mirrorlist=http/#mirrorlist=http/g /etc/yum.repos.d/*.repo \
&& yum update -y \
&& yum install -y centos-release-scl \

@datadog-prod-us1-5 datadog-prod-us1-5 Bot Oct 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Preserve the SCLo repository repair during the revert

On a fresh CentOS toolchain build, centos-release-scl creates repository files after the initial vault rewrite. Removing the second rewrite leaves those SCLo repositories pointing at retired mirrorlist.centos.org, causing the subsequent yum install of devtoolset-11 to fail. Preserve the SCLo repository repair after centos-release-scl installation independently of the certificate rollback.

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

@bouwkast bouwkast Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eliottness I'm unsure if this is now a new issue caused by the prior merge.

@eliottness eliottness Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bouwkast for the record: no, that one predates #9528. mirrorlist.centos.org is gone, so the stock centos7.build.dockerfile no longer built on a cold cache. #9528 fixed it by moving the SCLo repos to vault, and reverting would have brought the break back. Moot now that the VMs are rebuilt and this is closed. Thanks for handling it, and sorry again for the early merge.

# centos-release-scl adds SCLo repos that still point at the retired mirrorlist.centos.org: fix them too
&& sed -i s/mirror.centos.org/vault.centos.org/g /etc/yum.repos.d/CentOS-SCLo-*.repo \
&& sed -i s/^#.*baseurl=http/baseurl=http/g /etc/yum.repos.d/CentOS-SCLo-*.repo \
&& sed -i s/^mirrorlist=http/#mirrorlist=http/g /etc/yum.repos.d/CentOS-SCLo-*.repo \
&& yum install -y git \
gcc \
gcc-c++ \
Expand All @@ -25,21 +21,6 @@ RUN sed -i s/mirror.centos.org/vault.centos.org/g /etc/yum.repos.d/*.repo \
devtoolset-11 \
python3

# centos 7 is the glibc 2.17 baseline for linux-x64 (tracer/build/artifacts/requirements.json) and is EOL, so no
# fixed ca-certificates package will ship. Runs after the yum update above, which would otherwise restore them:
# remove the distrusted eMudhra roots "emSign ECC Root CA - G3" / "AAA Certificate Services" (IR-62343) from the
# p11-kit source bundle, re-extract every derived store and make NSS read the p11-kit trust store.
RUN set -eux; \
src=/usr/share/pki/ca-trust-source/ca-bundle.trust.p11-kit; \
awk 'function flush() { if (!drop) printf "%s", buf; buf = ""; drop = 0 } \
/^\[p11-kit-object-v1\]/ { flush() } \
/^label: "(Comodo AAA Services root|AAA Certificate Services|emSign ECC Root CA - G3)"$/ { drop = 1 } \
{ buf = buf $0 "\n" } \
END { flush() }' "$src" > /tmp/ca-bundle.trust.p11-kit; \
mv /tmp/ca-bundle.trust.p11-kit "$src"; \
update-ca-trust extract; \
ln -sf /usr/lib64/pkcs11/p11-kit-trust.so /usr/lib64/nss/libnssckbi.so

FROM base AS build-cmake

# build cmake version
Expand Down
27 changes: 0 additions & 27 deletions tracer/build/_build/docker/debian.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,6 @@
FROM mcr.microsoft.com/dotnet/runtime-deps:5.0-buster-slim AS base
ARG DOTNETSDK_VERSION

@chatgpt-codex-connector chatgpt-codex-connector Bot Oct 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Codex Security Review · Automatically triggered

P1 Badge Security: Restore the build images' CA distrust rules

An attacker with a compromised eMudhra intermediate key and network interception can exploit this when the image performs an uncached download. Deleting the rules here makes the Debian builder trust that chain again; it then downloads and executes https://apt.llvm.org/llvm.sh without a digest, and CI runs the resulting image with the checkout mounted before publishing Linux tracer packages. Restore the system, NSS, and NuGet distrust rules (and the equivalent Alpine/CentOS rules) rather than trading them for cache performance.

Useful? React with 👍 / 👎.

# buster is EOL, so no fixed ca-certificates package will ship: drop the distrusted eMudhra roots
# "emSign ECC Root CA - G3" / "AAA Certificate Services" (IR-62343) from the system trust store.
RUN sed -i -e 's|^mozilla/emSign_ECC_Root_CA_-_G3.crt|!&|' -e 's|^mozilla/Comodo_AAA_Services_root.crt|!&|' /etc/ca-certificates.conf \
&& rm -f /usr/share/ca-certificates/mozilla/emSign_ECC_Root_CA_-_G3.crt /usr/share/ca-certificates/mozilla/Comodo_AAA_Services_root.crt \
&& update-ca-certificates -f

# Based on https://github.com/dotnet/dotnet-docker/blob/34c81d5f9c8d56b36cc89da61702ccecbf00f249/src/sdk/6.0/bullseye-slim/amd64/Dockerfile
# and https://github.com/dotnet/dotnet-docker/blob/1eab4cad6e2d42308bd93d3f0cc1f7511ac75882/src/sdk/5.0/buster-slim/amd64/Dockerfile
ENV \
Expand Down Expand Up @@ -89,27 +83,6 @@ RUN curl -sSL https://github.com/dotnet/install-scripts/raw/2bdc7f2c6e00d60be57f
# Trigger first run experience by running arbitrary cmd
&& dotnet help

# The .NET SDK ships its own NuGet signing trust bundles (sdk/*/trustedroots/*.pem) that still contain the
# distrusted eMudhra / AAA Certificate Services roots (IR-62343): strip those certificates from them.
RUN for f in /usr/share/dotnet/sdk/*/trustedroots/*.pem; do \
d=$(mktemp -d) \
&& awk -v d="$d" '/-----BEGIN CERTIFICATE-----/ { n++; out = sprintf("%s/%05d.pem", d, n); incert = 1 } incert { print > out } /-----END CERTIFICATE-----/ { incert = 0; close(out) }' "$f" \
&& : > "$f" \
&& for c in "$d"/*.pem; do \
openssl x509 -noout -subject -in "$c" | grep -qE 'CN ?= ?(emSign ECC Root CA - G3|emSign Root CS CA - G3|AAA Certificate Services)' || cat "$c" >> "$f"; \
done \
&& rm -rf "$d" || exit 1; \
done

# buster's libnss3 compiles the distrusted roots into libnssckbi.so and no fixed NSS will ship for buster:
# point NSS at p11-kit's trust module so it reads the patched system store above instead.
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends p11-kit-modules \
&& trust_module=$(find /usr/lib -name p11-kit-trust.so | head -n 1) \
&& test -n "$trust_module" \
&& find / -xdev -name 'libnssckbi.so*' -exec ln -sf "$trust_module" {} \; \
&& rm -rf /var/lib/apt/lists/*

ENV \
DOTNET_ROOT=/usr/share/dotnet \
DOTNET_ROLL_FORWARD_TO_PRERELEASE=1 \
Expand Down
Loading