fm: add certificate diagnosis engine for expiring silo TLS certificates by smklein · Pull Request #11238 · oxidecomputer/omicron

smklein · 2026-09-04T16:59:02Z

Nexus serves each silo's external API with the silo's certificate whose leaf not_after is latest (ExternalEndpoint::best_certificate). When that certificate is about to expire or has expired, nothing told the operator.

This adds a fault management diagnosis engine (nexus/fm/src/diagnosis/certificate.rs) that opens a case per silo when the best certificate is inside a configurable warning window or already expired. The case carries one fact, BestCertificateExpiring or BestCertificateExpired, and requests a silo.certificate.expiring or silo.certificate.expired alert whenever that fact changes: entering the window, passing not_after, or a different (still expiring or expired) certificate becoming the best one. Unchanged input requests nothing. The case closes once a later-expiring certificate is installed or the silo is removed.

The warning window is a new FmConfig setting, certificate_expiry_warning_days (default 30, at most 3650). This value can be modified with omdb nexus fm-config set --certificate-expiry-warning-days <N>.

Supporting changes:

omicron-certificates: leaf_validity parses a PEM chain's leaf not_before/not_after into chrono timestamps.
nexus-types: DiagnosisEngineKind::Certificate, the CertificateFact payloads, the ObservedSiloCertificates analysis input, the two alert classes with V0 payloads, the FmConfig setting, and an input report section listing observed silos and certificates.
Database changes (schema/crdb/fm-certificate-de): diagnosis_engine and alert_class enum values, the fm_fact_certificate table (typed columns, one table per engine, as for disks and sagas), and the fm_config column with its CHECK constraint.
nexus-fm: Input now carries the FmConfig and the observed silo certificates (both required inputs). The fm_analysis task loads them from the same silo and certificate rows the external_endpoints task reads.

Not in scope, as follow-ups: not_before handling (to be added to best_certificate and the engine together), alerting on silos with no certificates at all, and exposing not_after in the external_endpoints task status.

Nexus serves each silo's external API with the silo's certificate whose leaf not_after is latest (ExternalEndpoint::best_certificate). When that certificate is about to expire or has expired, no later-expiring replacement is installed, and until now nothing told the operator. Add a fault management diagnosis engine that predicts the same choice and opens a case per silo when the best certificate is inside a configurable warning window or already expired. The case carries one fact (BestCertificateExpiring or BestCertificateExpired) and requests a silo.certificate.expiring or silo.certificate.expired alert whenever that fact changes. The case closes once a later-expiring certificate is installed or the silo is removed. Silos with no certificates open no case. Like best_certificate, the engine ignores not_before. The warning window is a new FmConfig setting, certificate_expiry_warning_days (default 30, at most 3650), settable via omdb nexus fm-config set --certificate-expiry-warning-days. Supporting changes: - omicron-certificates: leaf_validity parses a PEM chain's leaf not_before/not_after into chrono timestamps. - nexus-types: DiagnosisEngineKind::Certificate, CertificateFact payloads, the ObservedSiloCertificates analysis input, the two alert classes with V0 payloads, and the FmConfig setting. - Schema version 300 (fm-certificate-de): diagnosis_engine and alert_class enum values, the fm_fact_certificate table, and the fm_config column with its CHECK constraint. - nexus-fm: Input carries the FmConfig and observed silo certificates; the fm_analysis task loads them from the silo and certificate tables.

smklein · 2026-09-04T17:54:38Z

+    pub comment: String,
+
+    /// The silo this fact is about.
+    pub silo_id: Uuid,


Silos and certs don't have a strongly-typed UUID (yet), they probably should

A fact's comment is recorded when the fact is added and carried forward unchanged, so a relative phrase like "in 10days" or "1day ago" goes stale as sitreps advance. Record the absolute not_after only.

smklein · 2026-09-04T18:07:08Z

+    Ok(ParsedCertificateCase { silo_id, fact: kept, duplicate_facts })
+}
+
+pub(super) fn analyze(builder: &mut SitrepBuilder<'_>) -> anyhow::Result<()> {


A lot of this function is coping with "observing parent cases, reconciliation of facts". I'll annotate where the more important "new case opened, alert created" stuff happens.

smklein · 2026-09-04T18:07:44Z

+    for silo in silos.iter() {
+        let Some((desired, best)) = desired_fact(silo, reference_time, window)


This is where we flag certs which are "expired, or about to be"

smklein · 2026-09-04T18:10:55Z

+
+        let carried = parent.and_then(|(_, p)| p.fact.as_ref());
+        if carried.map(|(_, fact)| fact) == Some(&desired) {
+            continue;


we bail here if the fact was the same as the parent, so below this, the fact is new - meaning we need a new alert.

smklein · 2026-09-04T18:11:22Z

+            name: best.name.clone(),
+            not_after: best.not_after,
+        };
+        let alert_result = match &desired {


... and this is where we actually create alerts

smklein · 2026-09-04T18:43:02Z

+// License, v. 2.0. If a copy of the MPL was not distributed with this
+// file, You can obtain one at https://mozilla.org/MPL/2.0/.
+
+//! Silo TLS certificate alert types.


So, as an FYI, these alerts only go out to fleet-level operators. This isn't a new decision in this PR, it's sorta how alerts work generally.

However, certificate management is silo-scoped.

I bring this up because we don't send these alerts to "silo admins" right now - we send them to "fleet admins", who presumably will need to coordinate with silo admins to get the certificates patched.

This does technically expose certificate names within a silo to the fleet operator, but that SEEMS okay to me?

Yeah. I think we should probably be tracking the work needed to have silo-scoped (and also project-scoped) alerts. We punted on that from the initial implementation, but I think there are going to be a bunch of cases where we want them...and probably the same for Active Problems, to the extent that I'm wondering if I should rethink some of what I wrote in RFD 699 about authz.

In the meantime we should probably put together a ticket for having silo/project alerts so we can reference this there.

Filed #11244

hawkw · 2026-09-04T22:28:44Z

+/// from the Unix epoch.
+///
+/// `Asn1TimeRef` offers no direct conversion to a Unix timestamp, but
+/// `ASN1_TIME_diff` can compute the (days, seconds) difference between two


what is ASN1_TIME_diff?

oh i guess it's the name of the C function that Asn1Time::diff is a binding to.

hawkw · 2026-09-04T22:30:26Z

+    let epoch = Asn1Time::from_unix(0).map_err(CertificateError::Unexpected)?;
+    // `diff` computes `time - epoch`, split into whole days and the
+    // remaining seconds.
+    let diff = epoch.diff(time).map_err(CertificateError::Unexpected)?;


hm, do we know the errors that diff would return? is it possible that any of these would be time out of range-y?

hawkw · 2026-09-04T22:30:52Z

+    // `diff` computes `time - epoch`, split into whole days and the
+    // remaining seconds.
+    let diff = epoch.diff(time).map_err(CertificateError::Unexpected)?;
+    let secs = i64::from(diff.days) * SECS_PER_DAY + i64::from(diff.secs);


should we use checked/saturating arithmetic here, since the cert is user controlled and might be overflowy?

hawkw · 2026-09-04T22:32:36Z

    }
+
+    #[test]
+    fn test_leaf_validity_reads_leaf_certificate() {


this feels like it might perhaps wanna be a proptest someday.

hawkw · 2026-09-04T22:33:05Z

+        // Pin the leaf's validity window to exact second offsets from the
+        // Unix epoch. The root and intermediate certificates in the chain
+        // keep rcgen's default (much wider) window, so a correct result
+        // proves we read the leaf and not some other link in the chain.


this feels very claudeprose-y

hawkw · 2026-09-04T22:48:09Z

+        let config = self
+            .config
+            .ok_or(InvalidInputs::MissingInput { name: "config" })?;


i feel like this should maybe be its own error variant so that it can be turned into the "waiting for config" analysis status in the bg task

hawkw · 2026-09-04T22:49:36Z

+    // Expiry is judged against the input's deterministic "now"; see
+    // `Input::reference_time`.
+    let reference_time = input.reference_time();


thanks for adding this, i had been thinking we were gonna want something like this for a while now. (i was just gonna call it input.now(), but maybe your name is better!)

hawkw · 2026-09-04T22:52:37Z


 /// Summarizes the inputs to sitrep analysis.
 #[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
 pub struct InputReport {


i feel like the analysis' reference time belongs here too? interpreting the analysis results will almost certainly want to know what time we thought "now" was when we were doing stuff

hawkw · 2026-09-04T22:54:09Z

+        //
+        // The fault management certificate diagnosis engine
+        // (`nexus_fm::diagnosis::certificate`) predicts this choice from the
+        // same rule, so that it alerts when the certificate we will actually
+        // serve is expiring or expired. If the rule here changes, the engine
+        // (and `ObservedSiloCertificates::best_certificate`, which it uses)
+        // must change with it.


this really makes me feel like there ought to be a function someplace that both this code and FM could share, rather than "you have to remember to update the other thing too lol"...

hawkw · 2026-09-04T22:57:25Z

(i should add that it's neat to see DEs for simple, but high-impact purely software/config issues starting to be written! thanks for doing this!)

The certificate diagnosis engine's loader accepted any certificate whose PEM parsed, but ExternalEndpoints::new also drops certificates whose key rustls cannot load. Such a certificate could be chosen as the silo's best and hide an expiring one that is served. Run the loader's rows through TlsCertificate::try_from so both sides agree.

The "silo not found" warning was wrapped without a trailing backslash, leaving a run of interior whitespace in the text omdb prints.

ObservedSiloCertificates::best_certificate claimed callers depended only on not_after, but the engine stores the chosen certificate's id in its fact and re-alerts when it changes. Break ties toward the greatest id, document why, and test that equal-expiry certificates carry the fact unchanged across sitreps.

TlsCertificate now carries the leaf validity as a DateTime, so the served-certificate rule can be the same max_by_key expression the certificate diagnosis engine uses, instead of a hand-rolled loop over Asn1Time partial_cmp. Drop the parsed X509 field, which had no other readers left.

Both certificate fact kinds carry the same payload, so the nullable columns plus two identical per-kind CHECK constraints were NOT NULL spelled twice. Declare them NOT NULL, drop the Option fields and the unreachable missing-column error path in the Diesel model.

The loader hand-rolled the same pagination loop the datastore helper already provides, with an ad-hoc batch size. Use the helper for silos and the standard SQL_BATCH_SIZE for certificates.

parse_case returns NoFacts when a case has no facts, so a successfully parsed case always has one. Store it directly and derive the silo id from it instead of threading an Option through the reconcile loop.

smklein marked this pull request as draft September 4, 2026 17:19

smklein commented Sep 4, 2026

View reviewed changes

fm: name only the absolute expiration time in certificate fact comments

e8c3674

A fact's comment is recorded when the fact is added and carried forward unchanged, so a relative phrase like "in 10days" or "1day ago" goes stale as sitreps advance. Record the absolute not_after only.

smklein commented Sep 4, 2026

View reviewed changes

smklein marked this pull request as ready for review September 4, 2026 18:43

smklein requested review from hawkw and mergeconflict September 4, 2026 19:01

smklein assigned augustuswm and unassigned augustuswm Sep 4, 2026

smklein requested a review from augustuswm September 4, 2026 19:01

smklein mentioned this pull request Sep 4, 2026

Alerts should have more granular scoping than "fleet" #11244

Open

hawkw reviewed Sep 4, 2026

View reviewed changes

smklein added 5 commits September 4, 2026 16:23

fm: fix wrapped warning string in the certificate loader

4d7489d

The "silo not found" warning was wrapped without a trailing backslash, leaving a run of interior whitespace in the text omdb prints.

This was referenced Sep 4, 2026

fm: extract shared keyed-case adoption scaffolding from the saga and certificate engines #11246

Open

fm: extract shared keyed-case adoption scaffolding from the saga and certificate engines #11247

Closed

smklein added 2 commits September 4, 2026 16:55

fm: use silo_list_all_batched in the certificate loader

de88ede

The loader hand-rolled the same pagination loop the datastore helper already provides, with an ad-hoc batch size. Use the helper for silos and the standard SQL_BATCH_SIZE for certificates.

fm: make the parsed certificate case's fact non-optional

4a64593

parse_case returns NoFacts when a case has no facts, so a successfully parsed case always has one. Store it directly and derive the silo id from it instead of threading an Option through the reconcile loop.

		for silo in silos.iter() {
		let Some((desired, best)) = desired_fact(silo, reference_time, window)

Uh oh!

Conversation

smklein commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hawkw commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

smklein commented Sep 4, 2026 •

edited

Loading