Nexus holds on to specific service backends of crdb and crucible-pantry for too long

[askfongjojo]

While testing service failover on rack2, I noticed that nexus held on to the same cockroachdb backend without attempting to use any of the other nodes in the 5-node database cluster, causing requests to fail until the one it favored came back.

The same happened with pantry requests for disk import blocks / bulk writes. I haven't got to the point of seeing the TTL being exhausted. I tried waiting for up to 5 minutes and the request still couldn't succeed until the pantry zone Nexus has been using prior to its outage came up again.

[smklein]

I think ignoring the TTL makes sense (not that it's right, just that it makes sense) in the context of this code:

omicron/nexus/src/context.rs

Lines 167 to 195 in a29f08b

	let url = match &config.deployment.database {
	nexus_config::Database::FromUrl { url } => url.clone(),
	nexus_config::Database::FromDns => {
	info!(log, "Accessing DB url from DNS");
	let address = loop {
	match resolver
	.lookup_socket_v6(ServiceName::Cockroach)
	.await
	{
	Ok(address) => break address,
	Err(e) => {
	warn!(
	log,
	"Failed to lookup cockroach address: {e}"
	);
	tokio::time::sleep(std::time::Duration::from_secs(
	1,
	))
	.await;
	}
	}
	};
	info!(log, "DB address: {}", address);
	PostgresConfigWithUrl::from_str(&format!(
	"postgresql://root@{address}/omicron?sslmode=disable",
	))
	.map_err(|e| format!("Cannot parse Postgres URL: {}", e))?
	}
	};

Our API to interface with CockroachDB from Nexus involves "constructing a URL", and right now, we embed a single point-in-time address of CockroachDB into this URL.

We should probably avoid doing this, and embed the service name into the URL, so it actually use DNS during later lookups.

[askfongjojo]

Understood. The current behavior, in conjunction with #3613, results in an 1 in 15 probability that a user will not be able to use the system at all. I happened to be in this situation when my workstation always used the one nexus that had the first CRDB backend I brought down to test failover. :(

[jmpesp]

The same happened with pantry requests for disk import blocks / bulk writes. I haven't got to the point of seeing the TTL being exhausted. I tried waiting for up to 5 minutes and the request still couldn't succeed until the pantry zone Nexus has been using prior to its outage came up again.

In this case, there's no TTL: a specific pantry from the list of available ones is selected once and used for a disk until that disk is finalized.

One thing that could be done is to check of the selected pantry is still responsive and choose another one if not, but it was important not keep as much code out of the import chunk hot-path as possible, as any checks there will be multiplied by the number of chunks to import and slow down imports. That being said, slow imports are better than non-working imports :) I'll give this some thought.

[askfongjojo]

Sorry for not being clear in pantry's case. I understand the need to stay with the same pantry for the same disk snapshot. But the issue I ran into is with new/separate snapshot requests still holding on to the unresponsive one.

[luqmana]

#3783 partially addresses this for crdb. We now, grab all the cockroachdb hosts via internal dns at nexus startup and add them all to the connection string. Whenever a new connection is established it'll try the listed hosts in order and use the first one that successfully connects. While that's an improvement in that Nexus won't fail to serve requests if one crdb instance goes down, there are still some issues:

1. The list of hosts used are resolved and get fixed at Nexus startup. If new cockroachdb zones get spun up, it'll take a restart of Nexus for it to pick them up. As-is, I don't think we're yet dynamically spinning up new services and instead just creating the service plan during RSS.
2. Internal DNS returns records in the same order everytime (couldn't find existing issue so created one: Internal DNS should permute the order in which it responds for entries with multiple records. #3805). This means every Nexus zone tries to connect to the set of CRDB hosts in the same order.