Dogfooding findings from anthropic-lb: response cache on cachekit-rs 0.5.0 (SecureCache + CachekitIO/Redis)

[27Bslash6]

Dogfooding findings: cachekit-rs 0.5.0 as the response cache in anthropic-lb

Context: 27b-io/anthropic-lb now ships an opt-in,
client-side-encrypted response cache on /v1/messages built entirely on cachekit-rs 0.5.0
(SecureCache + CachekitIO/Redis backends, per-tenant HKDF with tenant_id = client_id).
This is the write-back required by that ticket (LAB-933 AC15): what worked, what fought us,
verified against the cachekit-rs-v0.5.0 tag source.

What worked well

• The L1-ciphertext guarantee is real and verifiable. SecureCache::set_with_ttl
  encrypts before both the L1 write and the backend write (client.rs:413-448). We could
  cite the code path in our security review instead of hand-waving.
• Per-tenant HKDF made per-client isolation nearly free. One shared backend Arc,
  one CacheKit per tenant via .encryption_from_bytes(master_key, tenant_id) — cross-tenant
  reads fail AEAD instead of relying on key-string discipline. Exactly what a multi-client
  proxy wants.
• Ok(None) on miss (CachekitIO 404 → None, Redis nil → None) keeps the fail-open
  caller trivial: one match covers hit/miss/error/timeout.
• The public Backend trait is a genuinely good test seam — an in-memory recording
  backend let us assert "no plaintext substring ever reaches storage" in CI.

Findings

1. The CachekitIO backend has no test seam (the Backend trait can't save you here)

url_validator rejects loopback/private IPs even with allow_custom_host = true
(good SSRF default!), and the internal reqwest client isn't pluggable. Net effect: an
integration test can never point the real CachekitIO backend at a local mock server, so
the SaaS wire path (auth header, X-TTL, 404→None mapping, urlencoded keys) is untestable
in a consumer's CI. We had to ship the SaaS round-trip as an #[ignore]d live test
requiring CACHEKIT_API_KEY.

Suggestion: a feature-gated escape hatch (cfg(feature = "test-insecure-hosts")) or a
pluggable base-URL validator/transport, so consumers can exercise the real wire code
against 127.0.0.1 in CI without weakening production builds.

2. RedisBackend cannot auto-reconnect outside the intent presets

auto_reconnect() is pub(crate); the only paths to it are CacheKit::production /
CacheKit::encrypted. But ::encrypted hardcodes tenant_id = "default" — so any
consumer needing per-tenant encryption over Redis (us) must use the builder, and the
builder path silently loses reconnection: after a Redis restart, every op errors until the
process restarts. With fail-open callers this degrades to "cache permanently off",
which is survivable but wrong.

Suggestion: expose .auto_reconnect(bool) on RedisBackendBuilder. More generally,
the intent presets don't compose — production has no encryption, encrypted has no
tenant choice — so the builder needs feature-parity with them.

3. No operation-timeout control on either backend

• RedisBackend: fred 9 ships with no command timeout → a hung Redis blocks get forever
  unless the caller wraps every call in tokio::time::timeout (we did).
• CachekitIO: fixed 30 s total / 10 s connect, not configurable — far too long for a
  fail-open cache in a request path with a ~250 ms budget.

Suggestion: .op_timeout(Duration) on both builders. Callers wrapping externally works
but times out the future, not the underlying connection slot.

4. Raw-bytes payloads pay a msgpack shape tax unless the consumer knows the trick

set/get always serialize via rmp_serde::to_vec_named, and a plain Vec<u8> encodes
as an int array (size + CPU tax on multi-hundred-KB response bodies). Nothing in the
docs points at serde_bytes; we found it by reading serializer/mod.rs, then added
serde_bytes::ByteBuf-style fields ourselves.

Suggestion: either document "wrap binary payloads with serde_bytes" prominently, or
add set_raw/get_raw(&[u8]) — proxies caching opaque bodies are a natural SDK consumer.

5. Doc nits that cost us real time

• SecureCache<'a> is a borrowing handle — it cannot be stored in a struct next to its
  CacheKit. Calling .secure()? per operation is cheap and correct, but nothing says so.
• RedisBackend::connect() returns a ConnectHandle whose drop semantics ("detaches, the
  presets drop it") are only discoverable from preset source.
• TTL floor is 1 s (validate_ttl) — worth a line in the set_with_ttl docs.

None of these are blockers — the integration shipped, encrypted end-to-end, with the
Backend-trait seam carrying most of the test load. Findings 1–3 are the ones we'd most
like to see move: they're the gap between "works in prod" and "provable in CI".