Prompted by discussion #169 (@mernen). Decide whether to switch all BLAKE2b hashing to BLAKE3 before the repo format stabilizes.
Why it might be worth it
Hashing is a first-order CPU cost on two paths where dedup skips everything downstream:
- File-cache miss with mostly-unchanged content (VM images, databases, mail files): the file is fully re-read, FastCDC-chunked, and every chunk keyed-hashed (
pipeline/worker.rs); chunks that dedup out then skip compression/encryption/upload via an in-memory index lookup. Per-byte CPU there is read + chunking + BLAKE2b.
check --verify-data (check/scan.rs): every chunk is decrypted, decompressed, and re-hashed. In none mode hashing dominates.
Within a file (or 64 MiB segment of a large file) hashing is single-threaded, so per-core hash throughput is the per-segment ceiling.
Measured (single-thread, 2 MiB chunks over 1 GiB, Apple Silicon)
| hash |
GB/s |
keyed BLAKE2b-256 (RustCrypto blake2 0.10, as used today) |
~1.4 |
blake3::keyed_hash (NEON) |
~2.15 |
~1.5× on ARM. The gap is substantially larger on x86-64 with AVX2/AVX-512 (RustCrypto blake2 is portable-only, ~1 GB/s; blake3 reaches multi-GB/s per core there), so the win is platform-dependent — biggest on the x86 servers/NAS boxes where CPU is scarcest.
Feasibility
No hard blocker: the only keyed use is chunk IDs (vykar-types/src/chunk_id.rs, Blake2bMac<U32>) and BLAKE3 has a first-class keyed mode. No BLAKE2 personalization/salt/parameter-block features are used anywhere (domain separation is via AEAD AAD).
Blast radius (repo-format break)
- Chunk IDs (
chunk_id.rs) — dedup identity, all existing repos invalidated
- Pack IDs (
pack_id.rs)
X-Content-BLAKE2b upload-integrity header + server-side verify/repack (vykar-server)- TOFU repo fingerprint (
repo/identity.rs)
- Local dedup-cache checksum + file-cache formats
- Plaintext-mode
chunk_id_key = BLAKE2b-256(repo_id) derivation
Next steps
Prompted by discussion #169 (@mernen). Decide whether to switch all BLAKE2b hashing to BLAKE3 before the repo format stabilizes.
Why it might be worth it
Hashing is a first-order CPU cost on two paths where dedup skips everything downstream:
pipeline/worker.rs); chunks that dedup out then skip compression/encryption/upload via an in-memory index lookup. Per-byte CPU there is read + chunking + BLAKE2b.check --verify-data(check/scan.rs): every chunk is decrypted, decompressed, and re-hashed. Innonemode hashing dominates.Within a file (or 64 MiB segment of a large file) hashing is single-threaded, so per-core hash throughput is the per-segment ceiling.
Measured (single-thread, 2 MiB chunks over 1 GiB, Apple Silicon)
blake20.10, as used today)blake3::keyed_hash(NEON)~1.5× on ARM. The gap is substantially larger on x86-64 with AVX2/AVX-512 (RustCrypto
blake2is portable-only, ~1 GB/s;blake3reaches multi-GB/s per core there), so the win is platform-dependent — biggest on the x86 servers/NAS boxes where CPU is scarcest.Feasibility
No hard blocker: the only keyed use is chunk IDs (
vykar-types/src/chunk_id.rs,Blake2bMac<U32>) and BLAKE3 has a first-class keyed mode. No BLAKE2 personalization/salt/parameter-block features are used anywhere (domain separation is via AEAD AAD).Blast radius (repo-format break)
chunk_id.rs) — dedup identity, all existing repos invalidatedpack_id.rs)X-Content-BLAKE2bupload-integrity header + server-side verify/repack (vykar-server)repo/identity.rs)chunk_id_key = BLAKE2b-256(repo_id)derivationNext steps
check --verify-data) on x86-64 and ARM, BLAKE2b vs BLAKE3docs/src/architecture.mdto state the rationale vs BLAKE3, not just vs SHA-256