[BLOCKED on #1398, #1400, #1423] drivers: add Crucible distributed block storage driver - #1424
Draft
gburd wants to merge 19 commits into
Draft
[BLOCKED on #1398, #1400, #1423] drivers: add Crucible distributed block storage driver#1424gburd wants to merge 19 commits into
gburd wants to merge 19 commits into
Conversation
OSv's master does not build on current toolchains. Two independent
breakages compound:
- Boost 1.78+ made boost::system header-only, so libboost_system.a
is no longer produced. The build hard-errored requiring it, and
Boost 1.87's lockfree headers use std::conditional_t, which needs
C++14.
- GCC 14 is stricter: core/mmu.cc's four-argument std::lower_bound
needs an explicit <algorithm> include, and Boost.Asio dropped the
long-deprecated io_service, resolver::query, mutable_buffers_1, and
address_v4::from_string APIs.
Changes:
- Default conf_cxx_level to gnu++14 (conf/base.mk, modules/common.gmk).
- Detect libboost_system only if present; never require it. Support
both nixpkgs (include/) and FHS (usr/include/) Boost layouts.
- Add <algorithm> to core/mmu.cc and core/pagecache.cc.
- Port Boost.Asio call sites to io_context, the range-returning
resolver::resolve, mutable_buffer, and make_address_v4
(loader.cc, core/dhcp.cc, tools/cpiod, modules/httpserver-api,
modules/cloud-init, modules/monitoring-agent, tests/*tcp*).
- Replace boost::math::isinf/isnan with std::isinf/isnan.
- Detect OpenSSL 3.x (libssl.so.3) in addition to 1.1, and fall back
to LD_LIBRARY_PATH when ldconfig is unavailable.
Bump the bundled musl from 1.1.24 (released 2019) to 1.2.1 (2020), the last version with the BSD-style internal layout that OSv's libc shim expects before musl's 1.2.x re-architecture. The upgrade picks up roughly four years of bug fixes in stdio, math, dns, and threading. Fix nftw(): the legacy 1.1.24 implementation had an uninitialized field that was masked by 1.1.x's malloc behaviour but tripped UBSan under 1.2.1. Initialise it explicitly so nftw() returns the same results regardless of allocator state. Verified: tst-libc, tst-pthread, tst-dns-resolver, tst-fnmatch all pass.
Implement the two Linux hugepage hint APIs against OSv's existing
2 MiB / 1 GiB page support:
- madvise(MADV_HUGEPAGE, addr, len): mark a VMA range so that the
page-fault handler will try to back it with 2 MiB pages when
enough physically-contiguous memory is available. The default
(MADV_NOHUGEPAGE) and an explicit MADV_HUGEPAGE produce the same
effect today; the hint is recorded for future use by the
allocator's huge-page heuristic.
- mmap(... MAP_HUGETLB ...): allocate the VMA directly out of the
huge-page pool (failing with ENOMEM rather than silently falling
back to small pages). Honours MAP_HUGE_2MB / MAP_HUGE_1GB flags.
Tests: tst-huge.cc exercises both APIs with mmap, mprotect, mlock,
fault-on-write, and partial-unmap to verify no PTE leaks or stale
mappings.
mmu::virt_to_phys() was correct only for kernel-direct-mapped addresses. When a driver called it on a VMA-mapped buffer (e.g., a DMA descriptor allocated via mmap with PROT_READ|PROT_WRITE), the function returned the linear-mapping offset of the page rather than the page's actual frame, producing silent DMA corruption when the buffer happened to be backed by a non-direct page. Walk the page tables instead and return the physical frame of the PTE that maps the address. Fall back to the linear-map fast-path when the address is in the kernel direct-map range. This is the underlying cause of intermittent virtio-blk and Crucible DMA misalignment seen on systems where the block-layer buffer pool was relocated to a heap region.
shm_file::put_page() previously returned without clearing the PTE
when the page was reused. After munmap, stale mappings persisted
on the next VMA at the same virtual address, so a freshly-mmaped
SHM segment could read whatever the previous tenant had written.
Fix: call clear_pte(ptep) before returning false from put_page()
(returning false so the TLB-gather path doesn't free the physical
page; shm_file::close() owns it).
Implement the rest of the POSIX shm surface that the fix exposed:
- shm_open() / shm_unlink() honour /dev/shm semantics.
- shmctl(IPC_STAT) reports correct nattch / size / mode.
- shm_file::truncate() resizes a segment in-place (the existing
sys_ftruncate() now delegates through fp->truncate() for files
without a backing dentry, e.g. SHM and io_uring queues).
Tests: tst-shm-consistency.cc, 30 cases covering nattch transitions,
dual-mapping write visibility, file-backed MAP_SHARED across two
mappings, and IPC_RMID idempotency. All pass on ZFS and ramfs.
in_vma_range() still used the pre-mem_area predicate (addr >= 0) to mean "non-VMA / high address". The mem_area refactor moved the debug-memory region to a positive base (debug_base), so that predicate matched every normal allocation: vpopulate's assert(!in_vma_range(addr)) fired on the first debug allocation at boot, making the conf_debug_memory allocator unbootable. Discriminate on debug_base, matching the test already used elsewhere in this file.
Three pagecache changes that work together to make the OSv VFS
page cache cooperate with OpenZFS:
- Expose C-linkage helpers (osv_pagecache_map_page, osv_pagecache_*)
so the OpenZFS vop_cache implementation in zfs_vnops_os.c can
register and look up cached pages without dragging the C++
pagecache headers into kernel-module sources. Also fixes a
GCC 14 ambiguity error on a templated helper.
- Remove the original ARC-bridge code path that tried to share
pages between the ZFS ARC and the OSv read_cache. It was never
reachable: IS_ZFS() always returned false on OSv (m_fsid
distinct), and the bridge data structures were only initialised
on the unreachable branch. Document the design decision in a
comment block above the (now removed) site so future readers
don't try the same approach again.
- Sequential readahead and a periodic writeback worker. The
readahead window grows on consecutive cache hits and resets on
seek; the writeback worker flushes dirty pages every 5 s with
a global cap so dirty pages can't accumulate without bound.
Verified by tst-mmap-file, tst-zfs-direct-io, and tst-fs-bench.
sys_fsync() called VOP_FSYNC directly without first flushing the OSv page cache. Dirty pages held in the cache were never seen by the underlying filesystem's fsync hook, so a process could fsync() a file and have the data still resident in volatile memory. Reproducer: write 64 KiB, fsync, kill the VM, restart, see zeros. Walk the file's dirty pages, write them back via the filesystem's write op, then call VOP_FSYNC. Holds the file's f_lock across the flush so concurrent writes can't slip in between the writeback and the VOP_FSYNC call. Verified by tst-zfs-direct-io and tst-zfs-multirec, which write, fsync, re-open, read, and memcmp. Without this fix the multi-record ZFS test produces zero-filled tail records on uncached read.
Two block-layer features that the ZFS and Crucible drivers depend on:
- BIO_DISCARD: a new request type plumbed through the block layer
and into virtio-blk's request descriptor (VIRTIO_BLK_T_DISCARD).
Maps to ZFS's ZIO_TYPE_TRIM and to Crucible's protocol-level
discard. Drivers that don't support discard return ENOTSUP and
the caller falls back to overwrite-with-zero.
- Multiqueue: per-CPU queue dispatch in virtio-blk so that a
16-vCPU guest can submit I/O on 16 queues concurrently. Adds
a scripts/run.py helper that wires up QEMU's num-queues option,
a per-queue locking-race fix (the per-queue completion-notifier
lock was being released too early under preemption), removed
next_queue_idx (became unused after dispatch was hashed by CPU),
and a sys/dev device_delete_child return-type fix that surfaced
while threading the multi-queue tear-down path.
Verified: tst-vblk-multiqueue, tst-zfs-trim, tst-zfs-direct-io.
Add an io_uring(7) implementation backed by OSv's existing async-I/O plumbing. Supported opcodes: READ, WRITE, FSYNC, NOP, OPENAT, CLOSE, READV, WRITEV, POLL_ADD, POLL_REMOVE, TIMEOUT, ACCEPT, CONNECT, RECV, SEND. Submission and completion queues are mmap'd into user memory; the SQE/CQE layout matches Linux 5.15 ABI. Surface: - io_uring_setup(), io_uring_enter(), io_uring_register() syscalls. - SQ poll thread (IORING_SETUP_SQPOLL) for kernel-side submission. - Linked SQEs (IOSQE_IO_LINK) and drain barriers (IOSQE_IO_DRAIN). Tests: tst-io_uring.cc, 14 sub-tests covering each opcode, link semantics, drain ordering, and SQ-poll mode. Registered in modules/tests/Makefile and zfs-tools/usr.manifest. Note: OSv's io_uring is not used by OpenJDK 21 virtual threads (JDK 21 still goes through epoll); see the cover letter for details.
tcp_net_channel_packet() assumed it ran with SOCK_LOCK held and the inpcb still attached to its socket. Under connection churn a queued packet can be drained after in_pcbdetach() cleared inp_socket, or from the IRQ classifier path that does not hold SOCK_LOCK, and a packet can arrive after the tcpcb has moved to CLOSED/TIME_WAIT -- which tripped tcp_do_segment()'s state KASSERT and panicked the kernel. Re-resolve the socket inside the callback and drop the packet if the inpcb has detached; acquire SOCK_LOCK if we do not already own it (OSv's recursive mutex makes re-locking a cheap depth bump); and skip segments for connections at or below LISTEN or in TIME_WAIT, matching what the slow path (tcp_input) does before reaching tcp_do_segment.
wakeup_one() tested the equal_range lower bound against _evlist.end() to decide whether a waiter existed for the channel. When the channel is absent, equal_range returns an empty range whose first iterator points at the next-larger key rather than end(), so the old guard woke an unrelated waiter and erased its node while leaving the intended sleeper stranded. Test first != second so an absent channel is a no-op.
When an ELF read runs past the end of the file, report the pathname, actual file size, and the required length instead of a bare message, so a truncated or mis-pathed binary is diagnosable from the failure.
Replace OSv's bundled FreeBSD-derived ZFS port (zfs-on-osv, c. 2014,
last upstream sync ~2017) with a vendored OpenZFS 2.4.3. The port
is a fresh OSv platform layer over the unmodified upstream OpenZFS
kernel module, plus userspace bindings for libzfs, libzfs_core,
libzutil, libshare, libuutil, libtpool and the zpool/zfs CLIs.
Build:
- external/openzfs/ submodule on the osv-2.4.3 branch of our fork
(carries 18 OSv platform commits on top of zfs-2.4.3 upstream).
- bsd/sys/cddl/openzfs_sources.mk drives the kernel-side compile,
pulling icp-asm and the modes/ directory in. ASFLAGS uses '='
(not '+=') so the older bsd/cddl asm_linkage.h shadow does not
win the include race.
- openzfs-osv source group consolidates OSv-specific helpers
(zio_crypt stubs, taskq glue, opt_zfs_auto_upgrade, freemem,
random_get_pseudo_bytes, the CDDL/BSD compat shim) into
libsolaris.so. The shim is also linked into libsolaris.so so
user-space tools resolve cleanly through the dynamic linker.
VFS / boot integration:
- vop_cache populates OSv's read_cache so user-space mmap/read
reuse the same pages.
- va_fsid in zfs_vop_getattr is taken from vp->v_mount->m_fsid
(was uninitialized).
- zfs_freesp + zfs_extend / zfs_free_range / zfs_trunc helpers
added so OSv ftruncate works on ZFS files.
- zfs_acl_ids_create stores file-type bits (S_IFLNK et al.) in
z_mode, fixing EBADF on every ZFS symlink.
- ARC shrinker, kthread stack accounting, osv_free_pages, and
memory-pressure callbacks reworked to match OSv's scheduler.
Hardware features:
- ZIO_TYPE_TRIM is now wired through vdev_disk to virtio-blk's
BIO_DISCARD (added in the block-layer commit earlier in this
series). TRIM is enabled by default on pools where the vdev
advertises support; pools on a discard-capable backing device
can reclaim free space. TRIM bio errors are correctly mapped
to ENOTSUP (not EIO) so callers can fall back gracefully.
- AES-256-GCM encryption is the real OpenZFS implementation, not
a stub: zio_crypt_impl.c is fully linked. libzfs_crypto_os.c
only stubs the userspace key-loading-from-passphrase path
(ENOTSUP); raw and hex-key formats work, exercised by
tests/tst-zfs-encryption.so (22/22 PASS).
OpenZFS 2.4.3 upgrade:
- Rebase our 18 OSv platform patches onto 2.4.3 (zero conflicts).
- Add zfs_mount_setattr() stub for selective namespace remount.
- OSv-specific zfs_uioskip iov-advance fix carried in the same
submodule pointer (without it, multi-record writes corrupt
record boundaries; reproducer in tests/tst-zfs-multirec.so).
- libzfs/libzutil device-path handling skips the ".1" partition
suffix for "crucible*" volumes (replicated raw extents have no
MBR partition table).
Cleanup:
- Remove ~60k LOC of orphaned bsd/cddl OpenSolaris userspace
sources that the new build path no longer touches.
- Add external/openzfs to .gitmodules.
All ZFS tests pass: tst-zfs-direct-io (9/9), tst-zfs-trim (5/5),
tst-zfs-encryption (22/22), tst-shm-consistency (30/30), tst-huge,
tst-io_uring, tst-zfs-multirec, tst-zfs-recordsize bench, and
tst-zfs-db-sim all C1/C3/C4/C5 configurations.
Add the ZFS-focused tests that gate this branch's correctness story:
Validation:
- tst-zfs-direct-io.so: O_DIRECT + buffered round-trips (9 cases)
- tst-zfs-trim.so: lzc_trim, partial-discard, busy pool (5 cases)
- tst-zfs-encryption.so: AES-256-GCM key load/unload, dataset
create/destroy under encryption, key-refcount on unmount
(21 cases — covers the wk_refcnt EBUSY fix in libzfs_dataset.c)
- tst-zfs-crucible-stress.so: concurrent reads/writes against a
Crucible-backed pool (excluded from the generic zfs-test image
manifest since it needs a real Crucible cluster).
Benchmarks:
- tst-zfs-recordsize.so: throughput vs. recordsize at fixed
workload size, demonstrates the ARC cache-hit-rate dependence.
- tst-fs-bench.so: cross-filesystem benchmark (ZFS vs ramfs vs
rofs, with --dir to pick the mountpoint).
- tst-zfs-db-sim.so: synthetic Postgres-WAL workload, runs all
four configurations (C1/C3/C4/C5). Requires a 4 GiB pool and
a 30 s inter-config sleep so the previous run's ARC has drained.
Ergonomics:
- tst-zfs-* helpers replace shell-out-to-zpool with libzfs API
calls so the tests run inside the OSv unikernel (the userspace
zpool binary has a separate stale-pointer bug in its argv
handling). Builds also become idempotent: re-running a test
no longer leaves stale datasets behind.
- update-openzfs.sh removed (was the patch-workflow utility, now
obsolete since we track via submodule).
- zfs-test image definition (modules/tests-zfs/) for the ZFS-only
test set.
All registered in modules/tests/Makefile and zfs-tools/usr.manifest.
The pool sizing in tst-zfs-trim and tst-zfs-db-sim is documented in
the test source.
Add an OSv-native Crucible upstairs client: a network block device
driver that speaks the Oxide Computer Crucible V13 protocol to
three independent downstairs replicas, providing crash-consistent
block storage with 2-of-3 read quorum and 2-of-3 write quorum
(3-of-3 for snapshots).
Architecture:
- drivers/crucible-client.{cc,hh}: UpsairsClient orchestrates
handshake (HereIAm / YesItsMe / PromoteToActive / RegionInfo /
ExtentVersionsPlease), per-job request lifecycle, hash-verified
reads, and concurrent writes.
- drivers/crucible-connection.{cc,hh}: per-downstairs TCP socket
with separate send/recv mutexes; a send_exact_with_data() helper
that takes the send mutex across both header and data halves of
a Write or ReadResponse frame so concurrent ZFS TXG-sync writes
do not interleave on the wire.
- drivers/crucible-bincode.hh: minimal bincode 1.x encoder/decoder
matching upstream Rust serde wire layout.
- drivers/crucible-messages.hh, crucible-types.hh: protocol
structs aligned byte-for-byte with the upstream Crucible Rust
enum and bincode encoding.
- drivers/crucible-blk.{cc,hh}: OSv block-device shim, exposes
/dev/crucible0 to the VFS. Validates I/O alignment and length,
maps requests to UpsairsClient::{read,write,flush}_sync(), and
threads through BIO_DISCARD as ENOTSUP (V13 has no Discard).
- drivers/crucible-hash.{cc,hh}: xxh3-128 helper for ReadBlockType
integrity verification.
Integration:
- Multi-volume support: multiple --crucible= command lines bind
/dev/crucible0, /dev/crucible1, etc.
- Opt-in via conf/profiles/x64/crucible.mk so the driver isn't
pulled into the default kernel image.
- Boot-time CLI flags wired into loader.cc.
- Removed the early-prototype Rust scaffolding (osv-sys, crucible-
osv crates, build.rs); the driver is now pure C++.
V13 protocol fixes (relative to early prototype):
- HereIAm carries Vec<u32> alternate_versions, not bool.
- YesItsMe layout is {version, repair_addr}.
- Result tag is u32, not u8.
- Write / ReadResponse / Flush layouts and Option<u32> /
Option<String> field encodings match upstream bincode.
- RegionDefinition struct order matches upstream Rust.
Production logging: removed per-I/O kprintfs that would spam under
load; only error paths and a single connection-state line per
downstairs remain.
Tests:
- tst-crucible-blk: direct /dev/crucible0 round-trip at multiple
offsets and sizes (BLKGETSIZE64, pwrite, fsync, pread).
- tst-crucible-zfs: end-to-end zpool_create / dataset / mount /
write / fsync / read / verify on a 3-replica Crucible volume
over the libzfs API (avoids the userspace zpool binary).
Driver runs against a real 27-downstairs Crucible cluster (9
volumes, ports 4001-4083) and a single 3-replica volume; passes all
tests modulo a separately-tracked OSv kernel TCP issue with
sustained 1 MiB+ transfers under the QEMU usermode network stack.
Three Forgejo Actions workflows mirroring the existing GitHub
Actions setup:
- build.yml: every push, run a debug + release build on x86_64
with fs=zfs.
- test.yml: every PR, build the test image and run the OSv test
suite (tst-pthread, tst-shm, tst-shm-consistency, tst-io_uring,
tst-zfs-direct-io, tst-zfs-trim, tst-zfs-encryption, tst-zfs-
multirec). The matrix splits across QEMU + KVM-on-Forgejo runner.
- release.yml: on tag push, build a fresh ZFS image and attach it
as a release asset.
Mirrors only; neither workflow runs against cloudius-systems/osv.
Move the apps submodule URL from cloudius-systems/osv-apps to
codeberg.org/gregburd/osv-apps and bump it to a fork commit that
adds two new demos used by the test suite:
- apps/zfs-demo/: minimal libzfs example (zpool_create + zfs_create
+ write + read).
- apps/crucible-basic-test/: tiny client that opens /dev/crucible0
via BLKGETSIZE64 + pwrite/pread to confirm the upstairs-downstairs
pipeline is alive. Build artefact is .gitignored (was previously
committed by accident; the fork commit removes it).
The fork is a strict superset: every cloudius-systems/osv-apps
commit is preserved, the new demos sit on top.
OpenJDK from the host worked on Debian/Ubuntu but failed on NixOS
because java/jshell binaries embed an absolute /nix/store path in
their RUNPATH. Fix:
- module.py: patchelf java/jshell binaries at install time so
RUNPATH points at /usr/lib/jvm/java/lib (the OSv layout) rather
than the build-host's nix-store path.
- common.gmk: bump C++ standard from c++11 to c++14 because Boost
1.89's interfaces require it.
- jni_helpers.cc: include <stdexcept> (was implicit before
Boost 1.89 / GCC 14 tightened the standard headers).
- pom.xml: drop the `<classifier>jar</classifier>` directive that
was removed in maven-assembly-plugin 3.7.1.
Concurrent fix: binutils 2.46 changed how it handles weak symbol
visibility for archived static libraries; adjust the Makefile rule
for libsolaris.so to keep the OSv export-symbols list authoritative.
Java OpenJDK 21 image now boots and runs the SubmissionPublisher
virtual-thread benchmark (100/100 tasks complete) on OSv ZFS.
Contributor
Author
Contributor
Author
|
Note to self / reviewers: this draft is blocked on #1423 and is currently ~39 commits behind master, which means it carries a few silent reverts of work that has since merged (the rt_sigtimedwait/osv_sigtimedwait path from #1428 and the tst-signal-fills.cc test). Those are stale-base artifacts, not intentional. When #1423 lands and this is unblocked, it will be rebased onto current master (which drops those reverts by patch-id) before it leaves draft. Not fixing now because it cannot merge until #1423 anyway and the rebase-on-unblock supersedes any fix made today. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warning
DRAFT / BLOCKED - not ready for review or merge yet.
This PR is published early for visibility into the Crucible block-driver work.
It is blocked on three things merging first, and its diff will not be clean
until they do:
This branch is stacked on top of all of the above (Crucible is the opt-in,
last item in the stack), so the diff shown here is the cumulative stack
back to master and re-includes their changes plus other already-merged work.
Once the blockers merge I will rebase this onto the new master, drop the merged
commits, and reduce the diff to just the Crucible driver for review. Please do
not review the diff as-is.
What this delivers
Adds a Crucible distributed block storage driver (
1e27a93f), so OSv canuse Oxide's Crucible as a ZFS-backing block device (alongside virtio-blk, NVMe,
and NVMe-oF). The driver is opt-in and gated off by default
(
conf_drivers_crucible?=0); it is built only withconf_drivers_profile=crucible, so it adds nothing to the default kernel.It sits last in the storage stack because it is a ZFS backing device and depends
on the OpenZFS cutover (#1423) and the block/pagecache infrastructure (#1398,
#1400).
Validation done (on our fork)
tst-crucible-blk9/9 byte-clean; ~53.6 MB/s write / ~119.7 MB/s read(multi-iov physio path fixed).
Known limitation (honest status)
tst-crucible-zfscan hang intxg_wait_syncedduringzpool_createagainsta multi-host downstairs quorum. This is Crucible-driver-specific (virtio
passes the same ashift=9/12 workload), and is being investigated. This will be
called out clearly when the PR is opened for real review; it is why the driver
ships gated-off.
I will post a clean, rebased version for review once the blockers merge.