libext: real fsync + page-cache bridge for ext2/3/4 - #1431
Conversation
4ddfdc8 to
3efa67b
Compare
c51b724 to
6c7cd9a
Compare
wkozaczuk
left a comment
There was a problem hiding this comment.
I have a concern about potential memory leak
There was a problem hiding this comment.
Pull request overview
This PR improves OSv’s libext (lwext4-based) filesystem integration to support Postgres-over-ext goals by making fsync(2) durable, bridging ext-backed mmap() reads into the shared page cache via vop_cache, and fixing ext inode deletion metadata so Linux e2fsck doesn’t complain after deletes.
Changes:
- Implement
ext_fsync()to flush lwext4’s device block cache sofsync()/fdatasync()are no longer no-ops. - Add a
vop_cacheimplementation (ext_map_cached_page()) that reads one page into an allocated page and maps it into OSv’s read page cache. - Set a non-zero inode
deletion_timebefore freeing inodes to satisfy Linuxe2fsck, and add a new ext4 read/write regression test.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| tests/tst-ext4-rw.cc | Adds a regression test for ext vop_cache mmap-read caching and fsync durability. |
| modules/tests/Makefile | Includes the new tst-ext4-rw.so in the built test set. |
| modules/libext/ext_vnops.cc | Implements ext_fsync, adds vop_cache page-cache warming, and fixes inode deletion time before freeing. |
| .gitignore | Ignores local ext_images/ used for ext4 disk images. |
Comments suppressed due to low confidence (2)
modules/libext/ext_vnops.cc:1526
- ext_map_cached_page() always allocates a fresh page and unconditionally passes it to pagecache::map_read_cached_page(). If another thread already inserted the same key, map_read_cached_page() returns false and the newly allocated page remains owned by this caller; currently it is leaked.
pagecache::map_read_cached_page((pagecache::hashkey *)uio->uio_iov->iov_base,
page);
uio->uio_resid = 0;
tests/tst-ext4-rw.cc:61
- If /data is not mounted read-write (or doesn't exist), creating written.dat will assert-fail. After adding SKIP handling above, it would be more consistent to SKIP here too so the test doesn't fail due to environment rather than a regression in libext.
fd = open(out.c_str(), O_CREAT | O_TRUNC | O_RDWR, 0644);
assert(fd >= 0);
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
6c7cd9a to
f935171
Compare
|
Good catch, you're exactly right -- it was a page leak, and I traced why the return went unchecked: this file forward-declares Fixed (folded into the page-cache-bridge commit):
if (!pagecache::map_read_cached_page(
(pagecache::hashkey *)uio->uio_iov->iov_base, page)) {
// Another thread cached the same key first; map_read_cached_page did
// not take ownership, so free our now-redundant copy to avoid a leak
// (mirrors the ZFS caller in core/pagecache.cc).
memory::free_page(page);
}Also removed the stray |
Two gaps in the ext (lwext4) filesystem module for the Postgres-over-ext goal. 1. fsync durability. ext mounts with lwext4 block-cache write-back enabled, so a write only reaches the disk when the block cache is flushed -- but vop_fsync was vop_nullop, so fsync(2)/fdatasync(2) on an ext file persisted nothing. Implement ext_fsync() to flush the device's block cache (like ext_sync does at unmount), making written data durable. The cache is shared per device, so this persists the file along with any other dirty buffers, which is correct if slightly more than the theoretical per-inode minimum. 2. Page-cache bridge (vop_cache). The vop_cache slot was null, so mmap faults on ext files went through the block layer on every fault, unlike ROFS and ZFS which populate the shared page cache. Add ext_map_cached_page(): on a VOP_CACHE call it reads one page-aligned page of file data into a freshly allocated page and hands it to pagecache::map_read_cached_page(), warming the read cache so subsequent faults and readahead are served from it. This is an allocate-and-copy bridge (lwext4's block-cache buffers are not page-aligned/shareable the way ROFS's read-around cache is); a zero-copy borrow-and-pin bridge like the ZFS ARC one can follow if it shows up hot. Because the module is built with -fno-rtti and <osv/pagecache.hh> pulls in <osv/trace.hh> (which uses typeid), the two page-cache symbols we need are declared minimally instead of including the heavy header (the uio carries the hashkey opaquely, so its layout is never needed). Add tests/tst-ext4-rw.cc: mmap a pre-populated ext4 file and verify the pattern survives the vop_cache bridge (first fault + cached re-read), then write a file, fsync it, and read it back (plus fdatasync and a fresh re-open). Verified on OSv under KVM with an ext4 second disk (created with mkfs.ext4 -b 4096 -O ^64bit,^metadata_csum, which lwext4 supports). Known pre-existing limitation (not introduced here, out of scope for this PR): libext's inode-delete path does not set the inode dtime, so Linux e2fsck flags "deleted inode has zero dtime" on a disk after OSv deletes a file. A fresh disk that OSv only reads/writes+fsyncs (no delete) fscks clean. Tracked as a follow-up in the ext write-path correctness work.
Follow-up to the fsync/page-cache work: libext's inode-delete path freed the inode from the bitmap but never set its on-disk deletion time (dtime), so after OSv created and deleted a file, Linux e2fsck flagged "Deleted inode NN has zero dtime" and reported the filesystem as still having errors. lwext4's own delete path marks the inode with ext4_inode_set_del_time(inode, -1L) before ext4_fs_free_inode(), but that symbol is not exported from liblwext4.so. Add a small ext_mark_inode_deleted() helper that sets inode->deletion_time = 0xffffffff directly (byte-order invariant, so no to_le32() needed) and call it before each ext4_fs_free_inode() in the module (unlink, rmdir, delete-on-last-close, delete-outstanding-on-unmount, and the dir_link allocation-rollback path). Verified: after OSv boots an ext4 second disk, mmaps/reads a file, writes and fsyncs another, then deletes it, `e2fsck -n -f` on the disk now exits 0 (clean), where before it reported the zero-dtime error. tst-ext4-rw still passes.
f935171 to
a17fcf7
Compare
|
Follow-up on the round of comments from the last push, now in the current tip (rebased onto master):
Needs a build to confirm the test compiles and skips cleanly without the ext4 disk, and passes with it; queued. |
|
Build-validated on x86_64: |
wkozaczuk
left a comment
There was a problem hiding this comment.
I left a comment about a leak.
What
Three fixes to the ext (lwext4) filesystem module toward the Postgres-over-ext
goal.
fsync durability. ext mounts with lwext4 block-cache write-back enabled,
so a write only reaches the disk when the block cache is flushed -- but
vop_fsyncwasvop_nullop, sofsync(2)/fdatasync(2)on an ext filepersisted nothing.
ext_fsync()now flushes the device's block cache (likeext_syncdoes at unmount), making written data durable.Page-cache bridge (
vop_cache). Thevop_cacheslot was null, so mmapfaults on ext files hit the block layer on every fault, unlike ROFS and ZFS
which populate the shared page cache.
ext_map_cached_page()reads onepage-aligned page into a freshly allocated page and hands it to
pagecache::map_read_cached_page(), warming the read cache so subsequentfaults and readahead are served from it. Allocate-and-copy for now (lwext4's
block-cache buffers are not page-aligned/shareable the way ROFS's read-around
cache is); a zero-copy borrow-and-pin bridge like the ZFS ARC one can follow.
Inode deletion time. The inode-delete path freed the inode from the
bitmap but never set its on-disk
dtime, so after OSv deleted a file, Linuxe2fsckflagged "deleted inode has zero dtime".ext_mark_inode_deleted()now sets it before each
ext4_fs_free_inode()(lwext4's ownext4_inode_set_del_timeis not exported fromliblwext4.so, so the field isset directly with the byte-order-invariant
0xffffffff).Because the module builds with
-fno-rttiand<osv/pagecache.hh>pulls in<osv/trace.hh>(typeid), the two page-cache symbols are declared minimallyinstead of including the heavy header (the uio carries the hashkey opaquely).
Testing
tests/tst-ext4-rw.cc: mmap a pre-populated ext4 file and verify the patternsurvives the
vop_cachebridge (first fault + cached re-read), then write afile,
fsyncit, and read it back (plusfdatasyncand a fresh re-open).Verified on OSv under KVM with an ext4 second disk (
mkfs.ext4 -b 4096 -O ^64bit,^metadata_csum, which lwext4 supports). After the run -- whichcreates, fsyncs, and deletes a file -- Linux
e2fsck -n -fon the disk exits 0(clean), confirming both the fsync durability and the dtime fix.