libaio: implement the Linux async I/O interface - #1433
Conversation
There was a problem hiding this comment.
Pull request overview
This PR implements the Linux libaio ABI/API in OSv to eliminate abort-based stubs and provide functional async I/O via detached worker threads, plus adds a basic regression test and wires it into the test build.
Changes:
- Implement
io_setup/io_submit/io_getevents/io_destroy/io_cancelincore/libaio.ccwith worker-thread execution and completion queuing. - Populate
include/api/libaio.hwith the Linux aio ABI structures/opcodes/flags needed for compatibility. - Add and register
tests/tst-libaio.ccin the tests module build.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 8 comments.
| File | Description |
|---|---|
| tests/tst-libaio.cc | Adds a regression test covering basic submit/getevents behavior and key safety paths. |
| modules/tests/Makefile | Registers the new libaio test module in the build. |
| include/api/libaio.h | Defines the libaio ABI types/opcodes/flags for consumers. |
| core/libaio.cc | Implements the libaio syscall-like entry points and completion machinery. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Replace the abort-based libaio stubs with a working implementation of the Linux-specific asynchronous I/O API/ABI (io_setup/io_submit/io_getevents/ io_destroy/io_cancel), plus a regression test. OSv has no user/kernel boundary, so async I/O is achieved the way io_uring does it: each submitted iocb runs on a short-lived detached worker thread that performs the blocking read/write/fsync via the VFS, records an io_event, and wakes any thread parked in io_getevents(). A context caps in-flight ops at the nr_events requested at io_setup(). Supported opcodes: PREAD/PWRITE, PREADV/PWRITEV, FSYNC/FDSYNC, NOOP, and IOCB_FLAG_RESFD eventfd notification. io_cancel returns EINVAL, since a blocking VFS call has no safe mid-flight cancellation point. Robustness at the trust boundary: - io_setup rejects nr_events <= 0 (EINVAL) and a null ctxp (EFAULT); a zero cap would otherwise mean "unbounded" and allow unbounded worker creation. - io_submit rejects a null ios array and null iocb entries (EFAULT), and does not count a failed thread allocation as inflight (which would make io_destroy hang). - io_getevents rejects a null events array (EFAULT) and an invalid timespec (EINVAL). - io_destroy now waits for both in-flight workers and any threads parked in io_getevents to leave before freeing the context, closing a use-after-free of ctx->mtx/cv when destroy races a concurrent io_getevents. Signed-off-by: Greg Burd <greg@burd.me>
|
Thanks (and thanks to Copilot). Addressed in the current tip (41d79a2, rebased onto master and squashed into one commit):
On the test-coverage note (RESFD notification and a non-read/write op): agreed that is worth adding; I will extend tst-libaio once I can run the image. Needs a build to validate; queued. |
|
Build-validated on x86_64: |
What
Implements the Linux async I/O interface (libaio), which was a latent crash
hazard.
io_setup()returned success, butio_submit(),io_getevents(),io_destroy()andio_cancel()wereUNIMPL()stubs that calledabort(). Aprogram that probed libaio, saw
io_setup()succeed, and then submitted any I/Owould take down the whole unikernel.
How
OSv has no user/kernel boundary, so "async" I/O is done the same way
core/io_uring.ccdoes it: each submitted iocb runs on a short-lived detachedworker thread that performs the blocking read/write/fsync through the VFS
(
sys_read/sys_write/sys_fsync, the same primitives io_uring uses), thenqueues an
io_eventand wakes any thread parked inio_getevents().io_setup(): allocate a realio_contextcapped atnr_eventsin-flight ops.io_submit(): dispatch each iocb to a detached worker; honor the in-flight cap(partial count if some were accepted, otherwise
EAGAIN, matching Linux).Supports
PREAD,PWRITE,PREADV,PWRITEV,FSYNC,FDSYNC,NOOP;unknown opcodes complete with
res == -EINVALrather than aborting.io_getevents(): block untilmin_nrevents are available or the optionaltimeout elapses, then copy out up to
nrevents. A bad fd surfaces asres == -EBADFin the event, not a crash.io_destroy(): mark the context draining and wait for all in-flight ops toretire (detached workers self-reap) before freeing, so no worker can touch a
freed context.
io_cancel(): returnEINVAL. Ops run to completion on a worker with no safemid-flight cancellation point;
EINVALis a valid Linux response.Best-effort eventfd notification is delivered when an iocb sets
IOCB_FLAG_RESFD.include/api/libaio.his filled in with the real Linux aio ABI (struct iocb,struct io_event,IO_CMD_*opcodes,IOCB_FLAG_*), which was previously onlyforward-declared.
Testing
tests/tst-libaio.cccovers the write/read round-trip (data andaio_dataecho), an 8-op batch, bad-fd (
-EBADF, no abort), and theio_geteventstimeout path. It passes on OSv under KVM on both single- and multi-vCPU
configurations.
Note
This is a self-contained worker-thread-per-iocb design, which is simple and
correct for typical libaio concurrency. If a caller submits very large batches
and thread-creation cost shows up, the worker loop can be swapped for a bounded
pool like the one in
core/io_uring.cc; that is left as a follow-up.(Recreated from #1413, which GitHub auto-closed when its branch was rebased onto current master. Same change, rebased and verified on master 3aba46c.)