Skip to content

libaio: implement the Linux async I/O interface - #1433

Open
gburd wants to merge 1 commit into
cloudius-systems:masterfrom
gburd:pr/libaio
Open

libaio: implement the Linux async I/O interface#1433
gburd wants to merge 1 commit into
cloudius-systems:masterfrom
gburd:pr/libaio

Conversation

@gburd

@gburd gburd commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

What

Implements the Linux async I/O interface (libaio), which was a latent crash
hazard. io_setup() returned success, but io_submit(), io_getevents(),
io_destroy() and io_cancel() were UNIMPL() stubs that called abort(). A
program that probed libaio, saw io_setup() succeed, and then submitted any I/O
would take down the whole unikernel.

How

OSv has no user/kernel boundary, so "async" I/O is done the same way
core/io_uring.cc does it: each submitted iocb runs on a short-lived detached
worker thread that performs the blocking read/write/fsync through the VFS
(sys_read/sys_write/sys_fsync, the same primitives io_uring uses), then
queues an io_event and wakes any thread parked in io_getevents().

  • io_setup(): allocate a real io_context capped at nr_events in-flight ops.
  • io_submit(): dispatch each iocb to a detached worker; honor the in-flight cap
    (partial count if some were accepted, otherwise EAGAIN, matching Linux).
    Supports PREAD, PWRITE, PREADV, PWRITEV, FSYNC, FDSYNC, NOOP;
    unknown opcodes complete with res == -EINVAL rather than aborting.
  • io_getevents(): block until min_nr events are available or the optional
    timeout elapses, then copy out up to nr events. A bad fd surfaces as
    res == -EBADF in the event, not a crash.
  • io_destroy(): mark the context draining and wait for all in-flight ops to
    retire (detached workers self-reap) before freeing, so no worker can touch a
    freed context.
  • io_cancel(): return EINVAL. Ops run to completion on a worker with no safe
    mid-flight cancellation point; EINVAL is a valid Linux response.

Best-effort eventfd notification is delivered when an iocb sets
IOCB_FLAG_RESFD.

include/api/libaio.h is filled in with the real Linux aio ABI (struct iocb,
struct io_event, IO_CMD_* opcodes, IOCB_FLAG_*), which was previously only
forward-declared.

Testing

tests/tst-libaio.cc covers the write/read round-trip (data and aio_data
echo), an 8-op batch, bad-fd (-EBADF, no abort), and the io_getevents
timeout path. It passes on OSv under KVM on both single- and multi-vCPU
configurations.

Note

This is a self-contained worker-thread-per-iocb design, which is simple and
correct for typical libaio concurrency. If a caller submits very large batches
and thread-creation cost shows up, the worker loop can be swapped for a bounded
pool like the one in core/io_uring.cc; that is left as a follow-up.

(Recreated from #1413, which GitHub auto-closed when its branch was rebased onto current master. Same change, rebased and verified on master 3aba46c.)

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements the Linux libaio ABI/API in OSv to eliminate abort-based stubs and provide functional async I/O via detached worker threads, plus adds a basic regression test and wires it into the test build.

Changes:

  • Implement io_setup/io_submit/io_getevents/io_destroy/io_cancel in core/libaio.cc with worker-thread execution and completion queuing.
  • Populate include/api/libaio.h with the Linux aio ABI structures/opcodes/flags needed for compatibility.
  • Add and register tests/tst-libaio.cc in the tests module build.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 8 comments.

File Description
tests/tst-libaio.cc Adds a regression test covering basic submit/getevents behavior and key safety paths.
modules/tests/Makefile Registers the new libaio test module in the build.
include/api/libaio.h Defines the libaio ABI types/opcodes/flags for consumers.
core/libaio.cc Implements the libaio syscall-like entry points and completion machinery.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread core/libaio.cc
Comment thread core/libaio.cc Outdated
Comment thread core/libaio.cc
Comment thread core/libaio.cc
Comment thread core/libaio.cc
Comment thread tests/tst-libaio.cc
Comment thread core/libaio.cc
Comment thread core/libaio.cc
Replace the abort-based libaio stubs with a working implementation of the
Linux-specific asynchronous I/O API/ABI (io_setup/io_submit/io_getevents/
io_destroy/io_cancel), plus a regression test.

OSv has no user/kernel boundary, so async I/O is achieved the way
io_uring does it: each submitted iocb runs on a short-lived detached
worker thread that performs the blocking read/write/fsync via the VFS,
records an io_event, and wakes any thread parked in io_getevents(). A
context caps in-flight ops at the nr_events requested at io_setup().

Supported opcodes: PREAD/PWRITE, PREADV/PWRITEV, FSYNC/FDSYNC, NOOP, and
IOCB_FLAG_RESFD eventfd notification. io_cancel returns EINVAL, since a
blocking VFS call has no safe mid-flight cancellation point.

Robustness at the trust boundary:
 - io_setup rejects nr_events <= 0 (EINVAL) and a null ctxp (EFAULT); a
   zero cap would otherwise mean "unbounded" and allow unbounded worker
   creation.
 - io_submit rejects a null ios array and null iocb entries (EFAULT), and
   does not count a failed thread allocation as inflight (which would make
   io_destroy hang).
 - io_getevents rejects a null events array (EFAULT) and an invalid
   timespec (EINVAL).
 - io_destroy now waits for both in-flight workers and any threads parked
   in io_getevents to leave before freeing the context, closing a
   use-after-free of ctx->mtx/cv when destroy races a concurrent
   io_getevents.

Signed-off-by: Greg Burd <greg@burd.me>
@gburd

gburd commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Thanks (and thanks to Copilot). Addressed in the current tip (41d79a2, rebased onto master and squashed into one commit):

  • Missing <unistd.h> for ::write: added.
  • io_setup allowing nr_events == 0 (uncapped) and null ctxp: now rejects nr_events <= 0 with EINVAL (a zero cap would mean "unbounded" and allow unbounded worker-thread creation), and a null ctxp with EFAULT.
  • io_submit not validating ios / ios[i]: a null ios array returns EFAULT; a null iocb entry returns EFAULT if nothing was accepted, otherwise the partial submitted count (Linux-style).
  • sched::thread::make() returning nullptr: a failed thread allocation is no longer counted as inflight (which would make io_destroy hang forever); it returns the partial count or EAGAIN.
  • io_getevents null events / invalid timespec: null events returns EFAULT; a negative or tv_nsec >= 1e9 timespec returns EINVAL.
  • io_destroy use-after-free vs a concurrent io_getevents: io_getevents now tracks itself in a waiters counter, and io_destroy sets destroying, wakes all, and waits for both inflight == 0 and waiters == 0 before freeing the context, so no worker or parked waiter still references ctx->mtx/cv at free time.

On the test-coverage note (RESFD notification and a non-read/write op): agreed that is worth adding; I will extend tst-libaio once I can run the image. Needs a build to validate; queued.

@gburd

gburd commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Build-validated on x86_64: image=tests builds (EXIT 0) and tst-libaio passes. The io_destroy-vs-concurrent-io_getevents waiters handshake was exercised specifically (no use-after-free, no hang, clean exit), and the new EINVAL/EFAULT input-validation cases pass without breaking the existing ones.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants