Skip to content

process: implement thread-backed fork()/vfork()/execve()/waitpid() (opt-in, off by default) - #1455

Open
gburd wants to merge 3 commits into
cloudius-systems:masterfrom
gburd:wip/feat-fork
Open

process: implement thread-backed fork()/vfork()/execve()/waitpid() (opt-in, off by default)#1455
gburd wants to merge 3 commits into
cloudius-systems:masterfrom
gburd:wip/feat-fork

Conversation

@gburd

@gburd gburd commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

What

Adds a thread-backed fork()/vfork()/execve()/waitpid() implementation to OSv, gated entirely behind a new CONFIG_fork kconfig option that defaults to OFF. With the flag off (the default), none of this code is compiled and OSv is byte-for-byte the current single-address-space kernel — fork()/vfork() return ENOSYS exactly as today. The feature is opt-in for workloads that need fork().

Why

OSv is a single-address-space unikernel, so a literal copy-on-write fork() is not its model, and fork() has historically been a stub. But a large class of real Linux programs use fork() — most commonly fork()+exec() to spawn a child program, plus system()/popen(). This provides the useful, compatible subset without disturbing the default OSv model.

How

With CONFIG_fork enabled:

  • fork()/vfork() create a child OSv thread that resumes in fork()'s caller and returns 0 in the child / the child pid in the parent (the classic twin return). The child runs on a private copy of the parent's user stack (so parent and child have independent locals after the return) and gets its own fresh OSv per-thread TLS block (own errno, etc.). Arch halves in arch/x64/fork.cc and arch/aarch64/fork.cc.
  • execve() launches the target as a fresh OSv application (its own ELF namespace) and does not return — making fork()+exec() work.
  • waitpid()/wait4()/wait() reap a child's exit status via a pid→child registry; SIGCHLD is raised to the parent on child exit (and SIGCHLD/SIGURG/SIGWINCH now correctly default to ignore rather than powering off the VM).
  • exit()/_exit() in a fork child ends only that child, not the whole unikernel.
  • pthread_atfork prepare/parent/child handlers are now actually run around fork() (they were a no-op stub; glibc/musl register these internally, e.g. to reset the malloc arena lock in the child).
  • sys_clone() routes the non-CLONE_THREAD (fork) case here when enabled.

Validation

tst-fork (new) passes 10/10 on both x86-64 and aarch64: twin return, private-stack isolation (parent's local intact after the child mutates its own copy), fork()+exec(), vfork(), and waitpid() reaping exit codes. Built and confirmed in both configurations: CONFIG_fork off builds with the fork code fully excluded and the kernel healthy; CONFIG_fork on builds and passes tst-fork.

Scope / honest limitations (documented in documentation/fork.md)

This base does not give the child private memory: it shares the parent's heap and globals (only the stack is copied). That is fine for fork()+exec(), system(), and children that only read shared state before exec/exit. A follow-up adds per-child copy-on-write address spaces (behind the same flag) for programs that need memory-isolated multi-process fork(). Also documented as not carried by this base: deep-call-chain child unwind, fork()-as-memory-snapshot (e.g. Redis BGSAVE), and a separate pre-existing fault in execve()'s new-ELF-namespace path.

Off by default, so there is no risk to existing OSv builds; this simply makes fork() available to those who opt in.

@gburd

gburd commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Added a fix for a fork-child lifecycle leak found while validating execve(): a forked child was created as an attached thread, so it held a shared_ptr to the app's application_runtime that nothing ever released (the parent reaps the child via waitpid/the pid registry, never join()). The runtime refcount therefore never hit zero, ~application_runtime never fired, and the loader's application::join() blocked forever -> OSv hung at shutdown after a successful fork+exec. Fix: create the fork child detached and dispose() it after its exit bookkeeping so the reaper releases application_runtime and OSv shuts down cleanly. Also confirmed execve() itself works (launches the target program via application::run in a new ELF namespace) and re-enabled tst-fork's fork+exec test to exec a real payload; added tst-execve. Validated on x86-64/KVM: tst-execve 3/3, tst-fork 10/10, 5x repeat clean, clean shutdown, no non-fork regressions.

gburd added a commit to gburd/osv-1 that referenced this pull request Jul 24, 2026
…d inherited (backend connection-socket wall)

OSv has ONE GLOBAL fd table (fs/vfs/kern_descrip.cc gfdt[]) shared by a
fork parent and child.  0a87423 handled a fork CHILD closing an fd it
inherited (drop only the childs ref, keep the shared slot for the
parent).  It did NOT handle the reverse: the top-level OWNER (a
postmaster) closing a connection fd that a LIVE forked child still needs.

PostgreSQLs postmaster->backend handoff hit exactly this:
  AcceptConnection() -> gfdt[N] = accepted socket (f_count=1)
  BackendStartup() -> fork() (snapshot fholds N -> f_count=2; child inherits N)
  postmaster: closesocket(s.sock) == close(N)   // no longer needed here
The postmaster is NOT a fork child, so fork_child_close_inherited_fd()
returned false and the normal fdclose() nulled the SHARED gfdt[N] slot.
The forked backend then either saw gfdt[N]==NULL -> getsockname()=EBADF,
or the freed fd was reused by the backends own startup open() for a
regular file (DTYPE_VNODE) -> getsockname()=ENOTSOCK.  Stock PG18.4 died
with FATAL: getsockname() failed: ENOTSOCK/EBADF (pqcomm.c pq_init) and
the psql connection was closed before any query was served.

gdb (KVM+hbreak on kern_getsockname) proved it: at the failing
getsockname(fd=31) the shared slot had been closed by the postmaster and
re-used -- gfdt[31] pointed at a file with f_type=1 (DTYPE_VNODE), not a
socket, so getsock_cap()s file_type()!=DTYPE_SOCKET returned ENOTSOCK.

Fix (all #if CONF_fork; non-fork path byte-identical):
 * fork_owner_close_inherited_fd(fd, fp): when a NON-child context closes
   an fd that a live child inherits (same fd->file), fdclose() keeps the
   shared gfdt slot (only the owners reference is fdrop()d) and records
   the fd in g_owner_released_fds -- the owner has relinquished the slot.
 * The LAST inheriting child to close/teardown such an owner-released fd
   clears the slot (fork_clear_gfdt_slot_if), so the fd can be reused.
   A slot the owner still holds is never cleared by a child (restores the
   0a87423 invariant, keeping tst-fork-socket green).
 * gfdt_lock is never nested inside g_fd_lock: the clear is deferred until
   after g_fd_lock is released.

Regression test tests/tst-fork-conn-socket.cc mirrors the handoff:
reproduces on HEAD (getsockname errno=EBADF/ENOTSOCK), passes with the fix.

Result: stock PG18.4 forked backend now passes getsockname (0 getsockname
errors across the run) and advances past the socket handoff.  The first
query is still blocked by a SEPARATE POSIX-shm/DSM gap (shm_open
/PostgreSQL.<n> ENOENT) and a net-RX fault -- distinct next walls.

Relates to shipping PR cloudius-systems#1455 (fork fd/socket inheritance).

Author: Greg Burd
gburd added a commit to gburd/osv-1 that referenced this pull request Jul 31, 2026
…d inherited (backend connection-socket wall)

OSv has ONE GLOBAL fd table (fs/vfs/kern_descrip.cc gfdt[]) shared by a
fork parent and child.  0a87423 handled a fork CHILD closing an fd it
inherited (drop only the childs ref, keep the shared slot for the
parent).  It did NOT handle the reverse: the top-level OWNER (a
postmaster) closing a connection fd that a LIVE forked child still needs.

PostgreSQLs postmaster->backend handoff hit exactly this:
  AcceptConnection() -> gfdt[N] = accepted socket (f_count=1)
  BackendStartup() -> fork() (snapshot fholds N -> f_count=2; child inherits N)
  postmaster: closesocket(s.sock) == close(N)   // no longer needed here
The postmaster is NOT a fork child, so fork_child_close_inherited_fd()
returned false and the normal fdclose() nulled the SHARED gfdt[N] slot.
The forked backend then either saw gfdt[N]==NULL -> getsockname()=EBADF,
or the freed fd was reused by the backends own startup open() for a
regular file (DTYPE_VNODE) -> getsockname()=ENOTSOCK.  Stock PG18.4 died
with FATAL: getsockname() failed: ENOTSOCK/EBADF (pqcomm.c pq_init) and
the psql connection was closed before any query was served.

gdb (KVM+hbreak on kern_getsockname) proved it: at the failing
getsockname(fd=31) the shared slot had been closed by the postmaster and
re-used -- gfdt[31] pointed at a file with f_type=1 (DTYPE_VNODE), not a
socket, so getsock_cap()s file_type()!=DTYPE_SOCKET returned ENOTSOCK.

Fix (all #if CONF_fork; non-fork path byte-identical):
 * fork_owner_close_inherited_fd(fd, fp): when a NON-child context closes
   an fd that a live child inherits (same fd->file), fdclose() keeps the
   shared gfdt slot (only the owners reference is fdrop()d) and records
   the fd in g_owner_released_fds -- the owner has relinquished the slot.
 * The LAST inheriting child to close/teardown such an owner-released fd
   clears the slot (fork_clear_gfdt_slot_if), so the fd can be reused.
   A slot the owner still holds is never cleared by a child (restores the
   0a87423 invariant, keeping tst-fork-socket green).
 * gfdt_lock is never nested inside g_fd_lock: the clear is deferred until
   after g_fd_lock is released.

Regression test tests/tst-fork-conn-socket.cc mirrors the handoff:
reproduces on HEAD (getsockname errno=EBADF/ENOTSOCK), passes with the fix.

Result: stock PG18.4 forked backend now passes getsockname (0 getsockname
errors across the run) and advances past the socket handoff.  The first
query is still blocked by a SEPARATE POSIX-shm/DSM gap (shm_open
/PostgreSQL.<n> ENOENT) and a net-RX fault -- distinct next walls.

Relates to shipping PR cloudius-systems#1455 (fork fd/socket inheritance).

Author: Greg Burd
@wkozaczuk
wkozaczuk requested a review from Copilot August 3, 2026 04:07

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an opt-in, thread-backed implementation of fork()/vfork() plus supporting execve()/waitpid() plumbing for OSv, gated behind CONFIG_fork (default OFF) to preserve the existing single-address-space behavior unless explicitly enabled.

Changes:

  • Introduces a fork-child registry + wait*() implementations, and routes the non-CLONE_THREAD sys_clone() case through the new fork() path when enabled.
  • Adds arch-specific fork_thread() implementations for x86-64 and aarch64 that resume execution in the fork caller on a copied user stack.
  • Adds documentation and new tests/payloads to validate fork/exec/wait behavior.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
tests/tst-fork.cc New functional test for fork twin-return, stack-copy behavior, fork+exec, vfork, and waitpid semantics
tests/tst-execve.cc New regression test for execve launching a fresh-namespace payload and clean shutdown/reaping
tests/payload-exit7.cc Simple exec payload used by the new fork/exec tests
runtime.cc Adds fork-off stubs and makes exit() terminate only a fork child thread when applicable
modules/tests/Makefile Adds the new tests/payload to the tests module build
Makefile Conditionally links arch fork object + fork implementation when conf_fork=1, adjusts wait object selection
linux.cc Routes non-CLONE_THREAD clone calls to fork emulation (when enabled) and rejects namespace clone flags
libc/signal.cc Adjusts default disposition for SIGCHLD/SIGURG/SIGWINCH to ignore (per POSIX)
libc/pthread.cc Implements actual pthread_atfork handler storage + invocation hooks
libc/process/waitpid.cc Provides fork-backed wait()/waitpid()/wait4() when enabled; ECHILD stub otherwise
libc/process/fork.cc New fork/vfork implementation, child registry, and wait backend
libc/process/execve.cc Implements execve via osv::application::run(..., new_program=true) when enabled; stub otherwise
include/osv/fork.hh New public header for fork emulation interfaces
documentation/fork.md New documentation describing semantics/limitations and build gating
conf/kconfig/threads Adds CONFIG_fork option (currently described as including COW/address-space changes)
arch/x64/fork.cc x86-64 fork trampoline + stack-copy implementation
arch/aarch64/fork.cc aarch64 fork trampoline + stack-copy implementation
Suppressed comments (1)

tests/tst-fork.cc:105

  • These fork tests will be executed by /testrunner.so in the default test suite. When CONFIG_fork is disabled (the default), fork()/vfork()/execve()/waitpid() are stubs and this test will fail rather than skip. Consider adding an early runtime probe to detect ENOSYS and skip cleanly when fork support is not compiled in.
    printf("=== tst-fork ===\n");
    test_fork_return();
    test_fork_exec();
    test_vfork();
    test_no_children();

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread include/osv/fork.hh
Comment on lines +41 to +44
// Register the current thread as a fork() child of @parent_pid with child pid
// @child_pid. Called on the child just before it resumes at the fork() return
// site. Sets up the child's exit hook so the parent's waitpid() can reap it.
void register_child(pid_t child_pid, pid_t parent_pid);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct - the header comment is wrong about the call site. register_child() runs in the parent before the child thread starts. Will fix the comment to describe the actual ordering (parent registers, then starts the child), which is what makes the wait*() lookup race-free.

Comment thread tests/tst-fork.cc
Comment on lines +93 to +97
int status = 0;
errno = 0;
pid_t w = waitpid(-1, &status, 0);
CHECK(w == -1, "waitpid with no children returns -1");
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed - the test should assert errno == ECHILD (both here and at line 101), not just the -1 return, so a wrong error code is caught. Will add the errno checks.

Comment thread documentation/fork.md
Comment on lines +15 to +17
All of the fork() machinery - the fork()/vfork()/execve()/waitpid()
implementations, the per-child address space, and the copy-on-write changes -
is gated behind the `CONFIG_fork` kconfig option (make variable `conf_fork`),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, the doc is internally inconsistent. This PR provides thread-backed fork + stack copy + wait/exec with a SHARED heap/globals; the per-child COW address space is later fork-stack work and should not be claimed here. Will rewrite this paragraph to describe only what this PR delivers and drop the COW/address-space wording until that work lands.

Comment thread conf/kconfig/threads
Comment on lines +9 to +20
config fork
prompt "Include fork()/vfork() support (per-child address space, off by default)"
bool
default n
help
Enable OSv's thread-backed fork()/vfork()/execve()/waitpid() emulation,
including the per-child address space with copy-on-write needed to give a
forked child private memory. This changes OSv's usual single-address-space
model (address-space switches on context switch between fork children), so
it is OFF by default: with this disabled, none of the fork code is compiled
in and OSv behaves exactly as before (fork() returns ENOSYS). Enable it
only for workloads that require fork() (e.g. multi-process programs).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same issue as the doc - the Kconfig help overstates the semantics. Will reword the prompt/help to match the actual behavior (thread-backed fork, shared address space, no COW isolation) and its limitations, so enabling CONFIG_fork is not misleading.

Comment thread libc/process/fork.cc
Comment on lines +39 to +43
struct child_state {
pid_t parent_pid;
bool exited = false;
int status = 0; // encoded: (exit_code & 0xff) << 8, or signal
shared_app_t execed_app; // set if the child execve()'d a program

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

child_state::execed_app is retained deliberately: it keeps the exec-replaced application object alive for the child until waitpid() reaps it (otherwise the shared_app_t could drop to zero refs while the child is still running). It is not read again because holding the reference is its whole purpose. Will add a clarifying comment so that is explicit rather than looking like dead state.

Comment thread libc/process/execve.cc
Comment on lines +76 to +84
try {
// new_program=true => fresh ELF namespace, so the exec'd program gets
// its own globals rather than colliding with the caller's.
child = osv::application::run(path, args, true,
envp ? &env : nullptr);
} catch (const osv::launch_error &e) {
// Could not load/exec the target - Linux returns ENOENT/ENOEXEC/EACCES.
return libc_error(ENOENT);
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair - mapping every launch_error to ENOENT is inaccurate. Will map an invalid/unloadable ELF to ENOEXEC and keep ENOENT only for the actual not-found case, and fix the comment above the catch to match.

Comment thread tests/tst-execve.cc
Comment on lines +34 to +40
pid_t pid = fork();
if (pid == 0) {
char *const argv[] = { (char*)"/tests/payload-exit7.so", nullptr };
char *const envp[] = { nullptr };
execve(argv[0], argv, envp);
_exit(99); // execve returned => it failed to launch the program
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch - with CONFIG_fork off (the default) fork() is a stub and these tests would fail in the default test suite. Will make both tst-fork and tst-execve skip cleanly (treat fork() returning -1/ENOSYS as skip) so the default configuration stays green.

@nyh

nyh commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

The "Why" in the commit message explains that:

But a large class of real Linux programs use fork() — most commonly fork()+exec() to spawn a child program, plus system()/popen(). This provides the useful, compatible subset without disturbing the default OSv model.

I'm trying to understand - for what kind of applications is this really "useful"? I can understand the specific example of fork()+exec() (see also #43), but even then, won't you have problems with typical code which assumes the parent and child have different file descriptors and the child can close the parent's file descriptors and open new ones (e.g., to do a pipe) and so on?

Can you give me an example of an application where this is useful?

@wkozaczuk wkozaczuk left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While some may find adding the fork()/execve() support to the unikernel controversial, I think it can be a win if we take a pragmatic approach to implement only what makes sense, document what works and what does not, make this optional as clearly you are trying to do.

Have you tried to look at how Unikraft implements it and how much of it? I know that nanos does not support fork, but they went through the exercise of running a threaded version of postgres (see https://nanovms.com/dev/tutorials/running-postgres-as-a-unikernel and https://github.com/postgrespro/postgresql.pthreads).

Comment thread linux.cc
errno = ENOSYS;
return -1;
}
return fork();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While the fork()/execve()/fork_thread() seem correct when called via the libc interface, and the new tests verify this, I am less confident it works via the syscall interface. Have you tried running a statically linked program or a dynamic one, but using the ld.so that executes the clone syscall to call fork?

The pthread_create() goes through a different call chain than the syscall clone, which calls clone_thread(). The clone_thread() for example has this extra assembly to make sure all necessary registers are restored from the parent so that the new spawned thread resumes exactly in the same state the parent does. I think something similar is necessary for the flavor of the clone syscall to support fork. It may be easier to tweak clone_thread to support fork.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, and I dug into this - the syscall/clone-direct path is a real coverage gap that I have not solved, and I do not want to overclaim it.

What works today, and what the tests cover, is the libc entry point. fork()/vfork() in libc/process/fork.cc capture the caller's continuation with __builtin_return_address(0)/__builtin_frame_address(0) and hand it to the arch fork_thread() (arch/x64/fork.cc, arch/aarch64/fork.cc), so the child resumes exactly at the application's fork() call site on a private stack copy. tst-fork/tst-execve exercise that path and pass on both arches.

The raw syscall(SYS_clone, ...) path (a static binary, or a loader that issues clone directly) is different, exactly as you describe. sys_clone() routes the non-CLONE_THREAD case to the same fork(), but fork() then captures its caller - which is sys_clone()/the syscall trampoline - not the application's syscall instruction. So the child would resume inside the dispatch code, and the full user register/sp state is not restored the way clone_thread() does it (that extra register-restore assembly you pointed at is precisely what is missing for the fork flavor). So the direct-clone fork is currently untested and, honestly, not correct.

For this PR I have documented that scope explicitly rather than claim more than is true: I added a "raw clone-syscall fork" entry to documentation/fork.md's "What does NOT work" section stating that the supported/tested path is libc fork(), that a program hitting this should be built against OSv's musl libc so it takes the libc path, and that teaching the arch clone/fork trampoline to restore the caller registers for the fork flavor (your "tweak clone_thread to support fork" suggestion) is the planned follow-up. I think that is the right sequencing: land the libc path that is validated, and do the syscall-path register-restore as a focused follow-up with its own static-binary/syscall(SYS_clone) test, rather than bolt an untested assembly change onto this PR.

I also rebased the branch onto current master while I was here (it had gone stale against the merged splice/membarrier/sig-dfl/iovcnt work), so it is a clean feature-only diff again.

gburd added 2 commits August 3, 2026 11:29
…pt-in)

OSv is a single-address-space unikernel, so a literal copy-on-write fork() is
not the model.  This adds a thread-backed fork() emulation that covers the
useful, compatible subset of fork semantics, gated entirely behind a new
CONFIG_fork kconfig option that defaults to OFF - so a default OSv build is
unchanged (none of this code is compiled in and fork()/vfork() return ENOSYS as
before).

With CONFIG_fork enabled:
- fork()/vfork() create a child OSv thread that resumes in fork()'s caller and
  returns 0 in the child / the child pid in the parent (the classic twin
  return).  The child runs on a private copy of the parent's user stack, so
  parent and child have independent locals after the return; it gets its own
  fresh OSv per-thread TLS block (own errno etc).
  Implemented in libc/process/fork.cc + arch/{x64,aarch64}/fork.cc.
- execve() launches the target as a fresh OSv application (its own ELF
  namespace) and does not return, making fork()+exec() work.
- waitpid()/wait4()/wait() reap a child's exit status via a pid->child
  registry; SIGCHLD is raised to the parent on child exit; SIGCHLD/SIGURG/
  SIGWINCH now correctly default to ignore (not poweroff).
- exit()/_exit() in a fork child ends only that child, not the whole unikernel.
- pthread_atfork prepare/parent/child handlers are now actually run around
  fork() (they were a no-op stub) - glibc/musl register these internally.
- sys_clone() routes the non-CLONE_THREAD (fork) case here when enabled.

Validated: tst-fork passes 10/10 on both x86-64 and aarch64 (twin return,
private-stack isolation, fork+exec, vfork, waitpid reaping).

Documented limitations (documentation/fork.md): the child shares the parent's
heap/globals (no per-process memory isolation in one address space - a follow-up
adds per-child copy-on-write address spaces behind the same flag); deep-call-
chain child unwind and fork-as-memory-snapshot (Redis BGSAVE) are not carried by
this base; execve()'s new-ELF-namespace path has a separate pre-existing fault.

Copyright (C) 2026 Greg Burd
execve() (CONF_fork) launches the target as a fresh OSv application in its own
ELF namespace via application::run(new_program=true) and, matching Linux, does
not return to the caller.  A successful exec DID launch the program (the
elf::program new-namespace construction path is fine), but after the exec'd
program finished the unikernel would hang at shutdown instead of powering off:
the loader's application::join() blocked forever on the top-level app's
_terminated flag.

Root cause is the fork child thread's lifecycle, not execve or elf::program.
fork_thread() (arch/{x64,aarch64}/fork.cc) creates the child as a normal
*attached* sched::thread.  At construction the thread captures a shared_ptr to
the current application's application_runtime (sched.cc sets _app_runtime =
app->runtime()).  Nothing ever join()s the fork child -- the parent reaps it
through the fork pid registry / waitpid(), not sched::thread::join() -- so the
thread object is never destroyed and its _app_runtime shared_ptr is never
released.  With that reference outstanding the application_runtime's use count
never reaches zero, ~application_runtime never runs, the app's _terminated is
never set, and application::join() waits forever.  (This bit every fork(), and
was most visible after fork()+execve() where the child adopts the caller app's
runtime.)

Fix: create the fork child detached and dispose it in its cleanup.
- arch/{x64,aarch64}/fork.cc: mark the child attr().detached(), so on
  completion it is handed to the thread reaper (which runs its set_cleanup()),
  rather than sitting forever waiting to be joined.
- libc/process/fork.cc: the child's cleanup now also calls
  sched::thread::dispose(child) after the existing bookkeeping (record exit
  status if it fell off the end, free the copied user stack).  Disposing the
  thread releases its _app_runtime reference, letting the owning application's
  runtime drop to zero so join() completes and OSv powers off.  This mirrors
  the default detached-thread cleanup ([this]{ dispose(this); }).

Both files are compiled only under CONF_fork, so default OSv builds are
unchanged.

Tests: tst-fork test 2 now execs a real payload (/tests/payload-exit7.so, which
prints a marker and exit(7)s) instead of the previous _exit(7) sentinel that
masked whether execve actually launched anything; a return from execve() is now
treated as a failure.  Added tst-execve.so (fork+execve launches the payload and
its exit code is reaped; missing path returns -1/ENOENT) and payload-exit7.so.

Validated on x86-64 (KVM disk boot): tst-fork 10/10 and tst-execve 3/3 pass with
the real exec payload and OSv shuts down cleanly (5/5 repeat runs, no hang).

Copyright (C) 2026 Greg Burd
@gburd

gburd commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Thank you - that framing is exactly the one I was aiming for, and it is good to hear it is the right instinct. This is deliberately pragmatic and opt-in:

  • Optional. It is off by default behind the fork configure flag (conf/kconfig/threads); a build that does not ask for fork gets OSv's historical "thread clones only, ENOSYS for a process clone" behavior unchanged. Nothing in the default image pays for it.
  • Only what makes sense. It implements the fork patterns that are actually reachable in a single-address-space unikernel: the fork+exec and fork+work-then-_exit flows, vfork() (which OSv's shared-address-space model serves more faithfully than fork's copy semantics), waitpid(), and SIGCHLD. It does not pretend to give process isolation.
  • Document what works and what does not. documentation/fork.md lists the real limits honestly: no memory isolation (child shares heap/globals, only the stack is copied), so fork()-as-a-memory-snapshot (Redis BGSAVE, a forking GC) is unsupported; the glibc-ARCH_SET_FS TLS-sharing case and its musl-build workaround; stack-internal pointers; namespace-unshare clone flags return ENOSYS; and now the raw-clone-syscall path (per your other comment).

On the references: I will look at how Unikraft implements fork and how far they take it before I do the syscall-path follow-up - if they have already solved the register-restore for a direct clone, that is the cleanest thing to mirror. And thank you for the nanos/threaded-postgres pointers (the nanovms writeup and postgrespro/postgresql.pthreads); the threaded-postgres route is a good sanity check on which fork behaviors a real workload actually depends on versus which can stay documented-unsupported.

The libc fork()/vfork() path is supported and tested (tst-fork, tst-execve).
A fork issued as a raw clone(2) syscall (a statically linked program, or a
dynamic loader calling the clone syscall directly) resumes the child inside
the syscall trampoline rather than at the application's own syscall
instruction, because fork()'s continuation is captured at the libc entry
point and the arch clone/fork trampoline does not restore the full caller
register/sp state for the fork flavor the way the thread clone path does.
Document that path as untested and unsupported for now, with the musl-libc
build as the supported workaround, and note the arch-trampoline follow-up.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants