numa: report real topology from getcpu and get_mempolicy by gburd · Pull Request #1438 · cloudius-systems/osv

gburd · 2026-07-13T11:25:03Z

What

With NUMA topology now discovered (the numa:: module from the base PR), make
the topology-query syscalls report it instead of a hardcoded single node.

How

sys_getcpu() now fills the node-out argument with numa::node_of_cpu() for
the calling CPU rather than always 0.
get_mempolicy():
- MPOL_F_NODE returns the calling CPU's node.
- MPOL_F_NODE | MPOL_F_ADDR returns the node backing the given address,
  resolved via a page-table walk (virt_to_phys_pt) and
  numa::node_of_phys().
- the allowed-nodes mask now sets a bit for every discovered node (not just
  node 0) and rejects a maxnode smaller than the node count with EINVAL.

set_mempolicy() stays a no-op: the topology is known but the physical
allocator is not yet node-aware, so a placement policy cannot be enforced. Its
comment is updated to say so; enforcement will come with the node-aware
allocator.

Adds numa::node_of_phys() to map a physical address to its node.

On a machine with no SRAT everything degrades to the previous single-node-0
behavior.

Testing

tests/tst-numa-mempolicy.cc checks getcpu's node is in range and matches
numa::node_of_cpu, get_mempolicy's MPOL_F_NODE and allowed-mask (bit count
== node count, maxnode-too-small EINVAL), and the MPOL_F_ADDR path. Verified
on OSv under KVM single-node and with a QEMU 2-node -numa config.

Note

Depends on #1418 (numa: discover NUMA topology from ACPI SRAT/SLIT). Until that
merges, this PR's diff shows both commits.

(Recreated from #1419, which GitHub auto-closed when its branch was rebased onto current master. Same change, rebased and verified on master 3aba46c.)

OSv had no notion of NUMA: the scheduler and allocator treat memory as flat, which leaves performance on the table on multi-socket bare-metal (the large large multi-socket bare-metal hosts). This is the first, discovery-only step. It adds a numa:: module that parses the ACPI SRAT (System Resource Affinity Table) and SLIT (System Locality Distance Information Table) at boot, right after acpi::init(), and exposes the topology: - numa::nr_nodes() / numa::available() - numa::node_of_cpu(cpu_id) -- resolved by correlating SRAT APIC ids with the APIC ids the MADT parse recorded on each sched::cpu - numa::distance(from, to) -- from SLIT (10 == local per ACPI convention), defaulting to 10 local / 20 remote when no SLIT is present - numa::memory_ranges() -- physical ranges tagged with their node It handles the SRAT CPU-affinity, x2APIC CPU-affinity and memory-affinity subtables, and only trusts SLIT if its locality count matches the node count SRAT reported. On a machine with no SRAT (the common single-node VM) it reports one flat node and available() == false; nothing changes behavior. This intentionally does NOT yet change allocation or scheduling; it only makes the topology available so a node-aware allocator, scheduler affinity, and mbind/get_mempolicy can build on it. Add tests/tst-numa.cc validating the invariants (>= 1 node, every CPU maps in range, distance diagonal == 10 and off-diagonal >= 10, memory ranges name valid nodes). Verified on OSv under KVM both without NUMA (reports 1 flat node) and with a QEMU 2-node -numa config (reports 2 nodes, 3 memory ranges, available).

With NUMA topology now discovered (numa:: module), make the topology-query syscalls report it instead of a hardcoded single node: - sys_getcpu() now fills the node-out argument with numa::node_of_cpu() for the calling CPU rather than always 0. - get_mempolicy(): - MPOL_F_NODE returns the calling CPU's node. - MPOL_F_NODE | MPOL_F_ADDR returns the node backing the given address, resolved via a page-table walk (virt_to_phys_pt) and numa::node_of_phys(). - the allowed-nodes mask now sets a bit for every discovered node (not just node 0), and rejects a maxnode smaller than the node count with EINVAL. set_mempolicy() stays a no-op: the topology is known but the physical allocator is not yet node-aware, so a placement policy cannot be enforced. Its comment is updated to say so; enforcement will come with the node-aware allocator. Adds numa::node_of_phys() to map a physical address to its node. On a machine with no SRAT everything degrades to the previous single-node-0 behavior. Add tests/tst-numa-mempolicy.cc checking getcpu's node is in range and matches numa::node_of_cpu, get_mempolicy's MPOL_F_NODE and allowed-mask (bit count == node count, maxnode-too-small EINVAL), and the MPOL_F_ADDR path. Verified on OSv under KVM single-node and with a QEMU 2-node -numa config. Depends on the "numa: discover NUMA topology from ACPI SRAT/SLIT" change.

Copilot

Pull request overview

This PR wires the newly introduced NUMA topology discovery (numa:: from the base PR) into Linux-compat syscalls so that getcpu()/get_mempolicy() report real node information (instead of hardcoded node 0), and adds tests to validate the behavior under both single-node and multi-node QEMU/KVM configs.

Changes:

Add core/numa.cc + include/osv/numa.hh to discover NUMA topology from ACPI SRAT/SLIT and expose node/distance/memory-range queries.
Update sys_getcpu() and get_mempolicy() to return node information derived from numa::node_of_cpu() / numa::node_of_phys().
Add NUMA topology and mempolicy tests and hook them into the tests module build.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/tst-numa.cc	New test validating basic NUMA discovery invariants (nodes, CPU mapping, distances, ranges).
tests/tst-numa-mempolicy.cc	New test validating `getcpu` node reporting and `get_mempolicy` node/mask behavior (including MPOL_F_ADDR path).
modules/tests/Makefile	Adds new NUMA tests to the tests image/build list.
Makefile	Builds the new NUMA core implementation (`core/numa.o`).
loader.cc	Initializes NUMA topology after ACPI init at boot.
linux.cc	Implements NUMA-aware behavior in `get_mempolicy()` and `sys_getcpu()`.
include/osv/numa.hh	Public NUMA discovery API.
core/numa.cc	ACPI SRAT/SLIT parsing + topology storage/query implementation.

Suppressed comments (2)

core/numa.cc:127

After switching sub to a byte pointer (needed to avoid void* arithmetic), the loop increment should advance the byte cursor directly. As written, sub = static_cast<void*>(sub) + s->Length; is also invalid C++ pointer arithmetic.

        sub = static_cast<void*>(sub) + s->Length;

core/numa.cc:94

The SRAT subtable walk only guards against Length == 0, but doesn't verify that Length stays within the SRAT table bounds. A malformed SRAT could make sub step past end and lead to out-of-bounds reads.

        if (s->Length == 0) {
            break;   // Guard against a malformed zero-length subtable.
        }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

gburd · 2026-08-03T12:51:57Z

+    void* sub = srat + 1;
+    void* end = static_cast<void*>(srat) + srat->Header.Length;
+    unsigned max_node = 0;
+
+    while (sub < end) {
+        auto s = static_cast<ACPI_SUBTABLE_HEADER*>(sub);


The void*+length arithmetic compiles under OSv's GCC (the GNU extension treats void* as char*), which is why this branch already builds and boots on KVM; but I will switch to explicit char* / static_cast to keep it standard C++ and avoid relying on the extension.

gburd · 2026-08-03T12:51:59Z

+    for (auto& r : s_mem_ranges) {
+        if (phys >= r.base && phys < r.base + r.length) {
+            return (int)r.node;
+        }
+    }


Will use a subtraction-based range check (phys - r.base < r.length) in node_of_phys() to avoid a base+length overflow on a malformed range.

gburd · 2026-08-03T12:52:00Z

+    uint64_t n = slit->LocalityCount;
+    // Only trust SLIT if it agrees with the node count we saw in SRAT.
+    if (n == 0 || n != s_nr_nodes) {
+        return;
+    }
+    s_distances.assign(slit->Entry, slit->Entry + n * n);


Will bound the SLIT parse: validate LocalityCount against header->Length before reading n*n entries, so a malformed/oversized table cannot read past the buffer.

gburd · 2026-08-03T12:52:01Z

+    if (!s_apic_to_node.empty() || !s_mem_ranges.empty()) {
+        s_available = true;
+        s_nr_nodes = max_node + 1;
+    }


Fair - proximity domain IDs can be sparse/large. I will map observed domains to a dense node index rather than using max_node+1 directly, so nr_nodes() reflects the real node count.

gburd · 2026-08-03T12:52:03Z

+        if (flags & MPOL_F_ADDR) {
+            // The node backing the given address, if we can resolve it.
+            int node = -1;
+            if (numa::available() && addr) {
+                auto phys = mmu::virt_to_phys_pt(addr);
+                node = numa::node_of_phys(phys);
+            }
+            *policy = node < 0 ? 0 : node;


Same as the get_mempolicy guard on the sibling PR: I will make the MPOL_F_NODE|MPOL_F_ADDR path return EFAULT on an unmapped user address instead of asserting in virt_to_phys_pt().

gburd · 2026-08-03T12:52:04Z

+	tst-numa.so \
+	tst-numa.so tst-numa-mempolicy.so \


Will keep a single tst-numa.so entry and add tst-numa-mempolicy.so alongside it.

gburd mentioned this pull request Jul 13, 2026

numa: report real topology from getcpu and get_mempolicy #1419

Closed

gburd force-pushed the pr/numa-mempolicy branch from 3c5b146 to 4d30f38 Compare July 15, 2026 13:23

gburd added 3 commits July 31, 2026 08:51

copyright: correct new-file attribution to the actual author

5a98418

gburd force-pushed the pr/numa-mempolicy branch from 4d30f38 to 5a98418 Compare July 31, 2026 12:51

wkozaczuk requested a review from Copilot August 3, 2026 03:58

Copilot started reviewing on behalf of wkozaczuk August 3, 2026 03:59 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

Navigation Menu

Uh oh!

Uh oh!

numa: report real topology from getcpu and get_mempolicy - #1438

numa: report real topology from getcpu and get_mempolicy#1438
gburd wants to merge 3 commits into
cloudius-systems:masterfrom
gburd:pr/numa-mempolicy

gburd commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

gburd Aug 3, 2026

Uh oh!

gburd Aug 3, 2026

Uh oh!

gburd Aug 3, 2026

Uh oh!

gburd Aug 3, 2026

Uh oh!

gburd Aug 3, 2026

Uh oh!

gburd Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

gburd commented Jul 13, 2026

What

How

Testing

Note

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

gburd Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

gburd Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

gburd Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

gburd Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

gburd Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

gburd Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants