fix(udp): size and align control message buffers from their payloads by dignifiedquire · Pull Request #778 · n0-computer/noq

dignifiedquire · 2026-07-29T10:07:02Z

Description

A version based on the comments form @flub to fix the alignment issues
Based on #777

Fixes #774.

Additional Fixes

1. ECN codepoints were dropped on every GRO-coalesced receive: Linux and Android, glibc as much as musl.

cmsg::LEN was 96 bytes. A coalesced receive on a v6 socket carries four control messages: IPV6_TCLASS 24 + IPV6_PKTINFO 40 + UDP_GRO 24 + SCM_TIMESTAMPNS 32 = 120. The kernel set MSG_CTRUNC and dropped the last one, which is the traffic class. RecvMeta::ecn came back None, so congestion control lost ECN feedback on exactly the traffic that matters most.

Before it, the worst case was 88 bytes and fit in 96; adding the 32-byte timestamp message pushed it over.

2. The startup assert that was supposed to prevent this checked the wrong thing.

UdpSocketState::new verified cmsg::LEN >= CMSG_SPACE(c_int) + CMSG_SPACE(in6_pktinfo) — 64 bytes, ignoring UDP_GRO and SCM_TIMESTAMPNS entirely. It passed while the real requirement was 120.

3. Nothing checked `MSG_CTRUNC`.

Only MSG_TRUNC (payload truncation) was handled, so the kernel telling us it had dropped our metadata went unread.

Breaking Changes

n/a

Notes & open questions

Have we fixed all the places? are there more platforms with special alignment?

Change checklist

Self-review.
Tests if relevant.
This PR was created by a human that thought critically about the
proposed change and wrote an as clear and concise description as
they could.
This PR isn't slop, and is carefully crafted to do have the intented effect.
cargo make passes locally.

The cmsg alignment breakage reproduces on both musl architectures, and x86_64 is what the static `iroh-relay` builds use, so cover it too. Each target runs on a runner of its own architecture, so the statically linked test binaries run natively.

github-actions · 2026-07-29T10:09:07Z

Documentation for this PR has been generated and is available at: https://n0-computer.github.io/noq/pr/778/docs/noq/

Last updated: 2026-07-29T14:56:00Z

github-actions · 2026-07-29T10:19:14Z

Performance Comparison Report

7149068106d3603518dae3a9faff6106d9af9cca - artifacts

Raw Benchmarks (localhost)

Scenario	noq	upstream	Delta	CPU (avg/max)
large-single	5315.9 Mbps	7992.4 Mbps	-33.5%	92.8% / 99.0%
medium-concurrent	5304.9 Mbps	7880.8 Mbps	-32.7%	94.1% / 99.0%
medium-single	3810.3 Mbps	4743.5 Mbps	-19.7%	93.8% / 101.0%
small-concurrent	3723.2 Mbps	5181.8 Mbps	-28.1%	99.8% / 152.0%
small-single	3402.7 Mbps	4651.3 Mbps	-26.8%	89.3% / 97.0%

Netsim Benchmarks (network simulation)

Condition	noq	upstream	Delta
ideal	2949.4 Mbps	4039.9 Mbps	-27.0%
lan	782.4 Mbps	810.4 Mbps	-3.4%
lossy	69.8 Mbps	69.8 Mbps	~0%
wan	83.8 Mbps	83.8 Mbps	~0%

Summary

noq is 28.2% slower on average

---

b32f763ea3394a4562fad58db8fbb7b1a75f5603 - artifacts

Raw Benchmarks (localhost)

Scenario	noq	upstream	Delta	CPU (avg/max)
large-single	5343.9 Mbps	7765.3 Mbps	-31.2%	96.5% / 98.0%
medium-concurrent	5405.3 Mbps	7861.4 Mbps	-31.2%	96.9% / 98.5%
medium-single	3586.1 Mbps	4699.9 Mbps	-23.7%	95.7% / 97.7%
small-concurrent	3786.0 Mbps	5328.5 Mbps	-28.9%	97.5% / 99.6%
small-single	3475.3 Mbps	4709.7 Mbps	-26.2%	96.3% / 98.7%

Netsim Benchmarks (network simulation)

Condition	noq	upstream	Delta
ideal	2995.5 Mbps	3928.4 Mbps	-23.7%
lan	782.4 Mbps	810.3 Mbps	-3.4%
lossy	69.8 Mbps	69.9 Mbps	~0%
wan	83.8 Mbps	83.8 Mbps	~0%

Summary

noq is 27.6% slower on average

---

e75075c71c40780d6e654dde893da4a890014b2a - artifacts

Raw Benchmarks (localhost)

Scenario	noq	upstream	Delta	CPU (avg/max)
large-single	5307.1 Mbps	7816.0 Mbps	-32.1%	96.7% / 149.0%
medium-concurrent	5276.1 Mbps	7831.8 Mbps	-32.6%	97.4% / 151.0%
medium-single	4041.4 Mbps	4749.3 Mbps	-14.9%	92.0% / 102.0%
small-concurrent	3601.7 Mbps	5220.3 Mbps	-31.0%	96.0% / 105.0%
small-single	3435.6 Mbps	4659.8 Mbps	-26.3%	96.9% / 155.0%

Netsim Benchmarks (network simulation)

Condition	noq	upstream	Delta
ideal	N/A	3769.0 Mbps	N/A
lan	N/A	796.4 Mbps	N/A
lossy	N/A	67.1 Mbps	N/A
wan	N/A	83.8 Mbps	N/A

Summary

noq is 28.5% slower on average

---

1da92dbc9001529a4a48c181af0c3f292ddeeccb - artifacts

Raw Benchmarks (localhost)

Scenario	noq	upstream	Delta	CPU (avg/max)
large-single	5364.1 Mbps	8072.7 Mbps	-33.6%	95.9% / 100.0%
medium-concurrent	5242.7 Mbps	7313.8 Mbps	-28.3%	96.8% / 151.0%
medium-single	3798.4 Mbps	4749.7 Mbps	-20.0%	98.4% / 151.0%
small-concurrent	3822.8 Mbps	5297.3 Mbps	-27.8%	94.8% / 103.0%
small-single	3315.6 Mbps	4744.2 Mbps	-30.1%	93.1% / 102.0%

Netsim Benchmarks (network simulation)

Condition	noq	upstream	Delta
ideal	2989.1 Mbps	4008.7 Mbps	-25.4%
lan	782.4 Mbps	810.3 Mbps	-3.4%
lossy	69.8 Mbps	69.9 Mbps	~0%
wan	83.8 Mbps	83.8 Mbps	~0%

Summary

noq is 27.5% slower on average

---

d3b7412c383e25d09c986cc86935425f8bec07fc - artifacts

Raw Benchmarks (localhost)

Scenario	noq	upstream	Delta	CPU (avg/max)
large-single	5364.8 Mbps	N/A	N/A	95.3% / 146.0%
medium-concurrent	5273.5 Mbps	N/A	N/A	95.7% / 146.0%
medium-single	3814.0 Mbps	N/A	N/A	97.9% / 149.0%
small-concurrent	3664.9 Mbps	N/A	N/A	94.0% / 101.0%
small-single	3388.0 Mbps	N/A	N/A	93.2% / 101.0%

---

b0cc69b557476eefec226cfde09549a107448580 - artifacts

Netsim Benchmarks (network simulation)

Condition	noq	upstream	Delta
ideal	2933.0 Mbps	4058.7 Mbps	-27.7%

Summary

noq is 27.7% slower on average

---

765b1207b5b14795e584e0eb7945d1357446a16d - artifacts

No results available

---

f317ebc7d6b54883c996c7f13585e31d2124303c - artifacts

Raw Benchmarks (localhost)

Scenario	noq	upstream	Delta	CPU (avg/max)
large-single	5283.2 Mbps	7829.9 Mbps	-32.5%	96.5% / 149.0%
medium-concurrent	5313.8 Mbps	7690.2 Mbps	-30.9%	97.3% / 150.0%
medium-single	3813.6 Mbps	4469.2 Mbps	-14.7%	89.1% / 96.9%
small-concurrent	3728.2 Mbps	5113.9 Mbps	-27.1%	95.7% / 104.0%
small-single	3379.1 Mbps	4492.6 Mbps	-24.8%	94.6% / 102.0%

Summary

noq is 27.3% slower on average

---

ec969f124aa0dadbea22cefe06d7116a94c01562 - artifacts

Raw Benchmarks (localhost)

Scenario	noq	upstream	Delta	CPU (avg/max)
large-single	5379.9 Mbps	7856.2 Mbps	-31.5%	93.7% / 98.5%
medium-concurrent	5332.1 Mbps	7676.6 Mbps	-30.5%	94.4% / 98.2%
medium-single	3352.3 Mbps	4749.5 Mbps	-29.4%	93.9% / 100.0%
small-concurrent	3681.0 Mbps	5164.1 Mbps	-28.7%	99.6% / 153.0%
small-single	3432.2 Mbps	4803.7 Mbps	-28.6%	98.5% / 152.0%

Netsim Benchmarks (network simulation)

Condition	noq	upstream	Delta
ideal	2885.8 Mbps	4023.0 Mbps	-28.3%
lan	782.4 Mbps	810.4 Mbps	-3.4%
lossy	69.8 Mbps	69.8 Mbps	~0%
wan	83.8 Mbps	83.8 Mbps	~0%

Summary

noq is 29.1% slower on average

`cmsg::decode` asserted `align_of::<T>() <= align_of::<C>()`, which is not what makes reading a payload sound. musl declares `cmsghdr` with `socklen_t` and `c_int` fields (alignment 4) where glibc uses `size_t` (alignment 8), so the `SCM_TIMESTAMPNS` arm tripped it on the first packet and every musl target aborted. The layouts are identical, only the declared alignment differs. What has to hold is that the buffer is aligned for the payloads it carries, so derive it from them. `Payload` is a union of every type we put into or read out of a control message, and `ControlBuf` takes its alignment from that instead of from `repr(align(8))` plus a runtime assert that the guess was right. The size was guessed the same way, and was too small. `cmsg::LEN` was 96 bytes where a v6 socket with GRO and timestamps enabled needs 120, so the kernel set `MSG_CTRUNC` and dropped `IPV6_TCLASS`: the ECN codepoint disappeared from every coalesced receive, on glibc as much as musl. `SEND_LEN` and `RECV_LEN` now count the messages each direction carries and take their size from `libc::CMSG_SPACE`, which is a `const fn`, so the platform's own macros do the arithmetic. `PAYLOAD_ALIGN` comes from the same place, the alignment `CMSG_LEN(0)` and `CMSG_SPACE` have in common, capped by the buffer's own. That spread is wider than musl: 4 bytes on Darwin and the Solarish platforms, 8 on 64 bit Linux, 16 on NetBSD and OpenBSD on sparc64. No constant in this file covers all of them. `Encoder::push` is private now, with one method per message we send, so the set `SEND_LEN` has to cover is declared in one place. The alignment check that remains is a `const` assert: a payload a platform cannot align is a compile error rather than a panic on the first packet. `gso_with_ecn` covers the truncation, which nothing else could: `gso` sends no ECN and the `ecn_*` tests send a single segment, so neither puts enough control messages on one datagram. Fixes #774.

Rewriting the module dropped them for the prelude and `std::` paths; the file used the imports throughout, so put them back.

The unix side has `payloads_are_aligned`; Windows had nothing checking that its buffer is sized and aligned for what the send path puts in it. Encoding the full set also exercises `SEND_LEN`, since `push` panics rather than overrun. Also adds the byte `IP_TOS` is decoded as to `Payload`, which listed every other payload but that one. It is smaller and less aligned than the rest, so neither constant moves.

…e payloads The payload union was doing two jobs: sizing a message slot, and aligning the buffer. Only the first is its own. What a buffer has to be aligned to is where the `CMSG_*` macros lay messages out, which is a property of the platform, so `PAYLOAD_ALIGN` now comes from `CMSG_LEN` and `CMSG_SPACE` alone and the buffer carries a plain `repr(align(16))`, asserted at compile time to be at least that. 16 is a literal because `repr(align)` takes no expression; NetBSD and OpenBSD on sparc64 are the strictest at 16, and a platform wanting more is a compile error rather than a silent misalignment. Taking the alignment from the payloads was sound but understated it: on FreeBSD and NetBSD every payload we use is 4 byte aligned, so the buffer claimed 4 where the platform lays messages out at 8. The union keeps its real job, which is telling us how large a payload slot has to be.

`repr(align(16))` covered the strictest platform but overshot everywhere else, and put a literal back in the file. Borrowing the alignment from a `usize`, what the `CMSG_*` macros round their offsets to, gives the same `PAYLOAD_ALIGN` on every target without over-aligning: the two 16 byte ABIs simply cap the guarantee at 8, which is more than any payload there needs.

They had grown into prose. Cut to the point in each, and reference the headers behind the claims the way the rest of the crate does: musl's and glibc's `cmsghdr` for the alignment they differ on, cmsg(3) for the macros, and the Winsock pktinfo structs.

Bare URLs are not links; wrap them so rustdoc renders them, including the two Winsock ones that were already there. musl and glibc are cited from their own SCMs rather than the GitHub mirrors.

It was there when the union carried the buffer's alignment, since the buffer holds headers as well as payloads, the C `union { struct cmsghdr hdr; char buf[N]; }` idiom. The alignment now comes from a `usize`, which is at least as strict as any platform's `cmsghdr`, so all the member did was feed `size_of::<Payload>()` a header that `CMSG_SPACE` already accounts for. No constant moves on any target we build: `in6_pktinfo` is the largest payload everywhere `cmsghdr` is not. A union named for payloads now holds only those.

dignifiedquire added 3 commits July 29, 2026 11:19

ci: run tests on arch64-musl

d9b7211

ci: also run the musl tests on x86_64

c64247e

The cmsg alignment breakage reproduces on both musl architectures, and x86_64 is what the static `iroh-relay` builds use, so cover it too. Each target runs on a runner of its own architecture, so the statically linked test binaries run natively.

ci: drop a duplicated comment in the musl job

1541e8c

dignifiedquire commented Jul 29, 2026

View reviewed changes

Comment thread noq-udp/src/cmsg/windows.rs Outdated

dignifiedquire commented Jul 29, 2026

View reviewed changes

Comment thread noq-udp/src/cmsg/windows.rs Outdated

dignifiedquire force-pushed the cmsg-buffers branch from 7149068 to b32f763 Compare July 29, 2026 10:27

n0bot Bot added this to iroh Jul 29, 2026

github-project-automation Bot moved this to 🚑 Needs Triage in iroh Jul 29, 2026

dignifiedquire added 2 commits July 29, 2026 13:00

style(udp): keep the mem:: and ptr:: prefixes in cmsg::windows

e75075c

Rewriting the module dropped them for the prelude and `std::` paths; the file used the imports throughout, so put them back.

dignifiedquire marked this pull request as ready for review July 29, 2026 11:36

dignifiedquire added 5 commits July 29, 2026 16:30

docs(udp): quote the reference links and cite them upstream

f317ebc

Bare URLs are not links; wrap them so rustdoc renders them, including the two Winsock ones that were already there. musl and glibc are cited from their own SCMs rather than the GitHub mirrors.

flub moved this from 🚑 Needs Triage to 🏗 In progress in iroh Jul 29, 2026

flub added this to the noq: Sprint 0 milestone Jul 29, 2026

IrosTheBeggar mentioned this pull request Jul 30, 2026

fix(p2p-sidecar): pin noq-udp 1.1.1 — musl binaries abort on first datagram IrosTheBeggar/mStream#790

Merged

Uh oh!

Conversation

dignifiedquire commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Additional Fixes

1. ECN codepoints were dropped on every GRO-coalesced receive: Linux and Android, glibc as much as musl.

2. The startup assert that was supposed to prevent this checked the wrong thing.

3. Nothing checked MSG_CTRUNC.

Breaking Changes

Notes & open questions

Change checklist

Uh oh!

github-actions Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance Comparison Report

Raw Benchmarks (localhost)

Netsim Benchmarks (network simulation)

Summary

Raw Benchmarks (localhost)

Netsim Benchmarks (network simulation)

Summary

Raw Benchmarks (localhost)

Netsim Benchmarks (network simulation)

Summary

Raw Benchmarks (localhost)

Netsim Benchmarks (network simulation)

Summary

Raw Benchmarks (localhost)

Netsim Benchmarks (network simulation)

Summary

Raw Benchmarks (localhost)

Summary

Raw Benchmarks (localhost)

Netsim Benchmarks (network simulation)

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dignifiedquire commented Jul 29, 2026 •

edited

Loading

3. Nothing checked `MSG_CTRUNC`.

github-actions Bot commented Jul 29, 2026 •

edited

Loading

github-actions Bot commented Jul 29, 2026 •

edited

Loading