GzipResponseAndCallback.commit() NPE on null HttpField array entry with Spring MVC ResponseBodyEmitter (12.0.34)

[zsait-clearstreet]

Jetty version(s)
Jetty: 12.0.34 (via Spring Boot BOM)

HTTP version
HTTP 1.1 and HTTP/2

Java version/vendor
openjdk version "21.0.10" 2026-01-20 LTS
OpenJDK Runtime Environment Corretto-21.0.10.7.1 (build 21.0.10+7-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.10.7.1 (build 21.0.10+7-LTS, mixed mode, sharing)

OS type/version
Linux (Kubernetes pod, Amazon Linux 2)

Description
When GzipHandler is enabled and a Spring MVC endpoint returns a ResponseBodyEmitter that streams a response ≥ the min-response-size threshold, GzipResponseAndCallback.commit() throws a NullPointerException on a null entry in the internal HttpField[] array. The exception is swallowed by the async infrastructure, leaving the stream permanently stalled — the opening bytes are sent to the client but no further data arrives and the connection eventually times out.

The bug does not reproduce when the response is below the threshold (GzipHandler skips compression and passes through). This makes it environment-dependent — e.g. sparse dev data never crosses the threshold while large prod result sets always do.

Stack trace:

java.lang.IllegalStateException: Failed to send [
	at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitter.send(ResponseBodyEmitter.java:217)
	at ..
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.is(String)" because "f" is null
	at org.eclipse.jetty.http.HttpFields.contains(HttpFields.java:371)
	at org.eclipse.jetty.ee10.servlet.ServletApiResponse.containsHeader(ServletApiResponse.java:117)
	at jakarta.servlet.http.HttpServletResponseWrapper.containsHeader(HttpServletResponseWrapper.java:68)
	at jakarta.servlet.http.HttpServletResponseWrapper.containsHeader(HttpServletResponseWrapper.java:68)
	at org.springframework.security.web.header.writers.ReferrerPolicyHeaderWriter.writeHeaders(ReferrerPolicyHeaderWriter.java:96)
	at org.springframework.security.web.header.HeaderWriterFilter.writeHeaders(HeaderWriterFilter.java:99)
	at org.springframework.security.web.header.HeaderWriterFilter$HeaderWriterResponse.writeHeaders(HeaderWriterFilter.java:132)
	at org.springframework.security.web.header.HeaderWriterFilter$HeaderWriterResponse.onResponseCommitted(HeaderWriterFilter.java:124)
	at org.springframework.security.web.util.OnCommittedResponseWrapper.doOnResponseCommitted(OnCommittedResponseWrapper.java:279)
	at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.flush(OnCommittedResponseWrapper.java:551)
	at org.springframework.web.context.request.async.StandardServletAsyncWebRequest$LifecycleServletOutputStream.flush(StandardServletAsyncWebRequest.java:415)
	at org.springframework.util.StreamUtils.copy(StreamUtils.java:137)
	at org.springframework.http.converter.StringHttpMessageConverter.writeInternal(StringHttpMessageConverter.java:128)
	at org.springframework.http.converter.StringHttpMessageConverter.writeInternal(StringHttpMessageConverter.java:44)
	at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:234)
	at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitterReturnValueHandler$DefaultSseEmitterHandler.sendInternal(ResponseBodyEmitterReturnValueHandler.java:315)
	at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitterReturnValueHandler$DefaultSseEmitterHandler.send(ResponseBodyEmitterReturnValueHandler.java:295)
	at org.springframework.web.servlet.mvc.method.annotation.ResponseBodyEmitter.send(ResponseBodyEmitter.java:211)
	... 5 common frames omitted

How to reproduce?

1. Enable GzipHandler with a min-response-size threshold (in Spring Boot: server.compression.enabled=true,
   server.compression.min-response-size=10240.
2. Implement a @RestController endpoint returning Response>.
3. In a background thread, write chunks via emitter.send(...) until the total response exceeds the threshold.
4. Send a request to the endpoint.

Expected: response streams to completion with gzip compres

Actual: first chunk is delivered, then the stream stalls ireaches the caller. Client times out with 1 byte received.

Workaround: exclude the affected endpoint(s) from GzipHand in a JettyServerCustomizer bean.

[lorban]

@zsait-clearstreet it seems you managed to create a simple reproducer, would you mind posting it as it would save us a lot of time?

Thanks!

[zsait-clearstreet]

@lorban Please see details and reproducer attached -- I had AI generate the reproducer and below report, but rest assured I am human and can provide more details if needed :)

---

jetty-15732-repro.zip

Reproducer attached (jetty-15732-repro.zip) — mvn test, three failing tests, ~15–25s total. Each test asserts the property that should hold, so the failures are the reproductions. Verified against Jetty 12.0.34, Java 21.0.10 (Corretto), macOS aarch64.

Please read the "what this does and does not show" section before spending time on it — I found a hole in my own analysis while putting it together, and the honest answer is that I can no longer explain our production failure with the mechanism the reproducer uses.

Two corrections to my original report

1. This is not Gzip specific. GzipHandler is only one way to get a concurrent mutation onto the response headers at commit time. The isolated test below involves no servlet layer at all, and the end-to-end test fails with no GzipHandler in the pipeline. We also continued to see the failure in production after setting server.compression.enabled=false. Apologies for the misleading title.

2. See below re: the reproducer's mechanism.

Test	Time	What it shows
HttpFieldsConcurrentMutationTest	~0.2s	MutableHttpFields corrupts under concurrent mutate/read — no servlet, no HTTP, no compression
AsyncResponseCommitRaceTest#withoutCompression	~5s	Same, end to end through the servlet stack, no GzipHandler
AsyncResponseCommitRaceTest#withCompression	~6s	Same, with GzipHandler, as originally filed

Root cause

HttpFields.contains(String) walks the iterator:

default boolean contains(String name)
{
    for (HttpField f : this)
    {
        if (f.is(name))   // f is null
            return true;
    }
    return false;
}

MutableHttpFields backs that iterator with a plain array and a plain size — _fields and _size are neither volatile nor guarded, so there is no happens-before edge between a mutating thread and a reading thread at all:

private HttpField[] _fields;
private int _size;

// iterator()
public boolean hasNext() { return _index < _size; }
public HttpField next()  { return _fields[_index++]; }

Three windows let a reader observe null at an index below _size:

1. Torn append.

_fields[_size++] = field;        // element store + non-atomic RMW of _size

A reader can observe the incremented _size while the element store is not yet visible to it.

2. Explicit null from remove. remove(int) decrements _size first and only later writes the null (line 479):

private void remove(int i)
{
    _size--;
    ...
    System.arraycopy(_fields, i + 1, _fields, i, _size - i);
    _fields[_size] = null;
}

A reader that captured _size before the decrement iterates to the old bound and reads exactly the slot being nulled. No reordering needed for this one.

3. Lost write into an orphaned array. If one thread grows via Arrays.copyOf and republishes _fields while another still holds the previous reference, the second thread's element lands in the discarded array while the shared _size advances. The live array keeps a permanent null at that index — which is why a later scan by Jetty's own code can fault on an array no application thread is touching any more.

For completeness: copyImmutable() copies to the same length and so introduces no nulls; put()'s in-place System.arraycopy(_fields, i + 1, _fields, i, _size-- - i-- - 1) leaves a stale duplicate rather than a null, but compounds with a reader's stale bound.

Two symptoms, one corrupted array

Once the array has a null hole, whichever thread touches it next faults. Both of these came out of every run we tried (three consecutive).

Reader side — as originally reported:

java.lang.NullPointerException: Cannot invoke
    "org.eclipse.jetty.http.HttpField.is(String)" because "f" is null
    at org.eclipse.jetty.http.HttpFields.contains(HttpFields.java:371)
    at org.eclipse.jetty.ee10.servlet.ServletApiResponse.containsHeader(ServletApiResponse.java:117)
    at jakarta.servlet.http.HttpServletResponseWrapper.containsHeader(HttpServletResponseWrapper.java:68)

Writer side — inside Jetty's own response preparation, no application frame in the stack:

java.lang.NullPointerException: Cannot invoke
    "org.eclipse.jetty.http.HttpField.isSameName(org.eclipse.jetty.http.HttpField)" because "f" is null
    at org.eclipse.jetty.http.MutableHttpFields.put(MutableHttpFields.java:322)
    at org.eclipse.jetty.http.HttpFields$Mutable.put(HttpFields.java:1364)
    at org.eclipse.jetty.server.internal.HttpChannelState$ChannelResponse.lockedPrepareResponse(HttpChannelState.java:1495)
    at org.eclipse.jetty.server.internal.HttpChannelState$ChannelCallback.completeChannelCallback(HttpChannelState.java:1598)
    at org.eclipse.jetty.server.internal.HttpChannelState$ChannelCallback.succeeded(HttpChannelState.java:1531)
    at org.eclipse.jetty.ee10.servlet.ServletChannel.onCompleted(ServletChannel.java:769)
    at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:428)

put carries the same unguarded 0.._size scan as the iterator.

What this reproducer does and does not show

To be straight about it: the end-to-end test drives the corruption from two application threads writing to one response (a CyclicBarrier forces their flushes to overlap). The Servlet API does not permit that, so on its own that test is arguably application misuse and you would be right to say so.

What I think is still worth your attention is the second stack above — once the array is corrupted, the fault surfaces inside lockedPrepareResponse on a container thread, and MutableHttpFields has no null guard on any of its 0.._size scans.

And I can no longer explain our production failure with this mechanism. Spring's ResponseBodyEmitter serialises every application entry point behind a ReentrantLock — send, complete, completeWithError, and the onTimeout/onError/onCompletion registrations — so our streaming thread and heartbeat thread could not have been in the write path simultaneously. I had assumed they could; they can't.

The remaining candidate is a container thread racing an application thread: an async timeout or completion running lockedPrepareResponse while an application thread is inside send(), since Jetty's internals do not take Spring's lock. Our setup had a 30-minute Spring async timeout against a 15-minute Jetty idle timeout, which fits the shape. I have not managed to isolate that one.

So the question I would most like answered: are a response's HttpFields expected to be confined to a single thread at commit time, and is the container permitted to mutate them while an application thread is in a write? If yes, that tells me where to look on our side. If no, then the missing null guards and the absent happens-before edges in MutableHttpFields are live issues in their own right, and I can aim a reproducer at the container/application collision instead.

On a fix

Null-guarding the readers would mask the symptom but leave the race. Making _fields and _size volatile closes the visibility window for a single appending writer, but remove's decrement-then-null and put's in-place compaction are compound operations that still aren't atomic against a concurrent reader.

The underlying question is whether HttpFields.Mutable is meant to be safe for concurrent access at all. If it isn't, then the response's field collection needs to be provably unreachable from two threads at commit time.

Possibly related: #11459, #12159, #12022, #11370.

Happy to reshape any of this — as a patch against your own test suite, or aimed at the container/application collision if you can confirm the threading expectation.

[lorban]

I had a look at your reproducer, it contains two tests: HttpFieldsConcurrentMutationTest and AsyncResponseCommitRaceTest.

HttpFieldsConcurrentMutationTest mutates a HttpFields.Mutable instance from two threads; since HttpFields.Mutable is not thread-safe, this isn't a bug.

AsyncResponseCommitRaceTest starts two threads that are synchronized to write at the same time to the ServletOutputStream. Servlet APIs are not thread-safe either, so this is again not a real bug.

If those are the actual root cause of your issue, then the problem is that you're using non-thread-safe APIs from multiple threads in parallel and you should sort out your code to avoid doing that. I'm surprised your AI did not point that out.