rustc emits a vacant vtable slot for a callable boxed async service (Rust 1.98)

[GlenDC]

View all comments

Observed on aarch64-apple-darwin.
It also consistently hits my CI runners since release of 1.98,
failing our CI... each commit (https://github.com/plabayo/rama)

The same Rama source works with Rust 1.97.1, crashes with
Rust 1.98.0, and works again with nightly-2026-07-16.

Rust 1.98 emits a zero method entry in a compiler-generated
vtable. Safe Rust dispatches through that entry and the process
segfaults at address zero.

Reproduction

Based on my CI failures I made a small example using our rama Repo,
note that it assumes TCP port 62017 is available:

git clone https://github.com/plabayo/rama.git
cd rama
git checkout --detach 0b4aa58e33e9d07db2718a3886c05b673ba9a70e

rustup toolchain install 1.98.0
just clean

CARGO_TARGET_DIR=target-ticket-198 \
cargo +1.98.0 build \
  -p rama-examples \
  --features=cli,tcp,http-full,proxy-full,boring \
  --bin http_mitm_proxy_boring

Than run:

./target-ticket-198/debug/http_mitm_proxy_boring \
  >/tmp/rama-server-198.log 2>&1 &
proxy_pid=$!

sleep 2

curl -skv --max-time 5 \
  --proxy http://john:secret@127.0.0.1:62017 \
  https://127.0.0.1:1/ \
  >/tmp/rama-curl-198.log 2>&1 || true

if kill -0 "$proxy_pid" 2>/dev/null; then
  echo "proxy is still alive"
  kill "$proxy_pid"
  wait "$proxy_pid" 2>/dev/null || true
else
  wait "$proxy_pid"
  echo "proxy exited with status $?"
fi

sed -n '1,120p' /tmp/rama-curl-198.log
sed -n '1,120p' /tmp/rama-server-198.log

Repeat with separate targets (w/ clean target directory) for 1.97.1 and nightly-2026-07-16 (tried to narrow down...).

What you should see is the following:

• 1.97.1: 502 Bad Gateway; proxy remains alive
• 1.98.0: CONNECT aborts; SIGSEGV at address zero
• nightly-2026-07-16: 502 Bad Gateway; proxy remains alive

In short, it is expected that the failed upstream connection should be converted into HTTP/1.1 502 Bad Gateway, and the proxy should remain alive.

Instead it is obsered in 1.98 that the CONNECT request is aborted. macOS reports SIGSEGV with pc=0. The generated caller loads the Service method from vtable + 0x18 and branches to the zero value stored there.

Diving into the code it seems that the Rust 1.98 vtable contains valid drop/size/alignment fields,
but the first method slot at offset 24 is zero. -Zprint-mono-items collects the erased caller on Rust 1.98, but
not EagerHttpProxyConnector's concrete Service::serve method. nightly-2026-07-16 collects both the method and its async closures.

The method may be reaching VtblEntry::Vacant because its predicates are incorrectly considered impossible.

Relevant rama types:

• DynService/BoxService
• EagerHttpProxyConnector::serve
• responder.boxed() boundary

Meta

macOS crash report:

Exception:       EXC_BAD_ACCESS (SIGSEGV)
Subtype:         KERN_INVALID_ADDRESS at 0x0000000000000000
Termination:     Segmentation fault: 11
Faulting thread: tokio-rt-worker
PC:              0x0000000000000000
LR:              0x0000000102511674

The link register resolves to:
rama_http::layer::upgrade::service::
  <UpgradeService<...> as Service<Request>>::serve::{closure#0}
  + 2956

Backtrace

Thread 2 Crashed: tokio-rt-worker

0  0x0000000000000000
1  <UpgradeService<...> as Service<Request>>::serve::{closure#0}
2  <tracing::instrument::Instrumented<
       Trace<...>::serve::{closure#0}
   > as Future>::poll
3  <rama_http_core::proto::h1::dispatch::Server<...>
       as Dispatch>::recv_msg::{closure#0}
4  rama_http_core::proto::h1::dispatch::Dispatcher<...>::poll_write
5  rama_http_core::proto::h1::dispatch::Dispatcher<...>::poll_catch
6  <rama_http_core::server::conn::http1::
       UpgradeableConnection<...> as Future>::poll
7  <rama_http_core::server::conn::auto::
       UpgradeableConnection<...> as Future>::poll
...
13 tokio::runtime::scheduler::multi_thread::worker::Context::run_task
14 tokio::runtime::scheduler::multi_thread::worker::Context::run

The program counter is exactly zero. The link register points inside UpgradeService::serve, at the indirect boxed Service call. Together with the emitted vtable containing zero at method offset 0x18, this indicates that dispatch branches through the null vtable entry.

This may be a recurrence of #152735/#153596 and appears closely related to #158148. It behaviour seems slightly different though: Rust 1.98 silently emits a vacant method slot and the resulting binary crashes, rather than rustc producing an ICE.

Note that this is for some reason the only location where it triggers despite:

• boxed services being used in many other places
• that we use that async dynamic trait dispatch trick for several other of our traits as well

Yet somehow only with this trait in this exact example it is failing... So there is hope I am just doing something weird here, as I havent found a stable rustc bug in 11 years that I used Rust in production.... So I'm really hoping to be wrong here...

Workaround

Found a work around but I would rather not have to add these to the codebase...

Because either I am (a) doing something bad since long and I rather just fix it properly in that case,
as it would mean I was just "lucky" so far. Or this is a real rust bug and in that case I rather help get a 1.98.1 through the door... So far my money is still on me doing something wrong ... somehow... but I don't see how.
AFAIK that code hasn't changed in any way, and it clearly reproduces on rust 1.98 while it does not on previous versions (e.g. 1.97.1) or nightly versions slightly there-after...

Work around

diff --git a/rama-http/src/layer/upgrade/service.rs b/rama-http/src/layer/upgrade/service.rs
index d21adcc8b..eaf030fae 100644
--- a/rama-http/src/layer/upgrade/service.rs
+++ b/rama-http/src/layer/upgrade/service.rs
@@ -125,7 +125,7 @@ pub struct UpgradeHandler<O> {
 
 enum UpgradeHandlerKind<O> {
     ResponseLocal {
-        responder: BoxService<Request, UpgradeOutput<Request, O>, O>,
+        responder: ResponseLocalResponder<O>,
         handler_error_sink: Arc<dyn ErrorSink>,
     },
     SeparateServices {
@@ -149,6 +149,13 @@ struct PreparedUpgrade<O> {
     continuation: UpgradeContinuation,
 }
 
+type ResponseLocalResponderFuture<O> =
+    Pin<Box<dyn Future<Output = Result<UpgradeOutput<Request, O>, O>> + Send + 'static>>;
+// Keep the generic `R: Service` predicate outside the erased call boundary.
+// Rust 1.98 can otherwise emit a vacant `BoxService` vtable entry here.
+type ResponseLocalResponder<O> =
+    Arc<dyn Fn(Request) -> ResponseLocalResponderFuture<O> + Send + Sync + 'static>;
+
 impl<O: Send + 'static> UpgradeHandler<O> {
     /// Register one service which returns its response-local upgrade handler.
     pub(crate) fn new<M, R, Sink>(matcher: M, responder: R, sink: Sink) -> Self
@@ -157,10 +164,15 @@ impl<O: Send + 'static> UpgradeHandler<O> {
         R: Service<Request, Output = UpgradeOutput<Request, O>, Error = O> + Clone,
         Sink: ErrorSink,
     {
+        let responder: ResponseLocalResponder<O> = Arc::new(move |request| {
+            let responder = responder.clone();
+            Box::pin(async move { responder.serve(request).await })
+        });
+
         Self {
             matcher: Box::new(matcher),
             kind: UpgradeHandlerKind::ResponseLocal {
-                responder: responder.boxed(),
+                responder,
                 handler_error_sink: Arc::new(sink),
             },
             _phantom: std::marker::PhantomData,
@@ -203,7 +215,7 @@ impl<O: Send + 'static> UpgradeHandler<O> {
             UpgradeHandlerKind::ResponseLocal {
                 responder,
                 handler_error_sink,
-            } => responder.serve(request).await.map(
+            } => responder(request).await.map(
                 |UpgradeOutput {
                      response,
                      request,

Prior to that I also had a work around which changed all boxed services ,not just this specific use case of it,
I also recall that working. Giving it here as well as extra info:

box service global work around

diff --git a/rama-core/src/service/svc.rs b/rama-core/src/service/svc.rs
--- a/rama-core/src/service/svc.rs
+++ b/rama-core/src/service/svc.rs
@@ -1,5 +1,6 @@
 //! [`Service`] and [`BoxService`] traits.
 
+use core::any::Any;
 use core::convert::Infallible;
 use core::fmt;
 use core::marker::PhantomData;
@@ -145,46 +146,41 @@ where
     }
 }
 
-/// Internal trait for dynamic dispatch of Async Traits,
-/// implemented according to the pioneers of this Design Pattern
-/// found at <https://rust-lang.github.io/async-fundamentals-initiative/evaluation/case-studies/builder-provider-api.html#dynamic-dispatch-behind-the-api>
-/// and widely published at <https://blog.rust-lang.org/inside-rust/2023/05/03/stabilizing-async-fn-in-trait.html>.
-trait DynService<Input> {
-    type Output;
-    type Error;
-
-    #[expect(clippy::type_complexity)]
-    fn serve_box(
-        &self,
-        input: Input,
-    ) -> Pin<Box<dyn Future<Output = Result<Self::Output, Self::Error>> + Send + '_>>;
-}
+// Keep dispatch separate from the erased value's vtable. Rust 1.98 can emit a
+// vacant method slot for the custom async-erasure trait with complex services.
+type ErasedService = dyn Any + Send + Sync + 'static;
+type BoxServiceFuture<'a, Output, Error> =
+    Pin<Box<dyn Future<Output = Result<Output, Error>> + Send + 'a>>;
+type ServeErasedFn<Input, Output, Error> =
+    for<'a> fn(&'a ErasedService, Input) -> BoxServiceFuture<'a, Output, Error>;
 
-impl<Input, T> DynService<Input> for T
+fn serve_erased<'a, Input, Output, Error, T>(
+    inner: &'a ErasedService,
+    input: Input,
+) -> BoxServiceFuture<'a, Output, Error>
 where
-    T: Service<Input>,
+    T: Service<Input, Output = Output, Error = Error>,
 {
-    type Output = T::Output;
-    type Error = T::Error;
-
-    fn serve_box(
-        &self,
-        input: Input,
-    ) -> Pin<Box<dyn Future<Output = Result<Self::Output, Self::Error>> + Send + '_>> {
-        Box::pin(self.serve(input))
-    }
+    #[expect(
+        clippy::unwrap_used,
+        reason = "the constructor stores the same service type as its dispatch function"
+    )]
+    let service = inner.downcast_ref::<T>().unwrap();
+    Box::pin(service.serve(input))
 }
 
 /// A boxed [`Service`], to serve Inputs with,
 /// for where you inputuire dynamic dispatch.
 pub struct BoxService<Input, Output, Error> {
-    inner: Arc<dyn DynService<Input, Output = Output, Error = Error> + Send + Sync + 'static>,
+    inner: Arc<ErasedService>,
+    serve: ServeErasedFn<Input, Output, Error>,
 }
 
 impl<Input, Output, Error> Clone for BoxService<Input, Output, Error> {
     fn clone(&self) -> Self {
         Self {
             inner: self.inner.clone(),
+            serve: self.serve,
         }
     }
 }
@@ -198,6 +194,7 @@ impl<Input, Output, Error> BoxService<Input, Output, Error> {
     {
         Self {
             inner: Arc::new(service),
+            serve: serve_erased::<Input, Output, Error, T>,
         }
     }
 }
@@ -223,7 +220,7 @@ where
 
         input: Input,
     ) -> impl Future<Output = Result<Self::Output, Self::Error>> + Send + '_ {
-        self.inner.serve_box(input)
+        (self.serve)(self.inner.as_ref(), input)
     }
 
     #[inline]

[theemathas]

Slightly smaller reproducer, with some LLM assistance (depends on rama with default-features = false and features =["std", "http-full"]).

Details

use rama::{
    Layer, Service, ServiceInput,
    http::{Body, Request},
    http::{
        client::EasyHttpWebClient,
        layer::upgrade::{EagerHttpProxyConnector, UpgradeLayer, mitm::HttpUpgradeMitmRelayLayer},
        matcher::MethodMatcher,
        proxy::mitm::HttpMitmRelay,
        ws::handshake::{
            matcher::HttpWebSocketRelayServiceRequestMatcher,
            mitm::{WebSocketRelayInput, WebSocketRelayOutput, WebSocketRelayService},
        },
    },
    layer::{ArcLayer, ConsumeErrLayer},
    net::client::{ConnectRequest, ConnectionError, EstablishedClientConnection},
    rt::Executor,
    service::service_fn,
};
use tokio::io::{AsyncRead, AsyncWrite};

use std::{convert::Infallible, sync::Arc};

async fn inspect_websocket_message(
    input: WebSocketRelayInput,
) -> Result<WebSocketRelayOutput, Infallible> {
    Ok(WebSocketRelayOutput {
        messages: vec![],
        extensions: input.extensions,
    })
}

trait AsyncReadWrite: AsyncRead + AsyncWrite {}
impl<T: AsyncRead + AsyncWrite> AsyncReadWrite for T {}

#[tokio::main]
async fn main() {
    let exec = Executor::default();
    let websocket_mitm_layer = HttpUpgradeMitmRelayLayer::new(
        exec.clone(),
        HttpWebSocketRelayServiceRequestMatcher::new(WebSocketRelayService::new(service_fn(
            inspect_websocket_message,
        ))),
    );
    let http_mitm_relay = HttpMitmRelay::new(exec.clone())
        .with_http_middleware((websocket_mitm_layer, ArcLayer::new()));
    let mitm_svc = Arc::new(ConsumeErrLayer::trace_as_debug().into_layer(http_mitm_relay));
    let connector = service_fn(|_req: ConnectRequest| async move {
        Err::<
            EstablishedClientConnection<
                ServiceInput<Box<dyn AsyncReadWrite + Send + Unpin>>,
                ConnectRequest,
            >,
            _,
        >(ConnectionError::unknown(""))
    });
    let connect = EagerHttpProxyConnector::new(connector, mitm_svc);
    let service = UpgradeLayer::new(exec, MethodMatcher::GET, connect)
        .into_layer(EasyHttpWebClient::default());
    let request = Request::builder().body(Body::empty()).unwrap();
    let _ = service.serve(request).await.unwrap();
}

[theemathas]

"Smaller" reproducer (thanks in part to LLM again): segfault-repro.zip

[theemathas]

Self-contained reproducer (205 lines, probably could be smaller)

Details

use std::convert::Infallible;
use std::marker::PhantomData;
use std::sync::Arc;

trait Service<Input>: Sized + Send + Sync + 'static {
    /// The type of the output returned by the service.
    type Output: Send + 'static;

    /// The type of error returned by the service.
    type Error: Send + 'static;
}
impl<S, Input> Service<Input> for Arc<S>
where
    S: Service<Input>,
{
    type Output = S::Output;
    type Error = S::Error;
}

fn service_fn<F, T, R, O, E>(f: F) -> ServiceFn<F, T, R, O, E>
where
    F: Factory<T, R, O, E>,
    R: Future<Output = Result<O, E>>,
{
    ServiceFn::new(f)
}

trait Factory<T, R, O, E>: Send + Sync + 'static
where
    R: Future<Output = Result<O, E>>,
{
}

impl<Input, F, R, O, E> Factory<(Input,), R, O, E> for F
where
    F: Fn(Input) -> R + Send + Sync + 'static,
    R: Future<Output = Result<O, E>>,
{
}

struct ServiceFn<F, T, R, O, E>
where
    R: Future<Output = Result<O, E>>,
{
    hnd: F,
    _t: PhantomData<fn(T, R, O) -> ()>,
}

impl<F, T, R, O, E> ServiceFn<F, T, R, O, E>
where
    R: Future<Output = Result<O, E>>,
{
    pub(crate) fn new(hnd: F) -> Self {
        Self {
            hnd,
            _t: PhantomData,
        }
    }
}

impl<F, T, R, O, E> Clone for ServiceFn<F, T, R, O, E>
where
    F: Factory<T, R, O, E> + Clone,
    R: Future<Output = Result<O, E>>,
{
    fn clone(&self) -> Self {
        Self {
            hnd: self.hnd.clone(),
            _t: PhantomData,
        }
    }
}

trait ServiceMatcher<Input>: Send + Sync + 'static {
    type Service: Send + 'static;
    type Error: Send + 'static;
    type ModifiedInput: Send + 'static;
}

trait Layer<S>: Sized {
    type Service;
}
impl<S, L1, L2> Layer<S> for (L1, L2)
where
    L1: Layer<L2::Service>,
    L2: Layer<S>,
{
    type Service = L1::Service;
}

#[derive(Debug, Clone, Copy, Default)]
pub struct ArcLayer;

impl ArcLayer {
    #[inline(always)]
    pub fn new() -> Self {
        Self
    }
}

impl<S> Layer<S> for ArcLayer {
    type Service = Arc<S>;
}

struct MyError;

pub(crate) struct Request;
pub(crate) struct Response<T = Body>(T);
pub(crate) trait StreamingBody {
    type Error;
}
pub(crate) struct Body;
impl StreamingBody for Body {
    type Error = MyError;
}

pub(crate) struct HttpClientService;

impl Service<Request> for HttpClientService {
    type Output = Response;
    type Error = MyError;
}

trait Trait {
    fn method(&self);
}

impl<M> Trait for M
where
    M: Layer<
            HttpClientService,
            Service: Service<Request, Output = Response, Error: Into<MyError>> + Clone,
        >,
{
    fn method(&self) {}
}

async fn inspect_websocket_message(
    _input: WebSocketRelayInput,
) -> Result<WebSocketRelayOutput, Infallible> {
    Ok(WebSocketRelayOutput)
}

#[derive(Clone)]
pub(crate) struct HttpUpgradeMitmRelayLayer<M>(pub(crate) M);

impl<M: Clone, S> Layer<S> for HttpUpgradeMitmRelayLayer<M> {
    type Service = HttpUpgradeMitmRelay<M, S>;
}

#[derive(Clone)]
#[expect(dead_code)]
pub(crate) struct HttpUpgradeMitmRelay<M, S> {
    nested_matcher_svc: M,
    inner_svc: S,
}

impl<M, S, ResBody> Service<Request> for HttpUpgradeMitmRelay<M, S>
where
    M: ServiceMatcher<Request>,
    S: Service<Request, Output = Response<ResBody>>,
    ResBody: StreamingBody<Error: Into<MyError>> + 'static,
{
    type Output = Response;
    type Error = S::Error;
}

#[derive(Debug, Clone)]
pub(crate) struct HttpWebSocketRelayServiceRequestMatcher<S>(pub(crate) S);

impl<S> ServiceMatcher<Request> for HttpWebSocketRelayServiceRequestMatcher<S>
where
    S: Clone + Send + Sync + 'static,
{
    type Service = S;
    type Error = Infallible;
    type ModifiedInput = Request;
}

#[derive(Debug, Clone)]
pub(crate) struct WebSocketRelayService<S>(pub(crate) S);

impl<S> WebSocketRelayService<S> {}

#[derive(Debug, Clone)]
pub(crate) struct WebSocketRelayInput;

#[derive(Debug, Clone)]
pub(crate) struct WebSocketRelayOutput;

impl<S> Service<()> for WebSocketRelayService<S>
where
    S: Service<WebSocketRelayInput>,
{
    type Output = ();
    type Error = Infallible;
}

fn main() {
    let websocket_mitm_layer = HttpUpgradeMitmRelayLayer(HttpWebSocketRelayServiceRequestMatcher(
        WebSocketRelayService(service_fn(inspect_websocket_message)),
    ));
    (&(websocket_mitm_layer, ArcLayer::new()) as &dyn Trait).method();
    println!("done");
}

[theemathas]

cc @lcnr in case you know what the types is going on here. (I'm currently unable to bisect this myself, due to some brokenness with fastly's servers for downloading older rust compilers, affecting only my geographical area.)