feat(ai-proxy): send LLM requests through ngx_http_ffi_client - #13778
Merged
shreemaan-abhishek merged 14 commits intoAug 18, 2026
Merged
shreemaan-abhishek merged 14 commits into
shreemaan-abhishek merged 14 commits into
Conversation
ai-proxy, ai-proxy-multi and ai-request-rewrite share ai-transport/http.lua for every outbound LLM request. It now prefers ngx_http_ffi_client, a C client whose object API matches lua-resty-http and which costs about a third of the outbound CPU time. The module only exists when the APISIX runtime was built with it, so the transport resolves the client once per worker and keeps lua-resty-http as the fallback. plugin_attr.ai-proxy.http_client pins the choice: auto (default), ffi, or lua-resty-http. Connection and Transfer-Encoding are now dropped from the forwarded headers. They describe the downstream connection, and passing them on desyncs the upstream one: the client frames the request body with Content-Length itself, and a forwarded "Connection: close" would also keep the connection out of the keepalive pool.
dosubot
Bot
added
size:L
Dropping Connection and Transfer-Encoding from the headers forwarded to the LLM upstream broke an existing ai-proxy-multi retry case: after a stream that dies before its first byte, the retry stopped finding an instance to re-pick and the request returned 502. Forwarding a downstream Connection header to a third-party upstream is still wrong, and the streaming read-error path still leaks the upstream connection without closing it, but neither belongs in a change about which HTTP client the transport builds.
…p_ffi_client plugin_attr.ai-proxy.http_client took auto, ffi or lua-resty-http, and auto made the choice implicit in what the runtime happened to carry. It now takes one of the two client names, ngx_http_ffi_client or lua-resty-http, and defaults to the first. A runtime built without the module still falls back rather than failing the request, and says so at warn level. Error level would put a line in the log for every AI request on such a runtime. t/plugin/ai-transport-http.t TEST 11 drives the lua-resty-http path against a real upstream with nothing stubbed, so the path that config selects is covered end to end rather than only through a stub.
…ing back Resolve the client from plugin_attr.ai-proxy.http_client through a name to module map, validate the name against an enum schema, and cache it only once it has loaded. A missing or unusable client is now an error the request carries, never a silent switch to the other one. Bump APISIX_RUNTIME to 1.3.12, the first runtime built with ngx_http_ffi_client.
Cosockets get their hostnames resolved by apisix/patch.lua, which routes them through core.resolver and so honours dns_resolver, /etc/hosts and the search domains. ngx_http_ffi_client dials from C and never touches a cosocket, so it saw only nginx's `resolver` and failed on every name that layer cannot answer. The transport now resolves the name the same way before handing the address to the C client, keeping the original for the Host header and the SNI. Pin the body-encoding and error-mapping cases to lua-resty-http: they stub resty.http, so under the C-client default they were driving the real client at a dead port instead of the stub.
dosubot
Bot
added
size:XL
| -- core.resolver, which honours dns_resolver, /etc/hosts and the search | ||
| -- domains. The C client dials on its own and only sees nginx's `resolver`, | ||
| -- so the name is resolved here and kept for the Host header and the SNI. | ||
| local function resolve_upstream_host(params) |
Contributor
Author
There was a problem hiding this comment.
ngx_http_ffi_client dials from C, so it never touches the cosocket that patch.lua wraps — it only sees nginx's resolver, which does not read /etc/hosts or our dns_resolver config. That is why every localhost upstream 500'd. Resolving here puts it back on the same path as every other socket in the gateway, and we keep the name for Host and SNI.
Member
There was a problem hiding this comment.
Could ngx_http_ffi_client provide a hook for DNS resolution? Otherwise, the DNS‑resolving code would have to be duplicated across many plugins.
membphis
reviewed
Aug 7, 2026
membphis
left a comment
membphis
left a comment
Member
There was a problem hiding this comment.
[P1] Please address these blockers before merge:
-
The current t/plugin/[a-k]*.t CI shard is failing on this head, and it contains the changed t/plugin/ai-transport-http.t coverage. Please identify the cause and make the relevant checks pass on this exact head.
-
Please make the ngx_http_ffi_client revision bundled in apisix-runtime 1.3.12 traceable, and add APISIX-level coverage that runs the real C client through buffered responses, SSE streaming, TLS/DNS, and keepalive. The new tests mostly stub the module, so they do not prove that the new default client includes and preserves the required framing, connection-lifecycle, and error-semantics fixes.
-
The linux_apisix_current_luarocks_in_customed_nginx job is also failing. Because custom runtimes may not include the C module, please verify the supported compatibility path on this head: either include the module or explicitly configure and test lua-resty-http as the fallback for that build.
1.3.15 is the first runtime built with ngx_http_ffi_client v0.1.2, which carries the three fixes this change depends on: request bodies containing CR or LF are no longer rejected, connection errors respect lua_socket_log_errors, and the trust store falls back to lua_ssl_trusted_certificate so verified TLS works on a bundled OpenSSL.
The selection tests stub resty.ngx_http_ffi_client, so module loading, the FFI boundary and the connection lifecycle were never exercised. Add cases that run the real client against local upstreams: a buffered request, an SSE response read to completion through body_reader, three requests over one pooled connection (asserted with the upstream's connection_requests counter), a hostname upstream that only resolves through core.resolver, and TLS with ssl_verify on and no per-call CA. They skip where the module is absent rather than failing for the wrong reason. Also cover a genuine require() failure: assigning a string to package.loaded makes require() succeed, so the old case only reached the non-table branch. A package.preload loader that raises reaches the pcall(require) failure path.
nic-6443
reviewed
Aug 10, 2026
The skip guard checked a hardcoded /usr/local/openresty path, which is not necessarily the runtime under test, and it checked the Lua binding rather than whether the module is compiled in. Ask the binary instead, falling back to bare nginx the way both CIs resolve it (neither sets TEST_NGINX_BINARY; both prepend $OPENRESTY_PREFIX/nginx/sbin to PATH). Also correct the skip count from 2 to 3: Test::Nginx adds a status-code assertion alongside response_body and no_error_log. The file now reports the same subtest total whether these blocks run or skip.
The selection, validation and DNS-parity logic sat inside the AI transport, so forward-auth, http-logger and anything else moving off lua-resty-http could not reuse it. It now lives in apisix/utils/http.lua alongside the other shared client factories: the module owns the client names, the schema fragment, loading and caching, and resolve_upstream_host(); the caller owns where the preference comes from and passes the name in. No behaviour change: the transport still reads plugin_attr.ai-proxy.http_client and the whole suite is unchanged.
t/sse_server_example/sse_server_example is the Go binary produced by ci/common.sh's start_sse_server_example; it is built at test time and must not be tracked.
AlinsRan
previously approved these changes
Aug 11, 2026
Ships ngx_http_ffi_client v0.1.3.
shreemaan-abhishek
dismissed stale reviews from AlinsRan and membphis
via
August 13, 2026 12:06
93dc395
The C client takes a resolver as of ngx_http_ffi_client v0.1.3, so the transport hands it core.resolver.parse_domain once at load instead of resolving each request itself.
…fi-client # Conflicts: # .requirements # ci/linux-install-openresty.sh
nic-6443
approved these changes
Aug 17, 2026
AlinsRan
approved these changes
Aug 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Why.
ai-proxy,ai-proxy-multiandai-request-rewriteall send theiroutbound LLM requests through
apisix/plugins/ai-transport/http.lua. That pathruns on every request the AI gateway proxies, and
lua-resty-httpspendsroughly three times the outbound CPU time of
ngx_http_ffi_client, the HTTPclient implemented as an nginx C module.
How.
plugin_attr.ai-proxy.http_clientnames the client:ngx_http_ffi_client, the default, orlua-resty-http. The name is validatedagainst an enum schema and mapped to a module, the module is loaded on the
first request, and the result is cached only once a client has actually loaded.
The two clients share an object API (
new,set_timeout,connect,request,res.body_reader,res:read_body,set_keepalive,close), sothe change is confined to which client the transport constructs; the providers
and the streaming path are untouched.
A client that is missing or cannot be created fails the request and the error
names the cause. The transport never falls back to the other client, so what
runs is always what the config asked for.
Name resolution. Cosockets have their hostnames resolved by
apisix/patch.lua, which routes them throughcore.resolverand so honoursdns_resolver,/etc/hostsand the search domains. The C client dials from Cand never touches a cosocket, so left alone it would see only nginx's
resolverand resolve names by different rules than the rest of the gateway.It takes a resolver as of v0.1.3 (api7/ngx_http_ffi_client#45), so the client
is handed
core.resolver.parse_domainonce, when the module loads. The clientresolves before it derives its connection-pool key and keeps the name for the
Hostheader and the SNI. A client too old to take a resolver fails therequest and names that as the cause, rather than silently resolving by its own
rules.
Runtime
APISIX_RUNTIMEmoves to 1.3.16, built withngx_http_ffi_clientv0.1.3 (api7/apisix-build-tools#488, released as
apisix-runtime/1.3.16;the tag is that PR's merge commit).
ci/linux-install-openresty.shpicks up thenew debug-deb checksums for both architectures.
Why this client revision. 1.3.13 and 1.3.14 pin v0.1.1, which is v0.1.0 plus a
LICENSE, a README section and a
configtweak, with byte-identical C sources.The releases that matter start at v0.1.2, and v0.1.3 is the first to carry all
four of the defects and gaps this cutover surfaced:
lua_socket_log_errors[error]linelua_ssl_trusted_certificateresolver, so the gateway's own resolution rules never apply to itTo check what a given build actually contains:
which prints the
--add-module=.../ngx_http_ffi_client-<tag>path. On thepublished 1.3.16 packages that reads
ngx_http_ffi_client-v0.1.3.v0.1.3 is v0.1.2 plus the resolver hook, and that hook is a change to the Lua
binding alone: the C sources are byte-identical between the two tags.
Performance record for the client:
https://github.com/api7/ngx_http_ffi_client/blob/63771541c5a229da8a840ab6008ba3771a22fb71/benchmark/results-full.md
A hand-built runtime without the module has to set
http_client: lua-resty-http, which runs exactly the code that ran before.Which issue(s) this PR fixes:
N/A
Checklist
Test plan
t/plugin/ai-transport-http.tcovers client selection with stubs, and thendrives the real C client against local upstreams so module loading, the FFI
boundary and the connection lifecycle are exercised rather than mocked:
pcall(require, ...)failure pathhttp_clientfails schema validationplugin_attrselectslua-resty-http, and that path runs against a real upstream with the C client stubbed to a sentinel so a selection regression cannot pass silentlycore.resolver.parse_domain, the gateway's own resolverbody_readerconnection_requestscounterlocalhost, which only resolves throughcore.resolver, with the name kept in theHostheaderssl_verifyon and no per-call CA, verifying againstlua_ssl_trusted_certificateTESTs 16-20 carry
--- skip_eval: 2: !-f "/usr/local/openresty/lualib/resty/ngx_http_ffi_client.lua"so they run wherever the module is installed, which includes CI on 1.3.16, and
skip on a runtime without it rather than failing for the wrong reason.
Beyond the suite, this exact code was run on a gateway carrying the pinned
client against real Azure OpenAI, comparing both clients on the same
routes:
[DONE][DONE]That covers three protocol adapters (
openai-chat,openai-responses,anthropic-messages), TLS with SNI, streaming, retry and connection reuse.The resolver hook was re-run against the same three providers. Two extra routes
went to upstreams named only in
/etc/hosts, which nginx'sresolvercannotanswer for: both returned 200 with the name intact in the
Hostheader theupstream received, and both failed with
<name> could not be resolvedonce theset_resolvercall was removed, which is what pins the behaviour on the hook.The three Azure hostnames happen to share one address, so the interleaved runs
also cover pooling across SNIs on a single peer:
ssl_server_nameis part ofthe client's pool key, so they never share a connection.
Reassembled streaming text is identical on both clients; only the raw byte count
differs, and it varies run-to-run within each client, so that is provider
variance rather than client behaviour.