Skip to content

feat(metrics): publish proxy metrics on a Prometheus endpoint - #31

Merged
hbrombeer merged 1 commit into
mainfrom
feat/proxy-metrics
Aug 3, 2026
Merged

hbrombeer merged 1 commit into
mainfrom
feat/proxy-metrics

Conversation

@hbrombeer

Copy link
Copy Markdown
Member

mc-router counts connections arriving at a region's front door. Each game server counts the players in its own world. Between them sits Velocity, and nothing measured it — a proxy holding players it cannot hand to a backend looks, from either side, like a quiet region.

What

The plugin binds a Prometheus endpoint when GROUNDS_METRICS_ENABLED=true (default false, port 9000, path /metrics):

Metric What it answers
velocity_players_online this proxy's own count. Against mc-router: does the router's view match the proxy's. Against the sum of minecraft_players_online: how many are on the proxy but not on a server — the transfer path, where a stuck player sits
velocity_servers_registered backends plugin-agones has registered. Zero with a healthy Fleet is discovery broken — invisible everywhere else, because Agones reports the servers Ready and the proxy simply has nowhere to send anyone
velocity_network_players what this proxy reports in the server list. Disagreeing with the sum of the proxies is a stale broadcast — the server list lying to everyone who has not joined yet

Plus Micrometer's JVM and process binders, under the same names the services and game servers publish, so one Grafana query covers all three. Every series carries region when REGION is set.

Deliberately not published: players per backend

Velocity knows it. But a backend's name is the Agones GameServer name — lobby-nl-ams1-tr9pf-s9fwt — which changes every time the Fleet replaces a pod. That is a label that churns with the workload, for a breakdown that already exists with stable labels: minecraft_players_online per app, and mc_router_server_active_connections per hostname.

Two deliberate choices

  • velocity_network_players is NaN, not 0, before the first broadcast. Zero would claim an empty network — exactly the lie onProxyPing already avoids by falling back to Velocity's own count. NaN is Prometheus' "no value" and renders as a gap. Asserted by a test.
  • A bind failure does not stop the proxy. Logged, startup continues. A proxy that cannot publish metrics is degraded; one that refuses to start because a port was taken takes every player in the region with it.

Gauges use strongReference(true) — Micrometer holds a gauge's state object weakly, so without it the snapshot is collected and every proxy gauge reports NaN hours later with the endpoint still up and the JVM series still correct. Same trap as the Minestom runtime.

Verification

  • ./gradlew build green — spotless + 73 tests, 9 of them new.
  • The new tests drive the real endpoint over HTTP on an ephemeral port: published names, the region label (and its absence when REGION is blank), gauges re-reading on every scrape rather than at registration, the NaN case, 404 on any other path, the socket closing on close(), and the config defaults plus a rejected path.
  • Velocity APIs checked against the actual velocity-api jar with javap, not assumed: ProxyServer.getPlayerCount(), getAllServers().

Follow-ups

  1. charts — grounds-velocity gains metrics.enabled (annotations + declared containerPort + env), same contract as grounds-gamemode
  2. grounds-pulumi — a Proxies dashboard
  3. deploy — turn it on for velocity and velocity-2

mc-router counts connections arriving at a region's front door, and each
game server counts the players in its own world. Between them sits Velocity,
and nothing measured it — a proxy holding players it cannot hand to a backend
looks, from either side, like a quiet region.

The plugin now binds a Prometheus endpoint when GROUNDS_METRICS_ENABLED=true
(default false, port 9000, path /metrics):

  velocity_players_online       this proxy's own count. Against mc-router it
                                says whether the router's view matches the
                                proxy's; against the sum of
                                minecraft_players_online it says how many are
                                on the proxy but not on a server
  velocity_servers_registered   backends plugin-agones has registered. Zero
                                with a healthy Fleet is discovery broken, and
                                it is invisible everywhere else — Agones
                                reports the servers Ready and the proxy simply
                                has nowhere to send anyone
  velocity_network_players      what this proxy reports in the server list

plus Micrometer's JVM and process binders, under the same names the services
and the game servers publish, so one query covers all three. Every series
carries `region` when REGION is set.

Deliberately not published: players per backend. Velocity knows it, but a
backend's name is the Agones GameServer name — lobby-nl-ams1-tr9pf-s9fwt —
which changes every time the Fleet replaces a pod. That is a label that churns
with the workload, and the same breakdown already exists with stable labels
(minecraft_players_online per app, mc_router_server_active_connections per
hostname).

velocity_network_players reports NaN, not 0, before service-player's first
broadcast. Zero would claim an empty network, which is exactly the lie the
ping handler avoids.

Gauges use strongReference(true): Micrometer holds a gauge's state object
weakly, so without it the snapshot is collected and every proxy gauge reports
NaN hours later with the endpoint still up. A failure to bind is logged and
the proxy keeps running — a proxy that cannot publish metrics is degraded, one
that refuses to start takes every player in the region with it.
@hbrombeer
hbrombeer merged commit 8889a04 into main Aug 3, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant