Skip to content

About

Self-hosted OpenTelemetry observability stack for Codex, Claude Code, and Gemini CLI, with Grafana dashboards for usage, tokens, estimated costs, latency, errors, tool activity, logs, metrics, and traces.

Topics

Resources

Stars

6 stars

Watchers

0 watching

Forks

Repository files navigation

AI CLI Observability

See where your AI coding time, tokens, and money go.

OpenTelemetry Grafana Docker Compose License

AI CLI Observability turns the built-in telemetry from Codex, Claude Code, and Gemini CLI into one self-hosted Grafana view. Track usage, latency, errors, tokens, tool activity, traces, and estimated API-equivalent cost without sending your telemetry to another SaaS.

Codex, Claude Code and Gemini CLI send OTLP over HTTP to the OpenTelemetry Collector. The collector writes logs to VictoriaLogs, traces to VictoriaTraces and metrics to VictoriaMetrics, and Grafana reads all three. The pricing exporter feeds price files to the collector and the EUR rate to VictoriaMetrics.

One Compose stack. Three AI CLIs. Full OpenTelemetry signals. Your data.

Text version of the diagram (Mermaid)
flowchart LR
    C[Codex] -->|OTLP/HTTP + bearer| O[OpenTelemetry Collector]
    A[Claude Code] -->|OTLP/HTTP + bearer| O
    G[Gemini CLI] -->|OTLP/HTTP + bearer| O
    P[Pricing exporter] -->|price files| O
    P -->|EUR rate + pricing metrics| M[VictoriaMetrics]
    O -->|logs| L[VictoriaLogs]
    O -->|metrics| M
    O -->|traces| T[VictoriaTraces]
    L --> D[Grafana]
    M --> D
    T --> D
Loading

The collector accepts OTLP/HTTP only:

Path Destination
/v1/logs VictoriaLogs
/v1/metrics VictoriaMetrics through Prometheus remote write
/v1/traces VictoriaTraces

Grafana and OTLP bind to 127.0.0.1 by default. Storage services are reachable only inside the Compose network.

What you can answer

Answer Signal
Which CLI and model consume the most tokens? Usage and token metrics
What would that activity cost through provider APIs? USD and EUR estimates
Where do slow requests and failures happen? Latency, errors, logs, and traces
Which tools are agents actually using? Immutable tool events

What ships

  • A ready-to-use AI CLI Tools - Overview Grafana dashboard
  • A Git AI - Code Attribution dashboard, fed by the optional git-ai exporter
  • Logs, metrics, and traces stored in the VictoriaMetrics stack
  • Client examples for Codex, Claude Code, and Gemini CLI
  • Collector-side Codex and Gemini cost estimation
  • Native Claude Code cost metrics
  • Live USD-to-EUR conversion for dashboard totals
  • Pinned container images and the VictoriaLogs Grafana plugin
  • An optional credit-reset dashboard for custom exporters

Safe by default

Local only Prompts off Estimates, not invoices
Grafana and OTLP bind to 127.0.0.1 by default. Prompt logging is disabled in all examples. Cost figures are estimates. Provider pricing changes.
Remote agents need a TLS proxy. Details in Privacy and security. Details in Cost estimation.

Note

This repository is a sanitized, single-host version of a real setup. It contains no production telemetry, credentials, hostnames, private network addresses, or identity-provider configuration.

Releases and updates

The stack uses one version for collector configuration, pricing and dashboards. The initial version is 0.1.0. Follow GitHub Releases and use the matching tagged sources and pricing image digest. RELEASING.md covers release assets, the optional GHCR Compose override, upgrades and rollback. Local builds below remain the default.

Step 1: start the stack

You need: Docker with Compose v2. Python 3.11 or newer is required only for the smoke test.

Copy the environment template and generate two secrets:

cp .env.example .env
openssl rand -hex 32
openssl rand -hex 32

Paste one value into OTEL_AUTH_TOKEN in .env and the other into GRAFANA_ADMIN_PASSWORD, then start the stack:

docker compose up -d --build
docker compose ps

Open http://localhost:3000, sign in with admin and your Grafana password, then connect a client below. Grafana provisions both dashboards and all three datasources automatically.

Prerequisites

  • Docker with Compose v2
  • Python 3.11 or newer for the smoke test
  • uv and Python 3.13 or newer for local pricing-exporter tests
  • Internet access to pull images and the Grafana VictoriaLogs plugin, and to refresh the exchange rate

Step 2: connect a client

The examples use http://localhost:4318. Export the collector token in the same process environment as the CLI:

export OTEL_AUTH_TOKEN="value-from-your-dot-env"

Codex

Merge clients/codex/config.toml.example into ~/.codex/config.toml. The example enables logs, metrics, traces, and runtime metrics while keeping raw prompt bodies disabled.

Codex exporter endpoints include their complete /v1/... paths.

Claude Code

Review and source clients/claude-code/telemetry.env.example before starting claude:

source clients/claude-code/telemetry.env.example
claude

Claude Code defaults to delta metric temporality. This setup selects cumulative temporality because Prometheus remote write cannot represent delta points.

Gemini CLI

Review and source clients/gemini-cli/telemetry.env.example before starting Gemini CLI. The file also contains an equivalent settings.json example. Keep the bearer header in the environment.

Gemini traces can contain more context than logs and metrics. Disable GEMINI_TELEMETRY_TRACES_ENABLED if that is not acceptable for your data.

Step 3: read the dashboards

Both dashboards are published on Grafana.com and can be imported by ID into any Grafana instance:

Dashboard Grafana.com ID
AI CLI Tools - Overview 25645
Git AI - Code Attribution 25803

The main dashboard combines:

  • LogsQL over immutable request and tool events in VictoriaLogs
  • PromQL over native CLI metrics in VictoriaMetrics
  • Pricing enrichment attributes added by the collector

Traces are available through the provisioned VictoriaTraces datasource in Grafana Explore.

The optional extras/credit-resets dashboard expects a separate custom metric and stays outside automatic provisioning when exact exporter-provided reset timestamps are preferred.

Publish on Grafana.com

Steps to publish a dashboard
  1. Load the provisioned dashboard and confirm its panels against real, sanitized data.
  2. In the dashboard toolbar, select Export and then Export as code.
  3. Under Advanced options, choose the Classic model.
  4. Enable Share dashboard with another instance to remove local deployment details, then download the JSON file.
  5. Sign in to Grafana.com, open My dashboards, and select Upload dashboard.
  6. Upload the JSON, complete the metadata, and select Save and Publish.

Do not upload .env, Grafana's database, screenshots with private labels, or a raw API response. Grafana documents the current process in Share dashboards and panels.

Cost estimation

Claude Code emits native cost data. Codex and Gemini estimates are calculated from token events and model prices supplied by the pinned LiteLLM data set, with explicit official overrides for missing models and stale prices. The pricing exporter also records pricing history in SQLite and exposes the USD-to-EUR rate to VictoriaMetrics.

Model coverage and pricing updates

Codex pricing covers gpt-6.2-astra, gpt-6.2-sol, GPT-6.1 Sol and Terra, GPT-6 Astra, Sol, and Luna, and GPT-5.6 Sol, Terra, and Luna, including cache writes, Fast/Priority, Flex, and long-context charges. OpenAI offers no Flex tier for the GPT-6.2 models, so a Flex event on them gets no estimate. Ultrafast is priced for GPT-6 Astra, GPT-6.1 Sol, and GPT-6.2 Astra and Sol only (6x for the first two, 3x for GPT-6.2 until 2027-02-28); other models stay unsupported. Earlier supported models retain their existing rules. Unknown internal aliases remain unsupported. Claude Fable 5.2 and 5.1, Opus 5.5, Opus 5, Sonnet 5.5, and Sonnet 5 are included in the price catalog; Claude request costs retain their native values without an extra promotion multiplier. Sonnet 5.5 and Sonnet 5 both cost $2/$10 per million input/output tokens.

Prices are loaded at exporter startup and interpolated when the collector starts. After a pricing update, rebuild the pricing exporter first, wait for it to become healthy, then force-recreate the collector to load the new files. A normal Compose update does not recreate a container just because a bind-mounted file changed. Historical request records retain the prices captured at ingestion.

Important

These values are estimates, not invoices. Provider pricing, service tiers, long-context multipliers, promotions, and model aliases change. The collector stores a pricing_as_of attribute so old telemetry remains interpretable.

Privacy and security

Prompt logging is disabled in all examples. That does not make arbitrary telemetry harmless: model names, session identifiers, tool activity, errors, and trace attributes can still disclose operational context.

  • Use a unique, random bearer token.
  • Keep .env and client configuration files containing secrets out of Git.
  • Restrict Grafana and storage backends to trusted networks.
  • Review emitted fields before increasing retention or adding users.
  • Do not enable prompt, tool-detail, or raw-body logging without an explicit retention and access policy.
  • Treat Grafana screenshots as data exports; inspect them before publication.

The collector's bearer token protects ingestion, not Grafana. Grafana uses its own administrator password.

Remote agents

Remote agent setup

The default Compose file is deliberately local-only. For remote agents, put a TLS-terminating reverse proxy in front of collector port 4318, expose only that port, and keep VictoriaMetrics, VictoriaLogs, and VictoriaTraces private. A private overlay network or firewall allowlist should limit who can reach the proxy.

Change client endpoints to https://otel.example.com while retaining the signal paths. Do not expose unauthenticated OTLP over plain HTTP.

Verification

Static checks:

python3 scripts/release.py
python3 -m unittest discover -s scripts/tests
python3 -m unittest discover -s git-ai-exporter/tests
docker compose config -q
jq empty grafana/dashboards/*.json extras/credit-resets/*.json
uv --directory pricing-exporter sync --frozen --python 3.13
uv --directory pricing-exporter run --frozen python -m unittest discover -s tests
Runtime smoke test

Runtime smoke test after docker compose up -d --build:

set -a
. ./.env
set +a
python3 scripts/smoke.py

The smoke test checks OTLP authentication, all three signal submissions, Grafana dashboard provisioning, log and metric ingestion, fresh trace retrieval, and datasource registration. It also exercises model pricing, cache writes, Fast/Priority/Flex, long context, native-cost preservation, and Claude dashboard queries through Grafana's authenticated proxy. It writes only synthetic records named ai-cli-observability-smoke. Run it against a test stack: its synthetic requests deliberately include nonzero costs.

Stop containers without deleting data:

docker compose down

Caution

docker compose down -v permanently deletes this stack's local telemetry and Grafana state.

Related projects

For Codex quotas and recent sessions in your GNOME panel, see Codex Usage Monitor.

For Claude Code, Claude Usage Tracker for Linux adds desktop quota monitoring and a customizable terminal statusline.

Reference documentation

License

Apache License 2.0. See LICENSE.

About

Self-hosted OpenTelemetry observability stack for Codex, Claude Code, and Gemini CLI, with Grafana dashboards for usage, tokens, estimated costs, latency, errors, tool activity, logs, metrics, and traces.

Topics

Resources

Stars

6 stars

Watchers

0 watching

Forks

Contributors

Languages