ClickHouse usage analytics: events/gauges tables with daily MV

[lohanidamodar]

Summary

Complete rewrite of the usage analytics library with a two-table architecture optimized for both real-time analytics and billing.

Architecture

• Events table (MergeTree) — raw request events with dedicated columns for path, method, status, resource, resourceId, country (LowCardinality), userAgent
• Gauges table (MergeTree) — simple resource snapshots (metric, value, time, tags)
• Daily MV (SummingMergeTree) — pre-aggregates events by metric + tenant + day for fast billing
• No periods — query-time aggregation via toStartOfHour/toStartOfDay instead of write-time fan-out
• Single write path — collect(metric, value, type, tags) routes to correct table; event columns auto-extracted from tags

Key Changes

• Two separate tables instead of one — events have 7 extra columns gauges don't need
• Plain MergeTree for both tables — raw appends, query-time aggregation
• Daily MV with minimal schema (metric, value, time, tenant)
• LowCardinality(String) for country column
• Bloom filter indexes on all filterable columns
• String tenant — setTenant(?string)
• Utopia Query for all read operations — parameterized queries, no SQL injection risk

API

Write

• collect(metric, value, type, tags) — buffer with auto-flush
• addBatch(metrics, type) — direct batch insert
• flush() — write buffered metrics

Read

• find(queries, type) / count(queries, type) / sum(queries, attr, type)
• getTotal(metric, queries, type) — SUM for events, argMax for gauges
• getTotalBatch(metrics, queries, type) — batch totals
• getTimeSeries(metrics, interval, start, end, queries, zeroFill, type)

Billing (Daily MV)

• findDaily(queries) / sumDaily(queries) / sumDailyBatch(metrics, queries)

Test Plan

• Unit tests for Metric schema, getters, validation
• Integration tests for ClickHouse and Database adapters
• PHPStan level max passing
• Linter passing
• Security audit — no SQL injection vulnerabilities
• CI green (CodeQL, Tests, Linter)

[loks0n]

This seems specific to Database adapter, should we move it to Databases.php?

[loks0n]

We're going to need a separate HttpLog.php type anyways, do we need these properties?

[greptile-apps]

Cross-type attribute fallback passes invalid columns to gauge queries

validateAttributeName() falls back to checking the opposite type's attributes. When find($queries, null) (or count/sum with $type = null) is called with event-specific filters like Query::equal('path', '/v1/'), parseQueries forwards the same $queries to both tables. For the gauge call, validateAttributeName('path', 'gauge') passes because it finds path in the event schema, generating WHERE \path` = {param:String}against the gauges table — which has no such column — causing a ClickHouseUNKNOWN_IDENTIFIER` server error at runtime.

The cross-type fallback should be removed so that an attribute is only valid if it exists in the target table's own schema:

// Remove lines 1138-1144 (the "Also check the other type's attributes" block)
throw new Exception("Invalid attribute name: {$attributeName}");

Any cross-table query with type-specific filters should require the caller to pass an explicit $type.

[greptile-apps]

purge() leaves stale aggregates in the daily MV table

DELETE FROM on the events and gauges tables does not cascade to the _events_daily SummingMergeTree table populated by the materialized view. After purge(), the daily table still contains all previously aggregated rows. Immediate re-insertion of the same metrics will double-count in the billing table, and any billing query against _events_daily will return inflated totals. This also means every test teardown (which calls purge()) leaves residual daily-table state that accumulates across runs.

[greptile-apps]

purge() ignores $type parameter, deletes all records regardless of type

The $type argument is not forwarded into convertQueriesToDatabase($queries) and no type discriminator is injected into $dbQueries. A call to purge([], Usage::TYPE_EVENT) will delete both events and gauges indiscriminately. Since addBatch() does not persist a type field in each document, the fix requires both write and read/delete paths to be aligned.

[greptile-apps]

INSERT retry creates duplicate rows in MergeTree

insert() wraps ClickHouse HTTP inserts in executeWithRetry(), which retries on HTTP 408/503/504 and on messages containing "timeout" or "connection". When ClickHouse commits the first attempt but the HTTP response is lost in transit (e.g. a proxy timeout), the retry re-sends the same payload. Since plain MergeTree does not deduplicate, every successful-but-unacknowledged attempt produces a permanent duplicate row — inflating event counts, gauge snapshots, and the _events_daily SummingMergeTree aggregates.

Consider one of the following mitigations:

• Set ClickHouse insert_deduplicate = 1 and use deterministic block checksums (requires ReplicatedMergeTree).
• Use async_insert mode with wait_for_async_insert = 1, which deduplicates at the server side.
• Or disable retries for INSERT operations and let the caller decide on retry semantics:

// Only retry SELECT/DDL; never retry INSERT
$this->executeWithRetry(
    $operation,
    fn ($e, $code) => false,   // never retry inserts
    $buildException
);

[greptile-apps]

Cross-type attribute fallback sends invalid columns to gauge queries

validateAttributeName falls back to checking the opposite type's schema (lines 1139–1144). When find/count/sum is called with $type = null, parseQueries passes the same queries to both tables. A caller passing Query::equal('path', '/v1/') passes validation for the gauge table because path is found in the event schema, generating WHERE \path` = {param:String}against the gauges table — which has no such column — and causing a ClickHouseUNKNOWN_IDENTIFIER` server error at runtime.

Remove the fallback block so each attribute is only valid for its own table's schema:

Suggested change

	// Also check the other type's attributes for cross-table queries
	$otherType = $type === 'event' ? 'gauge' : 'event';
	foreach ($this->getAttributes($otherType) as $attribute) {
	if ($attribute['$id'] === $attributeName) {
	return true;
	}
	}
	throw new Exception("Invalid attribute name: {$attributeName}");
	}