Skip to content

Commit 8f23752

Browse files
kyleconroykyleconroy
authored
Add goldeneye, a module that generates and checks the dialect seeds (#4605)
1 parent 4be42c6 commit 8f23752

25 files changed

Lines changed: 1445 additions & 311 deletions

File tree

‎.github/workflows/gen.yml‎

Lines changed: 15 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
1-
name: sqlc-pg-gen
1+
name: gen
22
on:
33
workflow_dispatch:
44
jobs:
5-
gen:
6-
name: sqlc-pg-gen
7-
runs-on: ubuntu-22.04
5+
generate:
6+
name: generate dialects
7+
runs-on: ubuntu-24.04
88
services:
99
postgres:
10-
image: postgres:15.0-alpine
10+
image: postgres:16
1111
env:
1212
POSTGRES_USER: postgres
1313
POSTGRES_PASSWORD: postgres
@@ -20,20 +20,18 @@ jobs:
2020
- uses: actions/checkout@v7
2121
- uses: actions/setup-go@v7
2222
with:
23-
go-version-file: go.mod
23+
go-version-file: internal/goldeneye/go.mod
2424
check-latest: true
25-
- run: go build -o sqlc-pg-gen ./internal/tools/sqlc-pg-gen
26-
- run: mkdir -p gen/contrib
27-
- run: ./sqlc-pg-gen gen
25+
- run: go run ./cmd/goldeneye install clickhouse
26+
working-directory: internal/goldeneye
27+
- run: go run ./cmd/goldeneye generate
28+
working-directory: internal/goldeneye
2829
env:
29-
PG_USER: postgres
30-
PG_HOST: localhost
31-
PG_DATABASE: postgres
32-
PG_PASSWORD: postgres
33-
PG_PORT: ${{ job.services.postgres.ports['5432'] }}
30+
POSTGRESQL_SERVER_URI: postgres://postgres:postgres@localhost:${{ job.services.postgres.ports['5432'] }}/postgres?sslmode=disable
3431
- name: Save results
3532
uses: actions/upload-artifact@v7
3633
with:
37-
name: sqlc-pg-gen-results
38-
path: gen
39-
34+
name: dialects
35+
path: internal/engine/*/dialect
36+
- name: Fail if the committed dialects differ
37+
run: git add -N internal/engine && git diff --exit-code --stat -- internal/engine

‎CLAUDE.md‎

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,24 @@ SQLC_TEST_CORE=1 go test ./internal/endtoend -run 'TestReplay/core'
143143
Go aborts a test binary on panic, so a case that panics the core analyzer ends
144144
the run early. Run a subset to get past one (`-run 'TestReplay/core/^select'`).
145145

146+
### Dialect Checks
147+
148+
The dialect seeds under `/internal/engine/<engine>/dialect/` are generated
149+
from a live database by `/internal/goldeneye`, a nested module, and its tests
150+
verify the committed files against one byte for byte. Engines whose database
151+
is not available skip.
152+
153+
```bash
154+
cd internal/goldeneye
155+
go run ./cmd/goldeneye install clickhouse # download the pinned clickhouse binary once
156+
POSTGRESQL_SERVER_URI="postgres://postgres:postgres@127.0.0.1:5432/postgres?sslmode=disable" go test ./...
157+
go run ./cmd/goldeneye generate postgresql # rewrite the files after a change
158+
```
159+
160+
The checks are not part of CI. The `gen` workflow
161+
(`.github/workflows/gen.yml`) generates the files on demand and uploads
162+
them as an artifact.
163+
146164
### Example Tests
147165

148166
- **Location:** `/examples/` directory
@@ -216,10 +234,13 @@ MYSQL_SERVER_URI="root:mysecretpassword@tcp(127.0.0.1:3306)/mysql?multiStatement
216234
- `/dolphin/` - MySQL parser (uses TiDB parser)
217235
- `/sqlite/` - SQLite parser
218236
- `/duckdb/` - DuckDB 2.0 parser (uses darkwing, the pure Go port of
219-
DuckDB's PEG parser); its dialect seeds are generated by
220-
`/internal/tools/sqlc-duckdb-gen` from a live DuckDB CLI
237+
DuckDB's PEG parser)
221238
- `<engine>/dialect/` - The engine's type system and standard library, as
222-
JSONL read by `/internal/core/seed`
239+
JSONL read by `/internal/core/seed`; the generated parts come from
240+
`/internal/goldeneye`
241+
- `/internal/goldeneye/` - Nested module that generates the dialect seeds
242+
under `/internal/engine/<engine>/dialect/` from a live database and checks
243+
the committed ones against it, one package per engine; see its README
223244
- `/internal/core/` - The analysis core: catalog, analyzer and dialect seeds
224245
- `/internal/compiler/` - Query compilation logic
225246
- `/internal/codegen/` - Code generation for different languages

‎Makefile‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
.PHONY: build build-endtoend test test-ci test-examples test-endtoend start psql mysqlsh proto
1+
.PHONY: build build-endtoend test test-ci test-examples test-endtoend test-goldeneye start psql mysqlsh proto
22

33
build:
44
go build ./...
@@ -26,8 +26,11 @@ test-ci: test-examples build-endtoend vet
2626
sqlc-dev:
2727
go build -o ~/bin/sqlc-dev ./cmd/sqlc/
2828

29-
sqlc-pg-gen:
30-
go build -o ~/bin/sqlc-pg-gen ./internal/tools/sqlc-pg-gen
29+
goldeneye:
30+
cd ./internal/goldeneye && go build -o ~/bin/goldeneye ./cmd/goldeneye
31+
32+
test-goldeneye:
33+
cd ./internal/goldeneye && go test ./...
3134

3235
sqlc-gen-json:
3336
go build -o ~/bin/sqlc-gen-json ./cmd/sqlc-gen-json

‎internal/engine/clickhouse/dialect/types.jsonl‎

Lines changed: 66 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -1,40 +1,66 @@
1-
{"name": "UInt8", "category": "N"}
2-
{"name": "UInt16", "category": "N"}
3-
{"name": "UInt32", "category": "N"}
4-
{"name": "UInt64", "category": "N"}
5-
{"name": "UInt128", "category": "N"}
6-
{"name": "UInt256", "category": "N"}
7-
{"name": "Int8", "category": "N"}
8-
{"name": "Int16", "category": "N"}
9-
{"name": "Int32", "category": "N"}
10-
{"name": "Int64", "category": "N"}
11-
{"name": "Int128", "category": "N"}
12-
{"name": "Int256", "category": "N"}
13-
{"name": "Float32", "category": "N"}
14-
{"name": "Float64", "category": "N"}
15-
{"name": "BFloat16", "category": "N"}
16-
{"name": "Decimal", "category": "N"}
17-
{"name": "Decimal32", "category": "N"}
18-
{"name": "Decimal64", "category": "N"}
19-
{"name": "Decimal128", "category": "N"}
20-
{"name": "Decimal256", "category": "N"}
21-
{"name": "Bool", "category": "B"}
22-
{"name": "String", "category": "S"}
23-
{"name": "FixedString", "category": "S"}
24-
{"name": "UUID", "category": "S"}
25-
{"name": "Date", "category": "D"}
26-
{"name": "Date32", "category": "D"}
27-
{"name": "DateTime", "category": "D"}
28-
{"name": "DateTime64", "category": "D"}
29-
{"name": "IPv4", "category": "S"}
30-
{"name": "IPv6", "category": "S"}
31-
{"name": "JSON", "category": "U"}
32-
{"name": "Enum8", "category": "U"}
33-
{"name": "Enum16", "category": "U"}
34-
{"name": "Nullable", "category": "U"}
35-
{"name": "LowCardinality", "category": "U"}
36-
{"name": "Array", "category": "A"}
37-
{"name": "Map", "category": "U"}
38-
{"name": "Tuple", "category": "U"}
39-
{"name": "Nested", "category": "U"}
40-
{"name": "Nothing", "category": "U"}
1+
{"name":"AggregateFunction","category":"U"}
2+
{"name":"Array","category":"A"}
3+
{"name":"BFloat16","category":"N"}
4+
{"name":"Bool","category":"B","aliases":["boolean"]}
5+
{"name":"Date","category":"D"}
6+
{"name":"Date32","category":"D"}
7+
{"name":"DateTime","category":"D","aliases":["TIMESTAMP"]}
8+
{"name":"DateTime32","category":"D"}
9+
{"name":"DateTime64","category":"D"}
10+
{"name":"Decimal","category":"N","aliases":["DEC","FIXED","NUMERIC"]}
11+
{"name":"Decimal128","category":"N"}
12+
{"name":"Decimal256","category":"N"}
13+
{"name":"Decimal32","category":"N"}
14+
{"name":"Decimal64","category":"N"}
15+
{"name":"Dynamic","category":"U"}
16+
{"name":"Enum","category":"U"}
17+
{"name":"Enum16","category":"U"}
18+
{"name":"Enum8","category":"U"}
19+
{"name":"FixedString","category":"S","aliases":["BINARY"]}
20+
{"name":"Float32","category":"N","aliases":["FLOAT","REAL","SINGLE"]}
21+
{"name":"Float64","category":"N","aliases":["DOUBLE","DOUBLE PRECISION"]}
22+
{"name":"IPv4","category":"S","aliases":["INET4"]}
23+
{"name":"IPv6","category":"S","aliases":["INET6"]}
24+
{"name":"Int128","category":"N"}
25+
{"name":"Int16","category":"N","aliases":["SMALLINT","SMALLINT SIGNED"]}
26+
{"name":"Int256","category":"N"}
27+
{"name":"Int32","category":"N","aliases":["INT","INT SIGNED","INTEGER","INTEGER SIGNED","MEDIUMINT","MEDIUMINT SIGNED"]}
28+
{"name":"Int64","category":"N","aliases":["BIGINT","BIGINT SIGNED","SIGNED"]}
29+
{"name":"Int8","category":"N","aliases":["BYTE","INT1","INT1 SIGNED","TINYINT","TINYINT SIGNED"]}
30+
{"name":"IntervalDay","category":"T"}
31+
{"name":"IntervalHour","category":"T"}
32+
{"name":"IntervalMicrosecond","category":"T"}
33+
{"name":"IntervalMillisecond","category":"T"}
34+
{"name":"IntervalMinute","category":"T"}
35+
{"name":"IntervalMonth","category":"T"}
36+
{"name":"IntervalNanosecond","category":"T"}
37+
{"name":"IntervalQuarter","category":"T"}
38+
{"name":"IntervalSecond","category":"T"}
39+
{"name":"IntervalWeek","category":"T"}
40+
{"name":"IntervalYear","category":"T"}
41+
{"name":"JSON","category":"U"}
42+
{"name":"LineString","category":"U"}
43+
{"name":"LowCardinality","category":"U"}
44+
{"name":"Map","category":"U"}
45+
{"name":"MultiLineString","category":"U"}
46+
{"name":"MultiPolygon","category":"U"}
47+
{"name":"Nested","category":"U"}
48+
{"name":"Nothing","category":"U"}
49+
{"name":"Nullable","category":"U"}
50+
{"name":"Object","category":"U"}
51+
{"name":"Point","category":"U"}
52+
{"name":"Polygon","category":"U"}
53+
{"name":"Ring","category":"U"}
54+
{"name":"SimpleAggregateFunction","category":"U"}
55+
{"name":"String","category":"S","aliases":["BINARY LARGE OBJECT","BINARY VARYING","BLOB","BYTEA","CHAR","CHAR LARGE OBJECT","CHAR VARYING","CHARACTER","CHARACTER LARGE OBJECT","CHARACTER VARYING","CLOB","GEOMETRY","LONGBLOB","LONGTEXT","MEDIUMBLOB","MEDIUMTEXT","NATIONAL CHAR","NATIONAL CHAR VARYING","NATIONAL CHARACTER","NATIONAL CHARACTER LARGE OBJECT","NATIONAL CHARACTER VARYING","NCHAR","NCHAR LARGE OBJECT","NCHAR VARYING","NVARCHAR","TEXT","TINYBLOB","TINYTEXT","VARBINARY","VARCHAR","VARCHAR2"]}
56+
{"name":"Time","category":"D"}
57+
{"name":"Time64","category":"D"}
58+
{"name":"Tuple","category":"U"}
59+
{"name":"UInt128","category":"N"}
60+
{"name":"UInt16","category":"N","aliases":["SMALLINT UNSIGNED","YEAR"]}
61+
{"name":"UInt256","category":"N"}
62+
{"name":"UInt32","category":"N","aliases":["INT UNSIGNED","INTEGER UNSIGNED","MEDIUMINT UNSIGNED"]}
63+
{"name":"UInt64","category":"N","aliases":["BIGINT UNSIGNED","BIT","SET","UNSIGNED"]}
64+
{"name":"UInt8","category":"N","aliases":["INT1 UNSIGNED","TINYINT UNSIGNED"]}
65+
{"name":"UUID","category":"S"}
66+
{"name":"Variant","category":"U"}

‎internal/engine/clickhouse/seed.go‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,15 @@ import (
77
"github.com/sqlc-dev/sqlc/internal/core/seed"
88
)
99

10+
// The dialect directory describes ClickHouse's type system. types.jsonl is
11+
// generated from system.data_type_families of the pinned ClickHouse release
12+
// by goldeneye (internal/goldeneye), which also checks it against one;
13+
// dialect.json and functions.jsonl are authored by hand, since ClickHouse
14+
// publishes no function signatures. Regenerate from internal/goldeneye with:
15+
//
16+
// go run ./cmd/goldeneye install clickhouse
17+
// go run ./cmd/goldeneye generate clickhouse
18+
//
1019
//go:embed dialect
1120
var dialectFS embed.FS
1221

‎internal/engine/duckdb/seed.go‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,10 @@ import (
99

1010
// The dialect directory describes DuckDB's type system. types.jsonl,
1111
// functions.jsonl and operators.jsonl are generated from a live DuckDB 2.0
12-
// CLI by sqlc-duckdb-gen (internal/tools/sqlc-duckdb-gen); dialect.json is
13-
// authored by hand. Regenerate with:
12+
// CLI by goldeneye (internal/goldeneye), which also checks them against one;
13+
// dialect.json is authored by hand. Regenerate from internal/goldeneye with:
1414
//
15-
// DUCKDB=/path/to/duckdb go run ./internal/tools/sqlc-duckdb-gen
15+
// DUCKDB=/path/to/duckdb go run ./cmd/goldeneye generate duckdb
1616
//
1717
//go:embed dialect
1818
var dialectFS embed.FS

‎internal/engine/postgresql/catalog.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ func NewCatalog() *catalog.Catalog {
1616
}
1717

1818
// genPGCatalog and genInformationSchema build the schemas sqlc knows
19-
// PostgreSQL by. Both read the dialect directory, which sqlc-pg-gen writes,
19+
// PostgreSQL by. Both read the dialect directory, which goldeneye writes,
2020
// and which the analysis core seeds its catalog from as well.
2121
func genPGCatalog() *catalog.Schema {
2222
return systemSchema("pg_catalog", pgCatalogFuncs())

‎internal/engine/postgresql/dialect/functions.jsonl‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1695,11 +1695,6 @@
16951695
{"name":"pg_export_snapshot","returns":"text"}
16961696
{"name":"pg_extension_config_dump","args":[{"type":"regclass"},{"type":"text"}],"returns":"void"}
16971697
{"name":"pg_extension_update_paths","args":[{"name":"name","type":"name"}],"returns":"record"}
1698-
{"name":"pg_file_rename","args":[{"type":"text"},{"type":"text"}],"returns":"boolean"}
1699-
{"name":"pg_file_rename","args":[{"type":"text"},{"type":"text"},{"type":"text"}],"returns":"boolean"}
1700-
{"name":"pg_file_sync","args":[{"type":"text"}],"returns":"void"}
1701-
{"name":"pg_file_unlink","args":[{"type":"text"}],"returns":"boolean"}
1702-
{"name":"pg_file_write","args":[{"type":"text"},{"type":"text"},{"type":"boolean"}],"returns":"bigint"}
17031698
{"name":"pg_filenode_relation","args":[{"type":"oid"},{"type":"oid"}],"returns":"regclass"}
17041699
{"name":"pg_function_is_visible","args":[{"type":"oid"}],"returns":"boolean"}
17051700
{"name":"pg_get_backend_memory_contexts","returns":"record"}
@@ -1772,7 +1767,6 @@
17721767
{"name":"pg_lock_status","returns":"record"}
17731768
{"name":"pg_log_backend_memory_contexts","args":[{"type":"integer"}],"returns":"boolean"}
17741769
{"name":"pg_log_standby_snapshot","returns":"pg_lsn"}
1775-
{"name":"pg_logdir_ls","returns":"record"}
17761770
{"name":"pg_logical_emit_message","args":[{"type":"boolean"},{"type":"text"},{"type":"bytea"}],"returns":"pg_lsn"}
17771771
{"name":"pg_logical_emit_message","args":[{"type":"boolean"},{"type":"text"},{"type":"text"}],"returns":"pg_lsn"}
17781772
{"name":"pg_logical_slot_get_binary_changes","args":[{"name":"slot_name","type":"name"},{"name":"upto_lsn","type":"pg_lsn"},{"name":"upto_nchanges","type":"integer"},{"name":"options","type":"text[]","mode":"v","has_default":true}],"returns":"record"}

‎internal/engine/postgresql/extension.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ import (
99
)
1010

1111
// loadExtension reads the named extension's function list from the dialect
12-
// directory, where sqlc-pg-gen writes one directory per extension. An
12+
// directory, where goldeneye writes one directory per extension. An
1313
// extension sqlc does not know is nil, which CREATE EXTENSION treats as
1414
// nothing to add.
1515
func loadExtension(name string) *catalog.Schema {

‎internal/engine/postgresql/seed.go‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,11 @@ import (
1717
// "WHERE bigint_col = 1" depends on.
1818
//
1919
// functions.jsonl and relations.jsonl are pg_catalog's functions and the
20-
// tables and views of pg_catalog and information_schema, all written by
21-
// sqlc-pg-gen. Both the analysis core and the catalog the legacy compiler
22-
// builds read them. Each contrib extension sqlc knows is a directory under
23-
// extensions/ holding the types and functions CREATE EXTENSION adds.
20+
// tables and views of pg_catalog and information_schema, all generated from
21+
// a live PostgreSQL 16 server by goldeneye (internal/goldeneye), which also
22+
// checks them against one. Both the analysis core and the catalog the legacy
23+
// compiler builds read them. Each contrib extension sqlc knows is a directory
24+
// under extensions/ holding the types and functions CREATE EXTENSION adds.
2425
//
2526
//go:embed dialect
2627
var dialectFS embed.FS

0 commit comments

Comments
 (0)