diff --git a/README.md b/README.md index 66ba69e..26d25d5 100644 --- a/README.md +++ b/README.md @@ -22,10 +22,7 @@ for SQL; 4× QPF is recommended for concurrent query performance. - [Choose an interface](#choose-an-interface) - [Integration paths](#integration-paths) - [Get a server running](#get-a-server-running) -- [Client examples](#client-examples) -- [Tested integration coverage](#tested-integration-coverage) - [Data tools](#data-tools) -- [Feature reference](#feature-reference) - [Documentation](#documentation) ## Choose an interface @@ -43,6 +40,12 @@ values without converting them to SQL rows. You can combine the two: upload a dataset, then ingest selected data into a SQL table. See the [dataset-to-SQL workflow](docs/dataset-sql-workflow.md). +Flight SQL supports queries, updates, prepared statements, bulk ingestion, +metadata, sessions and read-only query cancellation. SQL uses autocommit; +transactions and savepoints are unsupported. See the +[protocol capabilities](docs/flight-capabilities.md) and +[SQL reference](docs/sql-reference.md) for supported commands and limits. + ## Integration paths All three paths connect to the same Flight endpoint. Flight SQL defines SQL @@ -55,14 +58,14 @@ commands using Flight RPC; gRPC carries the requests and Arrow streams. | **Native Flight RPC** | Named dataset upload/download, discovery and custom actions | Python `pyarrow.flight`, Go Flight clients and other Flight SDKs | SDKs with Flight SQL support can also issue SQL commands directly, as the Go -tests do. Driver support and application validation are recorded separately in -[tested integration coverage](#tested-integration-coverage) below. +tests do. Driver support and application validation are recorded separately in the +[client compatibility matrix](docs/client-compatibility.md). ## Get a server running -[Run locally from source](docs/local-development.md) for prerequisites, Docker -Compose commands, source builds and the Python tour. For published binaries and -container images, follow [Install a release](#install-a-release). +[Install the latest release](#install-a-release) with Docker or a binary. For +Docker Compose, source builds and the Python tour, +[run locally from source](docs/local-development.md). See [build profiles](docs/build-profiles.md) for feature availability and [startup and readiness](docs/build-profiles.md#startup-and-readiness) before @@ -70,208 +73,98 @@ using a deployment health check. ### Install a release -You need a running Redis instance. For SQL, use Redis with **Query Engine -(Search)** enabled; we recommend **4× QPF** where supported. The SQL interface -uses the Redis ADBC driver's opinionated data model; -see [Redis requirements](#choose-an-interface). These commands start the Flight -server only. - -Choose a published tag from [Releases](https://github.com/redis/redis-apache-arrow-flight/releases). -Replace `vX.Y.Z` below with that tag. If no release is available for your -platform, [build from source](docs/local-development.md). - -```bash -VERSION=vX.Y.Z -``` +Use the [latest release](https://github.com/redis/redis-apache-arrow-flight/releases/latest). +You need a running Redis instance; SQL requires **Query Engine (Search)**. +We recommend **4× QPF** where supported. These commands start the Flight server +on port **8815**. -Choose **one** of the following options. - -#### Option 1: Download and run a binary - -Set `PLATFORM` to `linux-amd64`, `linux-arm64` or `darwin-arm64` (Apple Silicon -macOS), matching your machine. For example: - -```bash -PLATFORM=linux-amd64 -ARCHIVE="redis-flight-server-${VERSION}-${PLATFORM}" -curl -fLO "https://github.com/redis/redis-apache-arrow-flight/releases/download/${VERSION}/${ARCHIVE}.tar.gz" -curl -fLO "https://github.com/redis/redis-apache-arrow-flight/releases/download/${VERSION}/${ARCHIVE}.tar.gz.sha256" -shasum -a 256 -c "${ARCHIVE}.tar.gz.sha256" -``` +#### Docker -After the checksum passes, extract the archive and start the server. This -example connects to Redis running locally on port 6379: +With Docker Desktop and Redis running on your host: ```bash -tar -xzf "${ARCHIVE}.tar.gz" -"./${ARCHIVE}/redis-flight-server" -redis-uri redis://localhost:6379/0 -``` - -#### Option 2: Run a container - -The image tag is the release version without its leading `v`; `${VERSION#v}` -performs that conversion. With Docker Desktop and Redis running on your host: - -```bash -docker run --rm -p 8815:8815 \ +docker run --rm --pull=always -p 8815:8815 \ -e REDIS_URI=redis://host.docker.internal:6379/0 \ - "ghcr.io/redis/redis-apache-arrow-flight:${VERSION#v}" + ghcr.io/redis/redis-apache-arrow-flight:latest ``` +`--pull=always` fetches the latest published image each time you run the command. With Docker Engine on Linux, put Redis and Flight on the same Docker network and use the Redis container's name in `REDIS_URI` instead of `host.docker.internal`. -Both options expose Flight on port **8815**. Keep the server running while -trying the [client examples](#client-examples). - -## Client examples - -With a server listening on `localhost:8815`, use these addresses for local -connections without TLS: - -```text -Flight / ADBC: grpc://localhost:8815 -JDBC: jdbc:arrow-flight-sql://localhost:8815/?useEncryption=false -``` - -### Flight SQL from Python (ADBC) +#### Binary (macOS or Linux) -```python -import adbc_driver_flightsql.dbapi as flight_sql -import pyarrow as pa +Run this in a directory where you want to download the server. It detects your +platform, resolves the latest release, verifies its checksum and starts it +against local Redis. Requires `curl`, `tar` and `shasum`. -with flight_sql.connect("grpc://localhost:8815", autocommit=True) as conn, conn.cursor() as cur: - cur.adbc_ingest("orders", pa.table({"id": [1, 2, 3], "country": ["JPN", "USA", "JPN"]}), mode="replace") - cur.execute("SELECT country, COUNT(*) AS n FROM orders WHERE id >= ? GROUP BY country", parameters=(2,)) - print(cur.fetch_arrow_table()) -``` - -SQL uses autocommit; transactions are unsupported. Parameters may use `?` or -`$1`. The SQL dialect is documented by the -[Redis ADBC driver](https://github.com/redis/redis-adbc-driver#supported-sql). - -### Datasets from Python (pyarrow.flight) - -```python -import pyarrow as pa -import pyarrow.flight as flight - -client = flight.connect("grpc://localhost:8815") -table = pa.table({"id": [1, 2], "embedding": [[0.1, 0.2], [0.3, 0.4]]}) -desc = flight.FlightDescriptor.for_path("features", "v1") - -writer, results = client.do_put(desc, table.schema) -writer.write_table(table) -writer.done_writing() -results.read() # Consume the upload acknowledgement. -writer.close() - -info = client.get_flight_info(desc) -received = pa.concat_tables([ - client.do_get(endpoint.ticket).read_all() for endpoint in info.endpoints -]) -print(received) +```bash +( + set -eu + case "$(uname -s)-$(uname -m)" in + Darwin-arm64) PLATFORM=darwin-arm64 ;; + Linux-x86_64) PLATFORM=linux-amd64 ;; + Linux-aarch64|Linux-arm64) PLATFORM=linux-arm64 ;; + *) echo "No published binary for this platform; use Docker or build from source." >&2; exit 1 ;; + esac + RELEASE_URL=$(curl -fsSL -o /dev/null -w '%{url_effective}' \ + https://github.com/redis/redis-apache-arrow-flight/releases/latest) + VERSION=${RELEASE_URL##*/} + ARCHIVE="redis-flight-server-${VERSION}-${PLATFORM}" + DOWNLOAD="https://github.com/redis/redis-apache-arrow-flight/releases/download/${VERSION}" + curl -fLO "${DOWNLOAD}/${ARCHIVE}.tar.gz" + curl -fLO "${DOWNLOAD}/${ARCHIVE}.tar.gz.sha256" + shasum -a 256 -c "${ARCHIVE}.tar.gz.sha256" + tar -xzf "${ARCHIVE}.tar.gz" + "./${ARCHIVE}/redis-flight-server" -redis-uri redis://localhost:6379/0 +) ``` -Read every endpoint in the returned FlightInfo. For column/batch selection, -discovery, deletion and storage behavior, see the -[dataset reference](docs/data-access.md#datasets) and -[full Python example](docs/client-guide.md#datasets-from-python-pyarrowflight). - -### JDBC (DBeaver and others) - -Use `org.apache.arrow:flight-sql-jdbc-driver` with the JDBC URL above. Arrow -requires JVM options: Java 17+ needs -`--add-opens=java.base/java.nio=ALL-UNNAMED`; Java 23+ also needs -`--sun-misc-unsafe-memory-access=allow`. - -Follow the [JDBC setup guide](docs/client-guide.md#jdbc-dbeaver-and-others) for -authentication, TLS, session behavior and known driver limitations. For DBeaver, -follow the [driver setup and manual test guide](docs/dbeaver.md), including the -direct JDBC 19.0.0 JAR download and a screenshot walkthrough. Manual connection, -table navigation and a three-row SELECT passed on October 30, 2026; earlier -macOS accessibility automation crashes are recorded separately. - -### Other clients - -Go clients use `flightsql.NewClient` for SQL and -`flight.NewClientWithMiddleware` for datasets. See the -[client guide](docs/client-guide.md#other-clients), -[Python tour](examples/python/quickstart.py) and -[Go integration tests](server/integration_test.go). - -## Tested integration coverage - -All three integration paths have recorded passing checks. These are historical -results, with versions, dates and limitations in the -[client compatibility matrix](docs/client-compatibility.md). - -| Path | Tested client | Recorded passing coverage | Evidence | -|---|---|---|---| -| Flight SQL via ADBC | Python driver/manager 1.12.0 | Metadata, parameters, queries/updates, bulk ingestion and dataset-to-SQL import | [Python tour](examples/python/quickstart.py), [import example](examples/python/dataset_to_sql.py) | -| Flight SQL via JDBC | Flight SQL JDBC 19.0.0 on Java 25 | Metadata, prepared parameters, updates, sessions and concurrent statement/result reads | [JDBC smoke test](examples/jdbc/JdbcSmoke.java) | -| Native Flight RPC | PyArrow 26.0.0 | Upload/download, nested and dictionary types, multiple batches, column/batch selection, discovery and drop | [Python tour](examples/python/quickstart.py), [import example](examples/python/dataset_to_sql.py) | -| Native Flight RPC / Flight SQL SDK | Go arrow-go 18.8.0 | Datasets, SQL metadata, parameters, ingestion, scans, sessions, cancellation and cleanup; standalone/Cluster race suites | [Go integration tests](server/integration_test.go) and related server tests | - -Initial validation was recorded on October 20, 2026, with a Redis ADBC -v0.0.17 follow-up on October 22. On October 29, both Go race suites, both Python -examples and the JDBC smoke test passed again. Each Go suite had 80 passing -tests and one enterprise-only skip. CI is configured for Java 17/21/25; the -recorded local JDBC runs cover Java 25. See the -[core validation record](docs/client-compatibility.md#core-endpoint-validation-on-2026-10-29). -No integration suites were rerun for this documentation rewrite. +Release archives and container images are published by the Release workflow +once its checks finish. If a new release's artifacts are still being built, +wait for publication or [build from source](docs/local-development.md). +For a different Redis host, change `REDIS_URI` in Docker or `-redis-uri` above. +Keep the server running, then choose a [data tool guide](#data-tools). ## Data tools Choose the Flight SQL driver or native Flight API required by your workflow. A tool's generic Redis or PostgreSQL connector does not connect to this endpoint. SQL tools access SQL tables; named datasets use the Flight dataset API. +Each tool links to its setup guide; start with the +[shared prerequisites](docs/integrations/README.md). | Tool | Connection path | Tested status against this server | |---|---|---| -| Python / PyArrow | ADBC Flight SQL for SQL; `pyarrow.flight` for datasets | Both smoke scripts passed | -| Go | Flight SQL client or native Flight client | Integration/race suites passed | -| Java | Flight SQL JDBC or native Flight SDK | JDBC passed; native Java Flight untested | -| DBeaver | Generic Flight SQL JDBC 19.0.0 driver; [setup and manual tests](docs/dbeaver.md) | Community 26.2.3: manual connection, table navigation and three-row SELECT passed on 2026-10-30; [scope and limits](docs/client-compatibility.md#dbeaver-rerun-and-manual-validation-on-2026-10-30) | -| Other JDBC applications | Custom Flight SQL JDBC driver | Driver tested; individual applications untested | -| pandas | ADBC `pd.read_sql`, or Arrow `to_pandas()` | Reads, NULLs and empty results passed; ADBC `params=` unsupported | -| Polars | ADBC `pl.read_database`, or `pl.from_arrow()` | Reads, NULLs, empty results and batch iteration passed | -| DuckDB | Download an Arrow table, then query it locally | Local SQL over Flight SQL and Flight RPC Arrow results passed | -| R | `adbcdrivermanager` with the Flight SQL shared driver | DDL, writes, queries, NULLs and empty results passed; DBI/adbi untested | -| Apache Superset | [Version-specific guidance](docs/superset.md): direct Redis ADBC for 6.1; Flight SQL candidate for 7 | Tested Flight adapter blocked on 6.1; Superset 7 Flight SQL application validation pending | -| Tableau | Other Databases (JDBC) with Flight SQL JDBC | Desktop Free Edition 2026.3: connection, discovery and three-row preview (including NULL) passed; [read validation and scope](docs/validation/2026-11-01-tableau/README.md) | -| Metabase | Community Flight SQL plugin 0.8.0 with bundled JDBC 19.0.0 | Metabase 0.64.2.9 API reads, schema sync, generated queries and INSERT/UPDATE/DELETE Actions passed; [evidence and native-DDL limitation](docs/validation/2026-11-01-bi-jdbc/README.md) | -| Power BI | Custom ADBC Power Query connector | Untested; no connector shipped | -| Redis Insight | Connect directly to the backing Redis database | Untested; dataset payloads remain Arrow IPC | +| [Python / PyArrow](docs/integrations/python.md) | ADBC Flight SQL for SQL; `pyarrow.flight` for datasets | Both smoke scripts passed | +| [Go](docs/integrations/go.md) | Flight SQL client or native Flight client | Integration/race suites passed | +| [Java](docs/integrations/java.md) | Flight SQL JDBC or native Flight SDK | JDBC passed; native Java Flight untested | +| [DBeaver](docs/dbeaver.md) | Generic Flight SQL JDBC 19.0.0 driver | Community 26.2.3: manual connection, table navigation and three-row SELECT passed on 2026-10-30; [scope and limits](docs/client-compatibility.md#dbeaver-rerun-and-manual-validation-on-2026-10-30) | +| [Other JDBC applications](docs/integrations/jdbc-applications.md) | Custom Flight SQL JDBC driver | Driver tested; individual applications untested | +| [pandas](docs/integrations/pandas.md) | ADBC `pd.read_sql`, or Arrow `to_pandas()` | Reads, NULLs and empty results passed; ADBC `params=` unsupported | +| [Polars](docs/integrations/polars.md) | ADBC `pl.read_database`, or `pl.from_arrow()` | Reads, NULLs, empty results and batch iteration passed | +| [DuckDB](docs/integrations/duckdb.md) | Download an Arrow table, then query it locally | Local SQL over Flight SQL and Flight RPC Arrow results passed | +| [R](docs/integrations/r.md) | `adbcdrivermanager` with the Flight SQL shared driver | DDL, writes, queries, NULLs and empty results passed; DBI/adbi untested | +| [Apache Superset](docs/superset.md) | Direct Redis ADBC for 6.1; Flight SQL candidate for 7 | Tested Flight adapter blocked on 6.1; Superset 7 Flight SQL application validation pending | +| [Tableau](docs/integrations/tableau.md) | Other Databases (JDBC) with Flight SQL JDBC | Desktop Free Edition 2026.3: connection, discovery and three-row preview (including NULL) passed; [read validation and scope](docs/validation/2026-11-01-tableau/README.md) | +| [Metabase](docs/integrations/metabase.md) | Community Flight SQL plugin 0.8.0 with bundled JDBC 19.0.0 | Metabase 0.64.2.9 API reads, schema sync, generated queries and INSERT/UPDATE/DELETE Actions passed; [evidence and native-DDL limitation](docs/validation/2026-11-01-bi-jdbc/README.md) | +| [Power BI](docs/integrations/power-bi.md) | Custom ADBC Power Query connector | Untested; no connector shipped | +| [Redis Insight](docs/integrations/redis-insight.md) | Connect directly to the backing Redis database | Untested; dataset payloads remain Arrow IPC | The [October 29 client validation](docs/client-compatibility.md#flight-client-validation-on-2026-10-29) records dataframe and R results and application blockers. See the -[client compatibility matrix](docs/client-compatibility.md) for the scope of recorded checks and the [tool setup notes](docs/client-guide.md#data-tools) -for driver links and connector requirements. The +[client compatibility matrix](docs/client-compatibility.md) for versions, dates +and the scope of recorded checks. The [22-tool ADBC comparison](https://github.com/redis/redis-apache-arrow-flight/blob/main/docs/data-engineering-adbc-support.md) includes upstream Redis ADBC evidence; those results do not establish Flight server application compatibility. -## Feature reference - -### Flight SQL support - -The server supports queries and updates, prepared statements, bulk ingestion, -catalog/schema/table metadata, cookie sessions and explicit read-only query -cancellation. SQL runs in autocommit mode. Transactions, savepoints, Substrait -and bidirectional `DoExchange` are unsupported. - -Supported commands, optional features and errors are listed in the -[protocol capability matrix](docs/flight-capabilities.md). For prepared-handle -rules, ingestion options, statement execution and streaming limits, see the -[Flight SQL reference](docs/sql-reference.md). Streaming, pooling and other -capabilities depend on the [build profile](docs/build-profiles.md). - ## Documentation | Topic | Guide | |---|---| +| Data tool setup and examples | [Integration guides](docs/integrations/README.md) | | Local setup and source builds | [Run locally from source](docs/local-development.md) | | ADBC, native Flight and JDBC examples | [Client guide](docs/client-guide.md) | | DBeaver driver setup and manual tests | [DBeaver guide](docs/dbeaver.md) | diff --git a/docs/client-guide.md b/docs/client-guide.md index d8c3657..0bb77b6 100644 --- a/docs/client-guide.md +++ b/docs/client-guide.md @@ -58,8 +58,8 @@ with `--dataset-endpoint-batches`. Use the [Arrow Flight SQL JDBC driver](https://arrow.apache.org/java/current/flight_sql_jdbc_driver.html) (`org.apache.arrow:flight-sql-jdbc-driver`). The recorded local validation used -19.0.0 on Java 25; CI is configured for Java 17, 21 and 25. See -[tested integration coverage](../README.md#tested-integration-coverage). The URL is: +19.0.0 on Java 25; CI is configured for Java 17, 21 and 25. See the +[client compatibility matrix](client-compatibility.md). The URL is: ```text jdbc:arrow-flight-sql://localhost:8815/?useEncryption=false @@ -157,7 +157,7 @@ work with the results of either interface. | **Tableau** | Use [Other Databases (JDBC)](https://help.tableau.com/current/pro/desktop/en-gb/examples_otherdatabases_jdbc.htm) with Flight SQL JDBC 19.0.0 installed in Tableau's driver directory, then restart Tableau. Follow the [tested setup](validation/2026-11-01-tableau/README.md); this run needed no Tableau JVM configuration changes. Tableau's generic JDBC connector does not guarantee feature compatibility | Visual analysis and dashboards; requires driver setup and compatibility testing | Desktop Free Edition 2026.3 / JDBC 19.0.0: connection, catalog/schema/table/column discovery and three-row data preview passed. Read-only acceptance; worksheets, filters and aggregates remain untested. [Record](validation/2026-11-01-tableau/README.md) | | **Metabase** | Community [Flight SQL plugin](https://github.com/J0hnG4lt/metabase-flightsql-driver), with its Arrow JVM flags and a writable plugins directory. Enable Actions in connection details and database settings for writes | Native SQL and visual queries; custom SQL Actions for write-back | Metabase 0.64.2.9 / plugin 0.8.0: API reads, schema sync, generated queries and INSERT/UPDATE/DELETE Actions passed. Native DDL through the read-query endpoint failed; [validation](validation/2026-11-01-bi-jdbc/README.md) | | **Other JDBC applications** | Load `org.apache.arrow:flight-sql-jdbc-driver` and use the same JDBC URL, if the application supports custom JDBC drivers | Java applications and data tools that allow a custom driver | Driver only — Flight SQL JDBC 19.0.0 passed on Java 25; individual applications untested ([record](../docs/client-compatibility.md)). | -| **Python / PyArrow** | Use [ADBC Flight SQL](https://arrow.apache.org/adbc/current/driver/flight_sql.html) (`adbc_driver_flightsql`) for SQL and [`pyarrow.flight`](https://arrow.apache.org/docs/python/flight.html) for datasets; see [Client examples](../README.md#client-examples) | Queries, bulk ingest, Arrow table transfers and ML batches | Yes — ADBC Flight SQL and PyArrow Flight smoke scripts passed ([record](../intent/driver-v0.0.17/plan.md#validation)). | +| **Python / PyArrow** | Use [ADBC Flight SQL](https://arrow.apache.org/adbc/current/driver/flight_sql.html) (`adbc_driver_flightsql`) for SQL and [`pyarrow.flight`](https://arrow.apache.org/docs/python/flight.html) for datasets; see [Python integration guide](integrations/python.md) | Queries, bulk ingest, Arrow table transfers and ML batches | Yes — ADBC Flight SQL and PyArrow Flight smoke scripts passed ([record](../intent/driver-v0.0.17/plan.md#validation)). | | **pandas** | Pass an ADBC Flight SQL connection to [`pd.read_sql`](https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html) (pandas 2.2+), or convert a downloaded Arrow table with `table.to_pandas()` | Dataframe analysis and reporting | Yes — `pd.read_sql` results, NULLs and empty results passed; ADBC `params=` is unsupported ([record](../docs/client-compatibility.md#flight-client-validation-on-2026-10-29)). | | **Polars** | Pass an ADBC Flight SQL connection to [`pl.read_database`](https://docs.pola.rs/api/python/stable/reference/api/polars.read_database.html), or convert a downloaded Arrow table with `pl.from_arrow(table)` | Dataframe transformations and analytics | Yes — `pl.read_database`, NULLs, empty results and batch iteration passed. ([record](../docs/client-compatibility.md#flight-client-validation-on-2026-10-29)). | | **DuckDB** | Fetch an Arrow table through ADBC or Flight, then [query it locally in DuckDB](https://duckdb.org/docs/current/guides/python/import_arrow); this uses an Arrow transfer rather than a direct DuckDB connection to the server | Joining Redis results with local files or other Arrow data | Yes — local SQL over Arrow fetched through Flight SQL and Flight RPC passed. ([record](../docs/client-compatibility.md#flight-client-validation-on-2026-10-29)). | @@ -172,7 +172,7 @@ Flight / ADBC: grpc://localhost:8815 JDBC: jdbc:arrow-flight-sql://localhost:8815/?useEncryption=false ``` -See [tested integration coverage](../README.md#tested-integration-coverage) for the ADBC, +See the [client compatibility matrix](client-compatibility.md) for the ADBC, JDBC and native Flight checks. Tested status reflects recorded validation against this Flight server; the linked records give versions, dates, and scope. Tests of the Redis ADBC driver in the diff --git a/docs/data-engineering-adbc-support.md b/docs/data-engineering-adbc-support.md index 4ace670..dc40390 100644 --- a/docs/data-engineering-adbc-support.md +++ b/docs/data-engineering-adbc-support.md @@ -23,7 +23,7 @@ The latter does not imply that the tool can load the Redis driver. **Redis ADBC support** means loading this repository's driver locally. **Redis Arrow Flight SQL support** means connecting over the network to the -[Redis Arrow Flight server](https://github.com/alberttwong/redis-apache-arrow-flight#client-examples), +[Redis Arrow Flight server](https://github.com/redis/redis-apache-arrow-flight#data-tools), which runs the Redis ADBC driver on the server side. It requires a separate server deployment; it is not a built-in endpoint of Redis. Its Flight datasets API is a separate interface and does not establish Flight SQL compatibility. diff --git a/docs/integrations/README.md b/docs/integrations/README.md new file mode 100644 index 0000000..410a5a1 --- /dev/null +++ b/docs/integrations/README.md @@ -0,0 +1,83 @@ +# Data tool integration guides + +[Back to README data tools](../../README.md#data-tools). + +Start a [local server](../local-development.md) or +[install a release](../../README.md#install-a-release). SQL needs Redis with +Query Engine (Search); 4× QPF is recommended where supported. SQL uses the +Redis ADBC driver's [data model](../../README.md#choose-an-interface), and runs +in autocommit mode without transactions. Named Arrow datasets are separate +from SQL tables. + +Local examples assume an unauthenticated endpoint without TLS: + +```text +Flight / ADBC: grpc://localhost:8815 +JDBC: jdbc:arrow-flight-sql://localhost:8815/?useEncryption=false +``` + +Use an address reachable from the client. Inside a Docker Desktop container, +`host.docker.internal` reaches a Flight server on the host; `localhost` refers +to the container itself. For authenticated or TLS deployments, follow the +[connection options](../client-guide.md#jdbc-dbeaver-and-others) and +[server configuration](../server-guide.md#deployment-and-operations). + +## Guides + +| Integration | Guide | +| --- | --- | +| Python / PyArrow | [SQL and native datasets](python.md) | +| Go | [Flight SQL and native Flight](go.md) | +| Java | [JDBC application](java.md) | +| DBeaver | [Driver setup and manual tests](../dbeaver.md) | +| Other JDBC applications | [Custom driver configuration](jdbc-applications.md) | +| pandas | [SQL reads and parameter binding](pandas.md) | +| Polars | [SQL reads and batch iteration](polars.md) | +| DuckDB | [Local SQL over downloaded Arrow](duckdb.md) | +| R | [ADBC driver manager](r.md) | +| Apache Superset | [Version-specific connection guidance](../superset.md) | +| Tableau | [JDBC setup and read preview](tableau.md) | +| Metabase | [Community plugin and Actions](metabase.md) | +| Power BI | [Custom connector development path](power-bi.md) | +| Redis Insight | [Inspect the backing Redis database](redis-insight.md) | + +## Optional sample SQL table + +The pandas, Polars, DuckDB, Go, Java, JDBC and BI guides use +`flight_tool_sample`. Run these commands from the repository root against a +disposable backend. The table name must be unused; `mode="create"` refuses +to replace an existing table. + +```sh +python -m venv .venv-tools +. .venv-tools/bin/activate +python -m pip install -r examples/python/requirements-clients.txt +python - <<'PY' +import adbc_driver_flightsql.dbapi as sql +import pyarrow as pa + +with sql.connect("grpc://localhost:8815", autocommit=True) as conn, conn.cursor() as cur: + cur.adbc_ingest("flight_tool_sample", pa.table({ + "id": [1, 2, 3], + "label": ["one", None, "three"], + "amount": [1.5, 2.5, 3.5], + }), mode="create") +PY +``` + +Keep the table while following the guides. An ordered SELECT should return +three rows, with a NULL label on row 2; `SUM(amount)` is `7.5`. When done, drop +only the fixture you created: + +```sh +python - <<'PY' +import adbc_driver_flightsql.dbapi as sql +with sql.connect("grpc://localhost:8815", autocommit=True) as conn, conn.cursor() as cur: + cur.execute("DROP TABLE flight_tool_sample") + cur.fetch_arrow_table() +PY +``` + +These guides describe setup and expected checks. Historical passing results, +versions and application limits remain in the [compatibility matrix](../client-compatibility.md). +Creating a fixture through Python does not establish write support in another tool. diff --git a/docs/integrations/duckdb.md b/docs/integrations/duckdb.md new file mode 100644 index 0000000..bfe0bca --- /dev/null +++ b/docs/integrations/duckdb.md @@ -0,0 +1,44 @@ +# DuckDB + +[All integration guides](README.md). Fetch Arrow data from Flight, register it +in a local DuckDB connection, and query it there. This path transfers data +before DuckDB executes SQL; it does not require a DuckDB Flight extension. + +## Install and query a SQL result + +Follow the [Python environment and sample table setup](README.md#optional-sample-sql-table), +which includes DuckDB. Run: + +```python +import duckdb +import adbc_driver_flightsql.dbapi as sql + +with sql.connect("grpc://localhost:8815", autocommit=True) as conn, conn.cursor() as cur: + cur.execute("SELECT id, label, amount FROM flight_tool_sample ORDER BY id") + table = cur.fetch_arrow_table() + +with duckdb.connect() as local: + local.register("redis_rows", table) + result = local.execute("SELECT COUNT(*), SUM(amount) FROM redis_rows").fetchone() + assert result == (3, 7.5) + print(result) +``` + +Apply remote filters and select only needed columns in the ADBC query to limit +the transfer. `fetch_arrow_table()` materializes that result in client memory. +Subsequent DuckDB SQL runs locally; it does not update Redis. + +## Query a named dataset + +Download a dataset using the [Python Flight example](python.md#upload-and-download-a-dataset), +reading every returned endpoint. Inside a DuckDB connection, call +`local.register("features", received)` and query `features`. You can join the +registered data to local DuckDB tables or files. + +## Validation and cleanup + +Run `make clients-smoke` against a disposable server. Recorded coverage includes +local aggregates over Flight SQL results and ordered/NULL results over a native +Flight dataset; see [versions and limits](../client-compatibility.md#flight-client-validation-on-2026-10-29). +No direct DuckDB Flight connector or write-back path was tested. Remove the +sample table using the [shared cleanup](README.md#optional-sample-sql-table). diff --git a/docs/integrations/go.md b/docs/integrations/go.md new file mode 100644 index 0000000..29ddee4 --- /dev/null +++ b/docs/integrations/go.md @@ -0,0 +1,72 @@ +# Go + +[All integration guides](README.md). Use Arrow Go's Flight SQL client for SQL +and its native Flight client for named datasets. Both paths have recorded +passing integration/race coverage in the [compatibility matrix](../client-compatibility.md). + +## Install and query SQL + +Follow the [server and sample table setup](README.md). In a separate Go module, +install the repository's recorded Arrow version: + +```sh +go mod init example.com/flight-demo +go get github.com/apache/arrow-go/v18@v18.8.0 +``` + +Save as `main.go` and run `go mod tidy`, then `go run .`: + +```go +package main + +import ( + "context" + "fmt" + "log" + + "github.com/apache/arrow-go/v18/arrow/flight/flightsql" + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" +) + +func main() { + if err := run(); err != nil { log.Fatal(err) } +} + +func run() error { + ctx := context.Background() + client, err := flightsql.NewClient("localhost:8815", nil, nil, + grpc.WithTransportCredentials(insecure.NewCredentials())) + if err != nil { return err } + defer client.Close() + info, err := client.Execute(ctx, "SELECT * FROM flight_tool_sample ORDER BY id") + if err != nil { return err } + for _, ep := range info.Endpoint { + reader, err := client.DoGet(ctx, ep.Ticket) + if err != nil { return err } + for reader.Next() { fmt.Println(reader.RecordBatch()) } + err = reader.Err() + reader.Release() + if err != nil { return err } + } + return nil +} +``` + +Expect three rows across the returned record batches. Consume all endpoints, +check stream errors and release readers. Use `ExecuteUpdate` for DDL/DML and +`Prepare` for bound parameters. SQL is autocommit; transactions are unsupported. + +## Native datasets and validation + +Create a native client with `flight.NewClientWithMiddleware` from +`github.com/apache/arrow-go/v18/arrow/flight`. Use path descriptors for named +datasets, Arrow IPC over `DoPut`/`DoGet`, and consume upload acknowledgements. +The [dataset tests](../../server/integration_test.go) show complete upload, +download, selection and cleanup flows; the [dataset reference](../data-access.md#datasets) +defines descriptors, tickets and actions. Native datasets do not appear as SQL tables. + +From this repository, `make test-integration` and `make test-cluster` start the +Compose backends and run race suites. Use disposable backends: these tests +create tables, keys and ACL users. Without `REDIS_URI`, backend tests skip. +Remove the shared sample table with the [fixture cleanup](README.md#optional-sample-sql-table). diff --git a/docs/integrations/java.md b/docs/integrations/java.md new file mode 100644 index 0000000..0b9a6f2 --- /dev/null +++ b/docs/integrations/java.md @@ -0,0 +1,64 @@ +# Java + +[All integration guides](README.md). Flight SQL JDBC 19.0.0 is the validated +Java route. Native Java Flight SDK access is untested against this server. + +## Install and query through JDBC + +Follow the [server and sample table setup](README.md). Install a full JDK +(recorded local tests used Java 25), then download the verified driver from the +repository root: + +```sh +make bin/flight-sql-jdbc-driver-19.0.0.jar +``` + +For Maven applications use `org.apache.arrow:flight-sql-jdbc-driver:19.0.0`. +See the [official driver instructions](https://arrow.apache.org/java/current/flight_sql_jdbc_driver.html). +Save the following as `FlightQuery.java`: + +```java +import java.sql.*; + +public class FlightQuery { + public static void main(String[] args) throws Exception { + String url = "jdbc:arrow-flight-sql://localhost:8815/?useEncryption=false"; + try (Connection conn = DriverManager.getConnection(url); + PreparedStatement stmt = conn.prepareStatement( + "SELECT id, label FROM flight_tool_sample WHERE id >= ? ORDER BY id")) { + stmt.setInt(1, 2); + try (ResultSet rows = stmt.executeQuery()) { + while (rows.next()) { + System.out.println(rows.getLong(1) + ": " + rows.getString(2)); + } + } + } + } +} +``` + +On Java 25, run: + +```sh +java --add-opens=java.base/java.nio=ALL-UNNAMED \ + --sun-misc-unsafe-memory-access=allow \ + -cp bin/flight-sql-jdbc-driver-19.0.0.jar FlightQuery.java +``` + +Expect `2: null` and `3: three`. Java 17–22 must omit the +`--sun-misc-unsafe-memory-access` option; Java 23+ needs it for this setup. +Use `executeUpdate` for writes and verify them with a fresh SELECT. Keep +autocommit enabled and execute statements separately. + +## Validation and limits + +Run `make jdbc-smoke` against a disposable running server. Override the address +with `JDBC_URL` and the JDK with `JAVA=/path/to/bin/java`. The smoke suite tests +metadata, prepared parameters, writes, sessions and concurrent statement/result +reads. See the [JDBC caveats](../client-compatibility.md#jdbc-type-information-and-schema-caveats) +for empty `getTypeInfo()` and prepared statements without a result schema. + +For named datasets, use the [native Java Flight SDK](https://arrow.apache.org/java/current/flight.html) +with the server's [dataset API](../data-access.md#datasets); JDBC does not expose +that API. This is an unvalidated development route. Remove the shared sample +table with the [fixture cleanup](README.md#optional-sample-sql-table). diff --git a/docs/integrations/jdbc-applications.md b/docs/integrations/jdbc-applications.md new file mode 100644 index 0000000..e14f99e --- /dev/null +++ b/docs/integrations/jdbc-applications.md @@ -0,0 +1,45 @@ +# Other JDBC applications + +[All integration guides](README.md). Applications that accept a custom JDBC +driver can try Flight SQL. The JDBC driver is validated; each application's +metadata handling, generated SQL and write behavior still needs acceptance testing. + +## Configure a custom driver + +1. Start a server and create the [sample SQL table](README.md#optional-sample-sql-table). +2. Run `make bin/flight-sql-jdbc-driver-19.0.0.jar` from the repository root. + Add that JAR to the application's driver libraries, or use Maven coordinate + `org.apache.arrow:flight-sql-jdbc-driver:19.0.0`. +3. Set the driver class to `org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver`. +4. Set URL to `jdbc:arrow-flight-sql://localhost:8815/?useEncryption=false` + for the local unauthenticated endpoint. Use the actual host from the client. +5. Configure the application's JVM with + `--add-opens=java.base/java.nio=ALL-UNNAMED`. On Java 23+ also add + `--sun-misc-unsafe-memory-access=allow`, then restart the application. +6. Use autocommit. Browse catalog `redis`, schema `public`, and table + `flight_tool_sample`. + +For TLS, credentials, session behavior and long-result timeouts, follow the +[JDBC connection reference](../client-guide.md#jdbc-dbeaver-and-others) and +[official driver options](https://arrow.apache.org/java/current/flight_sql_jdbc_driver.html). + +## Check application behavior + +Run these statements separately in the application's SQL editor: + +```sql +SELECT id, label, amount FROM flight_tool_sample ORDER BY id; +SELECT COUNT(*) AS n, SUM(amount) AS total FROM flight_tool_sample; +``` + +Expect ids 1–3, a NULL label for id 2, and aggregate values `3` and `7.5`. +Check table/column discovery and generated filters separately. If writes are +part of the workflow, insert, update and delete a disposable row through that +application and verify each operation with a SELECT. Fixture creation through +another client does not prove application write support. + +Run `make jdbc-smoke` to distinguish a driver/server issue from an application +issue. The driver's empty `getTypeInfo()` can affect application type discovery; +see [known caveats](../client-compatibility.md#jdbc-type-information-and-schema-caveats). +SQL tools see SQL tables; named datasets require a native Flight client. +Finish with the [sample cleanup](README.md#optional-sample-sql-table). diff --git a/docs/integrations/metabase.md b/docs/integrations/metabase.md new file mode 100644 index 0000000..e49a732 --- /dev/null +++ b/docs/integrations/metabase.md @@ -0,0 +1,66 @@ +# Metabase + +[All integration guides](README.md). The validated route uses Metabase +0.64.2.9 with the community Flight SQL plugin 0.8.0, which bundles JDBC 19.0.0. +API reads, schema sync, generated queries and custom SQL Actions passed. +Browser/dashboard rendering was not tested. See [versions, commands and logs](../validation/2026-11-01-bi-jdbc/README.md). + +## Install the plugin + +Start a server using the [shared prerequisites](README.md). In the repository root: + +```sh +mkdir -p bin/metabase-plugins +curl -fL https://github.com/J0hnG4lt/metabase-flightsql-driver/releases/download/0.8.0/arrow-flight-sql.metabase-driver.jar \ + -o bin/metabase-plugins/arrow-flight-sql.metabase-driver.jar +echo 'b97e9cac4ac43d1011d3dc7100d49adee928684bddb33e42e4bd17e21d654b60 bin/metabase-plugins/arrow-flight-sql.metabase-driver.jar' | shasum -a 256 -c - +``` + +Use a **writable** plugins directory: the recorded read-only mount failed to +load the plugin. Configure the Metabase JVM with the flags used in validation: + +```text +--add-opens=java.base/java.nio=ALL-UNNAMED +--add-opens=java.base/sun.nio.ch=ALL-UNNAMED +--sun-misc-unsafe-memory-access=allow +--enable-native-access=ALL-UNNAMED +-Dio.netty.tryReflectionSetAccessible=true +``` + +The `sun.misc` option applies to Java 23+; older JVMs reject it. The +[plugin's own instructions](https://github.com/J0hnG4lt/metabase-flightsql-driver) +describe loading and configuration. Restart Metabase after installation. + +## Add the connection and read data + +Select the plugin's Arrow Flight SQL database type. Configure the Flight host, +port `8815`, catalog `redis`, and encryption disabled for the local example. +Use `host.docker.internal` when Metabase runs in Docker Desktop and Flight runs +on the host. Create the [shared sample](README.md#optional-sample-sql-table), +sync the database schema and inspect its columns. Run a native query: + +```sql +SELECT id, label, amount FROM flight_tool_sample ORDER BY id; +``` + +Expect ids 1–3 and a NULL label on row 2. Check a visual filter and grouped count +separately; API validation covered generated queries, not their rendered UI. + +## Write with Actions + +Enable Actions in the connection details and database settings. The recorded +API configuration sets `enable-actions` in details and `database-enable-actions` +in database settings. Use custom SQL Actions for INSERT, UPDATE and DELETE, +and verify effects with a SELECT after each action. The +[smoke script](../../examples/jdbc/metabase_smoke.py) shows the exact tested API calls. + +Do not use the native read-query endpoint as a DDL runner. The recorded DROP +failed with `Select statement did not produce a ResultSet for native query`; +an error does not guarantee that a write had no effects. Create/drop fixtures +through JDBC or ADBC. CSV uploads, implicit model actions and transactions +are outside the validated scope. + +For full reproduction, follow the [disposable Docker setup](../validation/2026-11-01-bi-jdbc/README.md#reproduction). +That recipe uses Flight port 18825 and a separate fixture, and refuses an +already configured Metabase. Remove this guide's sample using the +[shared cleanup](README.md#optional-sample-sql-table). diff --git a/docs/integrations/pandas.md b/docs/integrations/pandas.md new file mode 100644 index 0000000..35c8c7a --- /dev/null +++ b/docs/integrations/pandas.md @@ -0,0 +1,46 @@ +# pandas + +[All integration guides](README.md). pandas reads SQL through an ADBC Flight +SQL connection, or converts an Arrow table downloaded through either API. + +## Install and read + +Follow the [Python environment and sample table setup](README.md#optional-sample-sql-table). +It includes pandas and the Flight SQL driver. With the server running: + +```python +import pandas as pd +import adbc_driver_flightsql.dbapi as sql + +with sql.connect("grpc://localhost:8815", autocommit=True) as conn: + frame = pd.read_sql("SELECT * FROM flight_tool_sample ORDER BY id", conn) + print(frame) + assert frame.id.tolist() == [1, 2, 3] + assert pd.isna(frame.loc[1, "label"]) +``` + +## Bind parameters through ADBC + +The recorded pandas ADBC path raises `NotImplementedError` for +`pd.read_sql(..., params=...)`. Bind through the cursor, then convert: + +```python +with sql.connect("grpc://localhost:8815", autocommit=True) as conn, conn.cursor() as cur: + cur.execute("SELECT * FROM flight_tool_sample WHERE id >= ? ORDER BY id", (2,)) + selected = cur.fetch_arrow_table().to_pandas() + assert selected.id.tolist() == [2, 3] +``` + +Use the same `table.to_pandas()` conversion for a native Flight dataset +downloaded with the [Python guide](python.md#upload-and-download-a-dataset). +For writes, convert with `pyarrow.Table.from_pandas(frame, preserve_index=False)` +and ingest through ADBC into a new table; pandas `to_sql` is outside the +recorded coverage. + +## Validation and cleanup + +Run `make clients-smoke` from the repository root against a disposable server +(`FLIGHT_URI=grpc://localhost:8815` overrides the address). The +[recorded checks](../client-compatibility.md#flight-client-validation-on-2026-10-29) +cover reads, NULLs, empty results and the parameter workaround. Remove the +shared table with the [fixture cleanup](README.md#optional-sample-sql-table). diff --git a/docs/integrations/polars.md b/docs/integrations/polars.md new file mode 100644 index 0000000..88bafa4 --- /dev/null +++ b/docs/integrations/polars.md @@ -0,0 +1,42 @@ +# Polars + +[All integration guides](README.md). Use an ADBC Flight SQL connection for +SQL reads, or `pl.from_arrow()` for downloaded Arrow data. + +## Install and query + +Follow the [Python environment and sample table setup](README.md#optional-sample-sql-table), +which includes Polars. Then run: + +```python +import polars as pl +import adbc_driver_flightsql.dbapi as sql + +with sql.connect("grpc://localhost:8815", autocommit=True) as conn: + query = "SELECT * FROM flight_tool_sample ORDER BY id" + frame = pl.read_database(query, conn) + print(frame) + assert frame["id"].to_list() == [1, 2, 3] + assert frame["label"].null_count() == 1 + for batch in pl.read_database(query, conn, iter_batches=True, batch_size=2): + print(batch) +``` + +Keep the connection open while consuming the batch iterator. Batch sizes +depend on driver behavior; do not assume each batch has exactly two rows. + +## Arrow results and writes + +For a native dataset, follow the [Python download example](python.md#upload-and-download-a-dataset) +and use `pl.from_arrow(received)`. SQL tables and named datasets are separate. +For parameterized SQL, bind with an ADBC cursor before converting its Arrow +result. For ingestion, pass `frame.to_arrow()` to `cur.adbc_ingest` with a new +table name and `mode="create"`; Polars write APIs have not been validated here. + +## Validation and cleanup + +Run `make clients-smoke` against a disposable server. The +[recorded Polars checks](../client-compatibility.md#flight-client-validation-on-2026-10-29) +cover ordered reads, NULLs, empty results and batch iteration. They do not +establish arbitrary Polars API compatibility. Use the +[shared fixture cleanup](README.md#optional-sample-sql-table) when done. diff --git a/docs/integrations/power-bi.md b/docs/integrations/power-bi.md new file mode 100644 index 0000000..ce44819 --- /dev/null +++ b/docs/integrations/power-bi.md @@ -0,0 +1,38 @@ +# Power BI + +[All integration guides](README.md). This project ships no Power BI connector, +and no Power BI application integration has been validated. The available path +requires developing a custom ADBC-based Power Query connector. + +## Develop a connector + +Follow Microsoft's [ADBC connector development guide](https://learn.microsoft.com/en-us/power-query/adbc), +which uses a Flight SQL sample. Its driver configuration and `Adbc.Connection` +pattern are a starting point; the sample's SQL generation and metadata behavior +must be adapted to this server. + +1. Start the Flight server and create the [sample SQL table](README.md#optional-sample-sql-table). +2. Set up the Power Query SDK and Flight SQL driver configuration following + Microsoft's sample. Check the driver availability for the actual Power BI runtime. +3. Map host, port, encryption and authentication inputs to the Flight SQL + connection properties. The local endpoint is `grpc://localhost:8815`. +4. Adapt catalog/schema/table navigation for `redis` / `public`, Arrow-to-Power + Query type mappings and generated SQL to the [server's SQL reference](../sql-reference.md). +5. Build and load the custom connector using Microsoft's development workflow, + then test it in Power BI Desktop. + +These steps describe connector work still required, rather than an existing +plug-in installation. PostgreSQL and generic Redis connectors do not connect +to the Flight endpoint. + +## Acceptance checks + +Check connection, navigation, column types, ordered reads, NULLs and empty +results first. The sample table has three rows and `SUM(amount) = 7.5`. +Then test query folding for filters, projections and aggregates, refresh, and +the intended deployment/gateway runtime. Record the connector, driver and +Power BI versions with results before claiming compatibility. + +SQL is autocommit without transactions. Named Arrow datasets are separate from +SQL tables and need a native Flight client or explicit dataset-to-SQL import. +Use the [fixture cleanup](README.md#optional-sample-sql-table) when done. diff --git a/docs/integrations/python.md b/docs/integrations/python.md new file mode 100644 index 0000000..279b06d --- /dev/null +++ b/docs/integrations/python.md @@ -0,0 +1,72 @@ +# Python / PyArrow + +[All integration guides](README.md). Use ADBC Flight SQL for SQL tables and +`pyarrow.flight` for named datasets. Both paths have recorded passing smoke +checks; see [versions and scope](../client-compatibility.md). + +## Install and query SQL + +Follow the [prerequisites and sample table setup](README.md), which installs +the recorded Python client versions. Run in that environment: + +```python +import adbc_driver_flightsql.dbapi as sql + +with sql.connect("grpc://localhost:8815", autocommit=True) as conn, conn.cursor() as cur: + cur.execute("SELECT id, label, amount FROM flight_tool_sample WHERE id >= ? ORDER BY id", (2,)) + print(cur.fetch_arrow_table()) +``` + +Expect ids 2 and 3, including a NULL label for id 2. For writes, execute DDL or +DML separately, consume the response, and read back the affected rows. For bulk +ingestion use `cur.adbc_ingest(name, arrow_table, mode="create")`, as in the +sample setup. Transactions are unsupported; keep `autocommit=True`. + +## Upload and download a dataset + +This example uses a unique dataset name, reads every endpoint and removes its +dataset afterward. It does not create a SQL table. + +```python +import uuid +import pyarrow as pa +import pyarrow.flight as flight + +client = flight.connect("grpc://localhost:8815") +name = "guide_" + uuid.uuid4().hex +desc = flight.FlightDescriptor.for_path(name) +try: + source = pa.table({"id": [1, 2], "embedding": [[0.1, 0.2], [0.3, 0.4]]}) + writer, metadata = client.do_put(desc, source.schema) + with writer: + writer.write_table(source) + writer.done_writing() + print(metadata.read()) # Consume the upload acknowledgement. + info = client.get_flight_info(desc) + received = pa.concat_tables([ + client.do_get(ep.ticket).read_all() for ep in info.endpoints + ]) + assert received.equals(source) +finally: + try: + list(client.do_action(flight.Action("drop_dataset", name.encode()))) + finally: + client.close() +``` + +For discovery, projection, batch selection and storage details, see the +[dataset reference](../data-access.md#datasets). To turn a dataset into SQL +rows, follow the [dataset-to-SQL workflow](../dataset-sql-workflow.md). + +## Reproduce recorded checks + +Against a disposable running server, from the repository root: + +```sh +python examples/python/quickstart.py --uri grpc://localhost:8815 +python examples/python/dataset_to_sql.py --uri grpc://localhost:8815 +``` + +The scripts create and clean up fixtures; they also accept `--user` and +`--password`. See the [ADBC Flight SQL driver documentation](https://arrow.apache.org/adbc/current/driver/flight_sql.html) +for driver options and the [full client examples](../client-guide.md#examples). diff --git a/docs/integrations/r.md b/docs/integrations/r.md new file mode 100644 index 0000000..1fad229 --- /dev/null +++ b/docs/integrations/r.md @@ -0,0 +1,57 @@ +# R + +[All integration guides](README.md). The validated route uses +`adbcdrivermanager` and a Flight SQL shared library, with nanoarrow converting +the query stream to a data frame. DBI/adbi is untested. + +## Install the R packages and driver + +Start a server using the [shared prerequisites](README.md). Install the packages: + +```r +install.packages(c("adbcdrivermanager", "nanoarrow"), + repos = c("https://apache.r-universe.dev", "https://cloud.r-project.org")) +``` + +The [recorded run](../client-compatibility.md#flight-client-validation-on-2026-10-29) +used development builds of these packages and Flight SQL driver 1.12.0. Package +installation above follows the repositories' available versions; it does not +reproduce those development builds exactly. + +Obtain the [Flight SQL shared driver](https://arrow.apache.org/adbc/current/driver/flight_sql.html) +for R's OS and architecture. On Linux the library is +`libadbc_driver_flightsql.so`; use the corresponding library for your platform. +The recorded Linux arm64 test sourced it from the Python Flight SQL 1.12.0 +wheel. Python does not execute the R queries. The tested runtime could not +install `adbcflightsql`, so these instructions use the explicit library path. + +## Run the read/write example + +From the repository root, with R and the library installed: + +```sh +Rscript examples/r/flight_sql.R grpc://localhost:8815 /path/to/libadbc_driver_flightsql.so +# Equivalent Make target: +make r-smoke FLIGHT_URI=grpc://localhost:8815 FLIGHT_SQL_DRIVER=/path/to/libadbc_driver_flightsql.so +``` + +The [script](../../examples/r/flight_sql.R) creates a unique table, inserts three +rows, checks ordered results, NULLs, an empty result and a SUM of `7.5`, then +drops its fixture. Expect a `PASS R Flight SQL` line and session information. + +Its connection setup is: + +```r +library(adbcdrivermanager) +driver <- adbc_driver("/path/to/libadbc_driver_flightsql.so", + entrypoint = "AdbcDriverFlightSQLInit") +db <- adbc_database_init(driver, uri = "grpc://localhost:8815") +con <- adbc_connection_init(db, adbc.connection.autocommit = "true") +# Create statements and consume streams as in examples/r/flight_sql.R. +adbc_connection_release(con) +adbc_database_release(db) +``` + +Keep autocommit enabled. If loading fails, check library architecture, OS, +dependencies and the `AdbcDriverFlightSQLInit` entry point. The recorded checks +use a local unauthenticated endpoint; TLS/authentication remains outside that scope. diff --git a/docs/integrations/redis-insight.md b/docs/integrations/redis-insight.md new file mode 100644 index 0000000..d650a37 --- /dev/null +++ b/docs/integrations/redis-insight.md @@ -0,0 +1,39 @@ +# Redis Insight + +[All integration guides](README.md). Redis Insight connects to the backing +Redis database. It does not use the Flight endpoint. No Redis Insight +application validation is recorded for this project. + +## Connect to Redis + +1. Install [Redis Insight](https://redis.io/docs/latest/develop/tools/insight/). +2. Find the Redis host, port and database used by the Flight server's + `REDIS_URI` or `--redis-uri` setting. +3. Add that database in Redis Insight, supplying the corresponding Redis ACL + credentials and TLS settings. For a default local Redis this is + `localhost:6379`, database 0; Flight's default `8815` is a different service. +4. Open the key browser and inspect the database. + +For Redis Cluster or a managed deployment, select the matching connection +configuration and use addresses reachable from Redis Insight. Flight login +tokens and JDBC URLs are not Redis connection details. + +## Inspect SQL rows and datasets + +Create the [sample SQL table](README.md#optional-sample-sql-table) through ADBC, +then locate its row HASHes and Search index in Redis Insight. Their layout +follows the Redis ADBC driver's [rows and indexes conventions](https://github.com/redis/redis-adbc-driver/blob/main/docs/driver-reference.md#rows-and-indexes). +Arbitrary existing Redis keys do not automatically become SQL tables. + +Named datasets use the server's own keys (default prefix `arrowflight:`), with +Arrow IPC payloads. See [dataset storage](../data-access.md#datasets) for the +key layout and configurable prefix. Use a [native Flight client](python.md#upload-and-download-a-dataset) +to decode and inspect those values as Arrow tables. + +## Verify and clean up + +Confirm that a row ingested through Flight SQL is visible as a Redis HASH, +then query the table through Flight SQL to compare its values. Keep dataset +metadata and payloads consistent by using the Flight API for dataset changes. +Remove the sample table through the [fixture cleanup](README.md#optional-sample-sql-table). +These are suggested acceptance checks, not recorded Redis Insight passes. diff --git a/docs/integrations/tableau.md b/docs/integrations/tableau.md new file mode 100644 index 0000000..d0a34a5 --- /dev/null +++ b/docs/integrations/tableau.md @@ -0,0 +1,43 @@ +# Tableau + +[All integration guides](README.md). Tableau Desktop Free Edition 2026.3 with +Flight SQL JDBC 19.0.0 passed connection, metadata discovery and a three-row +preview. The acceptance scope is read-only; worksheets, aggregates, extracts, +joins and dashboards remain untested. See the [manual validation record](../validation/2026-11-01-tableau/README.md). + +## Install the driver + +Start the server and create the [sample SQL table](README.md#optional-sample-sql-table). +From the repository root: + +```sh +make bin/flight-sql-jdbc-driver-19.0.0.jar +``` + +Copy the JAR to Tableau's driver directory and restart Tableau. The validated +macOS setup used `~/Library/Tableau/Drivers/`. For other platforms, follow +[Tableau's JDBC driver locations](https://help.tableau.com/current/pro/desktop/en-gb/examples_otherdatabases_jdbc.htm). +The recorded macOS test needed no Tableau JVM changes, TDC overrides or custom connector. + +## Connect and preview + +1. Select **To a Server → More → Other Databases (JDBC)**. +2. Enter `jdbc:arrow-flight-sql://localhost:8815/?useEncryption=false`. +3. Keep the SQL92 dialect. Leave credentials blank for this local unauthenticated setup. +4. Sign in and select/type database `redis`, then schema `public`. +5. Add `flight_tool_sample` and click **Update Now** in the data preview. + +Expect three columns and three rows, including the NULL label on row 2. +The historical validation used a different fixture, `bi_jdbc_validation`, and +Flight port 18825; its exact commands and expected values are in the linked +record. The guide's shared sample uses the normal port 8815. + +## Scope and cleanup + +Preview success establishes that read path only. Test worksheet-generated SQL, +filters and aggregates before depending on them. Custom SQL may be wrapped as +a SELECT subquery, so it is not a general DDL/DML editor. No write support is +claimed. Authentication and TLS were not covered by the recorded run. + +Remove the sample through the [fixture cleanup](README.md#optional-sample-sql-table). +JDBC sees SQL tables; named Arrow datasets need the native Flight API. diff --git a/docs/sdlc/context.md b/docs/sdlc/context.md index a2ab68a..98353e1 100644 --- a/docs/sdlc/context.md +++ b/docs/sdlc/context.md @@ -56,7 +56,7 @@ All keys involved in one dataset commit/drop share a cluster hash tag. Uploads p Dataset tickets select generation, ordered column names, and a half-open batch range. With endpoint splitting enabled, consumers must read all endpoints. Approximate chunk size is 2 MiB; dataset read/write pages target 8 MiB and at most 64 batches. These values are not hard memory bounds for an oversized row/batch. Current readers handle LIST and HASH; old LIST-only readers cannot read new HASH generations. -Evidence: [dataset code](../../server/datasets.go), [IPC code](../../server/ipc.go), [SQL code](../../server/sqlserver.go), [feature reference](../../README.md#feature-reference). +Evidence: [dataset code](../../server/datasets.go), [IPC code](../../server/ipc.go), [SQL code](../../server/sqlserver.go), [protocol capabilities](../flight-capabilities.md). ## Selected defaults and boundaries diff --git a/docs/superset.md b/docs/superset.md index b0ff0cf..eedc79a 100644 --- a/docs/superset.md +++ b/docs/superset.md @@ -1,5 +1,7 @@ # Apache Superset: choosing a Redis connection +[All integration guides](integrations/README.md). + Guidance checked on 2026-11-01. Choose the connection path for your Superset version: