Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
243 changes: 68 additions & 175 deletions README.md

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions docs/client-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,8 @@ with `--dataset-endpoint-batches`.

Use the [Arrow Flight SQL JDBC driver](https://arrow.apache.org/java/current/flight_sql_jdbc_driver.html)
(`org.apache.arrow:flight-sql-jdbc-driver`). The recorded local validation used
19.0.0 on Java 25; CI is configured for Java 17, 21 and 25. See
[tested integration coverage](../README.md#tested-integration-coverage). The URL is:
19.0.0 on Java 25; CI is configured for Java 17, 21 and 25. See the
[client compatibility matrix](client-compatibility.md). The URL is:

```text
jdbc:arrow-flight-sql://localhost:8815/?useEncryption=false
Expand Down Expand Up @@ -157,7 +157,7 @@ work with the results of either interface.
| **Tableau** | Use [Other Databases (JDBC)](https://help.tableau.com/current/pro/desktop/en-gb/examples_otherdatabases_jdbc.htm) with Flight SQL JDBC 19.0.0 installed in Tableau's driver directory, then restart Tableau. Follow the [tested setup](validation/2026-11-01-tableau/README.md); this run needed no Tableau JVM configuration changes. Tableau's generic JDBC connector does not guarantee feature compatibility | Visual analysis and dashboards; requires driver setup and compatibility testing | Desktop Free Edition 2026.3 / JDBC 19.0.0: connection, catalog/schema/table/column discovery and three-row data preview passed. Read-only acceptance; worksheets, filters and aggregates remain untested. [Record](validation/2026-11-01-tableau/README.md) |
| **Metabase** | Community [Flight SQL plugin](https://github.com/J0hnG4lt/metabase-flightsql-driver), with its Arrow JVM flags and a writable plugins directory. Enable Actions in connection details and database settings for writes | Native SQL and visual queries; custom SQL Actions for write-back | Metabase 0.64.2.9 / plugin 0.8.0: API reads, schema sync, generated queries and INSERT/UPDATE/DELETE Actions passed. Native DDL through the read-query endpoint failed; [validation](validation/2026-11-01-bi-jdbc/README.md) |
| **Other JDBC applications** | Load `org.apache.arrow:flight-sql-jdbc-driver` and use the same JDBC URL, if the application supports custom JDBC drivers | Java applications and data tools that allow a custom driver | Driver only — Flight SQL JDBC 19.0.0 passed on Java 25; individual applications untested ([record](../docs/client-compatibility.md)). |
| **Python / PyArrow** | Use [ADBC Flight SQL](https://arrow.apache.org/adbc/current/driver/flight_sql.html) (`adbc_driver_flightsql`) for SQL and [`pyarrow.flight`](https://arrow.apache.org/docs/python/flight.html) for datasets; see [Client examples](../README.md#client-examples) | Queries, bulk ingest, Arrow table transfers and ML batches | Yes — ADBC Flight SQL and PyArrow Flight smoke scripts passed ([record](../intent/driver-v0.0.17/plan.md#validation)). |
| **Python / PyArrow** | Use [ADBC Flight SQL](https://arrow.apache.org/adbc/current/driver/flight_sql.html) (`adbc_driver_flightsql`) for SQL and [`pyarrow.flight`](https://arrow.apache.org/docs/python/flight.html) for datasets; see [Python integration guide](integrations/python.md) | Queries, bulk ingest, Arrow table transfers and ML batches | Yes — ADBC Flight SQL and PyArrow Flight smoke scripts passed ([record](../intent/driver-v0.0.17/plan.md#validation)). |
| **pandas** | Pass an ADBC Flight SQL connection to [`pd.read_sql`](https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html) (pandas 2.2+), or convert a downloaded Arrow table with `table.to_pandas()` | Dataframe analysis and reporting | Yes — `pd.read_sql` results, NULLs and empty results passed; ADBC `params=` is unsupported ([record](../docs/client-compatibility.md#flight-client-validation-on-2026-10-29)). |
| **Polars** | Pass an ADBC Flight SQL connection to [`pl.read_database`](https://docs.pola.rs/api/python/stable/reference/api/polars.read_database.html), or convert a downloaded Arrow table with `pl.from_arrow(table)` | Dataframe transformations and analytics | Yes — `pl.read_database`, NULLs, empty results and batch iteration passed. ([record](../docs/client-compatibility.md#flight-client-validation-on-2026-10-29)). |
| **DuckDB** | Fetch an Arrow table through ADBC or Flight, then [query it locally in DuckDB](https://duckdb.org/docs/current/guides/python/import_arrow); this uses an Arrow transfer rather than a direct DuckDB connection to the server | Joining Redis results with local files or other Arrow data | Yes — local SQL over Arrow fetched through Flight SQL and Flight RPC passed. ([record](../docs/client-compatibility.md#flight-client-validation-on-2026-10-29)). |
Expand All @@ -172,7 +172,7 @@ Flight / ADBC: grpc://localhost:8815
JDBC: jdbc:arrow-flight-sql://localhost:8815/?useEncryption=false
```

See [tested integration coverage](../README.md#tested-integration-coverage) for the ADBC,
See the [client compatibility matrix](client-compatibility.md) for the ADBC,
JDBC and native Flight checks. Tested status reflects recorded validation
against this Flight server; the linked
records give versions, dates, and scope. Tests of the Redis ADBC driver in the
Expand Down
2 changes: 1 addition & 1 deletion docs/data-engineering-adbc-support.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ The latter does not imply that the tool can load the Redis driver.

**Redis ADBC support** means loading this repository's driver locally.
**Redis Arrow Flight SQL support** means connecting over the network to the
[Redis Arrow Flight server](https://github.com/alberttwong/redis-apache-arrow-flight#client-examples),
[Redis Arrow Flight server](https://github.com/redis/redis-apache-arrow-flight#data-tools),
which runs the Redis ADBC driver on the server side. It requires a separate server
deployment; it is not a built-in endpoint of Redis. Its Flight datasets API is a
separate interface and does not establish Flight SQL compatibility.
Expand Down
83 changes: 83 additions & 0 deletions docs/integrations/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Data tool integration guides

[Back to README data tools](../../README.md#data-tools).

Start a [local server](../local-development.md) or
[install a release](../../README.md#install-a-release). SQL needs Redis with
Query Engine (Search); 4× QPF is recommended where supported. SQL uses the
Redis ADBC driver's [data model](../../README.md#choose-an-interface), and runs
in autocommit mode without transactions. Named Arrow datasets are separate
from SQL tables.

Local examples assume an unauthenticated endpoint without TLS:

```text
Flight / ADBC: grpc://localhost:8815
JDBC: jdbc:arrow-flight-sql://localhost:8815/?useEncryption=false
```

Use an address reachable from the client. Inside a Docker Desktop container,
`host.docker.internal` reaches a Flight server on the host; `localhost` refers
to the container itself. For authenticated or TLS deployments, follow the
[connection options](../client-guide.md#jdbc-dbeaver-and-others) and
[server configuration](../server-guide.md#deployment-and-operations).

## Guides

| Integration | Guide |
| --- | --- |
| Python / PyArrow | [SQL and native datasets](python.md) |
| Go | [Flight SQL and native Flight](go.md) |
| Java | [JDBC application](java.md) |
| DBeaver | [Driver setup and manual tests](../dbeaver.md) |
| Other JDBC applications | [Custom driver configuration](jdbc-applications.md) |
| pandas | [SQL reads and parameter binding](pandas.md) |
| Polars | [SQL reads and batch iteration](polars.md) |
| DuckDB | [Local SQL over downloaded Arrow](duckdb.md) |
| R | [ADBC driver manager](r.md) |
| Apache Superset | [Version-specific connection guidance](../superset.md) |
| Tableau | [JDBC setup and read preview](tableau.md) |
| Metabase | [Community plugin and Actions](metabase.md) |
| Power BI | [Custom connector development path](power-bi.md) |
| Redis Insight | [Inspect the backing Redis database](redis-insight.md) |

## Optional sample SQL table

The pandas, Polars, DuckDB, Go, Java, JDBC and BI guides use
`flight_tool_sample`. Run these commands from the repository root against a
disposable backend. The table name must be unused; `mode="create"` refuses
to replace an existing table.

```sh
python -m venv .venv-tools
. .venv-tools/bin/activate
python -m pip install -r examples/python/requirements-clients.txt
python - <<'PY'
import adbc_driver_flightsql.dbapi as sql
import pyarrow as pa

with sql.connect("grpc://localhost:8815", autocommit=True) as conn, conn.cursor() as cur:
cur.adbc_ingest("flight_tool_sample", pa.table({
"id": [1, 2, 3],
"label": ["one", None, "three"],
"amount": [1.5, 2.5, 3.5],
}), mode="create")
PY
```

Keep the table while following the guides. An ordered SELECT should return
three rows, with a NULL label on row 2; `SUM(amount)` is `7.5`. When done, drop
only the fixture you created:

```sh
python - <<'PY'
import adbc_driver_flightsql.dbapi as sql
with sql.connect("grpc://localhost:8815", autocommit=True) as conn, conn.cursor() as cur:
cur.execute("DROP TABLE flight_tool_sample")
cur.fetch_arrow_table()
PY
```

These guides describe setup and expected checks. Historical passing results,
versions and application limits remain in the [compatibility matrix](../client-compatibility.md).
Creating a fixture through Python does not establish write support in another tool.
44 changes: 44 additions & 0 deletions docs/integrations/duckdb.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# DuckDB

[All integration guides](README.md). Fetch Arrow data from Flight, register it
in a local DuckDB connection, and query it there. This path transfers data
before DuckDB executes SQL; it does not require a DuckDB Flight extension.

## Install and query a SQL result

Follow the [Python environment and sample table setup](README.md#optional-sample-sql-table),
which includes DuckDB. Run:

```python
import duckdb
import adbc_driver_flightsql.dbapi as sql

with sql.connect("grpc://localhost:8815", autocommit=True) as conn, conn.cursor() as cur:
cur.execute("SELECT id, label, amount FROM flight_tool_sample ORDER BY id")
table = cur.fetch_arrow_table()

with duckdb.connect() as local:
local.register("redis_rows", table)
result = local.execute("SELECT COUNT(*), SUM(amount) FROM redis_rows").fetchone()
assert result == (3, 7.5)
print(result)
```

Apply remote filters and select only needed columns in the ADBC query to limit
the transfer. `fetch_arrow_table()` materializes that result in client memory.
Subsequent DuckDB SQL runs locally; it does not update Redis.

## Query a named dataset

Download a dataset using the [Python Flight example](python.md#upload-and-download-a-dataset),
reading every returned endpoint. Inside a DuckDB connection, call
`local.register("features", received)` and query `features`. You can join the
registered data to local DuckDB tables or files.

## Validation and cleanup

Run `make clients-smoke` against a disposable server. Recorded coverage includes
local aggregates over Flight SQL results and ordered/NULL results over a native
Flight dataset; see [versions and limits](../client-compatibility.md#flight-client-validation-on-2026-10-29).
No direct DuckDB Flight connector or write-back path was tested. Remove the
sample table using the [shared cleanup](README.md#optional-sample-sql-table).
72 changes: 72 additions & 0 deletions docs/integrations/go.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Go

[All integration guides](README.md). Use Arrow Go's Flight SQL client for SQL
and its native Flight client for named datasets. Both paths have recorded
passing integration/race coverage in the [compatibility matrix](../client-compatibility.md).

## Install and query SQL

Follow the [server and sample table setup](README.md). In a separate Go module,
install the repository's recorded Arrow version:

```sh
go mod init example.com/flight-demo
go get github.com/apache/arrow-go/v18@v18.8.0
```

Save as `main.go` and run `go mod tidy`, then `go run .`:

```go
package main

import (
"context"
"fmt"
"log"

"github.com/apache/arrow-go/v18/arrow/flight/flightsql"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)

func main() {
if err := run(); err != nil { log.Fatal(err) }
}

func run() error {
ctx := context.Background()
client, err := flightsql.NewClient("localhost:8815", nil, nil,
grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil { return err }
defer client.Close()
info, err := client.Execute(ctx, "SELECT * FROM flight_tool_sample ORDER BY id")
if err != nil { return err }
for _, ep := range info.Endpoint {
reader, err := client.DoGet(ctx, ep.Ticket)
if err != nil { return err }
for reader.Next() { fmt.Println(reader.RecordBatch()) }
err = reader.Err()
reader.Release()
if err != nil { return err }
}
return nil
}
```

Expect three rows across the returned record batches. Consume all endpoints,
check stream errors and release readers. Use `ExecuteUpdate` for DDL/DML and
`Prepare` for bound parameters. SQL is autocommit; transactions are unsupported.

## Native datasets and validation

Create a native client with `flight.NewClientWithMiddleware` from
`github.com/apache/arrow-go/v18/arrow/flight`. Use path descriptors for named
datasets, Arrow IPC over `DoPut`/`DoGet`, and consume upload acknowledgements.
The [dataset tests](../../server/integration_test.go) show complete upload,
download, selection and cleanup flows; the [dataset reference](../data-access.md#datasets)
defines descriptors, tickets and actions. Native datasets do not appear as SQL tables.

From this repository, `make test-integration` and `make test-cluster` start the
Compose backends and run race suites. Use disposable backends: these tests
create tables, keys and ACL users. Without `REDIS_URI`, backend tests skip.
Remove the shared sample table with the [fixture cleanup](README.md#optional-sample-sql-table).
64 changes: 64 additions & 0 deletions docs/integrations/java.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Java

[All integration guides](README.md). Flight SQL JDBC 19.0.0 is the validated
Java route. Native Java Flight SDK access is untested against this server.

## Install and query through JDBC

Follow the [server and sample table setup](README.md). Install a full JDK
(recorded local tests used Java 25), then download the verified driver from the
repository root:

```sh
make bin/flight-sql-jdbc-driver-19.0.0.jar
```

For Maven applications use `org.apache.arrow:flight-sql-jdbc-driver:19.0.0`.
See the [official driver instructions](https://arrow.apache.org/java/current/flight_sql_jdbc_driver.html).
Save the following as `FlightQuery.java`:

```java
import java.sql.*;

public class FlightQuery {
public static void main(String[] args) throws Exception {
String url = "jdbc:arrow-flight-sql://localhost:8815/?useEncryption=false";
try (Connection conn = DriverManager.getConnection(url);
PreparedStatement stmt = conn.prepareStatement(
"SELECT id, label FROM flight_tool_sample WHERE id >= ? ORDER BY id")) {
stmt.setInt(1, 2);
try (ResultSet rows = stmt.executeQuery()) {
while (rows.next()) {
System.out.println(rows.getLong(1) + ": " + rows.getString(2));
}
}
}
}
}
```

On Java 25, run:

```sh
java --add-opens=java.base/java.nio=ALL-UNNAMED \
--sun-misc-unsafe-memory-access=allow \
-cp bin/flight-sql-jdbc-driver-19.0.0.jar FlightQuery.java
```

Expect `2: null` and `3: three`. Java 17–22 must omit the
`--sun-misc-unsafe-memory-access` option; Java 23+ needs it for this setup.
Use `executeUpdate` for writes and verify them with a fresh SELECT. Keep
autocommit enabled and execute statements separately.

## Validation and limits

Run `make jdbc-smoke` against a disposable running server. Override the address
with `JDBC_URL` and the JDK with `JAVA=/path/to/bin/java`. The smoke suite tests
metadata, prepared parameters, writes, sessions and concurrent statement/result
reads. See the [JDBC caveats](../client-compatibility.md#jdbc-type-information-and-schema-caveats)
for empty `getTypeInfo()` and prepared statements without a result schema.

For named datasets, use the [native Java Flight SDK](https://arrow.apache.org/java/current/flight.html)
with the server's [dataset API](../data-access.md#datasets); JDBC does not expose
that API. This is an unvalidated development route. Remove the shared sample
table with the [fixture cleanup](README.md#optional-sample-sql-table).
45 changes: 45 additions & 0 deletions docs/integrations/jdbc-applications.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Other JDBC applications

[All integration guides](README.md). Applications that accept a custom JDBC
driver can try Flight SQL. The JDBC driver is validated; each application's
metadata handling, generated SQL and write behavior still needs acceptance testing.

## Configure a custom driver

1. Start a server and create the [sample SQL table](README.md#optional-sample-sql-table).
2. Run `make bin/flight-sql-jdbc-driver-19.0.0.jar` from the repository root.
Add that JAR to the application's driver libraries, or use Maven coordinate
`org.apache.arrow:flight-sql-jdbc-driver:19.0.0`.
3. Set the driver class to `org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver`.
4. Set URL to `jdbc:arrow-flight-sql://localhost:8815/?useEncryption=false`
for the local unauthenticated endpoint. Use the actual host from the client.
5. Configure the application's JVM with
`--add-opens=java.base/java.nio=ALL-UNNAMED`. On Java 23+ also add
`--sun-misc-unsafe-memory-access=allow`, then restart the application.
6. Use autocommit. Browse catalog `redis`, schema `public`, and table
`flight_tool_sample`.

For TLS, credentials, session behavior and long-result timeouts, follow the
[JDBC connection reference](../client-guide.md#jdbc-dbeaver-and-others) and
[official driver options](https://arrow.apache.org/java/current/flight_sql_jdbc_driver.html).

## Check application behavior

Run these statements separately in the application's SQL editor:

```sql
SELECT id, label, amount FROM flight_tool_sample ORDER BY id;
SELECT COUNT(*) AS n, SUM(amount) AS total FROM flight_tool_sample;
```

Expect ids 1–3, a NULL label for id 2, and aggregate values `3` and `7.5`.
Check table/column discovery and generated filters separately. If writes are
part of the workflow, insert, update and delete a disposable row through that
application and verify each operation with a SELECT. Fixture creation through
another client does not prove application write support.

Run `make jdbc-smoke` to distinguish a driver/server issue from an application
issue. The driver's empty `getTypeInfo()` can affect application type discovery;
see [known caveats](../client-compatibility.md#jdbc-type-information-and-schema-caveats).
SQL tools see SQL tables; named datasets require a native Flight client.
Finish with the [sample cleanup](README.md#optional-sample-sql-table).
Loading