datafusion-variant

This crate provides a Variant SQL type and functions for querying semi-structured data using Apache DataFusion, and reading and writing such data to Apache Parquet files.

This crate is under active development and targets Apache Spark Variant behavior. Compatibility is incomplete, and APIs may change. See the roadmap.

What is Variant?

Variant is a type for storing semi-structured data, similar to JSON. Each value can hold objects, arrays, and primitive values, with different fields and types in each row.

For example, a single Variant column can contain all of these values, shown as JSON:

Value	Type
{"service":"api","status":200}	Object
["cached","retry"]	Array
42	Integer
"api"	String

Variant also supports "shredding" as an optimization for the common case where the values share a common structure. For more background, see Variant Type in Apache Parquet for Semi-Structured Data.

Quick start

First, enable Variant in your project by adding the following to Cargo.toml:

[dependencies]
datafusion-variant = { git = "https://github.com/datafusion-contrib/datafusion-variant.git" }
datafusion = { git = "https://github.com/apache/datafusion.git", rev = "484183dc34ea565190d9cd4a66db6d24789f95d0" } # match this crate
parquet = { version = "60", features = ["variant_experimental"] } # enable Parquet Variant support
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

Then, configure your DataFusion SessionContext to use the Variant functions, SQL field access, and VARIANT type with register_all. See the Apache DataFusion documentation for more details on creating a session and executing queries.

use datafusion::prelude::*;
use datafusion_variant::register_all;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let mut ctx = SessionContext::new();
    register_all(&mut ctx)?;
    Ok(())
}

Warning

Call register_all during session setup, before preparing statements. It replaces existing functions with the same names and the type planner, and clears prepared statements.

Congratulations -- you can now query semi-structured data with the Variant type!

Selective registration

To register individual functions, wrap implementations with ScalarUDF::new_from_impl or AggregateUDF::new_from_impl, then call ctx.register_udf or ctx.register_udaf. Use register_variant_get_functions for getters and casts from VARIANT, register_variant_conversion_functions for casts to VARIANT and to_variant_object, and register_variant_explode_functions for table functions. Install VariantExprPlanner with ctx.register_expr_planner(Arc::new(VariantExprPlanner)) for : access, and VariantTypePlanner with SessionStateBuilder::with_type_planner for SQL VARIANT syntax.

Common tasks

This section shows common queries with the Variant type. You can run the SQL examples like any other DataFusion SQL statement using the SessionContext, e.g. with ctx.sql(sql).await?.collect().await?.

This repository also includes a dfv command-line tool with the Variant functionality preinstalled to try the examples interactively. Run it from the repository root with:

# Drop `--release` for faster compile times if you aren't measuring performance.
cargo run --release

Convert JSON to Variant

parse_json converts JSON strings to Variant. You can use it to load semi-structured documents. For example, create an events table with a Variant column named event:

CREATE TABLE events AS
SELECT parse_json(json_data) AS event
FROM (VALUES
    ('{"service":"api","status":200,"latency_ms":12,"tags":["cached"]}'),
    ('{"service":"api","status":500,"latency_ms":85,"error":null}'),
    ('{"service":"api","status":504,"error":"Somethings not working"}')
) AS input(json_data);

Tip

For potentially malformed input, use try_parse_json(json_data) to return SQL NULL instead of failing the query.

Query

You can extract fields from a Variant column using the Spark SQL-style : field-access syntax or the variant_get UDF. For example, to select the service and latency_ms fields of events whose status is at least 400, run:

SELECT event:service::VARCHAR AS service,
       event:latency_ms::BIGINT AS latency_ms
FROM events
WHERE event:status::BIGINT >= 400;

+---------+------------+
| service | latency_ms |
+---------+------------+
| api     | 85         |
| api     | NULL       |
+---------+------------+

The : syntax is automatically rewritten to variant_get, so this is an equivalent query:

SELECT variant_get(event, 'service', 'VARCHAR') AS service,
       variant_get(event, 'latency_ms', 'BIGINT') AS latency_ms
FROM events
WHERE variant_get(event, 'status', 'BIGINT') >= 400;

+---------+------------+
| service | latency_ms |
+---------+------------+
| api     | 85         |
| api     | NULL       |
+---------+------------+

Store Variant in Parquet

You can write Variant columns to Parquet with the DataFusion COPY statement:

COPY events TO 'events.parquet' STORED AS PARQUET;

+-------+
| count |
+-------+
| 3     |
+-------+

The Variant column is stored in Parquet as a struct with metadata and value fields, as you can see with the parquet_metadata table function:

-- Note: parquet_metadata is part of dfv and datafusion-cli, not this crate.
SELECT path_in_schema, type FROM parquet_metadata('events.parquet');

+------------------+------------+
| path_in_schema   | type       |
+------------------+------------+
| "event.metadata" | BYTE_ARRAY |
| "event.value"    | BYTE_ARRAY |
+------------------+------------+

Read Variant from Parquet

You can read events.parquet back with the Variant column preserved.

Warning

DataFusion's Parquet reader ignores the schema metadata that identifies the Variant column by default today, so you must set datafusion.execution.parquet.skip_metadata to false first.

-- Preserve the schema information that identifies the Variant column.
SET datafusion.execution.parquet.skip_metadata = false;

SELECT variant_get(event, 'service', 'VARCHAR') AS service,
       event:tags[0]::VARCHAR AS first_tag -- Array indexing via `[]` operator
FROM 'events.parquet'
ORDER BY event:status::BIGINT;

+---------+-----------+
| service | first_tag |
+---------+-----------+
| api     | cached    |
| api     | NULL      |
| api     | NULL      |
+---------+-----------+

Convert between Variant and other SQL types

CAST(value AS VARIANT) converts SQL values to Variant. For example, to convert an integer and a decimal to Variant and show them as JSON, run:

SELECT CAST(42 AS VARIANT) AS int_value,
       CAST(1.23 AS VARIANT) AS decimal_value;

+-----------+---------------+
| int_value | decimal_value |
+-----------+---------------+
| 42        | 1.23          |
+-----------+---------------+

To convert a struct or list to Variant, use to_variant_object:

SELECT to_variant_object([1, 2, 3]) AS list_value,
       to_variant_object([{a: 1, b: 2}, {b: 3, a: 4}]) AS struct_list_value;

+------------+-------------------------------+
| list_value | struct_list_value             |
+------------+-------------------------------+
| [1,2,3]    | [{"a":1,"b":2},{"a":4,"b":3}] |
+------------+-------------------------------+

Inspect the schema of Variant values

Due to their dynamic nature, Variant values can have different fields and types in each row. You can inspect the schema of a Variant column using the following functions:

-- Get the schema of each Variant value
SELECT schema_of_variant(event) FROM events;

+----------------------------------------------------------------------------------+
| schema_of_variant(events.event)                                                  |
+----------------------------------------------------------------------------------+
| OBJECT<latency_ms: BIGINT, service: STRING, status: BIGINT, tags: ARRAY<STRING>> |
| OBJECT<error: VOID, latency_ms: BIGINT, service: STRING, status: BIGINT>         |
| OBJECT<error: STRING, service: STRING, status: BIGINT>                           |
+----------------------------------------------------------------------------------+

-- Print the merged schema (union) of all Variant values in the column
SELECT schema_of_variant_agg(event) AS schema FROM events;

+-------------------------------------------------------------------------------------------------+
| schema                                                                                          |
+-------------------------------------------------------------------------------------------------+
| OBJECT<error: STRING, latency_ms: BIGINT, service: STRING, status: BIGINT, tags: ARRAY<STRING>> |
+-------------------------------------------------------------------------------------------------+

Functions

See the generated SQL reference for signatures, arguments, and examples for documented scalar and aggregate functions.

Common pitfalls

Extract a field as a SQL type before filtering, aggregating, or joining on it. A missing field returns SQL NULL. A JSON null is a distinct Variant null value: use is_variant_null to detect it, and IS NULL to detect SQL NULL.

CAST(value AS VARIANT) accepts scalars and arrays; structs and string-keyed maps require to_variant_object.

More examples and benchmarks

For an interactive SQL shell with the Variant functions preinstalled, see dfv (#102).

See the benchmark guide for extraction, parsing, serialization, schema inference, and JSONBench workloads, including baseline comparison commands.

Contributing

Contributions are welcome, including compatibility fixes, regression tests, benchmarks, documentation, and integration examples.

Use a current stable Rust toolchain with rustfmt and clippy, and install the Protocol Buffers compiler (protoc).

Run the development checks from the repository root:

cargo test
cargo fmt --all -- --check
cargo clippy -- -D warnings

cargo test also checks this README's Rust example and SQL results.

Regenerate the SQL reference and check its examples:

cargo run --example generate_function_docs
cargo test --example generate_function_docs

For Spark compatibility changes, reference the upstream behavior and add regression coverage in the SQL tests.

The issue tracker contains individual tasks and discussions. Overall direction is tracked in #69.

References

• Apache Spark 4.2.0 documentation
• Parquet Variant encoding
• Arrow canonical extension types
• Custom types using metadata in DataFusion
• Iceberg Variant proposal — design background

License

Licensed under the Apache License, Version 2.0.