README.md

Spark Connect Rust Client

A fast, native Rust client for Apache Spark Connect - and a drop-in pyspark replacement with 100% public-API parity with PySpark 4.3.0. It builds spark.connect protobuf plans, manages the gRPC channel, and decodes Arrow results in Rust, speaking the same protocol and returning the same results as the reference client.

📖 Documentation

Full documentation lives at apache.github.io/spark-connect-rust

• installation, quickstart, the DataFrame / Columns / SQL / Reading & Writing / Streaming / Catalog / Types API, Rust UDFs via WebAssembly, and the architecture.

The Rust API reference (generated from the crate) is on docs.rs/apache-spark-connect.

Install

Python - a faster, drop-in replacement for the pyspark-client PyPI package (uninstall any existing pyspark / pyspark-client first):

pip install pyspark-client-rust

Your Spark Connect code then runs unchanged; use it exactly like PySpark.

Rust - the native crate:

[dependencies]
apache-spark-connect = "4.3"

Quickstart (Rust)

use spark_connect::{SparkSession, functions as f, lit};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let spark = SparkSession::builder()
        .remote("sc://localhost:15002")
        .get_or_create()?;

    let df = spark
        .range(1_000_000)?
        .select([(f::col("id") * lit(2)).alias("x")])
        .filter((f::col("x") % lit(3)).eq(lit(0)));

    println!("count = {}", df.count()?);
    df.show(20)?;
    Ok(())
}

See the documentation for the full API, running a Spark Connect server, and more.

User-Defined Functions (UDFs)

Write UDFs as plain Rust functions and call them directly — #[spark_wasm_udf] compiles them to WebAssembly and ships them to the executors:

#[spark_wasm_udf]
mod udfs {
    pub fn add_one(x: i64) -> i64 { x + 1 }
}

spark.range(5)?.select([udf::add_one(col("id"))?]).show(20)?;

See the WASM UDF guide for the full setup, SQL registration, supported types, and more.

Contributing

Issues are tracked in ASF JIRA under SPARK (GitHub Issues are disabled). See the contributing guide.

License

Apache License 2.0.