[HyukjinKwon]

What changes were proposed in this pull request?

SPARK-61154: Raise VALUE_NOT_BETWEEN for a literal of an int beyond the range of a long, in the Python client as PySpark does.

• A Python int (not a bool) that does not fit in a long is an error wherever it becomes a literal, instead of a double. This covers lit, the Column operators and comparisons, isin, between, when/otherwise, array, list literals and pivot values. The error is PySparkValueError VALUE_NOT_BETWEEN with arg_name "value", min "-9223372036854775808" and max "32767", as LiteralExpression._infer_type raises it (its maximum is JVM_SHORT_MAX, which PySpark writes this way).
• fillna and na.fill with such an int raise ValueError: Value out of range: <n>, as PySpark's proto raises for an out-of-range long.
• Floats, bools, NumPy scalars and ints within the range of a long are unchanged. createDataFrame and spark.sql(..., kwargs) keep their results, which already matched PySpark.

Official PySpark 4.3.0 for the same expressions (checked against the same server): df.where(F.col("a") == 2**63), F.lit(2**70), F.col("a") > 2**70, isin, between, when, array, pivot raise PySparkValueError VALUE_NOT_BETWEEN; fillna(2**70) raises ValueError Value out of range: 1180591620717411303424.

Why are the changes needed?

A Python int beyond the int64 range was silently converted to a double. The comparison df.where(F.col("a") == 2**63) returned the rows with a = 9223372036854775807 and a = 9223372036854775806, because the double rounds, and F.col("a") > 2**70 returned no rows. PySpark raises for these values.

Does this PR introduce any user-facing change?

Yes. An int beyond the range of a long raises PySparkValueError VALUE_NOT_BETWEEN where it used to become a double, and fillna raises ValueError for it.

How was this patch tested?

• test_int_literals_beyond_a_long_raise_value_not_between_as_pyspark (offline, no server): lit, ==, >, isin, between, when, array and list literals raise VALUE_NOT_BETWEEN with the message parameters of PySpark; fillna(2**70) raises the ValueError; the bounds of a long and a float beyond them are accepted. The same cases give the same results with official PySpark 4.3.0.
• The lib tests of both crates (590 passed), the build of all tests, also with wasm-udf, and the offline Python suites (2202 passed, 8 skipped), and the live tests of literals, isin, between, fillna, when and pivot (51 passed) against a local Spark 4.3.0 server; rustfmt passes, and clippy reports no diagnostics in the changed lines.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.