[python] Fix optional checksum compatibility for OSS and custom S3 endpoints

[XiaoHongbo-Hope]

Purpose

While upgrading PyArrow for LeRobot integration in #9821, we found that PyArrow 22/23 sends optional checksum trailers absent in 19. Real OSS rejects them with HTTP 400: STREAMING-UNSIGNED-PAYLOAD-TRAILER is not supported.

WHEN_REQUIRED fixes the tested writes by omitting optional checksums. A Content-MD5 header can retain checksum validation, but PyArrow exposes neither per-client checksum configuration nor per-part checksum headers. That alternative requires Arrow changes or another storage backend.

On PyArrow 22+, automatically default AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED for PyArrow-backed OSS and explicit S3 endpoints (including AWS), including subprocess reconstruction. No user setup is needed; explicit values win and required checksums remain enabled.

This is process-wide. Set fs.s3.checksum-compatibility.auto-configure=false on all relevant catalogs before client creation to prevent configuration; existing settings remain. Older PyArrow versions do not apply this default.

Validation

• PyArrow 19: 54 passed, 1 skipped; 23: 55 passed. Covers version boundaries, configuration, spawn/fork, and real Arrow HTTP writes.
• Real OSS (Hangzhou public endpoint): 19 default write/read passed; 22/23 failed by default and passed with WHEN_REQUIRED. Test objects and multipart uploads cleaned up. DLF was not rerun.
• Flake8, Python 3.6 syntax and diff checks pass.

[JingsongLi]

+1