Chore: remove unaltered python

[TaiPee]

Slim the FairGBM Python package by removing unaltered upstream files

Goal

Make the FairGBM Python package a lighter, easier-to-maintain fork of LightGBM —
mirroring what PR #58 did for the C++ sources. Instead of vendoring a full copy of
LightGBM's Python modules, we keep only the files FairGBM actually modifies and fetch
the untouched ones from pristine upstream at build time. This makes it trivial to see
exactly what FairGBM changes relative to upstream.

As a prerequisite for validating the change against a modern environment, this PR also
fixes two compatibility issues (scikit-learn ≥ 1.6 and NumPy ≥ 2.0) that otherwise block
a fresh install.

Thinking behind the changes

Which files are safe to remove?

I compared every file in python-package/fairgbm/ against a pristine checkout of
LightGBM v3.2.1 (the same upstream version pinned by the C++ overlay in PR #58,
confirmed by comparing basic.py/engine.py/dask.py against v3.2.1 vs v3.3.x).

File	Diff vs v3.2.1	Decision
callback.py	byte-identical	remove
libpath.py	byte-identical	remove
plotting.py	byte-identical	remove
engine.py	docstrings only (typo fixes + reworded gradient/Hessian notes), zero functional change	remove
basic.py, sklearn.py, compat.py, dask.py, __init__.py, setup.py, MANIFEST.in	real FairGBM changes	keep

These four modules use only relative imports and contain no lightgbm string references,
so upstream's copies drop into the fairgbm package verbatim — no content rewriting needed.

How to remove them without breaking imports

Unlike the C++ side (compiled), the Python package is imported directly, and the removed
modules are imported by __init__.py, engine.py, and basic.py. So they cannot simply
be deleted. The solution does the following at build time, setup.py
shallow-clones LightGBM v3.2.1 and copies the modules into the package before packaging.
MANIFEST.in (recursive-include fairgbm ... *.py) and include_package_data=True then
bundle them into the wheel/sdist automatically. Published wheels/sdists are unaffected and
need no network — only a build from a raw git checkout fetches upstream, the same trade-off
PR #58 introduces for C++.

Why the compatibility fixes are included

Simulating a fresh-user install on a current toolchain (Python 3.12, NumPy 2.1, pandas 2.2,
scikit-learn 1.8) surfaced two blockers unrelated to the file removal but required for the
package to work at all today:

• scikit-learn ≥ 1.6 renamed force_all_finite → ensure_all_finite (removed in 1.8),
  breaking FairGBMClassifier.fit().
• NumPy ≥ 2.0 changed np.array(obj, copy=False) to raise instead of copying when a
  copy is unavoidable, breaking data ingestion and cross-validation.

The NumPy fix has to be applied both to first-party code (basic.py) and to the fetched
upstream engine.py; the latter is patched by the overlay so it stays out of git while
still producing working code.

What changed

Three commits:

1. build(python-package): slim fork by fetching unaltered upstream modules

   • Deletes callback.py, libpath.py, plotting.py, engine.py (−1,522 lines).
   • Adds overlay_upstream_python_files() to setup.py: shallow + sparse clone of
     LightGBM v3.2.1, copies the four modules into fairgbm/. Idempotent and offline-safe
     (skips the fetch when the files are already present, e.g. in a published sdist).
   • The overlay also rewrites np.array(..., copy=False) → np.asarray(...) in the fetched
     files for NumPy ≥ 2.0 compatibility.
   • Git-ignores the four build-time files so they are never committed.

2. fix(python-package): support scikit-learn >= 1.6 validation kwargs

   • Wraps check_X_y/check_array in compat.py to translate force_all_finite →
     ensure_all_finite based on the installed scikit-learn signature (works across versions).

3. fix(python-package): NumPy >= 2.0 compatibility in array handling

   • Replaces the eight np.array(..., copy=False) calls in basic.py with np.asarray(...),
     preserving copy-if-needed behaviour on both NumPy 1.x and 2.x.

Testing

Validated end-to-end from a clean checkout (artifacts removed, pip uninstall), simulating
a new user:

• pip install ./python-package compiled the C++ library and built/installed the wheel
  (CMake 4.3.1, g++ 11.4, Python 3.12.5).
• Verified the installed wheel bundles callback.py, libpath.py, plotting.py,
  engine.py, and lib_lightgbm.so, and that the overlaid engine.py is NumPy-2-safe.
• Import + functional smoke test (FairGBMClassifier.fit / predict_proba) passes.
• Full tests/python_package_test/ suite: 118 passed, 11 skipped, 0 failed.
  (Skips are optional-dependency tests — aequitas, dask — via pytest.importorskip.)

Notes / follow-ups

• Build-time network: building from a git checkout now requires network access to fetch
  LightGBM v3.2.1 (mirrors PR Remove all unaltered C++ code #58). Published artifacts are unaffected.
• Pin drift: UPSTREAM_LIGHTGBM_TAG must stay in sync with the C++ FetchContent pin
  (v3.2.1); it is kept in a single constant.
• Relationship to PR Remove all unaltered C++ code #58: independent (disjoint files, no merge conflict, any order).
  Worth validating the combined C++ + Python build in CI once both land.
• The upstream Python files are MIT-licensed; their headers are left intact.