Skip to content

Support recoverable training continuation across scheduler allocations - #194

Merged
francois-drielsma merged 7 commits into
mainfrom
feature/training-allocation-continuation
Nov 18, 2026
Merged

francois-drielsma merged 7 commits into
mainfrom
feature/training-allocation-continuation

Conversation

@francois-drielsma

@francois-drielsma francois-drielsma commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Training that outlives a scheduler allocation needs a recoverable pause rather than successful scientific completion. This adds a compute-side continuation contract that checkpoints at collective boundaries and publishes a checksum-verified outcome after cleanup; scheduler submission and accounting stay in spine-prod.

  • Preserve optimizer, scheduler, RNG/loader progress, validation history, early-stopping patience, and best-checkpoint selection across allocation pauses.
  • Interrupt validation collectively, discard partial metrics, and replay pending validation before further training, including when the training limit has already been reached. Preserve partial logs for diagnosis.
  • Anchor scheduled validation/checkpointing to global iteration zero across resumes. Default initial validation off for random initialization and on for loaded weights; retain explicit validation.initial overrides.
  • Document the environment contract and add native CPU regression tests. New helper methods use typed signatures and NumPy-style docstrings; repository formatting/lint hooks pass.

Validation: 194 native driver/model/validation/CLI/continuation tests passed; 50 additional checkpoint, I/O manager, run-control and torch-runtime tests passed. Both use CPU PyTorch. The branch is based directly on current main (e8d9ca8), with no unrelated PIMM or other working-tree changes.

Site evidence from the equivalent v4 patch on NERSC: four allocations exited successfully, all four ranks advanced contiguously through iteration 3561, validation ran at global iterations 979/1959/2939, and checkpoint hashes verified. The production controller stopped at budget exhaustion after 58m03s of accounted runtime. Interrupted-validation replay is covered by CPU tests but remains pending as a fresh NERSC integration check; Polaris and multi-node GPU behavior have not been site-validated. A mixed-version v3 run failed with loader StopIteration; its cause remains unresolved and it is not acceptance evidence.

Companion controller work: DeepLearnPhysics/spine-prod#50. This PR exposes capability version 2; the existing JSON outcome schema remains version 1.

@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (e8d9ca8) to head (9013275).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##              main      #194    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files          444       445     +1     
  Lines        39748     39886   +138     
==========================================
+ Hits         39748     39886   +138     
Flag Coverage Δ
unittests 100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/spine/bin/cli.py 100.00% <100.00%> (ø)
src/spine/continuation.py 100.00% <100.00%> (ø)
src/spine/driver.py 100.00% <100.00%> (ø)
src/spine/model/manager.py 100.00% <100.00%> (ø)
src/spine/model/validation.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@francois-drielsma
francois-drielsma merged commit df897ad into main Nov 18, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant