Skip to content

feat(workflows): re-run a workflow from a chosen step, reusing earlier results - #684

Open
kthare10 wants to merge 1 commit into
developfrom
feat/rerun-from-step
Open

kthare10 wants to merge 1 commit into
developfrom
feat/rerun-from-step

Conversation

@kthare10

Copy link
Copy Markdown
Contributor

Closes #680

Changing only the plot at the end of a workflow used to re-run everything. With this change, a run, from the Run panel or from the assistant, can start from a chosen step and reuse what earlier runs left in the output folder for the steps before it. Keeping intermediates, so a re-run can start mid-workflow, is the follow-up #683.

How

Pegasus already skips a job whose outputs are all in the replica catalog. It had nothing to find: each run's catalog sits in its own submit directory, and generated workflows commonly set register_replica=False, so even --reuse finds nothing.

services/rerun.py writes the catalog itself, from files that exist on disk:

  • Replica file: lists earlier outputs of every step that isn't being re-run. The chosen step and everything downstream are left out. It goes to the planner as --inherited-rc-files, which has lower priority than the workflow's own catalog, so it can only add locations, never override one. Nothing in the user's workflow changes.
  • What counts as current: comes from the run directories (the abstract workflow copy Pegasus keeps, and the stampede DB), not from a record Studio keeps, so runs started from a terminal count too. The newest run in which a file's producing job succeeded owns that file. The file is reused only if it was written after that job started. This covers a stage-out that failed, a failed retry (it doesn't claim the file), and partial re-runs (jobs a run didn't run don't claim anything).
  • Preview: mirrors Pegasus's pruning rule, measured on 6.0.0.dev1. A job is dropped if all its outputs are listed. A job is also dropped if all its children were dropped and none of its own staged-out outputs is missing. The Run flow then reports what Pegasus actually planned, read from the DAG.

Where it shows up:

  • Run flow: rerun_from on the request. An unknown step fails the flow instead of quietly running everything, since running everything is what Re-run a workflow from a chosen step, reusing earlier results #680 was about.
  • API: GET /projects/{id}/steps and GET /projects/{id}/rerun-preview?from_step= (writes nothing).
  • Run panel: a "Start from" choice (default: the beginning) with the preview: what runs, what's reused and from which run, which intermediates were never kept. It resets after each Run, so a partial run is never inherited.
  • Assistant: a prepare_rerun MCP tool. It writes the replica file and returns the preview plus the pegasus-plan … --inherited-rc-files … command, which the assistant runs through the gated shell as it does any plan. The tool itself starts nothing. STUDIO_ENV.md tells the assistant to use it when only a step changed.

Verified

  • Pegasus 6.0.0.dev1, in a pod, planning a copy of climate-data-analysis-workflow (prep_data → analyze_station ×3 → aggregate), without submitting:
    • A replica file listing the analyze outputs: only aggregate is planned, with --cache and --inherited-rc-files alike.
    • One of those files missing: aggregate, that station's analyze, and prep_data are planned; the other two analyze jobs are reused.
    • Only aggregate's outputs listed: the analyze jobs are not dropped, because their staged outputs are missing.
    • Quoted paths in the replica file parse.
    • --reuse would reuse nothing, because the workflow registers no replicas.
  • rerun.py against that real run (preview only): from aggregate, 1 step runs and 4 are reused (6 files from run0001). From analyze_station, everything runs, and the preview names the three station_*.csv intermediates as the reason. Both match Pegasus above.
  • Tests: pytest 1086 passed (16 new in test_rerun.py, 2 in test_run_flow.py), tsc --noEmit clean, vitest passes.

Not verified

  • A submitted partial run end to end: the reused files staged in from output/ and aggregate running on the pool. Planning shows the stage-in jobs, but none of these runs was submitted.
  • The Run panel and the assistant tool in a browser on :develop.

Limits

  • Workflows with sub-workflows are refused with a message, since this doesn't look inside them.
  • The preview is computed from the workflow as last generated. Run regenerates first and looks the step up again by name, and the plan stage reports what actually ran.
  • A file that was overwritten with output from different generator parameters, but kept the same name, would be reused. The preview names the run each reused file came from.

…r results

Changing the plot at the end of a workflow cost a full re-run: Studio never
told Pegasus where earlier results were, and generated workflows commonly set
register_replica=False, so even `pegasus-plan --reuse` finds nothing.

services/rerun.py writes the catalog itself, from files that exist: a replica
file listing earlier outputs of every step not being re-run, passed to the
planner with --inherited-rc-files. Which files are current comes from the run
directories (the abstract workflow copy and the stampede DB): the newest run
in which a file's producer succeeded owns it, and the file is reused only if
written after that job started. The preview mirrors Pegasus's pruning as
measured on 6.0.0.dev1; the Run flow reports what Pegasus actually planned.

- Run flow: rerun_from on the request; an unknown step fails the flow
  rather than quietly running everything.
- API: GET /projects/{id}/steps and /projects/{id}/rerun-preview.
- Run panel: a "Start from" choice with the preview (what runs, what is
  reused and from which run, which intermediates were never kept).
- Assistant: a prepare_rerun MCP tool that writes the replica file and
  returns the pegasus-plan command, which runs through the gated shell; the
  Studio environment guide tells the assistant to use it.

Refs #680
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant