DM-53494: Add ability to run submit processes as batch jobs.

doc/changes/DM-53494.feature.rst

@@ -0,0 +1 @@
+Added ability to run submit processes as batch jobs on same cluster with shared filesystems.

doc/lsst.ctrl.bps/quickstart.rst

@@ -1613,6 +1613,49 @@ Parsl).
     responsibility to remove them once no longer needed.  The removal
     should be done regularly to avoid too many in single directory.
 
+.. _bps_submit_as_batch:
+
+Submit Stages as Batch Jobs
+---------------------------
+
+.. warning::
+
+   This feature is part of the ongoing work for submission to remote sites.
+   So details may change.  Also currently only the HTCondor plugin supports
+   this.
+
+In cases where one cannot run ``bps submit`` interactively (e.g., needs
+too much memory), BPS can run the submit processes as batch jobs after
+which the payload workflow will start running.
+
+The interactive submission process, ``bps batch-submit <submit yaml>``,
+is much shorter.  It will create a workflow with two jobs:
+
+- ``buildQuantumGraph`` which creates the quantum graph.
+- ``preparePayloadWorkflow`` which does the rest of the submission stages seen
+  when running ``bps submit``.  These include clustering, creation of the payload
+  workflow, and preparing the WMS-specific workflow.
+
+One can set runtime values specific to those jobs (e.g., ``requestMemory``) in
+sections with corresponding names similar to ``finalJob``.  Currently the
+logging-related command-line arguments aren't passed from ``bps batch-submit``
+to these jobs.  Instead, one can set ``bpsPreCommandOpts``, which has the
+same default as the payload job.
+
+Even with the new jobs, there is only one output run collection, one submit
+directory and one top level WMS ID to be used with BPS commands.
+
+``bps report`` will show these 2 new jobs same as the payload jobs.  They
+will be the only lines to appear in the report until the ``preparePayloadWorkflow``
+has finished at which time the expected payload lines should appear (from
+``pipetaskInit`` through ``finalJob``).
+
+``bps cancel`` can be used to abort the run during these new jobs or later
+during the running of the payload jobs.
+
+See the corresponding section in the WMS-plugin documentation for additional
+information.
+
 .. _bps-troubleshooting:
 
 Troubleshooting