update method layout docs and add a CI check for it

.github/PULL_REQUEST_TEMPLATE/new_sr_method.md

@@ -7,22 +7,42 @@ If you need help with the PR, feel free to tag @srbench-comp and we'll respond a
 -->
 ## Submission Checklist
 
+A submission spans **two** directories, using the same name in both. See
+[CONTRIBUTING.md](../../CONTRIBUTING.md#where-your-files-go) for the details.
+
 - [ ] title of this PR is meaningful, i.e. "adding method X"
-- [ ] A folder has been added to `algorithms/` with a meaningful name corresponding to your method name.
-- [ ] The added folder includes these elements:
-    - [ ] `metadata.yml` (**required**): A file describing your submission, following the descriptions in (algorithms/feat/metadata.yml). 
-    - [ ] `regressor.py` (**required**): a Python file that defines your method, named appropriately. See [algorithms/feat/regressor.py][regressor] for complete documentation. 
-      `regressor.py` contains:
-      - [ ] `est`: a sklearn-compatible `Regressor` object. 
-      - [ ]  `model(est, X=None)`: a function that returns a [**sympy-compatible**](https://www.sympy.org) string specifying the final model. It can optionally take the training data as an input argument. 
-      - [ ]  `eval_kwargs` *(optional)*: a dictionary that can specify method-specific arguments to `evaluate_model.py`.
-    - [ ] `LICENSE` *(optional)* A license file
-    - [ ] `environment.yml` *(optional)*: a [conda environment file](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file) that specifies dependencies for your submission. 
-  It will be used to update the baseline environment (`environment.yml` in the root directory). 
-  To the extent possible, conda should be used to specify the dependencies you need. 
-  If your method is part of conda, great! You can just put that in here and leave `install.sh` blank. 
-    - [ ] `requirements.txt` *(optional)*: a pypi requirements file. The script will run `pip install -r requirements.txt` if this file is found, before proceeding.
-    - [ ] `install.sh` *(optional)*: a bash script that installs your method **without sudo permissions**. 
-- [ ] I did not include source code; instead I used `install.sh` to pull it from a stable source repository. 
-
-- [ ] I locally tested that `bash local_ci.sh [method-folder-name]` runs successfully without error. 
+
+**`algorithms/<your-method>/`** — how to install your method
+
+- [ ] `metadata.yml` (**required**): A file describing your submission, following the descriptions in [algorithms/feat/metadata.yml][metadata]. `name`, `authors`, `email`, `description` and `url` are filled in.
+- [ ] `LICENSE` *(optional)* A license file
+- [ ] `environment.yml` *(optional)*: a [conda environment file](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file) that specifies dependencies for your submission.
+  It will be used to update the baseline environment (`base_environment.yml` in the root directory).
+  To the extent possible, conda should be used to specify the dependencies you need.
+  If your method is part of conda, great! You can just put that in here and leave `install.sh` blank.
+- [ ] `requirements.txt` *(optional)*: a pypi requirements file. The script will run `pip install -r requirements.txt` if this file is found, before proceeding.
+- [ ] `install.sh` *(optional)*: a bash script that installs your method **without sudo permissions**.
+- [ ] I did not include source code; instead I used `install.sh` to pull it from a stable source repository.
+
+**`experiment/methods/<your-method>/`** — how to call your method
+
+- [ ] `regressor.py` (**required**): a Python file that defines your method. See [experiment/methods/feat/regressor.py][regressor] for complete documentation.
+  `regressor.py` contains:
+  - [ ] `est`: a sklearn-compatible `Regressor` object.
+  - [ ] `model(est, X=None)`: a function that returns a [**sympy-compatible**](https://www.sympy.org) string specifying the final model. It can optionally take the training data as an input argument.
+  - [ ] `eval_kwargs` *(optional)*: a dictionary that can specify method-specific arguments to `evaluate_model.py`.
+- [ ] `__init__.py` (**required**): an empty file, so the harness can import your method.
+
+**Checks**
+
+- [ ] `python scripts/check_method_layout.py` passes.
+- [ ] I locally tested my method with:
+      ```
+      bash scripts/make_docker_compose_file.sh
+      docker compose build base
+      docker compose build <your-method>
+      docker compose run --rm <your-method> bash test.sh
+      ```
+
+[metadata]: https://github.com/cavalab/srbench/blob/master/algorithms/feat/metadata.yml
+[regressor]: https://github.com/cavalab/srbench/blob/master/experiment/methods/feat/regressor.py 

.github/workflows/ci-docker.yml

@@ -33,6 +33,19 @@ on:
       - '.github/workflows/**'
 
 jobs:
+  # Fast structural check. The build-and-test matrix below is generated from
+  # `ls algorithms/`, so a method added only under experiment/methods/ would
+  # otherwise get a full set of green checks without ever being built.
+  validate-layout:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - run: pip install pyyaml
+      - run: python scripts/check_method_layout.py
+
   list-algs:
     runs-on: ubuntu-latest
     outputs:
@@ -128,7 +141,8 @@ jobs:
 
   build-and-test:
     runs-on: ubuntu-latest
-    needs: 
+    needs:
+      - validate-layout
       - check-changes
       - print-changes
       - list-algs
@@ -139,7 +153,10 @@ jobs:
       matrix:
         alg: ${{ fromJson(needs.list-algs.outputs.matrix) }}
       fail-fast: false
-    if: always()
+    # always() keeps the matrix running even when check-changes reports no
+    # changes, but a layout failure means the images would be built from a
+    # method that cannot be imported -- don't spend ~27 docker builds on that.
+    if: always() && needs.validate-layout.result == 'success'
     steps:
       - uses: actions/checkout@v4
       - name: Check if algorithm has changed

CONTRIBUTING.md

@@ -26,25 +26,76 @@ You can leverage this code base and previous experimental results to do so.
 - An open-source method with a [scikit-learn compatible API](https://scikit-learn.org/stable/developers/develop.html)
 - Your method should be compatible with **Python 3.7 or higher** to ensure compatibility with conda-forge.
 - If your method uses a random seed, it should have a `random_state` attribute that can be set.
-- Methods must have their own folders in the `algorithms` directory (e.g., `algorithms/feat`). 
-This folder should contain:
-  1. `metadata.yml` (**required**): A file describing your submission, following the descriptions in [algorithms/feat/metadata.yml][metadata]. 
-  2. `regressor.py` (**required**): a Python file that defines your method, named appropriately. See [algorithms/feat/regressor.py][regressor] for complete documentation. 
-      It should contain:
-      -   `est`: a sklearn-compatible `Regressor` object. 
-      -   `model(est, X=None)`: a function that returns a [**sympy-compatible**](https://www.sympy.org) string specifying the final model. It can optionally take the training data as an input argument. See [guidance below](###-returning-a-sympy-compatible-model-string). 
-      -   `eval_kwargs` (optional): a dictionary that can specify method-specific arguments to `evaluate_model.py`.
-      -   We expect your algorithm to have a `max_time` parameter that lets us control the maximum execution time in seconds. When running the experiments in a cluster, we will give extra time to compensate for the overhead of initializing everything, and the maximum time considered is just the fit process. A signal `signal.SIGALRM` will be sent to your process if `fit(X, y)` exceeds the maximum time, and you can implement strategies to handle this signal. One idea is to store a random initial solution as the best and update it during the execution to ensure the `evaluate_model.py` script will find an equation to work on.
-  3. `LICENSE` *(optional)* A license file
-  4. `environment.yml` *(optional)*: a [conda environment file](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file) that specifies dependencies for your submission. 
-  It will be used to update the baseline environment (`environment.yml` in the root directory). 
-  To the extent possible, conda should be used to specify the dependencies you need. 
-  If your method is part of conda, great! You can just put that in here and leave `install.sh` blank. 
-  5. `requirements.txt` *(optional)*: a pypi requirements file. The script will run `pip install -r requirements.txt` if this file is found, before proceeding.
-  5. `install.sh` *(optional)*: a bash script that installs your method. 
+
+### Where your files go
+
+A submission spans **two** directories, and both are required.
+This is the single most common thing to get wrong, so it is worth reading closely.
+
+```
+algorithms/<your-method>/          # how to INSTALL your method
+├── metadata.yml                   #   required
+├── environment.yml                #   optional
+├── requirements.txt               #   optional
+├── install.sh                     #   optional
+├── Dockerfile                     #   optional
+└── LICENSE                        #   optional
+
+experiment/methods/<your-method>/  # how to CALL your method
+├── regressor.py                   #   required
+└── __init__.py                    #   required (an empty file)
+```
+
+The split follows from how the benchmark runs.
+`algorithms/<your-method>/` is copied into your Docker image when it is built, so it holds everything needed to *install* your method.
+`experiment/` is mounted into the running container, so `regressor.py` is read at *run* time and is never baked into the image.
+
+Use the **same directory name** in both places.
+Note that `metadata.yml` belongs with the install files in `algorithms/`, not next to `regressor.py`.
+
+You can check your layout before opening a PR:
+
+```bash
+python scripts/check_method_layout.py
+```
+
+CI runs this same check, and it will fail your PR if anything is out of place.
+
+#### `algorithms/<your-method>/`
+
+  1. `metadata.yml` (**required**): A file describing your submission, following the descriptions in [algorithms/feat/metadata.yml][metadata]. Please fill in `name`, `authors`, `email`, `description` and `url`.
+  2. `LICENSE` *(optional)* A license file
+  3. `environment.yml` *(optional)*: a [conda environment file](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file) that specifies dependencies for your submission.
+  It will be used to update the baseline environment (`base_environment.yml` in the root directory).
+  To the extent possible, conda should be used to specify the dependencies you need.
+  If your method is part of conda, great! You can just put that in here and leave `install.sh` blank.
+  4. `requirements.txt` *(optional)*: a pypi requirements file. The script will run `pip install -r requirements.txt` if this file is found, before proceeding.
+  5. `install.sh` *(optional)*: a bash script that installs your method.
   **Note: scripts should not require sudo permissions. The library and include paths should be directed to conda environment; the environmental variable `$CONDA_PREFIX` specifies the path to the environment.
   6. `Dockerfile` *(optional)*: we will try to dockerize all algorithms. You can optionally have a `Dockerfile` inside your `algorithms/your-submission` folder to describe specific images for running your algorithm. If no file is provided, it will use `alg-Dockerfile` for your container. You can specify the image as you like, as long as you have as minimal dependences the python packages described in `base_environment.yml`, as they are used to run the experiment scripts. See [this example](algorithms/tir/Dockerfile) in case you want to use a custom image. *Notice that there is a workflow to build the docker images and push them to dockerhub*.
-  7. **do not include your source code**. use `install.sh` to pull it from a stable source repository. 
+  7. **do not include your source code**. use `install.sh` to pull it from a stable source repository.
+
+#### `experiment/methods/<your-method>/`
+
+  1. `regressor.py` (**required**): a Python file that defines your method. See [experiment/methods/feat/regressor.py][regressor] for complete documentation.
+      It should contain:
+      -   `est`: a sklearn-compatible `Regressor` object.
+      -   `model(est, X=None)`: a function that returns a [**sympy-compatible**](https://www.sympy.org) string specifying the final model. It can optionally take the training data as an input argument. See [guidance below](#model-compatibility-with-sympy).
+      -   `eval_kwargs` (optional): a dictionary that can specify method-specific arguments to `evaluate_model.py`. Only these keys are recognized: `test_params`, `max_train_samples`, `scale_x`, `scale_y`, `pre_train`, `use_dataframe`.
+      -   We expect your algorithm to have a `max_time` parameter that lets us control the maximum execution time in seconds. When running the experiments in a cluster, we will give extra time to compensate for the overhead of initializing everything, and the maximum time considered is just the fit process. A signal `signal.SIGALRM` will be sent to your process if `fit(X, y)` exceeds the maximum time, and you can implement strategies to handle this signal. One idea is to store a random initial solution as the best and update it during the execution to ensure the `evaluate_model.py` script will find an equation to work on.
+      -   The harness looks for the time limit under any of these attribute names: `max_time`, `timeout_in_seconds`, `timeout`, `stop_time`, `time_limit`. If your estimator exposes none of them, it will simply be killed when it runs long.
+  2. `__init__.py` (**required**): an empty file, so the harness can import your method.
+
+### Testing your submission locally
+
+Build your image and run the same tests CI runs:
+
+```bash
+bash scripts/make_docker_compose_file.sh   # regenerates docker-compose.yml
+docker compose build base                  # the shared base image, needed once
+docker compose build <your-method>
+docker compose run --rm <your-method> bash test.sh
+```
 
 ### model compatibility with sympy
 
@@ -64,3 +115,7 @@ def model(est, X=None):
 ```
 
 2. The operators/functions in the model are available in [sympy's function set](https://docs.sympy.org/latest/modules/functions/index.html). 
+
+[metadata]: https://github.com/cavalab/srbench/blob/master/algorithms/feat/metadata.yml
+[regressor]: https://github.com/cavalab/srbench/blob/master/experiment/methods/feat/regressor.py
+

README.md

@@ -136,7 +136,7 @@ GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/cavalab/srbench.git
 
 A detailed guide on how to reproduce the experiments by yourself is provided in [`docs/user_guide.md`](./docs/user_guide.md).
 
-Once you get all the results, you nee to collate the results using the `collate` scripts in [`./postprocessing/scripts`](./postprocessing/scripts/collate_experiments_results.py)
+Once you get all the results, you need to collate them using the `collate` scripts in [`./postprocessing/`](./postprocessing/): [`collate_blackbox_results.py`](./postprocessing/collate_blackbox_results.py) and [`collate_groundtruth_results.py`](./postprocessing/collate_groundtruth_results.py).
 
 # References
 

algorithms/eql/metadata.yml

@@ -3,7 +3,7 @@ authors: # the participants
 email: alessandro.simon@tuebingen.mpg.de
 name: EQL # name of the submission /method
 description: | # anything you'd like here to describe the method.
-Implementation of the Equation Learner architecture as described in
-'Learning equations for extrapolation and control' (Sahoo et. al)
+  Implementation of the Equation Learner architecture as described in
+  'Learning equations for extrapolation and control' (Sahoo et. al)
 
 url: https://al.is.mpg.de/research_projects/symbolic-regression-and-equation-learning # a link to the project

docs/user_guide.md

@@ -7,7 +7,7 @@ should check out the [v2.0 release](https://github.com/cavalab/srbench/releases/
 
 ### Local install
 
-We have provided a [conda environment](../base_environment.yml), [configuration script](configure.sh), and [installation script](../scripts/install_algorithm.sh) that should make installation straightforward.
+We have provided a [conda environment](../base_environment.yml), [configuration script](../configure.sh), and [installation script](../scripts/install_algorithm.sh) that should make installation straightforward.
 The installation script is the same used internally when building the docker images.
 
 We've currently tested this on Ubuntu and CentOS. 
@@ -23,7 +23,7 @@ conda config --set solver libmamba
 1. Install the conda environment naming it `srbench`:
 
 ```bash
-conda env create -f environment.yml -n srbench
+conda env create -f base_environment.yml -n srbench
 conda activate srbench
 ```
 

experiment/methods/gplearn/regressor.py

@@ -37,4 +37,4 @@ def model(est, X=None):
 
 def complexity(est):
     #TODO: check
-    return len(re.split('\(|,',model(est)))
+    return len(re.split(r'\(|,',model(est)))