Repository navigation
V1 beta environment rebuild and experimental -alt_fs surfaces - #188
Merged
Merged
Conversation
…ainers -proc_surf -alt_fs runs FastSurfer, replaces its brain mask with FastScrubber's, and rebuilds the surfaces with run_fastsurfer.sh --surf_only. FastScrubber runs all 5 folds without test-time augmentation by default on the GPU or CPU; -alt_fs_tta adds it. FastSurfer's own mask and first surface logs are kept beside the rebuilt ones, and a directory given with -surf_dir is copied rather than modified. The default FastSurfer and -freesurfer paths are unchanged. The v1 beta container now builds in two variants, v1-beta-nocuda-DATE and v1-beta-cuda-DATE, which differ only in the PyTorch build of FastSurfer (CPU or CUDA 12.6) and FastScrubber (CPU or CUDA 12.4). Both include the FastScrubber environment and weights, and each build also writes a Docker image archive next to the Singularity image.
01_proc-surf.sh called micapipe_env, which init.sh defines only in the parent micapipe shell, so FastSurfer never ran inside the container. It now sources micapipe_env.sh itself; this affects the default FastSurfer path too. micapipe_env replaced LD_LIBRARY_PATH and dropped the NVIDIA driver libraries that singularity --nv adds, so PyTorch saw no GPU. Keep /.singularity.d/libs on the path. -alt_fs keeps the largest connected piece of the FastScrubber mask and fills its interior holes. On BIDS_CI sub-mri3T the mask had a detached 75 mL non-brain piece, which made mri_normalize fail with HISTOalloc(-2147483648); keeping the largest piece fixes it. FastScrubber's raw prediction is kept as mri/mask.fastscrubber.mgz.
utilities.sh sets CUDA_VISIBLE_DEVICES='' for every module, a workaround for a BIC-SGE failure in proc_dwi, so the CUDA container ran FastSurfer and FastScrubber on the CPU even with --nv. utilities.sh now keeps the caller's setting in MICAPIPE_CALLER_GPUS, and restore_gpus brings it back for FastSurfer and FastScrubber only, in a subshell. Every other module still sees no GPU.
FastScrubber left 1280 white matter voxels of BIDS_CI sub-mri7T outside its mask. They become zeros in norm.mgz, mri_normalize then finds a NaN white matter peak, and the rebuilt surfaces collapse (mean thickness 2.15 -> 0.76 mm). Rerunning the surfaces with FastSurfer's own mask reproduces the default result exactly, so the mask was the cause. clean_brain_mask.py now adds back every aseg.auto label except cerebral and cerebellar cortex and CSF before keeping the largest piece and filling holes: FastScrubber decides the outer boundary only.
mri_normalize takes the aseg white matter as control points. Where they fall outside mask.mgz, norm.mgz is 0, the white matter peak becomes NaN and the surfaces collapse without an error. Restoring deep brain labels was not enough: on BIDS_CI sub-mri7T 252 white matter voxels in detached aseg pieces stayed outside the mask (thickness 0.77 mm), and on sub-mri3T FastSurfer 2.6.0 labels a detached piece about 110 mm below the brain as white matter (10.7 mL). clean_brain_mask.py --trim now sets every label outside the final mask to 0 in aseg.auto.mgz and aseg.presurf.mgz, as in FreeSurfer, where the aseg never leaves the brain mask. mri_normalize then finds white matter at 110 and grey matter at 81-87 on both scans. -surf_dir with -alt_fs now also requires a complete FastSurfer run.
6.0.7.8 only has fsl-eddy-cuda-10.2, whose binary is eddy_cuda10.2 and whose CUDA 10.2 build will not run on current drivers. FSL dropped the CUDA eddy entirely in 6.0.7.20 and brought it back unsuffixed in 6.0.7.22 as fsl-eddy-cuda 2602.0, so 6.0.7.23 is the first current release that gives us plain eddy_cuda. The unsuffixed package cannot be added to 6.0.7.8: it needs fsl-newimage >= 2601.0, fsl-miscmaths >= 2412.6 and fsl-utils >= 2412.6, all newer than that release ships. Lock solved from FSL's own fsl-6.0.7.23_linux-64.yml: 732 packages, up from 693. Carried along by the release: fsl-pyfix 0.6.0 -> 0.10.0 and setuptools 80.9.0 -> 84.2.1. The legacy R/Octave FIX stays the default and is unaffected. docs/validation/v1-beta.md still says 6.0.7.8 on purpose: it records the 20261028 image. Both container variants need rebuilding and revalidating against this lock before that doc changes.
The help still said "Freesurfer 7"; the beta container ships FreeSurfer 8.2.0 (build-stamp freesurfer-linux-ubuntu22_x86_64-8.2.0-20260314). It also left the default unversioned, which made it hard to tell what -proc_surf actually runs. Both now name their versions. Under -post_structural the same entry was a copy of the -proc_surf one and claimed to reconstruct surfaces. It does not: 02_post-structural.sh only consults it to pick between an existing FreeSurfer and FastSurfer reconstruction when both are present.
The detached white matter piece that collapses the surfaces is now Deep-MI/FastSurfer#956. Worth naming here because it hits the default FastSurfer path too, not only -alt_fs, and because the issue records that 2.5.4 behaves the same way while 2.4.2 does not.
Bump FSL to 6.0.7.23 so the container ships eddy_cuda
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the v1 beta environment rebuild, plus an experimental alternative surface workflow. It covers most of the micapipe update list: the software upgrades, pyonlm, NiiVue, the Python environment rebuild, and the FIX model question.
Status: CPU engineering beta for lab review. Scientific equivalence is not established — see the validation notes at the bottom.
Rebuilt environments and containers
environments/holds one pinned environment per tool, each with an explicit conda lock, rather than one shared environment that tools fight over.containers/versions.jsonis the single record of what a build contains, andcontainers/carries the build scripts.Two variants are built:
nocuda(CPU only) andcuda, which uses the GPU with--nv/--gpus alland otherwise falls back to the CPU.readme.mdnotes that the previous two-stage Dockerfiles describe the old container and should not be used for this beta.FIX: the legacy R/Octave classifier stays
The existing training model was converted with FSL's own
convert_fix_model, and both classifiers were compared on three MICs subjects using identical legacy features. Label agreement was 94.19–98.99% and temporal correlation 0.951763–0.998045. None passed the engineering gate (agreement ≥95%, noise Jaccard ≥0.90, correlation ≥0.999, NRMSE ≤1%).So legacy FIX remains the default and R and Octave are retained in the container. The converted model is also present as an explicit Python option. Dropping R + Octave would have saved ~150 MB, and the old FIX folder another 2.1 GB — not worth it against an unproven classifier.
Experimental
-alt_fs-alt_fsruns FastSurfer, replaces its brain mask with a FastScrubber prediction, and rebuilds the surfaces. It targets 7T, where FastSurfer's mask tends to keep meninges, vessels, tentorium and sinuses.-alt_fs_ttaadds test-time augmentation; the default is all 5 folds without it, which on held-out data differs by Dice 0.9988–0.9992 and is 5–8× faster.FastScrubber only sets the outer boundary. micapipe adds back every deep-brain label from FastSurfer's own segmentation, keeps the largest connected component, fills interior holes, and then clears labels outside the new mask — without this, a few hundred stray white-matter voxels give
mri_normalizea NaN white-matter peak and the surfaces collapse (sub-mri7T: mean thickness 2.15 → 0.77 mm from 252 voxels). FastSurfer's segmentation, its original mask and the first-pass logs are all preserved.Defaults are untouched: plain FastSurfer and
-freesurferbehave exactly as before, and-alt_fsrefuses to combine with-freesurfer. Full write-up and caveats indocs/usage/alt_fs.md.QC fixes
beta.1 could report success when a non-executable image helper failed, producing text-only structural QC reports. The helper is now called through the selected interpreter, capture/PDF failures are checked, and the failure status is preserved through the QC wrapper and the pipeline.
Validation
docs/validation/v1-beta.mdandv1-beta-2.mdrecord what was actually run: native-version and dependency-isolation checks on two Linux servers, both FIX classifiers reproducing their reference labels, two structural pilots completing 09/09 with a passing 5TT check, real CPU LAMAReg/SynthSeg inference, the packaged pyonlm CLI plus all 25 upstream tests, offline FastSurfer CPU segmentation, offline NiiVue, and a standalone 41-volume diffusion pilot through topup/eddy_cpu andeddy_quad.Not established: manual visual sign-off, the complete DWI workflow, scientific equivalence of the converted FIX model, and
-alt_fsagainst manual masks on 3T. The-alt_fshole-filling and deep-label restoration have not yet been re-evaluated on the 10-session MICs cohort that motivated them.Reviewer notes
environments/locks/..debfiles (libxp6,libpng12-0) undercontainers/os/vendor/with recorded checksums, because the Ubuntu archive URLs 404.scripts/contains the one-off FIX comparison scripts behind the numbers above. They point at a server-local environment path and are kept as evidence, not as pipeline code — say the word and I'll drop them.eddy_cuda. GitHub will retarget it tov1when this merges.