Skip to content

V1 beta environment rebuild and experimental -alt_fs surfaces - #188

Merged
rcruces merged 10 commits into
v1from
enning/v1-beta-alt-fs
Nov 2, 2026
Merged

rcruces merged 10 commits into
v1from
enning/v1-beta-alt-fs

Conversation

@zihuaihuai

Copy link
Copy Markdown
Collaborator

This is the v1 beta environment rebuild, plus an experimental alternative surface workflow. It covers most of the micapipe update list: the software upgrades, pyonlm, NiiVue, the Python environment rebuild, and the FIX model question.

Status: CPU engineering beta for lab review. Scientific equivalence is not established — see the validation notes at the bottom.

Rebuilt environments and containers

environments/ holds one pinned environment per tool, each with an explicit conda lock, rather than one shared environment that tools fight over. containers/versions.json is the single record of what a build contains, and containers/ carries the build scripts.

tool version
FastSurfer 2.6.0 (FreeSurfer 8.2.0)
FreeSurfer 8.2.0
MRtrix3 3.0.8
ANTs 2.6.5
FSL 6.0.7.8
pyonlm zihuaihuai/pyonlm
NiiVue 0.69.0

Two variants are built: nocuda (CPU only) and cuda, which uses the GPU with --nv / --gpus all and otherwise falls back to the CPU.

readme.md notes that the previous two-stage Dockerfiles describe the old container and should not be used for this beta.

FIX: the legacy R/Octave classifier stays

The existing training model was converted with FSL's own convert_fix_model, and both classifiers were compared on three MICs subjects using identical legacy features. Label agreement was 94.19–98.99% and temporal correlation 0.951763–0.998045. None passed the engineering gate (agreement ≥95%, noise Jaccard ≥0.90, correlation ≥0.999, NRMSE ≤1%).

So legacy FIX remains the default and R and Octave are retained in the container. The converted model is also present as an explicit Python option. Dropping R + Octave would have saved ~150 MB, and the old FIX folder another 2.1 GB — not worth it against an unproven classifier.

Experimental -alt_fs

-alt_fs runs FastSurfer, replaces its brain mask with a FastScrubber prediction, and rebuilds the surfaces. It targets 7T, where FastSurfer's mask tends to keep meninges, vessels, tentorium and sinuses. -alt_fs_tta adds test-time augmentation; the default is all 5 folds without it, which on held-out data differs by Dice 0.9988–0.9992 and is 5–8× faster.

FastScrubber only sets the outer boundary. micapipe adds back every deep-brain label from FastSurfer's own segmentation, keeps the largest connected component, fills interior holes, and then clears labels outside the new mask — without this, a few hundred stray white-matter voxels give mri_normalize a NaN white-matter peak and the surfaces collapse (sub-mri7T: mean thickness 2.15 → 0.77 mm from 252 voxels). FastSurfer's segmentation, its original mask and the first-pass logs are all preserved.

Defaults are untouched: plain FastSurfer and -freesurfer behave exactly as before, and -alt_fs refuses to combine with -freesurfer. Full write-up and caveats in docs/usage/alt_fs.md.

QC fixes

beta.1 could report success when a non-executable image helper failed, producing text-only structural QC reports. The helper is now called through the selected interpreter, capture/PDF failures are checked, and the failure status is preserved through the QC wrapper and the pipeline.

Validation

docs/validation/v1-beta.md and v1-beta-2.md record what was actually run: native-version and dependency-isolation checks on two Linux servers, both FIX classifiers reproducing their reference labels, two structural pilots completing 09/09 with a passing 5TT check, real CPU LAMAReg/SynthSeg inference, the packaged pyonlm CLI plus all 25 upstream tests, offline FastSurfer CPU segmentation, offline NiiVue, and a standalone 41-volume diffusion pilot through topup/eddy_cpu and eddy_quad.

Not established: manual visual sign-off, the complete DWI workflow, scientific equivalence of the converted FIX model, and -alt_fs against manual masks on 3T. The -alt_fs hole-filling and deep-label restoration have not yet been re-evaluated on the 10-session MICs cohort that motivated them.

Reviewer notes

  • Large diff (~6,200 insertions), mostly generated conda lock files under environments/locks/.
  • Two vendored .deb files (libxp6, libpng12-0) under containers/os/vendor/ with recorded checksums, because the Ubuntu archive URLs 404.
  • scripts/ contains the one-off FIX comparison scripts behind the numbers above. They point at a server-local environment path and are kept as evidence, not as pipeline code — say the word and I'll drop them.
  • Bump FSL to 6.0.7.23 so the container ships eddy_cuda #187 stacks on this branch and bumps FSL to 6.0.7.23 for eddy_cuda. GitHub will retarget it to v1 when this merges.

zihuaihuai and others added 10 commits October 23, 2026 11:09
…ainers

-proc_surf -alt_fs runs FastSurfer, replaces its brain mask with
FastScrubber's, and rebuilds the surfaces with run_fastsurfer.sh
--surf_only. FastScrubber runs all 5 folds without test-time augmentation
by default on the GPU or CPU; -alt_fs_tta adds it. FastSurfer's own mask
and first surface logs are kept beside the rebuilt ones, and a directory
given with -surf_dir is copied rather than modified. The default
FastSurfer and -freesurfer paths are unchanged.

The v1 beta container now builds in two variants, v1-beta-nocuda-DATE and
v1-beta-cuda-DATE, which differ only in the PyTorch build of FastSurfer
(CPU or CUDA 12.6) and FastScrubber (CPU or CUDA 12.4). Both include the
FastScrubber environment and weights, and each build also writes a Docker
image archive next to the Singularity image.
01_proc-surf.sh called micapipe_env, which init.sh defines only in the
parent micapipe shell, so FastSurfer never ran inside the container. It
now sources micapipe_env.sh itself; this affects the default FastSurfer
path too.

micapipe_env replaced LD_LIBRARY_PATH and dropped the NVIDIA driver
libraries that singularity --nv adds, so PyTorch saw no GPU. Keep
/.singularity.d/libs on the path.

-alt_fs keeps the largest connected piece of the FastScrubber mask and
fills its interior holes. On BIDS_CI sub-mri3T the mask had a detached
75 mL non-brain piece, which made mri_normalize fail with
HISTOalloc(-2147483648); keeping the largest piece fixes it. FastScrubber's
raw prediction is kept as mri/mask.fastscrubber.mgz.
utilities.sh sets CUDA_VISIBLE_DEVICES='' for every module, a workaround
for a BIC-SGE failure in proc_dwi, so the CUDA container ran FastSurfer
and FastScrubber on the CPU even with --nv. utilities.sh now keeps the
caller's setting in MICAPIPE_CALLER_GPUS, and restore_gpus brings it back
for FastSurfer and FastScrubber only, in a subshell. Every other module
still sees no GPU.
FastScrubber left 1280 white matter voxels of BIDS_CI sub-mri7T outside
its mask. They become zeros in norm.mgz, mri_normalize then finds a NaN
white matter peak, and the rebuilt surfaces collapse (mean thickness
2.15 -> 0.76 mm). Rerunning the surfaces with FastSurfer's own mask
reproduces the default result exactly, so the mask was the cause.

clean_brain_mask.py now adds back every aseg.auto label except cerebral
and cerebellar cortex and CSF before keeping the largest piece and filling
holes: FastScrubber decides the outer boundary only.
mri_normalize takes the aseg white matter as control points. Where they
fall outside mask.mgz, norm.mgz is 0, the white matter peak becomes NaN
and the surfaces collapse without an error. Restoring deep brain labels
was not enough: on BIDS_CI sub-mri7T 252 white matter voxels in detached
aseg pieces stayed outside the mask (thickness 0.77 mm), and on sub-mri3T
FastSurfer 2.6.0 labels a detached piece about 110 mm below the brain as
white matter (10.7 mL).

clean_brain_mask.py --trim now sets every label outside the final mask to
0 in aseg.auto.mgz and aseg.presurf.mgz, as in FreeSurfer, where the aseg
never leaves the brain mask. mri_normalize then finds white matter at 110
and grey matter at 81-87 on both scans. -surf_dir with -alt_fs now also
requires a complete FastSurfer run.
6.0.7.8 only has fsl-eddy-cuda-10.2, whose binary is eddy_cuda10.2 and
whose CUDA 10.2 build will not run on current drivers. FSL dropped the
CUDA eddy entirely in 6.0.7.20 and brought it back unsuffixed in
6.0.7.22 as fsl-eddy-cuda 2602.0, so 6.0.7.23 is the first current
release that gives us plain eddy_cuda.

The unsuffixed package cannot be added to 6.0.7.8: it needs
fsl-newimage >= 2601.0, fsl-miscmaths >= 2412.6 and fsl-utils >= 2412.6,
all newer than that release ships.

Lock solved from FSL's own fsl-6.0.7.23_linux-64.yml: 732 packages, up
from 693. Carried along by the release: fsl-pyfix 0.6.0 -> 0.10.0 and
setuptools 80.9.0 -> 84.2.1. The legacy R/Octave FIX stays the default
and is unaffected.

docs/validation/v1-beta.md still says 6.0.7.8 on purpose: it records the
20261028 image. Both container variants need rebuilding and revalidating
against this lock before that doc changes.
The help still said "Freesurfer 7"; the beta container ships FreeSurfer
8.2.0 (build-stamp freesurfer-linux-ubuntu22_x86_64-8.2.0-20260314). It
also left the default unversioned, which made it hard to tell what
-proc_surf actually runs. Both now name their versions.

Under -post_structural the same entry was a copy of the -proc_surf one
and claimed to reconstruct surfaces. It does not: 02_post-structural.sh
only consults it to pick between an existing FreeSurfer and FastSurfer
reconstruction when both are present.
The detached white matter piece that collapses the surfaces is now
Deep-MI/FastSurfer#956. Worth naming here because it hits the default
FastSurfer path too, not only -alt_fs, and because the issue records
that 2.5.4 behaves the same way while 2.4.2 does not.
Bump FSL to 6.0.7.23 so the container ships eddy_cuda
@rcruces
rcruces merged commit 7aa9eb2 into v1 Nov 2, 2026
10 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants