HeFFTe: +cuda variant expects GPU-aware MPI and causes segfaults with non-CUDA MPI

[wspear]

When building heffte +cuda, the package silently builds with GPU-aware MPI enabled because heFFTe's CMakeLists.txt automatically turns on Heffte_ENABLE_GPU_AWARE_MPI=ON if accelerators are present. If the underlying ^mpi provider was not compiled with CUDA/ROCm support, this causes a silent compilation success followed by a hard segmentation fault at runtime whenever MPI attempts to touch the device pointers (e.g. inside MPI_Alltoallv).

Possible fixes:

• Add a variant("gpu_aware", default=False, description="Enable GPU-aware MPI") to heffte, and explicitly pass -DHeffte_DISABLE_GPU_AWARE_MPI=ON if it is disabled.
• Require depends_on("mpi +cuda", when="+cuda") ( problematic since not all MPI providers use that variant name).
• Auto-detect if the ^mpi provider is GPU-aware and adjust the CMake flags accordingly.

Example of failing heffte+cuda spec

 spack find -dvl heffte+cuda
-- linux-ubuntu20.04-ppc64le / %c,cxx,fortran=gcc@9.4.0 ---------
32vnlrw heffte@2.4.1+cuda~fftw~fortran~ipo~magma~mkl~python~rocm+shared build_system=cmake build_type=Release cuda_arch:=70 generator=make
22z6xms     cmake@3.31.11~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release
m2ir257         curl@8.20.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs:=shared,static tls:=openssl
3qyia2n             gnuconfig@2025-07-10 build_system=generic
f7expzy             nghttp2@1.67.1 build_system=autotools
k7ogqpj                 diffutils@3.12 build_system=autotools
f5vfau2             openssl@3.6.1~docs+shared build_system=generic certs=mozilla
vptxh4h                 ca-certificates-mozilla@2026-03-19 build_system=generic
v7ulq7w                 perl@5.42.0+cpanm+opcode+open+shared+threads build_system=generic
cgqf5vt                     berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc
mzouf3e                     bzip2@1.0.8~debug~pic+shared build_system=generic
iivgbku                     gdbm@1.26 build_system=autotools
ohr24dr                         readline@8.3 build_system=autotools patches=21f0a03,72dee13,e273643
77j3u4o                     less@692 build_system=autotools
dmbx5q3             pkgconf@2.5.1 build_system=autotools
bsembdx         ncurses@6.6~symlinks+termlib abi=none build_system=autotools patches=7a351bc
6yv3js5         zlib-ng@2.3.3+compat+new_strategies+opt+pic+shared build_system=autotools
woxw5tt     compiler-wrapper@1.1.0 build_system=generic
zn7lg76     cuda@11.4.4~allow-unsupported-compilers~dev build_system=generic
y24s4fe         coreutils@9.10~gprefix build_system=autotools
fvz2oi4         gzip@1.14 build_system=autotools
qyaxk7q         libxml2@2.13.9~http+pic~python+shared build_system=autotools
uo7q3km             libiconv@1.18 build_system=autotools libs:=shared,static
b7emxid             xz@5.8.3+pic build_system=autotools libs:=shared,static
a73nwa3     gcc@9.4.0+binutils+bootstrap~graphite+libsanitizer~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran'
eo2gqch     gcc-runtime@9.4.0 build_system=generic
iou6c5i     glibc@2.31 build_system=autotools
inplmf2     gmake@4.4.1~guile build_system=generic
l3erh4z     mpich@5.0.1~argobots~cuda+fortran~hwloc+hydra~level_zero+libxml2+pci~rocm+romio~slurm~vci~verbs~wrapperrpath~xpmem build_system=autotools datatype-engine=auto device=ch4 netmod=ofi pmi=default

Failed test execution and GDB Backtrace

root@c2e41f2e3b42:~/testsuite/validation_tests/heffte-cuda/build# which gdb
/spack/opt/spack/linux-ppc64le/gdb-17.1-n4d2fqkkilos6eyml2xdygvngpy2bmma/bin/gdb
root@c2e41f2e3b42:~/testsuite/validation_tests/heffte-cuda/build# /opt/mpi/pkgs/mpich-5.0.1/bin/mpiexec -n 2 -l ./examples/heffte_example_gpu
===================================================================================

=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

=   PID 89504 RUNNING AT c2e41f2e3b42

=   EXIT CODE: 139

=   CLEANING UP REMAINING PROCESSES

=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)

This typically refers to a problem with your application.

Please see the FAQ page for debugging suggestions

root@c2e41f2e3b42:/testsuite/validation_tests/heffte-cuda/build# /opt/mpi/pkgs/mpich-5.0.1/bin/mpiexec -n 2 bash -c 'echo "rank $PMI_RANK / $MPI_LOCALRANKID sees devices: $CUDA_VISIBLE_DEVICES"'

rank 0 / 0 sees devices:

rank 1 / 1 sees devices:

root@c2e41f2e3b42:/testsuite/validation_tests/heffte-cuda/build# /opt/mpi/pkgs/mpich-5.0.1/bin/mpiexec -n 2 gdb -q -batch -ex run -ex bt --args ./examples/heffte_example_gpu 2>&1 | tail -60

warning: Error disabling address space randomization: Operation not permitted

warning: Error disabling address space randomization: Operation not permitted

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/usr/lib/powerpc64le-linux-gnu/libthread_db.so.1".

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/usr/lib/powerpc64le-linux-gnu/libthread_db.so.1".

[New Thread 0x7fff9b73a890 (LWP 89545)]

[New Thread 0x7fff7bf3a890 (LWP 89544)]

[New Thread 0x7fff9af2a890 (LWP 89546)]

[New Thread 0x7fff7b72a890 (LWP 89547)]

[New Thread 0x7fff9764a890 (LWP 89548)]

[New Thread 0x7fff77e4a890 (LWP 89549)]

[New Thread 0x7fff9692a890 (LWP 89550)]

[New Thread 0x7fff7712a890 (LWP 89551)]
Thread 1 "heffte_example_" received signal SIGSEGV, Segmentation fault.

0x00007fff9bdefd5c in ?? () from /usr/lib/powerpc64le-linux-gnu/libc.so.6
Thread 1 "heffte_example_" received signal SIGSEGV, Segmentation fault.

0x00007fff7c5efd5c in ?? () from /usr/lib/powerpc64le-linux-gnu/libc.so.6

#0  0x00007fff9bdefd5c in ?? () from /usr/lib/powerpc64le-linux-gnu/libc.so.6

#0  0x00007fff7c5efd5c in ?? () from /usr/lib/powerpc64le-linux-gnu/libc.so.6

#1  0x00007fffb2550a4c in typerep_do_pack.part () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#1  0x00007fff92d50a4c in typerep_do_pack.part () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#2  0x00007fffb2550e68 in MPIR_Typerep_pack () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#2  0x00007fff92d50e68 in MPIR_Typerep_pack () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#3  0x00007fffb25127e0 in MPIDI_POSIX_eager_send () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#3  0x00007fff92d127e0 in MPIDI_POSIX_eager_send () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#4  0x00007fffb251aa24 in MPIDI_POSIX_mpi_isend () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#4  0x00007fff92d1aa24 in MPIDI_POSIX_mpi_isend () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#5  0x00007fffb251afbc in MPID_Isend () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#5  0x00007fff92d1afbc in MPID_Isend () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#6  0x00007fffb2525eb4 in MPIC_Isend () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#6  0x00007fff92d25eb4 in MPIC_Isend () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#7  0x00007fffb24436f8 in MPIR_Alltoallv_intra_scattered () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#7  0x00007fff92c436f8 in MPIR_Alltoallv_intra_scattered () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#8  0x00007fffb24f654c in MPIR_Alltoallv_allcomm_auto () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#8  0x00007fff92cf654c in MPIR_Alltoallv_allcomm_auto () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#9  0x00007fffb24f693c in MPIR_Alltoallv_impl () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#9  0x00007fff92cf693c in MPIR_Alltoallv_impl () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#10 0x00007fffb24f6ecc in MPIR_Alltoallv () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#10 0x00007fff92cf6ecc in MPIR_Alltoallv () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#11 0x00007fffb21307fc in PMPI_Alltoallv () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#11 0x00007fff929307fc in PMPI_Alltoallv () from /opt/mpi/pkgs/mpich-5.0.1/lib/libmpi.so.12

#12 0x00007fffb349ad70 in void heffte::reshape3d_alltoallv<heffte::tag::gpu, heffte::direct_packer, int>::apply_base<std::complex >(int, std::complex const*, std::complex, std::complex) const () from /spack/opt/spack/linux-ppc64le/heffte-2.4.1-32vnlrwrxgayecp235i5ljmdu77dzziw/lib/libheffte.so.2

#12 0x00007fff93c9ad70 in void heffte::reshape3d_alltoallv<heffte::tag::gpu, heffte::direct_packer, int>::apply_base<std::complex >(int, std::complex const*, std::complex, std::complex) const () from /spack/opt/spack/linux-ppc64le/heffte-2.4.1-32vnlrwrxgayecp235i5ljmdu77dzziw/lib/libheffte.so.2

#13 0x00007fffb349af48 in heffte::reshape3d_alltoallv<heffte::tag::gpu, heffte::direct_packer, int>::apply(int, std::complex const*, std::complex, std::complex) const () from /spack/opt/spack/linux-ppc64le/heffte-2.4.1-32vnlrwrxgayecp235i5ljmdu77dzziw/lib/libheffte.so.2

#13 0x00007fff93c9af48 in heffte::reshape3d_alltoallv<heffte::tag::gpu, heffte::direct_packer, int>::apply(int, std::complex const*, std::complex, std::complex) const () from /spack/opt/spack/linux-ppc64le/heffte-2.4.1-32vnlrwrxgayecp235i5ljmdu77dzziw/lib/libheffte.so.2

#14 0x00007fffb34bc2b0 in void heffte::compute_transform<heffte::tag::gpu, int, std::complex >(heffte::backend::data_manipulatorheffte::tag::gpu::stream_type, int, std::complex const*, std::complex, std::complex, unsigned long, unsigned long, std::array<std::unique_ptr<heffte::reshape3d_base, std::default_delete<heffte::reshape3d_base > >, 4ul> const&, std::array<heffte::executor_base*, 3ul> const&, heffte::direction) () from /spack/opt/spack/linux-ppc64le/heffte-2.4.1-32vnlrwrxgayecp235i5ljmdu77dzziw/lib/libheffte.so.2

#14 0x00007fff93cbc2b0 in void heffte::compute_transform<heffte::tag::gpu, int, std::complex >(heffte::backend::data_manipulatorheffte::tag::gpu::stream_type, int, std::complex const*, std::complex, std::complex, unsigned long, unsigned long, std::array<std::unique_ptr<heffte::reshape3d_base, std::default_delete<heffte::reshape3d_base > >, 4ul> const&, std::array<heffte::executor_base*, 3ul> const&, heffte::direction) () from /spack/opt/spack/linux-ppc64le/heffte-2.4.1-32vnlrwrxgayecp235i5ljmdu77dzziw/lib/libheffte.so.2

#15 0x0000000133dba438 in void heffte::fft3d<heffte::backend::cufft, int>::forward<std::complex, std::complex >(std::complex const*, std::complex, std::complex, heffte::scale) const ()

#15 0x0000000102e6a438 in void heffte::fft3d<heffte::backend::cufft, int>::forward<std::complex, std::complex >(std::complex const*, std::complex, std::complex, heffte::scale) const ()

#16 0x0000000133db627c in compute_dft(int) ()

#16 0x0000000102e6627c in compute_dft(int) ()

#17 0x0000000133db66ac in main ()

#17 0x0000000102e666ac in main ()

• Spack: 1.3.0.dev0 (spack/spack@307d3b0)
• Builtin repo: f020fd1
• Python: 3.8.10
• Platform: linux-ubuntu20.04-power9le

@G-Ragghianti @mkstoyanov @eugeneswalker