LUMI provides access to a
singularity runtime for running applications in
software containers. Currently, there are two major providers of the
singularity runtime, namely Singularity CE and
Apptainer, with the latter being a fork of the former. For most
cases these should be fully compatible. LUMI provides the
included in the HPE Cray Operating System (a SUSE Linux based OS) running on
LUMI - no modules need to be loaded to use
singularity on LUMI. You can
always check the version of singularity using the command
See the Apptainer/Singularity containers install page for details about creating LUMI compatible software containers.
The basics of running a container on LUMI¶
Applications in a container may be run by combining Slurm commands with
Singularity commands, e.g. to get the version of Ubuntu running in a container
stored as "ubuntu_22.04.sif", we may use
srun to execute the
$ srun --partition=<partition> --account=<account> singularity exec ubuntu_21.04.sif cat /etc/os-release
which prints something along the lines of
PRETTY_NAME="Ubuntu 22.04.1 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.1 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy
Binding network file systems in the container¶
By default, the network file system partitions, such as
/project are not accessible from the within the container. To
make them available, they need to be explicitly bound by passing the
-B/--bind command line option to
singularity exec/run. For instance
$ srun --partition=<partition> --account=<project_name> singularity exec -B /scratch/<project_name> ubuntu_21.04.sif ls /scratch/<account>
Since project folder paths like
/project/<project_name> are symlinks on LUMI, you must bind these full
paths to make them available in the container. Simply binding
/project will not work.
Running containerized MPI applications¶
Running MPI applications in a container requires that you either bind the host MPI (the MPI stack provided as part of the software stack available on the compute node) or install a LUMI compatible MPI stack in the container.
For MPI-enabled containers, the application inside the container must be dynamically linked to an MPI version that is ABI-compatible with the host MPI.
Using the host MPI¶
In order to properly make use of LUMI's high-speed network, it is necessary to
mount a few host system directories inside the container and set
LD_LIBRARY_PATH so that the necessary dynamic libraries are available at run
time. This way, the MPI installed in the container image is replaced by the
host's MPI stack.
All the necessary components are available in a module that can be installed by the user via EasyBuild
$ module load LUMI partition/<lumi-partition> EasyBuild-user $ eb singularity-bindings-system-cpeGNU-<toolchain-version>.eb -r
Running e.g. the OSU point-to-point bandwidth test container can then be done using
$ module load singularity-bindings $ srun --partition=<partition> --account=<account> --nodes=2 singularity run mpi_osu.sif
which gives the bandwidth measured for different message sizes, i.e. something along the lines of
# OSU MPI Bandwidth Test v5.3.2 # Size Bandwidth (MB/s) 1 3.00 2 6.01 4 12.26 8 24.53 16 49.83 32 97.97 64 192.37 128 379.80 256 716.64 512 1386.52 1024 2615.18 2048 4605.69 4096 6897.21 8192 9447.54 16384 10694.19 32768 11419.39 65536 11802.31 131072 11997.96 262144 12100.20 524288 12162.28 1048576 12207.27 2097152 12230.66 4194304 12242.46
Using the container MPI¶
MPI applications can also be run using an MPI stack installed in the container.
To do so, Slurm needs to be instructed to use the PMI-2 process management
interface by passing
which produces an output along the lines of
# OSU MPI Bandwidth Test v5.3.2 # Size Bandwidth (MB/s) 1 0.50 2 1.61 4 3.57 8 6.54 16 9.65 32 18.04 64 35.27 128 67.76 256 91.12 512 221.09 1024 278.88 2048 471.54 4096 917.02 8192 1160.74 16384 1223.41 32768 1397.97 65536 1452.23 131072 2373.07 262144 2104.56 524288 2316.71 1048576 2478.30 2097152 2481.68 4194304 2380.51
Note that this approach gives lower bandwidths, especially for the larger message sizes, than is the case when using the host MPI. In general, the performance obtained from using the container MPI might be quite low compared to the results obtained when using the host's MPI. For a more in-depth discussion of the topic of MPI in containers, we suggest that you read this introduction to MPI in containers.