LUMI container wrapper¶
The LUMI container wrapper is a set of tools which wrap software installations inside an Apptainer/Singularity container to improve startup times, reduce I/O load, and lessen the number of files on large parallel file systems.
Additionally, the LUMI container wrapper will generate wrappers so that installed software can be used (almost) as if it was not containerized. Depending on tool selection and settings, either the whole host file system or a limited subset is visible during execution and installation. This means that it is possible to wrap installations using e.g. mpi4py while still relying on the host provided MPI installation.
The LUMI container wrapper is a general purpose installation wrapper that supports wrapping:
- Existing installations on the filesystem: Mainly to reduce the I/O load and improve startup times, but may also be used to containerize existing installations that cannot be re-installed.
- Existing Singularity/Apptainer containers: Mainly to hide the need for using the container runtime from the user.
- Conda installations: Directly wrap a Conda installation based on a Conda environment file.
- Pip installations: Directly wrap a pip installation based on a requirements.txt file.
The LUMI container wrapper is NOT generally recommended for managing Conda/pip installations
For some use cases, the LUMI container wrapper is an excellent tool for managing Conda/pip installations. For other use cases, there are better alternatives. See the installing Python packages guide for an overview the recommended ways to manage Python installations, including Conda/pip installations.
The LUMI container wrapper has some limitations
Please be aware of the limitations when using the LUMI container wrapper before wrapping your installations.
The LUMI container wrapper is experimental software
As the LUMI container wrapper is still under development, some of the more advanced features might change in exact usage and API.
Examples of using the LUMI container wrapper¶
The tools provided by the LUMI container wrapper are accessible by loading the
lumi-container-wrapper module that is available in the
Once the module has been loaded, you can use one of the front-end tools
Wrapping a basic Conda installation¶
In order to wrap a basic Conda installation, create an installation directory
and run the
env.yml is a Conda environment file.
This file can be written manually, e.g:
or generated from an existing environment
Windows and MacOS will need to add the
--from-history flag to the export command
--explicit option only works if the existing environment is on a
Linux machine with x86 CPU architecture. Otherwise the result will not be
transferable to LUMI.
After the installation is done, you simply need to add the bin directory
<install_dir>/bin to your
Then, you can call
python or any other executables, Conda has installed, in
the same way as if you had activated the environment.
If you also need to install some additional pip packages, you can do so by
-r <req_file> argument e.g:
req.txt is a standard pip requirements.txt file.
The tool also supports using Mamba for installing packages. Enable
this feature by adding the
--mamba flag, e.g.
conda-containerize new --mamba
Make sure that you have read and understood the license terms for Miniconda as well as any used conda channels before using the command:
End-to-end example of wrapping a Conda installation¶
Using the previous
After the installation finishes, we can add the installation directory to our
PATH and use it like normal.
Modifying a wrapped Conda installation¶
As the wrapped installation resides in a container, it cannot be directly
modified. Small python packages can be added outside the container, in the
usual way, using
pip, but then the python packages are sitting on the
parallel file system which is not recommended for larger installations.
To actually modify the installation inside the wrapping container, you can use
update keyword together with the
--post-install <file> option which
specifies a bash script with commands to run to update the installation. The
commands are executed with the Conda environment activated.
<file> could e.g. contain
In this mode the whole host system is available including all software and modules.
Wrapping a plain pip installation¶
Sometimes you don't need a full-blown Conda environment, or you may prefer to
manage your python installations using pip. For this case you can use the
pip-containerize wrapper via
req.txt is a standard pip requirements.txt file. The above
notes and options for modifying a Conda installation apply to pip installations
Note that the python version used by
pip-containerize is the first python
executable found on the
PATH, so it's affected by loading modules.
This python used to installed pip packages cannot itself be container-based as nesting of containers is not possible.
Additionally, you may use the
--slim argument, which will use a pre-built
minimal python container with a much newer version of python as a base. Without
--slim argument, the whole host system is available. However, by using
--slim argument, the system installations (i.e /usr, /lib64 ...) are no
longer taken from the host, but are instead taken from the minimal python
Wrapping existing containers¶
The LUMI container wrapper also provides a tool to generate wrappers for
existing Apptainer/Singularity containers, so that they can be used
transparently without the need for prepending
singularity exec ..., or modify
scripts if switching between containerized versions of tools.
<container> can be a filepath or any URL accepted by
oras//:, or a local .sif file), and
-w needs to be an absolute
path (or comma-separated list) inside the container. Wrappers will then be
automatically created for the executables in the target directories or for the
More examples may be found in the LUMI container wrapper GitHub repository examples.
How it works¶
A description of how LUMI container wrapper works may be found in the LUMI container wrapper GitHub repository README file. A short discussion of how LUMI container wrapper works from the user perspective may be found in this GitHub issue.