Most software in the central LUMI software stacks is installed through EasyBuild. The central software stack is kept as compact as possible to ease maintenance and to avoid user confusion. E.g., packages for which users request special customisations will never be installed in the central software stack. Moreover, due to the technical implementation of a software stack on a system the size of LUMI, installing software can be disruptive, so new software is mostly made available during maintenance intervals.
This, however, does not mean that you may have to wait for weeks before you can get the software you need for your project on LUMI. We have made it very easy to install additional software in your home or project directories (where the latter is a better choice as you can then share it with the other people in your project). After installing, using the software requires not much more than loading a module that configures EasyBuild for local installations and running EasyBuild with a few recipes that can be supplied by the User Support Team or your national support team or that you may write yourself. And this software is then built in exactly the same way as it would be in a central installation.
Beginner's guide to installing software on LUMI¶
If you are new to EasyBuild and LUMI, it might be a good idea to first read through this chapter once, and then start software installations.
We support installing software with EasyBuild only in the LUMI software stacks, not in CrayEnv.
EasyBuild installs software through recipes that give instructions to create a single module that most of the time provides a single package. It will also tell EasyBuild on which other modules a package depends so that these can also be installed automatically if needed (through their own EasyBuild recipes).
An EasyBuild build recipe is a file with a name that consists of different components and ends with '.eb'. Consider, e.g., a build recipe for the software GROMACS:
The first part of the name,
GROMACS, is the name of the package. The second
part of the name,
2021.4 is the version of GROMACS, in this case the
The next part,
cpeGNU-22.08, denotes the so-called toolchain
used for the build. Each toolchain corresponds to a particular HPE Cray Programming
Environment, and the number (
22.08in this example) denotes the version of this
programming environment. The various EasyBuild toolchains on LUMI are:
|EasyBuild toolchain||HPE Cray PE|
The version number of the toolchain should match the version of the LUMI software stack or the installation will fail. (In fact, it is not just the version in the file name that should match but the version of the toolchain that is used in the recipe file.)
The last part of the name,
-PLUMED-2.7.4-CPU, is called the version suffix. Version suffixes are
typically used to distinguish different builds of the same version of the
package. In this case, it indicates that it is a build of the 2021.4 version
purely for CPU and also includes PLUMED as we have also builds without PLUMED
(which is not compatible with every GROMACS version).
EasyBuild build recipes are stored in repositories with a fixed directory structure. On LUMI we already provide two such repositories, one containing all the software that is installed in the central software stack and one that contains EasyBuild recipes that users can install themselves or use as a basis to make a customised installation of software. An overview of all recipes in these repositories is provided in the LUMI Software Library.
We encourage advanced users to also build up a user repository with their own EasyBuild recipes and manage it with a version control system as that will provide a good description of the software stack that was used for a project and is a good step towards reproducibility. This is discussed below in the "Advanced guide", section "Building your own EasyBuild repository".
Preparation: Set the location for your EasyBuild installation¶
By default our EasyBuild setup will install software in
However, this location can be changed by pointing the environment variable
EBU_USER_PREFIX to the directory where you want to create the software
installation. In most cases a subdirectory in your
directory is the best location to install software as that directory is both
permanent for the duration of your project and shared with all users in your
project so that everybody can use the software. It is a very good idea to set
this environment variable in your
(replacing 465000000 with the number of your project).
Tip for users with multiple projects
If you participate in multiple projects, you'll have to either have only
a very personal software setup in your home directory which no one else can
use, or a setup in each of the project directories, as sharing of project
directories across projects is not possible. Our modules can also support
only one user software setup at a time. However, you can always switch to
a different setup by changing the value of the
environment variable, but you should only do so when no modules are loaded,
not even the
LUMI module. Hence you should always do a
of at the very least
immediately before changing the value of
EBU_USER_PREFIX. If you fail to
do so, the old user module directories will not be removed from the module
search path, not even if you reload the
LUMI module, and you may get very
unexpected results from module load operations.
Do not change
EBU_USER_PREFIX when a
LUMI module is loaded
Changing the value of
EBU_USER_PREFIX while one of the
is loaded has side effects. When switching to a different version of the
LUMI module or reloading the current module to enable the new installation
directory, the module system will fail to first properly clean the
old user installation directories from the module search path, even when
module --force purge. This is a side effect of how Lmod works when
unloading modules and there is no easy workaround for this.
However, doing a
module --force unlod LUMI first and then changing the
EBU_USER_PREFIX and then reloading a
LUMI module will work.
From now on you will also see the software that you have installed yourself for
the selected version of the LUMI software stack and partition when you do
module avail. Also,
module spider will also search those directories.
Step 1: Load the LUMI software stack¶
The next step is to ensure that the right version of the software stack is
loaded. Assume that we want to install software in the
LUMI/22.08 stack, then
one needs to execute
This should also automatically load the right
partition module for the part
of LUMI you are on, as further detailed on the software
Though it is technically possible to cross-compile software for a different partition, it may not be without problems as not all install scripts that come with software do support cross-compiling and as tests may fail when compiling for a CPU with instructions that the host CPU does not support.
Step 2: Load EasyBuild¶
The next step to install software in the directory you have just indicated, is
to load the
This will print a line on the screen indicating where software will be installed as a confirmation. It will also create the directory structure for the user software installation if it does not yet exist, including the structure of the user repository discussed below in the "Advanced guide", section "Building your own EasyBuild repository". If you want more information about the full configuration of EasyBuild, you can execute
EasyBuild is configured so that it searches in the user repository and two repositories on the system. The current directory is not part of the default search path but can be easily added with a command line option. By default, EasyBuild will not install dependencies of a package and fail instead, if one or more of the dependencies cannot be found, but that is also easily changed on the command line.
Step 3: Install the package¶
To show how to actually install a package, we continue with our
If all needed EasyBuild recipes are in one of the repositories, all you need to do to install the package is to run
-r tells EasyBuild to also install dependencies that may not yet be
GROMACS-2021.4-cpeGNU-22.08-PLUMED-2.7.4-CPU.eb would not have been
in a repository, but in the current directory or one of its subdirectories,
you could use
The only difference is the dot added to the
-r flag. This adds the current directory to
the front of the search path. In general it doesn't hurt to always use the dot with
but performance may suffer if the current directory contains a lot of subdirectories they
will all be searched for EasyBuild recipes.
-r . or
-r flags should be omitted if you
want full control and install dependency by dependency before installing the
package (which may be very useful if building right away fails).
If you now type
module avail you should see the
module in the list. Note the relation between the name of the EasyBuild recipe and the module name and version of the module. This is only the case though if the EasyBuild recipe follows the EasyBuild guidelines for naming. If the guidelines are not followed and if EasyBuild needs to install this module as a dependency of another package, EasyBuild will fail to locate the build recipe.
GROMACS/2021.4-cpeGNU-22.08-PLUMED-2.7.4-CPU module can now be used just like
any other module on the system. To use the GROMACS module you don't need to load
That was only required for installing the package.
All you need to do to use the GROMACS module that we just installed, is
(i.e., loading the software stack in which we installed GROMACS and the GROMACS module that we installed).
A special case: Modules for singularity containers¶
We provide some EasyConfigs to build modules for singularity containers that we provide elsewhere on the system. These are marked in the LUMI Software Library with a "C" on a purple background in the list and a "singularity container" label on the page for the specific package.
These EasyConfigs will copy the container to a safe place in your user installation so that you can keep it if reproducibility is a concern for you. They will also install modules that define some standard variables that make it easy to locate the container and that set the appropriate bindings for the 'singularity' command. Some of the container modules also provide some wrapper scripts that make it easier to work with the container or can serve as an example for your own scripts to use the software in the container.
In many cases the singularity container file in your own directory space can be removed and the module will automatically pick up the central one (but check the documentation for the package in the LUMI Software Library, it will tell you if you can do so). Do keep in mind though that the centrally stored container file will be removed if we find problems with it, while the container may still be perfectly fine for you. E.g., some containers provide the RCCL communication library which is popular in AI applications, but requires a specific plugin to work well with the Slingshot 11 interconnect of LUMI. These containers often need to be rebuilt after a system upgrade, but they may still be perfectly fine for users who use only a single GPU or a single GPU node. If you want to keep using the older version though, it has to be installed in your own file space.
The containers we provide do in general not depend on any specific version of the Cray
Programming Environment and hence also not on a specific version of the LUMI software stack.
Hence LUMI provides a mechanism to install the container modules in a place where they will
be found by all partitions of all LUMI stacks and by the CrayEnv stack. To this end, you can
install in the dummy partition
Note that to subsequently use the container you do not need to load
Many containers come with documentation about their use. We encourage you to check
the documentation in the LUMI Software Library for the containers,
and to check the help provided by the module after installation (with
Some common problems¶
module availdoes not show the module.
There are two possible causes for this.
Lmod builds a cache of all modules on the system. EasyBuild will clear the cache so that it will be rebuilt after installing a software package and hence the newly installed modules should be found. In rare cases Lmod may be in a corrupt state. In those cases the best solution is to clear the cache (unless it happens right after running the
ebcommand to install a module):
and to log out and log in again to start with a clean shell.
If the problem occurs later on, e.g., while running a job, then a common cause is that you have a different version of the
partitionmodules loaded than used when installing the software package.
Note that even the LUMI CPU compute nodes have a newer processor than the login nodes and may benefit from processor-specific optimisations which is why they use a different
partitionmodule. If you load one of the versions of the
LUMImodule on the login nodes, it will automatically load
partition/Lwhile if you do the load on a regular LUMI-C compute node, it will load
In the example above, if the installation commands were executed on the login node, the software would have been installed in
partition/L, but if we then do a
module load LUMI/22.08on the compute nodes,
partition/Cwould have been selected. To get a GROMACS version in
partition/Cthat EasyBuild would build with compiler settings that are specific for the processors in the compute nodes, either do the compilation on a compute node or use cross-compiling by loading
LUMI/22.08in step 1 above.
EasyBuild complains that some modules are already loaded.
EasyBuild prefers to work in a clean environment with no modules loaded that are installed via EasyBuild except for a very select list, and will complain if other modules are loaded (though only fail if a module for one of the packages that you try to install is already loaded). It is best to take this warning serious and to install in a relatively clean shell as otherwise the installation process may pick up software libraries that it should not have used.
Advanced guide to EasyBuild on LUMI¶
Toolchains on Cray¶
Toolchains in EasyBuild contain at least a compiler, but can also contain an MPI library and a number of mathematical libraries (BLAS, LAPACK, ScaLAPACK and a FFT library). Programs compiled with different toolchains cannot be loaded together (though the module system will not always prevent this on LUMI).
The toolchains on LUMI are different from what you may be used to from non-Cray systems. On most systems, EasyBuild uses its own toolchains installed from within EasyBuild, but on LUMI we use toolchains that are based on the Cray Programming Environment. Four toolchains are currently implemented
cpeGNUis the equivalent of the Cray
cpeCrayis the equivalent of the Cray
cpeAOCCis the equivalent of the Cray
cpeAMDis the equivalent of the Cray
All four toolchains use
cray-mpich over the Open Fabric Interface library
craype-network-ofi) and Cray LibSci for the mathematical libraries, with the
releases taken from the Cray PE release that corresponds to the version number
cpeGNU/Cray/AOCC/AMD and PrgEnv-gnu/cray/aocc/amd
cpeAMD modules don't
load the corresponding
PrgEnv-* modules nor the
This is because in the current setup of LUMI both modules have their
problems and the result of loading those modules is not always as intended.
If you want to compile software that uses modules from the LUMI stack,
it is best to use one of the
modules to load the compiler and libraries rather than the matching
PrgEnv-* modules as those may not always load
all modules in the correct version.
Since the LUMI software stack does not support the EasyBuild common toolchains
(such as the EasyBuild intel and foss toolchains), one cannot use the default
EasyBuild build recipes without modifying them. Hence they are not included in
the robot search path of EasyBuild so that you don't accidentally try to
install them (and also removed from the search path for
eb -S or
--search to avoid any confusion that they might work).
Building your own EasyBuild repository¶
We advise users to maintain their own repository of EasyConfig files which they installed in their personal or project space. This may help to rebuild your environment for a later project on LUMI. It may even be a good idea to keep the repository on a personal GitHub or other version control service.
The repository is created automatically the first time
loaded. The directory is called
UserRepo and is in
$EBU_USER_PREFIX (or the
$HOME/EasyBuild if you don't set the environment variable).
It must be structured similarly to the main EasyBuild EasyConfig
EasyBuild recipes (
.eb files) should be in a subdirectory
easybuild/easyconfigs, leaving room for personal EasyBlocks also (which would
then go in the
easybuild/easyblocks subdirectory) and even personal
configuration files that overwrite some system options. This setup also
guarantees compatibility with some EasyBuild features for very advanced users
that go way beyond this page.
To store this repository on GitHub, you can follow the GitHub documentation, and in particular the page "Adding an existing project to GitHub using the command line".
Technical documentation on the toolchains on LUMI and the directory structure of EasyBuild can be found in the documentation of the LUMI-SoftwareStack GitHub repository.
If you want to get more familiar with EasyBuild and develop your own EasyBuild recipes, we suggest the following sources of information:
- EasyBuild documentation
- EasyBuild tutorials
- The EasyBuild YouTube channel contains recordings of a four-session tutorial given for the LUMI User Support Team by Kenneth Hoste (UGent), the lead developer of EasyBuild and Luca Marsella (CSCS)
- Technical documentation on our setup for developers
- LUMI EasyBuild recipes
- Main LUMI software stack GitHub repository
contains the full EasyBuild setup for LUMI, including the EasyBuild recipes
that we use for the central software stack and many others that we fully support
and consider of good quality. The clone on the system is automatically searched
- LUMI contributed EasyBuild recipes GitHub repository contains contributed EasyBuild recipes and other recipes developed by LUST that haven't been as thoroughly checked or are deemed not appropriate for central installation at this point. However, they are fully compatible with the setup on LUMI, with correct dependency versions etc.
- LUMI EasyBuild container installation recipes GitHub repository
contains the recipes that are used to ease access to the containers that are in
- Main LUMI software stack GitHub repository contains the full EasyBuild setup for LUMI, including the EasyBuild recipes that we use for the central software stack and many others that we fully support and consider of good quality. The clone on the system is automatically searched by the
- Other EasyBuild recipes for the Cray Programming Environment
- CSCS GitHub repository. Most of the recipes are for Piz Daint which uses slightly different toolchains. Moreover dependencies typically need updating as the software installation on LUMI is not in sync with the CSCS installation. The repository is particularly useful for CPU-only programs as the GPUs in their system are not compatible with those in LUMI.
- EasyBuild recipes that are not compatible with the Cray Programming Environment but that may sometimes be a good source to start developing compatible ones (if you're an EasyBuild expert):