Slurm partitions¶
The Slurm partition setup of LUMI prioritizes jobs that aim to scale out. As a consequence, most nodes are reserved for jobs that use all available resources within a node. However, some nodes are reserved for smaller allocations and debugging.
A list of the available partitions can be obtained using the sinfo command.
To get a shorter summary, use sinfo -s.
Slurm partitions allocatable by node¶
The following partitions are available for allocation by node. When using these partitions, your jobs use all resources available on the node and won't share the node with other jobs. Therefore, make sure that your application can take advantage of all the resources on the node as you will be billed for the complete node regardless of the resource actually used as detailed in the billing policy.
| Name | Max walltime | Max jobs | Max resources/job | Hardware partition used |
|---|---|---|---|---|
| standard-g | 2 days | 210 (200 running) | 1024 nodes | LUMI-G |
| standard | 2 days | 120 (100 running) | 512 nodes | LUMI-C |
| largemem | 2 day | 30 (20 running) | 1 nodes | LUMI-D |
Slurm partitions allocatable by resources¶
The following partitions are available for allocation by resources. This means that you can request a sub-node allocation: you can request only part of the resources (cores, gpus, and memory) available on the compute node. This also means that your job may share the node with other jobs.
| Name | Max walltime | Max jobs | Max resources/job | Hardware partition | Purpose |
|---|---|---|---|---|---|
| debug | 30 minutes | 2 (2 running) | 4 nodes | LUMI-C | Debugging and testing |
| small-g | 3 days | 210 (200 running) | 4 nodes | LUMI-G | Small GPU jobs |
| small | 3 days | 220 (200 running) | 4 nodes | LUMI-C | Small or memory intense jobs |
| lumid | 12 hours | 16 (8 running) | 8 GPUs (total) | LUMI-D | Visualisation |
| Name | Max walltime | Max jobs | Max resources/job | Hardware partition | Purpose |
|---|---|---|---|---|---|
| dev-g | 30 minutes | 1 (1 running) | 32 nodes | LUMI-G | Debugging |
| 1 hour | 1 (1 running) | 16 nodes | LUMI-G | Debugging | |
| 2 hours | 1 (1 running) | 8 nodes | LUMI-G | Debugging |
Notes about specific partitions
Debugging partitions¶
The debug and dev-g queues are meant for debugging and quick testing purposes, and not for production runs (and for dev-g, interactive development if the presence of a GPU is required to, e.g. correctly compile a code). Debugging has an interactive character which is why it has its own queue. Some examples of "interactive character":
- You are working in an interactive session, issuing commands in that session, e.g., manually trying a sequence from/for a job script.
- You have an interactive debugger attached to a job.
- You run a job with an interactive session attached to closely monitor the behaviour of a job (e.g., the load on the GPUs).
- You want to run a job under very close supervision, continuously monitoring the output of a job (e.g., using the tail command on one of the login nodes), so it is obvious that you cannot wait long for that job to start or that it should not start during the night.
- You run a short test, e.g., to judge the performance or correctness impact of a code change that you just made, that literally takes only a few minutes.
The max runtime on dev-g depends on the number of nodes one requests, as seen from the table: on dev-g the max resources one can ask per job is 16 node hours. This corresponds to allocating 16 nodes for 1 hour, more nodes for a shorter time, or part of a node for a longer time. The max run time for a job on the partition is 3 hours.
On dev-g, the maximum number of jobs is one per user and one per project. Both limits apply at the same time:
Per user: you can have only one job on dev-g at a time (running or in queue), even if you belong to several projects.
Per project: only one job per project is allowed on dev-g at a time. If another member of your project already has a job there, you will not be able to submit a new job to dev-g until theirs has finished.
Repeated abuse of these partitions might result in account suspension.
Small partitions¶
LUMI is optimized for large jobs (dozens of nodes). However not all applications can scale efficiently at large-scale or even at the node level. Allocating an entire node for a serial pre/post-processing job or for an application that can only use a single GPU is not an efficient use of the resources.
The kind of jobs described above should use the small partitions. On these partitions you can only allocate a few nodes but you can run for a longer period of time.
Large memory nodes¶
The LUMI-C large memory nodes (512GB and 1TB) are located in the
small partition. Therefore, to use these nodes, you need to
select the small partition (--partition=small). Then the LUMI-C large
memory nodes will be allocated if you request more memory than is available
in the LUMI-C standard compute nodes.
The nodes in the largemem partition are part of LUMI-D and have
4TB of memory per node. They are mostly meant for data-intensive pre- and
postprocessing and should not be the only compute resource used by your
project as there are only a limited number of those nodes.
Visualization nodes¶
LUMI-D GPU nodes are the only nodes in LUMI that have Nvidia GPUs. They are only intended for graphical hardware acceleration, to be used for visualisation purposes like Paraview. They are not a source of CUDA-compatible compute power for regular computations. Compute workloads should be done with codes suitable for the AMD GPUs of LUMI-G. Repeated abuse may result in account suspension or project termination.
Getting information about Slurm partitions¶
If you want more precise information about a particular partition, you can use the following command:
The output of this command will give you information about the defaults and
limits which applies to the <partition-name> partition.