Hands-on User Manual

Usage overview

SLURM manages cluster resources and allocates them to submitted jobs. Specify the resources a job needs when submitting it. Common options include:

  • -J <job_name>: sets the job name.
  • --time=DD-HH:MM:SS: sets the wall-time limit. The default depends on the partition (one day on the batch partition).
  • --mem-per-cpu=<value>: sets memory per CPU in MB or with a unit suffix such as G. The configured default is 2048 MB (DefMemPerCPU=2048).
  • --ntasks=<N>: sets the number of tasks, typically MPI ranks. The default is 1.
  • --cpus-per-task=<N>: sets the CPUs per task, typically OpenMP threads. The default is 1.
  • --nodes=<N>: sets the number of nodes. The default is 1.
  • --gpus=[type:]<number>: requests GPUs for the job.
  • --gpus-per-node=[type:]<number>: requests GPUs per node.
  • --gpus-per-socket=[type:]<number>: requests GPUs per socket.
  • --gpus-per-task=[type:]<number>: requests GPUs per task.

Important: The more accurately you specify job requirements, the more efficiently the cluster can be used.

Specify these requirements as options to sbatch or as #SBATCH directives in a batch script. For example:

#!/bin/bash
#SBATCH -J JobName
#SBATCH --time=08:00:00 # Walltime
#SBATCH --mem-per-cpu=4096 # memory/cpu (in MB)
#SBATCH --ntasks=2 # 2 tasks
#SBATCH --cpus-per-task=4 # number of cores per task
#SBATCH --nodes=1 # number of nodes
#SBATCH --gpus-per-node=nvidia_h200:2 # gpus h200 per node

CPU affinity and HyperThreading (SMT)

Compute nodes may have HyperThreading/SMT enabled. In the current clusters configuration, Slurm accounts logical CPUs (hardware threads) as schedulable resources (SelectTypeParameters=CR_CPU_Memory).

This means that, by default, a job requesting multiple CPUs may receive: - Different physical cores, or - Sibling SMT threads belonging to the same physical core.

⚠️ Note: For some CPU-bound workloads, using SMT siblings instead of independent physical cores may reduce performance.

For example, a task affinity like 0,32 usually means that the process is running on two hardware threads of the same physical core.

If your application benefits from using independent physical cores (typical in OpenMP, shared-memory, numerical simulations, etc.), you should explicitly request one thread per core using:

#SBATCH --hint=nomultithread

Example:

#!/bin/bash
#SBATCH -J JobName
#SBATCH -c 4
#SBATCH --hint=nomultithread
#SBATCH -t 00:10:00

This option instructs Slurm to avoid placing tasks on SMT sibling threads whenever possible, preferring allocation on separate physical cores.

💡 For CPU-bound applications, enabling --hint=nomultithread can significantly improve and stabilize performance.

You can inspect the CPU affinity assigned to your job using:

taskset -cp $$

or:

srun --cpu-bind=verbose ...

Job states

State Abbreviation Description
BOOT_FAIL BF Job terminated due to a launch or boot failure, typically caused by hardware issues (e.g., unable to boot a node or block and the job cannot be requeued).
CANCELLED CA Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
COMPLETED CD Job has terminated all processes on all nodes with an exit code of zero (successful completion).
COMPLETING CG Job is in the process of completing. Some processes on some nodes may still be active while SLURM cleans up resources.
CONFIGURING CF Job is being configured, typically waiting for nodes to be allocated before execution begins.
DEADLINE DL Job terminated upon reaching its deadline.
FAILED F Job terminated with a non-zero exit code or experienced another failure condition.
NODE_FAIL NF Job terminated due to failure of one or more allocated nodes.
OUT_OF_MEMORY OOM Job was terminated after exceeding its allocated memory.
PENDING PD Job is awaiting resource allocation.
PREEMPTED PR Job was terminated (or suspended) due to preemption by another job with higher priority.
RUNNING R Job currently has an allocation and is executing.
RESV_DEL_HOLD RD Job is being held because the requested reservation was deleted.
REQUEUE_HOLD RH Held job is being requeued.
REQUEUED RQ Completing or failed job has been requeued for another execution attempt.
RESIZING RS Job is changing its size (e.g., adding or releasing nodes).
REVOKED RV Job was revoked, typically due to dependency problems or system maintenance.
SIGNALING SI SLURM is signaling the job (e.g., sending SIGTERM or SIGKILL).
SPECIAL_EXIT SE Job terminated with a special exit condition defined by the system.
STOPPED ST Job has an allocation, but execution has been stopped with a SIGSTOP signal. CPUs remain allocated to the job.
SUSPENDED S Job has an allocation but execution is suspended and CPUs are released for other jobs.
TIMEOUT TO Job terminated after reaching its time limit.

Default priority

Slurm assigns each job a priority and schedules higher-priority jobs as resources become available. It recalculates priorities at regular intervals. Priorities may be based on several factors, each with a different weight:

  • Fairshare: your job will be given an initial score based on your share and your historical use of the cluster, with your recent use being given more weight.
  • Partition priority: the partition priority is based on job length. Longer jobs receive lower priority.
  • Job size: jobs with a higher ratio of cores to walltime receive higher priority.
  • Wait time: a job's priority increases as it waits. This bonus gradually increases over time.

Slurm calculates job priority from a base factor, weighted scores, resource (TRES) contributions, and a nice adjustment:

  • site_factor: the site-defined base contribution.
  • Weighted factors (each factor is multiplied by its corresponding priority weight):
    • PriorityWeightAge × age_factor
    • PriorityWeightAssoc × assoc_factor
    • PriorityWeightFairshare × fair-share_factor
    • PriorityWeightJobSize × job_size_factor
    • PriorityWeightPartition × partition_factor
    • PriorityWeightQOS × QOS_factor
  • TRES contributions, such as TRES_weight_cpu × TRES_factor_cpu, plus equivalent terms for other tracked resources.
  • nice_factor: the job's nice adjustment.

Configured weights:

PriorityWeightFairShare = 10000  
PriorityWeightJobSize = 1000  
PriorityWeightPartition = 10  
PriorityWeightQOS = 1000

Running a job

Most common Slurm commands

  • sbatch: submits a batch job script.
  • scancel: cancels a running or pending job.
  • srun: runs a command on the allocated node(s).
  • sbcast: transfers files to the compute nodes allocated for the job.
  • sattach: connects to the standard input, output, and error streams of an existing job or job step.
  • squeue: displays the job queue and job status.

Commonly used Slurm variables

  • $SLURM_JOBID: job ID.
  • $SLURM_JOB_NODELIST: nodes allocated to the job.
  • $SLURM_NNODES: number of allocated nodes.
  • $SLURM_SUBMIT_DIR: directory from which the job was submitted.
  • $SLURM_ARRAY_JOB_ID: job ID of the array.
  • $SLURM_ARRAY_TASK_ID: index of the current array task.

Temporary folders

The following temporary folders are created for each job by the job_container/tmpfs plugin, which provides job-specific, private temporary file system space. Their contents are removed when the job completes. These folders are intended for high-I/O operations. To take advantage of these high-performance file systems, stage the required files in and out.

File System Job Path Real Path
local disk /tmp /scratch/$SLURM_JOBID/.$SLURM_JOBID
local memory /dev/shm New mount for every job

Examples of batch jobs

Submit a batch job with sbatch:

sbatch --partition=sci -t 00:01:00 --wrap "sleep 30; echo hello world"

  • The -t option sets the maximum run time for the job allocation.

  • If no time limit is specified, Slurm applies the maximum time limit for the default partition.

  • The --wrap option tells Slurm to run the following string as a simple shell script.

Examples of interactive jobs

Interactive job selecting partition.

srun --partition=sci --pty bash

Interactive job selecting partition and node.

salloc --partition=sci -w sci01

Interactive job selecting partition, memory per cpu, cores and walltime.

salloc --partition=sci --mem-per-cpu=8G -c 4 -t 06:00:00

Interactive job selecting partition, GPU type and count, memory per CPU, cores and walltime.

salloc --partition=sci --mem-per-cpu=16G -c 4 -t 06:00:00 --gpus=t4:1

Interactive job selecting a partition, memory per CPU, cores, and walltime, then starting Python interactively.

srun --partition=sci --mem-per-cpu=4G -c 2 -t 06:00:00 --pty python3

💡 When using srun in ce200 partition, specify --mpi=none.

💡 In an interactive session, unset Slurm environment variables before launching a nested job with unset $(compgen -v | grep "^SLURM").

Monitoring your work on the cluster

Jobs are scheduled according to their relative priority. Use squeue to list batch jobs. To request estimated start times, use squeue --start.

sbatch -t 00:01:00 --wrap "sleep 30; echo hello world"
Submitted batch job 2

squeue
JOBID PARTITION NAME USER  ST TIME NODES NODELIST(REASON)
2     main      wrap user R  0:02 1     wn061
  • JOBID: the job identifier.
  • PARTITION: the partition to which the job was submitted.
  • NAME: the job name.
  • USER: the submitting user.
  • ST: the current job state.
  • TIME: the elapsed time.
  • NODES: the number of allocated nodes.
  • NODELIST(REASON): the allocated node(s), or the reason a pending job is waiting.

Use sacct -j <JOBID> --format=JobID,MaxRSS,ReqMem to compare peak memory usage (MaxRSS) with requested memory (ReqMem). If MaxRSS is close to ReqMem, consider requesting more memory; if it is much lower, you may be able to request less.

Connecting to a compute node during job execution

While a job is running, you can open an interactive shell on the node where it is executing:

srun --jobid <JOBID> --overlap --pty bash -i

Note: This only works while the job is actively running.

For multi-node jobs (e.g. MPI), you can select a specific node with -w:

srun --jobid <JOBID> --overlap -w <NODENAME> --pty bash -i

Job efficiency

The seff Perl utility is available to check a specific job's efficiency:

$ seff 2
Job ID: 2
Cluster: sci
User/Group: <username>/<group>
State: COMPLETED (exit code 0)
Cores: 1
CPU Utilized: 00:00:00
CPU Efficiency: 0.00% of 00:00:31 core-walltime
Job Wall-clock time: 00:00:31
Memory Utilized: 860.00 KB
Memory Efficiency: 0.04% of 2.00 GB (2.00 GB/core)

Job outputs

The default output files are written to the submission directory as slurm-$SLURM_JOBID.out and slurm-$SLURM_JOBID.err. Use these options to choose different output and error file names: - -o <filename>: sets the standard output file (for example, -o test.out). - -e <filename>: sets the standard error file (for example, -e test.err).

In output and error file names, sbatch replaces the following percent-based filename patterns with job information:

Note: Keep these patterns exactly as shown in the filename; Slurm replaces them when it creates the files.

  • %%: a literal percent sign.
  • %A: the master job ID of a job array.
  • %a: the array task index.
  • %J: the job ID and step ID, separated by a period (for example, 128.0).
  • %j: the job ID.
  • %N: the short hostname. This creates a separate file for each node.
  • %n: the node index within the job, starting at 0. This creates a separate file for each node.
  • %s: the step ID.
  • %t: the task index (rank) within the job. This creates a separate file for each task.
  • %u: the username.
  • %x: the job name.

Email notifications from jobs

SLURM can send email notifications when a job starts, finishes, fails, or is requeued. Add the following directives to the batch script:

#!/bin/bash
#SBATCH --mail-user=your_user@domain.com
#SBATCH --mail-type=BEGIN,END,FAIL,REQUEUE

You can also set these options directly in the sbatch command:

sbatch --mail-user=your_user@domain.com --mail-type=BEGIN,END,FAIL,REQUEUE my_job.sh

Common event types are:

  • NONE: disables e-mail notifications.
  • BEGIN: job starts running.
  • END: job completes successfully.
  • FAIL: job terminates with a non-zero exit code.
  • REQUEUE: job is requeued.
  • ALL: enables all of the above notifications.

This is especially useful for long-running jobs, jobs that depend on other tasks, or when you need to know quickly if a calculation ended with an error.

Canceling jobs

To cancel a pending or running job, run:

scancel <JOBID>

To cancel all your jobs, use the -u option with your username:

scancel -u "$USER"

Slurm scripts examples

Serial example

/nfs/admin/slurm/ops/slurm-examples/serial.sh

#!/bin/bash
#-----------------------------------------------------------------
# Example SLURM job script to run serial applications
#----------------------------------------------------------------- 

#SBATCH -J mysimplejob            # Job name
#SBATCH -o mysimplejob.%j.out     # Specify stdout output file (%j expands to jobId)
# No partition specification needed
#SBATCH -n 1                      # Total number of tasks
#SBATCH -t 01:30:00               # Run time (hh:mm:ss) - 1.5 hours
#SBATCH --mem=8G                  # Total memory demanded - 8 GB RAM

# Load any necessary modules
# Loading modules in the script ensures a consistent environment.
ml hdf5-1.14.5/gcc-8.5.0-q24s5

# Launch the executable named "h5perf_serial"
h5perf_serial
# check the hostname
hostname

Shared memory (OpenMP) example

/nfs/admin/slurm/ops/slurm-examples/compileOpenMP.sh

#!/bin/bash
module load intel-oneapi-compilers-2024.2.0/gcc-8.5.0-st4zi
icx -qopenmp -o omphello ./omphello.c

#module load gcc-13.3.0/gcc-8.5.0-rt6fd
#gcc -fopenmp -o omphello ./omphello.c

/nfs/admin/slurm/ops/slurm-examples/OpenMP.sh

#!/bin/bash
#----------------------------------------------------
# Example SLURM job script to run OpenMP applications
#----------------------------------------------------
#SBATCH -J openmp_job          # Job name
#SBATCH -o openmp_job.o%j      # Name of stdout output file(%j expands to jobId)
#SBATCH -e openmp_job.o%j      # Name of stderr output file(%j expands to jobId)
#SBATCH -c 8                   # Cores per task requested (1 task job)
#SBATCH --hint=nomultithread   # Disabled Sibling SMT threads   
#SBATCH -t 00:10:00            # Run time (hh:mm:ss) - 10 min
#SBATCH --mem-per-cpu=3G       # Memory per core demandes (24 GB in total: 3GB * 8 cores)

# This example will run an OpenMP application using 8 threads
module load intel-oneapi-compilers-2024.2.0/gcc-8.5.0-st4zi

# Run the OpenMP application
./omphello

Distributed memory (MPI) examples

/nfs/admin/slurm/ops/slurm-examples/compileMPI.sh

#!/bin/bash
module load openmpi-5.0.5/gcc-8.5.0-slurm-ogwvq
mpifort -o pi ./pi3f90.f90

/nfs/admin/slurm/ops/slurm-examples/MPI.sh

#!/bin/bash
#----------------------------------------------------
# Generic SLURM script -- MPI Hello World
#
# This script requests 2 nodes and 20 cores/node
# for a total of 40 MPI tasks.
#----------------------------------------------------
#SBATCH -J mpijob              # Job name
#SBATCH -o mpijob.%j.out       # stdout; %j expands to jobid
#SBATCH -e mpijob.%j.err       # stderr; skip to combine stdout and stderr
#SBATCH -N 2                   # Number of nodes, not cores (64 cores/node)
#SBATCH -n 40                  # Total number of MPI tasks (if omitted, n=N)
#SBATCH --ntasks-per-node=20   # MPI tasks per node
#SBATCH -t 00:30:00            # max time
#SBATCH --mem-per-cpu=1G       # memory per core demanded

# Desactivar UCX (usa TCP y shared-memory)
# export OMPI_MCA_pml=ob1
# export OMPI_MCA_btl=self,vader,tcp

# Indicar explícitamente el interfaz de red (Solo necesario en la particion geocean)
# https://www.mail-archive.com/search?l=users@lists.open-mpi.org&q=subject:%22Re\%3A+\[OMPI+users\]+Unable+to+run+complicated+MPI+Program%22&o=newest&f=1
# export OMPI_MCA_btl_tcp_if_include=eth0

ml openmpi-5.0.5/gcc-8.5.0-slurm-ogwvq

srun ./pi                   # Do not add "-n" or "-np" options here. SLURM infers the
                            # process count from the "-N" and "-n" directives above.

Hybrid example (MPI+OpenMPI)

/nfs/admin/slurm/ops/slurm-examples/compileMPIOpenMP.sh /nfs/admin/slurm/ops/slurm-examples/help_fortran_find_core_id.c /nfs/admin/slurm/ops/slurm-examples/hybrid.f90 /nfs/admin/slurm/ops/slurm-examples/hybrid.c

#!/bin/bash
# FORTRAN
# INTEL
#ml iimpi/2022b
#icx -c help_fortran_find_core_id.c
#mpiifort -qopenmp -o hybrid ./hybrid.f90 help_fortran_find_core_id.o
# GNU
#ml gompi/2022b
#gcc -c help_fortran_find_core_id.c
#mpif90 -fopenmp -ffree-line-length-256 -o hybrid ./hybrid.f90 help_fortran_find_core_id.o

# C
ml iimpi/2022b
mpiicc -cc=icx -qopenmp -Wimplicit-function-declaration -o hybrid ./hybrid.c
# GNU
#ml gompi/2022b
#mpicc -fopenmp o hybrid ./hybrid.c

/nfs/admin/slurm/ops/slurm-examples/MPIOpenMP.sh

#!/bin/bash
#SBATCH -J MPIOpenMP -o %x-%J.out
#SBATCH -t 00:20:00
#SBATCH -n 8 --ntasks-per-node=4 -c 8

# Desactivar UCX (usa TCP y shared-memory)
# export OMPI_MCA_pml=ob1
# export OMPI_MCA_btl=self,vader,tcp

# Indicar explícitamente el interfaz de red (Solo necesario en la particion geocean)
# https://www.mail-archive.com/search?l=users@lists.open-mpi.org&q=subject:%22Re\%3A+\[OMPI+users\]+Unable+to+run+complicated+MPI+Program%22&o=newest&f=1
# export OMPI_MCA_btl_tcp_if_include=eth0

ml iimpi/2022b
#ml gompi/2022b
srun --cpu_bind=verbose ./hybrid

💡 Use the --exclusive option to request exclusive access to the allocated node(s).

To submit a job with exclusive access to its node(s), use sbatch --exclusive:

srun --cpu_bind=verbose,mask_cpu:
0x000000ff,
0x0000ff00,
0x00ff0000,
0xff000000
./hybrid

Job array example

/nfs/admin/slurm/ops/slurm-examples/JobArray.sh

#!/bin/bash
#-----------------------------------------------------------------
# Example SLURM job array script
#-----------------------------------------------------------------
#SBATCH -J jobarray # Job name
#SBATCH -o %x-%A-%a.out # Specify stdout output file (%A expands to array jobId, %a expands to array task id)
##SBATCH -a 1,6,16-32
#SBATCH --array=0-15:4
#SBATCH -n 1  # Total number of tasks and memory per core
#SBATCH -t 00:10:00 # Run time (hh:mm:ss) - 1.5 hours

# Load any necessary modules
# Loading modules in the script ensures a consistent environment.
# module load ...

# run the task:
./task.sh

GPU job example

/nfs/admin/slurm/ops/slurm-examples/compileCUDA.sh

#!/bin/bash
#cuda samples repo needed:
#git clone -b v12.1 https://github.com/NVIDIA/cuda-samples.git
#Compilation should be done on a gpu node:
#salloc -c 8 --gpus 1
ml CUDA/12.1.1 GCC
nvcc -o deviceQuery_cuda12 -I./cuda-samples/Common \
cuda-samples/Samples/1_Utilities/deviceQuery/deviceQuery.cpp

/nfs/admin/slurm/ops/slurm-examples/CUDA.sh

#!/bin/bash
#----------------------------------------------------
# Example SLURM job script to run CUDA applications
#----------------------------------------------------
#SBATCH -J gpu_job                # Job name
#SBATCH -o gpu_job.o%j            # Name of stdout output file(%j expands to jobId)
#SBATCH -e gpu_job.o%j            # Name of stderr output file(%j expands to jobId)
#SBATCH -c 32 --mem-per-cpu=2G   # Cores per task requested (1 task job)
#SBATCH --gpus=nvidia_h200:2      # Options for requesting 2 GPUs
#SBATCH -t 01:30:00               # Run time (hh:mm:ss) - 1.5 hours

# Run the CUDA application
module load CUDA/12.1.1
./deviceQuery_cuda12

Enroot + Pyxis example

How to use Enroot

Pull the image from docker hub

enroot import docker://ubuntu

Create the container

enroot create library+ubuntu+latest.sqsh

Start the container

enroot start library+ubuntu+latest

If you need to run something as root inside the container, you can use the --root option.

enroot start --root library+ubuntu+latest

List the existing containers

enroot list -f

Remove a container

enroot remove library+ubuntu+latest
How to use Pyxis

Run a command on a node

srun -p main cat /etc/os-release
…
PRETTY_NAME="Rocky Linux 8.10 (Green Obsidian)"

Run the same command, but now inside of a container

srun -p main --container-image=$(pwd)/library+ubuntu+latest.sqsh --container-name=ubuntu cat /etc/os-release
…
PRETTY_NAME="Ubuntu 24.04.3 LTS"
…

Mount a file from the host and run the command on it, from inside the container

srun -p main --container-name=ubuntu --container-mounts=/etc/os-release:/etc/os-release cat /etc/os-release
…
PRETTY_NAME="Rocky Linux 8.10 (Green Obsidian)"

To see more options

srun --help | grep container

Execute the sbatch script inside a container image, a real application example GROMACS

#!/bin/bash
#SBATCH -p main -t 30:00
#SBATCH --container-mounts /var/spool/slurm,/nfs/home/<group>/<username>/slurm-sci/stmv:/host_pwd
#SBATCH --container-workdir=/host_pwd
#SBATCH --container-image=nvcr.io#hpc/gromacs:2021.3
# Alternatively, use a local SquashFS image:
##SBATCH --container-image=/nfs/home/<group>/<username>/slurm-sci/hpc+gromacs+2021.3.sqsh
#SBATCH --container-name hpc+gromacs+2021.3
export GMX_ENABLE_DIRECT_GPU_COMM=1
/usr/local/gromacs/avx2_256/bin/gmx mdrun -ntmpi 8 -ntomp 16 -nb gpu -pme gpu -npme 1 -update gpu -bonded gpu -nsteps 100000 -resetstep 90000 -noconfout -dlb no -nstlist 300 -pin on -v -gpu_id 0123