When running parallel jobs, SLURM will automatically set up some default process affinity. This means that every task spawned by srun (each MPI rank on an MPI execution) will be pinned to a specific core or set of cores within every computing node.
However, the default affinity may not be what you would expect, and depending on the application it could have a significant impact in performance.
Slurm Reference
You may see all the possible options when it comes to setting up Affinity in SLURM in the official documentation pages https://slurm.schedmd.com/mc_support.html
Below are some examples of how the affinity is setup by default in the different cases.
Understanding CPU layout
Every node has a total of 256 virtual cores (128 physical). Every core will have an ID, with IDs 0 and 128 being the two hardware threads on the same physical core. See Atos HPCF: System overview for all the details.
MPI single threaded execution
Default setup
This is the simplest case. Slurm will define affinity at the level of physical cores, but allow the task to use the two hardware threads. In this example, we run a 128 task MPI job with default settings, on a single node:
Disabling multithread use
If you want to restrict your process to just one of the hardware threads, you may use the --hint=nomultithread option
By defining the nomultithread, the maximum number of threads is halved from 256 to 128.
Hybrid MPI + OpenMP execution
Default setup - Least ideal
Slurm will allocate a number of cores for each task, and will bind each process to a group of cores equal to the number of threads. However, all threads may run in any of the cores in the group for the same process. In the following example we run a 4 task MPI program, with every rank spawning 4 threads.
In some cases with low number of tasks/threads, it may be necessary to force that binding defining the hint:
#SBATCH --hint=multithread
Binding OpenMP threads to single cores - Best if hyper-threading desired
If you want to bind every thread to a single core, then you may use the OpenMP variable OMP_PLACES.
As you can see, this is not using all the physical cores in the node.
Disabling multithread use - Best if no hyper-threading desired
If you want to avoid having two threads sharing the same physical core and maximise the use of all physical cores in the node, you may use the --hint=nomultithread option.
Further customisation
Only recommended for expert users who want to have full control of how the binding and distribution is done.
If you wish to further customise how the binding and task/thread distribution is done, check the man pages for sbatch an srun, or check the online documentation.
You may also use most flags as both sbatch options/directives or directly on the srun line. If passed onto srun, they will override the configuration coming from the job. Note that srun will fail if trying to use more resources than allocated for the job.
You may use the --cpu-bind option to fine tune how the binding is done. All the possible values are defined in the official man pages. There are the main highlights:
- Binding can be disabled altogether passing
--cpu-bind=notosrun. - You may see the actual binding mask applied by passing
--cpu-bind=verbose. - Custom masks/maps can be defined and passed to
srunwith--cpu-bind=map_cpuand--cpu-bind=mask_cpu.
You can also control how the processes/threads are distributed and bound with the -m or --distribution option in srun. The default is block:cyclic. The first element refers to the distribution of tasks among nodes, and in this case block would distribute them in such a way that consecutive tasks would share a node. The second element is how the CPUs are dstributed within the node. As default cyclic will distribute allocated CPUs for binding to a given task consecutively from the same socket, and from the next consecutive socket for the next task, in a round-robin fashion across sockets.