Page History

...

1. Load the xthi module with:

   No Format
   module load xthi

   
   

2. Run the program interactively to familiarise yourself with the ouptut:

   No Format
   $ xthi Host=ac6-200 MPI Rank=0 CPU=128 NUMA Node=0 CPU Affinity=0,128

   As you can see,  only 1 process and 1 thread are run, and they may run on one of two virtual cores assigned to my session (which correspond to the same physical CPU). If you try to run with 4 OpenMP threads, you will see they will effectively fight each other for those same two cores, impacting the performance of your application but not anyone else in the login node:

   No Format
   $ OMP_NUM_THREADS=4 xthi Host=ac6-200 MPI Rank=0 OMP Thread=0 CPU=128 NUMA Node=0 CPU Affinity=0,128 Host=ac6-200 MPI Rank=0 OMP Thread=1 CPU= 0 NUMA Node=0 CPU Affinity=0,128 Host=ac6-200 MPI Rank=0 OMP Thread=2 CPU=128 NUMA Node=0 CPU Affinity=0,128 Host=ac6-200 MPI Rank=0 OMP Thread=3 CPU= 0 NUMA Node=0 CPU Affinity=0,128

   
   

3. Create a new job script fractional.sh to run xthi with 2 MPI tasks and 2 OpenMP threads, submit it and check the output to ensure the right number of tasks and threads were spawned. 

   Here is a job template to start with:

   Code Block
   language bash title fractional.sh collapse true
   #!/bin/bash #SBATCH --output=fractional.out # TODO: Add here the missing SBATCH directives for the relevant resources # Define the number of OMP_NUM_THREADSOpenMP threads export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1} # Load xthi tool module load xthi # TODO: Add here the line to run xthi # Hint: use srun

   
   

   Expand
   title Solution
   Using your favourite editor, create a file called fractional.sh with the following content: Code Block language bash title fractional.sh #!/bin/bash #SBATCH --output=fractional.out # Add here the missing SBATCH directives for the relevant resources #SBATCH --ntasks=2 #SBATCH --cpus-per-task=2 module load  xthi # AddDefine herethe thenumber lineof toOpenMP run xthi # Hint: use srunthreads export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1} # Load xthi tool module load xthi srun -c $SLURM_CPUS_PER_TASK xthi You need to request 2 tasks, and 2 cpus per task in the job. Then we will use srun to spawn our parallel run, which should inherit the job geometry requested, except the cpus-per-task, which must be explicitly passed to srun. You can submit it with sbatch: No Format sbatch fractional.sh The job should be run shortly. When finished, a new file called fractional.out should appear in the same directory. You can check the relevant output with: No Format grep -v ECMWF-INFO fractional.out You should see an output similar to: No Format $ grep -v ECMWF-INFO fractional.out Host=ad6-202 MPI Rank=0 OMP Thread=0 CPU= 5 NUMA Node=0 CPU Affinity=5,133 Host=ad6-202 MPI Rank=0 OMP Thread=1 CPU=133 NUMA Node=0 CPU Affinity=5,133 Host=ad6-202 MPI Rank=1 OMP Thread=0 CPU=137 NUMA Node=0 CPU Affinity=9,137 Host=ad6-202 MPI Rank=1 OMP Thread=1 CPU= 9 NUMA Node=0 CPU Affinity=9,137 Info title Srun automatic cpu binding You can see srun automatically ensures certain binding of the cores to the tasks. If you were to instruct srun to avoid any cpu binding with --cpu-bind=none, you would see something like: No Format $ grep -v ECMWF-INFO fractional.out Host=aa6-203 MPI Rank=0 OMP Thread=0 CPU=136 NUMA Node=0 CPU Affinity=4,8,132,136 Host=aa6-203 MPI Rank=0 OMP Thread=1 CPU= 8 NUMA Node=0 CPU Affinity=4,8,132,136 Host=aa6-203 MPI Rank=1 OMP Thread=0 CPU=132 NUMA Node=0 CPU Affinity=4,8,132,136 Host=aa6-203 MPI Rank=1 OMP Thread=1 CPU= 4 NUMA Node=0 CPU Affinity=4,8,132,136 Here all processes/threads could run in any of the cores assigned to the job, potentially having them hopping from cpu to cpu during the program's execution

   
   

4. Can you ensure each one of the OpenMP threads runs on a single physical core, without exploiting the hyperthreading, for optimal performance?

   Expand
   title Solution
   In order to ensure each thread gets their own core, you can use the environment variable OMP_PLACES=threads. Then, to make sure only physical cores are used for performance, we need to use the --hint=nomultithread directive: Code Block language bash title fractional.sh #!/bin/bash #SBATCH --output=fractional.out # Add here the missing SBATCH directives for the relevant resources #SBATCH --ntasks=2 #SBATCH --cpus-per-task=2 #SBATCH --hint=nomultithreadno multithread module# load xthi # Add here the line to run xthi # Hint: use srunDefine the number of OpenMP threads export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1} # Ensure proper OpenMP thread CPU pinning export OMP_PLACES=threads # Load xthi tool module load xthi srun -c $SLURM_CPUS_PER_TASK xthi You can submit the modified job with sbatch: No Format sbatch fractional.sh You should see an output similar to the following one, where each thread is in a different core with a number lower than 128: No Format $ grep -v ECMWF-INFO fractional.out Host=ad6-201 MPI Rank=0 OMP Thread=0 CPU=18 NUMA Node=1 CPU Affinity=18 Host=ad6-201 MPI Rank=0 OMP Thread=1 CPU=20 NUMA Node=1 CPU Affinity=20 Host=ad6-201 MPI Rank=1 OMP Thread=0 CPU=21 NUMA Node=1 CPU Affinity=21 Host=ad6-201 MPI Rank=1 OMP Thread=1 CPU=22 NUMA Node=1 CPU Affinity=22

   
   

...

1. If not already on HPCF, open a session on hpc-login.

2. Create a new job script parallel.sh to run xthi with 32 MPI tasks and 4 OpenMP threads, leaving hyperthreading enabled. Submit it and check the output to ensure the right number of tasks and threads were spawned. Take note of what cpus are used, and how much SBUs you used.

   Here is a job template to start with:

   Code Block
   language bash title parallel.sh collapse true
   #!/bin/bash #SBATCH --output=parallel-%j.out #SBATCH --qos=np # TODO: Add here the missing SBATCH directives for the relevant resources   module load xthi # Define the number of OpenMP threads export OMP_NUM_PLACES=threads srun -c $SLURMTHREADS=${SLURM_CPUS_PER_TASK xthi :-1} # Ensure proper OpenMP thread CPU pinning export OMP_PLACES=threads # Load xthi tool module load xthi srun -c $SLURM_CPUS_PER_TASK xthi

   
   

   Expand
   title Solution
   Using your favourite editor, create a file called parallel.sh with the following content: Code Block language bash title paralell.sh #!/bin/bash #SBATCH --output=parallel-%j.out #SBATCH --qos=np # Add here the missing SBATCH directives for the relevant resources #SBATCH --ntasks=32 #SBATCH --cpus-per-task=4 module load xthi # Define the number of OpenMP threads export OMP_NUM_PLACES=threads srun -c $SLURMTHREADS=${SLURM_CPUS_PER_TASK xthi:-1} # Ensure proper OpenMP thread CPU pinning export OMP_PLACES=threads # Load xthi tool module load xthi srun -c $SLURM_CPUS_PER_TASK xthi You need You need to request 32 tasks, and 4 cpus per task in the job. Then we will use srun to spawn our parallel run, which should inherit the job geometry requested, except the cpus-per-task, which must be explicitly passed to srun. You can submit it with sbatch: No Format sbatch parallel.sh The job should be run shortly. When finished, a new file called parallel-<jobid>.out should appear in the same directory. You can check the relevant output with: No Format grep -v ECMWF-INFO $(ls -1 parallel-*.out | tail -n1) You should see an output similar to: No Format Host=ac2-4046 MPI Rank= 0 OMP Thread=0 CPU= 0 NUMA Node=0 CPU Affinity= 0 Host=ac2-4046 MPI Rank= 0 OMP Thread=1 CPU=128 NUMA Node=0 CPU Affinity=128 Host=ac2-4046 MPI Rank= 0 OMP Thread=2 CPU= 1 NUMA Node=0 CPU Affinity= 1 Host=ac2-4046 MPI Rank= 0 OMP Thread=3 CPU=129 NUMA Node=0 CPU Affinity=129 Host=ac2-4046 MPI Rank= 1 OMP Thread=0 CPU= 2 NUMA Node=0 CPU Affinity= 2 Host=ac2-4046 MPI Rank= 1 OMP Thread=1 CPU=130 NUMA Node=0 CPU Affinity=130 Host=ac2-4046 MPI Rank= 1 OMP Thread=2 CPU= 3 NUMA Node=0 CPU Affinity= 3 Host=ac2-4046 MPI Rank= 1 OMP Thread=3 CPU=131 NUMA Node=0 CPU Affinity=131 ... Host=ac2-4046 MPI Rank=30 OMP Thread=0 CPU=116 NUMA Node=7 CPU Affinity=116 Host=ac2-4046 MPI Rank=30 OMP Thread=1 CPU=244 NUMA Node=7 CPU Affinity=244 Host=ac2-4046 MPI Rank=30 OMP Thread=2 CPU=117 NUMA Node=7 CPU Affinity=117 Host=ac2-4046 MPI Rank=30 OMP Thread=3 CPU=245 NUMA Node=7 CPU Affinity=245 Host=ac2-4046 MPI Rank=31 OMP Thread=0 CPU=118 NUMA Node=7 CPU Affinity=118 Host=ac2-4046 MPI Rank=31 OMP Thread=1 CPU=246 NUMA Node=7 CPU Affinity=246 Host=ac2-4046 MPI Rank=31 OMP Thread=2 CPU=119 NUMA Node=7 CPU Affinity=119 Host=ac2-4046 MPI Rank=31 OMP Thread=3 CPU=247 NUMA Node=7 CPU Affinity=247 Note the following facts: Both the main cores (0-127) and hyperthreads (128-256) were used. You get consecutive threads on the same physical CPU (0 with 128, 1 with 129...). There are physical cpus entirely unused, since their cpu number does show in the output. In terms of SBUs, this job cost: No Format $ grep SBU $(ls -1 parallel-*.out | tail -n1) [ECMWF-INFO -ecepilog] SBU : 6.051

   
   

3. Modify the parallel.sh job geometry (number of tasks,  threads and use of hyperthreading) so that you fully utilise all the physical cores, and only those, i.e. 0-127.

   Expand
   title Solution
   Without using hyperthreading, an Atos HPCF node has 128 phyisical cores available. Any combination of tasks and threads that adds up to that figure will fill the node. Examples include 32 tasks x 4 threads, 64 tasks x 2 threads or 128 single-threaded tasks. For this example, we picked the first one: Code Block language bash title paralell.sh #!/bin/bash #SBATCH --output=parallel-%j.out #SBATCH --qos=np # Add here the missing SBATCH directives for the relevant resources #SBATCH --ntasks=32 #SBATCH --cpus-per-task=4 #SBATCH --hint=nomultithread module load xthi # Define the number of OpenMP threads export OMP_NUM_PLACES=threads srun -c $SLURMTHREADS=${SLURM_CPUS_PER_TASK xthi:-1} # Ensure proper OpenMP thread CPU pinning export OMP_PLACES=threads # Load xthi tool module load xthi srun -c $SLURM_CPUS_PER_TASK xthi You can submit it You can submit it with sbatch: No Format sbatch parallel.sh The job should be run shortly. When finished, a new file called parallel-<jobid>.out should appear in the same directory. You can check the relevant output with: No Format grep -v ECMWF-INFO $(ls -1 parallel-*.out | tail -n1) You should see an output similar to: No Format Host=ac3-2015 MPI Rank= 0 OMP Thread=0 CPU= 0 NUMA Node=0 CPU Affinity= 0 Host=ac3-2015 MPI Rank= 0 OMP Thread=1 CPU= 1 NUMA Node=0 CPU Affinity= 1 Host=ac3-2015 MPI Rank= 0 OMP Thread=2 CPU= 2 NUMA Node=0 CPU Affinity= 2 Host=ac3-2015 MPI Rank= 0 OMP Thread=3 CPU= 3 NUMA Node=0 CPU Affinity= 3 Host=ac3-2015 MPI Rank= 1 OMP Thread=0 CPU= 4 NUMA Node=0 CPU Affinity= 4 Host=ac3-2015 MPI Rank= 1 OMP Thread=1 CPU= 5 NUMA Node=0 CPU Affinity= 5 Host=ac3-2015 MPI Rank= 1 OMP Thread=2 CPU= 6 NUMA Node=0 CPU Affinity= 6 Host=ac3-2015 MPI Rank= 1 OMP Thread=3 CPU= 7 NUMA Node=0 CPU Affinity= 7 ... Host=ac3-2015 MPI Rank=30 OMP Thread=0 CPU=120 NUMA Node=7 CPU Affinity=120 Host=ac3-2015 MPI Rank=30 OMP Thread=1 CPU=121 NUMA Node=7 CPU Affinity=121 Host=ac3-2015 MPI Rank=30 OMP Thread=2 CPU=122 NUMA Node=7 CPU Affinity=122 Host=ac3-2015 MPI Rank=30 OMP Thread=3 CPU=123 NUMA Node=7 CPU Affinity=123 Host=ac3-2015 MPI Rank=31 OMP Thread=0 CPU=124 NUMA Node=7 CPU Affinity=124 Host=ac3-2015 MPI Rank=31 OMP Thread=1 CPU=125 NUMA Node=7 CPU Affinity=125 Host=ac3-2015 MPI Rank=31 OMP Thread=2 CPU=126 NUMA Node=7 CPU Affinity=126 Host=ac3-2015 MPI Rank=31 OMP Thread=3 CPU=127 NUMA Node=7 CPU Affinity=127 Note the following facts: Only the main cores (0-127) were used. Each thread gets one and only one cpu pinned to it. All the phyisical cores are in use In terms of SBUs, this job cost: No Format $ grep SBU $(ls -1 parallel-*.out | tail -n1) [ECMWF-INFO -ecepilog] SBU : 5.379

   
   

4. Modify the parallel.sh job geometry so it still runs on the np qos QoS, but only with 2 tasks and 2 threads. Check the SBU cost. Since the execution is 32 times smaller, did it cost 32 times less than the previous? Why?

   Expand
   title Solution
   Let's use the following job: Code Block language bash title paralell.sh #!/bin/bash #SBATCH --output=parallel-%j.out #SBATCH --qos=np # Add here the missing SBATCH directives for the relevant resources #SBATCH --ntasks=2 #SBATCH --cpus-per-task=2 #SBATCH --hint=nomultithread module load xthi export OMP_PLACES=threads srun -c $SLURM_CPUS_PER_TASK xthi You can submit it with sbatch: No Format sbatch fractional.sh The job should be run shortly. When finished, a new file called parallel-<jobid>.out should appear in the same directory. You can check the relevant output with: No Format grep -v ECMWF-INFO $(ls -1 parallel-*.out | tail -n1) You should see an output similar to: No Format Host=ac2-3073 MPI Rank=0 OMP Thread=0 CPU= 0 NUMA Node=0 CPU Affinity= 0 Host=ac2-3073 MPI Rank=0 OMP Thread=1 CPU= 1 NUMA Node=0 CPU Affinity= 1 Host=ac2-3073 MPI Rank=1 OMP Thread=0 CPU=16 NUMA Node=1 CPU Affinity=16 Host=ac2-3073 MPI Rank=1 OMP Thread=1 CPU=17 NUMA Node=1 CPU Affinity=17 In terms of SBUs, this job costof SBUs, this job cost: No Format $ grep SBU $(ls -1 parallel-*.out | tail -n1) [ECMWF-INFO -ecepilog] SBU : 4.034 This is in a similar scale to the previous one which 32 times bigger one. The reason behind it is that on the np QoS the allocation is done in full nodes. The SBU cost takes into account the allocated nodes for a given period of time, no matter how they are used. You may compare the cost of your last parallel job and your last fractional, with the same geometry (2x2): No Format $ grep -h SBU $(ls -1 parallel-*.out | tail -n1) fractional.out [ECMWF-INFO -ecepilog] SBU : 4.034 This is in a similar scale to the previous one which 32 times bigger one. The reason behind it is that on the np QoS the allocation is done in full nodes. The SBU cost takes into account the allocated nodes for a given period of time, no matter how they are used. [ECMWF-INFO -ecepilog] SBU : 0.084