Job Directives
Any shell script can be submitted as a Slurm job with no modifications. In such a case, sensible default values will be applied to the job. However, you can configure the script to fit your needs through job directives. In Slurm, these are just special comments in your script, usually at the top just after the shebang line, with the form:
...
| Code Block | ||
|---|---|---|
| ||
#!/bin/bash # The job name #SBATCH --job-name=helloworld # Set the error and output files #SBATCH --output=hello-%J.out #SBATCH --error=hello-%J.out # Set the initial working directory #SBATCH --workdirchdir=/scratch/us/usxauser # Choose the queue #SBATCH --–qosqos=express # Wall clock time limit #SBATCH --time=00:05:00 # Send an email on failure #SBATCH --mail-type=FAIL # This is the job echo “Hello World!” sleep 30 |
...
| Note |
|---|
You may need to change the QOS depending on the platform. Check the queues available before submitting. |
| Info |
|---|
You can also use these options as command line arguments to sbatch. |
General directives
| Directive | Description | Default |
|---|---|---|
| Project account for resource accounting and billing purposes. | default project account for the user |
|
| A descriptive name of the job | Script name |
-- |
chdir=... |
| Working directory of the job. The output and error files can be defined relative to this directory | submitting directory | |
| Path to the file where standard output is redirected. Special placeholders for job id (%j) and the execution node (%N) | slurm-%j.out |
| Path to the file where standard error is redirected. Special placeholders for job id (%j) and the execution node (%N) | output value |
|
|
| Quality of Service (or queue) where the job is to be submitted |
| . Check the available queues for the platform. | nf or ef |
|
| Wall clock limit of the job. Note that this is not cpu time limit The format can be: m, m:s, h:m:s, d-h, d-h:m or d-h:m:s | qos default time limit |
--mail-type= |
<type> | Notify user by email when certain event types occur. Valid values are: BEGIN, END, FAIL, REQUEUE and ALL | disabled |
--mail-user= |
<email> | email address to send the email | submitting user |
Directives for resource allocation
| Directive | Description | Default |
|---|---|---|
|
| Allocate resources for the specified number of parallel tasks. Note that a job requesting more than one must be submitted to a parallel queue. There might not be any parallel queue configured on the cluster | 1 |
| Allocate <nodes> number of nodes to the job | 1 |
| Allocate <threads> number of cpus for every task. Use for threaded applications. | 1 |
| Allocate a maximum of <tasks> tasks on every node. | node capacity |
| Allocate <threads> threads on every core (HyperThreading) | core thread capacity |
| Use or not hyperthreaded cores and define the binding accordingly. | not defined |
| Allocate <mem> memory on each node | 8 GB for serial and fractional jobs(*i, *f and *l QoS), 240 for parallel jobs (*p QoS) |
| Tip |
|---|
See man sbatch or https://slurm.schedmd.com/sbatch.html for the complete list of directives and their options. |
| Info |
You can also use these options as command line arguments to sbatch. |
Job variables
Inside a job, you can benefit from some variables defined by SLURM automatically. Some examples are:
- SLURM_JOBID
- SLURM_NODELIST
SLURM_SUBMIT_DIR
...
| Tip | ||
|---|---|---|
For a complete list of variables defined by slurm, you submit a job which runs
|
Job arrays
Job arrays offer a mechanism for submitting and managing collections of similar jobs quickly and easily. The array index values are specified using the --array or -a option of the sbatch command. The option argument can be specific array index values, a range of index values, and an optional step size as shown in the examples below. Jobs which are part of a job array will have the environment variable SLURM_ARRAY_TASK_ID set to its array index value.
| Code Block |
|---|
# Submit a job array with index values between 0 and 31
$ sbatch --array=0-31 -N1 tmp
# Submit a job array with index values of 1, 3, 5 and 7
$ sbatch --array=1,3,5,7 -N1 tmp
# Submit a job array with index values between 1 and 7
# with a step size of 2 (i.e. 1, 3, 5 and 7)
$ sbatch --array=1-7:2 -N1 tmp |
| Tip | |||||
|---|---|---|---|---|---|
The --array option can also be used inside the job script as a job directive. For example:
|