Interactive Jobs¶
Allocation¶
salloc is used to allocate resources in real time to run an interactive batch job. Typically, this is used to allocate resources and spawn a shell. The shell is then used to execute srun commands to launch parallel tasks.
"interactive" QOS on Perlmutter¶
Perlmutter has a dedicated interactive QOS to support medium-length interactive work. This QOS is intended to deliver nodes for interactive use within 6 minutes of the job request.
Warning
On Perlmutter, if you have not set a default account, salloc may fail with the following error message:
salloc: error: Job request does not match any supported policy.
salloc: error: Job submit/allocate failed: Unspecified error
Avoid frequent short-lived interactive sessions
The interactive QOS is meant to be used interactively. If you need to submit many short jobs, use a single active salloc session rather than repeated independent salloc commands.
Perlmutter GPU nodes¶
The constraint flag (-C or --constraint) must be set to gpu for any GPU-enabled jobs (-C gpu or --constraint gpu).
When using srun in an salloc session, you must also explicitly request GPU resources. Use the --gpus (-G), --gpus-per-node, or --gpus-per-task flag to make the allocated GPUs visible to your srun command. Otherwise, you will see errors / complaints similar to the following:
no CUDA-capable device is detected
No Cuda device found
Shared node allocation (1 or 2 GPUs)¶
If you only need 1 or 2 GPUs, use the shared_interactive QOS:
salloc --constraint gpu --gpus 1 --qos shared_interactive --time 01:00:00 --account mxxx
Whole node allocation¶
To get an interactive session with access to a whole GPU node (4 GPUs), run a command like the following:
salloc --constraint gpu --gpus 4 --nodes 1 --qos interactive --time 01:00:00 --account mxxxx
Perlmutter CPU nodes¶
The constraint flag (-C or --constraint) must be set to cpu for any jobs on CPU-only nodes (-C cpu or --constraint cpu).
Shared node allocation (up to half of one node)¶
To request an interactive session on up to half of the resources (CPUs and memory) on a CPU node, use the shared_interactive QOS in combination with a resource request of up to half of the node's resources. For example, to get an interactive session on 1/4 of the node, use
salloc --constraint cpu --cpus-per-task 64 --qos shared_interactive --time 01:00:00 --account mxxxx
which requests 64 of the node's 256 logical cores.
Whole node allocation¶
You can initiate an interactive session with access to an entire CPU-only node by issuing a command like:
salloc --constraint cpu --nodes 1 --qos interactive --time 01:00:00 --account mxxxx
Limits¶
If resources are not readily available for the requested interactive job, it is automatically canceled after 6 minutes. To allow a job to wait for resources for a longer time, one can use the optional --immediate flag to specify the number of seconds that the job should wait for available resources:
# wait for up to 10 minutes
salloc --nodes 1 --qos interactive --time 01:00:00 --constraint <node type> --account mxxxx --immediate=600
There is a maximum node limit of 4 nodes for interactive jobs on both cpu and gpu partitions. For more details see QOS Limits and Charges.