The Slurm system is undergoing several upgrades that have been planned over the last couple of weeks, and will streamline its current functionalities and add new features.

The changes will be applied on Tuesday, June 10th, at 2pm. You should not experience any disruptions to your running jobs, but the Slurm commands and monitoring may momentarily be unreachable. New jobs submitted after the service time will need to follow the new partition naming scheme, and will benefit from the new features. Some of the GPU nodes will need to be drained in the next couple of days to complete the maintenance, and will be brought back as soon as possible.

The changes are as follows. The User’s Guide page on Slurm has been updated to reflect them.

1. Simplified partitions

There will now only be two partitions: submit and submit-gpu

The submit partition is intended for CPU usage, and unifies the previous submit and submit-1gbs partitions.

All machines with GPUs, which were previously distributed across submit-gpu, submit-gpu-a30, and other machines, are now collected under the submit-gpu partition only.

2. Features/constraints

We have added support for selecting for features (or constraints) when submitting jobs. This is the new mechanism for selecting GPU type, and connection speed.

For the gpus, the available features are: nvidia_a30, nvidia_gtx1080, Tesla_v100.
For the connection speed, the available features are: 100gpb, 1gbs.
You can check which nodes have what features with: scontrol show nodes
You can select features via the –constraint option. e.g. sbatch job.sh –constraint=100gbs.

3. SSH to compute node

We have added a feature that allows you to SSH directly to a compute node where one of your jobs is running. Your connection will be under the same resource limitations as the job itself, and will allow you to interactively debug or monitor your job. You can find which machine your job is running on with the “squeue” command, and will you then be able to just “ssh submitXX.mit.edu” as long as your job is running. If it’s not, the SSH service will tell you that you cannot connect there.

4. Miscellanea

We have optimized the distribution of resources across the partitions, added new compute nodes, and re-enabled some old ones. We are working on refurbishing other GPU nodes, which are always in high demand.

Please reach out to us if you have any questions about the changes, or any feature requests.