SubMIT has grown by about 520 CPU cores! Since October 10, 30 machines of the MIT CMS Tier-3 (T3_US_MIT) are part of the SubMIT Slurm cluster. They are available in a new partition called tier3 to all SubMIT users.

These machines served for many years as the batch farm of the MIT Tier-3, still running CentOS 7, which reached its end of life in 2024. We have now reinstalled them with AlmaLinux 9 and integrated them into SubMIT. For your jobs they look just like the rest of SubMIT: the same operating system, user accounts and Slurm version, and your /home/submit and /work/submit directories, /ceph/submit, /scratch and CVMFS are all available. The partition is open to all users, with the same time limits as submit (up to 6 days). To use it, add #SBATCH --partition=tier3 to your batch script. Or let Slurm pick whichever partition has free CPUs first with #SBATCH --partition=submit,tier3.

There is a catch, though: these machines are considerably older than the rest of SubMIT. A few things to keep in mind:

• Older CPUs. Most nodes have Intel Xeon CPUs from 2010–2013, so each core is slower than on the submit nodes. Plan for longer run times.
• No AVX2, and mostly no AVX. Programs built for newer CPUs stop with Illegal instruction. Examples are code compiled with -march=native on a login node, or packages like TensorFlow. Rebuild with -march=x86-64-v2, or keep these jobs on submit.
• 1 Gbit/s network. Most submit nodes have 100 Gbit/s links. Jobs that read or write a lot of data are better kept on submit.
• Less memory. Most nodes have 16 cores and about 21 GB of RAM, so please request memory explicitly with --mem-per-cpu.

This makes the tier3 partition a great place for many independent, CPU-heavy jobs that do not move a lot of data, such as simulations, fits or toy studies. It also takes load off the submit partition, which helps everybody.

You can follow how the new machines are doing on our Ganglia page (http://submit08.mit.edu/ganglia/) under “Tier 3”. All details are in the Users Guide: https://submit.mit.edu/submit-users-guide/running.html#the-tier3-partition.

As always, let us know about your experience with the new partition, or contact us at submit-help@mit.edu if something does not work as expected!