{"id":1161,"date":"2026-10-20T13:46:54","date_gmt":"2026-10-20T13:46:54","guid":{"rendered":"https:\/\/submit.mit.edu\/?p=1161"},"modified":"2026-10-20T13:46:54","modified_gmt":"2026-10-20T13:46:54","slug":"over-500-more-cpus-on-submit-from-refurbished-hardware","status":"publish","type":"post","link":"https:\/\/submit.mit.edu\/?p=1161","title":{"rendered":"Over 500 more CPUs on SubMIT from refurbished hardware"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">SubMIT has grown by about 520 CPU cores! Since October 10, 30 machines of the MIT CMS Tier-3 (T3_US_MIT) are part of the SubMIT Slurm cluster. They are available in a new partition called <code>tier3<\/code> to all SubMIT users.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">These machines served for many years as the batch farm of the MIT Tier-3, still running CentOS 7, which reached its end of life in 2024. We have now reinstalled them with AlmaLinux 9 and integrated them into SubMIT. For your jobs they look just like the rest of SubMIT: the same operating system, user accounts and Slurm version, and your <code>\/home\/submit<\/code> and <code>\/work\/submit<\/code> directories, <code>\/ceph\/submit<\/code>, <code>\/scratch<\/code> and CVMFS are all available. The partition is open to all users, with the same time limits as <code>submit<\/code> (up to 6 days). To use it, add <code>#SBATCH --partition=tier3<\/code> to your batch script. Or let Slurm pick whichever partition has free CPUs first with <code>#SBATCH --partition=submit,tier3<\/code>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">There is a catch, though: these machines are considerably older than the rest of SubMIT. A few things to keep in mind:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Older CPUs.<\/strong> Most nodes have Intel Xeon CPUs from 2010\u20132013, so each core is slower than on the <code>submit<\/code> nodes. Plan for longer run times.<\/li>\n\n\n\n<li><strong>No AVX2, and mostly no AVX.<\/strong> Programs built for newer CPUs stop with <code>Illegal instruction<\/code>. Examples are code compiled with <code>-march=native<\/code> on a login node, or packages like TensorFlow. Rebuild with <code>-march=x86-64-v2<\/code>, or keep these jobs on <code>submit<\/code>.<\/li>\n\n\n\n<li><strong>1 Gbit\/s network.<\/strong> Most <code>submit<\/code> nodes have 100 Gbit\/s links. Jobs that read or write a lot of data are better kept on <code>submit<\/code>.<\/li>\n\n\n\n<li><strong>Less memory.<\/strong> Most nodes have 16 cores and about 21 GB of RAM, so please request memory explicitly with <code>--mem-per-cpu<\/code>.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">This makes the <code>tier3<\/code> partition a great place for many independent, CPU-heavy jobs that do not move a lot of data, such as simulations, fits or toy studies. It also takes load off the <code>submit<\/code> partition, which helps everybody.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can follow how the new machines are doing on our Ganglia page (<a href=\"http:\/\/submit08.mit.edu\/ganglia\/\" target=\"_blank\" rel=\"noreferrer noopener\">http:\/\/submit08.mit.edu\/ganglia\/<\/a>) under \u201cTier 3\u201d. All details are in the Users Guide: <a href=\"https:\/\/submit.mit.edu\/submit-users-guide\/running.html#the-tier3-partition\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/submit.mit.edu\/submit-users-guide\/running.html#the-tier3-partition<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As always, let us know about your experience with the new partition, or contact us at submit-help@mit.edu if something does not work as expected!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>SubMIT has grown by about 520 CPU cores! Since October 10, 30 machines of the MIT CMS Tier-3 (T3_US_MIT) are [&hellip;]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[],"class_list":["post-1161","post","type-post","status-publish","format-standard","hentry","category-news"],"_links":{"self":[{"href":"https:\/\/submit.mit.edu\/index.php?rest_route=\/wp\/v2\/posts\/1161","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/submit.mit.edu\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/submit.mit.edu\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/submit.mit.edu\/index.php?rest_route=\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/submit.mit.edu\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1161"}],"version-history":[{"count":1,"href":"https:\/\/submit.mit.edu\/index.php?rest_route=\/wp\/v2\/posts\/1161\/revisions"}],"predecessor-version":[{"id":1162,"href":"https:\/\/submit.mit.edu\/index.php?rest_route=\/wp\/v2\/posts\/1161\/revisions\/1162"}],"wp:attachment":[{"href":"https:\/\/submit.mit.edu\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1161"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/submit.mit.edu\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1161"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/submit.mit.edu\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1161"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}