{"id":885,"date":"2025-06-10T02:25:28","date_gmt":"2025-06-10T02:25:28","guid":{"rendered":"https:\/\/submit.mit.edu\/?p=885"},"modified":"2025-06-10T02:25:30","modified_gmt":"2025-06-10T02:25:30","slug":"notice-slurm-upgrades","status":"publish","type":"post","link":"https:\/\/submit.mit.edu\/?p=885","title":{"rendered":"Notice: Slurm upgrades"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">The Slurm system is undergoing several upgrades that have been planned over the last couple of weeks, and will streamline its current functionalities and add new features.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>The changes will be applied on Tuesday, June 10th, at 2pm. <\/strong>You should not experience any disruptions to your running jobs, but the Slurm commands and monitoring may momentarily be unreachable. New jobs submitted after the service time will need to follow the new partition naming scheme, and will benefit from the new features.<strong> Some of the GPU nodes will need to be drained in the next couple of days to complete the maintenance, and will be brought back as soon as possible<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The changes are as follows. The <a href=\"https:\/\/submit.mit.edu\/submit-users-guide\/running.html\">User&#8217;s Guide page on Slurm<\/a> has been updated to reflect them.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>1. Simplified partitions<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>There will now only be two partitions: submit and submit-gpu<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The submit partition is intended for CPU usage, and unifies the previous submit and submit-1gbs partitions.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">All machines with GPUs, which were previously distributed across submit-gpu, submit-gpu-a30, and other machines, are now collected under the submit-gpu partition only.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>2. Features\/constraints<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We have added\u00a0support for selecting for features (or constraints) when submitting jobs.\u00a0<strong>This is the new mechanism for selecting GPU type, and connection speed.<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For the gpus, the available features are: nvidia_a30, nvidia_gtx1080, Tesla_v100.<br>For the connection speed, the available features are: 100gpb, 1gbs.<br>You can check which nodes have what features with: scontrol show nodes<br>You can select features via the &#8211;constraint option. e.g. sbatch job.sh &#8211;constraint=100gbs.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>3. SSH&nbsp;to compute node<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We have added a feature that\u00a0allows you to SSH directly to a compute node where one of your jobs is running. Your connection will be under the same resource limitations as the job itself, and will allow you to interactively debug or monitor your job.\u00a0You can find which machine your job is running on with the &#8220;squeue&#8221; command, and will you then be able to just &#8220;ssh submitXX.mit.edu&#8221; as long as your job is running. If it&#8217;s not, the SSH service will tell you that you cannot connect there.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>4. Miscellanea<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We have optimized the distribution of resources across the partitions, added new compute nodes, and re-enabled some old ones. We are working on refurbishing other GPU nodes, which are always in high demand.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Please reach out to us if you have any questions about the changes, or any feature requests.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>The Slurm system is undergoing several upgrades that have been planned over the last couple of weeks, and will streamline [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[],"class_list":["post-885","post","type-post","status-publish","format-standard","hentry","category-news"],"_links":{"self":[{"href":"https:\/\/submit.mit.edu\/index.php?rest_route=\/wp\/v2\/posts\/885","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/submit.mit.edu\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/submit.mit.edu\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/submit.mit.edu\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/submit.mit.edu\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=885"}],"version-history":[{"count":3,"href":"https:\/\/submit.mit.edu\/index.php?rest_route=\/wp\/v2\/posts\/885\/revisions"}],"predecessor-version":[{"id":888,"href":"https:\/\/submit.mit.edu\/index.php?rest_route=\/wp\/v2\/posts\/885\/revisions\/888"}],"wp:attachment":[{"href":"https:\/\/submit.mit.edu\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=885"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/submit.mit.edu\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=885"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/submit.mit.edu\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=885"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}