Slurm installation and upgrading
Contents
Jump to our top-level Slurm page: Slurm batch queueing system
To get started with Slurm see the Slurm_Quick_Start Administrator Guide. See also CECI Slurm Quick Start Tutorial.
Hardware optimization for the slurmctld master server
SchedMD recommends that the slurmctld server should have only a few, but very fast CPU cores, in order to ensure the best responsiveness.
The file system for /var/spool/slurmctld/
should be mounted on the fastest possible disks (SSD or NVMe if possible).
Create global user accounts
There must be a uniform user and group name space (including UIDs and GIDs) across the cluster,
see the Slurm_Quick_Start Administrator Guide.
It is not necessary to permit user logins to the control hosts (ControlMachine or BackupController), but the users and groups must be configured on those hosts.
To restrict user login by SSH, use the AllowUsers
parameter in /etc/ssh/sshd_config
.
Slurm and MUNGE require consistent UID and GID across all servers and nodes in the cluster, including the slurm and munge users.
It is very important to avoid UID and GID below 1000, as defined in the standard configuration file /etc/login.defs
by the parameters UID_MIN, UID_MAX, GID_MIN, GID_MAX
,
see also https://en.wikipedia.org/wiki/User_identifier.
Create the users/groups for slurm and munge, for example:
export MUNGEUSER=1005
groupadd -g $MUNGEUSER munge
useradd -m -c "MUNGE Uid 'N' Gid Emporium" -d /var/lib/munge -u $MUNGEUSER -g munge -s /sbin/nologin munge
export SlurmUSER=1001
groupadd -g $SlurmUSER slurm
useradd -m -c "Slurm workload manager" -d /var/lib/slurm -u $SlurmUSER -g slurm -s /bin/bash slurm
and make sure that these same users are created identically on all nodes. This must be done prior to installing RPMs (which would create random UID/GID pairs if these users don’t exist).
Please note that UIDs and GIDs up to 1000 are currently reserved for the CentOS system users, see this article and the file /etc/login.defs.
MUNGE authentication service
The MUNGE authentication plugins identifies the user originating a message. You should read the Munge_installation guide and the Munge_wiki.
The MUNGE RPM for RHEL7 is in the EPEL repository, where you install the newest version of epel-release RPM for EL7, for example:
CentOS8: dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
CentOS7: yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
RHEL7: yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
Warning: EPEL contains Slurm 20.11 (January 2021) and most likely you do not want automatic Slurm updates from EPEL.
If so, you must add manually to the file /etc/yum.repos.d/epel.repo
the exclude
line:
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
#baseurl=https://download.fedoraproject.org/pub/epel/7/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch&infra=$infra&content=$contentdir
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
exclude=slurm*
Install the MUNGE RPM packages from the EPEL repository:
yum install munge munge-libs munge-devel
To download packages directly (using statically in compute nodes):
RHEL/CentOS 7: https://dl.fedoraproject.org/pub/epel/7/x86_64/m/
MUNGE configuration and testing
By default MUNGE uses an AES AES-128 cipher and SHA-256 HMAC (Hash-based Message Authentication Code). Display these encryption options by:
munge -C
munge -M
On the Head/Master node (only) create a secret key to be used globally on every node (see the Munge_installation guide):
dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key
chown munge: /etc/munge/munge.key
chmod 400 /etc/munge/munge.key
Alternatively use this command (slow):
/usr/sbin/create-munge-key -r
NOTE: For a discussion of using /dev/random
in stead of /dev/urandom
(pseudo-random) as recommended in the Munge_installation guide,
see Myths about /dev/urandom.
Securely propagate /etc/munge/munge.key
(e.g., via SSH) to all other hosts within the same security realm:
scp -p /etc/munge/munge.key hostXXX:/etc/munge/munge.key
Make sure to set the correct ownership and mode on all nodes:
chown -R munge: /etc/munge/ /var/log/munge/
chmod 0700 /etc/munge/ /var/log/munge/
Then enable and start the MUNGE service on all nodes:
systemctl enable munge
systemctl start munge
Run some tests as described in the Munge_installation guide:
munge -n
munge -n | unmunge # Displays information about the MUNGE key
munge -n | ssh somehost unmunge
remunge
Increase munged number of threads
On busy servers such as the slurmctld server, the munged daemon could become a bottleneck, see the presentation Field Notes 5: From The Frontlines of Slurm Support in the Slurm_publications page. On such servers it is recommended to increase the number of munged threads, see man munged.
On RHEL/CentOS systems the procedure is:
The current Munge 0.5.11 does not honor an options file in
/etc/sysconfig/
, see PR 68.Edit the Systemd unit file:
export EDITOR=vim # Configure your favorite editor systemctl edit munge --full
Alternatively, you can copy the Systemd unit file:
cp /usr/lib/systemd/system/munge.service /etc/systemd/system/munge.service
Edit this line in the unit file:
ExecStart=/usr/sbin/munged --num-threads 10
Restart the munged service:
systemctl daemon-reload systemctl restart munge
Build Slurm RPMs
See the Slurm_Quick_Start Administrator Guide, especially the section below this text:
Optional Slurm plugins will be built automatically when the configure script detects that the required build requirements are present.
Build dependencies for various plugins and commands are denoted below:
You must decide which Slurm plugins to activate in the RPM packages which you build, especially items such as:
MySQL for accounting support
cgroup Task Affinity
MUNGE support
Lua Support
PAM support
NUMA Affinity
Install prerequisites
Install Slurm prerequisites as well as several optional packages that enable Slurm plugins as described in the Slurm_Quick_Start guide:
yum install rpm-build gcc python3 openssl openssl-devel pam-devel numactl numactl-devel hwloc hwloc-devel munge munge-libs munge-devel lua lua-devel readline-devel rrdtool-devel ncurses-devel gtk2-devel libibmad libibumad perl-Switch perl-ExtUtils-MakeMaker xorg-x11-xauth http-parser-devel json-c-devel
Enable the EPEL repository:
CentOS 8:
dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
CentOS 7:
dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
and install the following packages from EPEL:
yum install libssh2-devel man2html
If you use the recommended:
AuthType=auth/munge
in slurm.conf or slurmdbd.conf, then you must also install:
yum install munge munge-libs munge-devel
If you want to build the Slurm REST API daemon named slurmrestd (from Slurm 20.02 and newer),
or if you want to use the slurm.conf ResumeProgram
and SuspendProgram
from the Power_Saving_Guide,
then you make sure to install these prerequisites before building RPMs:
yum install http-parser-devel json-c-devel
Install MariaDB database
Important: If you want to enable accounting, you must install the MariaDB (a replacement for MySQL) version 5.5 from CentOS7/RHEL7 packages before you build Slurm RPMs:
yum install mariadb-server mariadb-devel
CentOS8/RHEL8 has the newer MariaDB version 10.3 which is installed by:
dnf install mariadb-server mariadb-devel
NOTICE: Do not forget to configure the database as described in the Slurm database page!
Needs testing: Alternatively, you can install the MariaDB version 10.3 database from the CentOS 7 Software Collections (SCL) Repository:
yum install centos-release-scl
yum install rh-mariadb103-mariadb-server rh-mariadb103-mariadb-devel rh-mariadb103-mariadb rh-mariadb103-mariadb-backup
Install the latest MariaDB version
Optional: Install the latest MariaDB version. This is not required, and installation is somewhat involved.
For best results with RPM and DEB packages, use the Repository Configuration Tool. Configure the Yum repository as instructed and read the MariaDB Yum page.
For building Slurm you need to install these MariaDB 10.4 (or later) packages:
yum install MariaDB-client MariaDB-shared MariaDB-devel
The MariaDB-shared package contains the required shared libraries for Slurm. The slurmdbd server host will need to install also:
yum install MariaDB-server MariaDB-backup
Install MySQL Python tools
If you will use Ansible to manage the database, Ansible needs this Python package:
yum install MySQL-python # CentOS7/RHEL7
dnf install python3-mysql # CentOS8/RHEL8
Build Slurm packages
Get the Slurm source code from the Slurm_download page. At this point you must decide whether to build in Slurm plugins, for example, mysql for accounting (see above).
Set the version (for example, 22.05.8) and build Slurm RPM packages by:
export VER=22.05.8
rpmbuild -ta slurm-$VER.tar.bz2 --with mysql # Includes accounting support with the slurm-slurmdbd package
rpmbuild -ta slurm-$VER.tar.bz2 --without mysql # No slurm-slurmdbd accounting support
The --with mysql
option is not strictly necessary because the slurm-slurmdbd
package will be built by default,
but using this option will catch the scenario where your forgot to install the mariadb-devel
packages as described above, see also bug_8882
and this mailing list posting.
If you want to build the Slurm REST API daemon named slurmrestd (from Slurm 20.02 and newer):
rpmbuild -ta slurm-$VER.tar.bz2 --with mysql --with slurmrestd
Note: On RHEL 9 (and derivatives) you must (currently) disable LTO in the SPEC file, see bug_14565.
The RPM packages will typically be in $HOME/rpmbuild/RPMS/x86_64/
and should be installed on all relevant nodes.
Installing RPMs
The RPMs to be installed on the head node, compute nodes, and slurmdbd node can vary by configuration, but here is a suggested starting point:
Head/Master Node (where the slurmctld daemon runs), Compute, and Login nodes:
export VER=22.05.8-1 yum install slurm-$VER*rpm slurm-devel-$VER*rpm slurm-perlapi-$VER*rpm slurm-torque-$VER*rpm slurm-example-configs-$VER*rpm
On the master node explicitly enable the slurmctld service:
systemctl enable slurmctld
The slurm-torque package could perhaps be omitted, but it does contain a useful
/usr/bin/mpiexec
wrapper script.Only if the database service will run on the Head/Master node: Install the database service RPM:
export VER=22.05.8-1 yum install slurm-slurmdbd-$VER*rpm
Explicitly enable the service:
systemctl enable slurmdbd
On Compute nodes you may additionally install the slurm-pam_slurm RPM package to prevent rogue users from logging in:
yum install slurm-pam_slurm-$VER*rpm
You may consider this RPM as well with special PMIx libraries:
yum install slurm-libpmi-$VER*rpm
Explicitly enable the service:
systemctl enable slurmd
Database-only (slurmdbd service) node:
export VER=22.05.8-1 yum install slurm-$VER*rpm slurm-devel-$VER*rpm slurm-slurmdbd-$VER*rpm
Explicitly enable the service:
systemctl enable slurmdbd
Servers (from Slurm 20.02 and newer) which should offer slurmrestd (which can be used also by normal users) should install also this package:
yum install slurm-slurmrestd-$VER*rpm
The slurmctld server and the login nodes would typically include slurmrestd.
Study the configuration information in the Quick Start Administrator_Guide.
Update Systemd service files
On CentOS/RHEL 8 (EL8) systems the Slurm daemons may fail starting up at reboot, when Slurm is running in configless mode, apparently due to DNS failures. This is actually due to the daemons starting too soon, before the network is fully online. The issue is tracked in bug_11878.
The solution (which may be solved in 21.08) is to modify the Systemd service files for slurmd, slurmctld and slurmdbd, for example:
cp /usr/lib/systemd/system/slurmd.service /etc/systemd/system/
and edit the line in the service file:
After=munge.service network.target remote-fs.target
into:
After=munge.service network-online.target remote-fs.target
The network-online target will ensure that the network is online before starting the daemons. Reboot the system to verify the daemon startup.
This modification may be beneficial on all Systemd systems, including EL8 and EL7.
Configure Slurm logging
The Slurm logfile directory is undefined in the RPMs since you have to define it in slurm.conf. See SlurmdLogFile and SlurmctldLogFile in the slurm.conf page, and LogFile in the slurmdbd.conf page.
Check your logging configuration with:
# grep -i logfile /etc/slurm/slurm.conf SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdLogFile=/var/log/slurm/slurmd.log # scontrol show config | grep -i logfile SlurmctldLogFile = /var/log/slurm/slurmctld.log SlurmdLogFile = /var/log/slurm/slurmd.log SlurmSchedLogFile = /var/log/slurm/slurmsched.logIf log files are configured, you have to create the log file directory manually:
mkdir /var/log/slurm chown slurm.slurm /var/log/slurmSee the more general description in Bug_8272.
Upgrading Slurm
New Slurm updates are released rather often. Follow the Upgrades instructions in the Slurm_Quick_Start page, see also presentations by Tim Wickberg in the Slurm_publications page. Pay attention to these statements:
You may upgrade at most by 2 major versions, see the Upgrades page:
Slurm daemons will support RPCs and state files from the two previous major releases (e.g. a version 16.05.x SlurmDBD will support slurmctld daemons and commands with a version of 16.05.x, 15.08.x or 14.11.x).
In other words, when changing the version to a higher release number (e.g from 14.11.x to 15.08.x) always upgrade the slurmdbd daemon first.
Be mindful of your configured SlurmdTimeout and SlurmctldTimeout values.
The recommended upgrade order is that versions may be mixed as follows:
slurmdbd >= slurmctld >= slurmd >= commands
If you use a database, also make sure to:
Make a database dump (see Slurm database) prior to the slurmdbd upgrade.
Start the slurmdbd service manually after the upgrade in order to avoid timeouts (see bug_4450). In stead of starting the slurmdbd service, it is most likely necessary to start the daemon manually. If you use the
systemctl
command, it is very likely to exceed a system time limit and kill slurmdbd before the database conversion has been completed.The recommended way to perform the slurmdbd database upgrade is therefore:
time slurmdbd -D -vvv
See further info below.
This command can report current jobs that have been orphaned on the local cluster and are now runaway:
sacctmgr show runawayjobs
Database upgrade from Slurm 17.02 and older
If you are upgrading from Slurm 17.02 and older to Slurm 17.11 and newer, you must be extremely cautious about long database update times, since in Slurm 17.11 (and newer) some database structures were changed. Read the mailing list thread Extreme long db upgrade 16.05.6 -> 17.11.3, where Lech Nieroda states:
To sum it up, the issue affects those users who still have 17.02 or prior versions, use their distribution defaults for mysql/mariadb from RHEL6/CentOS6 and RHEL7/CentOS7, have millions of jobs in their database and would like to upgrade slurm without upgrading mysql.
The patch is also available from and is discussed in bug_6796.
Furthermore, the 17.11 Release Notes states:
NOTE FOR THOSE UPGRADING SLURMDBD: The database conversion process from SlurmDBD 16.05 or 17.02 may not work properly with MySQL 5.1 (as was the default version for RHEL 6).
Upgrading to a newer version of MariaDB or MySQL is strongly encouraged to prevent this problem.
and the 18.08 Release Notes added:
NOTE FOR THOSE UPGRADING SLURMDBD:
The database conversion process from SlurmDBD 16.05 or 17.02 may not work properly with MySQL 5.1 or 5.5 (as was the default version for RHEL 6).
Upgrading to a newer version of MariaDB or MySQL is strongly encouraged to prevent this problem.
NOTE: MariaDB version 5.5 is the default database version delivered with RHEL7/CentOS 7!
More recent MariaDB versions 10.x can be downloaded from the MariaDB_repository. Some further information:
This MariaDB blog explains the upgrade process from 5.5 to 10.x.
The patch in the above thread should be applied manually to Slurm 17.11 before upgrading the database from 17.02 or 16.05 to 17.11 (do not upgrade by more than 2 Slurm releases!).
Upgrade of MySQL/MariaDB
If you restore a database dump (see Slurm database) onto a different server running a newer MySQL/MariaDB version, for example upgrading MySQL 5.1 on CentOS 6 to MariaDB 5.5 on CentOS 7, there are some extra steps.
See Upgrading from MySQL to MariaDB about running the mysql_upgrade command:
mysql_upgrade
whenever major (or even minor) version upgrades are made, or when migrating from MySQL to MariaDB.
It may be necessary to restart the mysqld service or reboot the server after this upgrade (??).
Slurm database modifications required for MariaDB 10.2.1 and above
In MariaDB 10.2.1 and above there are some important changes to Slurm database tables, please read instructions in the page Slurm database modifications required for MariaDB 10.2.1 and above (with a reference to bug_15168). This has been resolved in Slurm 22.05.7.
Make a dry run database upgrade
Optional but strongly recommended: You can test the database upgrade procedure before doing the real upgrade.
In order to verify and time the slurmdbd database upgrade you may make a dry_run upgrade for testing before actual deployment.
Here is a suggested procedure:
Drain a compute node running the current Slurm version and use it for testing the database.
Install the database RPM packages and configure the database EXACTLY as described in the Slurm database page:
yum install mariadb-server mariadb-devel
Configure the MySQL/MariaDB database as described in the Slurm database page.
Copy the latest database dump file (
/root/mysql_dump
, see Slurm database) from the main server to the compute node. Load the dump file into the testing database:time mysql -u root -p < /root/mysql_dump
If the dump file is in some compressed format:
time zcat mysql_dump.gz | mysql -u root -p time bzcat mysql_dump.bz2 | mysql -u root -p
The MariaDB/MySQL password will be asked for. Reading in the database dump may take many minutes depending on the size of the dump file, the storage system speed, and the CPU performance. The
time
command will report the time usage.Verify the database contents on the compute node by making a new database dump and compare it to the original dump.
The following actions must be performed on the drained compute node.
First stop the regular slurmd daemons on the compute node:
systemctl stop slurmd
Install the OLD (the cluster’s current version, say, NN.NN) additional slurmdbd database RPMs as described above:
VER=NN.NN yum install slurm-slurmdbd-$VER*rpm
Information about building RPMs is in the Slurm installation and upgrading page.
Make sure that the
/etc/slurm
directory exists (it is not needed in configless Slurm clusters):$ ls -lad /etc/slurm drwxr-xr-x. 5 root root 4096 Feb 22 10:12 /etc/slurm
Copy the configuration files from the main server to the compute node:
/etc/slurm/slurm.conf /etc/slurm/slurmdbd.conf
Important: Edit these files to replace the database server name by
localhost
so that all further actions take place on the compute node, not the real database server.Configure this in
slurmdbd.conf
:DbdHost=localhost StorageHost=localhost
Configure this in
slurm.conf
:AccountingStorageHost=localhost
Set up files and permissions:
chown slurm: /etc/slurm/slurmdbd.conf chmod 600 /etc/slurm/slurmdbd.conf touch /var/log/slurm/slurmdbd.log chown slurm: /var/log/slurm/slurmdbd.log
Make sure that slurmdbd is running, and start it if necessary:
systemctl status slurmdbd systemctl start slurmdbd
Make some query to test slurmdbd:
sacctmgr show user -s
If all is well, stop the slurmdbd before the upgrade below:
systemctl stop slurmdbd
At this point you have a Slurm database server running an exact copy of your main Slurm database!
Now it is time to do some testing. Update all Slurm RPMs to the new version (say, 22.05.8-1) built as shown above:
export VER=22.05.8-1 yum update slurm*$VER*.rpm
If you use the auto_tmpdir RPM package, you have to remove it first because it will block the upgrade:
yum remove auto_tmpdir
See also Temporary job directories
Perform and time the actual database upgrade:
time slurmdbd -D -vvv
and wait for the output:
slurmdbd: debug2: accounting_storage/as_mysql: as_mysql_roll_usage: Everything rolled up
and do a Control-C. Please note that the database table conversions may take several minutes or longer, depending on the size of the tables.
Write down the timing information from the
time
command, since this will be the expected approximate time when you later perform the real upgrade. However, the storage system performance is important for all database operations, so timings may vary substantially between servers.Now start the service as usual:
systemctl start slurmdbd
Make some query to test slurmdbd:
sacctmgr show user -s
and make some other tests to verify that slurmdbd is responding correctly.
When all tests have been completed successfully, reinstall the compute node to its default installation.
Upgrading on CentOS 7
Let’s assume that you have built the updated RPM packages for CentOS 7 and copied them to the current directory so you can use yum
on the files directly.
Prerequisites before upgrading
If you have installed the pdsh tool, there may be a module that has been linked against a specific library version libslurm.so.30
,
and yum
will then refuse to update the slurm* RPMs.
You must first do:
yum remove pdsh-mod-slurm
and then later rebuild and install pdsh-mod-slurm, see the Slurm batch queueing system page.
Upgrade slurmdbd
The upgrading steps for the slurmdbd host are:
Stop the slurmdbd service:
systemctl stop slurmdbd
Make a mysqldump of the MySQL/Mariadb database (see above).
If also upgrading to MariaDB 10.2.1 (and above) from an older version, there are some important changes to Slurm database tables, please read instructions in the page Slurm database modifications required for MariaDB 10.2.1 and above (with a reference to bug_15168). This has been resolved in Slurm 22.05.7.
Update all RPMs:
export VER=22.05.8-1 yum update slurm*$VER*.rpm
Start the slurmdbd service manually after the upgrade in order to avoid timeouts (see bug_4450). In stead of starting the slurmdbd service, it is most likely necessary to start the daemon manually. If you use the
systemctl
command, it is very likely to exceed a system time limit and kill slurmdbd before the database conversion has been completed. Perform and time the actual database upgrade:time slurmdbd -D -vvv
The completion of the database conversion may be printed as:
slurmdbd: debug2: accounting_storage/as_mysql: as_mysql_roll_usage: Everything rolled up
Then do a Control-C. Please note that the database table conversions may take several minutes or longer, depending on the size of the tables.
Restart the slurmdbd service normally:
systemctl start slurmdbd
Make some query to test slurmdbd:
sacctmgr show user -s
Upgrade slurmctld
The upgrading steps for the slurmctld host are:
Change the timeout values in slurm.conf to:
SlurmctldTimeout=3600 SlurmdTimeout=3600
and copy
/etc/slurm/slurm.conf
to all nodes (not needed in configless Slurm clusters). Then reconfigure the running daemons and test the timeout andStateSaveLocation
values:scontrol reconfigure scontrol show config | grep Timeout scontrol show config | grep StateSaveLocation
Stop the slurmctld service:
systemctl stop slurmctld
Make a backup copy of the
StateSaveLocation
(check your configuration first)/var/spool/slurmctld
directory:tar czf $HOME/var.spool.slurmctld.tar.gz /var/spool/slurmctld/*
Make sure the contents of the tar-ball file looks correct!
Upgrade the RPMs, for example:
export VER=22.05.8-1 yum update slurm*$VER-*.rpm
Enable and restart the slurmctld service:
systemctl enable slurmctld systemctl restart slurmctld
Check the cluster nodes’ health using
sinfo
and check for anyNodes ... not responding
errors inslurmctld.log
. It may be necessary to restart all theslurmd
on all nodes:clush -ba systemctl restart slurmd
Restore the previous timeout values in slurm.conf (item 1.).
Note: The compute nodes should be upgraded at your earliest convenience.
Install slurm-libpmi
On the compute nodes, only, you may consider this RPM as well with special PMIx libraries:
yum install slurm-libpmi-$VER*rpm
Upgrade MPI applications
MPI applications such as OpenMPI may be linked against the /usr/lib64/libslurm.so
library.
In this context you must understand the remark in the Upgrades page:
The libslurm.so version is increased every major release.
So things like MPI libraries with Slurm integration should be recompiled.
Sometimes it works to just symlink the old .so name(s) to the new one, but this has no guarantee of working.
In the thread Need for recompiling openmpi built with –with-pmi? it has been found that:
It looks like it is the presence of lib64/libpmi2.la and lib64/libpmi.la that is the "culprit". They are installed by the slurm-devel RPM.
Openmpi uses GNU libtool for linking, which finds these files, and follow their "dependency_libs" specification, thus linking directly to libslurm.so.
Slurm version 16.05 and later no longer installs the libpmi*.la files. This should mean that if your OpenMPI was built against Slurm 16.05 or later, there should be no problem (we think), but otherwise you probably must rebuild your MPI applications and install them again at the same time that you upgrade the slurmd on the compute nodes.
To check for the presence of the “bad” files, go to your software build host and search:
locate libpmi2.la
locate libpmi.la
TODO: Find a way to read relevant MPI libraries like this example:
readelf -d libmca_common_pmi.so
Upgrade slurmd on nodes
First determine which Slurm version the nodes are running:
clush -bg <partition> slurmd -V # Using ClusterShell
pdsh -g <partition> slurmd -V | dshbak # Using PDSH
See the Slurm batch queueing system page about ClusterShell or PDSH.
The quick and usually OK procedure would be to simply update the RPMs (here: version 22.05.8-1) on all nodes:
clush -bw <nodelist> 'yum -y update /some/path/slurm*22.05.8-1-*.rpm'
This would automatically restart and enable slurmd on the nodes without any loss of running batch jobs.
For the compute nodes running slurmd the safe procedure could be:
Drain all desired compute nodes in a <nodelist>:
scontrol update NodeName=<nodelist> State=draining Reason="Upgrading slurmd"
Nodes will change from the DRAINING to the DRAINED state as the jobs are completed. Check which nodes have become DRAINED:
sinfo -t drained
Stop the slurmd daemons on compute nodes:
clush -bw <nodelist> systemctl stop slurmd
Update the RPMs (here: version 22.05.8-1) on nodes:
clush -bw <nodelist> 'yum -y update /some/path/slurm*22.05.8-1-*.rpm'
and make sure to install also the new
slurm-slurmd
andslurm-contribs
packages.Now enable the slurmd service:
clush -bw <nodelist> systemctl enable slurmd
For restarting slurmd there are two alternatives:
Restart slurmd or simply reboot the nodes in the DRAINED state:
clush -bw <nodelist> systemctl daemon-reload clush -bw <nodelist> systemctl restart slurmd or simply reboot: clush -bw <nodelist> shutdown -r now
Reboot the nodes automatically as they become idle using the RebootProgram as configured in slurm.conf, see the scontrol reboot option and explanation in the man-page:
scontrol reboot [ASAP] [NodeList]
Return upgraded nodes to the IDLE state:
scontrol update NodeName=<nodelist> State=resume
Finally, restore the timeout values in slurm.conf to their defaults, for example:
SlurmctldTimeout=600
SlurmdTimeout=300
and copy /etc/slurm/slurm.conf
to all nodes. Then reconfigure the running daemons:
scontrol reconfigure
Again, consult the Upgrades page before you start!
Log file rotation
The Slurm log files may be stored in /var/log/slurm
, and they may grow rapidly on a busy system.
Especially the slurmctld.log
file on the controller machine may grow very big.
Therefore you probably want to configure logrotate to administer your log files.
On RHEL and CentOS the logrotate configuration files are in the /etc/logrotate.d/
directory.
Manual configuration is required because the SchedMD RPM files do not contain the logrotate setup, see bug_3904 and bug_2215 and bug_4393. See also the section LOGGING at the end of the slurm.conf page with an example logrotate script.
First install the relevant RPM:
yum install logrotate
Create the following script /etc/logrotate.d/slurm
which will rotate and compress the slurmctld log file on a weekly basis:
/var/log/slurm/*.log {
compress
missingok
nocopytruncate
nodelaycompress
nomail
notifempty
noolddir
rotate 5
sharedscripts
size=5M
create 640 slurm root
postrotate
pkill -x --signal SIGUSR2 slurmctld
pkill -x --signal SIGUSR2 slurmd
pkill -x --signal SIGUSR2 slurmdbd
exit 0
endscript
}
Warning: Do not run scontrol reconfig
or restart slurmctld to rotate the log files, since this will incur a huge overhead.
See the NEWS file for changes related to SIGUSR2:
Modify all daemons to re-open log files on receipt of SIGUSR2 signal. This is much than using SIGHUP to re-read the configuration file and rebuild various tables.