Dell R650 server 

This page contains information out Dell PowerEdge R650 could servers which we have deployed in our cluster.

Documentation and software 

Dell Support provides R650 information:

R650_manuals.
R650_downloads.
iDRAC9_manuals.
Dell blog: Intel Ice Lake - BIOS Characterization for HPC.

Booting and BIOS configuration 

Press F2 during start-up to enter the BIOS and firmware setup menus. Go to the BIOS Settings menu.

Minimal configuration of a new server or motherboard 

At our site the following minimal settings are required for a new server or a new motherboard. Remaining settings will be configured by racadm.

The Dell iDRAC9 (BMC) setup is accessed via the System Setup menu item iDRAC Settings:

In the System Summary page read the NIC iDRAC MAC Address from this page for configuring the DHCP server.
In the Network page set the Enable IPMI over LAN to Enabled.

Go to the System Setup menu item Device Settings and select the Integrated NIC items:

In the NIC Main Configuration Page select NIC Configuration. We use NIC port 3 (1 Gbit) as the system’s NIC.
Read the NIC Ethernet MAC Address from this page for configuring the DHCP server.
Select the Legacy Boot Protocol item PXE.

Performance settings for HPC 

The following BIOS settings should be made on HPC nodes:

[Key=BIOS.Setup.1-1#SysProfileSettings]
SysProfile=PerfOptimized
[Key=BIOS.Setup.1-1#SysProfileSettings]
WorkloadProfile=HpcProfile
[Key=BIOS.Setup.1-1#DellControlledTurbo]
ControlledTurbo=Disabled
[Key=BIOS.Setup.1-1#IntegratedDevices]
SnoopHldOff=Roll8KCycles

The CLI commands are:

racadm set BIOS.SysProfileSettings.SysProfile PerfOptimized
racadm set BIOS.SysProfileSettings.WorkloadProfile HpcProfile
racadm set BIOS.DellControlledTurbo.ControlledTurbo Disabled
racadm set BIOS.IntegratedDevices.SnoopHldOff Roll8KCycles
racadm jobqueue create BIOS.Setup.1-1

and then reboot the server.

The SnoopHldOff value (should be 8K or 4K, the default 256 is too small) is important for high performance Omni-Path or Infiniband network fabrics.

System Thermal settings 

System Thermal Profile settings can be changed based on the need to maximize performance or power efficiency. This can make CPU thermal throttling less likely.

Read the document Custom Cooling Fan Options for Dell EMC PowerEdge Servers.

In the BIOS setup screen, select iDRAC->Thermal and configure Thermal profile = Maximum performance.

Read the current settings:

racadm get System.ThermalSettings

For HPC applications set the fans to high performance:

racadm set System.ThermalSettings.ThermalProfile "Maximum Performance"
racadm set System.ThermalSettings.MinimumFanSpeed 25

Processor settings menu 

You may want to verify the following BIOS settings on HPC nodes:

Disable Hyperthreading by Logical Processor = Disabled.
Virtualization Technology = Disabled.
Dell Controlled Turbo = Disabled.
Sub NUMA Cluster = Enabled.

The Sub NUMA Cluster (SNC, replaces the older Cluster-on-Die (COD) implementation) has been shown to improve performance, see BIOS characterization for HPC with Intel Cascade Lake processors. This will cause each processor socket to have two NUMA domains for the two memory controllers, so a dual-socket server will have 4 NUMA domains.

Display the NUMA domains by:

$  numactl --hardware
available: 4 nodes (0-3)
...

PXE boot setup 

Go to the System Setup menu item Device Settings and select the Integrated NIC items:

In the NIC Main Configuration Page select NIC Configuration. We use NIC port 3 (1 Gbit) as the system’s NIC.
Read the NIC Ethernet MAC Address from this page for configuring the DHCP server.
Select the Legacy Boot Protocol item PXE.
Set Wake On LAN to Enabled.
Set the Boot Retry Count = 3 if desired.
Disable PXE boot for all unused NICs (port 1).

Press Finish to save all settings.

iDRAC (BMC) setup 