Dell BIOS settings common to servers 

Booting and BIOS configuration 

Press the F2 Function_key during start-up to enter system setup menus.

Minimal configuration of a new server or motherboard 

At our site the following minimal settings are required for a new server or a new motherboard. Remaining settings will be configured by racadm.

The Dell iDRAC9 (BMC) setup is accessed via the System Setup menu item iDRAC Settings:

In the System Summary page read the NIC iDRAC MAC Address from this page for configuring the DHCP server.
In the Network page set the Enable IPMI over LAN to Enabled.

Go to the System Setup menu item Device Settings and select the Integrated NIC items:

In the NIC Main Configuration Page select NIC Configuration. We use NIC port 3 (1 Gbit) as the system’s NIC.
Read the NIC Ethernet MAC Address from this page for configuring the DHCP server.
Select the Legacy Boot Protocol item PXE.

Boot Sequence menu:

Click the Boot Sequence item to move PXE boot up above the hard disk boot.

PXE boot setup 

Go to the System Setup menu item Device Settings and select the Integrated NIC items:

In the NIC Main Configuration Page select NIC Configuration. We use NIC port 3 (1 Gbit) as the system’s NIC.
Read the NIC Ethernet MAC Address from this page for configuring the DHCP server.
Select the Legacy Boot Protocol item PXE.
Set Wake On LAN to Enabled.
Set the Boot Retry Count = 3 if desired.
Disable PXE boot for all unused NICs (port 1).

Press Finish to save all settings.

It is possible to request a one-time PXE boot from the BMC using this IPMItool raw command:

ipmitool -I lanplus -H <BMC-address> -U <username> -P <password> raw 0x00 0x08 0x05 0xa0 0x04 0x00 0x00 0x00

The FreeIPMI command ipmi-raw may also be used.

Boot settings menu 

Press the F2 Function_key during start-up to enter the BIOS and firmware setup menus. Go to the BIOS Settings menu.

Boot Mode = UEFI.
In the Boot Sequence menu:
- Click the Boot Sequence item to move PXE boot up above the hard disk boot (if desired).
- Verify that the correct devices are selected in Boot Option Enable/Disable.

If UEFI boot mode is selected, the following must be enabled before installing the OS for the first time:

In the Boot Setting menu:
- Hard-disk Drive Placeholder = Enabled

UEFI Secure Boot configuration 

First set the system’s Boot Mode to UEFI, see above.

The UEFI Secure_Boot configuration can be set in the System BIOS Settings menu:

Click on the System Security item.
Set the Secure Boot item to Enabled.
Click on the Finish buttons to exit System Setup, and the system will restart in Secure_Boot mode.

Memory settings menu 

Memory Operating Mode = Optimizer Mode.
Node interleaving = Disabled.
Opportunistic Self-Refresh = Disabled.
ADDDC setting = Disabled.

Adaptive Double DRAM Device Correction (ADDDC) that is available when a system is configured with memory that has x4 DRAM organization (32GB, 64GB DIMMs). ADDDC is not available when a system has x8 based DIMMs (8GB, 16GB) and is immaterial in those configurations. For HPC workloads, it is recommended that ADDDC be set to disabled when available as a tunable option. See BIOS characterization for HPC with Intel Cascade Lake processors.

Processor settings menu 

Disable Hyperthreading by Logical Processor = Disabled.
Virtualization Technology = Disabled.
Dell Controlled Turbo = Disabled.
Sub NUMA Cluster = Enabled.

The Sub NUMA Cluster (SNC, replaces the older Cluster-on-Die (COD) implementation) has been shown to improve performance, see BIOS characterization for HPC with Intel Cascade Lake processors. This will cause each processor socket to have two NUMA domains for the two memory controllers, so a dual-socket server will have 4 NUMA domains.

Display the NUMA domains by:

$  numactl --hardware
available: 4 nodes (0-3)
...

System Thermal settings 

System Thermal Profile settings can be changed based on the need to maximize performance or power efficiency. This can make CPU thermal throttling less likely.

Read the document Custom Cooling Fan Options for Dell EMC PowerEdge Servers.

In the BIOS setup screen, select iDRAC->Thermal and configure Thermal profile = Maximum performance.

Read the current settings:

racadm get System.ThermalSettings

For HPC applications set the fans to high performance:

racadm set System.ThermalSettings.ThermalProfile "Maximum Performance"
racadm set System.ThermalSettings.MinimumFanSpeed 25

A MinimumFanSpeed value of 255 indicates the Default setting. Values between 21 (the default) and 100 may be used, but high values consume lots of power and generate noise. For HPC systems a MinimumFanSpeed of 40 to 50 may perhaps be useful.

To configure the default thermal settings:

racadm set System.Thermalsettings.ThermalProfile "Default Thermal Profile Settings"
racadm set System.ThermalSettings.MinimumFanSpeed 255