.. _Dell_R650: ================ Dell R650 server ================ .. Contents:: This page contains information out Dell PowerEdge R650_ could servers which we have deployed in our cluster. .. _R650: https://www.dell.com/en-us/work/shop/povw/poweredge-r650 Documentation and software ========================== Dell Support_ provides R650_ information: * R650_manuals_. * R650_downloads_. * iDRAC9_manuals_. * Dell blog: `Intel Ice Lake - BIOS Characterization for HPC `_. .. _R650_manuals: https://www.dell.com/support/home/us/en/04/product-support/product/poweredge-r650/manuals .. _R650_downloads: https://www.dell.com/support/home/us/en/04/product-support/product/poweredge-r650/drivers .. _Support: https://www.dell.com/support/home/us/en/04/product-support/product/poweredge-r650/research .. _iDRAC: https://en.wikipedia.org/wiki/Dell_DRAC .. _iDRAC9_manuals: https://www.dell.com/support/home/us/en/19/products/software_int/software_ent_systems_mgmt/remote_ent_sys_mgmt/rmte_ent_sys_idrac9 Booting and BIOS configuration ============================== Press **F2** during start-up to enter the BIOS and firmware setup menus. Go to the *BIOS Settings* menu. Minimal configuration of a new server or motherboard ---------------------------------------------------- At our site the following minimal settings are required for a new server or a new motherboard. Remaining settings will be configured by ``racadm``. The Dell iDRAC9_ (BMC) setup is accessed via the *System Setup* menu item *iDRAC Settings*: * In the *System Summary* page read the NIC **iDRAC MAC Address** from this page for configuring the DHCP server. * In the *Network* page set the **Enable IPMI over LAN** to **Enabled**. Go to the *System Setup* menu item *Device Settings* and select the *Integrated NIC* items: * In the NIC *Main Configuration Page* select *NIC Configuration*. We use **NIC port 3** (1 Gbit) as the system's NIC. * Read the NIC **Ethernet MAC Address** from this page for configuring the DHCP server. * Select the **Legacy Boot Protocol** item **PXE**. Boot settings menu ------------------ * The **Boot Mode** may be either **BIOS** or **UEFI**. * In the *Boot Sequence* menu: * Click the **Boot Sequence** item to move PXE boot up above the hard disk boot (if desired). * Verify that the correct devices are selected in *Boot Option Enable/Disable*. If UEFI boot mode is selected, the following must be enabled before installing the OS for the first time: * In the **Boot Setting** menu: * **Hard-disk Drive Placeholder = Enabled** Performance settings for HPC ---------------------------- The following BIOS settings should be made on HPC nodes:: [Key=BIOS.Setup.1-1#SysProfileSettings] SysProfile=PerfOptimized [Key=BIOS.Setup.1-1#SysProfileSettings] WorkloadProfile=HpcProfile [Key=BIOS.Setup.1-1#DellControlledTurbo] ControlledTurbo=Disabled [Key=BIOS.Setup.1-1#IntegratedDevices] SnoopHldOff=Roll8KCycles The CLI commands are:: racadm set BIOS.SysProfileSettings.SysProfile PerfOptimized racadm set BIOS.SysProfileSettings.WorkloadProfile HpcProfile racadm set BIOS.DellControlledTurbo.ControlledTurbo Disabled racadm set BIOS.IntegratedDevices.SnoopHldOff Roll8KCycles racadm jobqueue create BIOS.Setup.1-1 and then reboot the server. The *SnoopHldOff* value (should be 8K or 4K, the default 256 is too small) is important for high performance Omni-Path or Infiniband network fabrics. System Thermal settings ----------------------- System Thermal Profile settings can be changed based on the need to maximize performance or power efficiency. This can make **CPU thermal throttling** less likely. Read the document `Custom Cooling Fan Options for Dell EMC PowerEdge Servers `_. In the BIOS setup screen, select **iDRAC->Thermal** and configure **Thermal profile = Maximum performance**. Read the current settings:: racadm get System.ThermalSettings For HPC applications set the fans to high performance:: racadm set System.ThermalSettings.ThermalProfile "Maximum Performance" racadm set System.ThermalSettings.MinimumFanSpeed 25 Processor settings menu ----------------------- You may want to verify the following BIOS settings on HPC nodes: * Disable Hyperthreading by **Logical Processor** = **Disabled**. * **Virtualization Technology** = **Disabled**. * **Dell Controlled Turbo** = **Disabled**. * **Sub NUMA Cluster** = **Enabled**. The *Sub NUMA Cluster* (SNC_, replaces the older Cluster-on-Die (COD) implementation) has been shown to improve performance, see `BIOS characterization for HPC with Intel Cascade Lake processors `_. This will cause each processor socket to have **two NUMA domains** for the two memory controllers, so a dual-socket server will have 4 NUMA domains. Display the NUMA domains by:: $ numactl --hardware available: 4 nodes (0-3) ... .. _SNC: https://software.intel.com/content/www/us/en/develop/articles/intel-xeon-processor-scalable-family-technical-overview.html System Profile Settings menu ---------------------------- * **System Profile** = **Performance**. System Security menu -------------------- * **AC Power Recovery** = **Last** state. Miscellaneous Settings menu --------------------------- * **Keyboard NumLock** = **Off**. PXE boot setup ============== Go to the *System Setup* menu item *Device Settings* and select the *Integrated NIC* items: * In the NIC *Main Configuration Page* select *NIC Configuration*. We use **NIC port 3** (1 Gbit) as the system's NIC. * Read the NIC **Ethernet MAC Address** from this page for configuring the DHCP server. * Select the **Legacy Boot Protocol** item **PXE**. * Set **Wake On LAN** to **Enabled**. * Set the **Boot Retry Count = 3** if desired. * Disable PXE boot for all unused NICs (port 1). Press *Finish* to save all settings. iDRAC (BMC) setup ================= The Dell iDRAC9_ (BMC) setup is accessed via the *System Setup* menu item *iDRAC Settings*: * In the *System Summary* page read the NIC **iDRAC MAC Address** from this page for configuring the DHCP server. * In the *Network* page set the **Enable IPMI over LAN** to **Enabled**. * In the *User Configuration* page set the *User 2* (**root**) Administrator user name and change the **password**. The Dell iDRAC_ **default password** for *root* is **calvin** and you will be asked to change this at the first login. **IMPORTANT:** The iDRAC9_ keyboard layout is **US English**! Do not use characters that differ from the US layout! * Optional: In the *Thermal* page set Thermal: **Maximum Performance**. Press *Finish* to save all settings. .. _iDRAC9: https://www.dell.com/support/article/us/en/04/sln311300/idrac9-home?lang=en