Introduction

Niflheim overview

Niflheim currently (as of January 2022) consists of the following hardware:

  • A total of 679 compute nodes.
  • The nodes contain a total of 24560 CPU cores.
  • Aggregate theoretical peak performance of more than 1785 TeraFLOPS (TFLOPS) or 1.785 PetaFLOPS.
  • Total RAM memory is 247 TB.

CPU architectures

Niflheim comprises several generations of hardware for a total of 679 compute nodes:

Dell Ice Lake servers

  • 96 56-core nodes Dell R650 with two Intel IceLake Xeon_Gold_6348 28-core CPUs (a total of 5376 cores) running at 2.60 GHz Base frequency (up to 3.50 GHz in Turbo_mode) and including AVX-512 vector instructions.

    Peak speed: 4659 GFLOPS/node, 447 TFLOPS in total for all nodes.

    The RAM memory is 512 GB (9.14 GB/core) 3200 MHz DDR4 dual-rank memory, 49.1 TB in total for all nodes.

    Each server has a 480 GB local SSD hard disk.

    The network interconnect is 100 Gbit/s Cornelis_Networks (previously Intel) OmniPath.

    Installed in December 2021.

SuperMicro servers

  • 8 40-core nodes from Nextron/SuperMicro with Intel Xeon_Gold_5218R 20-core CPUs @2.10 GHz (a total of 320 cores). Hyperthreading is enabled so that each node has 80 threads or virtual CPUs.

    Peak speed: 2688 GFLOPS/node, 21 TFLOPS in total for all nodes.

    The RAM memory is 198 GB (4 nodes) and 768 GB (4 nodes), 3.81 TB in total for all nodes.

    These servers are equipped with Nvidia GPUs.

    Each server has a 960 GB local NVMe hard disk and a 1 Gbit Ethernet.

    Installed in December 2020 and July 2021.

Dell Cascade Lake Refresh servers

  • 128 40-core nodes Dell R640 with two Intel Cascade_Lake Xeon_Gold_6242R 20-core CPUs (a total of 5120 cores) running at 3.10 GHz Base frequency (up to 4.10 GHz in Turbo_mode) and including AVX-512 vector instructions.

    Peak speed: 3968 GFLOPS/node, 508 TFLOPS in total for all nodes.

    The RAM memory is 384 GB (9.6 GB/core) 2933 MHz DDR4 dual-rank memory, 49.1 TB in total for all nodes.

    Each server has a 240 GB local SSD hard disk.

    The network interconnect is 100 Gbit/s Intel OmniPath.

    Installed in October 2020.

Dell Skylake servers

  • 208 40-core nodes Dell C6420 and R640 with two Intel Skylake Xeon_Gold_6148 20-core CPUs (a total of 8320 cores) running at 2.40 GHz Base frequency (up to 3.70 GHz in Turbo_mode) and including AVX-512 vector instructions.

    Peak speed: 3072 GFLOPS/node, 639 TFLOPS in total for all nodes.

    The RAM memory type is 2666 MHz DDR4 dual-rank memory:

    • 196 C6420 nodes have 384 GB of memory (9.6 GB/core), 75.3 TB in total for all nodes.
    • 12 R640 nodes have 768 GB of memory (19.2 GB/core), 9.2 TB in total for all nodes.

    Each server has a 240 GB local SSD hard disk.

    The network interconnect is 100 Gbit/s Intel OmniPath.

    Installed in April 2019.

Huawei Broadwell servers

  • 192 24-core nodes Huawei XH620 v3 with two Intel Broadwell Xeon_E5-2650_v4 12-core CPUs (a total of 4608 cores) running at 2.20 GHz (up to 2.90 GHz in Turbo_mode).

    Peak speed: 845 GFLOPS/node, 162 TFLOPS in total for all nodes.

    The RAM memory type is 2400 MHz DDR4 dual-rank memory:

    • 180 nodes have 256 GB of memory (10.7 GB/core), 46.1 TB in total for all nodes.
    • 12 nodes have 512 GB of memory (21.3 GB/core), 6.1 TB in total for all nodes.

    Each server has a 240 GB local SSD hard disk.

    The network interconnect is 100 Gbit/s Intel OmniPath.

    Installed in December 2016, March 2017, November 2017.

Dell Ivy Bridge servers

  • 47 16-core nodes Dell PowerEdge C8220 with two Intel Ivy_Bridge Xeon_E5-2650_v2 8-core CPUs (a total of 816 cores) running at 2.60 GHz (up to 3.40 GHz in Turbo_mode).

    Peak speed: 166 GFLOPS/node, 8 TFLOPS in total for all nodes.

    The RAM memory type is 1833 MHz DDR3 dual-rank memory:

    • 27 nodes have 128 GB of memory (8 GB/core), 3.5 TB in total for all nodes.
    • 20 nodes have 256 GB of memory (16 GB/core, 5.1 TB in total for all nodes).

    Each server has a 300 GB local SSD hard disk. The network interconnect is QDR Infiniband.

    Installed in May 2014, April 2015 and June 2015.

File servers

Several Linux file servers are available for the departmental user groups. Each group is assigned a file-system on one of the existing file servers. Depending on disk requirements, group file-systems can be from 1 TB and up.

The file servers are standard Linux servers with large disk arrays, sharing the file-systems using NFS. We do not use any parallel file servers (for example, Lustre etc.).

The file server total available disk spaces are:

  • Server niflfs1: 108 TB
  • Server niflfs3: 87 TB
  • Server niflfs4: 90 TB
  • Server niflfs5: 90 TB
  • Server niflfs6: 106 TB
  • Server niflfs7: 106 TB
  • Server niflfs8: 163 TB
  • Server niflfs9: 163 TB

A maximum disk capacity of 913 TB disk space is available for user applications.

Niflheim: Hardware (last edited 2022-03-30 10:42:32 by OleHolmNielsen)