OLD Niflheim6: Getting Started information for Niflheim users

WARNING: This page has been obsoleted. Please refer to Niflheim7_Getting_started in stead.

Login to Niflheim

Authorized users of Niflheim may log in (using SSH version 2) to the interactive front-end nodes. Currently Niflheim consists of CentOS_6 compute nodes and their corresponding login nodes.

The login nodes (only accessible from the DTU network) are:

  • muspel.fysik.dtu.dk: Login node for compatibility with older Intel CPU type.
    • An 8-CPU (dual-processor, 4-core + Hyperthreading = 16 "virtual" cores), 48 GB of RAM.
    • CPUs: Intel(R) Xeon(R) CPU X5570 @ 2.93GHz Nehalem.

The login nodes are always updated to the latest CentOS_6 releases.

Windows PC users

Windows PC users are recommended to use the free Windows SSH client PuTTY for command-line SSH login to Niflheim. If you need to display graphical tools from Niflheim on your Windows PC, PuTTY will happily forward your X11 (a.k.a. X-Windows) display onto your PC. However, in order to Windows to actually display X11, you need to install an X11 server on your PC. Please see the X11_on_Windows page for further information.

Login access restrictions

Please note that for reasons of security, SSH login access is only possible from the known Internet domains of authorized DCSC grant holders and testing users.

Please log in to your department's network first, then connect to Niflheim from the authorized departmental network.

CAMD users may be authorized to use the departmental SSH gateway machine for access - please contact the IT service staff for details. Students should contact their advisor in order to be authorized for SSH access.

Installed software

Software installed on Niflheim is managed using modules. Please consult the list of available software https://wiki.fysik.dtu.dk/niflheim/Installed_software.

If you're developing a code, please login to login nodes of the same CPU architecture and version of the operating system, as the compute nodes on which your code is going to run.

Please read about available Compilers.

Running jobs

Submit jobs from one of the login nodes into the batch system as described in Batch_jobs.

Filesystems

Note: External (non-CAMD) users: please ignore this section and read https://wiki.fysik.dtu.dk/niflheim/Getting_Started#user-data-disks instead.

CAMD users

While being logged in to your CAMD desktop PC, /home/camp/$USER is your home directory on the local CAMD file server (separate from Niflheim). It has regular backup, however is not intended to store large files (type the quota -s command to see your storage limits).

While running large calculations you have two choices:

  • Run in the batch system on Niflheim, by submitting a job from one of the login nodes as described in Batch_jobs. This is the preferred way.

  • Run them locally, while logged in to your desktop PC - in this case start calculations below the /scratch/$USER directory. To see how much disk space is available type:

    df -h /scratch

    Note that /scratch/$USER has no backup so don't store your valuable data there.

While logged in to one of the login nodes your files are under /home/camp/$USER. Note that despite the same name this is a different filesystem from the one available from your PC.

Note that Niflheim's filesystem is available, as a convenience, from your CAMD desktop PC (without being logged in into a login node) under /home/niflheim/$USER. Please read more about the filesystem at https://wiki.fysik.dtu.dk/niflheim/Getting_Started#user-data-disks.

File transfer to and from Niflheim

If you need to transfer files to and from Niflheim, please use SSH's transfer method scp (Secure Copy).

You can also synchronize directories between Niflheim and your local (CAMD) machine in a simple way by using rsync over an SSH connection. On your local machine you may find these commands useful:

From Niflheim to your local machine:
rsync -av -e ssh surt.fysik.dtu.dk:niflheim_directory/ local_directory/

From your local machine to Niflheim:
rsync -av -e ssh local_directory/ thul.fysik.dtu.dk:niflheim_directory/

(Note that trailing ``/`` is important with rsync - read the rsync man-page first).

Another useful option to rsync is --exclude-from=FILE that allows one to exclude files/directories specified in the file FILE. Note that paths in FILE must be relative to the root directory of the source, e.g. niflheim_directory/ in the first example above.

If the disk on your local machine is formatted as a Windows FAT/FAT32 filesystem (for example, on an external USB disk) we suggest using these flags with rsync:

rsync -rlptv --modify-window=1 -e ssh thul.fysik.dtu.dk:niflheim_directory/ local_directory/

Windows users may use WinSCP or FileZilla, to do scp or sftp operations.

User data disks

Each user has a home directory on the group's file server.

For performance reasons it is not permitted to perform heavy disk I/O operations over extended periods of time, and the local node scratch disks (see below) must be used for heavy I/O operations.

Each user filesystem has a default disk quota, and when required increased disk quotas can be granted to individual users. Use the command:

quota -s

to display your disk quota.

Every night a daily and/or weekly backup is made to a separate backup server. This may be used to restore corrupted or deleted user files from the backups, please contact support@fysik.dtu.dk if you have a request.

CAMD users

CAMD user home directories ($HOME) are in the file system:

/home/camp/

An alias for this is /home/niflheim.

The user $HOME directories will be soft-linked to the new file server /home/niflheim1 during the ongoing file server migration process.

New CAMD fileservers

New CAMD fileserver (from April 2014) serves the file system /home/niflheim1 (server name: vid).

Users projects requiring large datasets (~50 GB to Terabytes) must use one of the new CAMD fileservers at /home/niflheim2 (server name: hrid).

Nightly snapshot backups of these file servers are made to a large backup server, and available as /home/snapshots-niflheimX (X=1,2).

Old CAMD N5200 file server

The Niflheim NFS fileserver gjoll (IBM N5200/NetApp servers) currently serves the /home/camp home directory until file migrations have been completed.

The N5200 NFS fileservers have a very useful snapshot feature enabling users to recover earlier versions of files that were deleted or corrupted, for example. In each and every directory on these servers you can go to the snapshot directory:

cd .snapshot

In the read-only .snapshot subdirectory you will find several subdirectories which contain point-in-time snapshots of all user files in the current directory. At Niflheim we have configured two snapshots taken at midnight, meaning that you have access to your files exactly as they were on the previous two midnights (older snapshots are automatically removed). The snapshot directories are named:

nightly.0/   # Most recent snapshot
nightly.1/   # The previous snapshot

You can go into these read-only subdirectories and recover (copy) any files to your normal directory structure.

Warning: an automatic process removes periodically all files starting with temp and corresponding to the regular expression:

tmp[a-zA-Z0-9_\-][a-zA-Z0-9_\-][a-zA-Z0-9_\-][a-zA-Z0-9_\-][a-zA-Z0-9_\-][a-zA-Z0-9_\-]

from user's .snapshot directories.

Nanotech and MEK users

The Linux NFS file server gunnthra service the home directories /home/mic and /home/mek.

Default disk quotas have been agreed with the respective group leaders.

Nightly snapshot backups of these file servers are made to a large backup server, and available as /home/snapshots-{mic,mek}.

Node disk space

Each cluster node (including the login nodes) have individual temporary spaces for users' scratch files. Each user has a directory /scratch/$USER for this purpose, and any job temporary files must be created on this local directory. Please make sure to move or remove your files away from the scratch space during your batch job, since communication from the login nodes to the compute nodes is prohibited.

The /scratch filesystems have a size of 45 GB to 65 GB, depending on the type of node.

Notice: Files in /scratch may be removed without notice in case of system problems.

No backup is done of files on the nodes' local disks, including the login servers' local disks.

CAMD users /home/data archive

CAMD users may obtain access to a filesystem referred to as /home/data from both the Niflheim login nodes as well as the CAMD desktop Linux PCs. This /home/data should be used to archive old data that are no longer needed in current calculations, but which must nevertheless be kept around for some time. Please contact the IT staff if you need to have a directory created for you in /home/data.

Data transfer between the Niflheim fileserver /home/camp and the /home/data will be much faster if performed on the Niflheim login nodes.

Do not move large amounts of data using your desktop PC - thank you !

We recommend you to use the rsync command on a login node (see above) to transfer data, for example:

thul# rsync -av /home/camp/$USER/subdir/ /home/data/$USER/subdir/

If you need to transfer large data sets (many Gigabytes), please contact the IT staff so that we can help you in the most efficient way.

The /home/data filesystem currently has a capacity of 6.9 TB with RAID-6 data protection, so the disk capacity is quite sufficient at the moment, and the data should be relatively safe from hardware errors. No backup is done of files on the /home/data filesystem because we do not have the capacity to do so !

Personal backup for departing users

Note that people leaving CAMD may obtain a copy of their data made onto a portable USB harddisk formatted with ext3 or ntfs formats.

Note: the ntfs format is portable and can be read on Windows, Linux, and MAC. Writing ntfs from CentOS_5 system requires additional configuration How to Mount an NTFS Filesystem, similarly there exist instructions for enabling ntfs write access on MAC.

Please contact the IT staff at least 2 weeks in advance.

Note an USB hard disk may be mounted on CentOS_5 with root user write access only. You need to mount it with write access for all users, e.g.:

mkdir /tmp/usbdisk && mount -t nfts /dev/sdb1 /tmp/usbdisk # as root

Reporting problems

Please report problems to the Niflheim_support by E-mail.

Niflheim: Old_Niflheim6_Getting_Started (last edited 2017-09-03 19:47:21 by OleHolmNielsen)