This is a series of exercises that will walk you through the basic tasks as you get on board the Atos HPCF.

Prerequisites

In order to follow this tutorial, these are the prerequisites you must fulfil before starting:

• You must have a valid ECMWF account with privileges to access HPCF or ECS. If you only have access to ECS, you may need to skip certain exercises involving HPCF.
• You must have 2 Factor Authentication enabled with TOTP.
• You must be able to connect with at least one of the following methods:
  • Using the Virtual Desktop Infrastructure (VDI). See the corresponding documentation to get started.
  • Using Teleport SSH from your end user device .

Logging into Atos HPCF and ECS

First of all, let's try to connect to the computing services via SSH:

Accessing a login node

1. Access the default login node of the ATOS HPCF or ECS and take note of what node you are in 

   Solution - HPCF

   ssh hpc-login
   hostname

   Solution - ECS

   ssh ecs-login
   hostname

2. Open a new tab in your terminal and connect again. Did you get the same hostname? Why is that?

   Answer

   hpc-login hostname is an alias to a load-balanced service of login nodes. You may land on a different one every time you connect.

   ecs-login is an alias to a specific login node of the ECS virtual cluster. It is not automatically load-balanced, so you will typically land on the same node on consecutive connections.

   Both aliases will always point to a working login node, and the actual node and complex behind it may change depending on the load, system sessions or outages.

3. Now, from your open SSH session on Atos HPCF or ECS, connect to the main login alias again. Did it ask for a password? Can you set your account up so jumps between hosts are done without a password?

   Answer

   Password-less SSH between ECMWF hosts such as Atos HPCF or ECS nodes, or VDI hosts is not set up by default. If you were asked for a password, you can run the following command from your Atos HPCF, ECS or VDI session to set up  key-based authentication:

   ssh-key-setup

   After this you should be able to jump between hosts without having to introduce your password.

   Besides being convenient, this setup is also necessary for other tools such as ECACCESS or ecinteractive to work properly.
   

Interactive session

Standard sessions on login nodes do not guarantee access to dedicated resources such as cpus or memory, and strict limits on those are imposed.

1. Can you get a dedicated interactive session with 10 GB of memory and 4 cpus for 8 hours?

   Solution

   You can use ecinteractive. It is installed and available on all the Atos HPCF and ECS nodes, as well as the VDI, so you can run it from there

   ecinteractive -c 4 -m 10 -t 8:00

   This will create an interactive job with the requested configuration and land you on a shell in a given node.

   If you are connecting from your own computer via teleport, you can download it and run it there (no Windows native support, only Mac, Linux or WSL supported).

2. Log out of that interactive session. Can you reattach to it?

   Solution

   Your job kept running in the background, and there can only be one interactive job per user. You can attach as many concurrent shells to the same interactive session, for example in different terminal tabs, with:

   ecinteractive

3. Cancel your interactive session

   Solution

   ssh hpc-login
   hostname

Storage spaces

We will now explore the different options when it comes to storing your data.

Main filesystems

1. Connect to Atos HPCF or ECS main login node. What is your default filesystem? Can you try 4 different ways to accessing that space?

   Answer

   The default directory is your HOME directory, which is /home/$USER. It is a dedicated personal space for you, and you can always come back to that with either of the following commands:

   cd
   cd ~
   cd $HOME
   cd /home/$USER

   Your HOME directory is accessible across all Atos HPCF, ECS, VDI and EcFlow services.

2. There are 3 more main storage spaces. Create an empty file called del.me on each one of them? Check that they have been created with ls, and then remove them with rm.

   Answer

   Besides HOME, you also have also access to PERM, HPCPERM and SCRATCH. Like HOME, they are all dedicated personal spaces with their corresponding environment variable. Using those environment variables over hardcoded paths is strongly recommended.

   You can use touch to create the test files:

   touch $PERM/del.me
   touch $HPCPERM/del.me
   touch $SCRATCH/del.me

   Check they exist with:

   ls -l $PERM/del.me
   ls -l $HPCPERM/del.me
   ls -l $SCRATCH/del.me

   Remove them with:

   rm $PERM/del.me
   rm $HPCPERM/del.me
   rm $SCRATCH/del.me

3. How much space have you used in each of your main 4 filesystems? How much can you store?

   Answer

   All the filesystems have quotas enforced. You can check them with the quota command

   quota

   For HOME and PERM, the snippet should look similar to:

   Quota for $HOME:
   home_b             user    1234        <space used>   <space limit>       <number of files stored>       -   *
   
   Quota for $PERM
   POSIX User      1234    <space used>   <space limit>       <number of files stored>       none

   For SCRATCH and HPCPERM the format is slightly different:

   Project quota for $SCRATCH and $SCRATCHDIR:
   Disk quotas for prj 1000001798 (pid 1000001798):
        Filesystem    used   quota   limit   grace   files   quota   limit   grace
          /ec/res4     XXX     YYY     YYY       -     ZZZ     WWW     WWW       -
   
   Project quota for $HPCPERM:
   Disk quotas for prj 2000001798 (pid 2000001798):
        Filesystem    used   quota   limit   grace   files   quota   limit   grace
          /ec/res4     XXX     YYY     YYY       -     ZZZ     WWW     WWW       -

4. If you are on the VDI, open a new terminal there. Can you access your HOME, PERM, SCRATCH and HPCPERM ?

   Answer

   HOME and PERM are NFS-based Filesystems, which are mounted on all user computing platforms at ECMWF. You may access them with $HOME and $PERM environment variables:

   ls $HOME
   ls $PERM

   However, SCRATCH and HPCPERM  are Lustre Based filesystem only available on the Atos HPCF, so  they are not available on other computing platforms such as VDI or ecFlow VMs and the corresponding environment variables are therefore not defined.

5. EXTRA: For long term archival purposes, users with access to HPCF may also use ECFS. Files will be stored in ECMWF's Data Handling System on Tape. Create a small text file and copy it to your ECFS space, then ensure it is there, retrieve it and remove it.

   Solution

   echo "hello world" > test_file.txt
   ecp test_file.txt ec:
   els -l ec:test_file.txt
   ecp ec:test_file.txt retrieved_test_file.txt
   diff test_file.txt retrieved_test_file.txt
   erm ec:test_file.txt

Temporary spaces

There are a number of temporary spaces you can use in your session or job.

1. Create a file called testfile on the $TMPDIR, $SCRATCHDIR and /tmp/.

   Solution

   touch $TMPDIR/testfile
   touch $SCRATCHDIR/testfile
   touch /tmp/testfile

2. Open another session in the same login node with ssh $HOSTNAME. Can you find the files you have created earlier?

   Solution

   ls -l $TMPDIR/testfile
   ls -l $SCRATCHDIR/testfile
   ls -l /tmp/testfile

   You will not see the files you created in any of those locations, since every session or job will have a different location. This includes /tmp, which is also a dedicated ramdisk for session.

Filesystem Usage

Can you decide what would be the best filesystem to use in the following cases? Why would you make that choice?

1. Store the source code, scripts and configuration of your programs and workflows

   Answer

   HOME would be the preferred choice. They are typically small but important files, so convenience of backups, snapshots and availability on all computing platforms is more important than parallel performance. 

2. Store Climate Files to be used by your model runs on Atos HPCF.

   Answer

   HPCPERM is the right choice for big files that are going to be used concurrently by parallel applications such as NWP models.

3. Working directory for your jobs.

   Answer

   SCRATCH is the go to place for your daily work. Plenty of space, good parallel performance for output data that is transient by nature. Remember to move the data you want to keep after your job somewhere else, since files not used for 30 days will be automatically deleted.

4. Store data that that you use frequently, which is considerable in size.

   Answer

   PERM if accessibility from other computing platforms or the need of snapshots is important. You can see PERM as an extension to your HOME space.

   HPCPERM, if I/O performance is more important than, especially if they are going to be used in parallel jobs on Atos HPCF.

5. Store data for longer term which is considerable in size, such as experiment results. You are not going to use it often.

   Answer

   ECFS would be the right place for longer term archival or storing backups. This is by far the place where you can store  However, data on tapes needs to be retrieved to another disk space before it can be used, so it is costly in terms of time.

   In order to use ECFS efficiently, remember to store fewer but bigger files, so it is a good idea to use tools like tar or zip to bundle together big directories with lots of files.

6. Temporary files that you don't need beyond the end of the session or job

   Answer

   $TMPDIR if performance is important and size is small, since TMPDIR is either in memory (for parallel jobs on HPCF), or on SSD disk.

   $SCRATCHDIR if size of the files is big and does not fit TMPDIR.

Recovering Deleted files

1. Imagine you have accidentally deleted ~/.profile in your HOME directory. Can you get back the latest version?

   Solution

   You can use the snapshots . You can list all the versions available with:

   ls -l ~/.snapshot/*/.profile

   To recover, you would just need to copy the file back into place. 

   For longer time spans, use the utility home_snap to get the locations

2. Imagine you have accidentally deleted a file in your PERM directory. Can you get back the latest version?

   Solution

   You can use the snapshots . You can list all the versions available with:

   perm_snap

   Note that the snapshots are less frequent in PERM.

3. Imagine you have accidentally deleted a file in your SCRATCH or HPCPERM directories. Can you get back the latest version?

   Solution

   Unfortunately there are no snapshots or backups for those filesystems, so the data has been lost permanently.

Managing your software stack environment

Atos HPCF and ECS computing platforms offer a wide range of software, libraries and tools.

Basic software environment management

1. You want to use CDO, a popular tool to manipulate climate and NWP model data. What do you need to do to get the following result?
   

   $ cdo --version
   Climate Data Operators version X.Y.Z (https://mpimet.mpg.de/cdo)
   System: x86_64-pc-linux-gnu
   ...

   Solution

   If you run the command without any prior action, you may get:

   $ cdo --version
   -bash: cdo: command not found

   Many software packages and tools are not part of your default environment, and need to be explicitly loaded via modules.

   So the following commands would be sufficient to get to the desired result:

   module load cdo
   cdo --version

   ml shortcut

   You can also use the ml shortcut to load the module

   ml cdo

   Note that we did not ask for any specific version. In those cases, you will get the one defined as default.

2. How many versions of CDO can be used at ECMWF? Can you pick the newest?

   Solution

   There are hundreds of different packages with their corresponding different versions installed at ECMWF. You can use:

   module avail

   To see what modules can be loaded at any time.

   However, not all modules can be loaded at any time, some will only become available if a certain combination of modules is loaded.

   You can also use the following command for an overview or all the packages that are installed, including those that may not be visible in module avail:

   module spider

   In this case we are only interested in CDO so we can do either:

   module avail cdo

   or 

   module spider cdo

   To load the newest, you can either explicitly pick up the latest version explicitly, so assuming that it was "X.Y.Z":

   module load cdo/X.Y.Z

   But you can also use the module tag "new":

   module load cdo/new

   or also ask for the latest with:

   module --latest load cdo

   No swap needed

   If you had another version of the module loaded, the system will automatically swap it by the new one requested.

3. Load the netcdf4 module. Can you see what modules do you have loaded in your environment now?

   Solution

   To load the netcdf4 module just do:

   module load netcdf4

   Then, you can see what your software environment looks like with:

   module list

   or with just the shortcut:

   ml

   You should see both the CDO and netcdf4, beside the default modules loaded in your environment.

4. Remove the netcdf4 module from your environment and check it is gone/

   Solution

   To unload the netcdf4 module just do:

   module unload netcdf4

   or with just the shortcut:

   ml -netcdf4

   Then, you can see what your software environment looks like with:

   module list

5. Can you restore the default environment you had when you logged in? Check that the environment is back to the desired state.

   Solution

   If you log out of your session, next time you log in you will start with a fresh default environment. Modules are only loaded for that specific session.

   However, if you don't want to log out, you can also reset your module environment with:

   module reset

   You can then check the effects with

   module list

   reset vs purge

   There is a subtile difference between module reset and module purge. While the former will go back the default environment, which typically contains some default modules, the latter will completely unload all modules and leave you with a blank environment.