We will now explore the different options when it comes to storing your data.
Connect to Atos HPCF or ECS main login node. What is your default filesystem? Can you try 4 different ways to accessing that space?
The default directory is your HOME directory, which is /home/$USER. It is a dedicated personal space for you, and you can always come back to that with either of the following commands:
Your HOME directory is accessible across all Atos HPCF, ECS, VDI and EcFlow services. |
There are 3 more main storage spaces. Create an empty file called del.me
on each one of them. Then, check that they have been created with ls
, and then remove them with rm
.
Besides HOME, you also have also access to PERM, HPCPERM and SCRATCH. Like HOME, they are all dedicated personal spaces with their corresponding environment variable. Using those environment variables over hardcoded paths is strongly recommended. You can use touch to create the test files:
Check they exist with:
Remove them with:
|
How much space have you used in each of your main 4 filesystems? How much can you store?
All the filesystems have quotas enforced. You can check them with the quota command
For
For
|
If you are on the VDI, open a new terminal there. Can you access your HOME
, PERM,
SCRATCH
and HPCPERM
?
However, |
For long term archival purposes, most users may also use ECFS. Files will be stored in ECMWF's Data Handling System on Tape. Create a small text file and copy it to your ECFS space, then ensure it is there, retrieve it and remove it.
Users from Cooperating States do not have full access to the ECFS service, but only to a temporary storage domain on tape for 90 days. They should therefore not use ECFS for longer term archival. |
For users with full access to the ECFS service, you can use the
Cooperating State users can only use the
|
There are a number of temporary spaces you can use in your session or job.
Create a file called testfile
on the $TMPDIR
, $SCRATCHDIR
and /tmp/
.
|
Open another session in the same login node with ssh $HOSTNAME
. Can you find the files you have created earlier?
You will not see the files you created in any of those locations, since every session or job will have a different location. This includes /tmp, which is also a dedicated ramdisk for session. |
Can you decide what would be the best filesystem to use in the following cases? Why would you make that choice?
Store the source code, scripts and configuration of your programs and workflows
|
Store Climate Files to be used by your model runs on Atos HPCF.
|
Working directory for your jobs.
|
Store data that that you use frequently, which is considerable in size.
|
Store data for longer term which is considerable in size, such as experiment results. You are not going to use it often.
ECFS would be the right place for longer term archival or storing backups. This is by far the place where you can store However, data on tapes needs to be retrieved to another disk space before it can be used, so it is costly in terms of time. In order to use ECFS efficiently, remember to store fewer but bigger files, so it is a good idea to use tools like tar or zip to bundle together big directories with lots of files. |
Temporary files that you don't need beyond the end of the session or job
$TMPDIR if performance is important and size is small, since TMPDIR is either in memory (for parallel jobs on HPCF), or on SSD disk. $SCRATCHDIR if size of the files is big and does not fit TMPDIR. |