No filesystems from other platforms are cross-mounted, so you will need to copy over what you need.
First of all, make sure you have set up password-less authentication as described in HPC2020: How to connect before proceeding.
For transfers, we recommend using rsync which will transfer the files over an ssh connection. For that, you will need to have Teleport configured with the apropriate settings in your ssh config file.
Any file transfer tool that supports SSH and the ProxyJump feature should work, such as the command line tools sftp or scp. Alternatively, you may also use the Linux Virtual Desktop and its folder sharing capabilites to copy local files to your ECMWF's HOME or PERM.
| Note |
|---|
All examples are done with the generic HPCF login node hpc-login, but you should use |
...
Transferring a directory tree between
...
your computer and the Atos HPCF
...
| Warning | ||
|---|---|---|
| ||
If you are transferring your HOME from Reading platforms into Atos HPCF, make sure you do not transfer your dot files and directories in your home top level directory, such as your ~/.profile, ~/.bashrc, ~/.config or ~/.local. They may interfere with the local configuration on the Atos HPCF. Instead, we suggest you carefully select and transfer your files and directories in your home that you really need to keep. |
with rsync
This is the recommended and most versatile option. You may initiate the transfer from ECGATE your computer for the standard set of filesystems:
| No Format |
|---|
user@ecgb11user@yourlaptop:~> rsync -avz $SCRATCH/mydataset hpc-login:/scratch/user/ |
or alternatively from Atos HPCF:
| No Format |
|---|
user@aa6-100:~> rsync -avz ecgate:/scratch/group/user/mydataset $SCRATCH/ |
Both solutions are equivalent, and This command can be run multiple times. Only new , since only new or modified files will be transferred.
| Tip |
|---|
You may add the |
The same idea can be applied to transfer files from your Cray HPCF filesystems.
...
You can use the $HOME environment variable to refer to your files in the rsync command provided this is protected by single quotes to ensure the variable is expanded on the remote host.
For example, to copy a directory from $HOME on ecgate to $HOME on Atos HPCF, initiating the transfer from ecgate, use:
| No Format |
|---|
rsync -avz $HOME/mydataset 'hpc-login:$HOME/' |
...
An alternative option is to use the $HOME variable with pattern substitution. For example, to copy a directory tree from $SCRATCH on ecgate to $SCRATCH on Atos HPCF running the command on ecgate use:
| No Format |
|---|
user@ecgb11:~> rsync -avz $SCRATCH/mydataset 'hpc-login:${HOME/home/scratch}/' |
or from $PERM on ecgate to $PERM on Atos HPCF running the command on the latter use:
| No Format |
|---|
user@aa6-100:~> rsync -avz 'ecgate:${HOME/home/perm}/mydataset' $PERM/ |
The single quotes are needed here to ensure the $HOME variable takes its value on the remote rather than the local host.
...
| group | ecmwf |
|---|
Transferring files from local Reading-based workstation disks /var/tmp or /hugetmp
Direct transfers from your workstation to the spaces in Bologna are discouraged, since they would not use the dedicated network link between the two data centres and transfer rates will be poor. Instead, you may force the transfers to jump through ecgate for best results. For example, to transfer a directory and its contents under your local disk onto the PERM space in Bologna:
Transferring a single file between your computer and the Atos HPCF with scp
You may initiate the transfer from your computer:
| No Format |
|---|
user@yourlaptop:~> scp myfile hpc-login:/scratch/user/ |
Note that running the command multiple times will always overwrite the file on destination.
Transferring a single file between your computer and the Atos HPCF with sftp
You may initiate the transfer from your computer:
| No Format |
|---|
user@yourlaptop:~> sftp hpc-login
sftp> cd /scratch/user
sftp> put myfile |
Note that running the command multiple times will always overwrite the file on destination.
| No Format |
|---|
user@leap42-workstation:~> rsync -avz -e "ssh -x -o ProxyCommand='ssh -q -x -W %h:%p ecgate'" /var/tmp/user/mydataset hpc-login:/perm/user/mydataset |
Transferring files from Reading-based workstation filesystems $HOME and $PERM
...
Advanced: High Performance Transfers with bbcp
You may also use a specialised tool called bbcp for best transfer rates. It is available on Atos HPCF, as well as Reading Based platforms Cray HPCF and ECGATE. This tool is not as flexible as rsync when it comes to updating existing or partial copies, but it should be quicker when doing a one-off transfer.
For example, if you wanted to transfer a directory called mydataset in your SCRATCH in Reading, you could initiate the transfer from ECGATE:
| No Format |
|---|
user@ecgb11user@yourlaptop:~> bbcp -rp -s 10 -P 2 $SCRATCH/mydataset hpc-login:/scratch/user/ |
or alternatively from Atos HPCF:
| No Format |
|---|
user@aa6-100:~> bbcp -rp -s 10 -P 2 ecgate:/scratch/group/user/mydataset $SCRATCH/ |
The options above would perform a recursive copy, preserving file permissions and using 10 parallel streams. It would also report progress every 2 seconds.
You may find all the details in the bbcp official documentation.
...