Versions Compared

Old Version 35

changes.mady.by.user Carsten Maass

Saved on

compared with

New Version Current

changes.mady.by.user Bojan Kasic

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

ECFS is ECMWF's File Storage system. ECFS is a file oriented client-server application, providing facilities to archive and retrieve files between your local workstation or server HPC/ECS and the Data Handling System (DHS), which is based on HPSS (High Performance Storage System). The system is non-transparent: you must issue explicit commands to store files into ECFS, or to retrieve them back into local (client) storage.

...

Excerpt

Each ECFS user has an ecfs_audit file, placed in each of their ECFS domains. These files contain a list of all files you own in the relevant ECFS domai, excluding any backup copies. To list the audit file in your ec: domain use the following command:

els -la ec:ecfs_audit

Small audit files can be inspected quickly using the ecat command, e.g.

ecat ec:ecfs_audit - | less 

To examine the audit file in more detail you might want to copy it to your disk space in $HOME, $PERM or $SCRATCH, e.g. :

ecp ec:ecfs_audit $SCRATCH

For a complete list of available ECFS commands please refer to the ECFS user documentation or read the relevant man pages on ecgate.

The ecfs_audit file lists your files (no directories) in the format 

--    uid gid        size(bytes) creation    last_access path    today=2018-09-13
...
*+    uid gid            114176  20120307     20120702  /uid/directory1/file1
*     uid gid            114176  20120307     20120702  /uid/directory2/file1
b+    uid gid             19363  20141208     20141208  /uid/directory2/file2
...
Total files =924    gigabytes = 282.40779780224
total directories = 52 total files not accessed since 20170312 = 865

where the columns have the following meaning:

ColumnComment
--

Attribute:

b  - file is backed-up
+  - file has not been accessed for over 18 months
*  - file is not backed-up
o  - file is offline
O  - file is offline, with backup-up

uid User-id of the file's owner
gid The file's Unix group
size(bytes)
File size in bytes
creation
File creation date in format YYYYMMDD
last_access
Date of last access, e.g. read, in format YYYYMMDD
path
Full path with a file name in ec:

The audit file's creation date in the format "today= YYYY-MM-DD" is stated in its first line. At the bottom the total of your ECFS content is reported in terms of number and (binary) volume. The very last line gives the number of directories and the number of files not accessed for 18 months.

...

  • DON'T archive many small files separately. ECFS is most efficient at handling a small number of large files. Thus 
  • DO tar and compress (or gzip) many small files into one large file when archiving (say) a directory or use the ecfsdir command. 
  • DON'T copy in/out the same files frequently.
  • If you wish to archive files for a short period only (less than 90 days) DO store them in the ectmp: domain. Then they will be automatically deleted after 90 days without any further action from you.

  • Check the existence of a local copy before getting the ECFS version of a file:

    #!/bin/ksh
if [ ! -r $SCRATCH/myfile ]
then
  ecp ec:myfile $SCRATCH/.  
fi


  • If it is required to store a large number ( > 5000) of files into ECFS then DO contact the Service Desk ECMWF Support Portal in advance to discuss the most efficient way to store/retrieve this large number.

...