...
Within this Framework, Member State users can use the Atos HPCF and ECGATE ECS services (HPCF resources are needed to use the HPCF / HPC service). In general, users should minimise the number of systems they use. Use of the HPC service is recommended for all computationally intensive (e.g. for running a numerical model) time-critical option 2 and 3 activity. The ECGATE (ECS) service should be used only for work that is not excessively computationally intensive such as post-processing output graphically before it is transferred to their Member State. Member State time-critical work may also need to use additional systems outside ECMWF after processing at ECMWF has been performed, for example to run other models using data produced by their work at ECMWF. It is not the purpose of this document to provide guidelines on how to run work which does not make use of ECMWF computing systems.
...
| Note | ||
|---|---|---|
The ecFlow environment is not set up by default for users on the Atos HPCF and ECGATE systems. Users will have to load the ecFlow environment using
|
...
- Your work requires input data which is produced by any of the ECMWF models. In such case it is possible to set up a specific dissemination stream which will send the required data to the HPCF. ECPDS allows for the "local" dissemination to a specific User ID (the User ID used to run time-critical work) so that only this recipient User ID can see the data and is similar to the standard dissemination to remote sites. This ”local” dissemination option is the recommended option. The recipient User ID is responsible for the regular clean-up of the received data.
If produced by ECMWF, your required data will also be available in the FDB and will remain online for a limited (variable depending on the model) amount of time. You can access these data using the usual ”mars” command. If your suite requires access to data which may no longer be contained in the FDB then your suite needs to access these data before they are removed from the FDB and temporarily store them in one of your disk storage areas.
Warning For no reason should any of your time-critical suite tasks depend on data only available from the Data Handling System (MARS archive or ECFS). Beware that using a keyword value of "ALL" in any mars request will automatically redirect it to the MARS archive (DHS). Note also that we recommend you do not use abbreviations for a MARS verb, parameter or value in your mars requests. If too short, these abbreviations may become ambiguous if a new verb, parameter or value name is added to the mars language.
- Your work requires input data which is available at ECMWF but not produced by an ECMWF model. For example, your work requires observations normally available on the GTS e.g. if you are interested in running some data assimilation work at ECMWF. In such a case you can obtain the required observations from /ec/vol/msbackup/ on the Atos HPCF where they are stored by a regular extraction task which runs as part of the ECMWF operational suite. For any other data you may need for your time-critical activity and which is available at ECMWF, please ask your User Services technical contact.
Your work requires input data which is neither produced by any of the ECMWF models nor available at ECMWF. You will then be responsible for setting up the required ”acquisition” tasks and establish their level of time criticality. For example, your suite may need some additional observations which improve the quality of your assimilation but your work can also run without them in case there is a delay/problem in their arrival at ECMWF. Please see the section ”Data transfers” for advice on how to transfer incoming data.
...
| Note |
|---|
Note that, by default, ectrans transfers are asynchronous; the successful completion of the ectrans command does not mean your file has been transferred successfully. You may want to use the option ”-put” to request synchronous transfers. |
...
Incoming data - sending data
...
to ECMWF
We recommend ectrans (option -get) to upload some data from a remote site to ECMWF. Other options, including the use of ECPDS in acquisition mode, may be considered in specific situations. Please discuss your requirements with your User Services technical contact.
...
The User IDs authorised to run Option 2 work have access to all Atos HPCF complexes and you are advised to implement your suite so it can run on any of the clusters in case a specific cluster is unavailable for an extended period of time.
...