Introduction

For the projects  like TIGGE, S2S, UERRA or LC_WFV it is important to have good checking tools deployed on the both ends of the processing chain, provider's and ECMWF's ones, to avoid common operational problems related to preparation and dispatching the data.

Find below some examples what we use to do at ECMWF as a part of the operational data processing suites. If you are our data provider, please try to install similar checks on your end to minimise our common future support effort. It would help us handling increasing number of similar projects in a sustainable way and without additional resources. 

Checking data delay

It should be checked that the data production and dispatching is done within expected time frame. This can be of high importance as some data products cannot be produced any more after a longer delay from whatever reason (e.g. full model results which are needed are kept only for limited time).

There are many ways how to check the delays, depending on the data provider's production chain e.g..

• regular cron based check
• automated log file analysis by software like splunk

Checking data content

Following checks should be done before providing the data:

• Data encoding check

  • the proper data encoding to the required data exchange format (WMO GRIB2) can be checked by the dedicated ECMWF tools

• Comparing data content against reference

  • the number of the fields must be as expected
  • the actual field list must be the same as expected
    • see example in Comparing data content against reference how to create a reference fields list from your input data and compare it against the actual data content

• Data quality check

  • the best and most complete quality control can be done by a custom made tools only, e.g. during the forecast computation by the model
  • the basic quality control can be done using the dedicated ECMWF tools which are simply checking the allowed limits (min/max values in each grid point) of each parameter

Checking data transfer success

If data is pushed to our premises, the following steps can ensure smooth dispatching:

• Make the transferred file "invisible"

  • e.g. by adding the dot before the file name and rename it only after the transfer is finished
  • that way the ECMWF's automated acquisition action will not be triggered prematurely

• Compare the file sizes before and after the transfer

  • the file size must be identical to ensure the file was not corrupted during the transfer (should be done still before the file renaming)