It is important to have good checking tools deployed on the both ends of the processing chain, provider's and ECMWF's ones, to avoid common operational problems related to preparation and dispatching the data for the project.

Find below some examples what we use to do at ECMWF as a part of the operational data processing suites. Please install similar checks on your end to minimise future support effort. Only by doing that we could handle increasing number of similar projects in a sustainable way and without additional resources. 

Checking data delay

It should be checked that the data production and dispatching is done within expected time frame. This can be of high importance as some data products cannot be produced any more after a longer delay from whatever reason (e.g. full model results which are needed are kept only for limited time).

There are many ways how to check the delays, depending on the data provider's production chain e.g..

• regular cron based check
• automated log file analysis by software like splunk

Checking data content

Following checks should be done before providing the data:

• Data encoding check

  • the proper data encoding to the required data exchange format (WMO GRIB2) can be checked by the dedicated ECMWF tools

• Checking the file content against reference

  • the number of the fields must be as expected
  • the actual field list must be the same as expected
    • example how to create a reference fields list from your input data and compare it against the actual data content

• Data quality check

  • the best and most complete quality control can be done by a custom made tools only, e.g. during the forecast computation by the model
  • the basic quality control can be done using the dedicated ECMWF tools which are simply checking the allowed limits (min/max values in each grid point) of each parameter

Checking data transfer success

If data is pushed to ECMWF, the following basic check should be done after each file transfer:

• Make the transferred file "invisible"

  • e.g. by adding the dot before the file name and rename it only after the transfer is finished
  • that way the ECMWF's automated acquisition action will not be triggered prematurely

• Compare the file sizes before and after the transfer

  • the file size must be identical to ensure the file was not corrupted during the transfer
  • do it still before the file renaming