Description

For the projects  TIGGE, S2S and UERRA the exact data format, WMO compliant GRIB2,  is required to allow easy data processing and intercomparison. Read in Data encoding checking tools more information about tigge_check and grib_check.py tools. Apart from encoding verification, theses tools can do also some basic quality control by checking the allowed value ranges, if they were defined, for each parameter.

Please be aware that some previously allowed valid ranges can become obsolete at some point e.g. due to model upgrade to higher resolution meaning in general different values of some parameters. Because of that a continuous set up updates are needed so it is important to use the latest versions of the tools. Also for some parameters the allowed ranges must not be defined due to various reasons. In some cases the allowed limits for different data sets or data origins  might also exist because of known  special features or deficiencies of some related model's outputs.

Examples of tigge_check usage

Checking TIGGE data
module load eccodes
load eccodes 2.9.2 Library (ECCODES_DIR, ECCODES_VERSION, ECCODES_INCLUDE, ECCODES_LIB, ECCODES_INCLUDE_DIR, ECCODES_LIB_DIR)
# example with one data value ranges warning
tigge_check -u -w -v
tigge_mogreps-g_2019021600_00.grib2
warning: tigge_mogreps-g_2019021600_00.grib2, field 513 (Potential vorticity, paramId=60), matched definition (class:ti, model:glob, origin:egrr) 
  => maximum value 1296779.249971 is not in [0.000001,100.000000]

Examples of grib_check.py usage

grib_check options
usage: grib_check.py [-h] [-v VERBOSITY] [-w] [-f] [-his] [-l [LISTING]]
                     [inp_file [inp_file ...]]

positional arguments:
  inp_file              enter input file name(s)

optional arguments:
  -h, --help            show this help message and exit
  -v VERBOSITY, --verbosity VERBOSITY
                        increase output verbosity [0-2]
  -w, --warning         warnings are treated as errors..
  -f, --fail            force abort for undefined parameters..
  -his, --history       add value ranges history, if exists, to listing..
  -l [LISTING], --listing [LISTING]
                        list defined parameters (search if argument provided!)
Checking LC-WVF data
export BIN=/home/ma/emos/def/s2s/s2s_prod/bin

python $BIN/grib_check.py lw.grib2
warning: lw.grib2, field 700 (Peak wave period, paramId=140231), matched definition (default) => minimum value -9.699028 is not in [0.000000,35.000000]
warning: lw.grib2, field 718 (Peak wave period, paramId=140231), matched definition (default) => maximum value 41.463626 is not in [0.000000,35.000000]
Checking parameter set up

List all defined limits

python $BIN/grib_check.py -l
165   [10 metre U wind component]   [min <-150, -0.1> max <0.1, 150>] (default)  
                                   [min <-150, -0.1> max <0.1, 150>] (class:ti, model:lam, suiteName:mogreps-mo-eua)  
166   [10 metre V wind component]   [min <-100, -1> max <0.5, 100>] (default)  
                                   [min <-100, 10> max <-10, 100>] (class:ti, model:lam, suiteName:mogreps-mo-eua)  
260260   [10 metre wind direction]   [min <0, 1> max <359, 360.1>] (default) 

...

See allowed limits for parameters with paramId containing 141

python $BIN/grib_check.py -l 141
228141   [Snow depth water equivalent]   [min <-1e-10, 0> max <100, 15000>] (default)  
                                        [min <-1e-10, 0> max <100, 40000>] (class:s2, origin:cwao)  
                                        [min <-1, 0> max <100, 100000>] (class:ti)

See allowed limits for parameters with paramId containing 141 including past set up history

python $BIN/grib_check.py -l 141 -his
228141   [Snow depth water equivalent]   [min <-1e-10, 0> max <100, 15000>] (default)  

# History of changes:  
# 
#    1.version (valid from: 2017-01-01) 
#     min <-1e-10, 0> 
#     max <100, 15000> 
#        info:See long history of changes in tigge_check.h! 

228141   [Snow depth water equivalent]   [min <-1e-10, 0> max <100, 40000>] (class:s2, origin:cwao)  

# History of changes:  
# 
#    1.version (valid from: 2013-02-11) 
#     min <-1e-10, 0> 
#     max <100, 40000> 
#        info:s2s.z_s2s_c_cwao_20130211000000_glob_prod_sl_000.grib2, field 23: snow_depth_water_equivalent_sfc.glob.s2 maximum value 30452.5 is not in [100,15000] 

228141   [Snow depth water equivalent]   [min <-1, 0> max <100, 100000>] (class:ti)  

# History of changes:  
# 
#    3.version (valid from: 2017-08-08) 
#     min <-1, 0> 
#     max <100, 100000> 
#        info:z_tigge_c_RKSL_20170808000000_glob_prod_cf_0240_000.grib2, field 119 (Snow depth water equivalent, paramId=228141), matched definition (default) =>  maximum value 
94537.8 is not in [100,15000] 
#    2.version (valid from: 2017-08-08) 
#     min <-1, 0> 
#     max <100, 90000> 
#        info:warning: tigge_mogreps-g_2017080812_00.grib2, field 445 => maximum value 67640.3 is not in [100,15000] 
#    1.version (valid from: 2017-10-17) 
#     min <-1, 0> 
#     max <100, 90000> 
#        info:warning: ncmrwf_tigge_20170807_000.grib2, field 1749 (Snow depth water equivalent, paramId=228141), matched definition (default) =>  maximum value 78601.367188 is 
not in [100.000000,15000.000000]

Installating grib_check.py

Get a Visibility for Confluence license to continue

Your Visibility for Confluence license does not match the LicenseType<COMMUNITY> license on this Confluence installation. Please get a LicenseType<COMMUNITY> license for Visibility for Confluence and try again.

The prerequisite is  to have eccodes python 3 modules available in the system. The tool itself consist of two python scripts attached to this page:

The set up file contains mostly a python dictionary with the allowed ranges for parameters. These ranges  can be  easily tuned and can be specific for specific selection of metadata if needed (e.g. MARS class, level type etc).

Please be aware that this tool is not part of any package yet as  is still under development. For example some code refactoring or performance optimisation might be needed.