You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Description

For the projects like TIGGE, S2S or UERRA the exact data format, WMO compliant GRIB2 in those cases,  is required to allow easy data processing and intercomparison. To check that the encoding is as requested, one can use tigge_check tool which is a part of ecCodes package.

The tigge_check can do also very basic quality control by checking allowed value ranges for each parameter (with -v option). Some allowed ranges can become obsolete at some point e.g. due to model upgrade to higher resolution meaning in general different values of some parameters.  There is another newer better maintainable tool doing similar basic quality check called grib_check.py.

Examples

tigge_check options
$ tigge_check
tigge_check [options] grib_file grib_file ...
   -l: check local area model fields
   -v: check value ranges
   -w: warnings are treated as errors
   -g: write good gribs
   -b: write bad  gribs
   -z: return 0 to calling shell
   -s: check s2s fields
   -r: check s2s reforecast fields
   -u: check uerra fields
Checking UERRA data

# example with one data value ranges warning and one encoding error
> tigge_check -u -w -v -z *.grib2
> warning: an.enda.tcw.sfc.0.fp.grib2, field 1 [total_column_water_sfc]: total_column_water_sfc maximum value 31.5638 is not in [50,150]
  => [31.5638,150]
> fc.oper.2t.sfc.2.grib2 , field 1 [temperature_hl]: invalid height level 3

Checking S2S reforecast data

> tigge_check -r *.grib2

Performance tip to speed up checking big files 

There is a new tool  (ecCodes v>=2.6.0) called codes_split_file which is useful for parallellising decoding/checking tasks like tigge_check.



NAME    codes_split_file
DESCRIPTION
        Split an input file (GRIB, BUFR etc) into chunks of roughly the same size.


        The output files are named input_1, input_2 etc. This is much faster than grib_copy/bufr_copy.
USAGE
        codes_split_file [-v] nchunks input
OPTIONS
        -v  Print the count of messages and files created

If one has a very large inputfile with 1000s of messages, instead of running one process which sequentially checks each message in thefile, one cansplitthefileinto 8 chunks and run the checking code in parallel on the 8 outputfiles.

set -e

# Assume you have 8 cores
codes_split_file 8 my_big.grib

# Now you will have my_big.grib_01, my_big.grib_02, ... my_big.grib_08
for f in my_big.grib_*; do
  # Run check in the background. Now multiple processes are running in parallel
  tigge_check $f &
done

# With the 'wait' command you can force the execution of the script to pause until a
# all background jobs have finished executing before continuing the execution
# of your script
wait

# Now clean up the split files
rm -f my_big.grib_*


  • No labels