Versions Compared

Old Version 1

changes.mady.by.user Paul Dando

Saved on

compared with

New Version Current

changes.mady.by.user Unknown User (nagc)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Using the

...

 

GRIB tools


The GRIB tools are part of the ECMWF GRIB API Library. They are a set of command line programs for interactive and batch decoding and processing of GRIB data

They provide ready and tested solutions to the most common processing of GRIB data and work with both GRIB edition 1 and GRIB edition 2.

All of the tools use a common syntax:

Code Block
grib_<tool> [options] grib_file grib_file … [output_grib]

There are tools for getting information about the GRIB API installation and the keys available

  • grib_info,   grib_keys

There are tools to inspect the content of and compare GRIB messages

  • grib_ls,   grib_dump,   grib_get,   grib_get_data,  grib_compare

There are tools for counting and copying some messages

  • grib_count,   grib_copy   

There is a tool for making changes to the content of a GRIB message

  • grib_set

There is an advanced tool with a macro language provided via a rules file which allows for more sophisticated manipulation of GRIB files

  • grib_filter

There is a tool for converting from GRIB to netCDF

  • grib_to_netcdf


You will find some example data in $SCRATCH/ectrain/trx/openifs or you can use output from one of your own experiments. 

Let's play !

Getting help with the tools

...

  1. Specifying the information to display:

    Code Block
    grib_ls -p dataDate,dataTime,stepRange,shortName,typeOfLevel,level ICMGGepc8+000000


  2. Specifying the type of key (s: string, i: integer).  To output the centre as a string (default):

    Code Block
    grib_ls -p centre:s ICMGGepc8+000000

    or as an integer:

    Code Block
    grib_ls -p centre:i ICMGGepc8+000000


  3. Listing specific namespace information.  Try:

    Code Block
    grib_ls -n parameter ICMGGepc8+000000

    and

    Code Block
    grib_ls -n statistics ICMGGepc8+000000

    Other namespaces to try are time, vertical and geography.

  4. Order the output.  Try:

    Code Block
    grib_ls -B "level desc" ICMSHepc8+000000

    Compare and contrast with:

    Code Block
    grib_ls -B "level:i desc" ICMSHepc8+000000


  5. The where option:  IS

    Code Block
    grib_ls -w shortName=stl1 ICMGGepc8+000000

    IS NOT

    Code Block
    grib_ls -w shortName!=stl1 ICMGGepc8+000000

    OR

    Code Block
    grib_ls -w shortName=stl1/z ICMGGepc8+000000

    AND

    Code Block
    grib_ls -w shortName=q,level=10 ICMGGepc8+000000


  6. Specifying the step unit.  Compare the output from:

    Code Block
    grib_ls -n time ICMGGepc8+000000

    and

    Code Block
    grib_ls -s stepUnits=m -n time ICMGGepc8+000012

    The stepUnits can be set to s,m,h,3h,6h,12h,D,M,Y,10Y,30Y or C.  The result must be expressible as an integer (try setting stepUnits=D above). This is a temporary change for display purposes only !

Note

grib_ls is a tool meant mainly for interactive use.  If you want to use the tools in a shell script then take a look at grib_get instead. It works in much the same way as grib_ls but is less verbose and also fails if an error occurs.

Obtaining detailed information about a GRIB message with grib_dump

  1. To print all of the keys available for a particular GRIB message:

    Code Block
    grib_dump -w shortName=stl1 ICMGGepc8+000012


  2. To obtain a listing which is more like the WMO tables:

    Code Block
    grib_dump -O -w shortName=stl1 ICMGGepc8+000012


  3. To obtain even more detail:

    Code Block
    grib_dump -OatH -w shortName=stl1 ICMGGepc8+000012


Printing the data values

For grid point data, the grib_get_data tool prints a list of latitude, longitude and data values from one or more GRIB files. 

  1. Try

    Code Block
    grib_get_data -w shortName=z  ICMGGepc8+000000


  2. The format of the data values can be controlled with a C-style format statement.  Try:

    Code Block
    grib_get_data -w shortName=z -F"%.4f" ICMGGepc8+000000


  3. Other information can be printed along with the lat-lon-values triplet:

    Code Block
    grib_get_data -p dataDate,dataTime,stepRange -w shortName=stl1 -F"%.4f" ICMGGepc8+000000


Warning

If you need the values in a program then do not be tempted to use grib_get_data to output the information to a file and then read the file in your program.  It is more efficient to use the appropriate Fortran 90, C or Python interface to GRIB API to access the data values directly.

Using the 'smart' copy

The structure of GRIB allows files to be simply copied and concatenated with the UNIX 'cat' command.  However, it's sometimes desirable to do something more sophisticated.  The grib_copy tool provides a way to copy only selected content from GRIB files.

  1. To copy all of the pressure level data at 1000 hPa from a number of separate files and store the output in a single file, try:

    Code Block
    grib_copy -w level=1000 ICMSHepc8* lev1000.grib


    Note

    Remember that you can check the output file with grib_ls to see the result !


  2. To do the same with only temperature field and to store in descending forecast step order, try:

    Code Block
    grib_copy -w level=1000,shortName=t -B "stepRange:i desc" ICMSHepc8* lev1000_ordered.grib


  3. Now let's gather all of the temperature fields and store in order of decreasing level and decreasing step:

    Code Block
    grib_copy -w shortName=t -B "level:i desc, stepRange:i desc" ICMSHepc8* all_t_ordered.grib


    Note

    It is important when using the "order by" (-B) option to specify the key type with, e.g., "level:i" to avoid unexpected (unwanted ?) results.  Remember that the key values are strings by default.  Usually, we want a numeric order.


  4. Key names can be used to specify the output file names.  Try:

    Code Block
    grib_copy ICMGGepc8* "[shortName]_[level].grib[edition]"

    This provides a very convenient way to filter GRIB messages into separate files.

Setting key/value pairs

The grib_set tool allows key values to be set or modified.  It is also able to make simple global changes to the data values themselves.

  1. Suppose we want to use the soil temperature on level 1 to provide a sea-surface temperature.  We can change the header information for this parameter and set it to sea-surface temperature

    Code Block
    grib_set -S -s shortName=sst -w shortName=stl1 ICMGGepc8+* sst.grib


    Note

    By default, all parameters are copied to the output file.  The 'strict' option (-S) writes only those messages that are changed to the output file.  To see the difference, try running grib_set again but without the -S option !


  2. This is still not encoded like a true sea-surface temperature.  To do this, we need also to change the typeOfLevel:

    Code Block
    grib_set -S -s shortName=sst,typeOfLevel=surface -w shortName=stl1 ICMGGepc8+* sst.grib


  3. An offset can be added to all data values:

    Code Block
    grib_set -S -s offsetValuesBy=-273.15 -w shortName=stl1 ICMGGepc8+* stl1_in_degC.grib


    Note

    Remember: use "grib_ls -n statistics" to make a quick check of the values in output file.


  4. The values can be multiplied by a constant factor:

    Code Block
    grib_set -S -s scaleValuesBy=0.102 -w shortName=z ICMGGepc8+* gh.grib


  5. Some values are read only and cannot be changed.  Try:

    Code Block
    grib_set -S -s scaleValuesBy=0.000001,units="kg cm**-2" -w shortName=tcw ICMGGepc8+000000 tcm_kgcm-2.grib

    The units key cannot be set as it is read only.  the only way to change the units key is by changing the parameter (either the shortName or paramId), assuming there is a suitable parameter defined to express Total column water in units of kg/cm2.

  6. All data values can be set to a constant

    Code Block
    grib_set -S -d 1.0 -w shortName=tcw ICMGGepc8+000000 out.grib


    Note

    The resulting file is small as it contains only the reference value. This is the method used to create the ifs_sample files used to provide 'template' GRIBs for model output.


  7. Key names can be used to specify the output file names.  This works in the same way as for grib_copy. Try:

    Code Block
    grib_set -S -s scaleValuesBy=0.102,shortName=gh -w shortName=z ICMGGepc8+* "[shortName]_[dataDate]_[dataTime]_[stepRange].grib"


Warning

grib_set is very powerful but it cannot be used to:

  • make transformations to the data representation
  • transform data from one grid representation to another
  • select sub-areas of data


Resources and further reading

  1. The WMO FM 92 GRIB Manuals can be obtained from www.wmo.int/pages/prog/www/WMOCodes.html
  2. The ECMWF GRIB API manual is available atat https://softwareconfluence.ecmwf.int/wiki/display/GRIB/Home
  3. ECMWF parameter database: http://oldapps.ecmwf.int/publicationscodes/manualsgrib/d/gribapi/paramparam-db/
  4. GRIB 1 keyshttp://oldapps.ecmwf.int/publicationscodes/manuals/d/gribapi/fm92grib/format/grib1/
  5. GRIB 2 keyshttp://oldapps.ecmwf.int/publicationscodes/manualsgrib/d/gribapiformat/fm92/grib2/
  6. Edition independent keys http://oldapps.ecmwf.int/publicationscodes/manualsgrib/dformat/gribapi/keys/ edition-independent/
  7. MARS specific keys http://apps.ecmwf.int/codes/grib/format/mars/
  8. The GRIB API Tools are documented at GRIB tools
  9. Details of the GRIB API Fortran 90 interface:Fortran package grib_api
  10. Example Python scripts and Fortran 90 and C programs can be found at GRIB API examples
  11. ECMWF GRIB API training course material: GRIB API: Library and Tools

...


  1. Excerpt Include
    Credits
    Credits
    nopaneltrue