The default behaviour for grib_compare without any option is to perform a bit by bit comparison of the two messages. If the messages are found to be bitwise different then grib_compare switches to a "key based" mode to find out which coded keys are different. To see how grib_compare works we first set the shortName=2d (2 metre dew point temperature) in the file regular_latlon_surface.grib1
> grib_set -s shortName=2d regular_latlon_surface.grib1 2d.grib1 |
> grib_compare regular_latlon_surface.grib1 2d.grib1 -- GRIB #1 -- shortName=2t paramId=167 stepRange=0 levelType=sfc level=0 packingType=grid_simple gridType=regular_ll -- long [indicatorOfParameter]: [167] != [168] |
> grib_compare -b indicatorOfParameter regular_latlon_surface.grib1 2d.grib1 |
Two grib messages can be very different because they have different edition, but they can contain the same identical information in the header and the same data. To see how grib_compare can help in comparing messages with different edition we do
> grib_set edition=2 reduced_gaussian_model_level.grib1 reduced_gaussian_model_level.grib2 |
> grib_compare reduced_gaussian_model_level.grib1 reduced_gaussian_model_level.grib2 -- GRIB #1 -- shortName=t paramId=130 stepRange=0 levelType=ml level=1 packingType=grid_simple gridType=reduced_gg -- long [totalLength]: [10908] != [10996] long [editionNumber]: [1] != [2] long [section1Length]: [52] != [21] [table2Version] not found in 2nd field [gridDefinition] not found in 2nd field [indicatorOfParameter] not found in 2nd field [indicatorOfTypeOfLevel] not found in 2nd field [yearOfCentury] not found in 2nd field [unitOfTimeRange] not found in 2nd field [P1] not found in 2nd field [P2] not found in 2nd field [numberIncludedInAverage] not found in 2nd field [numberMissingFromAveragesOrAccumulations] not found in 2nd field [centuryOfReferenceTimeOfData] not found in 2nd field [reservedNeedNotBePresent] not found in 2nd field [perturbationNumber] not found in 2nd field [numberOfForecastsInEnsemble] not found in 2nd field [padding_local1_1] not found in 2nd field long [section2Length]: [896] != [17] [pvlLocation] not found in 2nd field [dataRepresentationType] not found in 2nd field long [latitudeOfFirstGridPoint]: [87864] != [87863799] long [latitudeOfLastGridPoint]: [-87864] != [-87863799] long [longitudeOfLastGridPoint]: [357188] != [357187500] [padding_grid4_1] not found in 2nd field long [section4Length]: [9948] != [770] [dataFlag] not found in 2nd field |
> grib_compare -e reduced_gaussian_model_level.grib1 reduced_gaussian_model_level.grib2 -- GRIB #1 -- shortName=t paramId=130 stepRange=0 levelType=ml level=1 packingType=grid_simple gridType=reduced_gg -- string [param]: [130.128] != [130] |
> grib_compare -ve reduced_gaussian_model_level.grib1 reduced_gaussian_model_level.grib2 reduced_gaussian_model_level.grib2 comparing centre as string comparing paramId as long comparing units as string comparing name as string comparing shortName as string comparing typeOfLevel as string comparing level as long comparing pv as double (184 values) tolerance=0 using compare_double_absolute comparing bitmapPresent as long comparing latitudeOfFirstGridPointInDegrees as double (1 values) tolerance=0.0005 using compare_double_absolute comparing longitudeOfFirstGridPointInDegrees as double (1 values) tolerance=0.0005 using compare_double_absolute comparing latitudeOfLastGridPointInDegrees as double (1 values) tolerance=0.0005 using compare_double_absolute comparing longitudeOfLastGridPointInDegrees as double (1 values) tolerance=0.0005 using compare_double_absolute comparing iDirectionIncrementInDegrees is set to missing in both fields comparing N as long comparing iScansNegatively as long comparing jScansPositively as long comparing jPointsAreConsecutive as long comparing pl as long comparing gridType as string comparing packedValues as double (6114 values) tolerance=0 using compare_double_absolute comparing domain as string comparing levtype as string comparing levelist as long comparing date as long comparing time as long comparing step as long comparing param as string -- GRIB #1 -- shortName=t paramId=130 stepRange=0 levelType=ml level=1 packingType=grid_simple gridType=reduced_gg -- string [param]: [130.128] != [130] comparing class as string comparing type as string comparing stream as string comparing expver as string |
Some options are provided to compare only a set of keys in the messages. The option -H is used to compare only the headers coded in the message, it doesn't compare the data values. The option "-c key1:[i/d/s/n],key2:[i/d/s/n],... " can be used to compare a set of keys or namespaces. The letter after the colon is optional and it is used to force the type used in the comparison which is otherwise assumed to be the native type of the key. The possible types are:
set bitsPerValue=10;
set values={1,2.5,3,4,5,6,70};
write "first.grib1";
set values={1,2.5,5,4,5,6,70};
write "second.grib1";
|
> grib_compare -H first.grib1 second.grib1 |
> grib_compare -c data:n first.grib1 second.grib1 -- GRIB #1 -- shortName=t paramId=130 stepRange=0 levelType=ml level=1 packingType=grid_simple gridType=reduced_gg -- double [packedValues]: 1 out of 7 different max absolute diff. = 2.0000000000000000e+00, relative diff. = 0.4 max diff. element 2: 3.00000000000000000000e+00 5.00000000000000000000e+00 tolerance=0.0000000000000000e+00 packingError: [0.0625005] [0.0625005] values max= [70] [70] min= [1] [1] |
> grib_compare -A 2 -c data:n first.grib1 second.grib1 |
> grib_compare -R packedValues=0.4 -c data:n first.grib1 second.grib1 |
> grib_set -s changeDecimalPrecision=0 first.grib1 third.grib1 |
> grib_compare -P -c data:n first.grib1 third.grib1 |
> grib_compare -c data:n first.grib1 third.grib1 -- GRIB #1 -- shortName=t paramId=130 stepRange=0 levelType=ml level=1 packingType=grid_simple gridType=reduced_gg -- double [packedValues]: 1 out of 7 different max absolute diff. = 5.0000000000000000e-01, relative diff. = 0.166667 max diff. element 1: 2.50000000000000000000e+00 3.00000000000000000000e+00 tolerance=0.0000000000000000e+00 packingError: [0.0625005] [0.5] values max= [70] [70] min= [1] [1] |
When we already know that the fields are not numerically identical, but have similar statistical characteristics we can compare their statistics namespaces:
> grib_compare -c statistics:n first.grib1 third.grib1 -- GRIB #1 -- shortName=t paramId=130 stepRange=0 levelType=ml level=1 packingType=grid_simple gridType=reduced_gg -- double [avg]: [1.30714285714285711748e+01] != [1.31428571428571423496e+01] absolute diff. = 0.0714286, relative diff. = 0.00543478 tolerance=0 double [sd]: [2.32907531796090587761e+01] != [2.32589679873534969090e+01] absolute diff. = 0.0317852, relative diff. = 0.00136471 tolerance=0 double [skew]: [2.02295027950165895447e+00] != [2.02385673400705590197e+00] absolute diff. = 0.000906455, relative diff. = 0.000447885 tolerance=0 double [kurt]: [2.12697527593972246507e+00] != [2.12906658242618895827e+00] absolute diff. = 0.00209131, relative diff. = 0.000982264 tolerance=0 |
> grib_compare -A 0.052 -c statistics:n first.grib1 third.grib1 -- GRIB #1 -- shortName=t paramId=130 stepRange=0 levelType=ml level=1 packingType=grid_simple gridType=reduced_gg -- double [avg]: [1.30714285714285711748e+01] != [1.31428571428571423496e+01] absolute diff. = 0.0714286, relative diff. = 0.00543478 tolerance=0.052 |
When a file contains several fields and some keys are different, it is useful to have a summary report of the keys found different in the messages. This can be obtained with the option -f. We change few keys in a file:
> grib_set -w typeOfLevel=surface -s step=48 tigge_pf_ecmwf.grib2 out.grib2 |
> grib_compare -f tigge_pf_ecmwf.grib2 out.grib2 -- GRIB #9 -- shortName=skt paramId=235 stepRange=96 levelType=sfc level=0 packingType=grid_simple gridType=regular_ll -- long [forecastTime]: [96] != [48] -- GRIB #10 -- shortName=sd paramId=228141 stepRange=96 levelType=sfc level=0 packingType=grid_simple gridType=regular_ll -- long [forecastTime]: [96] != [48] -- GRIB #11 -- shortName=sf paramId=228144 stepRange=0-96 levelType=sfc level=0 packingType=grid_simple gridType=regular_ll -- long [dayOfEndOfOverallTimeInterval]: [26] != [24] long [lengthOfTimeRange]: [96] != [48] ... output deleted ## ERRORS SUMMARY ####### ## ## Summary of different key values ## forecastTime ( 3 different ) ## dayOfEndOfOverallTimeInterval ( 11 different ) ## lengthOfTimeRange ( 11 different ) ## ## 14 different messages out of 38 |
We can change the order of the messages in a file using grib_copy with the -B option:
> grib_copy -B typeOfLevel tigge_pf_ecmwf.grib2 out.grib2 |
> grib_compare -f tigge_pf_ecmwf.grib2 out.grib2 -- GRIB #1 -- shortName=10u paramId=165 stepRange=96 levelType=sfc level=10 packingType=grid_simple gridType=regular_ll -- long [discipline]: [0] != [2] long [totalLength]: [1555] != [990] long [parameterCategory]: [2] != [0] long [parameterNumber]: [2] != [22] long [scaledValueOfFirstFixedSurface]: [10] != [0] long [typeOfSecondFixedSurface]: [255] != [106] scaleFactorOfSecondFixedSurface is set to missing in 1st field is not missing in 2nd field scaledValueOfSecondFixedSurface is set to missing in 1st field is not missing in 2nd field long [numberOfValues]: [684] != [239] double [referenceValue]: [-1.57229328155517578125e+01] != [4.15843811035156250000e+01] absolute diff. = 57.3073, relative diff. = 1.3781 tolerance=3.8147e-06 long [binaryScaleFactor]: [-10] != [-15] long [bitsPerValue]: [16] != [24] long [section6Length]: [6] != [92] long [bitMapIndicator]: [255] != [0] long [section7Length]: [1373] != [722] Different size for "codedValues" [684] [239] ... very long output |
> grib_compare -r tigge_pf_ecmwf.grib2 out.grib2 |
To copy only the pressure levels from a file
> grib_copy -w levtype=pl ../data/tigge_pf_ecmwf.grib2 out.grib |
To copy only the fields that are not on pressure levels from a file
> grib_copy -w levtype!=pl ../data/tigge_pf_ecmwf.grib2 out.grib |
To copy only the first three fields from a file
> grib_copy -w count=1/2/3 ../data/tigge_pf_ecmwf.grib2 out.grib |
A grib_file with multi field messages can be converted in single field messages with a simple grib_copy.
> grib_copy multi.grib simple.grib |
Use the square brackets to insert the value of a key in the name of the output file (This is a good way to split a large GRIB file)
> grib_copy in.grib 'out_[shortName].grib' |
To copy fields whose typeOfLevel is either 'surface' or 'meanSea'
> grib_copy -w typeOfLevel=surface/meanSea orig.grib out.grib |
To copy selected fields and apply sorting (sorted by level in ascending order)
> grib_copy -w typeOfLevel=heightAboveGround -B'level:i asc' tigge_af_ecmwf.grib2 out.grib |
To dump in a WMO documentation style with hexadecimal octet values (-H).
> grib_dump -OH ../data/reduced_gaussian_model_level.grib1 |
To add key aliases and type information.
> grib_dump -OtaH ../data/reduced_gaussian_model_level.grib1 |
To obtain all the key names (computed keys included) available in a grib file.
> grib_dump -D ../data/regular_latlon_surface.grib1 |
The grib_filter processes sequentially all grib messages contained in the input files and applies the rules to each one of them. Input messages can be written to the output by using the "write" statement. The write statement can be parameterised so that output is sent to multiple files depending on key values used in the output file name. If we write a rules_file containing the only statement:
write "../data/split/[centre]_[date]_[dataType]_[levelType].grib[editionNumber]"; |
> grib_filter rules_file ../data/tigge_pf_ecmwf.grib2 > ls ../data/split ecmf_20060619_pf_sfc.grib2 ecmf_20060630_pf_sfc.grib2 ecmf_20070122_pf_pl.grib2 ecmf_20070122_pf_pt.grib2 ecmf_20070122_pf_pv.grib2 ecmf_20070122_pf_sfc.grib2 |
The key values in the file name can also be obtained in a different format by indicating explicitly the type required after a colon.
write "../data/split/[centre:i]_[date]_[dataType:i]_[levelType].grib[editionNumber]"; |
> grib_filter rules_file ../data/tigge_pf_ecmwf.grib2 > ls ../data/split 98_20060619_4_sfc.grib2 98_20060630_4_sfc.grib2 98_20070122_4_pl.grib2 98_20070122_4_pt.grib2 98_20070122_4_pv.grib2 98_20070122_4_sfc.grib2 |
Other statements are allowed in the grib_filter syntax:
# Temperature
if ( level == 850 && indicatorOfParameter == 11 ) {
print "found indicatorOfParameter=[indicatorOfParameter] level=[level] date=[date]";
transient oldtype = type ;
set identificationOfOriginatingGeneratingSubCentre=98;
set gribTablesVersionNo = 128;
set indicatorOfParameter = 130;
set localDefinitionNumber=1;
set marsClass="od";
set marsStream="kwbc";
# Negatively/Positively Perturbed Forecast
if ( oldtype == 2 || oldtype == 3 ) {
set marsType="pf";
set experimentVersionNumber="4001";
}
# Control Forecast
if ( oldtype == 1 ) {
set marsType="cf";
set experimentVersionNumber="0001";
}
set numberOfForecastsInEnsemble=11;
write;
print "indicatorOfParameter=[indicatorOfParameter] level=[level] date=[date]";
print;
}
|
Here is an example of an IF statement comparing a key with a string. Note you have to use the "is" keyword for strings and not "==", and to negate you add the "!" before the whole condition:
# Select Geopotential Height messages which are not on a Reduced Gaussian Grid
if (shortName is "gh" && !(gridType is "reduced_gg" )) {
set step = 72;
}
|
The switch statement is an enhanced version of the if statement. Its syntax is the following:
switch (key1) {
case val1:
# block of rules;
case val2:
# block of rules;
default:
# block of rules
}
|
processing paramId=[paramId] [shortName] [stepType]
switch (shortName) {
case tp :
set stepType=accum;
case 10u :
set typeOfLevel=surface;
default:
}
|
grib_get fails if a key is not found.
> grib_get -p gribname ../data/tigge_pf_ecmwf.grib2 |
To get the step of the first GRIB message in a file:
> grib_get -w count=1 -p step ../data/tigge_pf_ecmwf.grib2 |
To get a latitude, longitude, value list, skipping the missing values(=9999)
> grib_get_data ../data/reduced_gaussian_model_level.grib2 |
If you want to define your missing value=1111 and to print the string 'missing' in place of it
> grib_get_data -m 1111:missing ../data/reduced_gaussian_model_level.grib2 |
If you want to print the value of other keys with the data value list
> grib_get_data -p centre,level,step ../data/reduced_gaussian_model_level.grib2 |
By default grib_index_build will index on the MARS keys.
> grib_index_build ../data/reduced*.grib1 ../data/regular*.grib1 ../data/reduced*.grib2 |
To specify a custom list of keys to index on, use the -k option.
> grib_index_build -k paramId,dataDate ../data/reduced*.grib1 ../data/regular*.grib1 ../data/reduced*.grib2 |
Without options a default list of keys is printed. The default list is different depending on the type of grib message.
> grib_ls ../data/reduced*.grib1 ../data/regular*.grib1 ../data/reduced*.grib2 |
To print offset and count number in file use the keys offset and count Also the total count in a set of files is available as countTotal
> grib_ls -p offset,count,countTotal ../data/reduced*.grib1 |
To list only a subset of messages use the -w (where option). Only the pressure levels are listed with the following line.
> grib_ls -w levelType=pl ../tigge_pf_ecmwf.grib2 |
All the grib messages not on pressure levels are listed as follows:
> grib_ls -w levelType!=pl ../tigge_pf_ecmwf.grib2 |
To get the closest grid point to a latitude/longitude.
> grib_ls -l 51.46,-1.33,1 -p paramId,name ../data/reduced_gaussian_surface.grib2 ../data/reduced_gaussian_surface.grib2 paramId shortName value 167 2t 282.002 1 of 1 messages in ../data/reduced_gaussian_surface.grib2 1 of 1 total messages in 1 files Input Point: latitude=51.46 longitude=-1.33 Grid Point chosen #3 index=749 latitude=51.63 longitude=0.00 distance=93.81 (Km) Other grid Points <ul><li>1 - index=845 latitude=48.84 longitude=0.00 distance=306.86 (Km) </li><li>2 - index=944 latitude=48.84 longitude=356.40 distance=333.66 (Km) </li><li>3 - index=749 latitude=51.63 longitude=0.00 distance=93.81 (Km) </li><li>4 - index=844 latitude=51.63 longitude=356.25 distance=168.37 (Km) |
To get a list ordered by the 'level' key (ascending order).
> grib_ls -B 'level:i asc' tigge_af_ecmwf.grib2 |
To set productDefinitionTemplateNumber=2 only for the fields with productDefinitionTemplateNumber=11
> grib_set -s productDefinitionTemplateNumber=2 -w productDefinitionTemplateNumber=11 ../data/tigge_pf_ecmwf.grib2 out.grib2 |
To set productDefinitionTemplateNumber=2 only for the fields for which productDefinitionTemplateNumber is not equal to 11
> grib_set -s productDefinitionTemplateNumber=2 -w productDefinitionTemplateNumber!=11 tigge_pf_ecmwf.grib2 out.grib2 |
When a key is not used all the bits of its value should be set to 1 to indicate that it is missing. Since the length (number of octet) is different from a key to another, the value that we have to code for missing keys is not unique. To give an easy way to set a key to missing a string "missing" or "MISSING" is accepted by grib_set as follows:
> grib_set -s scaleFactorOfFirstFixedSurface=missing,scaledValueOfFirstFixedSurface=MISSING ../data/regular_latlon_surface.grib2 out.grib2 |
To set scaleFactorOfSecondFixedSurface to missing only for the fields for which scaleFactorOfSecondFixedSurface is not missing:
> grib_set -s scaleFactorOfSecondFixedSurface=missing -w scaleFactorOfSecondFixedSurface!=missing tigge_pf_ecmwf.grib2 out.grib2 |
It's possible to produce a grib edition 2 file from a grib edition 1 just changing the edition number with grib_set. At this stage of development all the geography parameters, level and time information is correctly translated, for the product definition extra set calls must be done. To do this properly, \ref grib_filter is suggested.
grib_set -s edition=2 ../data/reduced_gaussian_pressure_level.grib1 |
With grib edition 2 is possible to compress data using the jpeg algorithm. To change packing algorithm from grid_simple (simple packing) to grid_jpeg (jpeg2000 packing):
> grib_set -s packingType=grid_jpeg ../data/regular_gaussian_model_level.grib2 out.grib2 |
It's possible to ask ecCodes to calculate the number of bits per value needed to pack a given field with a fixed number of decimal digits of precision. For example if we want to pack a temperature expressed in Kelvin with 1 digits of precision after the decimal point we can set changeDecimalPrecision=1
> grib_set -s changeDecimalPrecision=1 ../data/regular_latlon_surface.grib2 ../data/out.grib2 |
Produce a NetCDF file from grib edition 1
> grib_to_netcdf -o output.nc input.grib1 |
If your grib file has analysis and 6-hour forecast, then ignore keys 'type' and 'step'. Thus type=an/fc and step=00/06 will not be considered as netcdf dimensions.
> grib_to_netcdf -I type,step -o output.nc input.grib |
Do not use time of validity. If time of validity is used, it means the 1D time coordinate is considered as date+time+step, otherwise 3 different dimensions are created. The default behaviour is to use the time of validity.
> grib_to_netcdf -T -o output.nc input.grib |
Produce NetCDF with data type of FLOAT (32bit floating point, for higher precision). Note these types were chosen to provide a reasonably wide range of trade-offs between data precision and number of bits required for each value
> grib_to_netcdf -D NC_FLOAT -o output.nc input.grib |