A page Page to detail some of the changes users should expect when sub-hourly data is supported in ecCodes.

The sub-hourly version of ecCodes adds support for data encoded in units below hours, namely in minutes (m) or seconds (s).

We will compare the before and after behaviour to illustrate what the sub-hourly version brings to ecCodes.

Decoding data (grib_ls & grib_get)

Let's take an instantaneous example with the following temporal properties:

======================   SECTION_4 ( length=1138, padding=0 )   ======================
...
18        indicatorOfUnitOfTimeRange = 0 [Minute (grib2/tables/28/4.4.table) ]
19-22     forecastTime = 45
...

We can see that this is a field at 45 minutes into the forecast.

Now let's compare the two versions:

support will arrive in ecCodes 2.34.0 in GRIB2 only.

See also the Behaviour of sub-hourly ecCodes with sub-hourly data from other centres, from DWD, HARMONIE and MET EIREANN.

Introduction

Full support for sub-hourly data in GRIB2 only will arrive along with ecCodes 2.34.0. This will bring sub-hourly support to ecCodes, which will give support for GRIB2 data encoded in units below hours, namely in minutes (m) or seconds (s).

The key points:

• Getting the step/stepRange will show the unit.
• Setting the step/stepRange will default to the largest possible canonical unit that can be used.
• Forcing a unit of your choice can be done using the stepunits key. This must be specified before the step itself.
• Hourly steps are currently kept without a unit to preserve compatibility with current behaviour.
  The plan in future will be to standardise this, and give the unit in all cases, e.g. 1 → 1h.
  Users can test this future behaviour by setting the environment variable "export ECCODES_GRIB_HOURLY_STEPS_WITH_UNITS=1".
  

The step units supported in GRIB2 can be found here.

We will compare the before and after behaviour to illustrate what the sub-hourly version brings to ecCodes in more detail below.

Decoding data (grib_ls & grib_get)

Instantaneous GRIB message

Let's take an instantaneous example with the following temporal properties:

======================   SECTION_4 ( length=1138, padding=0 )   ======================
...
18        indicatorOfUnitOfTimeRange = 0 [Minute (grib2/tables/28/4.4.table) ]
19-22     forecastTime = 45
...

We can see that this is a field at 45 minutes into the forecast.

Now let's compare the two versions:

You can see that the unit is added to the step, and that this is also the case within the MARS namespace.

Statistically processed GRIB message

Now let's take a statistically processed example with the following temporal properties:

======================   SECTION_4 ( length=61, padding=0 )    ======================
...
18        indicatorOfUnitOfTimeRange = 0 [Minute (grib2/tables/28/4.4.table) ]
19-22     forecastTime = 45
...
50        typeOfStatisticalProcessing = 2 [Maximum (grib2/tables/28/4.10.table) ]
...
52        indicatorOfUnitForTimeRange = 0 [Minute (grib2/tables/28/4.4.table) ]
53-56     lengthOfTimeRange = 15
...

We can see that this is a maximum starting at 45 minutes into the forecast, taken over a 15 minute interval, and thus ending at 60 minutes into the forecast.

Now let's compare the two versions:

You can see that the unit is added to the startStep/endStep/stepRange, and that within the MARS namespace, where the endStep is used for indexing, the unit is also added.

Encoding data (grib_set)

A key point in the way the sub-hourly encoding works is that:

Note that hourly steps are currently kept without a unit to preserve compatibility with current behaviour. The plan in future will be to unify this, and give, for example, "1h" in the above case.

This future behaviour can be activated by setting the environment variable "export ECCODES_GRIB_HOURLY_STEPS_WITH_UNITS=1".

Instantaneous GRIB message

Let's return to the instantaneous example we had above, and set the step to some different values to see the behaviour:

grib_get -p step in.grib2
0m

grib_set -s step=59m in out.grib2
grib_get -p step out.grib2
59m

grib_set -s step=60m in.grib2 out.grib2
grib_get -p step out.grib2
1

grib_set -s step=3600s in.grib2 out.grib2
grib_get -p step out.grib2
1

grib_set -s step=61m in.grib2 out.grib2
grib_get -p step out.grib2
61m

grib_set -s step=3660s in.grib2 out.grib2
grib_get -p step out.grib2
61m

You can see that the representation defaults to the largest possible unit in the above cases, with both 60m and 3600s mapping to 1 and 3660s mapping to 61m.

However, you may wish to fix the unit to prevent this conversion. This is performed by passing the stepunits key in addition to step. Please note that the stepunits key must be passed before the step key. Let's look at an example below:

grib_set -s stepunits=m,step=60 in.grib2 out.grib2
grib_get -p step out.grib2
60m

grib_set -s stepunits=s,step=3600 in.grib2 out.grib2
grib_get -p step out.grib2
3600s

grib_set -s stepunits=s,step=3660 in.grib2 out.grib2
grib_get -p step out.grib2
3660s

You can see that the representation has not changed to use the largest possible unit, with both 60m, 3600s and 3660s remaining as specified.

Let's take a look at the GRIB section 4 to clearly show this. We compare first without using the stepunits key and then with using the stepunits key.

Without stepunits key:

Statistically processed GRIB message

In a statistically processed example, we have the unit of the start of the interval, as well as the unit of the time interval over which the processing is performed. Therefore, in this case, we preserve the largest possible unit in which both the startStep and endStep can be represented.
Let's return to the statistically processed example we had above, and change the processing interval via stepRange to see this behaviour:

grib_get -p startStep,endStep,stepRange in.grib2
45m 60m 45m-60m

grib_set -s stepRange=1-75m in.grib2 out.grib2 # <-------- could equivalently specify 60m-75m , 60m-4500s , ...
grib_get -p startStep,endStep,stepRange out.grib2
60m 75m 60m-75m

grib_set -s stepRange=3600s-120m in.grib2 out.grib2 # <-------- could equivalently specify 1-2 , 60m-120m , 60m-7200s , ...
grib_get -p startStep,endStep,stepRange out.grib2
1 2 1-2

Again, you may wish to fix the unit to prevent this conversion. This is performed by passing the stepunits key in addition to the stepRange key. Please note that the stepunits key must be passed before the stepRange key. Let's look at an example below:

grib_get -p startStep,endStep,stepRange in.grib2
45m 60m 45m-60m

grib_set -s stepunits=m,stepRange=60-120 in.grib2 out.grib2
grib_get -p startStep,endStep,stepRange out.grib2
60m 120m 60m-120m  # <----------------------------- Has not changed to 1 2 1-2

grib_set -s stepunits=s,stepRange=3600-4500 in.grib2 out.grib2
grib_get -p startStep,endStep,stepRange out.grib2
3600s 4500s 3600s-4500s  # <----------------------------- Has not changed to 60m 75m 60m-75m

You can see that the representation has not changed to use the largest possible unit, with both 60m-120m and 3600s-4500s remaining as specified.

Let's take a look at the GRIB section 4 to clearly show this. We compare first without using the stepunits key and then with using the stepunits key.

Without stepunits key:

You can see that the unit is added to the startStep/endStep/stepRange, and that within the MARS namespace, where the endStep is used for indexing, the new key stepunits is added.

With stepunits key:

Awkward units

ecCodes supports a unified unit conversion helper function between GRIB1 and GRIB2. This is done by using the "-s" option within a grib_get.

With the unit now specified for all sub-hourly fields, this means a unit must be displayed when using the unit conversion helper function with units of "15m" and "30m". In these awkward cases we must specify a separator between the step or stepRange and its unit.

We use the separator "x", i.e. multiples of, which is intuitive and does not provoke issues if the step or stepRange is further processed with regex or splitting tools.

Now let's compare the two versions:

Since we must have that the absence of the unit means we have hourly data, we also apply this to all other awkward units, such as "3h" and "6h", as can be seen below: