IFS data selection information for conventional observations

Interpretation of data selection information.

The IFS monthly denylist/allowlist information is sent out via e-mail to provide specific details about data usage in the ECMWF data assimilation system. 

The IFS data usage decisions are not only determined by the data selection: There are also other data selection algorithms and a variety of quality control mechanisms in place.

Stations present in the exclusion list (the denylist) are not necessarily of poor quality. There are many reasons why stations may be excluded from assimilation for a period of time:

• Large differences between observed and model values: This may be due to a problem with either the model or the observations. Observations may be removed while investigations are carried out, or for a period of time until the situation has improved. These decisions are reviewed each month based on the most recent monitoring statistics. The EUMETNET Observations Programme routinely follow up on suspect stations (see below).
• New stations: These will be passively monitored (thus 'denylisted') for a certain period of time until their quality has been assessed and they can be actively used (assimilated) by the IFS data assimilation.
• Representativity: The model is unable to represent small-scale feature below the scale resolved by the global ECMWF NWP model. The data assimilation system cannot make good use of observations in places where the lack of model resolution creates large errors, for example in complex terrain, on coastlines and steep orography.
• TAC to BUFR migration: The transition from TAC to BUFR is at ECMWF managed through updates to the data selection. This is clearly indicated in the change document, i.e. denoted by subtype of the observations in the allowlist (former whitelist) sections. There are some complications due to this migration (in the data selection file some entries effectively say "use BUFR from this station"), see the overview at https://confluence.ecmwf.int/display/TCBUF/ .
• Some surface pressure observations are bias corrected at ECMWF.  If another NWP centre doesn't bias correct this station the data may be detrimental.

The data selection information shared in the monthly e-mails is tailored for use with the IFS global model and needs to be used with caution in other NWP systems, especially those with higher spatial resolution, where representativeness is much less of an issue than in the (relatively) lower-resolution global model. It should be noted that parts of the information is encoded in the form of allowlist, that is, listing stations to be used. Please see the User's Guide to Data Selection for more information on data selection. Table for codetypes and subtypes can be found here.

ECMWF aims to implement IFS data selection changes in the second week of each month. During the first week, analysts evaluate automated and manual proposals for changes and run model experiments to test the data selection change. Operational change requests are then submitted in the second week of the month (typically Monday or Tuesday). Occasionally operational IT constraints mean the change is postponed by a few days.

Every three months there is a synchronisation with WMO's OSCAR station database. This generally warrants a short data selection change proposal of its own.

If you would like to be added to the e-mail distribution of this data selection information please send an email to ifs_conventional_blacklist-request@lists.ecmwf.int  with the word subscribe or unsubscribe as Subject. Please note that this service is available to ECMWF Member and Cooperating States only.

Resolution of station issues

Data quality for in situ observations is routinely monitored by the EUMETNET Observations Programme. Their information should be relied upon for routine monitoring of station quality, identification of issues and issues resolution. Please refer to 'Observations Programme/Observation Monitoring' at the EUMETNET/EUCOS portal: https://portal.eumetnet.eu

Data selection change documents

Data selection files