At the WMO Commission for Basic Systems (CBS) ET-OWFPS Meeting on 22-24 October 2014 in Geneva, a draft document on standardized surface verification of deterministic NWP products was agreed. The document specifies the content of the future exchange of surface scores between global centres and is available here: Standardized Surface Verification for Deterministic NWP

ECMWF will be collecting and archiving the scores. Data will be exchanged in an ASCII-based format which is simple but flexible and is similar to the format used WMO exchange of upper-air scores.

Data format

Data file consists of score values and corresponding metadata in an ASCII format.

Every score value is described by the full set of key attributes, like its parameter, station id, month, step etc. Parameters describing one score value at one station are organised into a record. Each record corresponds to one score value. A record is a collection of pairs key=value separated by commas. A record spans one line. Value of the key which is not given in the current record is inherited from the previous record, except for the value parameter v which has to be present in each record.

Each record has the following format:

centre=centre, model=model_id, d=yyyymm, t=time, s=forecast_step, st=station_id, lat=latitude, lon=longitude, lam= model_grid_latitude , lom= model_grid_longitude , se=station_elevation, me=model_orography_elevation, par=parameter, sc=score, th=event_thresholds, n=sample_size, v=mean_value

If the value is not available the record either should not be given at all or the value should be set to NIL (v=nil); every record must contain the key v (value of key v is not inheritable from previous record).

Example

centre=ecmf,model=0001_fc,d=201410,t=0,s=24,st=11520,lat=50.02,lon=14.38,se=302,me=248,par=tp24,sc=me,ev=na,n=31,v=0.95
s=48,v=1.15
s=24,sc=ct,th=5,v=1,16,12,2
s=48,v=3/13/11/4

Values of keys

centre (4-characters string) is the WMO identifier of the originating centre (ammc, cwao, ecmf, edzw, egrr, kwbc, lfpw, rjtd, rksl, rums etc);
model_id (a string, not containing a comma or vertical bar) is free model identifier assigned by the originating centre (to distinguish between potentially different models provided by the centre);
yyyymm is the month of the mean, where yyyy is the year and mm is the month (01-12);
time is the validity time (in hours UTC) of the forecasts verified;
forecast_step is the length of the forecast (in hours);
station_id (a number) is the WMO ID of the observation station verifying the forecasts;
latitude is the latitude of the observation station verifying the forecasts;
longitude is the longitude of the observation station verifying the forecasts;
model_grid_latitude (non-mandatory) is the latitude of the model grid point used to extract the forecast at the observation location;
model_grid_longitude (non-mandatory) is the longitude of the model grid point used to extract the forecast at the observation location;
station_elevation is the elevation of the observation station above the mean sea level in meters;
model_orography_elevation is the elevation of the model orography at the observation location;
parameter is the verified model output parameter:

parameter	description	unit
t2m	air temperature at 2 meters above the model orography, corrected to the actual station elevation using the constant lapse rate 6.5K/1000m	K
td2m	dewpoint at 2 meters above the model orography	K
rh2m	relative humidity at 2 meters above the model orography	%
tp06	total precipitation accumulated over previous 6 hours	mm
tp24	total precipitation accumulated over previous 24 hours	mm
ff10m	wind speed at 10 meters above the model orography	m/s
dd10m	wind direction at 10 meters above the model orography	deg
tcc	total cloud cover	0-1 (convert to okta for contingency table)

score is the name of the verification score or statistic:

score

description

me

mean error (bias)

mae

mean absolute error

rmse

root mean square error

ct

contingency table values,
scanned from bottom up, then left to right

The rank of a contingency table is defined by number of values in the key event_thresholds
(rank_of_ct = len(event_thresholds)+1)

NB. Please note how those contingency tables are constructed.
For an event like "10m wind speed >15m/s" a corresponding contingency
table would traditionally be

	obs	non-obs
fc	H	F
non-fc	M	R

but following the definition of 2x2 contingency table on the left the table should be

	ob<15	15≤ob
fc<15	R	M
15≤fc	F	H

2x2	ob<th1	th1≤ob
fc<th1	A	B
th1≤fc	C	D

v=C/A/D/B

3x3	ob<th1	th1≤ob<th2	th2≤ob
ob<th1	A	B	C
th1≤ob<th2	D	E	F
th2≤ob	G	H	I

v=G/D/A/H/E/B/I/F/C

4x4	ob<th1	th1≤ob<th2	th2≤ob<th3	th3≤ob
ob<th1	A	B	C	D
th1≤ob<th2	E	F	G	H
th2≤ob<th3	I	J	K	L
th3≤ob	M	N	O	P

v=M/I/E/A/N/J/F/B/O/K/G/C/P/L/H/D

event_thresholds is a value or values of threshold(s) defining events for contingency tables; number of values in event_thresholds defines the rank of the contingency table; multiple values are separated by a forward slash /

event_threshold	description
15	threshold for a 2x2 contingency table, e.g. if par=ff10m this is an event of 10m wind speed
5/10/15	thresholds for a 4x4 contingency table for 10m wind speed

sample_size is number of observations used to compute the monthly mean at the given station;
mean_value is the value or values of the score mean computed based on the forecast initiating at time time UTC, verifying in the month yyyymm, for the forecast length forecast_step hours; in case of nxn contingency table these are the n² values delimited by forward slashes (see the score table above)

Procedures for exchange of data

Coming soon...