...

Issued by: DWD/A. Mayer, A. C. Mikalsen, H. Konrad, B. Würzler and KNMI/J. F. Meirink

Date: 28/03/2019

Ref: C3S_D312b_Lot1.0.4.8_201903_Updated_KPIs_v1.0

...

List of datasets covered by this document

...

Scope of the document

In this document, the general quality assessment approach for C3S_312b_Lot1 is outlined. This includes a description of common validation methods, including metrics and terminology, as well as a practical approach for the derivation of Key Performance Indicators (KPIs) and their evaluation for the Interim Climate Data Records (ICDRs). This approach introduces an objective quality measure across the different datasets provided within the project.

...

The data records provided within C3S_312b_Lot1 need to be quality assured before they are made publicly available. In order to harmonize the quality assurance of the different data records, a common approach needs to be agreed upon. In this document, common validation methods and metrics are introduced. Moreover, a concrete method for the derivation and evaluation of KPIs for ICDRs is proposed. This method involves an assessment whether ICDRs are consistent with their corresponding Thematic Climate Data Records (TCDRsCDRs) in terms of deviations from a reference data record. The document also includes a revision of the KPIs for the TCDRs CDRs that have been provided in the tender for this project [D2].

...

This document focuses on the fourth quality indicator: validation. Firstly, it describes a general approach for product validation, including the adopted metrics and terminology. Secondly, it proposes a guideline for the establishment of KPIs for the ICDRs, based on the corresponding TCDRsCDRs, and a concrete method for the evaluation of these KPIs on a frequent (quarterly) basis.

...

For these reasons, we present here a new approach to monitor ICDR chains newly generated for the Climate Data Store (CDS). It is a general strategy which aims, for the first time in this project, to provide a unified objective evaluation of the different ICDRs. Given that very different opinions on “the best” approach may exist, the aim here is to propose a practical concept and common strategy for all ICDRs delivered to the CDS. The primary aim is to allow shortcomings in the ICDRs to be detected with respect to the TCDR CDR performance for which the KPIs were initially designed.

...

Our idea is to define KPIs to perform a unified, quick, but reliable quality check of the ICDRs. As ICDRs are released every 3 months, performing a comprehensive, in-depth validation with each release is not feasible. Therefore, we rather propose to compare the performance of new ICDRs with a suitable reference dataset with the performance of the (deeply validated) TCDRs CDRs against the reference dataset.

If the ICDR differs from the reference dataset in the same way the TCDR CDR does, we conclude that the ICDR is of the same quality as the TCDR CDR (i.e. the test indicates that the key performance of the ICDR is good/positive). But if the ICDR, statistically speaking, differs unusually widely from the reference dataset, we infer that we need to have a deeper look into the ICDR to identify possible errors (i.e. the test indicates that the key performance of our ICDR is bad/negative). To decide whether the ICDR differs, in a statistical sense, unusually widely from the reference dataset, we perform a statistical hypothesis test, a binomial test. Therefore, we predefine a reference Performance Target PT, i.e. a well-chosen range where most of the TCDR differences (say 95%) are found. Then, we check how many values of the ICDR lie within this range. We get three possible outcomes:

...

The approach to verify the KPI accuracy is to first identify a suitable reference dataset (subscript r below), against which differences of (near-) global mean values can be computed for the tested dataset (i.e. the TCDRCDR/ICDR that is delivered to the CDS; subscript t below). The chosen reference dataset should have as much spatial and temporal overlap with the tested dataset as possible. As most of our data have global coverage, a reference dataset with global coverage would consequently be preferential. Any choice, however, will most probably be a subjective one in the end. If we feel that there is no suitable reference data set available (in time), we will directly take the actual TCDR CDR as reference data set, although this implicitly assumes that the true time series is stable in terms of mean value and variability.

3.2.2

...

CDR: Basis for the derivation of performance targets

The TCDR CDR will be used to derive PTs for the ICDR as follows. We compute spatially averaged (preferentially global mean) values of both the tested dataset,

...

has a strong seasonal cycle, it is advised to de-seasonalise it in a further step.

If the actual TCDR CDR is taken as reference data set, we define its long-term mean as the reference, i.e. 

...

The ICDR is an extension of an existing TCDR CDR and will be assessed on the basis of the TCDR CDR distribution.

As for the TCDR, the differences 

...

 to distinguish it from the TCDRCDR), where 

y_{i}

 is either the global mean of the reference dataset for that time slice or the long-term global mean of the TCDRCDR. The same reference dataset needs to be used, or otherwise a significant deviation of TCDR CDR and ICDR biases would be expected simply due to the different references. The KPI check is carried out by verifying that 95 % of the values in the difference time series 

...

. This procedure allows monitoring the quality of the ICDR with respect to the TCDR CDR to a certain degree.

The proposal [D1] states that “[if] the KPI is negative, then the project team will assess the data product in more detail investigating possible causes for the quality deterioration.” The above procedure does not involve an indication of good data by KPI > 0 and bad data by KPI < 0 anymore. Consequently, we define a new criterion for the need of detailed assessment in the following.

...

1. 
   

   Mathinline
   T_{lower} \leq d_{i} \leq T_{upper} \quad \text{('successful') with a probability of } p_{0}

   
   

2. 
   

   Mathinline
   T_{lower} > d_{i} \quad \text{or} \quad d_{i} > T_{upper} \quad \text{with a probability of}\ 1 - p_{0}

   i.e. the probability p_n,k to draw k successful (option 1) samples in a total of n samples ( d₁, …, d_n ) is given by the binomial distribution:

   Mathdisplay
   p_{n,k} = \tbinom{n}{k} p_{0}^k (1-p_{0})^{n-k}, \quad (7)

   
   

Given k 'successful' ICDR difference values in a total of n, we test the hypothesis that the probability for one single sample of the ICDR difference value to fall into the 95% interval of the TCDR CDR is also 95% or higher (null hypothesis), i.e.

...

, i.e. the ICDR bias deviates significantly from the TCDR CDR bias.

By defining a significance level of 5%, α=0.05, and computing the cumulative probability 

...

• 
  

  Mathinline
  p_{n,k}^{cum} < \alpha: p_{0} < 0.95, \quad \text{i.e. reject null hypothesis; the ICDR bias deviates from the TCDRCDR bias;} \textbf{ the ICDR needs to be assessed in detail.}

  
  

• 
  

  Mathinline
  p_{n,k}^{cum} \geq \alpha: p_{0} \geq 0.95, \quad \text{null hypothesis verified; the ICDR bias performs at least equally well as the TCDRCDR bias;} \textbf{ no further assessment of the ICDR is required at this stage.}

  
  

In python, for example, 

p_{n,k}^{cum}

can be computed by scipy.stats.binom_test(k, n, p₀, alternative='less'). Matlab and R have similar commands on offer.

...

 based on the TCDR is indicated by thin black lines (not to be confused with the thick black lines indicating the periods for the TCDR CDR and the ICDR). The values corresponding to this interval are

...

. Panel B shows that k=9 of the n=10 ICDR difference values are successful, i.e. fall inside the 95% interval of the TCDRCDR.

...

 so the ICDR bias complies with the TCDR CDR bias and no action is required. Note that

...

range to pass the binomial test with a significance level of 95% (see e.g. http://www.fuellenbach-online.de/fh/pdf/binomialverteilung.pdf, pp. 2-–6).

3.3 Concluding remarks

We drop the initially foreseen KPI stability for ICDR checks completely since the stability itself, as defined in Eq. (5), is already included in existing validation reports for brokered TCDRsCDRs, and will be included in non-brokered datasets in the C3S validation documents. The main reason for doing so is that the ICDR periods are expected to be too short for detecting trends in noisy (or rather highly variable) data. The above approach for the KPI accuracy/bias will also enable us to detect a significant trend in the ICDR bias, because eventually the ICDR bias will deviate significantly from the TCDR CDR bias if it grows consistently smaller or larger over time. Consequently, we propose to investigate possible trends in the bias once the binomial test for KPI accuracy/bias indicates the need for action.

...

If an ICDR bias deviates significantly from the TCDR CDR bias for comprehensible and documentable reasons, the criterion of 95% of all ICDR difference values falling inside the 95% interval from the TCDR CDR or the significance level might have to be relaxed.

The usage of the percentiles in the TCDR CDR difference values implies that – in contrast to the concept of 'accuracy' – the optimal situation for the ICDR is not a near-zero bias, but a bias close to the mean/median bias of the TCDRCDR. We think that this makes sense as zero bias in the ICDR could be problematic if the TCDR CDR bias is clearly off-zero.

Any ICDR that is not simply the extension of a TCDR CDR would obviously be treated differently than proposed here.

In addition, we are aware of possible data quality variations or missing data within the ICDRs since accuracy and completeness of time-averaged satellite data can differ due to instrumental, transmission or system errors. More precisely, for monthly means the number of processed orbits per month determines the data quality and whether there is missing data for a whole month or for certain parts of the globe. Please note that we will not do any previewing of the data set quality and completeness before the KPI evaluation. Any further data analysis or deeper gap analysis is only performed in the case of a rejection of the performance targets.

4.

...

Revision and statement of the

...

KPIs for CDRs

So far, C3S_312b_Lot1 delivered several brokered TCDRs to the CDS. In advance, these data sets have been selected also under the condition that their performance is compliant with GCOS requirements or close to them. In the following, we report on their data quality based on the two Key Performance Indicators “accuracy” and “stability”, as prescribed in the Contractors Tender [D1]. As we became aware of cases where the targets for the KPIs were not well defined or wrongly cited, we critically reviewed and updated these values. Since all brokered data sets have been produced outside C3S_312b_Lot1 and consequently cannot be altered, we mostly took the original data set requirements defined in the respective project, in which they were produced, as required KPI values, if available. For brokered data sets from CM SAF, for instance, we provide original CM SAF requirements as required KPI values. For those data sets which will be produced within C3S_312b_Lot1 or which will be brokered at a later stage within C3S_312b_Lot1, the required KPI values will be specified in the course of the project.

4.1 Revision of KPIs for TCDRs

The performance targets for accuracy and stability have now been specified according to Table 2, which contains, for all datasets already brokered to the CDS, properly revised entries of the table given in Annex E to the Contractors Tender [D2].

4.2 Statement on the KPIs for CDRs

In the following, we give an account of the quality of the so far provided TCDRs with respect to the KPIs provided in Table 2. Since these TCDRs are brokered products, we mostly refer to validation activities carried out in previous projects. For the detailed evaluation, see the respective validation reports.

4.2.1 Cloud properties CDR brokered from CM SAF CLARA-A2

All four cloud products fulfill the CM SAF target requirements for accuracy and stability (which were, for CLARA-A2, adopted as KPI values within C3S_312b_Lot1) with respect to at least one qualified reference data set. CLARA-A2 was compared to synoptic observations (SYNOP data), observations from the CALIOP lidar onboard the CALIPSO satellite, the PATMOS-x (short for 'AVHRR Pathfinder Atmospheres-Extended') data set, observations from the International Satellite Cloud Climatology Project (ISCCP), measurements from the Moderate Resolution Imaging Spectroradiometer (MODIS), and the microwave-based liquid water path climatology from the University of Wisconsin. For more information see the CM SAF validation report. It encompasses in-depth validation of level-2b (instantaneous) and level 3 (monthly mean) data. The level-3 daily data were not tested explicitly, but the mix of level-2 (instantaneous) and level-3 (monthly data) studies provides enough information to ensure their quality.

For further details, see the Validation Report CM SAF Cloud, albedo, radiation data record, AVHRR-based, Edition 2 (CLARA-A2), Cloud Products, Ref: SAF/CM/DWD/VAL/GAC/CLD, available at https://www.cmsaf.eu/SharedDocs/Literatur/document/2016/saf_cm_smhi_val_gac_cld_2_3_pdf.

4.2.2 Precipitation CDR brokered from GPCP

The GPCP monthly v2.3 precipitation rates fulfil both the accuracy target (0.3 mm/d) and the stability target (0.034 mm/d/decade) when compared to the TMPA 3B43 v7 monthly product. Spatial averages in this case are computed covering the TRMM/TMPA latitudinal window between 50°S and 50°N.

The GPCP daily v1.3 precipitation rates fulfils the stability target (0.034 mm/d/decade) when compared to the TMPA 3B42 v7 daily product, again in the TRMM/TMPA window, see above. However, the accuracy target of 0.3 mm/d is violated on 7.5% of all available dates. Higher temporal and spatial variability can be expected for a dataset at higher resolution, both temporally and spatially, as is the case for GPCP daily v1.3 vs. GPCP monthly v2.3. Consequently, it can be expected that the same target (0.3 mm/d) will not be equally applicable to both datasets. The targets were initially formulated based on the HOAPS v4.0 validation report which does not include a daily product either. In this sense, we recommend to accept this violation in 7.5% of all cases as the expected behavior of a high-resolution dataset.

For further details, see the Validation Report CM SAF SSM/I and SSMIS products, HOAPS version 4.0, Ref: SAF/CM/DWD/VAL/HOAPS, available at https://www.cmsaf.eu/SharedDocs/Literatur/document/2017/saf_cm_dwd_val_hoaps4_1_2_pdf.

4.2.3 Surface Radiation Budget CDR brokered from CM SAF CLARA-A2

The predefined requirements for the surface incoming shortwave radiation are given in CM SAF Product Requirement Document (PRD), Annex A. The validation of the surface radiation datasets was conducted against surface measurements from the Baseline Surface Radiation Network (BSRN) [Ohmura et al., 1998].

All products in the brokered CLARA-A2 dataset fulfil the updated GCOS requirements regarding the horizontal and temporal resolution. The GCOS requirements for accuracy and stability are formulated for the global mean and are not relevant for individual reference measurement networks (e.g. BSRN). Besides this, there also exists the general problem with comparison of area-to-point measurements.

The KPI values for accuracy have been selected as following: SIS monthly means – 10 W/m² that corresponds to CM SAF target accuracy, SIS daily means – 20 W/m² that corresponds to CM SAF threshold accuracy, SDL monthly means – 10 W/m² that corresponds to CM SAF target accuracy and SOL monthly means – 10 W/m² that corresponds to CM SAF target accuracy. These targets are met by all datasets and the KPIs are fulfilled. Considering the uncertainty of the surface observations of 5 W/m², less than 5% of the SIS monthly means exceed the target requirements. For the SIS daily means about 20% exceed the threshold accuracy. This number can be explained by higher temporal resolution and thus less random biases averaging out. For the SDL monthly means less than 14% exceed the target accuracy and for the SOL monthly means less than 32% exceed the target accuracy.

Stability requirements for the Surface Radiation Budget TCDR have been defined as follows: SIS monthly means - 1.5 W/(m² dec), SIS daily means – 1.5 W/(m² dec), SDL and SOL monthly means – 0.5 W/(m² dec). Within the range of uncertainty the CLARA-A2 dataset agrees with the surface reference data. Achieved stability for the SIS monthly means (after 1994) is less than 1 W/(m² dec), achieved stability for the SDL and SOL monthly means is less than 1 W/(m² dec).

For further details, see the Validation Report CM SAF Cloud, Albedo, Radiation dataset, AVHRR-based, Edition 2 (CLARA-A2), Surface Radiation Products, Code: SAF/CM/DWD/VAL/GAC/RAD

4.1 Updated KPI values for the ICDRs

For steady quality control of our new ICDR chains, we will apply the evaluation approach outlined in Chapter 3 above. Therefore, we set the KPI values according to Table 2 which contains entries for all ICDRs for which the respective TCDRs have already been brokered. Missing entries relating to ICDRs, whose TCDRs have not been brokered yet will be given at a later stage. They will serve as lower and upper bounds for the binomial test as they correspond to the 2.5^th and 97.5^th percentiles mentioned in Chapter 3. The last column of Table 2 briefly describes important data set specific aspects of the analysis, e. g. the reference data set used. We will regularly give account of our test results in the Quarterly Reports.

4.2 Exemplary application to the first delivery of the GPCP ICDR

As a non-synthetic example, we evaluate the KPI accuracy for the first delivery of the GPCP precipitation ICDR.

4.2.1 GPCP monthly v2.3

The 2.5- and 97.5-quantiles of the spatially averaged differences of precipitation for GPCP monthly v2.3 TCDR, covering the time period Jan 1979 to Dec 2017, are -0.175 mm/d and +0.189 mm/d, respectively (see Table 2). The respective reference is TMPA 3B43 v7, which is only available between 50°S and 50°N, and from Jan 1998. Consequently, all relevant values are obtained by averaging over this spatial TMPA window from Jan 1998.

The first delivery of the ICDR comprises Jan 2018 to Dec 2018, i.e. 12 realizations of monthly means. For all of these, the spatially averaged precipitation differences between GPCP and the reference product fall inside the 2.5- and 97.5-quantiles found in the monthly TCDR ensemble. It is thus

p_{n,k}^{cum}=1 \geq 0.05

, so that we conclude that the ICDR at this stage is of at least the same quality as the TCDR and therefore passes the described test.

4.2.2 GPCP daily v1.3

The GPCP daily v1.3 TCDR covers the period 1st Oct 1996 to 31st Dec 2017. Our reference here is TMPA 3B42 v7, which is, like its monthly realization (see above), available from Jan 1998 between 50°S and 50°N. The respective 2.5- and 97.5-quantiles of spatially averaged differences are -0.332 mm/d and 0.343 mm/d, respectively (see Table 2).

The first delivery of the ICDR comprises the same period as for the monthly product, 1st Jan 2018 to 31st Dec 2018, i.e. 365 realizations. As the TMPA 3B42 reference data set is yet missing for one day (31st Dec 2018), 364 realizations remain. For 361 of these, the spatially averaged precipitation differences between GPCP and the reference product fall inside the 2.5- and 97.5-quantiles found in the daily TCDR ensemble. It is thus

p_{n,k}^{cum} \approx 0.99 \geq 0.05

, so that we conclude here, too, that the ICDR quality is sufficient. (Only a number of 338 or less would have indicated that the ICDR's quality is lower than the TCDR's in this context.)

5. Revision and statement of the KPIs for TCDRs

So far, C3S_312b_Lot1 delivered several brokered TCDRs to the CDS. In advance, these data sets have been selected also under the condition that their performance is compliant with GCOS requirements or close to them. In the following, we report on their data quality based on the two Key Performance Indicators “accuracy” and “stability”, as prescribed in the Contractors Tender [D1]. As we became aware of cases where the targets for the KPIs were not well defined or wrongly cited, we critically reviewed and updated these values. Since all brokered data sets have been produced outside C3S_312b_Lot1 and consequently cannot be altered, we mostly took the original data set requirements defined in the respective project, in which they were produced, as required KPI values, if available. For brokered data sets from CM SAF, for instance, we provide original CM SAF requirements as required KPI values. For those data sets which will be produced within C3S_312b_Lot1 or which will be brokered at a later stage within C3S_312b_Lot1, the required KPI values will be specified in the course of the project.

5.1 Revision of KPIs for TCDRs

The performance targets for accuracy and stability have now been specified according to Table 3, which contains, for all datasets already brokered to the CDS, properly revised entries of the table given in Annex E to the Contractors Tender [D2].

5.2 Statement on the KPIs for TCDRs

In the following, we give an account of the quality of the so far provided TCDRs with respect to the KPIs provided in Table 3. Since these TCDRs are brokered products, we mostly refer to validation activities carried out in previous projects. For the detailed evaluation, see the respective validation reports.

5.2.1 Cloud properties TCDR brokered from CM SAF CLARA-A2 (D3.3.16-v1.0):

All four cloud products fulfill the CM SAF target requirements for accuracy and stability (which were, for CLARA-A2, adopted as KPI values within C3S_312b_Lot1) with respect to at least one qualified reference data set. CLARA-A2 was compared to synoptic observations (SYNOP data), observations from the CALIOP lidar onboard the CALIPSO satellite, the PATMOS-x (short for 'AVHRR Pathfinder Atmospheres-Extended') data set, observations from the International Satellite Cloud Climatology Project (ISCCP), measurements from the Moderate Resolution Imaging Spectroradiometer (MODIS), and the microwave-based liquid water path climatology from the University of Wisconsin. For more information see the CM SAF validation report. It encompasses in-depth validation of level-2b (instantaneous) and level 3 (monthly mean) data. The level-3 daily data were not tested explicitly, but the mix of level-2 (instantaneous) and level-3 (monthly data) studies provides enough information to ensure their quality.

For further details, see the Validation Report CM SAF Cloud, albedo, radiation data record, AVHRR-based, Edition 2 (CLARA-A2), Cloud Products, Ref: SAF/CM/DWD/VAL/GAC/CLD, available at https://www.cmsaf.eu/SharedDocs/Literatur/document/2016/saf_cm_smhi_val_gac_cld_2_3_pdf.

5.2.2 Precipitation TCDR brokered from GPCP (D3.3.1-v1.0):

The GPCP monthly v2.3 precipitation rates fulfil both the accuracy target (0.3 mm/d) and the stability target (0.034 mm/d/decade) when compared to the TMPA 3B43 v7 monthly product. Spatial averages in this case are computed covering the TRMM/TMPA latitudinal window between 50°S and 50°N.
The GPCP daily v1.3 precipitation rates fulfils the stability target (0.034 mm/d/decade) when compared to the TMPA 3B42 v7 daily product, again in the TRMM/TMPA window, see above. However, the accuracy target of 0.3 mm/d is violated on 7.5% of all available dates. Higher temporal and spatial variability can be expected for a dataset at higher resolution, both temporally and spatially, as is the case for GPCP daily v1.3 vs. GPCP monthly v2.3. Consequently, it can be expected that the same target (0.3 mm/d) will not be equally applicable to both datasets. The targets were initially formulated based on the HOAPS v4.0 validation report which does not include a daily product either. In this sense, we recommend to accept this violation in 7.5% of all cases as the expected behavior of a high-resolution dataset.

For further details, see the Validation Report CM SAF SSM/I and SSMIS products, HOAPS version 4.0, Ref: SAF/CM/DWD/VAL/HOAPS, available at https://www.cmsaf.eu/SharedDocs/Literatur/document/20172016/saf_cm_dwd_val_hoaps4gac_1rad_2_1_pdf.

...

4.2.

...

4 Water vapour profiles CDR brokered from

...

ROM SAF

...

The monthly-mean gridded humidity profile data have been validated against ERA-Interim reanalysis data, as a part of the ROM SAF validation procedures. The compliance of the observed data with the ROM SAF target accuracy of 3% is checked within six broad latitude-height regions (low-, mid-, high latitudes, and 0-8 km, 8-12 km). The data comply with the target requirements for the whole time period, except for a single month in a single region (mid-latitudes, 0-8 km, December 2013) where it is just slightly above the target. This can most likely be explained by wintertime high variability northern hemisphere conditions leading to larger uncertainties both in the observed and reference data.

For further details, see the ROM SAF Level 3 Validation Report, Ref: SAF/ROM/DMI/REP/GRD/001, available at http://www.romsaf.org/product_documents/romsaf_vr_grd_rep.pdf.

4.2.5 Water vapour UTH CDR brokered from CM SAF

Validation of the MW UTH TCDR was performed against the MW-equivalent UTH calculated from ERA-Interim reanalysis. The UTH from AMSU-B (NOAA-15, NOAA-16, NOAA-17) and MHS (NOAA-18, MetOp-A, MetOp-B) instruments was examined. The data record fulfills the CM SAF target requirements (see CM SAF Products Requirements Document, SAF/CM/DWD/PRD/2.10) and the GCOS requirements in terms of both accuracy and decadal stability within ±60° latitude.

For further details, see the CM SAF Validation Report, Ref: SAF/CM/UKMO/VAL/UTH, DOI: 10.5676/EUM_SAF_CM/UTH/V001, available at https://www.cmsaf.eu/SharedDocs/Literatur/document/2018/saf_cm_ukmo_val_uth_1_3_pdf.

4.2.6 Earth Radiation Budget CDR brokered from CERES

The CERES team invests a lot of effort in the comprehensive validation of the CERES products. The main findings are discussed during bi-annual science team meetings and are summarised in the so-called "Data Quality Summary (DQS)" that any users of the data is expected to read. For the 4th edition of the Energy Balanced And Filed (EBAF), the DQS reports accuracies of 2.5 W/m² as well for the reflected shortwave radiation as for the emitted longwave radiation. In terms of stability, the record seems to remain stable within 0.3 W/m²/decade for the SW and 0.2 W/m²/decade for the LW.

For further details, see the CERES_EBAF_Ed4.0 Data Quality Summary (January 12, 2018), available at https://ceres.larc.nasa.gov/documents/DQ_summaries/CERES_EBAF_Ed4.0_DQS.pdf.

5. Application of the KPI evaluation approach to ICDRs

5.1 Updated KPI values for the ICDRs

For steady quality control of our new ICDR chains, we will apply the evaluation approach outlined in Chapter 3 above. Therefore, we set the KPI values according to Table 3 which contains entries for all ICDRs for which the respective TCDRs have already been brokered. Missing entries relating to ICDRs, whose TCDRs have not been brokered yet will be given at a later stage. They will serve as lower and upper bounds for the binomial test as they correspond to the 2.5^th and 97.5^th percentiles mentioned in Chapter 3. The last column of Table 3 briefly describes important data set specific aspects of the analysis, e. g. the reference data set used. We will regularly give account of our test results in the Quarterly Reports.

5.2 Exemplary application to the first delivery of the GPCP ICDR

As a non-synthetic example, we evaluate the KPI accuracy for the first delivery of the GPCP precipitation ICDR.

5.2.1 GPCP monthly v2.3

The 2.5- and 97.5-quantiles of the spatially averaged differences of precipitation for GPCP monthly v2.3 CDR, covering the time period Jan 1979 to Dec 2017, are -0.169 mm/d and +0.185 mm/d, respectively (see Table 3). The respective reference is TMPA 3B43 v7, which is only available between 50°S and 50°N, and from Jan 1998. Consequently, all relevant values are obtained by averaging over this spatial TMPA window from Jan 1998.

The first delivery of the ICDR comprises Jan 2018 to Dec 2018, i.e. 12 realizations of monthly means. For all of these, the spatially averaged precipitation differences between GPCP and the reference product fall inside the 2.5- and 97.5-quantiles found in the monthly CDR ensemble. It is thus

p_{n,k}^{cum}=1 \geq 0.05

, so that we conclude that the ICDR at this stage is of at least the same quality as the CDR and therefore passes the described test.

5.2.2 GPCP daily v1.3

The GPCP daily v1.3 TCDR covers the period 1st Oct 1996 to 31st Dec 2017. Our reference here is TMPA 3B42 v7, which is, like its monthly realization (see above), available from Jan 1998 between 50°S and 50°N. The respective 2.5- and 97.5-quantiles of spatially averaged differences are -0.314 mm/d and 0.341 mm/d, respectively (see Table 3).

The first delivery of the ICDR comprises the same period as for the monthly product, 1st Jan 2018 to 31st Dec 2018, i.e. 365 realizations. As the TMPA 3B42 reference data set is yet missing for one day (31st Dec 2018), 364 realizations remain. For 361 of these, the spatially averaged precipitation differences between GPCP and the reference product fall inside the 2.5- and 97.5-quantiles found in the daily CDR ensemble. It is thus

p_{n,k}^{cum} \approx 0.99 \geq 0.05

, so that we conclude here, too, that the ICDR quality is sufficient. (Only a number of 338 or less would have indicated that the ICDR's quality is lower than the CDR's in this context.)

The predefined requirements for the surface incoming shortwave radiation are given in CM SAF Product Requirement Document (PRD), Annex A. The validation of the surface radiation datasets was conducted against surface measurements from the Baseline Surface Radiation Network (BSRN) [Ohmura et al., 1998].

All products in the brokered CLARA-A2 dataset fulfil the updated GCOS requirements regarding the horizontal and temporal resolution. The GCOS requirements for accuracy and stability are formulated for the global mean and are not relevant for individual reference measurement networks (e.g. BSRN). Besides this, there also exists the general problem with comparison of area-to-point measurements.

The KPI values for accuracy have been selected as following: SIS monthly means – 10 W/m² that corresponds to CM SAF target accuracy, SIS daily means – 20 W/m² that corresponds to CM SAF threshold accuracy, SDL monthly means – 10 W/m² that corresponds to CM SAF target accuracy and SOL monthly means – 10 W/m² that corresponds to CM SAF target accuracy. These targets are met by all datasets and the KPIs are fulfilled. Considering the uncertainty of the surface observations of 5 W/m², less than 5% of the SIS monthly means exceed the target requirements. For the SIS daily means about 20% exceed the threshold accuracy. This number can be explained by higher temporal resolution and thus less random biases averaging out. For the SDL monthly means less than 14% exceed the target accuracy and for the SOL monthly means less than 32% exceed the target accuracy.

Stability requirements for the Surface Radiation Budget TCDR have been defined as follows: SIS monthly means - 1.5 W/(m² dec), SIS daily means – 1.5 W/(m² dec), SDL and SOL monthly means – 0.5 W/(m² dec). Within the range of uncertainty the CLARA-A2 dataset agrees with the surface reference data. Achieved stability for the SIS monthly means (after 1994) is less than 1 W/(m² dec), achieved stability for the SDL and SOL monthly means is less than 1 W/(m² dec).

For further details, see the Validation Report CM SAF Cloud, Albedo, Radiation dataset, AVHRR-based, Edition 2 (CLARA-A2), Surface Radiation Products, Code: SAF/CM/DWD/VAL/GAC/RAD, available at https://www.cmsaf.eu/SharedDocs/Literatur/document/2016/saf_cm_dwd_val_gac_rad_2_1_pdf.

5.2.4 Water vapour profiles TCDR brokered from ROM SAF (D3.3.9-v1.0):

The monthly-mean gridded humidity profile data have been validated against ERA-Interim reanalysis data, as a part of the ROM SAF validation procedures. The compliance of the observed data with the ROM SAF target accuracy of 3% is checked within six broad latitude-height regions (low-, mid-, high latitudes, and 0-8 km, 8-12 km). The data comply with the target requirements for the whole time period, except for a single month in a single region (mid-latitudes, 0-8 km, December 2013) where it is just slightly above the target. This can most likely be explained by wintertime high variability northern hemisphere conditions leading to larger uncertainties both in the observed and reference data.

For further details, see the ROM SAF Level 3 Validation Report, Ref: SAF/ROM/DMI/REP/GRD/001, available at http://www.romsaf.org/product_documents/romsaf_vr_grd_rep.pdf.

5.2.5 Water vapour UTH TCDR brokered from CM SAF (D3.3.12-v1.0):

Validation of the MW UTH TCDR was performed against the MW-equivalent UTH calculated from ERA-Interim reanalysis. The UTH from AMSU-B (NOAA-15, NOAA-16, NOAA-17) and MHS (NOAA-18, MetOp-A, MetOp-B) instruments was examined. The data record fulfills the CM SAF target requirements (see CM SAF Products Requirements Document, SAF/CM/DWD/PRD/2.10) and the GCOS requirements in terms of both accuracy and decadal stability within ±60° latitude.

For further details, see the CM SAF Validation Report, Ref: SAF/CM/UKMO/VAL/UTH, DOI: 10.5676/EUM_SAF_CM/UTH/V001, available at https://www.cmsaf.eu/SharedDocs/Literatur/document/2018/saf_cm_ukmo_val_uth_1_3_pdf.

5.2.6 Earth Radiation Budget TCDR brokered from CERES (D3.3.20-v1.0):

The CERES team invests a lot of effort in the comprehensive validation of the CERES products. The main findings are discussed during bi-annual science team meetings and are summarised in the so-called "Data Quality Summary (DQS)" that any users of the data is expected to read. For the 4th edition of the Energy Balanced And Filed (EBAF), the DQS reports accuracies of 2.5 W/m² as well for the reflected shortwave radiation as for the emitted longwave radiation. In terms of stability, the record seems to remain stable within 0.3 W/m²/decade for the SW and 0.2 W/m²/decade for the LW.

For further details, see the CERES_EBAF_Ed4.0 Data Quality Summary (January 12, 2018), available at https://ceres.larc.nasa.gov/documents/DQ_summaries/CERES_EBAF_Ed4.0_DQS.pdf.

...

...

Anchor
references references

References

Immler, F. J., J. Dykema, T. Gardiner, D. N. Whiteman, P. W. Thorne, and H. Vömel (2010), Reference Quality upper-air measurements: Guidance for developing GRUAN data products, Atmos. Meas. Tech., 3(5), 1217–1231, doi:10.5194/amt-3-1217-2010.

Loew, A., Bell, W., Brocca, L., Bulgin, C. E., Burdanowitz, J., Calbet, X., Donner, R. V., Ghent, D., Gruber, A., Kaminski, T., Kinzel, J., Klepp, C., Lambert, J.-C., Schaepman-Strub, G., Schröder, M., and Verhoelst, T. (2017): Validation practices for satellite based earth observation data across communities, Rev. Geophys., https://doi.org/10.1002/2017RG000562 ^{Image Modified} .

Merchant, C. J., Paul, F., Popp, T., Ablain, M., Bontemps, S., Defourny, P., Hollmann, R., Lavergne, T., Laeng, A., de Leeuw, G., Mittaz, J., Poulsen, C., Povey, A. C., Reuter, M., Sathyendranath, S., Sandven, S., Sofieva, V. F., and Wagner, W. (2017): Uncertainty information in climate data records from Earth observation, Earth Syst. Sci. Data, 9, 511-527, https://doi.org/10.5194/essd-9-511-2017 ^{Image Modified} .

Weatherhead, E. C., Reinsel, G. C., Taio, G. C., Meng, X.-L., Choi, D., Cheang, W.-K., Keller, T., DeLuisi, J., Wuebbles, D. J., Kerr, J. B., Miller, A. J., Oltmans, S. J., and Frederick, J. E. (1998), Factors affecting the detection of trends: Statistical considerations and applications to environmental data, J. Geophys. Res., 103, 17,149–17,161, doi:10.1029/98JD00995.

...

...