The Climate Data Store (CDS) is the cornerstone infrastructure which supports the implementation of the Copernicus Climate Change Service (C3S). It enables the provision of Essential Climate Variables (ECVs), climate analyses, reanalyses, projections and indicators at temporal and spatial scales relevant to adaptation and mitigation strategies for various sectoral and societal benefit areas.
The CDS is designed as a distributed system which provides improved access to local and remote datasets via a powerful service-oriented architecture.
The CDS offers seamless web-based and API-based search and retrieve facilities to access climate data and information. In addition, the CDS also provides a generic software toolbox that allows users to develop web-based applications that make use of the datasets available in the CDS.
The data provided by the CDS are free and open data, subject to the user agreeing to the relevant dataset licence(s). For further details, please see this article from the ECMWF Newsletter 151 and this C3S User Learning Services video.
The CDS provides a Catalogue which lists C3S data and products, including observations, reanalyses, seasonal forecasts and climate projections. Users can search through the catalogue, and can filter the entries by means of both faceted search (Product type, Variable domain, Spatial coverage, or Temporal coverage) and textual search.
Users can request data from the CDS using a variety of methods:
1) The CDS web interface is an interactive system: the user fills a web form to construct their valid query and can then choose between three options:
Please make sure that you select all the mandatory fields before submitting a request. ![]() |
This is shown in the following figures:
Figure 1 Web interface | Figure 2 Request options on the web interface |
2) The CDS API is a service providing programmatic access to CDS data in Python (using the CDS API client). Users need to have a CDS account to use it with their related CDS API credentials. For a full description and some useful examples, please see How to use the CDS API.
If users have more than one CDS API key (e.g. you also have a Atmosphere Data Store (ADS) account), you have to pass credentials in explicitly and the following method can be used:
import cdsapi import yaml with open('/path/to/ads/cdsapirc', 'r') as f: credentials = yaml.safe_load(f) c = cdsapi.Client(url=credentials['url'], key=credentials['key']) c.retrieve("dataset-short-name", {... sub-selection request ...}, "target-file") |
We strongly suggest to construct CDS API requests by using the CDS web interface of the relevant dataset and using the 'Show API request' button to get the code.
For the most common CDS API issues, please see Common Error Messages for CDS Requests. |
For non-Python users, please note that the CDS API is REST-based so it can be wrapped by any programming language. Please see this example with the R package: Download CDS ERA5 data using R.
Users can also set the PROXY within the CDS API script:
import requests import cdsapi session = requests.Session() session.proxies = { 'http': 'http://10.10.10.10:8000', 'https': 'http://10.10.10.10:8000', } client = cdsapi.Client(session=session) |
3) The CDS Toolbox is a set of software which enables users to develop custom-made applications via an on-line interface (Figure 3). The applications can make use of the content of the CDS to analyse, monitor and predict the evolution of both climate drivers and impacts. The CDS Toolbox is a catalogue of software that can be classified as:
The CDS Toolbox documentation provides a description of the CDS Toolbox content including Tutorials, Applications Gallery and Guides. In addition, a graphical, user-friendly tour is presented when the user first connects to the CDS Toolbox editor .
A new Toolbox forum is also available for user to find answers and share knowledge.
Figure 3 CDS Toolbox.
Please note: CDS datasets are covered by one or more licences, and users have to accept them in order to download datasets (whether through the web interface, or the CDS API, or the toolbox). At the time this article was written, the acceptance of the relevant licences can only be carried out through the CDS web interface for the relevant dataset. In some cases, data can has multiple licenses because it comes from different providers. |
As the CDS is a distributed platform, CDS datasets are hosted at a number of different locations. Some data are stored on the CDS disks; others are hosted on the data providers' storage systems, e.g. the ECMWF Meteorological Archival and Retrieval System (MARS).
Table 2 presents the ERA5 datasets which are available on CDS disks ('MARS internal)'. Table 3 and Table 4 show the CDS datasets which are stored in the tape library (referred to as 'MARS external'), and in the data providers archives respectively.
The way the user selects the items to include in a single CDS request can make a significant difference in terms of performance. Please see the page Efficiency tips for how to build an efficient request.
On the CDS, requests and relevant results are generally cached for 1.5 to 2 days, depending on the user load on the CDS system. Thereafter, files are deleted starting from the oldest. Please note that cached data is used to fulfil a CDS request only if a given CDS request is exactly identical to a previous one.
After sending a request, the user can track its state on the 'Your Requests' page of the CDS web interface. Users can also check the status of the CDS system at the live page. There are five different states of a request:
User can check the live status of the CDS queues here https://cds.climate.copernicus.eu/live/queue
Please note:
|
Limits are set on usage of CDS resources to ensure an appropriate level of performance.
They are divided in three categories: per-user, global and system, and these limits are shown in Table 1 (last reviewed on ).
The CDS will queue requests which would otherwise cause any of these limits to be exceeded. Please note that these limits are changed from time to time according to the current workload of the system and number of concurrent tasks.
The 'live' status of the limits values can be checked here https://cds.climate.copernicus.eu/live/limits
|
Moreover, CDS data requests also have limits in terms of number of fields and volume size which are different for each dataset (see the tables below). These values are dependent on each dataset's structure and where the data is stored. Again, these restrictions are introduced to help the system maintain good performance and minimise the queue time.
When a user CDS web request exceeds the number of fields limit, an information message appears at the bottom of the web interface page. Note that these limits are also enforced for requests sent via the CDS API and the CDS Toolbox.
Figure 4 Information message when a user CDS web request exceeds the number of fields limit.
The tables below summarise the number of fields and volume size limits, as well as the major features of all CDS datasets. Table 2 shows the datasets stored in 'MARS internal', whereas Table 3 presents the datasets stored in 'MARS external', and Table 4 shows datasets maintained by external data providers.
|
|
For ERA5 data requests, please see: How to download ERA5. The first example in CDS API and CDS Toolbox examples section below shows how to efficiently download a whole year of hourly data for 2m temperature (grib format, for an area subset) from the CDS, by asking for one month of data per request.
reanalysis-era5-complete
, do not appear in the CDS catalogue on the web interface, but users can still retrieve the data through the CDS API. In these cases, users can make use of a subset of the ECMWF MARS keywords in their CDS API request, and should also follow the MARS efficiency rule of thumb (the idea is to request as much data as possible from the same tape file or to reduce the number of tapes involved.).CDS API Request for hourly data of one variable looping though months and years ( e.g 2m temperature for all months in 2018 and 2019)
#!/usr/bin/env python3 import cdsapi c = cdsapi.Client() first_year = 2018 last_year = 2019 for year in range(first_year, last_year + 1): for month in range(1, 13): print("=========================================================") print("Downloading {year}-{month:02d}".format(year=year, month=month)) c.retrieve( 'reanalysis-era5-single-levels', { 'product_type': 'reanalysis', 'variable': '2m_temperature', 'year': str(year), 'month': "{month:02d}".format(month=month), 'day': [ '01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', ], 'time': [ '00:00', '01:00', '02:00', '03:00', '04:00', '05:00', '06:00', '07:00', '08:00', '09:00', '10:00', '11:00', '12:00', '13:00', '14:00', '15:00', '16:00', '17:00', '18:00', '19:00', '20:00', '21:00', '22:00', '23:00', ], 'area': [ 90, 170, 80, 180, ], 'format': 'grib', }, "{year}-{month:02d}.grib".format(year=year, month=month)) |
CDS API Request for Temperature data from the ERA5 hourly dataset on pressure levels ('renalysis-era5-pressure-levels') on 11-10-2018, from 08:00 to 18:00 UTC on pressure level 1000 hPa.
import cdsapi c = cdsapi.Client() c.retrieve( 'reanalysis-era5-pressure-levels', { 'product_type':'reanalysis', 'variable':'temperature', 'pressure_level':'1000', 'year':'2018', 'month':'10', 'day':'11', 'time':[ '08:00','09:00','10:00', '11:00','12:00','13:00', '14:00','15:00','16:00', '17:00','18:00' ], 'format':'grib' }, 'download.grib') |
CDS Toolbox request for Temperature data from the ERA5 hourly dataset on pressure levels ('renalysis-era5-pressure-levels') on 11-10-2018, from 08:00 to 18:00 UTC on pressure level 1000 hPa.
import cdstoolbox as ct data = ct.catalogue.retrieve( 'reanalysis-era5-pressure-levels', { 'product_type':'reanalysis', 'variable':'temperature', 'pressure_level':'1000', 'year':'2018', 'month':'10', 'day':'11', 'time':[ '08:00','09:00','10:00', '11:00','12:00','13:00', '14:00','15:00','16:00', '17:00','18:00' ], 'format':'grib' }) |
CDS API request for Temperature data from ERA5 complete dataset ('reanalysis-era5-complete') for July 2019 on all model levels (1-137)
import cdsapi c = cdsapi.Client() c.retrieve('reanalysis-era5-complete', { 'class': 'ea', 'date': '2019-07-01/to/2019-07-31', 'expver': '1', 'levelist': '1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30/31/32/33/34/35/36/37/38/39/40/41/42/43/44/45/46/47/48/49/50/51/52/53/54/55/56/57/58/59/60/61/62/63/64/65/66/67/68/69/70/71/72/73/74/75/76/77/78/79/80/81/82/83/84/85/86/87/88/89/90/91/92/93/94/95/96/97/98/99/100/101/102/103/104/105/106/107/108/109/110/111/112/113/114/115/116/117/118/119/120/121/122/123/124/125/126/127/128/129/130/131/132/133/134/135/136/137', 'levtype': 'ml', 'param': '130', 'step': '0', 'stream': 'oper', 'time': '09:00:00', 'type': '4v', }, 'output') |
This document has been produced in the context of the Copernicus Climate Change Service (C3S). The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of C3S on behalf of the European Union (Delegation Agreement signed on 11/11/2014 and Contribution Agreement signed on 22/07/2021). All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose. The users thereof use the information at their sole risk and liability. For the avoidance of all doubt , the European Commission and the European Centre for Medium - Range Weather Forecasts have no liability in respect of this document, which is merely representing the author's view. |