EUMETSAT is establishing has established a new set of data access services that will bring are bringing the full data catalog catalogue online.  Some more General information can be found here:  https://wwwuser.eumetsat.int/data-access-our-data.

Knowledge bases for each of the new data services can be found through this landing page: https://eumetsatspace.atlassian.net/wiki/spaces/EUM/overview

These services are currently in their pilot phase, and are provided on a best-effort basis. Even now the services are open, it will still take time to fully populate them.

.

In addition, some selected datasets can be accessed from EWC VMs from the EWC object storage. Please contact us, if you need additional data which is not provided in the Data Store, or you have special conditions for access. 

Option 1: Using EUMETSAT data services

To avoid duplication, the information on how to use the new data services can be found on the following links:

• General description of all the data services
• Direct link to the Data Store (catalogue and download service) documentation
• Direct link to the Data Tailor (format conversion service) documentation

• Getting started guide for EUMETSAT data services, explaining what service to use for what purpose
• EUMDAC - recommended tool for simplifying and automating downloads from the EUMETSAT Data Store

In the future, it may become possible to directly access the object storage behind the Data Store for applications that need very high throughput (e.g. mass data processing).  Please contact us if you have tried the existing possibilities and believe you need this capability.

Option 2: Using EUMETSAT data buckets

Setting up your instance

...

In order to mount the buckets as filesystem, you need to create the Virtual Machine instances in the Morpheus UI. You can follow the Provision a new instance - web Knowledge Base Article to get started.

This article describes how to mount EUMETSAT data buckets as FUSE mounts. The possibility to mount the buckets into as filesystem is provided to as convenience. Readers are advised to note that mounting buckets via FUSE will provide significantly worse performance than accessing the data directly from workloads, for example using python.  Whenever possible, users are advised to use the buckets directly from the workloads.

Tenants having Morpheus account at EUMETSAT Morpheus, can mount the data buckets using ready workflow. This article describes how to run the workflow and how to mount the data manually. One can naturally use the same steps to any other buckets as well by modifying bucket name, service endpoint, credentials, and potentially project id.

Available EUMETSAT data buckets are listed below with the corresponding mount point in case of using ready workflow. You can get access keys for the data buckets from support.

EUMETSAT Data Buckets:

2016

Bucket	Mount point	Time range
ascat.metop-abc.level-1b.native	/eumetsatdata/ascat	2012-10-22T11:00:00Z - 2013-04-24T08:57:00Z
cmsaf-cloud-type.meteosat-0-degree.level-2.netcdf-4	/eumetsatdata/cloud	2004-2017
geo-hsr-simulation.0-degree.level-15.netcdf-4	/eumetsatdata/geo-hsr-test-data
iasi.metop-a.level-1c.native	/eumetsatdata/iasi	2007-
2020
mviri.meteosat-0-degree.level-15.native	/eumetsatdata/mviri	1981-2017
nwc-saf.0-degree.level-2	/eumetsatdata/nwcsaf	2019-2021
nwc-saf.0-degree.level-2-{year}	/eumetsatdata/nwcsaf-2022	2022 onwards
nwc-saf.0-degree.level-2-cf	/eumetsatdata/nwcsaf-cf	2019-2020
seviri.meteosat-0-degree.level-15.native	/eumetsatdata/seviri	2020
drought-and-vegetation.prototype-datacube.netcdf	/eumetsatdata/psdc

Using the data directly from workloads 

It is advisable to use the data directly from workloads (scripts / applications). Please consult e.g.

• s3 using Python libraries
• Object storage: How to use s3cmd and s3fs
• s3 from command line

for more help. 

Mounting the buckets using FUSE

Following describes how to mount EUMETSAT data buckets as FUSE mounts. The possibility to mount the buckets into as filesystem is provided to as convenience. Readers are advised to note that mounting buckets via FUSE will provide significantly worse performance than accessing the data directly from workloads, for example using python.  Whenever possible, users are advised to use the buckets directly from the workloads.

Tenants having Morpheus account at EUMETSAT Morpheus, can mount the data buckets using ready workflow. This article describes how to run the workflow and how to mount the data manually. One can naturally use the same steps to any other buckets as well by modifying bucket name, service endpoint, credentials, and potentially project id.

Using the Workflow

1. Go to instances and select the instance of interest
   
2. Select Actions → Run Workflow
   
   
3. Select Mount EUMETSAT databuckets from Workflow drop-down meny and click Execute. Command options are not needed.
   
   

...

1. Make sure you have latest rclone installed

   Code Block
   curl https://rclone.org/install.sh | sudo bash

   
   

2. Create mount directory, i.e. for ascat:

   Code Block
   sudo mkdir -p /eumetsatdata/ascat && sudo chown -R $USER:$USER /eumetsatdata/ascat && chmod 755 /eumetsatdata/ascat

   
   

3. Put rclone like listed below config into a config file. For example, into /etc/eum_bucket_creds (keys without brackets) 

   Code Block
   sudo sh -c "cat >>/etc/eum_bucket_creds <<-EOF [eumetsat-ascat] type = s3 provider = Ceph env_auth = false access_key_id={{ aws_access_key_id }} secret_access_key={{ aws_secret_access_key }} endpoint = https://s3.waw3-1.cloudferro.com EOF

   
   

4. Now you can already mount the data with command:

   Code Block
   sudo rclone --config /etc/eum_bucket_creds mount --allow-other --daemon eumetsat-ascat:ascat.metop-abc.level-1b.native /eumetsatdata/ascat

...

1. Link rclone

   Code Block
   sudo ln -s /usr/bin/rclone /sbin/mount.rclone

   
   

2. Create systemd service:, for example for ascat:

   Code Block

   sudo sh -c "cat >>/etc/systemd/system/eum-data-ascat.service" <<-EOF [Unit] Description=Mount EUMETSAT data bucket ascat After=network-online.target [Mount] Type=rclone What=eumetsat-ascat:ascat.metop-abc.level-1b.native Where=/eumetsatdata/ascat Options=ro,allow_other,args2env,vfs-cache-mode=writes,config=/etc/eum_bucket_creds,cache-dir=/var/rclone,vfs-cache-max-size=512Mi,vfs-cache-poll-interval=30m' EOF

   
   

3. Add corresponding line into /etc/fstab:

   Code Block
   echo "eumetsat-ascat:847632567fbf45408555318627075a26:ascat.metop-abc.level-1b.native /eumetsatdata/ascat rclone ro,noauto,nofail,x-systemd.automount,args2env,allow_other,vfs_cache_mode=writes,config=/etc/eum_bucket_creds,cache_dir=/var/cache/rclone,vfs-cache-max-size=512Mi,vfs-cache-poll-interval=30m 0 0" | sudo tee -a /etc/fstab

   
   

4. Mount the folder:

   Code Block
   sudo mount /eumetsatdata/ascat

...