This page contains a guide for building and using OpenIFS 48r1 in a linux based container using Docker.

While this page focuses on the containerised installation, the details presented can be used as a recipe for the underlying dependencies required to install and run OpenIFS 48r1 on a linux system.

If you have access to the ECMWF HPCF (ATOS hpc2020) and are planning to run OpenIFS on that system please refer to OpenIFS 48r1 User Guide.

Contents:

Creation of a docker build directory

The first step is to make a directory from which the OpenIFS container can be created and cd into that directory, e.g.

mkdir <openifs-docker-build-dir>
cd <openifs-docker-build-dir>

Once inside the <openifs-docker-build-dir> download the OpenIFS release package, e.g.

wget https://sites.ecmwf.int/openifs/openifs-data/src/48r1/openifs-48r1.tar.gz
tar -xvzf openifs-48r1.tar.gz

Further, if the SCM is required, then also download the SCM package to <openifs-docker-build-dir>

wget https://sites.ecmwf.int/openifs/openifs-data/scm/48r1/scm_openifs_48r1.tar.gz
tar -xzvf scm_openifs_48r1.tar.gz

Once OpenIFS 48r1 has been extracted to your <openifs-docker-build-dir> , then copy the Dockerfile from the release to <openifs-docker-build-dir>

cp  <openifs-docker-build-dir>/openifs-48r1/scripts/docker/gcc-docker-48r1/Dockerfile <openifs-docker-build-dir>/.

This cp step is important because the Dockerfile and the OpenIFS release and SCM package, i.e., openifs-48r1 and scm_openifs, need to be in the same directory to build the container.

Build the OpenIFS docker image

The following command builds the docker image from the <openifs-docker-build-dir> , with the docker command executed in the same directory as the Dockerfile .

docker build --tag "openifs-48r1" .

Note the trailing '.' to build in the current dir, which is required.

This command runs the build process for the OpenIFS image using gcc:11.2.0-bullseye as the base image, which is a debian Linux distribution with gcc 11.2 (at the time of writing, similar to the gcc on the ATOS). After downloading the image, the Dockerfile installs the following software

    apt install -y git && \
    apt install -y cmake && \
    apt install -y python3 python3-ruamel.yaml python3-yaml python3-venv && \
    apt install -y libomp-dev && \
    apt install -y libboost-dev libboost-date-time-dev libboost-filesystem-dev libboost-serialization-dev libboost-program-options-dev&& \
    apt install -y netcdf-bin libnetcdf-dev libnetcdff-dev && \
    apt install -y libatlas-base-dev && \
    apt install -y liblapack-dev && \
    apt install -y libeigen3-dev && \
    apt install -y bison && \ 
    apt install -y flex && \
    apt install -y vim && \
    apt install -y wget bc 
## Install python3-pip and the dependencies to run 
## plotscm.py with SCM output
    apt install -y python3-pip && \
    pip3 install numpy && \
    pip3 install netcdf4 && \
    pip3 install matplotlib && \
    pip3 install pandas && \
    pip3 install xarray
## Download and build openmpi
https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.5.tar.gz

Once these repository packages and the python librarie are installed, open-mpi is downloaded and built. Then the openifs user is created and the openifs-48r1 directory is copied to the image.

On a Macbook pro (M1) the initial build of the container, takes about 5 minutes. If you decide to change the Dockerfile following a build, you can just execute the same command and, depending on what you change, subsequent builds can be a lot quicker.

At the time of writing open-mpi is downloaded and built as part of the image creation. This is quite slow, so in the future we will investigate the use of a standard library
If a fresh build is required but an image has already been built, then execute the following command, which is the same as above but with no-cache

docker build --no-cache --tag "openifs-48r1" .

The tag name does not need to be provided but it is useful for identifying images.

Running OpenIFS-test using the docker image

Once a build has completed, the image can be checked by typing docker images , e.g.,

docker images

REPOSITORY          TAG       IMAGE ID       CREATED       SIZE
openifs-48r1      latest    f72ea92f010f   5 hours ago   2.15GB

And the container can be run using the following

docker run -it 'openifs-48r1'

This command will open an interactive session in the new container, in which the entry directory is the openifs-48r1 directory, e.g.

docker run -it 'openifs-48r1'

OpenIFS environment variables are:
------------------------------------------------------
OIFS_ARCH=./arch/ecmwf/hpc2020
OIFS_CYCLE=48r1
OIFS_DATA_DIR=/perm/openifs/oifs_data/48r1/48r1
OIFS_EXEC=/home/openifs/openifs-48r1/build/bin/ifsMASTER.DP
OIFS_EXPT=/home/openifs/openifs-expt
OIFS_HOME=/home/openifs/openifs-48r1
OIFS_LOGFILE=/home/openifs/openifs-48r1/oifs_test_log.txt
OIFS_RUN_SCRIPT=/home/openifs/openifs-48r1/scripts/exp_3d
OIFS_TEST=/home/openifs/openifs-48r1/scripts/build_test


SCM environment variables are:
------------------------------------------------------
SCM_EXEC=/home/openifs/openifs-48r1/build/bin/MASTER_scm.DP
SCM_LOGFILE=/home/openifs/openifs-expt/scm_openifs/48r1/scm-projects/ref48r1/scm_run_log.txt
SCM_PROJDIR=/home/openifs/openifs-expt/scm_openifs/48r1/scm-projects
SCM_RUNDIR=/home/openifs/openifs-expt/scm_openifs/48r1/scm-projects/ref48r1
SCM_TEST=/home/openifs/openifs-48r1/scripts/scm
SCM_VERSIONDIR=/home/openifs/openifs-expt/scm_openifs/48r1

openifs@d1bd89ccc47f:~/openifs-48r1$ ls
CHANGES  COPYING  NOTICE  arch        ifs-source  oifs-config.edit_me.sh  scripts
CITE     LICENSE  README  bundle.yml  ifs-test    openifs-bundle

In the above code block the openifs config (oifs-config.edit_me.sh) is sourced automatically so the container has the correct environment for OpenIFS

Once inside the container, the openifs-test.sh script can be run, e.g.,

openifs@d1bd89ccc47f:~/openifs-48r1$ $OIFS_TEST/openifs-test.sh -cbt --arch=""

where :

-c creates source directory in $OIFS_HOME, which is used to bring all the sources, e.g. ifs-source, ifs-test and any packages in the bundle (ecbuild, eccodes etc.), together in preparation for the build

-b builds the source. This step creates the directory build in $OIFS_HOME, which is used to build and store

the OpenIFS double and single precision master executables (ifsMASTER.DP and ifsMASTER.SP, respectively) , which are used to run 3-D OpenIFS. The executables are located in $OIFS_HOME/build/bin.
- The location and name of the executable for OpenIFS is defined in the platform configuration file (oifs-config.edit_me.sh) as $OIFS_EXEC.

the double and single precision Single Column Model (SCM) executables (MASTER_scm.DP and MASTER_scm.SP, respectively), which are used to run the SCM derived from OpenIFS. The executables are located in $OIFS_HOME/build/bin.
- The location and name of the executable for the SCM is defined in the platform configuration file (oifs-config.edit_me.sh) as $SCM_EXEC.

-t will run the ifs-test t21 tests, which comprise of

21 3-D OpenIFS forecast-only tests with and without chemistry
1 SCM test (based on TWP-ICE)

--arch="" means no arch file is required because all the libraries and paths have been set-up locally in the docker container.

For more information on the openifs-test option please refer to Getting started and the linked page Openifs-test - build options.

On a Macbook pro (M1) the build of OpenIFS, takes about 10 minutes, while successful completion of ifstest takes just over 3 minutes.

Running the SCM in the docker container

Once the above has been completed, you are ready to run the SCM. The scm_openifs contains 3 testcases, each representative of different cloudy regimes

DYCOMS - marine stratocumulus case
BOMEX - trade-wind cumulus case
TWPICE - A multi-day deep convective case

The SCM is run using the callscm script, which is a wrapper for the main run.scm. Both scripts can found in $SCM_TEST , which is set in the oifs-config.edit_me.sh file to ${OIFS_HOME}/scripts/scm .

callscm includes default settings, which are the all cases, with a 450 s timestep and an experiment name of ref-oifs-scm . To run with these settings, enter the following

cd $OIFS_HOME
$SCM_TEST/callscm

callscm (with defaults, i.e. no arguments) will run the DYCOMS case with the SCM and create an output directory in $SCM_RUNDIR/scmout_DYCOMS_ref-oifs-scm_450s , which contains the diagnostic output from the SCM. In addition, the file scm_run_log.txt will be created in $SCM_RUNDIR . This file contains the print output from the SCM, which is useful for checking all the sources and paths for a simulation.

A user can change the defaults by using the available command-line options

callscm -h -c <case_name or list of case_names> -t <timestep or list of timesteps>
        -x <expt_name>
where :
-h is help which returns basic usage options and exits
-c case_name or list of case_names (space delimited) of the case study
   used for namelist and output directory. Default list is
   "DYCOMS BOMEX TWPICE" 
-t timestep or list of timesteps in seconds. The default is 450s. An
   example of a list is "1800 900 300" 
-x expt_name shortname to identify experiment. Default is ref-oifs-scm

For example, if a user wanted to run the BOMEX case with timesteps of 1800 s and 900 s and an experiment name of "bomex_test", they would enter the following

$SCM_TEST/callscm -c BOMEX -t "1800 900" -x "bomex_test"

This command results in the following output directories $SCM_RUNDIR/scmout_BOMEX_bomex_test_900s and scmout_BOMEX_bomex_test_1800s.

Example plotting script

Once the SCM has successfully, it will produce output in $SCM_RUNDIR/scmout_<casename>_<expt_name>_<timestep>, which contains some netcdf files.

As part of the scm_openifs download package we provide a python script $SCM_RUNDIR/plotscm.py , which can be used to plot some profiles at a certain time, 2-d time height plots and some scalar timeseries. This script will work without any changes for the DYCOMS case and the default settings of callscm by type the following

cd $SCM_RUNDIR
python3 plotscm.py

For the defaults, the above will produce plot_DYCOMS directory, which contains png plots

To plot the other cases or experiments with different names and/or timesteps to the defaults, at present, the user will need to edit plotscm.py appropriately.

Basic docker commands and functionality

Check for existing containers

From a terminal on the host system, it is possible to list running and exited containers with the following

List running containers

$ docker ps

CONTAINER ID   IMAGE            COMMAND   CREATED        STATUS         PORTS     NAMES
d1bd89ccc47f   openifs-48r1   "bash"    15 hours ago   Up 5 seconds             beautiful_pasteur

If no container is running on your system, then only CAPITAL headings are returned with docker ps

List all containers (running and exited)

$ docker ps -a

CONTAINER ID   IMAGE               COMMAND       CREATED        STATUS                     PORTS     NAMES
d1bd89ccc47f   openifs-48r1      "bash"        15 hours ago   Exited (0) 4 seconds ago             beautiful_pasteur

Notice that the STATUS is Exited, rather than Up, as in (1)

Start an existing container

If docker ps -a shows an exited container it can be restarted using the following

$ docker start -i <Container ID> 
# e.g. 
$ docker ps -a

CONTAINER ID   IMAGE               COMMAND       CREATED        STATUS                     PORTS     NAMES
d1bd89ccc47f   openifs-48r1      "bash"        15 hours ago   Exited (0) 4 seconds ago             beautiful_pasteur

$ docker start -i  d1bd89ccc47f

Exiting the container

The container can be exited by typing exit from the active container.
A container can be stopped from a terminal on the host system
docker stop <Container ID>
It is also possible to stop multiple containers at once by adding more than one ID.
Using docker stop will attempt a clean shutdown. It is also possible to stop the container by killing it
docker kill <Container ID>

Removing containers and images

Remove a container with the following command

docker rm <Container ID>

It is possible to remove multiple containers at once by adding more than one ID.

It is worth noting that if a container is running it cannot be removed without being forced or being stopped first

Removing a container will result in a loss of all data and any code changes etc from the container