This page contains a guide for building and using OpenIFS 48r1 at model version 48R1.1 in a linux based container using Docker. 

The instructions presented assume that docker is installed on the system of interest. If docker is not installed please speak with local system administration to find out if installation is permitted and how to do it. The specifics of this page have been tested with docker desktop on ubuntu 22.04 and Apple Macbook M1

While this page focuses on the containerised installation, the details presented, particularly the list of libraries to install, can be used as a recipe for the underlying dependencies required to install and run OpenIFS 48r1 on a linux system.

...

Contents:

...

Creation of a docker build directory

The first step is to make a directory from which the OpenIFS container can be created and cd into that directory, e.g.

...

Once inside the <openifs-docker-build-dir>  clone or branch from the git repository with download the OpenIFS release package from https://sites.ecmwf.int/openifs/openifs-data/src/48r1/openifs-48r1.tar.gz,  e.g.

git# download clonethe openifs--single-branch --branch release ssh://git@git48r1.tar.gz model package from https://sites.ecmwf.int/openifs/openifs-data/src/48r1/openifs-48r1.tar.gz to <openifs-docker-build-dir> and untar :
tar -xvzf openifs-48r1.tar.gz

Then also download the SCM package to <openifs-docker-build-dir> from https://sites.ecmwf.int/

...

openifs/openifs-data/scm/48r1/scm_openifs_48r1.tar.

...

gz . 

# download the scm_openifs_48r1.tar.gz https://sites.ecmwf.int/openifs/openifs-data/scm/48r1/scm_openifs_48r1.tar.gz to <openifs-docker-build-dir> and untar :
tar -xzvf scm_openifs_48r1.tar.gz

At present, the SCM package is required for the build of the container.

Once OpenIFS 48r1 and the SCM package have

Once OpenIFS 48r1 has been extracted to your <openifs-docker-build-dir> ,  then copy the Dockerfile from the release to <openifs-docker-build-dir> 

cp  <openifs-docker-build-dir>/openifs-48r1.1/scripts/docker/gcc-docker-48r1.1/Dockerfile <openifs-docker-build-dir>/.

This cp step is important because the Dockerfile  and the OpenIFS release and SCM package, i.e., openifs-48r1 .1and scm_openifs, need to be in the same directory to build the container.

Build the OpenIFS docker image

Before starting to build the container

...

, it is necessary to set the OIFS_EXPT environment variable in oifs-config.edit_me.sh, i.e. 

open <openifs-docker-build-dir>/openifs-48r1/oifs-config.edit_me.sh  with a text editor, then change the following line

export OIFS_EXPT=${PERM}

to 

export OIFS_EXPT=/home/openifs/openifs-expt

The above change will ensure that the defined experiment directory is consistent to the experiment directory set-up during the build of the container. 

...

The following command builds the docker image from the <openifs-docker-build-dir> , with the docker command executed in the same directory as the Dockerfile .

docker build --tag "openifs-48r1.1" . 

Note the trailing '.' to build in the current dir, which is required.

...

    apt install -y git && \
    apt install -y cmake && \
    apt install -y python3 python3-ruamel.yaml python3-yaml python3-venv && \
    apt install -y libomp-dev && \
    apt install -y libboost-dev libboost-date-time-dev libboost-filesystem-dev libboost-serialization-dev libboost-program-options-dev&& \
    apt install -y netcdf-bin libnetcdf-dev libnetcdff-dev && \
    apt install -y libatlas-base-dev && \
    apt install -y liblapack-dev && \
    apt install -y libeigen3-dev && \
    apt install -y bison && \ 
    apt install -y flex && \
    apt install -y vim && \
    apt install -y wget bc 
## Install python3-pip and the dependencies to run 
## plotscm.py with SCM output
    apt install -y python3-pip && \
    pip3 install numpy && \
    pip3 install netcdf4 && \
    pip3 install matplotlib && \
    pip3 install pandas && \
    pip3 install xarray
## Download and build openmpi
https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.5.tar.gz 

Once these repository packages and the python librarie are installed, open-mpi is downloaded and built, . Then the openifs user is created and the openifs-48r1.1  directory is copied to the image.

...

docker build --no-cache --tag "openifs-48r1.1" .

Running OpenIFS-test using the docker image

Once a build has completed, the image can be checked by typing docker images , e.g.,

docker images

REPOSITORY          TAG       IMAGE ID       CREATED       SIZE
openifs-48r1.1      latest    f72ea92f010f   5 hours ago   2.15GB

...

docker run -it 'openifs-48r1.1'

This command will open an interactive session in the new container, in which the entry directory is the openifs-48r1 .1 directory, e.g.

docker run -it 'openifs-48r1.1'

OpenIFS environment variables are:
------------------------------------------------------
OIFS_ARCH=./arch/ecmwf/hpc2020
OIFS_CYCLE=48r1
OIFS_DATA_DIR=/perm/openifs/oifs_data/48r1/48r1
OIFS_EXEC=/home/openifs/openifs-48r1/build/bin/ifsMASTER.DP
OIFS_EXPT=/home/openifs/openifs-expt
OIFS_HOME=/home/openifs/openifs-48r1.1
OIFS_LOGFILE=/home/openifs/openifs-48r1.1/oifs_test_log.txt
OIFS_RUN_SCRIPT=/home/openifs/openifs-48r1/scripts/exp_3d
OIFS_TEST=/home/openifs/openifs-48r1/scripts/build_test


SCM environment variables are:
------------------------------------------------------
SCM_EXEC=/home/openifs/openifs-48r1/build/bin/MASTER_scm.DP
SCM_LOGFILE=/home/openifs/openifs-expt/scm_openifs/48r1/scm-projects/ref48r1/scm_run_log.txt
SCM_PROJDIR=/home/openifs/openifs-expt/scm_openifs/48r1/scm-projects
SCM_RUNDIR=/home/openifs/openifs-expt/scm_openifs/48r1/scm-projects/ref48r1
SCM_TEST=/home/openifs/openifs-48r1/scripts/scm
SCM_VERSIONDIR=/home/openifs/openifs-expt/scm_openifs/48r1

openifs@d1bd89ccc47f:~/openifs-48r1.1$48r1$ ls
CHANGES  COPYING  NOTICE  arch        ifs-source  oifs-config.edit_me.sh  scripts
CITE     LICENSE  README  bundle.yml  ifs-test    openifs-bundle

...

openifs@d1bd89ccc47f:~/openifs-48r1.1$ ./scripts48r1$ $OIFS_TEST/openifs-test.sh -cbt -s dockerj 2 --arch=""

where :

-c will clean the directory (not necessary for a fresh image or new container)   creates source directory in $OIFS_HOME, which is used to bring all the sources, e.g. ifs-source, ifs-test and any packages in the bundle (ecbuild, eccodes etc.), together in preparation for the build

-b  builds the source. This step creates the directory build in $OIFS_HOME, which is used to build and store-b will configure the bundle and build OpenIFS, i.e.,

• the OpenIFS double and single precision master executables (ifsMASTER.DP and ifsMASTER.SP, respectively) , which are used to run 3-D OpenIFS. The executables are located in  $OIFS_HOME/build/bin.
  • The location and name of the executable for OpenIFS is defined in the platform configuration file (oifs-config.edit_me.sh) as $OIFS_EXEC.

• the double and single precision Single Column Model (SCM) executables (MASTER_scm.DP and MASTER_scm.SP, respectively), which are used to run the SCM derived from OpenIFS. The executables are located in  $OIFS_HOME/build/bin.
  • The location and name of the executable for the SCM is defined in the platform configuration file (oifs-config.edit_me.sh) as $SCM_EXEC.

-t will run the ifs-test t21 tests, which comprise of

• 21 3-D OpenIFS forecast-only tests with and without chemistry
• 1 SCM test (based on TWP-ICE)

-j is an optional argument that allows the user to define the number of threads used to create and build openifs, with the above example using 2 threads. It should be noted that 

• If -j is not supplied as a command line argument, as in the above example, the number of threads will default to 8 threads, which can be problematic for older and lower spec systems. For example, it will not work with a 4 core intel m3. The above example uses 2 threads, which should work on most systems.
• The number of threads can be increased to speed up the build step, but how much the number of threads can be increased to will depend on the system spec. Hence, please speak the local system administration for guidance about appropriate settings. 

--arch="" means no arch file is required because all the libraries and paths have been set-up locally in the docker container. 

For more information on the openifs-test option please refer to Getting started and the linked page Openifs-test - build options.s docker defines the system as docker, which makes the script use the correct build commands

On a Macbook pro (M1) the build of OpenIFS, takes about 10 minutes, while successful completion of ifstest takes just over 3 minutes.

Notes about containers

Exiting the container

• The container can be exited by typing exit from the active container. Once exited all data and changes made within the container will be lost. Hence, if necessary, it is important to
  • Push any code changes back to a repo
  • If data is required either set-up a data volume or copy the data back to a local machine, using docker cp 

Check for existing containers, stopping, restarting

Running the SCM in the docker container

Once the above has been completed, you are ready to run the SCM. The scm_openifs  contains 3 testcases, each representative of different cloudy regimes

• DYCOMS - marine stratocumulus case
• BOMEX - trade-wind cumulus case 
• TWPICE - A multi-day deep convective case

The SCM is run using the callscm script, which is a wrapper for the main run.scm. Both scripts can found in $SCM_TEST , which is set in the oifs-config.edit_me.sh file to ${OIFS_HOME}/scripts/scm . 

callscm  includes default settings, which are the all cases, with a 450 s timestep and an experiment name of ref-oifs-scm . To run with these settings, enter the following

cd $OIFS_HOME
$SCM_TEST/callscm

callscm  (with defaults, i.e. no arguments) will run the DYCOMS case with the SCM and create an output directory in $SCM_RUNDIR/scmout_DYCOMS_ref-oifs-scm_450s , which contains the diagnostic output from the SCM. In addition, the file scm_run_log.txt will be created in $SCM_RUNDIR . This file contains the print output from the SCM, which is useful for checking all the sources and paths for a simulation. 

A user can change the defaults by using the available command-line options

callscm -h -c <case_name or list of case_names> -t <timestep or list of timesteps>
        -x <expt_name>
where :
-h is help which returns basic usage options and exits
-c case_name or list of case_names (space delimited) of the case study
   used for namelist and output directory. Default list is
   "DYCOMS BOMEX TWPICE" 
-t timestep or list of timesteps in seconds. The default is 450s. An
   example of a list is "1800 900 300" 
-x expt_name shortname to identify experiment. Default is ref-oifs-scm

 For example, if a user wanted to run the BOMEX case with timesteps of 1800 s and 900 s and an experiment name of "bomex_test", they would enter the following

$SCM_TEST/callscm -c BOMEX -t "1800 900" -x "bomex_test"

This command results in the following output directories $SCM_RUNDIR/scmout_BOMEX_bomex_test_900s  and scmout_BOMEX_bomex_test_1800s.

Example plotting script

Once the SCM has successfully, it will produce output in $SCM_RUNDIR/scmout_<casename>_<expt_name>_<timestep>, which contains some netcdf files. 

As part of the scm_openifs  download package we provide a python script $SCM_RUNDIR/plotscm.py , which can be used to plot some profiles at a certain time, 2-d time height plots and some scalar timeseries. This script will work without any changes for the DYCOMS case and the default settings of callscm by type the following

cd $SCM_RUNDIR
python3 plotscm.py

 For the defaults, the above will produce plot_DYCOMS  directory, which contains png plots

To plot the other cases or experiments with different names and/or timesteps to the defaults, at present, the user will need to edit plotscm.py appropriately. 

Basic docker commands and functionality

Check for existing containers

From a terminal on the host system, it is possible to list running and exited containers with the following

1. List running containers

   Code Block
   language bash theme Midnight
   $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d1bd89ccc47f openifs-48r1 "bash" 15 hours ago Up 5 seconds beautiful_pasteur

    If no container is running on your system, then only CAPITAL headings are returned with docker ps 

2. List all containers (running and exited)

   Code Block
   language bash theme Midnight
   $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d1bd89ccc47f openifs-48r1 "bash" 15 hours ago Exited (0) 4 seconds ago beautiful_pasteur

   Notice that the STATUS  is Exited, rather than Up, as in (1)

Start an existing container

If docker ps -a shows an exited container it can be restarted using the following

$ docker start -i <Container ID> 
# e.g. 
$ docker ps -a

CONTAINER ID   IMAGE               COMMAND       CREATED        STATUS                     PORTS     NAMES
d1bd89ccc47f   openifs-48r1      "bash"        15 hours ago   Exited (0) 4 seconds ago             beautiful_pasteur

$ docker start -i  d1bd89ccc47f

Exiting the container

1. The container can be exited by typing exit from the active container.

2. A container can be stopped from a terminal on the host system

   Code Block
   language bash theme Midnight
   docker stop <Container ID>

   It is also possible to stop multiple containers at once by adding more than one ID.

3. Using docker stop  will attempt a clean shutdown. It is also possible to stop the container by killing it 

   Code Block
   language bash theme Midnight
   docker kill <Container ID>

   
   

Removing containers and images

Remove a container with the following command

docker rm <Container ID>

It is possible to remove multiple containers at once by adding more than one ID. 

It is worth noting that if a container is running it cannot be removed without being forced or being stopped first

...