Dockerfiles
Two configurations are possible depending on how OpenIFS might be used in a container:
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
|
The Dockerfile describes the build process of the container image. Examples are provided in OpenIFS from version 43r3v1 onwards. The naming convention for Dockerfiles is:
Dockerfiles
The Dockerfile describes the build process of the container image. Examples for several of these files are provided in the OpenIFS git repository. The naming convention for Dockerfiles is as such:
Panel | ||||
---|---|---|---|---|
| ||||
Dockerfile.oifs<MODELRELEASE>.<GITHASH>oifs<RELEASE>.<NOTE>.<ARCH>.<TYPE> MODELRELEASE: A string generated from the IFS cycle, release number and OpenIFS version RELEASE: The OpenIFS release , e.g. 40r1v2 for CY40R1 OpenIFS release v2. GITHASH: The first five to seven characters of the OpenIFS repository git commit from which this image is built. 43r3v1. NOTENOTE: An optional comment string that describes Describes features of this build, in our examples this is either 'user' or 'root'. ARCH: The The architecture for which this image is built, e.g. x86_64, amd64, i386 etc.. TYPE: dev, test or bld. bld (build) to be used only for production images, such as from existing OpenIFS releases. dev should be used for images created from development branch commits.Type of build. Here 'bld' is used but could be changed to 'dev' or 'test' for example. |
Example: Dockerfile.oifs40r1v2oifs43r3v1.41537.user.x86_64.bld.bld
will generate an image of the OpenIFS 43r3v1 release.
Example Build Process
This section describes the generation process of a container image from the Dockerfile. You need the following files in your build directory:
1. Start by navigating to the docker directory in the OpenIFS distribution:
Code Block |
---|
% cd tools/docker |
2. Make a copy of one of the Dockerfile templates that can be found in this directory. There are two versions:
Code Block | ||||
---|---|---|---|---|
% ls Dockerfile*
Dockerfile.oifs43r3v1 | ||||
Panel | ||||
| ||||
lrwxrwxrwx 1 45 Dockerfile -> Dockerfile.oifs40r1v2.415374.root.x86_64. bld -rw-r----- 1 2.1K bld Dockerfile.oifs40r1v2 oifs43r3v1.415374.root user.x86_64.bld -rw-r----- 1 21M oifs40r1v2.415374d.tar.gz |
The Dockerfile can be obtained from the git repository. As described through the naming convention it will generate an image of OpenIFS 40r1 version 2. The partial git hash relates the Dockerfile (and image) to a specific git commit in the OpenIFS repository (in this case it relates to 415374d which is tagged as model release v2
). The note 'root
' indicates that when the image is loaded as a container the user will have root privileges. This will allow us to explore the directory structure of the container image. For convenience it is recommended to create a symbolic link to the generic name Dockerfile
.
The tar archive oifs40r1v2.41537.tar.gz
is created from the model sources after they have been checked out from the git repository (again the partial commit is specified). The Dockerfile will expect the tar archive in the same directory and the file name is specified.
The following command builds the image oifs40r1v2.415374.root. The generic command is docker build -t <image_name>
however at ECMWF four variables need to be set for network proxies in order to access the internet from within the container.
Panel | ||||
---|---|---|---|---|
| ||||
|
This starts the generation process of the image which contains the minimum of software that is required to run OpenIFS. The image is based on a Ubuntu Linux LTS version and in a first step the necessary developer tools are installed (e.g. GNU compiler, MPI and maths libraries). Afterwards the ecCodes library is obtained via download from the web and compiled with minimum settings. Then the OpenIFS sources are added from the tar archive, required environment variables are set and the model binaries are compiled. During a last step various file permissions are set and the model executable is moved to a globally accessible location.
At the end of the build process the successful image creation is shown as: Successfully tagged oifs40r1v2.415374.root:latest
Now we can verify the that the image is available and we will load it into a container using the docker run
command:
Panel | ||||
---|---|---|---|---|
| ||||
|
Our command line prompt has changed as we are now 'root
' inside the container. A file listing shows the directory structure inside the container.
Panel | ||||
---|---|---|---|---|
| ||||
root@38b1649e05b9:/# ls -F |
The OpenIFS model is installed in /oifs
. The ecCodes library is found in its default destination under /usr/local/lib
.
In order to run the acceptance test the file /oifs/t21test/job
needs editing:
EXPID=epc8
MASTER=/usr/local/bin/master.exe
When setting GRIB_SAMPLES_PATH
replace grib_api
with eccodes
.
In order to run the executable with mpirun as root the following option needs to be added: $OIFS_RUNCMD --allow-run-as-root $MASTER -e $EXPID
With the command 'exit
' the container is removed and all created or changed files in the container are lost. The next section will show how results can be retained and OpenIFS experiments can be run using a container.
Docker Image with User Account
Dockerfiles with the note 'user
' instead of 'root
' in their filename contain an additional build step wherein a user account is created in the Docker image.
In this case, once the image has been loaded into a container, we have taken on the identity 'oifs_user
' and no longer have root privileges. We are now in the home directory /home/oifs_user
and we have no longer access to in the files and sub-directories under /oifs
. As the model binaries were moved to /usr/local/bin
they can be accessed and an experiment can be run from the home directory location. The model sources are hidden from the user and only a copy of the t21test directory is available in the home directory.
Running OpenIFS Experiments in a Container
In this section we describe a method how the containerised version of OpenIFS can be used to run a case study interactively on the user's workstation (i.e. no batch job submission). Due to the temporary nature of containers all model results that are created in an experiment need to be stored outside the container. One possible method is to mount an external experiment directory inside the container. Data written to the mounted directory will be retained once the container is removed.
We assume an experiment directory has been created and shall be located at /scratch/rd/user/exp/.
Some preparation is required as this directory needs to contain all the necessary experiment data. Sub-directories are allowed however symbolic links to other file system locations will not work; hence the symbolic links created by oifs_run
at its first run will need to be manually created as sub-directories.
This experiment directory is mounted to the container when it is invoked:
Panel | ||||
---|---|---|---|---|
| ||||
|
A linked duplicate of the experiment directory can now be found inside the container in sub-directory /exp
with read and write permissions. If a Dockerfile with user account is used then the experiment directory needs to be mounted within the user home directory of the container.
In order to mount the external experiment directory successfully all the files or sub-directories therein need to have full read-write-executable access: chmod -R 777 /scratch/rd/user/exp
All the files in the mounted directory that were newly created or modified are owned by the container user, and seen from outside the container their file ownership will be different.
Invoking the Container from the OpenIFS run script
An alternative method of using OpenIFS in a container consists of including the docker call inside the oifs_run
script, replacing the execution of the model binary with mpirun. This method is also only suitable for running the model interactively (i.e. no batch job submission with aprun or srun). The modification in the script is as follows:
Panel | ||||
---|---|---|---|---|
| ||||
|
When using this method the Docker container environment remains relatively "concealed" from the model user and requires no further interaction with it.
Batch Job Submission
The use of Docker containers when running OpenIFS on HPC facilities has been tested with local computing support on the Piz Daint Cray XC50 at the Swiss National Supercomputing Centre in Decmber 2019. At present we do not yet offer this capability at ECMWF. This is work in progress and any updates will be reported here when available.
.
Crib Sheet: Important Docker commands
This section contains a listing of frequently used Docker commands some of which are specific to the ECMWF computing environment.
Start the Docker deamon on your machine (ECMWF):
Panel | ||||
---|---|---|---|---|
| ||||
|
which is actually: sudo /usr/bin/systemctl status docker
Which images are on my machine:
Panel | ||||
---|---|---|---|---|
| ||||
|
Which containers are running:
Panel | ||||
---|---|---|---|---|
| ||||
|
.bld
% cp Dockerfile.oifs43r3v1.user.x86_64.bld Dockerfile |
The note 'root
' indicates that OpenIFS is installed into a system directory /usr/local
whereas the 'user
' version installs into the user's account. Both Dockerfiles create a user called 'oifs'.
Which you choose depends on your application. The 'root' version might be more suited to a training workshop for example. We'll use the 'user
' version in this example.
3. Put a copy of the OpenIFS distribution tarfile as downloaded into the same directory as the Dockerfile.
Info |
---|
Make sure the version number of the tarfile matches that specified in the Dockerfile, the build process will unpack this file inside the container. |
You should have these files in your build directory (your version numbers may be different):
Code Block |
---|
-rw-r--r-- 1 glenn staff 4255 6 May 17:35 Dockerfile
-rw-r--r-- 1 glenn staff 30611901 6 May 18:13 oifs43r3v1.tar.gz |
4. Build the OpenIFS Docker image.
The following command builds the image oifs43r3v1.user. Change 'user' to 'root' if building the other variant.
Code Block |
---|
% docker build -t oifs43r3v1.user . # note the trailing '.' to build in the current dir |
Panel | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
For the docker image on the workstations at ECMWF four variables need to be set for network proxies in order to access the internet from within the container.
|
This runs the build process of the image which contains the minimum of software that is required to run OpenIFS.
The image is based on a Ubuntu Linux LTS version. After downloading the base Ubuntu image, the Dockerfile executes the following steps: the necessary developer tools are installed (e.g. GNU compiler, MPI and maths libraries).; the ecCodes library is downloaded and compiled; the OpenIFS sources are unpacked from the tar archive, and the model binaries are compiled. The last step sets file permissions and the model executable is moved to an install location.
At the end of the build process the successful image creation is shown as: Successfully tagged oifs43r3v1.user:latest
Running the docker image
We can verify that the image is available and load it into a container using the docker run
command:
Code Block |
---|
% docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
oifs43r3v1.user latest 982f6e82bb93 39 minutes ago 873MB
ubuntu latest 72300a873c2c 13 days ago 64.2MB
% docker run -it oifs43r3v1.user
oifs@40a923f11202:~$ |
Our command line prompt has changed as we are now the user 'oifs
' inside the container.
A directory listing shows the following structure (your version numbers may be different):
Code Block |
---|
oifs@40a923f11202:~$ ls
oifs43r3v1
oifs@40a923f11202:~$ ls oifs43r3v1
CHANGES COPYING NOTICE READMEs examples make oifs-config.sh t21test tools
CITE LICENSE README bin fcm oifs-config.editme.sh src t21test_xios
oifs@40a923f11202:~$ |
The compiled model executables can be found in and can be moved to another install location:
Code Block |
---|
oifs@40a923f11202:~$ ls oifs43r3v1/make/gnu-opt/oifs/bin
getres.exe grib_set_vtable.exe master.exe spinterp.exe timeint.exe vod2uv.exe
gptosp.exe intsst.exe rgrid.exe sptogp.exe uvtovod.exe |
The ecCodes library is found in its default destination under /usr/local/lib
.
Info |
---|
If using the 'root' Dockerfile, the install location will be in /usr/local and not the home directory of the 'oifs' user. In order to run the acceptance test as the root user the file
In order to run the executable with the command |
With the command 'exit
' the container is removed and all created or changed files in the container are lost. The next section will show how results can be retained and OpenIFS experiments can be run using a container.
Running OpenIFS experiments in a container
Due to the temporary nature of containers all model results that are created in an experiment need to be stored outside the container. One possible method is to mount an external experiment directory inside the container. Data written to the mounted directory will be retained once the container is removed.
Assume an experiment directory at /scratch/user/exp/.
Info |
---|
Sub-directories are allowed however symbolic links to other file system locations will not work; the symbolic links created by |
This experiment directory is mounted to the container when it is invoked:
Code Block |
---|
docker run -v /scratch/user/exp:/home/oifs/exp:rw -it oifs43r3v1.user |
A mount of the experiment directory can be found inside the container in sub-directory /exp
with read and write permissions.
In order to mount the external experiment directory successfully, all the files or sub-directories need to have full read-write-executable access: chmod -R 777 /scratch/user/exp
All the files in the mounted directory that were newly created or modified are owned by the container user, seen from outside the container their file ownership will be different.
Invoking the container from the OpenIFS run script
An alternative method of using OpenIFS in a container consists of including the docker call inside the oifs_run
script, replacing the execution of the model binary with mpirun.
This method is only suitable for running the model interactively (i.e. no batch job submission). The modification in the script is as follows:
Panel | ||||
---|---|---|---|---|
| ||||
|
When using this method the Docker container environment remains "concealed" from the model user and requires no further interaction with it.
Batch Job Submission
The use of Docker containers when running OpenIFS on HPC facilities has been tested successfully and with good scalability on the Piz Daint Cray XC50 at the Swiss National Supercomputing Centre in December 2019 using local computing support. At present we do not yet offer this capability at ECMWF.
Crib Sheet: Important Docker commands
This section contains a list of frequently used Docker commands.
Start the Docker deamon on your machine (if not already running):
Panel | ||||
---|---|---|---|---|
| ||||
|
which is actually: sudo /usr/bin/systemctl status docker
ECMWF users may need to contact servicedesk to request permission to run Docker.
Which images are on my machineBuild docker image:
Panel | ||||
---|---|---|---|---|
| ||||
|
At ECMWF: docker build -t oifs --build-arg http_proxy="$http_proxy" --build-arg ftp_proxy="$ftp_proxy" --build-arg https_proxy="$https_proxy" --build-arg no_proxy="$no_proxy" .
|
Which containers are runningRun docker images in container:
Panel | ||||
---|---|---|---|---|
| ||||
|
Do this first: docker login eccr.ecmwf.int
|
Build docker image:
Panel | ||||
---|---|---|---|---|
| ||||
|
Info |
---|
At ECMWF, use the proxy arguments: |
|
|
|
Run docker images in container:
Panel | ||||
---|---|---|---|---|
| ||||
|
Excerpt Include | ||||||
---|---|---|---|---|---|---|
|
Then push it to harbor, manually specifying version number. Careful: Existing version numbers are overwritten! docker push eccr.ecmwf.int/openifs/oifs:0.0.1
Pull image from repository into memory: docker pull eccr.ecmwf.int/openifs/oifs:0.0.1