View Source

Ways to modify OpenIFS

There are several ways in which code changes to OpenIFS could be managed and these are described in this article.

We do not recommend that the source code in the oifs/src directory is edited directly as this can make it difficult to track changes and update to new versions. We recommend all code changes are maintained outside the main src directory.

Copy entire source

Copy the entire oifs/src to (for example) oifs/mysrc and work on the copy. The build configuration files can also be copied if required to, say, oifs/mymake. For instance you might make a new version of oifs/make/fcmcfg/oifs.cfg for your own changes.

Multiple copies of the src directory could be used for different experiments. The use of a version control system is recommended to track changes. The difficulty with this approach is the full source needs to be copied which makes integrating changes different and changed files are hard to see.

Copy only changed/new source

Leave code in oifs/src 'as-is' and place only modified or new code in separate directories but link to pre-compiled code in oifs/make. This has the advantage that multiple code developments can be maintained in separate directories; code changes are easy to see and integrating new versions of the model source straightforward. A version control system can also be used.

This makes use of a facility provided by the FCM build software called 'inherited builds'. This is explained below in a series of examples.

Inherited builds with FCM

In essence an inherited build is one in which a pre-build (i.e. compilation) of OpenIFS is used or inherited by a new FCM configuration file placed in a separate directory. The new FCM configuration file only needs to specify what's different about the build, either the sources or compilation settings. The rest of the information is obtained by telling FCM to look at the already compiled OpenIFS. A user may have a number of these separate configuration files each in separate directories for developing or testing independent code changes.

The information on this page is correct for OpenIFS version 38r1. For OpenIFS 40r1 please change 'build.prop' to 'oifs.prop'. This page will be updated with new examples.

For assistance either email openifs-support@ecmwf.int or use the OpenIFS User Forums.

oifs_inherit_examples.tar.gz.

Save the tarfile in your oifs directory, where the main OpenIFS tarfile was unpacked.

This tarfile will create several directories: mymake, mysrc1, mysrc2 and mysrc3.

Note! Before running these examples, make sure you have built the model by using the 'fcm make' command in the 'oifs/make' directory in the normal way as these examples rely on that build being present

Example1: Modifying compilation options

HowTo customise compiling OpenIFS explains how environment variables and editing the configuration files can alter the compiler settings for OpenIFS. In this simple example, an inherited build is used to achieve the same effect by creating a separate configuration file.

The advantage of this approach is that the source code and configuration files of OpenIFS are left unaltered and do not need any environment variables to be set.

Create directories for your own configuration files and source code. For example, create 'oifs/mymake' in which to put your own configuration files and 'oifs/mysrc' to put your own source code:

% cd oifs
% mkdir mysrc mymake
% ls
COPYING  ChangeLog  INSTALL  LICENSE  NOTICE  README  html  make  mymake  mysrc  src  t21test
% cd mymake

As usual the OpenIFS source code is in 'src' with the compilation configurations files used by the FCM build software in 'make/fcmcfg'.

In the 'oifs/mymake' directory, create a file called 'change-fcflags.cfg' with the following content:

use = ../make
include = ../make/fcmcfg/oifs.cfg
#  Add array bound checking to some routines for debugging
build.prop{fc.flags}[ifs/phys_radi/uvclr.F90] = $OIFS_FFLAGS -fcheck=bounds

The 'use' statement tells FCM that a pre-compiled version of OpenIFS exists in the directory '../make'. This pre-compiled version is 'inherited' through the 'use' statement. All targets (i.e. the object files), source and compilation settings are inherited.

Now run the fcm command to build the model with this change (make sure you have already compiled OpenIFS in the normal way in the oifs/make directory as this relies on the .o files being available):

% fcm make -v -f change-fcflags.cfg
[init] make
[init] make config-parse
[info] config-file=/openifs/inherit_tests/oifs/mymake/change-fcflags.cfg
[info] config-file= - /openifs/inherit_tests/oifs/make/fcmcfg/oifs.cfg
[info] config-file= -  - /openifs/inherit_tests/oifs/make/fcmcfg/x86_64-gnu-opt.cfg
[info] use=/openifs/inherit_tests/oifs/make
[info] sources: total=2192, analysed=0, elapsed-time=0.6s, total-time=0.0s
[info] target-tree-analysis: elapsed-time=13.3s
[info] compile    0.2 M uvclr.o              <- ifs/phys_radi/uvclr.F90
[info] link       1.7 M master.exe           <- programs/master.F90
[info] compile   targets: modified=1, unchanged=2070, total-time=0.2s
[info] compile+  targets: modified=0, unchanged=626, total-time=0.0s
[info] ext-iface targets: modified=0, unchanged=1247, total-time=0.0s
[info] install   targets: modified=0, unchanged=120, total-time=0.0s
[info] link      targets: modified=1, unchanged=0, total-time=1.7s
[info] TOTAL     targets: modified=2, unchanged=4063, elapsed-time=25.3s
[done] make build          # 26.3s
[done] make                # 26.6s

Notice the lines:

[info] compile    0.2 M uvclr.o              <- ifs/phys_radi/uvclr.F90
[info] link       1.7 M master.exe           <- programs/master.F90

showing that FCM has noted the change in the compiler options for the file uvclr.F90, recompiled it and relinked the main executable.

If you now check the contents of this directory, you'll see:

% ls -R
build  change-fcflags.cfg
./build:
bin  o
./build/bin:
master.exe
./build/o:
uvclr.o

As in the main 'make' directory, FCM has created a 'build' subdirectory which in this case only contains the object file of the routine uvclr and a new executable. You now have the original executable in 'oifs/make/build/bin/master.exe' and this modified one for testing/debugging.

To run this new executable as a test, edit the job script in the t21test directory and alter the line: MASTER= to point to the location of the newly built executable.

Experiment with this example by adding additional lines in the change-fcflags.cfg file and verify that the 'fcm make' command shows those routines being recompiled.

To verify the new compiler options are being used add the -vv flag to the fcm make command to see all output i.e. fcm make -vv or look in the FCM log file which can be found in the file.fcm-make/log.

Example 2: Editing existing code

In this example, we'll edit an existing source file. First make a new directory to hold the modified source code and copy a source file:

% mkdir -p mysrc/ifs/control
% cp src/ifs/control/cnt0.F90 mysrc/ifs/control/cnt0.F90
  Edit mysrc/ifs/control/cnt0.F90 using your favourite editor to add some print statements.

The path to any copies of OpenIFS code must be the same as they are laid out under the 'src' directory. Otherwise, the build command 'fcm' will not be able to match the sources.

Then create the configuration file for FCM that looks like this:

use = ../make
build.source = $HERE/../mysrc
build.prop{fc.flags}[ifs/control/cnt0.F90] = -g -O1 -fcheck=bounds

As in example 1, the 'use' statement tells the build command FCM where to find the pre-compiled OpenIFS and all that's needed here are the changes. In this case, we now need to tell FCM where our new source code is using the 'build.source' statement - the special fcm $HERE variable means "the directory which contains this configuration file". We'll also change the compiler options for this file. Note we don't need to include the oifs.cfg file as in the previous example, as this file doesn't contain any references to the OIFS_FFLAGS variable.

% fcm make -v -f change-src1.cfg
[init] make
[init] make config-parse
[info] config-file=/openifs/inherit_tests/oifs/mymake/change-src1.cfg
[info] use=/openifs/inherit_tests/oifs/make
[done] make dest-init      # 0.0s
[init] make build 
[info] sources: total=2192, analysed=0, elapsed-time=0.4s, total-time=0.0s
[info] target-tree-analysis: elapsed-time=13.9s
[info] compile    0.1 M cnt0.o               <- ifs/control/cnt0.F90
[info] link       1.7 M master.exe           <- programs/master.F90
[info] compile   targets: modified=1, unchanged=2070, total-time=0.1s
[info] compile+  targets: modified=0, unchanged=626, total-time=0.0s
[info] ext-iface targets: modified=0, unchanged=1247, total-time=0.0s
[info] install   targets: modified=0, unchanged=120, total-time=0.0s
[info] link      targets: modified=1, unchanged=0, total-time=1.7s
[info] TOTAL     targets: modified=2, unchanged=4063, elapsed-time=25.6s
[done] make build          # 26.2s
[done] make                # 26.4s

Note that FCM has: seen the new version of cnt0.F90, recompiled it and relinked it to the pre-compiled object code in oifs/make. The new executable can be found in the build/bin sub-directory. As in example 1, to try out this new executable in the t21test directory edit the script and change the line MASTER= to point to the location of this executable.

A more advanced example with modules

We'll now add to this and change a fortran file with a MODULE. In this case, FCM will notice that the module interface has changed and recompile all source files (that we have not changed) that 'USE' this module:

% mkdir mysrc/ifs/module
% cp src/ifs/module/yemct0.F90 mysrc/ifs/module
Edit yemct0.F90 - add a new logical variable

The module YEMCT0 is only USE'd in the file ifs/control/cnt3.F90. We have not made any changes to cnt3.F90 but adding a new variable to the source file yemct0.F90 will change its interface and cnt3.F90 will need to be recompiled.

We do not need to make any changes to the configuration file:

% fcm make -v -f change-src.cfg 
[init] make
[init] make config-parse
[info] config-file=/openifs/inherit_tests/oifs/mymake/change-src2.cfg
[info] use=/openifs/inherit_tests/oifs/make
[done] make dest-init
[init] make build 
[info] analyse  0.0 ifs/module/yemct0.F90
[info] analyse  0.0 ifs/control/cnt0.F90
[info] sources: total=2192, analysed=2, elapsed-time=0.5s, total-time=0.0s
[info] target-tree-analysis: elapsed-time=13.3s
[info] compile    0.0 M yemct0.o             <- ifs/module/yemct0.F90
[info] compile+   0.0 M yemct0.mod           <- ifs/module/yemct0.F90
[info] ext-iface  0.0 U cnt3.intfb.h         <- ifs/control/cnt3.F90
[info] compile    0.0 U cnt2.o               <- ifs/control/cnt2.F90
[info] ext-iface  0.0 U cnt0.intfb.h         <- ifs/control/cnt0.F90
[info] compile    0.1 U cnt3.o               <- ifs/control/cnt3.F90
[info] compile    0.1 M cnt0.o               <- ifs/control/cnt0.F90
[info] link       1.6 M master.exe           <- programs/master.F90
[info] compile   targets: modified=2, unchanged=2069, total-time=0.3s
[info] compile+  targets: modified=1, unchanged=625, total-time=0.0s
[info] ext-iface targets: modified=0, unchanged=1247, total-time=0.0s
[info] install   targets: modified=0, unchanged=120, total-time=0.0s
[info] link      targets: modified=1, unchanged=0, total-time=1.6s
[info] TOTAL     targets: modified=4, unchanged=4061, elapsed-time=25.1s
[done] make build          # 25.7s
[done] make                # 25.9s

The lines:

[info] analyse  0.0 ifs/module/yemct0.F90
[info] analyse  0.0 ifs/control/cnt0.F90

show that FCM has noticed we now have 2 modified source code files and has analysed them for changes.

The following lines:

[info] compile    0.0 M yemct0.o             <- ifs/module/yemct0.F90
[info] compile+   0.0 M yemct0.mod           <- ifs/module/yemct0.F90
[info] ext-iface  0.0 U cnt3.intfb.h         <- ifs/control/cnt3.F90
[info] compile    0.0 U cnt2.o               <- ifs/control/cnt2.F90
[info] ext-iface  0.0 U cnt0.intfb.h         <- ifs/control/cnt0.F90
[info] compile    0.1 U cnt3.o               <- ifs/control/cnt3.F90
[info] compile    0.1 M cnt0.o               <- ifs/control/cnt0.F90
[info] link       1.6 M master.exe           <- programs/master.F90

show that not only have the new versions of yemct0.F90 and cnt0.F90 been compiled, but also the unchanged, original, source for cnt3.F90 has been recompiled because it has a USE statement for yemct0. Also, because cnt3 is called by cnt2, which again we haven't altered, then this has is also recompiled.

The directory build/o now contains .o files for cnt0, cnt2, cnt3 and yemct0 and the new executable in build/bin/master.exe.

The directory mysrc we used here can contain as many source files from the main src directory as you like; either changed or unchanged.

Example 3: Adding new code

In this last example, we'll extend example 2 but this time add some new code and modify OpenIFS to call it.

% mkdir mysrc/newsrc
% cd mysrc/newsrc
% cat > newsub.F90 << EOF
subroutine newsub
print *,'hello openifs'
end subroutine
EOF
% cd ../ifs/control
% vi cnt0.F90
Edit cnt0.F90 and add 'call newsub' to the start of the executable code.

Having added a new subroutine and a call to it from our modified cnt0.F90 file, we need to make an additional change to the configuration file:

use = ../make
build.source = $HERE/../mysrc
build.ns-incl = newsrc algor ifsaux ifs surf trans openifs programs/master.F90
build.prop{fc.flags}[ifs/control/cnt0.F90] = -g -O1 -fcheck=bounds

The statement build.ns-incl lists the directories under mysrc (and the original src) that FCM is allowed to analyse and compile code. The new directory we created needs to be added to this list. Compare it with the same statement in the default configuration file: oifs/make/fcmcfg/oifs.cfg. Without this addition, the command 'fcm make' would result in an undefined reference to the subroutine newsub as it will not have compiled the code.

With this additional line, when we compile we get:

% fcm make -v -f new-src.cfg 
[init] make
[init] make config-parse
[info] config-file=/openifs/inherit_tests/oifs/mymake/new-src.cfg
[done] make dest-init
[init] make build
[info] analyse  0.0 newsrc/newsub.F90
[info] analyse  0.0 ifs/module/yemct0.F90
[info] analyse  0.0 ifs/control/cnt0.F90
[info] sources: total=2193, analysed=3, elapsed-time=0.5s, total-time=0.0s
[info] target-tree-analysis: elapsed-time=13.2s
[info] compile    0.0 M yemct0.o             <- ifs/module/yemct0.F90
[info] compile+   0.0 M yemct0.mod           <- ifs/module/yemct0.F90
[info] ext-iface  0.0 U cnt3.intfb.h         <- ifs/control/cnt3.F90
[info] compile    0.0 U cnt2.o               <- ifs/control/cnt2.F90
[info] ext-iface  0.0 U cnt0.intfb.h         <- ifs/control/cnt0.F90
[info] compile    0.1 U cnt3.o               <- ifs/control/cnt3.F90
[info] compile    0.1 M cnt0.o               <- ifs/control/cnt0.F90
[FAIL] make build          # 25.4s
[FAIL] make                # 25.6s
[FAIL] mpif90 -obin/master.exe /media/hugetmp/Backedup/openifs/inherit_tests/oifs/make/build/o/master.o -L/var/tmp/tmpdir/user/jtmp.6858/LvpNtq0TxD -lmaster -fopenmp /home/rd/openifs/software/grib_api/1.9.18/grib_api-gcc-4.5.0/lib/libgrib_api_f90.a /home/rd/openifs/software/grib_api/1.9.18/grib_api-gcc-4.5.0/lib/libgrib_api.a -L/usr/local/apps/lapack/3.4.1/LP64 -llapack -lblas -lm # rc=1
[FAIL] /var/tmp/tmpdir/user/jtmp.6858/LvpNtq0TxD/libmaster.a(cnt0.o): In function `cnt0':
[FAIL] /media/hugetmp/Backedup/openifs/inherit_tests/oifs/mysrc3/ifs/control/cnt0.F90:101: undefined reference to `newsub_'
[FAIL] collect2: ld returned 1 exit status

FCM correctly analysed the new code now that the configuration file lists the directory containing it, but FCM failed to compile the file newsrc/newsub.F90.

Why? The reason is that the code does not have enough information to determine where the subroutine 'newsub' comes from. As it's not a module (no USE statement), nor has an explicit interface (no #include "newsub.intfb.h") FCM will assume it comes from an external library.

We can correct the problem in two ways.

Solution 1. The preferred way is to add an explicit interface for the subroutine call making it clear the source code is not from an external library. This also has the advantage that the compiler can check arguments passed to the subroutine are correct. Interface files for subroutines are generated automatically by FCM and do not need to be created by the user.

#include "su0yoma.intfb.h"
#include "su0yomb.intfb.h"
#include "newsub.intfb.h"
!     ------------------------------------------------------------------
call newsub

Building the model now works:

% fcm make -v -f new-src.cfg
[init] make
[init] make build
[info] analyse  0.0 newsrc/newsub.F90
[info] analyse  0.0 ifs/module/yemct0.F90
[info] analyse  0.0 ifs/control/cnt0.F90
[info] sources: total=2193, analysed=3, elapsed-time=0.5s, total-time=0.0s
[info] target-tree-analysis: elapsed-time=13.4s
[info] compile    0.0 M newsub.o             <- newsrc/newsub.F90
[info] ext-iface  0.0 M newsub.intfb.h       <- newsrc/newsub.F90
[info] compile    0.0 M yemct0.o             <- ifs/module/yemct0.F90
[info] compile+   0.0 M yemct0.mod           <- ifs/module/yemct0.F90
[info] ext-iface  0.0 U cnt3.intfb.h         <- ifs/control/cnt3.F90
[info] compile    0.0 U cnt2.o               <- ifs/control/cnt2.F90
[info] ext-iface  0.0 U cnt0.intfb.h         <- ifs/control/cnt0.F90
[info] compile    0.1 U cnt3.o               <- ifs/control/cnt3.F90
[info] compile    0.1 M cnt0.o               <- ifs/control/cnt0.F90
[info] link       1.7 M master.exe           <- programs/master.F90
[info] compile   targets: modified=3, unchanged=2069, total-time=0.3s
[info] compile+  targets: modified=1, unchanged=625, total-time=0.0s
[info] ext-iface targets: modified=1, unchanged=1247, total-time=0.0s
[info] install   targets: modified=0, unchanged=120, total-time=0.0s
[info] link      targets: modified=1, unchanged=0, total-time=1.7s
[info] TOTAL     targets: modified=6, unchanged=4061, elapsed-time=25.2s
[done] make build          # 25.8s
[done] make                # 26.0s

The compilation now succeeds. Note that as well as analysing the new code, FCM has also compiled newsub.F90 and created the interface for it, newsub.intfb.h (found in the build/include subdirectory).

Solution 2. Another way to correct the problem is to explicitly tell FCM that the master program depends on the code in our new directory. If you have many new source files and they do not use interface statements you will need to use this approach instead (or as well as - we recommend using interfaces wherever possible).

In this case, rather than edit cnt0.F90 (though its still preferable to include the #include "newsub.intfb.h" statement in cnt0.F90), add another statement to the FCM configure file:

use = ../make
build.source = $HERE/../mysrc
build.ns-incl = newsrc algor ifsaux ifs surf trans openifs programs/master.F90
build.prop{ns-dep.o}[programs/master.F90] = newsrc algor trans surf ifs ifsaux openifs
build.prop{fc.flags}[ifs/control/cnt0.F90] = -g -O1 -fcheck=bounds

The statement 'build.prop{ns-dep.o}' means the file 'programs/master.F90' depends on the object files (.o) created from the source in the directories listed. By adding the entry 'newsrc' to the list (see the original statement in oifs/make/fcmcfg/oifs.cfg) this will force FCM to compile all the source code it finds in the newsrc directory. If this behaviour is not desired the statement build.ns-excl can be used to list directories and/or files that should be excluded from the compilation.

Tips

We hope these examples serve to illustrate how to alter and add to the OpenIFS source code using FCM and although simple, the examples show the basics on which to build more complex changes. Whether you choose to use inherited builds or work with copies of the code is largely down to personal preference but we offer a number of suggestions to make life easier:

If not using inherited builds, always work with copies of the code from the main 'src' directory, never edit the files in src directly - unless you are using a source versioning system, tracking changes and integrating new versions of the code is difficult.
Inherited builds work well with small and large code changes and help to separate code changes and additions. Any code patches to the main src are automatically linked with inherited builds whereas if you maintain separate copies of the source you'll need to update every copy.
Only copy the FCM configuration files from make/fcmcfg if you need to and then only oifs.cfg rather than the hardware and compiler settings. It helps to keep all hardware & compiler specific configuration files in one place (ie. make/fcmcfg) and either make links to them from other directories or, if you are not using inherited builds, use 'include' statements in your modified oifs.cfg file to always refer to them in the default location make/fcmcfg.