Page History

...

1. Ensure your environment is clean by running:

   No Format
   module reset

   
   

2. Create a directory for this tutorial and cd into it:

   No Format
   mkdir -p compiling_tutorial cd compiling_tutorial

   
   

3. We are going to use a simple program that will display versions of different libraries linked to it. Create a file called versions.c using your favourite editor with the following contents:

   Code Block
   language cpp title versions.c collapse true
   #include <stdio.h> #include <hdf5.h> #include <netcdf.h> #include <eccodes.h> int main() { #if defined(__INTEL_LLVM_COMPILER) printf("Compiler: Intel LLVM %d\n", __INTEL_LLVM_COMPILER); #elif defined(__INTEL_COMPILER) printf("Compiler: Intel Classic %d\n", __INTEL_COMPILER); #elif defined(__clang_version__) printf("Compiler: Clang %s\n", __clang_version__); #elif defined(__GNUC__) printf("Compiler: GCC %d.%d.%d\n", __GNUC__, __GNUC_MINOR__, __GNUC_PATCHLEVEL__); #else printf("Compiler information not available\n"); #endif     // HDF5 version unsigned majnum, minnum, relnum; H5get_libversion(&majnum, &minnum, &relnum); printf("HDF5 version: %u.%u.%u\n", majnum, minnum, relnum); // NetCDF version printf("NetCDF version: %s\n", nc_inq_libvers()); // ECCODES version printf("ECCODES version: "); codes_print_api_version(stdout); printf("\n"); return 0; }

   
   

4. Try to naively compile this program with:

   No Format
   $CC -o versions versions.c

   
   

5. The compilation above fails as it does not know where to find the different libraries. We need to add some additional flags so the compiler can find both the include headers and link to the actual libraries. 

   1. Let's use the existing software installed on the system with modules, and benefit from the corresponding environment variables *_DIR which are defined in them to manually construct the include and library flags:

      No Format
      $CC -o versions versions.c -I$HDF5_DIR/include -I$NETCDF4_DIR/include -I/$ECCODES_DIR/include -L$HDF5_DIR/lib -lhdf5 -L$NETCDF4_DIR/lib -lnetcdf -L$ECCODES_DIR/lib -leccodes

      Load the appropriate modules so that the line above completes successfully and generates the versions executable:

      Expand
      title Solution
      You will need to load the following modules to. have those variables defined.: No Format module load hdf5 netcdf ecmwf-toolbox $CC -o versions versions.c -I$HDF5_DIR/include -I$NETCDF4_DIR/include -I/$ECCODES_DIR/include -L$HDF5_DIR/lib -lhdf5 -L$NETCDF4_DIR/lib -lnetcdf -L$ECCODES_DIR/lib -leccodes The versions executable should now be in your current directory: No Format ls versions

      
      

   2. Run ./versions. You will get an error such as the one below:

      No Format
      $ ./versions ./versions: error while loading shared libraries: libhdf5.so.200: cannot open shared object file: No such file or directory

      
      

      While you passed the location of the libraries at compile time, the program cannot not find the libraries at runtime. Inspect the executable with ldd to see what libraries are missing

      Expand
      title Solution
      ldd is a utility that prints the shared libraries required by each program or shared library specified on the command line: No Format $ ldd versions linux-vdso.so.1 (0x00007ffffada9000) libhdf5.so.200 => not found libnetcdf.so.19 => not found libeccodes.so => not found libc.so.6 => /lib64/libc.so.6 (0x000014932ff36000) /lib64/ld-linux-x86-64.so.2 (0x00001493302fb000)

      
      

   3. Can you make that program run successfully?

      Expand
      title Solution
      While you passed the location of the libraries at compile time, the program cannot not find the libraries at runtime. There are two solutions: Use the environment variable LD_LIBRARY_PATH- not recommended for long term Use the environment variable LD_LIBRARY_PATH. Check that ldd with the environment variable defined reports all libraries found: No Format LD_LIBRARY_PATH=$HDF5_DIR/lib:$NETCDF4_DIR/lib:$ECCODES_DIR/lib ldd ./versions Use  rpath  - robust solution  Use the  rpath strategy to engrave the library paths into the actual executable at link time, so it always knows where to find them at runtime. Rebuild your program with: No Format $CC -o versions versions.c -I$HDF5_DIR/include -I$NETCDF4_DIR/include -I/$ECCODES_DIR/include -L$HDF5_DIR/lib -Wl,-rpath,$HDF5_LIB -lhdf5 -L$NETCDF4_DIR/lib -Wl,-rpath,$NETCDF4_DIR/lib -lnetcdf -L$ECCODES_DIR/lib -Wl,-rpath,$ECCODES_DIR/lib -leccodes Check that ldd with the environment variable defined reports all libraries found: No Format unset LD_LIBRARY_PATH ldd ./versions Final version For convenience, all those software modules define the *_INCLUDE and *_LIB variables: No Format module show hdf5 netcdf4 ecmwf-toolbox | grep -e _INCLUDE -e _LIB You can use those in for your compilation directly, with the following simplified compilation line: No Format $CC -o versions versions.c $HDF5_INCLUDE $NETCDF4_INCLUDE $ECCODES_INCLUDE $HDF5_LIB $NETCDF4_LIB $ECCODES_LIB Now you can run your program without any additional settings: No Format $ ./versions Compiler: GCC 8.5.0 HDF5 version: <HDF5 version> NetCDF version: <NetCDF version> of <date> $ ECCODES version: <ecCodes version>

      
      

6. Can you rebuild the program so it uses the "old" versions of all those libraries in modules? Ensure the output of the program matches the versions loaded in modules? Do the same with the latest. 

   Expand
   title Solution
   You need to load the desired versions or the modules: No Format module load hdf5/old netcdf4/old ecmwf-toolbox/old And then rebuild and run the program: No Format $CC -o versions versions.c $HDF5_INCLUDE $NETCDF4_INCLUDE $ECCODES_INCLUDE $HDF5_LIB $NETCDF4_LIB $ECCODES_LIB ./versions The output should match the versions loaded by the modules:  No Format echo $HDF5_VERSION $NETCDF4_VERSION $ECCODES_VERSION Repeat the operation with  No Format module load --latest hdf5 netcdf4 ecmwf-toolbox

   
   

7. To simplify the build process, let's create a simple Makefile for this program. With your favourite editor, create a file called Makefile in the same directory with the following contents:

   Code Block
   language bash title Makefile collapse true
   # # Makefile # # Make sure all the relevant modules are loaded before running make EXEC = versions # TODO: Add the necessary variables into CFLAGS and LDFLAGS definition CFLAGS = LDFLAGS = all: $(EXEC) %: %.c $(CC) -o $@ $^ $(CFLAGS) $(LDFLAGS) test: $(EXEC) ./$(EXEC) clean: rm -f $(EXEC)

   You can test it works by running:

   No Format
   make clean ldd test

   
   

   Expand
   title Solution
   Edit the Makefile and add the *_INCLUDE and *_LIB variables which are defined by the modules: Code Block language bash title Makefile collapse true # # Makefile # # Make sure all the relevant modules are loaded before running make EXEC = versions CFLAGS = $(HDF5_INCLUDE) $(NETCDF4_INCLUDE) $(ECCODES_INCLUDE) LDFLAGS = $(HDF5_LIB) $(NETCDF4_LIB) $(ECCODES_LIB) all: $(EXEC) %: %.c $(CC) -o $@ $^ $(CFLAGS) $(LDFLAGS) test: $(EXEC) ./$(EXEC) ldd: $(EXEC) @ldd $(EXEC) | grep -e netcdf.so -e eccodes.so -e hdf5.so clean: rm -f $(EXEC) Then run it with: No Format make clean ldd test

   
   

...

#!/bin/bash
#SBATCH -J build_cdo
#SBATCH -o %x.out
#SBATCH -n 8

set -x
set -e
set -u
set -o pipefail

# Get the URL and VERSION for the latest CDO available
URL=$(curl -s https://code.mpimet.mpg.de/projects/cdo/files | grep attachments/download  | sed -e "s:.*href=\"\(.*\)\".*:https\://code.mpimet.mpg.de\1:" | head -n 1)
VERSION=$(echo $URL | sed -e "s:.*/cdo-\(.*\).tar.gz:\1:")

# TODO: Define installation prefix and Buildbuild directory
# Hint: Use somewhere in your PERM for installation.
PREFIX=
BUILDDIR=

# Move to our BUILD DIRECTORY
mkdir -p $BUILDDIR
cd $BUILDDIR

# Download source
[ -f cdo-$VERSION.tar.gz ] || wget $URL
[ -d cdo-$VERSION ] || tar xvf cdo-$VERSION.tar.gz
cd cdo-$VERSION

# TODO: Define the environment for the build 


# TODO: Configure the build


# Make sure we start a clean build
make clean

# Build
make -j $SLURM_NTASKS

# Install
make install

# Check installed binary
$PREFIX/bin/cdo -V

We will take a step-by-step approach, taking the build_cdo.sh template above as the starting point.

First thing we need to do is to decide where to install your personal CDO. A good choice would be using your PERM space, and we may use the same structure as the production installation. Because in the script we already have a variable called $VERSION containing the CDO version to install, we can also use that. This way we could have multiple versions of the same package installed alongside should we require it in the future:

# Define installation prefix and build directory
PREFIX=$PERM/apps/cdo/$VERSION

Next comes the decision on what to use for the build itself. Since it is a relatively small build, for performance you might use $TMPDIR which is local to the node.

BUILDDIR=$TMPDIR

However, if you are going to submit it as a batch job, the directory will be wiped at the end. While you are putting the build script together, it may be more practical to have the build directory somewhere that is not deleted after a failed build so you can inspect output files and troubleshoot. As an example, we could pick the following directory in PERM:

BUILDDIR=$PERM/apps/cdo/build

Let's look at the environment for the build. All of dependencies listed above are already available, so we may leverage that by loading all the corresponding modules:

# Define the environment for the build 
module load hdf5 netcdf4 aec ecmwf-toolbox proj cmor udunits

At this point, we need to refer to the installation instructions of this package in the official documentation. We can see it is a classic autotools package, which is typically built with the configure - make - make install sequence. We should then look at the configure --help to see how to enable all the extra options to configure the build:

# Configure the build
./configure --help && exit

Since we are just getting some short help, we can just run the script locally to get the configure output.

bash ./build_cdo.sh

We should inspect the output of the configure help command, and identify what options are to be used:

  --with-szlib=<yes|no|directory> (default=no)  
  --with-hdf5=<yes|no|directory> (default=no)
  --with-netcdf=<yes|no|directory> (default=no)
  --with-udunits2=<directory>
  --with-cmor=<directory> Specify location of CMOR library.
  --with-eccodes=<yes|no|directory> (default=no)
  --with-proj=<directory> Specify location of PROJ library for cartographic
                          projections.

Since all those dependencies are not installed on system paths, we will need to specify the installation directory for each one of them. We may then use the *_DIR environment variables defined by the corresponding modules we load just before.

We will also define where to install the package with the --prefix option. Let's amend the configure line with:

# Configure the build
./configure --prefix=$PREFIX --with-hdf5=$HDF5_DIR --with-netcdf=$NETCDF4_DIR --with-szlib=$AEC_DIR  --with-eccodes=$ECCODES_DIR  --with-proj=$proj_DIR --with-cmor=$CMOR_DIR --with-udunits2=$UDUNITS_DIR

Note that for PROJ, since the variable $PROJ_DIR has a special meaning in the package itself, we must use the lowercase version $proj_DIR.

We are now ready to attempt our first build.  Submit it the build script to the batch system with:

sbatch build_cdo.sh

While it builds, we may keep an eye on the progress with:

tail -f build_cdo.out

At this point CDO build and installation should complete successfully, but the execution of the newly installed CDO at the end fails with:

$ grep -v ECMWF-INFO build_cdo.out | tail
make[1]: Leaving directory '/etc/ecmwf/nfs/dh2_perm_a/user/apps/cdo/build/cdo-2.3.0/test/pytest'
make[1]: Entering directory '/etc/ecmwf/nfs/dh2_perm_a/user/apps/cdo/build/cdo-2.3.0'
make[2]: Entering directory '/etc/ecmwf/nfs/dh2_perm_a/user/apps/cdo/build/cdo-2.3.0'
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/etc/ecmwf/nfs/dh2_perm_a/user/apps/cdo/build/cdo-2.3.0'
make[1]: Leaving directory '/etc/ecmwf/nfs/dh2_perm_a/user/apps/cdo/build/cdo-2.3.0'
+ /perm/user/apps/cdo/2.3.0/bin/cdo -V
/perm/user/apps/cdo/2.3.0/bin/cdo: error while loading shared libraries: libproj.so.25: cannot open shared object file: No such file or directory

If we inspect the resulting binary with ldd, we will notice there are a few libraries that cannot be found at runtime:

$ ldd $PERM/apps/cdo/2.3.0/bin/cdo
        linux-vdso.so.1 (0x00007ffc3fbb6000)
        libproj.so.25 => not found
        libeccodes.so => not found
        libcmor.so => /usr/local/apps/cmor/3.7.1/lib/libcmor.so (0x00001484b970d000)
        libudunits2.so.0 => /usr/local/apps/udunits/2.2.28/lib/libudunits2.so.0 (0x00001484b94ed000)
        libexpat.so.1 => /lib64/libexpat.so.1 (0x00001484b92b1000)
        libnetcdf.so.19 => /usr/local/apps/netcdf4/4.9.1/GNU/8.5/lib/libnetcdf.so.19 (0x00001484b8e86000)
        libbz2.so.1 => /lib64/libbz2.so.1 (0x00001484b8c75000)
        libzstd.so.1 => /lib64/libzstd.so.1 (0x00001484b89d1000)
        libxml2.so.2 => /lib64/libxml2.so.2 (0x00001484b8669000)
        libcurl.so.4 => /lib64/libcurl.so.4 (0x00001484b83db000)
        libhdf5_hl.so.200 => /usr/local/apps/hdf5/1.12.2/GNU/8.5/lib/libhdf5_hl.so.200 (0x00001484b81ba000)
        libhdf5.so.200 => /usr/local/apps/hdf5/1.12.2/GNU/8.5/lib/libhdf5.so.200 (0x00001484b7bca000)
        libz.so.1 => /lib64/libz.so.1 (0x00001484b79b2000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00001484b77ae000)
        libsz.so.2 => /lib64/libsz.so.2 (0x00001484b75ab000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00001484b738b000)
        libuuid.so.1 => /usr/local/apps/cmor/3.7.1/lib/libuuid.so.1 (0x00001484b7187000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00001484b6df2000)
        libm.so.6 => /lib64/libm.so.6 (0x00001484b6a70000)
        libgomp.so.1 => /lib64/libgomp.so.1 (0x00001484b6838000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00001484b6620000)
        libc.so.6 => /lib64/libc.so.6 (0x00001484b625b000)
        /lib64/ld-linux-x86-64.so.2 (0x000014850b1a3000)
        libjson-c.so.4 => /usr/local/apps/cmor/3.7.1/lib/libjson-c.so.4 (0x00001484b604c000)
        libgfortran.so.5 => /lib64/libgfortran.so.5 (0x00001484b5bcd000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00001484b59a6000)
        libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00001484b577f000)
        libidn2.so.0 => /lib64/libidn2.so.0 (0x00001484b5561000)
        libssh.so.4 => /lib64/libssh.so.4 (0x00001484b52f2000)
        libpsl.so.5 => /lib64/libpsl.so.5 (0x00001484b50e1000)
        libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00001484b4e4d000)
        libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00001484b4964000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00001484b470f000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00001484b4425000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00001484b420e000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00001484b400a000)
        libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00001484b3dbb000)
        liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00001484b3bab000)
        libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00001484b399e000)
        libaec.so.0 => /lib64/libaec.so.0 (0x00001484b3796000)
        libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00001484b3555000)
        libunistring.so.2 => /lib64/libunistring.so.2 (0x00001484b31d4000)
        librt.so.1 => /lib64/librt.so.1 (0x00001484b2fcc000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00001484b2dbb000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00001484b2bb7000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00001484b29a0000)
        libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00001484b2782000)
        libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00001484b2561000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00001484b2337000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00001484b210e000)
        libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00001484b1e8a000)

We are missing the PROJ and ecCodes libraries. We will need to explicitly set RPATHs when we build CDO to make sure the libraries are found. In Autotools packages, and as shown in the configure help we ran earlier, we may pass any extra link flags through the LDFLAGS environment variable. We can amend our build script setting that variable just after loading the dependant modules:

# Define the environment for the build 
module load hdf5 netcdf4 aec ecmwf-toolbox proj cmor udunits

# We will need to explicitly set rpath for proj and eccodes since CDO build system will not
export LDFLAGS="-Wl,-rpath,$proj_DIR/lib -Wl,-rpath,$ECCODES_DIR/lib"

If we submit the the build job again and wait for it to complete, we should see something like:

$ grep -v ECMWF-INFO build_cdo.out | tail -n 25
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/etc/ecmwf/nfs/dh2_perm_a/user/apps/cdo/build/cdo-2.3.0'
make[1]: Leaving directory '/etc/ecmwf/nfs/dh2_perm_a/user/apps/cdo/build/cdo-2.3.0'
+ /perm/user/apps/cdo/2.3.0/bin/cdo -V
Climate Data Operators version 2.3.0 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-linux-gnu
CXX Compiler: g++ -std=gnu++17 -g -O2 -fopenmp -pthread
CXX version : g++ (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15)
C Compiler: gcc -g -O2 -fopenmp -pthread -pthread
C version : gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15)
F77 Compiler: gfortran -g -O2
F77 version : GNU Fortran (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15)
Features: 7/503GB 16/256threads c++17 OpenMP45 Fortran pthreads HDF5 NC4/HDF5/threadsafe OPeNDAP sz udunits2 proj cmor sse2                                                                
Libraries: yac/3.0.1 NetCDF/4.9.1 HDF5/1.12.2 proj/9.1.1 cmor/3.7.1
CDI data types: SizeType=size_t
CDI file types: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5 nczarr
     CDI library version : 2.3.0
 cgribex library version : 2.1.1
 ecCodes library version : 2.30.2
  NetCDF library version : 4.9.1 of Feb  9 2023 13:54:09 $
    exse library version : 1.5.0
    FILE library version : 1.9.1

For reference, this is the complete job script:

#!/

#!/bin/bash
#SBATCH -J build_cdo
#SBATCH -o %x.out
#SBATCH -n 8

set -x
set -e
set -u
set -o pipefail

# Get the URL and VERSION for the latest CDO available
URL=$(curl -s https://code.mpimet.mpg.de/projects/cdo/files | grep attachments/download  | sed -e "s:.*href=\"\(.*\)\".*:https\://code.mpimet.mpg.de\1:" | head -n 1)
VERSION=$(echo $URL | sed -e "s:.*/cdo-\(.*\).tar.gz:\1:")

# Define installation prefix and Buildbuild directory
PREFIX=$PERM/apps/cdo/$VERSION
#BUILDDIR=$TMPDIR
BUILDDIR=$PERM/apps/cdo/build

# Move to our BUILD DIRECTORY
mkdir -p $BUILDDIR
cd $BUILDDIR

# Download source
[ -f cdo-$VERSION.tar.gz ] || wget $URL
[ -d cdo-$VERSION ] || tar xvf cdo-$VERSION.tar.gz
cd cdo-$VERSION

# Define the environment for the build 
module load aec hdf5 netcdf4 udunitsaec ecmwf-toolbox proj cmor ecmwf-toolboxudunits

# We will need to explicitly set rpath for proj and eccodes since CDO build system will not
export LDFLAGS="-Wl,-rpath,$proj_DIR/lib -Wl,-rpath,$ECCODES_DIR/lib"

# Configure the build
./configure --prefix=$PREFIX --with-eccodeshdf5=$ECCODES$HDF5_DIR --with-hdf5netcdf=$HDF5$NETCDF4_DIR --with-netcdfszlib=$NETCDF4$AEC_DIR  --with-szlibeccodes=$AEC$ECCODES_DIR  --with-proj=$proj_DIR --with-cmor=$CMOR_DIR --with-udunits2=$UDUNITS_DIR$UDUNITS_DIR

# Make sure we start a clean build
make clean

# Build
make -j $SLURM_NTASKS

# Install
make install

# Check installed binary
$PREFIX/bin/cdo -V

You can submit it to the batch system with

sbatch build_cdo.sh

While it builds, you may want to keep an eye on the progress with:

tail -f build_cdo.out