Compiling - Atos HPCF and ECS Introduction Tutorial

The following exercises will help you understand how to build your own software on the ATOS HPCF or ECS.

Building a simple program

1. Ensure your environment is clean by running:

   module reset

2. Create a directory for this tutorial and cd into it:

   mkdir -p compiling_tutorial
   cd compiling_tutorial

3. We are going to use a simple program that will display versions of different libraries linked to it. Create a file called versions.c using your favourite editor with the following contents:

    Expand source

   #include <stdio.h>
   #include <hdf5.h>
   #include <netcdf.h>
   #include <eccodes.h>
   
   int main() {
       #if defined(__INTEL_LLVM_COMPILER)
           printf("Compiler: Intel LLVM %d\n", __INTEL_LLVM_COMPILER);
       #elif defined(__INTEL_COMPILER)
           printf("Compiler: Intel Classic %d\n", __INTEL_COMPILER);
       #elif defined(__clang_version__)
           printf("Compiler: Clang %s\n",  __clang_version__);
       #elif defined(__GNUC__)
           printf("Compiler: GCC %d.%d.%d\n", __GNUC__, __GNUC_MINOR__, __GNUC_PATCHLEVEL__);
       #else
           printf("Compiler information not available\n");
       #endif
       
       // HDF5 version
       unsigned majnum, minnum, relnum;
       H5get_libversion(&majnum, &minnum, &relnum);
       printf("HDF5 version: %u.%u.%u\n", majnum, minnum, relnum); 
   
       // NetCDF version
       printf("NetCDF version: %s\n", nc_inq_libvers());
   
       // ECCODES version
       printf("ECCODES version: ");
       codes_print_api_version(stdout);
       printf("\n");
   
       return 0;
   }
   

4. Try to naively compile this program with:

   $CC -o versions versions.c

5. The compilation above fails as it does not know where to find the different libraries. We need to add some additional flags so the compiler can find both the include headers and link to the actual libraries. 

   1. Let's use the existing software installed on the system with modules, and benefit from the corresponding environment variables *_DIR which are defined in them to manually construct the include and library flags:

      $CC -o versions versions.c -I$HDF5_DIR/include -I$NETCDF4_DIR/include -I/$ECCODES_DIR/include -L$HDF5_DIR/lib -lhdf5 -L$NETCDF4_DIR/lib -lnetcdf -L$ECCODES_DIR/lib -leccodes

      Load the appropriate modules so that the line above completes successfully and generates the versions executable:

      Solution

      You will need to load the following modules to. have those variables defined.:

      module load hdf5 netcdf ecmwf-toolbox
      $CC -o versions versions.c -I$HDF5_DIR/include -I$NETCDF4_DIR/include -I/$ECCODES_DIR/include -L$HDF5_DIR/lib -lhdf5 -L$NETCDF4_DIR/lib -lnetcdf -L$ECCODES_DIR/lib -leccodes

      The versions executable should now be in your current directory:

      ls versions

   2. Run ./versions. You will get an error such as the one below:

      $ ./versions
      ./versions: error while loading shared libraries: libhdf5.so.200: cannot open shared object file: No such file or directory
      
      

      While you passed the location of the libraries at compile time, the program cannot not find the libraries at runtime. Inspect the executable with ldd to see what libraries are missing

      Solution

      ldd is a utility that prints the shared libraries required by each program or shared library specified on the command line:

      $ ldd versions
              linux-vdso.so.1 (0x00007ffffada9000)
              libhdf5.so.200 => not found
              libnetcdf.so.19 => not found
              libeccodes.so => not found
              libc.so.6 => /lib64/libc.so.6 (0x000014932ff36000)
              /lib64/ld-linux-x86-64.so.2 (0x00001493302fb000)

      
      

   3. Can you make that program run successfully?

      Solution

      While you passed the location of the libraries at compile time, the program cannot not find the libraries at runtime. There are two solutions:

      Use the environment variable LD_LIBRARY_PATH- not recommended for long term

      Use the environment variable LD_LIBRARY_PATH. Check that ldd with the environment variable defined reports all libraries found:

      LD_LIBRARY_PATH=$HDF5_DIR/lib:$NETCDF4_DIR/lib:$ECCODES_DIR/lib ldd ./versions

      Use  rpath  - robust solution 

      Use the  rpath strategy to engrave the library paths into the actual executable at link time, so it always knows where to find them at runtime. Rebuild your program with:

      $CC -o versions versions.c -I$HDF5_DIR/include -I$NETCDF4_DIR/include -I/$ECCODES_DIR/include -L$HDF5_DIR/lib -Wl,-rpath,$HDF5_LIB -lhdf5 -L$NETCDF4_DIR/lib -Wl,-rpath,$NETCDF4_DIR/lib -lnetcdf -L$ECCODES_DIR/lib -Wl,-rpath,$ECCODES_DIR/lib -leccodes

      Check that ldd with the environment variable defined reports all libraries found:

      unset LD_LIBRARY_PATH
      ldd ./versions

      Final version

      For convenience, all those software modules define the *_INCLUDE and *_LIB variables:

      module show hdf5 netcdf4 ecmwf-toolbox | grep -e _INCLUDE -e _LIB

      You can use those in for your compilation directly, with the following simplified compilation line:

      $CC -o versions versions.c $HDF5_INCLUDE $NETCDF4_INCLUDE $ECCODES_INCLUDE $HDF5_LIB $NETCDF4_LIB $ECCODES_LIB

      Now you can run your program without any additional settings:

      $ ./versions
      Compiler: GCC 8.5.0
      HDF5 version: <HDF5 version>
      NetCDF version: <NetCDF version> of <date> $
      ECCODES version: <ecCodes version>

6. Can you rebuild the program so it uses the "old" versions of all those libraries in modules? Ensure the output of the program matches the versions loaded in modules? Do the same with the latest. 

   Solution

   You need to load the desired versions or the modules:

   module load hdf5/old netcdf4/old ecmwf-toolbox/old

   And then rebuild and run the program:

   $CC -o versions versions.c $HDF5_INCLUDE $NETCDF4_INCLUDE $ECCODES_INCLUDE $HDF5_LIB $NETCDF4_LIB $ECCODES_LIB
   ./versions

   The output should match the versions loaded by the modules: 

   echo $HDF5_VERSION $NETCDF4_VERSION $ECCODES_VERSION

   Repeat the operation with 

   module load --latest hdf5 netcdf4 ecmwf-toolbox

7. To simplify the build process, let's create a simple Makefile for this program. With your favourite editor, create a file called Makefile in the same directory with the following contents:

   Makefile Expand source

   #
   # Makefile
   #
   # Make sure all the relevant modules are loaded before running make
   
   EXEC = versions
   
   # TODO: Add the necessary variables into CFLAGS and LDFLAGS definition
   CFLAGS = 
   LDFLAGS = 
   
   all: $(EXEC)
   
   %: %.c 
   	$(CC) -o $@ $^ $(CFLAGS) $(LDFLAGS)
   
   test: $(EXEC)
   	./$(EXEC)
   
   clean:
   	rm -f $(EXEC)

   You can test it works by running:

   make clean ldd test

   Solution

   Edit the Makefile and add the *_INCLUDE and *_LIB variables which are defined by the modules:

   Makefile Expand source

   #
   # Makefile
   #
   # Make sure all the relevant modules are loaded before running make
   
   EXEC = versions
   
   CFLAGS = $(HDF5_INCLUDE) $(NETCDF4_INCLUDE) $(ECCODES_INCLUDE)
   LDFLAGS = $(HDF5_LIB) $(NETCDF4_LIB) $(ECCODES_LIB)
   
   all: $(EXEC)
   
   %: %.c 
   	$(CC) -o $@ $^ $(CFLAGS) $(LDFLAGS)
   
   test: $(EXEC)
   	./$(EXEC)
   
   ldd: $(EXEC)
   	@ldd $(EXEC) | grep -e netcdf.so -e eccodes.so -e hdf5.so 
   
   clean:
   	rm -f $(EXEC)

   Then run it with:

   make clean ldd test

Using different toolchains: prgenv

So far we have used the default compiler toolchain to build this program. Because of the installation paths of the library, it is easy to see both the version of the library used as well as the compiler flavour with ldd:

$ make ldd
        libhdf5.so.200 => /usr/local/apps/hdf5/<HDF5 version>/GNU/8.5/lib/libhdf5.so.200 (0x000014f612b7d000)
        libnetcdf.so.19 => /usr/local/apps/netcdf4/<NetCDF version>/GNU/8.5/lib/libnetcdf.so.19 (0x000014f611f2a000)
        libeccodes.so => /usr/local/apps/ecmwf-toolbox/<ecCodes version>/GNU/8.5/lib/libeccodes.so (0x000014f611836000)

1. Rebuild the program with:

2. 1. The default GNU GCC compiler.
   2. The default Classic Intel compiler.
   3. The default LLVM-based Intel compiler.
   4. The default AMD AOCC.

   Use the following command to test and show what versions of the libraries are being used at any point:

   make clean ldd test

   Solution

   You can perform this test with the following one-liner, exploiting the prgenv module:

   for pe in gnu intel intel-llvm amd; do ml prgenv/$pe; make clean ldd test; done

   Pay attention to the following aspects:

   • The Lmod module command informs you that it has reloaded the corresponding modules when changing the prgenv. This ensures the libraries used in your program are built with the same compiler for maximum compatibility.
   • The compiler command changes automatically, since we are using the environment variable $CC in the Makefile.
   • The include and library flags in the compilation lines are adapted automatically based on the libraries loaded.
   • The final binary is linked with the corresponding libraries for the version of the compiler as shown by ldd output.