Page History

...

The following exercises will help you understand how to build your own software on the ATOS HPCF or ECS.

Table of Contents

...

Before we start...

Ensure your environment is clean by running:
No Format
module reset
Create a directory for this tutorial and cd into it:
No Format
mkdir -p compiling_tutorial cd compiling_tutorial

Building simple programs

With your favourite editor, create three hello world programs, one in C, one in C++ and one in Fortran

Code Block

language	cpp
title	hello.c
collapse	true

#include <stdio.h>

int main(int argc, char** argv)
{
    printf("Hello World from a C program\n");
    return 0;
}

Code Block

language	cpp
title	hello++.cc
collapse	true

#include <iostream>

int main()
{
  std::cout << "Hello World from a C++ program!" << std::endl;
}

Code Block

language	cpp
title	hellof.f90
collapse	true

       program hello
          print *, "Hello World from a Fortran Program!"
       end program hello

Compile and run each one of them with the GNU compilers (gcc, g++, gfortran)

Expand

title	Solution

We can build them with:

No Format
gcc -o hello hello.c g++ -o hello++ hello++.cc gfortran -o hellof hellof.f90

All going well, we should have now the executables that we can run

No Format
$ for exe in hello hello++ hellof; do ./$exe; done Hello World from a C program Hello World from a C++ program! Hello World from a Fortran Program!

Now, use the generic environment variables for the different compilers ($CC, $CXX, $FC) and rerun, you should see no difference to the above results.

Expand

title	Solution

We can rebuild them with:

No Format
$CC -o hello hello.c $CXX -o hello++ hello++.cc $FC -o hellof hellof.f90

We can now run them exactly in the same way:

No Format
$ for exe in hello hello++ hellof; do ./$exe; done Hello World from a C program Hello World from a C++ program! Hello World from a Fortran Program!

Tip

title	Always use the environment variables for compilers

We always recommend using the environment variables since it will make it easier for you to switch to a different compiler.

...

We are now going to use a simple program that will display versions of different libraries linked to it. Create a file called versions.c using your favourite editor with the following contents:

Code Block

language	cpp
title	versions.c
collapse	true

#include <stdio.h>
#include <hdf5.h>
#include <netcdf.h>
#include <eccodes.h>

int main() {
    #if defined(__INTEL_LLVM_COMPILER)
        printf("Compiler: Intel LLVM %d\n", __INTEL_LLVM_COMPILER);
    #elif defined(__INTEL_COMPILER)
        printf("Compiler: Intel Classic %d\n", __INTEL_COMPILER);
    #elif defined(__clang_version__)
        printf("Compiler: Clang %s\n",  __clang_version__);
    #elif defined(__GNUC__)
        printf("Compiler: GCC %d.%d.%d\n", __GNUC__, __GNUC_MINOR__, __GNUC_PATCHLEVEL__);
    #else
        printf("Compiler information not available\n");
    #endif
    
    // HDF5 version
    unsigned majnum, minnum, relnum;
    H5get_libversion(&majnum, &minnum, &relnum);
    printf("HDF5 version: %u.%u.%u\n", majnum, minnum, relnum); 

    // NetCDF version
    printf("NetCDF version: %s\n", nc_inq_libvers());

    // ECCODES version
    printf("ECCODES version: ");
    codes_print_api_version(stdout);
    printf("\n");

    return 0;
}

Try to naively compile this program with:
No Format
$CC -o versions versions.c

The compilation above fails as it does not know where to find the different libraries. We need to add some additional flags so the compiler can find both the include headers and link to the actual libraries.

Let's use the existing software installed on the system with modules, and benefit from the corresponding environment variables *_DIR which are defined in them to manually construct the include and library flags:

No Format
$CC -o versions versions.c -I$HDF5_DIR/include -I$NETCDF4_DIR/include -I/$ECCODESI$ECCODES_DIR/include -L$HDF5_DIR/lib -lhdf5 -L$NETCDF4_DIR/lib -lnetcdf -L$ECCODES_DIR/lib -leccodes

Load the appropriate modules so that the line above completes successfully and generates the versions executable:

Expand

title	Solution

You will need to load the following modules to have those variables defined:

No Format

module load hdf5 netcdfnetcdf4 ecmwf-toolbox
$CC -o versions versions.c -I$HDF5_DIR/include -I$NETCDF4_DIR/include -I/$ECCODESI$ECCODES_DIR/include -L$HDF5_DIR/lib -lhdf5 -L$NETCDF4_DIR/lib -lnetcdf -L$ECCODES_DIR/lib -leccodes

The versions executable should now be in your current directory:

No Format
ls versions

Run ./versions. You will get an error such as the one below:

No Format
./versions: error while loading shared libraries: libhdf5.so.200: cannot open shared object file: No such file or directory

While you passed the location of the libraries at compile time, the program cannot not find the libraries at runtime. Inspect the executable with ldd to see what libraries are missing

Expand

title	Solution

ldd is a utility that prints the shared libraries required by each program or shared library specified on the command line:

No Format

$ ldd versions
        linux-vdso.so.1 (0x00007ffffada9000)
        libhdf5.so.200 => not found
        libnetcdf.so.19 => not found
        libeccodes.so => not found
        libc.so.6 => /lib64/libc.so.6 (0x000014932ff36000)
        /lib64/ld-linux-x86-64.so.2 (0x00001493302fb000)

Can you make that program run successfully?

Expand

title	Solution

While you passed the location of the libraries at compile time, the program cannot not find the libraries at runtime. There are two solutions:

Use the environment variable LD_LIBRARY_PATH- not recommended for long term

Use the environment variable LD_LIBRARY_PATH. Check that ldd with the environment variable defined reports all libraries found:

No Format
LD_LIBRARY_PATH=$HDF5_DIR/lib:$NETCDF4_DIR/lib:$ECCODES_DIR/lib ldd ./versions

Rebuild with rpath - robust solution

Use the rpath strategy to engrave the library paths into the actual executable at link time, so it always knows where to find them at runtime. Rebuild your program with:

No Format

$CC -o versions versions.c -I$HDF5_DIR/include -I$NETCDF4_DIR/include -I/$ECCODES_DIR/include -L$HDF5_DIR/lib -Wl,-rpath,$HDF5_LIB -lhdf5 -L$NETCDF4_DIR/lib -Wl,-rpath,$NETCDF4_DIR/lib -lnetcdf -L$ECCODES_DIR/lib -Wl,-rpath,$ECCODES_DIR/lib -leccodes

Check that ldd with the environment variable defined reports all libraries found:

No Format
unset LD_LIBRARY_PATH ldd ./versions

Final version

For convenience, all those software modules define the *_INCLUDE and *_LIB variables:

No Format
module show hdf5 netcdf4 ecmwf-toolbox \| grep -e _INCLUDE -e _LIB

You can use those in for your compilation directly, with the following simplified compilation line:

No Format
$CC -o versions versions.c $HDF5_INCLUDE $NETCDF4_INCLUDE $ECCODES_INCLUDE $HDF5_LIB $NETCDF4_LIB $ECCODES_LIB

Now you can run your program without any additional settings:

No Format
$ ./versions Compiler: GCC 8.5.0 HDF5 version: <HDF5 version> NetCDF version: <NetCDF version> of <date> $ ECCODES version: <ecCodes version>

Can you rebuild the program so it uses the "old" versions of all those libraries in modules? Ensure the output of the program matches the versions loaded in modules? Do the same with the latest.

Expand

title	Solution

You need to load the desired versions or the modules:

No Format
module load hdf5/old netcdf4/old ecmwf-toolbox/old

And then rebuild and run the program:

No Format
$CC -o versions versions.c $HDF5_INCLUDE $NETCDF4_INCLUDE $ECCODES_INCLUDE $HDF5_LIB $NETCDF4_LIB $ECCODES_LIB ./versions

The output should match the versions loaded by the modules:

No Format
echo $HDF5_VERSION $NETCDF4_VERSION $ECCODES_VERSION

Repeat the operation with

No Format
module load --latest hdf5 netcdf4 ecmwf-toolbox

To simplify the build process, let's create a simple Makefile for this program. With your favourite editor, create a file called Makefile in the same directory with the following contents:

Note

title	Watch the indentation

Make sure that the indentations at the beginning of the lines are tabs and not spaces!

Code Block

language	bash
title	Makefile
collapse	true

#
# Makefile
#
# Make sure all the relevant modules are loaded before running make

EXEC = hello hello++ hellof versions

# TODO: Add the necessary variables into CFLAGS and LDFLAGS definition
CFLAGS = 
LDFLAGS = 

all: $(EXEC)

%: %.c 
	$(CC) -o $@ $^ $(CFLAGS) $(LDFLAGS)

test%: $(EXEC)%.cc
	./$(EXEC)CXX) -o $@ $^

ldd%: $(EXEC)%.f90
	@ldd $(EXECF90) | -o $@ $^

test: $(EXEC)
	@for exe in $(EXEC); do ./$$exe; done

ldd: versions
	@ldd versions | grep -e netcdf.so -e eccodes.so -e hdf5.so 

clean:
	rm -f $(EXEC)

You can test it works by running:

No Format
make clean test ldd test

Expand

title	Solution

Edit the Makefile and add the *_INCLUDE and *_LIB variables which are defined by the modules:

Code Block

language	bash
title	Makefile
collapse	true

#
# Makefile
#
# Make sure all the relevant modules are loaded before running make

EXEC = versions

CFLAGS = $(HDF5_INCLUDE) $(NETCDF4_INCLUDE) $(ECCODES_INCLUDE)
LDFLAGS = $(HDF5_LIB) $(NETCDF4_LIB) $(ECCODES_LIB)

all: $(EXEC)

%: %.c 
	$(CC) -o $@ $^ $(CFLAGS) $(LDFLAGS)

test%: %.cc
	$(EXEC)
	./$(EXEC)

lddCXX) -o $@ $^

%: %.f90
	$(F90) -o $@ $^

test: $(EXEC)
	@ldd@for exe in $(EXEC); do ./$$exe; done

ldd: versions
	@ldd versions | grep -e netcdf.so -e eccodes.so -e hdf5.so 

clean:
	rm -f $(EXEC)

Then run it with:

No Format
make clean test ldd test

Using different toolchains: prgenv

...

Rebuild the program with:

The default GNU GCC compiler.
The default Classic Intel compiler.
The default LLVM-based Intel compiler.
The default AMD AOCC.

Use the following command to test and show what versions of the libraries are being used at any point:

No Format
make clean ldd test

Expand

title	Solution

You can perform this test with the following one-liner, exploiting the prgenv module:

No Format
for pe in gnu intel intel-llvm amd; do module load prgenv/$pe; make clean test ldd test; echo "******************"; done

Pay attention to the following aspects:

The Lmod module command informs you that it has reloaded the corresponding modules when changing the prgenv. This ensures the libraries used in your program are built with the same compiler for maximum compatibility.
The compiler command changes automatically, since we are using the environment variable $CC in the Makefile.
The include and library flags in the compilation lines are adapted automatically based on the libraries loaded.
The final binary is linked with the corresponding libraries for the version of the compiler as shown by ldd output.

Rebuild the program with the "new" GNU GCC compiler. Use the same command as above to test and show what versions of the libraries are being used at any point.

Expand

title	Solution

This time we need to be on the GNU prgenv, but also select the "new" gcc compiler instead of just the default.

Remember you can look at the versions available in modules and their corresponding labels with:

No Format
module avail gcc

This sequence of commands should do the trick:

No Format
module load prgenv/gnu gcc/new make clean test ldd test

Rebuild the program with the Classic Intel compiler once again, but this time reset your module environment once the executable has been produced and before running it. What happens when you run it?

Expand

title	Solution

Let's look at the following sequence of commands:

No Format
module load prgenv/intel make clean allversions module reset make test

The result will be something similar to:

No Format
./versions: error while loading shared libraries: libifport.so.5: cannot open shared object file: No such file or directory

Inspecting the executable with ldd throws some missing libraries at runtime:

No Format

$ ldd versions |  grep "not found"
        libcilkrts.so.5 => not found
        libifcoremt.so.5 => not found
        libifport.so.5 => not found
        libimf.so => not found
        libintlc.so.5 => not found
        libirng.so => not found
        libsvml.so => not found
        ...

That shows how, for Intel-built programs, you must have the intel environment set up at both compile and run times.

...

Space shortcuts

Page tree

Versions Compared

Old Version 14

New Version Current

Key

Building simple programs

Using different toolchains: prgenv