This section shows how to configure and run OpenIFS and XIOS together.
First of all, it is necessary that OpenIFS is compiled with XIOS (see OpenIFS: Enable XIOS during compilation) and the XIOS executable is available in the experiment directory. OpenIFS should be run as usual (see 6.3 OpenIFS Running a Forecast), but adding the argument '--xios' to oifs_run. This will use by default 1 XIOS server, but in case of needing more servers, it can be done with '--xios --xios-nproc <arg>'.
It is also necessary to enable the 'LXIOS' switch in the 'NAMCT0' namelist (fort.4 file).
It is expected that users have knowledge on the use of XIOS which includes the creation of NetCDF files, computation of diagnostics, interpolations, data compression, interpretation of performance parameters, etc. Official documentation such as the user guide or tutorials is available on the webpage: https://forge.ipsl.jussieu.fr/ioserver. However, this section will explain XIOS aspects that are particular to OpenIFS.
OpenIFS and XIOS are controlled using different XML files:
Let's see in more detail how to configure different features particular to OpenIFS-XIOS.
In context_ifs.xml different FullPos variables can be tuned. There is a group of variables that control the spectral fitting of FullPos and a second group that contains four different variables named 'NFPCLI', 'LFPQ', 'LTRACEFP' and 'RFPCORR'. More information about all these variables can be found at How to control OpenIFS output and in the latest FullPos user guide.
There are two available optimizations that might be useful to improve the execution time under some circumstances. It is not possible to predict in which conditions, as many factors have to be taken into consideration. For this reason it is necessary to test them. They are disabled by default, but can be enabled in context_ifs.xml:
It is possible to define different vertical levels to be interpolated from model levels in axis_def_ifs.xml. In particular, it is possible to define pressure levels, theta levels and PV levels. For example, if you want to define four theta levels, you should edit 'n_glo' and 'value':
<axis id="theta_levels" long_name="vertical theta (potential temperature) levels" n_glo="4" value="(0,3)[300 320 340 360]" /> |
In addition, it is possible to output fields with a subset of the vertical levels. This XIOS feature is known as zoom and you have to specify the indexes (from the original axis) of the levels that you want to output, and NOT the level values themselves. From the previous example, if you are only interested in theta levels 320 and 360, you should edit 'index' as follows:
<zoom_axis index="(0,1)[1 3]" /> |
Note that indexes go from '0' to 'n_glo - 1'.
If the user chooses to output data in a regular lat-lon grid, by default fields use a grid with 256 latitudes and 512 longitudes. However, it is possible to change the sizes of the regular grid or even to declare more than one regular grid. Domains for regular grids are declared in domain_def_ifs.xml. There are two key attributes:
In addition, it is necessary to specify '<generate_rectilinear_domain />' to indicate that a horizontal interpolation is needed. The attribute 'write_weight' controls whether to write interpolation weights or not.
New axes and domains might be defined to output fields, but it is recommended for 2D fields and mandatory for 3D fields to declare new grids made up of these new axes and domains. Grids are declared in grid_def_ifs.xml, so different possible combinations of axes (axis_def_ifs.xml) and domains (domain_def_ifs.xml) should go here. It is then possible to make use of this new grid to output fields (see next section).
With a well-defined regular lat-lon grid it is possible to use it to output fields. file_def_ifs.xml controls the netCDF files to be output with the corresponding fields. To output a field in a regular lat-lon grid it is necessary to use the attribute 'grid_ref'. For example, to output the 3D temperature field in a 256x512 lat-lon grid and model levels, you can use this XML code:
<field field_ref="t" name="t" grid_ref="regular_ml" operation="instant" /> |
where 'regular_ml' is defined in grid_def_ifs.xml as follows:
<grid id="regular_ml" description="3D interpolated regular grid with hybrid model levels" > <domain domain_ref="regular" /> <axis axis_ref="model_levels" /> </grid> |
It is very important to distinguish the difference between attributes 'output_freq' (output frequency) and 'freq_op' (sampling frequency), and how 'NFRHIS' and 'NFRPOS' are intrinsically related to 'freq_op'.
While 'output_freq' and 'freq_op' are attributes that belong to XIOS, 'NFRHIS' and 'NFRPOS' variables belong to FullPos. Although it would have been desirable to automatically set up these two FullPos variables during runtime (instead of giving the 'responsibility' to the user, which might be a source of unnecessary errors), it is necessary to manually set up them in context_ifs.xml because the XIOS API does not give the possibility of doing it.
That being said, let's see how to correctly set up them. The definition and use of the two XIOS attributes is the following one:
Two examples to illustrate how these two variables work and how 'NFRHIS' and 'NFRPOS' must be set up.
Example 1: you only want to output instant values of 't' and 'cc' fields every 6 hours.
Example 2: you want to output these three variables:
Note that always 'NFRHIS' = 'NFRPOS'. It is also possible to specify these two FullPos variables in hours. To do so, you have to use negative values:
This section reviews different aspects that can considerably affect the computational performance of OpenIFS-XIOS and consequently increase the total execution time.
When XIOS is run with more than one server and compiled with parallel netCDF support, there is the possibility to output single netCDF files, i.e. all servers write in parallel into one netCDF file. However, this 'one_file' mode has a limited computational efficiency as it does not scale well when outputting a large volume of data for high resolution configurations.
On the contrary, the 'multiple_file' mode achieves a good computational efficiency as it scales with many resources. However, each XIOS server writes its own netCDF file, so output data is splitted between all these partial files. It is required later to use a post-process script to combine all these chunked netCDF files into a single one.
In order to output fields in a regular lat-lon grid XIOS needs to perform a horizontal interpolation. However, this operation is very expensive depending on different parameters (number of fields, size of the fields, frequency, etc), so it can have a large impact on the total execution time. This happens because interpolations are performed on the client side of XIOS, which means that OpenIFS processes have to first compute these interpolations before resuming the time stepping. The recommendation is to only use horizontal interpolations when it is necessary and not systematically.
It is very important to set up 'NFRHIS' and 'NFRPOS' FullPos variables at a frequency equal to the great common divisor (gcd) of all the different values of 'freq_op' so that it produces correct data and does not waste computational resources. This means that if we set up 'NFRHIS' and 'NFRPOS' with a value that is smaller than the gcd, then we would call FullPos unnecessarily (post-processed data will not be sent to XIOS as it is not required). This increases the computational cost with no gains in the accuracy of the results.
XIOS only supports fields in grid-point space, so it is not possible to directly output OpenIFS native spectral fields. To overcome this issue, the OpenIFS-XIOS integration makes use of the internal TRANS package to transform spectral fields to grid-point fields before sending them to XIOS. This operation is applied in conjunction with the FullPos post-processing, so it is also controlled with the 'NFRHIS' and 'NFRPOS' variables. This transformation operation is not cheap, and as said previously, it is very important to correctly set up 'NFRHIS' and 'NFRPOS' to make a responsible use.
When OpenIFS is run on a cluster that uses the Lustre filesystem, it is important to pay attention to the striping. Lustre stores a file in one or more Object Storage Target (OST) devices. By default, it uses one OST, but it is possible to divide the file into chunks that are stored in different OSTs. This is known as striping, and it is used for performance purposes, especially for really big files.
It is therefore important to set up a striping for each netCDF at least as equal as to the number of XIOS servers when using the 'one_file' mode. This allows each XIOS server to write into a different OST. If the number of OSTs per file was not changed, it could affect the performance of the whole system, including the rest of the users that are performing I/O at that moment because it could fill up the OST where the big netCDF file would be. When such OST runs out of space, it also affects other user's jobs.
Please note: as XIOS is not used operationally at ECMWF, we can only provide help on a best effort basis. We encourage users to post questions on XIOS to the OpenIFS User Forums to get help from other users in the OpenIFS community. |