Login to ECMWF Cray
Each participant will have a training user account on the ECMWF Cray system. This is different from the user account on the classroom computers.
Begin by logging onto the Cray login nodes:
Code Block | ||
---|---|---|
| ||
ssh trcrayXX@ccbtroifsXX@ccb <<< use your Cray user id Password: xxxxxxxx trcray@ccbtroifs@ccb-login2:~> |
The contents of the account should look like:
Code Block | ||
---|---|---|
| ||
trcray1@ccbtroifs1@ccb-login2:~> ls bin modules.sh oifs38r1 scratch |
If any are missing please let us know.
Directory: oifs38r1
This directory contains the OpenIFS source code and OpenIFS executable.
...
Info | ||
---|---|---|
| ||
Examine the files in the oifs38r1 directory if you are unfamiliar with the OpenIFS model. |
Directory: scratch
This is the directory in which we will run OpenIFS.
Setting the correct environment
...
...
Each time you login and before doing any work in this account, make sure you have the correct environment loaded to run and work with OpenIFS.
Code Block | ||
---|---|---|
| ||
source bin/modules.sh
|
This will load the appropriate modules for OpenIFS and set the choice of compiler and build environment. It is good practise to use something similar for your own OpenIFS installation.
Creating the OpenIFS ensemble experiment
An ensemble experiment consists of a 'control' and multiple 'perturbed' forecasts, or 'ensemble members'. For the operational ECMWF ensemble forecast, 50 members are used.
For this exercise, to save computer time, each participant will run a single, different 'ensemble member' and we will create the complete forecast ensemble from the individual ensembles run by the class as a whole. A later exercise will show how to do this from the model output GRIB files.
The first exercises show how to create the experiment directories, prepare the jobs and submit them to the ECMWF Cray.
createENS command
Use the command createENS
to create the ensemble experiment:
Info | ||
---|---|---|
Please only use the 2013102400 start date to ensure we can create the full ensemble from everyone's runs. Use the createENS command to create an ensemble with just a single member
|
Code Block |
---|
createENS --help usage: createENS [-h] -d <startdate/enddate/hr_inc> [-m <members>] Set up ensemble forecast for the OpenIFS 2015 workshop optional arguments: -h, --help show this help message and exit -d <startdate/enddate/hr_inc>, --date <startdate/enddate/hr_inc> List of date(s) to retrieve, format: YYYYMMDDHH. The only dates available are: 2013102400/2013102700/24 -m <members>, --members <members> Number of ensemble members to create e.g. 3 (default 1) |
Info | ||
---|---|---|
| ||
Decide which of the available dates to use. Use the createENS command to create an ensemble with just a single member. |
Info |
---|
Exercise
Change to the experiment directory labelled by the date chosen:
Code Block | ||||
---|---|---|---|---|
| ||||
cd scratch/ENFO/2013102400/ # your date may be different |
There are two experiment directories: cf000 and pf001. cf000 is for the control forecast which is not used here. pf001 is the single perturbation forecast.
Code Block | |||||
---|---|---|---|---|---|
| examine
| ||||
cd pf001
ls |
Understand what the different files in the experiment directory are for.
Initial files
ICM* : these are the initial starting files. ICMGG* are the initial gridpoint files, ICMSH* are the initial spectral fields.
Note that part of the file name is 'oifs'. This is known as the 'experiment id' (exptid). It is also encoded into the Grib file itself.
Use the 'grib_ls' command to examine the initial files:
Code Block |
---|
grib_ls ICMGGgc1dINITICMGGoifsINIT grib_lsICMSHgc1dINIT ICMSHoifsINIT |
Fortran namelist
The file: namelistfc is the model 'NAMELIST'. It contains a list of variable settings or 'switches' that control what the model does. These variables are grouped into separate fortran namelists.
The model will read this file when it starts up. There are many options to control the model. For more information, it is best and recommended to check the comments in the code. Please ask if you are interested.
Creating the OpenIFS namelist
Set the ensemble member
In IFS, each ensemble member uses the stochastic physics schemes to generate uncertainty. A random number 'seed' is used by the stochastic scheme to generate a different forecast. This random number seed is changed by altering the ensemble member value in the model's namelist. Each ensemble member must have a unique number and therefore random number seed, in order to produce a different forecast.
Exercise
Info |
---|
Make sure you are in the 'pf001' directory. Make a copy of the namelist file and edit it to set the ensemble member value to the one you are given.Make sure you are in the 'pf001' directory. |
Code Block |
---|
cp namelistfc mynamelist
vigedit mynamelist # or use your favourite text editor; vi, emacs etc. |
and make the following changes to the namelist variables:
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
CTYPE="pf", ! change the type of forecast to 'perturbed' from 'control' NENSFNB=2, ! change the ensemble member number to your assigned value. |
The variable NENSFNB is important as this determines the random number seed used by the stochastic schemes, by altering the ensemble member value.
Code Block | ||||
---|---|---|---|---|
| ||||
diff mynamelist namelistfc |
Note |
---|
This is different from how the ensemble would be run in practice. Ensemble member '2' would be run in directory pf002, member number '3' in pf003 and so on. But to keep this exercise simple, everyone will use the same experiment directory name but with a different ensemble member number. |
Create and submit Cray batch job
The next steps are to create and submit a batch job to run on the ECMWF Cray XC30 system.
To create the batch job, use the oifs_run command. This creates a small batch job file ready to submit.
Panel | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||
For this workshop, exercise the only arguments to use are: -e, -l, -f, -x and -q. Other options are available which can be left to their default values. Note that ! the -f argument, the length of the forecast, is specified as: dNN, for example d10 means '10 days'. | ||||||||||||
Info | ||||||||||||
Exercise
...
Use the oifs_run
command to create the job.
...
The exptid to use with the '-e
...
'
...
options passed to oifs_run
must be "oifs". It is part of the filename of the initial files for the model (the files starting with "ICM").
The '-l' option is the name of the fortran namelist file you edited above.
...
Code Block | ||||
---|---|---|---|---|
| ||||
oifs_run -e oifs -l mynamelist |
This will create a file 'job1
'.
Submit this job:
Code Block |
---|
qsub job1 |
Check the job status:
Code Block |
---|
qstatu |
These forecasts complete in about 15mins.
Info |
---|
If you get the error message:
execute the command:
to correct set the environment. |
During the course of the model run, files with names ICM*+* like ICMGGoifs+00000, ICMGGoifs+00012 etc will appear in the experiment directory.
These are the model output files. There is 1 per output interval. In these experiments the output interval is every 3 hours.
Checking the model output
After the qstatu command shows that the job has completed, there will be a 'batch log file' in the pf001 directory:
Code Block | ||||
---|---|---|---|---|
| ||||
oifs_trcray1troifs1.o1481792 | ||||
Info | ||||
Exercise
Using your favourite text editor, check this file to make sure the model has run correctly. It's usually best to start from the bottom of the file and scroll up.
If you see:
Code Block |
---|
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
OPENIFS JOB FAILED
!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
the model run has failed.
The output from each model run goes into a directory named according to this run number. e.g. the default is 'output1'. So job2 would write the output to 'output2' and so on.
If the model fails, there are 2 files to look at in the appropriate 'output' directory:
NODE.001_01 : this is the output from the model as it is running (i.e. all the print/write fortran statements). This is a large file, so in case of errors, start from the bottom up!
oifs.log : this is where the model writes any error messages.
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
When the model completes the forecast successfully, the following files will be found in the output directory. Files with '+' in the name are the model output files. ICMSH are the spectral fields, ICMGG are the gridpoint fields.
ICMGGoifs+000000 ICMGGoifs+000048 ICMGGoifs+000096 ICMGGoifs+000144 ICMSHoifs+000016 ICMSHoifs+000064 ICMSHoifs+000112 ICMSHoifsICMGGgc1d+000000 ICMGGgc1d+000048 ICMGGgc1d+000096 ICMGGgc1d+000144 ICMSHgc1d+000016 ICMSHgc1d+000064 ICMSHgc1d+000112 ICMSHgc1d+000160 oifs.log ICMGGgc1dICMGGoifs+000016 ICMGGgc1dICMGGoifs+000064 ICMGGgc1dICMGGoifs+000112 ICMGGgc1dICMGGoifs+000160 ICMSHgc1dICMSHoifs+000032 ICMSHgc1dICMSHoifs+000080 ICMSHgc1dICMSHoifs+000128 NODE.001_01 ICMGGgc1dICMGGoifs+000032 ICMGGgc1dICMGGoifs+000080 ICMGGgc1dICMGGoifs+000128 ICMSHgc1dICMSHoifs+000000 ICMSHgc1dICMSHoifs+000048 ICMSHgc1dICMSHoifs+000096 ICMSHgc1dICMSHoifs+000144 ifs.stat |
Run number
Each model run has a number. This is '1' by default but can be changed using the -x option to the oifs_run
command.
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
oifs_run -e gc1d -l mynamelist -x 2 -f d5 -q |
Preparing files for Metview
The next step is to take the output of the successful run and prepare it for the Metview based exercises.
As the data files and Metview scripts have been optimized by selecting specific areas, we have made a script available to do the work.
Code Block | ||||
---|---|---|---|---|
| ||||
oifs_to_mv -e pf001 |
This will generate files in scratch/metview.
This task is only to prepare the output fields for Metview. The tutor will then use these files and combine them with the other ensemble members to create the final files that will be used in the plotting exercises.
If errors are encountered please let us know. If the command completes successfully (it will take a while), this task is complete. There is no need to inspect the files.
Info |
---|
This script may take a few minutes to run. |
Excerpt Include | ||||||
---|---|---|---|---|---|---|
|