| Horizontal Navigation Bar | ||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
By modifying this variable, it is possible to control where and how a job file will run.
The ECF_JOB variable contains the job file path and ECF_JOBOUT contains
the path of a file where the output of the job will be written. The default command:
| Note |
|---|
| ECF_JOB_CMD = %ECF_JOB% 1> %ECF_JOBOUT% 2>&1 & |
Let us run the tasks on a remote machine. For that, we could use the UNIX command ssh.
We assume that all the files are visible on all the hosts, e.g. using NFS.
In the examples below replace the string ?????? with a host name of your choice.
| Note |
|---|
The environment of a task running on a remote host can be different from that of a task running locally. It depends on how your system is set up. The head.h should already be using the correct PATH, to allow child commands to be used. > export PATH=$PATH:/usr/local/apps/ecflow/ |
To use ssh requires your public key to be available on the destination machine.
Check if you can log on to the remote machine through ssh without a password check.
If you need to enter a password you will need to add your public key on the destination machine. To do this issue the following commands:
| Code Block | ||||
|---|---|---|---|---|
| ||||
REMOTE_HOST=?????? # change me ssh $USER@$REMOTE_HOST mkdir -p \$HOME/.ssh # if you are prompted for a password use your Training password that was provided cat $HOME/.ssh/id_rsa.pub || ssh-keygen -t rsa -b 2048 cat $HOME/.ssh/id_rsa.pub | ssh $USER@$REMOTE_HOST 'cat >> $HOME/.ssh/authorized_keys' |
Modify the family f5 so that all its tasks will run on another machine in the classroom.
Text
| Code Block |
|---|
# Definition of the suite test
suite test
edit ECF_INCLUDE "$HOME/course"
edit ECF_HOME "$HOME/course"
limit l1 2
family f5
edit HOST ??????
edit ECF_OUT /tmp/$USER
edit ECF_JOB_CMD "ssh %HOST% 'mkdir -p %ECF_OUT%/%SUITE%/%FAMILY% && %ECF_JOB% > %ECF_JOBOUT% 2>&1 &'"
inlimit l1
edit SLEEP 20
task t1
task t2
task t3
task t4
task t5
task t6
task t7
task t8
task t9
endfamily
endsuite |
If your login shell is csh, you should define ECF_JOB_CMD as:
| Code Block |
|---|
edit ECF_JOB_CMD "ssh %HOST% 'mkdir -p %ECF_OUT%/%SUITE%/%FAMILY%; %ECF_JOB% >& %ECF_JOBOUT%'" |
Python
In python modify the function create_family_f5() created in the earlier page, to add HOST,ECF_OUT,ECF_LOGHOST,ECF_LOGPORT, and ECF_JOB_CMD:
| Code Block | ||||
|---|---|---|---|---|
| ||||
#!/usr/bin/env python2.7 import os from ecflow import Defs,Suite,Family,Task,Edit,Trigger,Complete,Event,Meter,Time,Day,Date,Label, \ RepeatString,RepeatInteger,RepeatDate,InLimit,Limit def create_family_f5() : return Family("f5", InLimit("l1"), Edit(SLEEP=20, HOST='?????', ECF_OUT = '/tmp/%s' % os.getenv("USER"), ECF_LOGHOST='%HOST%', ECF_LOGPORT='?????', # port=$((35000 + $(id -u))) run this on the command line ECF_JOB_CMD,="ssh %HOST% 'mkdir -p %ECF_OUT%/%SUITE%/%FAMILY%; %ECF_JOB% > %ECF_JOBOUT% 2>&1 &'"), [ Task('t{}'.format(i)) for i in range(1,10) ] ) print ("Creating suite definition") home = os.path.join(os.getenv("HOME"),"course") defs = Defs( Suite("test", Edit(ECF_INCLUDE=home,ECF_HOME=home), Limit("l1",2), create_family_f5())) print(defs) print("Checking job creation: .ecf -> .job0") print(defs.check_job_creation()) print("Checking trigger expressions") assert len(defs.check()) == 0,defs.check() print("Saving definition to file 'test.def'") defs.save_as_defs("test.def") |
Logserver
We can view the output on the remote machine (class??) by using a log server.
This assumes you have defined variables ECF_LOGHOST and ECF_LOGPORT in your definition.
Launch the log server on a remote machine:
| Code Block | ||
|---|---|---|
| ||
ssh $USER@class01 /usr/local/apps/ecflow/5.5.1/bin/ecflow_logserver.sh -d /tmp/$USER -m /tmp/$USER:/tmp/$USER |
What to do
- Modify PATH environment variable in head.h
- Change the suite definition
- Replace the suite definition
- It may not work immediately. Have a look in the file $HOME/course/host.port.ecf.log to see why.
- Add a uname -n to your ecf ECF script to see what machine the task is running on.
- What do you need to do in order to have the task /test/f5/t9 run on another machine? Try your solution.
Launch the logserver on a remote machine:
| Code Block | ||
|---|---|---|
| ||
ssh class01 /usr/local/apps/ecflow/4.8.0/bin/start_logserver -d /tmp/$USER -m /tmp/$USER:/tmp/$USER |
Edit the definition file:
| Code Block |
|---|
HOST=class01
edit ECF_LOGHOST $HOST
edit ECF_LOGPORT $(35000 + $(id -u))
edit HOST $HOST
edit ECF_OUT /tmp/$USER |
You can test the logserver from the command line, requesting the output from a task:
| Code Block |
|---|
telnet class01 -p $(35000 + $(id -u))
# get $ECF_JOBOUT
# list $ECF_JOBOUT |
- Create a log server, to access the remote output
| Horizontal Navigation Bar | ||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...