<div id="glossary"> <span id="index-0"></span><span id="id1"></span> <dl > <dt id="term-aborted">aborted</dt> <dd><p >Is a <a href="#term-node"><em >node</em></a> <a href="#term-status"><em >status</em></a>.</p> <p >When the ECF_JOB_CMD fails or the <a href="#term-job-file"><em >job file</em></a> sends a <a href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> –abort <a href="#term-child-command"><em >child command</em></a>, then the task is placed into a aborted state.</p> </dd> <dt id="term-active">active</dt> <dd><p >Is a <a href="#term-node"><em >node</em></a> <a href="#term-status"><em >status</em></a>.</p> <p >If <a href="#term-job-creation"><em >job creation</em></a> was successful, and <a href="#term-job-file"><em class="xref std std-term">job file</em></a> has started, then the <a href="#term-ecflow-client"><em >ecflow_client</em></a> –init <a href="#term-child-command"><em class="xref std std-term">child command</em></a> is received by the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> and the <a href="#term-task"><em >task</em></a> is placed into a active state</p> </dd> <dt id="term-autocancel">autocancel</dt> <dd><p >autocancel is a way to automatically delete a <a href="#term-node"><em class="xref std std-term">node</em></a> which has completed.</p> <p>The delete may be delayed by an amount of time in hours and minutes or expressed in days. Any node may have a single autocancel attribute. If the auto cancelled node is referenced in the <a href="#term-trigger"><em >trigger</em></a> expression of other nodes it may leave the node waiting. This can be solved by making sure the <a href="#term-trigger"><em class="xref std std-term">trigger</em></a> expression also checks for the <a href="#term-unknown"><em >unknown</em></a> state. i.e.:</p> <div ><pre>trigger node_to_cancel == complete or node_to_cancel == unknown</pre> </div> <p>This guards against the ‘node_to_cancel’ being undefined or deleted</p> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Autocancel" title="ecflow.Autocancel"><tt ><span >ecflow.Autocancel</span></tt></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_autocancel" title="ecflow.Node.add_autocancel"><tt class="xref py py-class docutils literal"><span >ecflow.Node.add_autocancel</span></tt></a>. For text BNF see <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-autocancel"><tt ><span >autocancel</span></tt></a></p> </dd> <dt id="term-check-point">check point</dt> <dd><p >The check point file is like the <a class="reference internal" href="#term-suite-definition"><em >suite definition</em></a>, but includes all the state information.</p> <p>It is periodically saved by the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>.</p> <p>It can be used to recover the state of the node tree should server die, or machine crash.</p> <p>By default when a <a href="#term-ecflow-server"><em >ecflow_server</em></a> is started it will look to load the check point file.</p> <p >The default check point file name is <host>.<port>.ecf.check. This can be overridden by the ECF_CHECK environment variable</p> </dd> <dt id="term-child-command">child command</dt> <dd><p >Child command’s(or task requests) are called from within the <a href="#term-ecf-script"><em >ecf script</em></a> files. They include:</p> <blockquote> <div><table border="1" > <colgroup> <col width="37%" /> <col width="63%" /> </colgroup> <thead valign="bottom"> <tr ><th >Child Command</th> <th >Description</th> </tr> </thead> <tbody valign="top"> <tr ><td><a href="#term-ecflow-client"><em >ecflow_client</em></a> –init</td> <td>Sets the <a href="#term-task"><em class="xref std std-term">task</em></a> to the <a href="#term-active"><em >active</em></a> <a href="#term-status"><em class="xref std std-term">status</em></a></td> </tr> <tr ><td><a href="#term-ecflow-client"><em >ecflow_client</em></a> –event</td> <td>Set an event</td> </tr> <tr ><td><a href="#term-ecflow-client"><em >ecflow_client</em></a> –meter</td> <td>Change a meter</td> </tr> <tr ><td><a href="#term-ecflow-client"><em >ecflow_client</em></a> –label</td> <td>Change a label</td> </tr> <tr ><td><a href="#term-ecflow-client"><em >ecflow_client</em></a> –wait</td> <td>wait for a expression to evaluate.</td> </tr> <tr ><td><a href="#term-ecflow-client"><em >ecflow_client</em></a> –abort</td> <td>Sets the <a href="#term-task"><em class="xref std std-term">task</em></a> to the <a href="#term-aborted"><em >aborted</em></a> <a href="#term-status"><em class="xref std std-term">status</em></a></td> </tr> <tr ><td><a href="#term-ecflow-client"><em >ecflow_client</em></a> –complete</td> <td>Sets the <a href="#term-task"><em class="xref std std-term">task</em></a> to the <a href="#term-complete"><em >complete</em></a> <a href="#term-status"><em class="xref std std-term">status</em></a></td> </tr> </tbody> </table> </div></blockquote> <p >The following environment variables must be set for the child commands. ECF_NODE, ECF_NAME ,ECF_PASS and ECF_RID. See <a href="#term-ecflow-client"><em >ecflow_client</em></a>.</p> </dd> <dt id="term-clock">clock</dt> <dd><p >A clock is an attribute of a <a href="#term-suite"><em >suite</em></a>.</p> <p>A gain can be specified to offset from the given date.</p> <p>The hybrid and real clock’s always runs in phase with the system clock (UTC in UNIX) but can have any offset from the system clock.</p> <p>The clock can be :</p> <blockquote> <div><ul > <li><a href="#term-hybrid-clock"><em class="xref std std-term">hybrid clock</em></a></li> <li><a href="#term-real-clock"><em class="xref std std-term">real clock</em></a></li> <li><a href="#term-virtual-clock"><em class="xref std std-term">virtual clock</em></a></li> </ul> </div></blockquote> <p><a href="#term-time"><em class="xref std std-term">time</em></a>, <a href="#term-day"><em >day</em></a> and <a href="#term-date"><em >date</em></a> and <a href="#term-cron"><em class="xref std std-term">cron</em></a> <a href="#term-dependencies"><em >dependencies</em></a> work a little differently under the clocks.</p> <p>The default clock type is hybrid.</p> <p>If the <a href="#term-ecflow-server"><em >ecflow_server</em></a> is <a class="reference internal" href="#term-shutdown"><em >shutdown</em></a> or <a href="#term-halted"><em class="xref std std-term">halted</em></a> the job <a href="#term-scheduling"><em >scheduling</em></a> is suspended. If this suspension is left for period of time, then it can affect task submission under <strong>hybrid</strong> and <strong>real</strong> clocks. In particular it will affect <a href="#term-task"><em >task</em></a> s with <a href="#term-time"><em class="xref std std-term">time</em></a>, <a href="#term-today"><em >today</em></a> or <a href="#term-cron"><em >cron</em></a> <a href="#term-dependencies"><em class="xref std std-term">dependencies</em></a>.</p> <blockquote> <div><ul> <li><p ><a href="#term-dependencies"><em >dependencies</em></a> with time series, can result in missed time slots:</p> <div ><pre>time 10:00 20:00 00:15 # If server is suspended > 15 minutes, time slots can be missed time +00:05 20:00 00:15 # start 5 minutes after the start of the suite, then every 15m until 20:00</pre> </div> </li> <li><p >When the server is placed back into <a class="reference internal" href="#term-running"><em >running</em></a> state any time <a href="#term-dependencies"><em >dependencies</em></a> with an expired time slot are submitted straight away. i.e if <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> is <a href="#term-halted"><em class="xref std std-term">halted</em></a> at 10:59 and then placed back into <a href="#term-running"><em class="xref std std-term">running</em></a> state at 11:20</p> <blockquote> <div><p>time 11:00</p> </div></blockquote> <p>Then any <a href="#term-task"><em class="xref std std-term">task</em></a> with a expired single time slot dependency will be submitted straight away.</p> </li> </ul> </div></blockquote> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Clock" title="ecflow.Clock"><tt ><span >ecflow.Clock</span></tt></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Suite.add_clock" title="ecflow.Suite.add_clock"><tt class="xref py py-class docutils literal"><span >ecflow.Suite.add_clock</span></tt></a>. For text BNF see <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-clock"><tt ><span >clock</span></tt></a></p> </dd> <dt id="term-complete">complete</dt> <dd><p >Is a <a href="#term-node"><em >node</em></a> <a href="#term-status"><em >status</em></a>.</p> <p>The node can be set to complete:</p> <blockquote > <div><p>By the <a href="#term-complete-expression"><em >complete expression</em></a></p> <p>At job end when the <a href="#term-task"><em >task</em></a> receives the <a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> –complete <a href="#term-child-command"><em >child command</em></a></p> </div></blockquote> </dd> <dt id="term-complete-expression">complete expression</dt> <dd><p >Force a node to be complete <strong>if</strong> the expression evaluates, without running any of the nodes.</p> <p>This allows you to have tasks in the suite which a run only if others fail. In practice the node would need to have a <a href="#term-trigger"><em >trigger</em></a> also.</p> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Expression" title="ecflow.Expression"><tt ><span >ecflow.Expression</span></tt></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_complete" title="ecflow.Node.add_complete"><tt class="xref py py-class docutils literal"><span >ecflow.Node.add_complete</span></tt></a></p> </dd> <dt id="term-cron">cron</dt> <dd><p >Like <a href="#term-time"><em >time</em></a>, cron defines time dependency for a <a href="#term-node"><em class="xref std std-term">node</em></a>, but it will be repeated indefinitely</p> <div ><pre>cron 11:00 cron 10:00 22:00 00:30 # <start> <finish> <increment></pre> </div> <p>When the node becomes complete it will be <a href="#term-queued"><em >queued</em></a> immediately. This means that the suite will never complete, and the output is not directly accessible through <a href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a></p> <p>If tasks abort, the <a href="#term-ecflow-server"><em >ecflow_server</em></a> will not schedule it again.</p> <p>If the time the job takes to complete is longer than the interval a “slot” is missed, e.g.</p> <div ><pre>cron 10:00 20:00 01:00</pre> </div> <p>if the 10:00 run takes more than an hour, the 11:00 run will never occur.</p> <p>If the cron defines months, days of the month, or week days or a single time slot the it relies on a day change, hence if a <a href="#term-hybrid-clock"><em >hybrid clock</em></a> is defined, then it will be set to <a href="#term-complete"><em >complete</em></a> at the beginning of the <a href="#term-suite"><em class="xref std std-term">suite</em></a>, without running the corresponding job. Otherwise under a hybrid clock the <a href="#term-suite"><em >suite</em></a> would never <a href="#term-complete"><em class="xref std std-term">complete</em></a>.</p> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Cron" title="ecflow.Cron"><tt ><span >ecflow.Cron</span></tt></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_cron" title="ecflow.Node.add_cron"><tt class="xref py py-class docutils literal"><span >ecflow.Node.add_cron</span></tt></a>. For text BNF see <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-cron"><tt ><span >cron</span></tt></a></p> </dd> <dt id="term-date">date</dt> <dd><p >This defines a date dependency for a node.</p> <p>There can be multiple date dependencies. The European format is used for dates, which is: dd.mm.yy as in 31.12.2007. Any of the three number fields can be expressed with a wildcard <cite>*</cite> to mean any valid value. Thus, 01.*.* means the first day of every month of every year.</p> <p>If a <a href="#term-hybrid-clock"><em class="xref std std-term">hybrid clock</em></a> is defined, any node held by a date dependency will be set to <a href="#term-complete"><em >complete</em></a> at the beginning of the <a href="#term-suite"><em class="xref std std-term">suite</em></a>, without running the corresponding job. Otherwise under a hybrid clock the <a href="#term-suite"><em >suite</em></a> would never <a href="#term-complete"><em class="xref std std-term">complete</em></a>.</p> <p >For python see: <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Date" title="ecflow.Date"><tt ><span >ecflow.Date</span></tt></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_date" title="ecflow.Node.add_date"><tt class="xref py py-class docutils literal"><span >ecflow.Node.add_date</span></tt></a>. For text BNF see <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-date"><tt ><span >date</span></tt></a></p> </dd> <dt id="term-day">day</dt> <dd><p >This defines a day dependency for a node.</p> <p>There can be multiple day dependencies.</p> <p>If a <a href="#term-hybrid-clock"><em class="xref std std-term">hybrid clock</em></a> is defined, any node held by a day dependency will be set to <a href="#term-complete"><em >complete</em></a> at the beginning of the <a href="#term-suite"><em class="xref std std-term">suite</em></a>, without running the corresponding job. Otherwise under a hybrid clock the <a href="#term-suite"><em >suite</em></a> would never <a href="#term-complete"><em class="xref std std-term">complete</em></a>.</p> <p >For python see: <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Day" title="ecflow.Day"><tt ><span >ecflow.Day</span></tt></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_day" title="ecflow.Node.add_day"><tt ><span >ecflow.Node.add_day</span></tt></a>. For text BNF see <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-day"><tt ><span >day</span></tt></a></p> </dd> <dt id="term-defstatus">defstatus</dt> <dd><p >Defines the default <a href="#term-status"><em >status</em></a> for a task/family to be assigned to the <a href="#term-node"><em >node</em></a> when the begin command is issued.</p> <p>By default <a href="#term-node"><em class="xref std std-term">node</em></a> gets queued when you use begin on a <a href="#term-suite"><em class="xref std std-term">suite</em></a>. defstatus is useful in preventing suites from running automatically once begun or in setting tasks complete so they can be run selectively.</p> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.DState" title="ecflow.DState"><tt ><span >ecflow.DState</span></tt></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_defstatus" title="ecflow.Node.add_defstatus"><tt class="xref py py-class docutils literal"><span >ecflow.Node.add_defstatus</span></tt></a>. For text BNF see <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-defstatus"><tt ><span >defstatus</span></tt></a></p> </dd> <dt id="term-dependencies">dependencies</dt> <dd><p >Dependencies are attributes of node, that can suppress/hold a <a href="#term-task"><em class="xref std std-term">task</em></a> from taking part in <a href="#term-job-creation"><em >job creation</em></a>.</p> <p>They include <a href="#term-trigger"><em >trigger</em></a>, <a href="#term-date"><em >date</em></a>, <a href="#term-day"><em class="xref std std-term">day</em></a>, <a href="#term-time"><em >time</em></a>, <a href="#term-today"><em >today</em></a>, <a href="#term-cron"><em class="xref std std-term">cron</em></a>, <a href="#term-complete-expression"><em >complete expression</em></a>, <a href="#term-inlimit"><em >inlimit</em></a> and <a href="#term-limit"><em >limit</em></a>.</p> <p>A <a href="#term-task"><em class="xref std std-term">task</em></a> that is dependent can not be started as long as some dependency is holding it or any of its <strong>parent</strong> <a href="#term-node"><em class="xref std std-term">node</em></a> s.</p> <p >The <a href="#term-ecflow-server"><em >ecflow_server</em></a> will check the dependencies every minute, during normal <a class="reference internal" href="#term-scheduling"><em class="xref std std-term">scheduling</em></a> <strong>and</strong> when any <a href="#term-child-command"><em class="xref std std-term">child command</em></a> causes a state change in the <a href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a>.</p> </dd> <dt id="term-directives">directives</dt> <dd><p >Directives start with a % charater. This is referred to as <a href="#term-ecf-micro"><em class="xref std std-term">ECF_MICRO</em></a> character.</p> <p>The directives are used in two main context.</p> <blockquote> <div><ul> <li><p >Preprocessing directives. In this case the directive starts as the <strong>first</strong> character on a line in a <a href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> file. See the table below which shows the allowable values. Only one directive is allowed on the line.</p> </li> <li><p >Variable directives. We use two <a class="reference internal" href="#term-ecf-micro"><em class="xref std std-term">ECF_MICRO</em></a> characters ie %VAR%, in this case they can occur <strong>anywhere</strong> on the line and in any number.</p> <div ><pre>%CAR% %TYPE% %WISHLIST%</pre> </div> <p>These directives take part in <a href="#term-variable-substitution"><em >variable substitution</em></a>.</p> <p>If the micro characters are not paired (i.e uneven) then <a class="reference internal" href="#term-variable-substitution"><em class="xref std std-term">variable substitution</em></a> can not take place hence an error message is issued.</p> <div ><pre>port=%ECF_PORT # error issued since '%' micro character are not paired.</pre> </div> <p>However an uneven number of micro character are allowed, <strong>If</strong> the line begins with ‘#’ comment charcter.</p> <div ><div ><pre><span ># This is a comment line with a single micro character % no error issued</span> <span ># port=%ECF_PORT again no error issued</span> </pre></div> </div> </li> </ul> </div></blockquote> <p>Directives are expanded during <a href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a>. Examples include:</p> <table border="1" > <colgroup> <col width="22%" /> <col width="78%" /> </colgroup> <thead valign="bottom"> <tr ><th >Symbol</th> <th >Meaning</th> </tr> </thead> <tbody valign="top"> <tr ><td>%include <filename></td> <td>%ECF_INCLUDE% directory is searched for the <tt class="file docutils literal"><span >filename</span></tt> and the contents included into the job file. If that variable is not defined ECF_HOME is used. If the ECF_INCLUDE is defined but the file does not exist, then we look in ECF_HOME. This allows specific files to be placed in ECF_INCLUDE and the more general/common include files to be placed in ECF_HOME. This is the recommended format</td> </tr> <tr ><td>%include “filename”</td> <td>Include the contents of the file: %ECF_HOME%/%SUITE%/%FAMILY%/filename into the job.</td> </tr> <tr ><td>%include filename</td> <td>Include the contents of the file <tt ><span >filename</span></tt> into the output. The only form that can be used safely must start with a slash ‘/’</td> </tr> <tr ><td>%includenopp filename</td> <td>Same as %include, but the file is not interpreted at all.</td> </tr> <tr ><td>%comment</td> <td>Start’s a comment, which is ended by %end directive. The section enclosed by %comment - %end is removed during <a class="reference internal" href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a></td> </tr> <tr ><td>%manual</td> <td>Start’s a manual, which is ended by %end directive. The section enclosed by %manual - %end is removed during <a class="reference internal" href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a>. The manual directive is used to create the <a href="#term-manual-page"><em >manual page</em></a> show in <a href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a>.</td> </tr> <tr ><td>%nopp</td> <td>Stop pre-processing until a line starting with %end is found. No interpretation of the text will be done( i.e. no variable substitutions)</td> </tr> <tr ><td>%end</td> <td>End processing of %comment or %manual or %nopp</td> </tr> <tr ><td>%ecfmicro CHAR</td> <td>Change the directive character, to the character given. If set in an include file the effect is retained for the rest of the job( or until set again). It should be noted that the ecfmicro directive specified in the <a href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> file, does <strong>not</strong> effect the variable substitution for ECF_JOB_CMD, ECF_KILL_CMD or ECF_STATUS_CMD variables. They still use <a href="#term-ecf-micro"><em class="xref std std-term">ECF_MICRO</em></a>. If no ecfmicro directive exists, we default to using <a href="#term-ecf-micro"><em class="xref std std-term">ECF_MICRO</em></a> from the <a href="#term-suite-definition"><em >suite definition</em></a></td> </tr> </tbody> </table> </dd> <dt id="term-ecf-file-location-algorithm">ecf file location algorithm</dt> <dd><p ><a href="#term-ecflow-server"><em >ecflow_server</em></a> and job creation checking uses the following algorithm to locate the ‘.ecf’ file corresponding to a <a href="#term-task"><em >task</em></a>:</p> <ul> <li><p >ECF_SCRIPT</p> <p>First it uses the generated variable ECF_SCRIPT to locate the script. This variable is generated from: ECF_HOME/<path to task>.ecf</p> <p>Hence if the task path is /suite/f1/f2/t1, then ECF_SCRIPT=ECF_HOME/suite/f1/f2/t1.ecf</p> </li> <li><p >ECF_FILES</p> <p>Second it checks for the user defined ECF_FILES variable. If defined the value of this variable must correspond to a directory. This directory is searched in reverse order.</p> <p>i.e lets assume we have a task: /o/12/fc/model and ECF_FILES is defined as: /home/ecmwf/emos/def/o/ECFfiles</p> <p>The ecFlow will use the following search pattern.</p> <blockquote> <div><ol > <li>/home/ecmwf/emos/def/o/ECFfiles/o/12/fc/model.ecf</li> <li>/home/ecmwf/emos/def/o/ECFfiles/12/fc/model.ecf</li> <li>/home/ecmwf/emos/def/o/ECFfiles/fc/model.ecf</li> <li>/home/ecmwf/emos/def/o/ECFfiles/model.ecf</li> </ol> </div></blockquote> </li> <li><p >ECF_HOME</p> <p>Thirdly it searchs for the script in reverse order using ECF_HOME (i.e like ECF_FILES)</p> </li> </ul> <blockquote > <div><p>If this fails, than the <a href="#term-task"><em >task</em></a> is placed into the <a href="#term-aborted"><em class="xref std std-term">aborted</em></a> state. We can check that file can be located before loading the suites into the server.</p> <blockquote> <div><ul > <li><a href="/wiki/display/ECFLOW/Checking+job+creation#checking-job-creation"><em>Checking job creation</em></a></li> <li><a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Defs.check_job_creation" title="ecflow.Defs.check_job_creation"><tt class="xref py py-class docutils literal"><span >ecflow.Defs.check_job_creation</span></tt></a></li> </ul> </div></blockquote> </div></blockquote> </dd> <dt id="term-ecf-script">ecf script</dt> <dd><p >The ecFlow script refers to an ‘.ecf’ file.</p> <p>The script file is transformed into the <a href="#term-job-file"><em >job file</em></a> by the <a href="#term-job-creation"><em class="xref std std-term">job creation</em></a> process.</p> <p>The base name of the script file <strong>must</strong> match its corresponding <a href="#term-task"><em class="xref std std-term">task</em></a>. i.e t1.ecf , corresponds to the task of name ‘t1’. The script if placed in the ECF_FILES directory, may be re-used by multiple tasks belonging to different families, providing the <a href="#term-task"><em class="xref std std-term">task</em></a> name matches.</p> <p>The ecFlow script is similar to a UNIX shell script.</p> <p >The differences, however, includes the addition of “C” like pre-processing <a href="#term-directives"><em >directives</em></a> and ecFlow <a class="reference internal" href="#term-variable"><em class="xref std std-term">variable</em></a>‘s. Also the script <em>must</em> include calls to the <strong>init</strong> and <strong>complete</strong> <a href="#term-child-command"><em >child command</em></a> s so that the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> is aware when the job starts(i.e changes state to <a href="#term-active"><em class="xref std std-term">active</em></a>) and finishes ( i.e changes state to <a href="#term-complete"><em class="xref std std-term">complete</em></a>)</p> </dd> <dt id="term-ecf-dummy-task">ECF_DUMMY_TASK</dt> <dd><p >This is a user variable that can be added to <a href="#term-task"><em class="xref std std-term">task</em></a> to indicate that there is no associated <a href="#term-ecf-script"><em >ecf script</em></a> file.</p> <p>If this variable is added to <a href="#term-suite"><em >suite</em></a> or <a href="#term-family"><em class="xref std std-term">family</em></a> then all child tasks are treated as dummy.</p> <p >This stops the server from reporting an error during <a href="#term-job-creation"><em class="xref std std-term">job creation</em></a>.</p> </dd> <dt id="term-ecf-job">ECF_JOB</dt> <dd><p >This is a generated <a href="#term-variable"><em >variable</em></a>. If defines the path name location of the job file.</p> <p >The variable is composed as: ECF_HOME/ECF_NAME.job<ECF_TRYNO></p> </dd> <dt id="term-ecf-jobout">ECF_JOBOUT</dt> <dd><p >This is a generated <a href="#term-variable"><em >variable</em></a>. This variable defines the path name for the job output file. The variable is composed as:</p> <blockquote > <div>ECF_HOME/ECF_NAME.ECF_TRYNO</div></blockquote> </dd> <dt id="term-ecf-micro">ECF_MICRO</dt> <dd>This is a generated <a href="#term-variable"><em >variable</em></a>. The default value is %. This variable is used in <a href="#term-variable-substitution"><em >variable substitution</em></a> during command invocation and default directive character during <a href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a>. It can be overriden, but must be replaced by a single character.</dd> <dt id="term-ecf-name">ECF_NAME</dt> <dd>This is a generated <a href="#term-variable"><em >variable</em></a>. It defines the path name of the task.</dd> <dt id="term-63">ECF_SCRIPT</dt> <dd>This is a generated <a href="#term-variable"><em >variable</em></a>. If defines the path name for the <a href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a></dd> <dt id="term-ecf-tryno">ECF_TRYNO</dt> <dd><p >This is a generated <a href="#term-variable"><em >variable</em></a> that is used in file name generation. It represents the current try number for the <a href="#term-task"><em >task</em></a>.</p> <p >After <strong>begin</strong> it is set to 1. The number is advanced if the job is re-run. It is re-set back to 1 after a re-queue. It is used in output and <a href="#term-job-file"><em >job file</em></a> numbering. (i.e It avoids overwriting the <a href="#term-job-file"><em >job file</em></a> output during multiple re-runs)</p> </dd> <dt id="term-ecflow">ecFlow</dt> <dd><p >Is the ECMWF work flow manager.</p> <p>A general purpose application designed to schedule a large number of computer process in a heterogeneous environment.</p> <p >Helps computer jobs design, submission and monitoring both in the research and operation departments.</p> </dd> <dt id="term-ecflow-client">ecflow_client</dt> <dd><p >This executable is a command line program; it is used for all communication with the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>.</p> <p>To see the full range of commands that can be sent to the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> type the following in a UNIX shell:</p> <blockquote> <div>ecflow_client –help</div></blockquote> <p>This functionality is also provided by the <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#client-server-python-api"><em>Client Server API</em></a>.</p> <p>The following variables affect the execution of ecflow_client.</p> <p>Since the <a href="#term-ecf-script"><em >ecf script</em></a> can call ecflow_client( i.e <a href="#term-child-command"><em class="xref std std-term">child command</em></a>) then typically some are set in an include header. i.e. <a href="/wiki/display/ECFLOW/Understanding+Includes#head-h"><em>head.h</em></a>.</p> <table border="1" > <colgroup> <col width="12%" /> <col width="49%" /> <col width="8%" /> <col width="31%" /> </colgroup> <thead valign="bottom"> <tr ><th >Variable Name</th> <th >Explanation</th> <th >Compulsory</th> <th >Example</th> </tr> </thead> <tbody valign="top"> <tr ><td>ECF_NODE</td> <td>Name of the host running the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></td> <td>Yes</td> <td>pikachu</td> </tr> <tr ><td>ECF_NAME</td> <td>Path to the task</td> <td>Yes</td> <td>/suite/family/task</td> </tr> <tr ><td>ECF_PASS</td> <td>Jobs password.</td> <td>Yes</td> <td>(generated)</td> </tr> <tr ><td>ECF_RID</td> <td>Remote id. Allow easier job kill, and disambiguate a zombie from the real job.</td> <td>Yes</td> <td>(generated)</td> </tr> <tr ><td>ECF_PORT</td> <td>Port number of the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></td> <td>No</td> <td>3141.Must match <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></td> </tr> <tr ><td>ECF_TRYNO</td> <td>The number of times the job has run. This is allocated by the server and used in job/output file name generation.</td> <td>No</td> <td>(generated)</td> </tr> <tr ><td>ECF_HOSTFILE</td> <td>File that lists alternate hosts to try, if connection to main host fails</td> <td>No</td> <td>/home/user/avi/.ecfhostfile</td> </tr> <tr ><td>ECF_TIMEOUT</td> <td>Maximum time is seconds for the client to deliver message</td> <td>No</td> <td>24*3600 (default value)</td> </tr> <tr ><td>ECF_DENIED</td> <td>If server denies client communication and this flag is set, exit with an error. Avoids 24hr hour connection attempt to <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>.</td> <td>No</td> <td>1</td> </tr> <tr ><td>NO_ECF</td> <td>If set exit’s ecflow_client immediately with success. This allows the scripts to be tested independent of the server</td> <td>No</td> <td>1</td> </tr> </tbody> </table> </dd> <dt id="term-ecflow-server">ecflow_server</dt> <dd><p >This executable is the server.</p> <p>It is responsible for <a href="#term-scheduling"><em >scheduling</em></a> the jobs and responding to <a href="#term-ecflow-client"><em >ecflow_client</em></a> requests</p> <p>Multiple servers can be run on the same machine/host providing they are assigned a unique port number.</p> <p>The server record’s all request’s in the log file.</p> <p>The server will periodically(See ECF_CHECKINTERVAL) write out a <a href="#term-check-point"><em class="xref std std-term">check point</em></a> file.</p> <p>The following environment variables control the execution of the server and may be set before the start of the server. ecflow_server will start happily with out any of these variables being set, since all of them have default values.</p> <table border="1" > <colgroup> <col width="15%" /> <col width="51%" /> <col width="34%" /> </colgroup> <thead valign="bottom"> <tr ><th >Variable Name</th> <th >Explanation</th> <th >Default value</th> </tr> </thead> <tbody valign="top"> <tr ><td>ECF_HOME</td> <td>Home for all the <a href="#term-ecflow"><em >ecFlow</em></a> files</td> <td>Current working directory</td> </tr> <tr ><td>ECF_PORT</td> <td>Server port number. Must be unique</td> <td>3141</td> </tr> <tr ><td>ECF_LOG</td> <td>History or log file</td> <td><host>.<port>.ecf.log</td> </tr> <tr ><td>ECF_CHECK</td> <td>Name of the checkpoint file</td> <td><host>.<port>.ecf.check</td> </tr> <tr ><td>ECF_CHECKOLD</td> <td>Name of the backup checkpoint file</td> <td><host>.<port>.ecf.check.b</td> </tr> <tr ><td>ECF_CHECKINTERVAL</td> <td>Interval in second to save <a href="#term-check-point"><em >check point</em></a> file</td> <td>120</td> </tr> <tr ><td>ECF_LISTS</td> <td>White list file. Controls read/write access to the server for each user</td> <td><host>.<port>.ecf.lists</td> </tr> </tbody> </table> <p >The server can be in several states. The default when first started is <a href="#term-halted"><em class="xref std std-term">halted</em></a>, See <a href="#term-server-states"><em >server states</em></a></p> </dd> <dt id="term-ecflowview">ecflowview</dt> <dd><p >ecflowview executable is the GUI based client, that is used to visualise and monitor the hierarchical structure of the <a href="#term-suite-definition"><em >suite definition</em></a></p> <blockquote > <div><p>state changes in the <a href="#term-node"><em >node</em></a>‘s and the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>, using colour coding</p> <p>Attributes of the nodes and any <a href="#term-dependencies"><em class="xref std std-term">dependencies</em></a></p> <p><a href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> file and the corresponding <a class="reference internal" href="#term-job-file"><em >job file</em></a></p> </div></blockquote> </dd> <dt id="term-event">event</dt> <dd><p >The purpose of an event is to signal partial completion of a <a href="#term-task"><em class="xref std std-term">task</em></a> and to be able to trigger another job which is waiting for this partial completion.</p> <p>Only tasks can have events and they can be considered as an attribute of a <a href="#term-task"><em class="xref std std-term">task</em></a>.</p> <p>There can be many events and they are displayed as nodes.</p> <p>The event is updated by placing the –event <a class="reference internal" href="#term-child-command"><em >child command</em></a> in a <a href="#term-ecf-script"><em >ecf script</em></a>.</p> <p>An event has a number and possibly a name. If it is only defined as a number, its name is the text representation of the number without leading zeroes.</p> <p>For python see: <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Event" title="ecflow.Event"><tt ><span >ecflow.Event</span></tt></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_event" title="ecflow.Node.add_event"><tt class="xref py py-class docutils literal"><span >ecflow.Node.add_event</span></tt></a> For text BNF see <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-event"><tt ><span >event</span></tt></a></p> <p >Events can be referenced in <a href="#term-trigger"><em >trigger</em></a> and <a href="#term-complete-expression"><em class="xref std std-term">complete expression</em></a> s.</p> </dd> <dt id="term-extern">extern</dt> <dd><p >This allows an external <a href="#term-node"><em >node</em></a> to be used in a <a href="#term-trigger"><em class="xref std std-term">trigger</em></a> expression.</p> <p>All <a href="#term-node"><em class="xref std std-term">node</em></a>‘s in <a href="#term-trigger"><em >trigger</em></a>‘s must be known to <a href="#term-ecflow-server"><em >ecflow_server</em></a> by the end of the load command. No cross-suite <a href="#term-dependencies"><em >dependencies</em></a> are allowed unless the names of tasks outside the suite are declared as external. An external <a href="#term-trigger"><em class="xref std std-term">trigger</em></a> reference is considered unknown if it is not defined when the <a href="#term-trigger"><em >trigger</em></a> is evaluated. You are strongly advised to avoid cross-suite <a href="#term-dependencies"><em class="xref std std-term">dependencies</em></a>.</p> <p >Families and suites that depend on one another should be placed in a single <a href="#term-suite"><em class="xref std std-term">suite</em></a>. If you think you need cross-suite dependencies, you should consider merging the suites together and have each as a top-level family in the merged suite. For BNF see <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-extern"><tt ><span >extern</span></tt></a></p> </dd> <dt id="term-family">family</dt> <dd><p >A family is an organisational entity that is used to provide hierarchy and grouping. It consists of a collection of <a href="#term-task"><em >task</em></a>‘s and families.</p> <p>Typically you place tasks that are related to each other inside the same family, analogous to the way you create directories to contain related files. For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Family" title="ecflow.Family"><tt ><span >ecflow.Family</span></tt></a>. For BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-family"><tt ><span >family</span></tt></a></p> <p >It serves as an intermediate <a href="#term-node"><em >node</em></a> in a <a href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a>.</p> </dd> <dt id="term-halted">halted</dt> <dd>Is a <a href="#term-ecflow-server"><em >ecflow_server</em></a> state. See <a class="reference internal" href="#term-server-states"><em >server states</em></a></dd> <dt id="term-hybrid-clock">hybrid clock</dt> <dd><p >A hybrid <a href="#term-clock"><em >clock</em></a> is a complex notion: the date and time are not connected.</p> <p>The date has a fixed value during the complete execution of the <a href="#term-suite"><em class="xref std std-term">suite</em></a>. This will be mainly used in cases where the suite does not <a class="reference internal" href="#term-complete"><em >complete</em></a> in less than 24 hours. This guarantees that all tasks of this suite are using the same <a href="#term-date"><em class="xref std std-term">date</em></a>. On the other hand, the time follows the time of the machine.</p> <p>Hence the <a href="#term-date"><em class="xref std std-term">date</em></a> never changes unless specifically altered or unless the suite restarts, either automatically or from a begin command.</p> <p >Under a hybrid <a href="#term-clock"><em >clock</em></a> any <a href="#term-node"><em class="xref std std-term">node</em></a> held by a <a href="#term-date"><em >date</em></a> or <a href="#term-day"><em class="xref std std-term">day</em></a> dependency will be set to complete at the beginning of the suite. (i.e without its job ever running). Otherwise the <a href="#term-suite"><em class="xref std std-term">suite</em></a> would never <a href="#term-complete"><em >complete</em></a>.</p> </dd> <dt id="term-inlimit">inlimit</dt> <dd><p >The inlimit works in conjunction with <a class="reference internal" href="#term-limit"><em >limit</em></a>/<a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Limit" title="ecflow.Limit"><tt ><span >ecflow.Limit</span></tt></a> for providing simple load management</p> <p>inlimit is added to the <a href="#term-node"><em >node</em></a> that needs to be limited.</p> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.InLimit" title="ecflow.InLimit"><tt ><span >ecflow.InLimit</span></tt></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_inlimit" title="ecflow.Node.add_inlimit"><tt class="xref py py-class docutils literal"><span >ecflow.Node.add_inlimit</span></tt></a>. For text BNF see <a href="#term-inlimit"><em class="xref std std-term">inlimit</em></a></p> </dd> <dt id="term-job-creation">job creation</dt> <dd><p >Job creation or task invocation can be initiated manually via <a href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a> but also by the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> during <a href="#term-scheduling"><em >scheduling</em></a> when a <a href="#term-task"><em class="xref std std-term">task</em></a> (and <em>all</em> of its parent <a class="reference internal" href="#term-node"><em >node</em></a> s) is free of its <a href="#term-dependencies"><em >dependencies</em></a>.</p> <p>The process of job creation includes:</p> <blockquote> <div><p>o Generating a unique password ECF_PASS, which is placed in <a href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> during <a href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a>. See <a href="/wiki/display/ECFLOW/Understanding+Includes#head-h"><em>head.h</em></a></p> <p>o Locating <a href="#term-ecf-script"><em >ecf script</em></a> files , corresponding to the <a href="#term-task"><em class="xref std std-term">task</em></a> in the <a href="#term-suite-definition"><em >suite definition</em></a>, See <a href="#term-ecf-file-location-algorithm"><em >ecf file location algorithm</em></a></p> <p>o <a href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a> the contents of the <a class="reference internal" href="#term-ecf-script"><em >ecf script</em></a> file</p> </div></blockquote> <p>The steps above transforms an <a href="#term-ecf-script"><em >ecf script</em></a> to a <a href="#term-job-file"><em class="xref std std-term">job file</em></a> that can be submitted by performing <a href="#term-variable-substitution"><em >variable substitution</em></a> on the ECF_JOB_CMD <a href="#term-variable"><em class="xref std std-term">variable</em></a> and invoking the command.</p> <p>The running jobs will communicate back to the <a href="#term-ecflow-server"><em >ecflow_server</em></a> by calling <a href="#term-child-command"><em >child command</em></a>‘s.</p> <p>This causes <a href="#term-status"><em >status</em></a> changes on the <a class="reference internal" href="#term-node"><em >node</em></a>‘s in the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> and flags can be set to indicate various events.</p> <p>If a <a href="#term-task"><em class="xref std std-term">task</em></a> is to be treated as a dummy task( i.e. is used as a scheduling task) and is not meant to to be run, then a variable of name <a href="#term-ecf-dummy-task"><em class="xref std std-term">ECF_DUMMY_TASK</em></a> can be added.</p> <div ><div ><pre><span >task</span><span >.</span><span >add_variable</span><span >(</span><span >"ECF_DUMMY_TASK"</span><span >,</span><span >""</span><span >)</span> </pre></div> </div> </dd> <dt id="term-job-file">job file</dt> <dd><p >The job file is created by the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> during <a href="#term-job-creation"><em >job creation</em></a> using the <a href="#term-ecf-tryno"><em class="xref std std-term">ECF_TRYNO</em></a> <a href="#term-variable"><em >variable</em></a></p> <p>It is derived from the <a href="#term-ecf-script"><em >ecf script</em></a> after expanding the pre-processing <a href="#term-directives"><em >directives</em></a>.</p> <p>It has the form <task name>.job<<a href="#term-ecf-tryno"><em class="xref std std-term">ECF_TRYNO</em></a>>”, i.e. t1.job1.</p> <p>Note job creation checking will create a job file with an extension with zero. i.e ‘.job0’. See <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Defs.check_job_creation" title="ecflow.Defs.check_job_creation"><tt class="xref py py-class docutils literal"><span >ecflow.Defs.check_job_creation</span></tt></a></p> <p >When the job is run the output file has the <a class="reference internal" href="#term-ecf-tryno"><em class="xref std std-term">ECF_TRYNO</em></a> as the extension. i.e t1.1 where ‘t1’ represents the task name and ‘1’ the <a href="#term-ecf-tryno"><em class="xref std std-term">ECF_TRYNO</em></a></p> </dd> <dt id="term-label">label</dt> <dd><p >A label has a name and a value and is a way of <strong>displaying</strong> information in <a href="#term-ecflowview"><em >ecflowview</em></a></p> <p>By placing a label <a href="#term-child-command"><em >child command</em></a> s in the <a href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> the user can be informed about progress in <a href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a>.</p> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Label" title="ecflow.Label"><tt ><span >ecflow.Label</span></tt></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_label" title="ecflow.Node.add_label"><tt class="xref py py-class docutils literal"><span >ecflow.Node.add_label</span></tt></a>. For text BNF see <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-label"><tt ><span >label</span></tt></a></p> </dd> <dt id="term-late">late</dt> <dd><p >Define a tag for a node to be late.</p> <p>Suites cannot be late, but you can define a late tag for submitted in a suite, to be inherited by the families and tasks. When a node is classified as being late, the only action <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> takes is to set a flag. <a href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a> will display these alongside the <a href="#term-node"><em class="xref std std-term">node</em></a> name as an icon (and optionally pop up a window).</p> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Late" title="ecflow.Late"><tt ><span >ecflow.Late</span></tt></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_late" title="ecflow.Node.add_late"><tt class="xref py py-class docutils literal"><span >ecflow.Node.add_late</span></tt></a>. For text BNF see <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-late"><tt ><span >late</span></tt></a></p> </dd> <dt id="term-limit">limit</dt> <dd><p >Limits provide simple load management by limiting the number of tasks submitted by a specific <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>. Typically you either define limits on <a href="#term-suite"><em >suite</em></a> level or define a separate suite to hold limits so that they can be used by multiple suites.</p> <p>The limit max value can be changed on the command line</p> <div ><pre>>ecflow_client --alter change limit_max <limit-name> <new-limit-value> <path-to-limit> >ecflow_client --alter change limit_max limit 2 /suite</pre> </div> <p>It can also be changed in python</p> <div ><div ><pre><span >#!/usr/bin/env python2.7</span> <span >import</span> <span >ecflow</span> <span >try</span><span >:</span> <span >ci</span> <span >=</span> <span >ecflow</span><span >.</span><span >Client</span><span >()</span> <span >ci</span><span >.</span><span >alter</span><span >(</span><span >"/suite"</span><span >,</span><span >"change"</span><span >,</span><span >"limit_max"</span><span >,</span><span >"limit"</span><span >,</span> <span >"2"</span><span >)</span> <span >except</span> <span >RuntimeError</span><span >,</span> <span >e</span><span >:</span> <span >print</span> <span >"Failed: "</span> <span >+</span> <span >str</span><span >(</span><span >e</span><span >)</span> </pre></div> </div> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Limit" title="ecflow.Limit"><tt ><span >ecflow.Limit</span></tt></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_limit" title="ecflow.Node.add_limit"><tt class="xref py py-class docutils literal"><span >ecflow.Node.add_limit</span></tt></a>. For BNF see <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-limit"><tt ><span >limit</span></tt></a> and <a href="#term-inlimit"><em >inlimit</em></a></p> </dd> <dt id="term-manual-page">manual page</dt> <dd><p >Manual pages are part of the <a href="#term-ecf-script"><em >ecf script</em></a>.</p> <p>This is to ensure that the manual page is updated when the <a href="#term-ecf-script"><em class="xref std std-term">ecf script</em></a> is updated. The manual page is a very important operational tool allowing you to view a description of a task, and possibly describing solutions to common problems. The <a href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a> can be used to extract the manual page from the script file and is visible in <a href="#term-ecflowview"><em >ecflowview</em></a>. The manual page is the text contained within the %manual and %end <a href="#term-directives"><em class="xref std std-term">directives</em></a>. They can be seen using the manual button on <a href="#term-ecflowview"><em >ecflowview</em></a>.</p> <p>The text in the manual page in <strong>not</strong> included in the <a href="#term-job-file"><em class="xref std std-term">job file</em></a>.</p> <p>There can be multiple manual sections in the same <a class="reference internal" href="#term-ecf-script"><em >ecf script</em></a> file. When viewed they are simply concatenated. It is good practice to modify the manual pages when the script changes.</p> <p >The manual page may have the %include <a class="reference internal" href="#term-directives"><em class="xref std std-term">directives</em></a>.</p> </dd> <dt id="term-meter">meter</dt> <dd><p >The purpose of a meter is to signal proportional completion of a task and to be able to trigger another job which is waiting on this proportional completion.</p> <p>The meter is updated by placing the –meter <a class="reference internal" href="#term-child-command"><em >child command</em></a> in a <a href="#term-ecf-script"><em >ecf script</em></a>.</p> <p>For python see: <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Meter" title="ecflow.Meter"><tt ><span >ecflow.Meter</span></tt></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_meter" title="ecflow.Node.add_meter"><tt class="xref py py-class docutils literal"><span >ecflow.Node.add_meter</span></tt></a>. For text BNF see <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-meter"><tt ><span >meter</span></tt></a></p> <p >Meter’s can be referenced in <a class="reference internal" href="#term-trigger"><em >trigger</em></a> and <a href="#term-complete-expression"><em >complete expression</em></a> expressions.</p> </dd> <dt id="term-node">node</dt> <dd><p ><a href="#term-suite"><em >suite</em></a>, <a href="#term-family"><em >family</em></a> and <a href="#term-task"><em class="xref std std-term">task</em></a> form a hierarchy. Where a <a href="#term-suite"><em class="xref std std-term">suite</em></a> serves as the root of the hierarchy. The <a href="#term-family"><em class="xref std std-term">family</em></a> provides the intermediate nodes, and the <a href="#term-task"><em class="xref std std-term">task</em></a> provide the leaf’s.</p> <p>Collectively <a href="#term-suite"><em >suite</em></a>, <a href="#term-family"><em >family</em></a> and <a href="#term-task"><em class="xref std std-term">task</em></a> can be referred to as nodes.</p> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node" title="ecflow.Node"><tt ><span >ecflow.Node</span></tt></a>.</p> </dd> <dt id="term-pre-processing">pre-processing</dt> <dd><p >Pre-processing takes place during <a class="reference internal" href="#term-job-creation"><em >job creation</em></a> and acts on <a href="#term-directives"><em >directives</em></a> specified in <a href="#term-ecf-script"><em >ecf script</em></a> file.</p> <p>This involves:</p> <blockquote > <div><p>o expanding any include file <a href="#term-directives"><em >directives</em></a>. i.e similar to ‘c’ language pre-processing</p> <p>o removing comments and manual <a href="#term-directives"><em >directives</em></a></p> <p>o performing <a href="#term-variable-substitution"><em >variable substitution</em></a></p> </div></blockquote> </dd> <dt id="term-queued">queued</dt> <dd><p >Is a <a href="#term-node"><em >node</em></a> <a href="#term-status"><em >status</em></a>.</p> <p >After the begin command, the task <strong>without</strong> a <a href="#term-defstatus"><em class="xref std std-term">defstatus</em></a> are placed into the queued state</p> </dd> <dt id="term-real-clock">real clock</dt> <dd>A <a href="#term-suite"><em class="xref std std-term">suite</em></a> using a real <a href="#term-clock"><em >clock</em></a> will have its <a href="#term-clock"><em class="xref std std-term">clock</em></a> matching the clock of the machine. Hence the <a href="#term-date"><em class="xref std std-term">date</em></a> advances by one day at midnight.</dd> <dt id="term-repeat">repeat</dt> <dd><p >Repeats provide looping functionality. There can only be a single repeat on a <a href="#term-node"><em >node</em></a>.</p> <blockquote> <div><p>repeat day step [ENDDATE] # only for suites</p> <p>repeat integer VARIABLE start end [step]</p> <p>repeat enumerated VARIABLE first [second [third ...]]</p> <p>repeat string VARIABLE str1 [str2 ...]</p> <p>repeat file VARIABLE filename</p> <p>repeat date VARIABLE yyyymmdd yyyymmdd [delta]</p> </div></blockquote> <p>The repeat VARIABLE can be used in <a href="#term-trigger"><em >trigger</em></a> and <a href="#term-complete-expression"><em class="xref std std-term">complete expression</em></a> expressions.</p> <p>If a “repeat date” VARIABLE is used in a trigger expression then date arithmetic is used, when the expression uses addition and subtraction. i.e</p> <div ><div ><pre><span >defs</span> <span >=</span> <span >ecflow</span><span >.</span><span >Defs</span><span >()</span> <span >s1</span> <span >=</span> <span >defs</span><span >.</span><span >add_suite</span><span >(</span><span >"s1"</span><span >);</span> <span >t1</span> <span >=</span> <span >s1</span><span >.</span><span >add_task</span><span >(</span><span >"t1"</span><span >)</span><span >.</span><span >add_repeat</span><span >(</span> <span >ecflow</span><span >.</span><span >RepeatDate</span><span >(</span><span >"YMD"</span><span >,</span><span >20090101</span><span >,</span><span >20091231</span><span >,</span><span >1</span><span >)</span> <span >);</span> <span >t2</span> <span >=</span> <span >s1</span><span >.</span><span >add_task</span><span >(</span><span >"t2"</span><span >)</span><span >.</span><span >add_trigger</span><span >(</span><span >"t1:YMD - 1 eq 20081231"</span><span >);</span> <span >assert</span> <span >t2</span><span >.</span><span >evaluate_trigger</span><span >(),</span> <span >"Expected trigger to evaluate. 20090101 - 1 == 20081231"</span> </pre></div> </div> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_repeat" title="ecflow.Node.add_repeat"><tt class="xref py py-class docutils literal"><span >ecflow.Node.add_repeat</span></tt></a>, <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Repeat" title="ecflow.Repeat"><tt ><span >ecflow.Repeat</span></tt></a>, <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.RepeatDate" title="ecflow.RepeatDate"><tt ><span >ecflow.RepeatDate</span></tt></a>, <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.RepeatEnumerated" title="ecflow.RepeatEnumerated"><tt class="xref py py-class docutils literal"><span >ecflow.RepeatEnumerated</span></tt></a>, <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.RepeatInteger" title="ecflow.RepeatInteger"><tt class="xref py py-class docutils literal"><span >ecflow.RepeatInteger</span></tt></a>, <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.RepeatDay" title="ecflow.RepeatDay"><tt ><span >ecflow.RepeatDay</span></tt></a> For text BNF see <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-repeat"><tt ><span >repeat</span></tt></a></p> </dd> <dt id="term-running">running</dt> <dd>Is a <a href="#term-ecflow-server"><em >ecflow_server</em></a> state. See <a class="reference internal" href="#term-server-states"><em >server states</em></a></dd> <dt id="term-scheduling">scheduling</dt> <dd><p >The <a href="#term-ecflow-server"><em >ecflow_server</em></a> is responsible for <a href="#term-task"><em >task</em></a> scheduling.</p> <p >It will check <a href="#term-dependencies"><em >dependencies</em></a> in the <a href="#term-suite-definition"><em >suite definition</em></a> every minute. If these <a href="#term-dependencies"><em >dependencies</em></a> are free, the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> will submit the task. See <a href="#term-job-creation"><em class="xref std std-term">job creation</em></a>.</p> </dd> <dt id="term-server-states">server states</dt> <dd><p >The following tables reflects the <a class="reference internal" href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> capabilities in the different states</p> <table border="1" > <colgroup> <col width="22%" /> <col width="16%" /> <col width="16%" /> <col width="19%" /> <col width="26%" /> </colgroup> <thead valign="bottom"> <tr ><th >State</th> <th >User Request</th> <th >Task Request</th> <th >Job Scheduling</th> <th >Auto-Check-pointing</th> </tr> </thead> <tbody valign="top"> <tr ><td><a href="#term-running"><em >running</em></a></td> <td>yes</td> <td>yes</td> <td>yes</td> <td>yes</td> </tr> <tr ><td><a href="#term-shutdown"><em >shutdown</em></a></td> <td>yes</td> <td>yes</td> <td>no</td> <td>yes</td> </tr> <tr ><td><a href="#term-halted"><em >halted</em></a></td> <td>yes</td> <td>no</td> <td>no</td> <td>no</td> </tr> </tbody> </table> </dd> <dt id="term-shutdown">shutdown</dt> <dd>Is a <a href="#term-ecflow-server"><em >ecflow_server</em></a> state. See <a class="reference internal" href="#term-server-states"><em >server states</em></a></dd> <dt id="term-status">status</dt> <dd><p >Each <a href="#term-node"><em >node</em></a> in <a href="#term-suite-definition"><em >suite definition</em></a> has a status.</p> <p>Status reflects the state of the <a href="#term-node"><em >node</em></a>. In <a href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a> the background colour of the text reflects the status.</p> <p><a href="#term-task"><em class="xref std std-term">task</em></a> status are: <a href="#term-unknown"><em >unknown</em></a>, <a href="#term-queued"><em class="xref std std-term">queued</em></a>, <a href="#term-submitted"><em >submitted</em></a>, <a href="#term-active"><em class="xref std std-term">active</em></a>, <a href="#term-complete"><em >complete</em></a>, <a href="#term-aborted"><em class="xref std std-term">aborted</em></a> and <a href="#term-suspended"><em >suspended</em></a></p> <p ><a href="#term-ecflow-server"><em >ecflow_server</em></a> status are: <a href="#term-shutdown"><em class="xref std std-term">shutdown</em></a>, <a href="#term-halted"><em >halted</em></a>, <a href="#term-running"><em class="xref std std-term">running</em></a> this is shown on the root node in <a href="#term-ecflowview"><em >ecflowview</em></a></p> </dd> <dt id="term-submitted">submitted</dt> <dd><p >Is a <a href="#term-node"><em >node</em></a> <a href="#term-status"><em >status</em></a>.</p> <p >When the <a href="#term-task"><em >task</em></a> <a href="#term-dependencies"><em >dependencies</em></a> are resolved/free the <a href="#term-ecflow-server"><em >ecflow_server</em></a> places the task into a submitted state. However if the ECF_JOB_CMD fails, the task is placed into the <a href="#term-aborted"><em class="xref std std-term">aborted</em></a> state</p> </dd> <dt id="term-suite">suite</dt> <dd><p >A suite is organisational entity. It is serves as the root <a href="#term-node"><em class="xref std std-term">node</em></a> in a <a href="#term-suite-definition"><em >suite definition</em></a>. It should be used to hold a set of jobs that achieve a common function. It can be used to hold user <a href="#term-variable"><em class="xref std std-term">variable</em></a> s that are common to all of its children.</p> <p>Only a suite node can have a <a href="#term-clock"><em >clock</em></a>.</p> <p >It is a collection of <a href="#term-family"><em >family</em></a>‘s, <a href="#term-variable"><em class="xref std std-term">variable</em></a>‘s, <a href="#term-repeat"><em >repeat</em></a> and a single <a href="#term-clock"><em class="xref std std-term">clock</em></a> definition. For a complete list of attributes look at BNF for <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-suite"><tt ><span >suite</span></tt></a>. For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Suite" title="ecflow.Suite"><tt ><span >ecflow.Suite</span></tt></a>.</p> </dd> <dt id="term-suite-definition">suite definition</dt> <dd><p >The suite definition is the hierarchical <a href="#term-node"><em class="xref std std-term">node</em></a> tree.</p> <p>It describes how your <a href="#term-task"><em >task</em></a>‘s run and interact.</p> <p>It can built up using:</p> <blockquote> <div><ul> <li><p >Ascii text file by following the rules defined in the ecFlow <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammer"><em>Definition file Grammar</em></a>.</p> <p>Hence any language can be used, to generate this format.</p> </li> <li><p ><a href="/wiki/display/ECFLOW/ecFlow+Python+Api#suite-definition-python-api"><em>Suite Definition API</em></a></p> </li> </ul> </div></blockquote> <p >Once the definition is built, it can be loaded into the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>, and started. It can be monitored by <a href="#term-ecflowview"><em >ecflowview</em></a></p> </dd> <dt id="term-suspended">suspended</dt> <dd><p >Is a <a href="#term-node"><em >node</em></a> state. A <a href="#term-node"><em >node</em></a> can be placed into the suspended state via a <a href="#term-defstatus"><em >defstatus</em></a> or via <a href="#term-ecflowview"><em class="xref std std-term">ecflowview</em></a></p> <p >A suspended <a href="#term-node"><em >node</em></a> including any of its children can not take part in <a href="#term-scheduling"><em >scheduling</em></a> until the node is resumed.</p> </dd> <dt id="term-task">task</dt> <dd><p >A task represents a job that needs to be carried out. It serves as a leaf <a href="#term-node"><em >node</em></a> in a <a href="#term-suite-definition"><em >suite definition</em></a></p> <p>Only tasks can be submitted.</p> <p>A job inside a task <a href="#term-ecf-script"><em >ecf script</em></a> should generally be re-entrant so that no harm is done by rerunning it, since a task may be automatically submitted more than once if it aborts.</p> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Task" title="ecflow.Task"><tt ><span >ecflow.Task</span></tt></a>. For text BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-task"><tt ><span >task</span></tt></a></p> </dd> <dt id="term-time">time</dt> <dd><p >This defines a time dependency for a node.</p> <p>Time is expressed in the format [h]h:mm. Only numeric values are allowed. There can be multiple time dependencies for a node, but overlapping times may cause unexpected results. To define a series of times, specify the start time, end time and a time increment. If the start time begins with ‘+’, times are relative to the beginning of the suite or, in repeated families, relative to the beginning of the repeated family.</p> <p>If the time the job takes to complete is longer than the interval a “slot” is missed, e.g.</p> <div ><pre>time 10:00 20:00 01:00</pre> </div> <p>if the 10:00 run takes more than an hour, the 11:00 run will never occur.</p> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Time" title="ecflow.Time"><tt ><span >ecflow.Time</span></tt></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_time" title="ecflow.Node.add_time"><tt class="xref py py-class docutils literal"><span >ecflow.Node.add_time</span></tt></a>. For BNF see <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-time"><tt ><span >time</span></tt></a></p> </dd> <dt id="term-today">today</dt> <dd><p >Like <a href="#term-time"><em >time</em></a>, but If the suites begin time is <strong>past</strong> the time given for the “today” command the <a href="#term-node"><em class="xref std std-term">node</em></a> is free to run (as far as the time dependency is concerned).</p> <p>For example</p> <div ><pre>task x today 10:00</pre> </div> <p>If we begin or re-queue the <a href="#term-suite"><em >suite</em></a> at 9.00 am, then the <a href="#term-task"><em class="xref std std-term">task</em></a> in held until 10.00 am. However if we begin or re-queue the suite at 11.00am, the <a class="reference internal" href="#term-task"><em >task</em></a> is run immediately.</p> <p>No lets look at time</p> <div ><pre>task x time 10:00</pre> </div> <p>If we begin or re-queue the <a href="#term-suite"><em >suite</em></a> at 9.00am, then the <a href="#term-task"><em class="xref std std-term">task</em></a> in held until 10.00 am. If we begin or re-queue the <a href="#term-suite"><em >suite</em></a> at 11.00am, the <a href="#term-task"><em class="xref std std-term">task</em></a> is still held.</p> <p>If the time the job takes to complete is longer than the interval a “slot” is missed, e.g.</p> <div ><pre>today 10:00 20:00 01:00</pre> </div> <p>if the 10:00 run takes more than an hour, the 11:00 run will never occur.</p> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Today" title="ecflow.Today"><tt ><span >ecflow.Today</span></tt></a>. For text BNF see <a class="reference internal" href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-today"><tt ><span >today</span></tt></a></p> </dd> <dt id="term-trigger">trigger</dt> <dd><p >Triggers defines a dependency for a <a class="reference internal" href="#term-task"><em >task</em></a> or <a href="#term-family"><em class="xref std std-term">family</em></a>.</p> <p>There can be only one trigger dependency per <a href="#term-node"><em >node</em></a>, but that can be a complex boolean expression of the <a class="reference internal" href="#term-status"><em >status</em></a> of several nodes. Triggers should be avoided on suites. A node with a trigger can only be activated when its trigger has expired. A trigger holds the node as long as the trigger’s expression evaluation returns false.</p> <p>Trigger evaluation occurs when ever the <a href="#term-child-command"><em >child command</em></a> communicates with the server. i.e whenever there is a state change in the suite definition.</p> <p>The keywords in trigger expressions are: <a href="#term-unknown"><em >unknown</em></a>, <a href="#term-suspended"><em class="xref std std-term">suspended</em></a>, <a href="#term-complete"><em >complete</em></a>, <a href="#term-queued"><em class="xref std std-term">queued</em></a>, <a href="#term-submitted"><em >submitted</em></a>, <a href="#term-active"><em class="xref std std-term">active</em></a>, <a href="#term-aborted"><em >aborted</em></a> and <strong>clear</strong> and <strong>set</strong> for <a class="reference internal" href="#term-event"><em >event</em></a> status.</p> <p>Triggers can also reference Node attributes like <a class="reference internal" href="#term-event"><em >event</em></a>, <a href="#term-meter"><em class="xref std std-term">meter</em></a>, <a href="#term-variable"><em >variable</em></a>, <a href="#term-repeat"><em class="xref std std-term">repeat</em></a> and generated variables. Trigger evaluation for node attributes uses integer arithmetic:</p> <ul > <li><a href="#term-event"><em class="xref std std-term">event</em></a> has the integer value of 0(clear) and set(1)</li> <li><a href="#term-meter"><em class="xref std std-term">meter</em></a> values are integers hence they are used as is</li> <li><a href="#term-variable"><em class="xref std std-term">variable</em></a> value is converted to an integer, otherwise 0 is used. See example below</li> <li><a href="#term-repeat"><em class="xref std std-term">repeat</em></a> <em>string</em> : We use the index values as integers. See example below</li> <li><a href="#term-repeat"><em class="xref std std-term">repeat</em></a> <em>enumerated</em> : We use the index values as integers. See example below</li> <li><a href="#term-repeat"><em class="xref std std-term">repeat</em></a> <em>integer</em> : Use the implicit integer values</li> <li><a href="#term-repeat"><em class="xref std std-term">repeat</em></a> <em>date</em> : Use the date values as integers. Use of plus/minus on repeat date variable uses date arithmetic</li> </ul> <p>Here are some examples</p> <div ><pre>suite trigger_suite task a event EVENT meter METER 1 100 50 edit VAR_INT 12 edit VAR_STRING "captain scarlett" # This is not convertible to an integer, if referenced will use '0' family f1 edit SLEEP 2 repeat string NAME a b c d e f # This has values: a(0), b(1), c(3), d(4), e(5), f(6) i.e index family f2 repeat integer VALUE 5 10 # This has values: 5,6,7,8,9,10 family f3 repeat enumerated red green blue # red(0), green(1), blue(2) task t1 repeat date DATE 19991230 20000102 # This has values: 19991230,19991231,20000101,20000102 endfamily endfamily endfamily family f2 task event_meter trigger /suite/a:EVENT == set and /suite/a:METER >= 30 task variable trigger /suite/a:VAR_INT >= 12 and /suite/a:VAR_STRING == 0 task repeat_string trigger /suite/f1:NAME >= 4 task repeat_integer trigger /suite/f1/f2:VALUE >= 7 task repeat_date trigger /suite/f1/f2/f3/t1:DATE >= 19991231 task repeat_date2 # Using plus/minus on a repeat DATE will use date arithmetic # Since the starting value of DATE is 19991230, this task will run # straight away trigger /suite/f1/f2/f3/t1:DATE - 1 == 19991229 endfamily endsuite</pre> </div> <p>What happens when we have multiple node attributes of the same name, referenced in trigger expressions ?</p> <div ><pre>task foo event blah meter blah 0 200 50 edit blah 10 task bar trigger foo:blah >= 0</pre> </div> <p>In this case ecFlow will use the following precedence:</p> <ul > <li><a href="#term-event"><em class="xref std std-term">event</em></a></li> <li><a href="#term-meter"><em class="xref std std-term">meter</em></a></li> <li><a href="#term-variable"><em class="xref std std-term">variable</em></a></li> <li><a href="#term-repeat"><em class="xref std std-term">repeat</em></a></li> <li>generated variables</li> </ul> <p>Hence in the example above expression ‘foo:blah >= 0’ will reference the event.</p> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Expression" title="ecflow.Expression"><tt ><span >ecflow.Expression</span></tt></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_trigger" title="ecflow.Node.add_trigger"><tt class="xref py py-class docutils literal"><span >ecflow.Node.add_trigger</span></tt></a></p> </dd> <dt id="term-unknown">unknown</dt> <dd><p >Is a <a href="#term-node"><em >node</em></a> <a href="#term-status"><em >status</em></a>.</p> <p >This is the default <a href="#term-node"><em >node</em></a> <a href="#term-status"><em class="xref std std-term">status</em></a> when a <a href="#term-suite-definition"><em >suite definition</em></a> is loaded into the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></p> </dd> <dt id="term-user-commands">user commands</dt> <dd>User commands are any client to server requests that are <strong>not</strong> <a href="#term-child-command"><em >child command</em></a> s.</dd> <dt id="term-variable">variable</dt> <dd><p >ecFlow makes heavy use of different kinds of variables.There are several kinds of variables:</p> <blockquote> <div><p>Environment variables: which are set in the UNIX shell before the <a href="#term-ecflow"><em class="xref std std-term">ecFlow</em></a> starts. These control <a href="#term-ecflow-server"><em >ecflow_server</em></a>, and <a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> .</p> <p>suite definition variables: Also referred to as user variables. These control <a href="#term-ecflow-server"><em >ecflow_server</em></a>, and <a class="reference internal" href="#term-ecflow-client"><em class="xref std std-term">ecflow_client</em></a> and are available for use in <a href="#term-job-file"><em >job file</em></a>.</p> <p>Generated variables: These are generated within the <a class="reference internal" href="#term-suite-definition"><em >suite definition</em></a> node tree during <a href="#term-job-creation"><em >job creation</em></a> and are available for use in the <a href="#term-job-file"><em >job file</em></a>.</p> </div></blockquote> <p>Variables can be referenced in <a href="#term-trigger"><em >trigger</em></a> and <a href="#term-complete-expression"><em class="xref std std-term">complete expression</em></a> s . The value part of the variable should be convertible to an integer otherwise a default value of 0 is used.</p> <p >For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.Node.add_variable" title="ecflow.Node.add_variable"><tt class="xref py py-class docutils literal"><span >ecflow.Node.add_variable</span></tt></a>. For BNF see <a href="/wiki/display/ECFLOW/Definition+file+Grammar#grammar-token-variable"><tt ><span >variable</span></tt></a></p> </dd> <dt id="term-variable-inheritance">variable inheritance</dt> <dd><p >When a <a href="#term-variable"><em >variable</em></a> is needed at <a href="#term-job-creation"><em class="xref std std-term">job creation</em></a> time, it is first sought in the <a href="#term-task"><em class="xref std std-term">task</em></a> itself.</p> <p>If it is not found in the <a href="#term-task"><em >task</em></a>, it is sought from the task’s parent and so on, up through the <a href="#term-node"><em class="xref std std-term">node</em></a> levels until found.</p> <p>For any <a href="#term-node"><em class="xref std std-term">node</em></a>, there are two places to look for variables.</p> <p >Suite definition variables are looked for first, and then any generated variables.</p> </dd> <dt id="term-variable-substitution">variable substitution</dt> <dd><p >Takes place during <a href="#term-pre-processing"><em class="xref std std-term">pre-processing</em></a> <em>or</em> command invocation.(i.e ECF_JOB_CMD,ECF_KILL_CMD,etc)</p> <p>It involves searching each line of <a href="#term-ecf-script"><em >ecf script</em></a> file or command, for <a href="#term-ecf-micro"><em >ECF_MICRO</em></a> character. typically ‘%’</p> <p>The text between two % character, defines a variable. i.e %VAR%</p> <p>This variable is searched for in the <a href="#term-suite-definition"><em >suite definition</em></a>.</p> <p>First the suite definition variables( sometimes referred to as user variables) are searched and then Repeat variable name, and finally the generated variables.If no variable is found then the same search pattern is repeated up the node tree.</p> <p>The value of the <a href="#term-variable"><em >variable</em></a> is replaced between the % characters.</p> <p>If the micro character are not paired and an error message is written to the log file, and the task is placed into the <a href="#term-aborted"><em >aborted</em></a> state.</p> <p>If the variable is not found in the <a href="#term-suite-definition"><em >suite definition</em></a> during pre-processing then <a href="#term-job-creation"><em >job creation</em></a> fails, and an error message is written to the log file, and the task is placed into the <a href="#term-aborted"><em class="xref std std-term">aborted</em></a> state. To avoid this, variables in the <a href="#term-ecf-script"><em >ecf script</em></a> can be defined as:</p> <div ><pre>%VAR:replacement%</pre> </div> <p >This is similar to %VAR% but if VAR is not found in the <a href="#term-suite-definition"><em class="xref std std-term">suite definition</em></a> then ‘replacement’ is used.</p> </dd> <dt id="term-virtual-clock">virtual clock</dt> <dd><p >Like <a href="#term-real-clock"><em >real clock</em></a> until the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> is suspended (i.e <a class="reference internal" href="#term-shutdown"><em >shutdown</em></a> or <a href="#term-halted"><em class="xref std std-term">halted</em></a>), the suites <a href="#term-clock"><em class="xref std std-term">clock</em></a> is also suspended.</p> <p>Hence will honour relative times in <a href="#term-cron"><em >cron</em></a>, <a href="#term-today"><em class="xref std std-term">today</em></a> and <a href="#term-time"><em >time</em></a> dependencies. It is possible to have a combination of hybrid/real and virtual.</p> <p >More useful when we want complete adherence to time related dependencies at the expense being out of sync with system time.</p> </dd> <dt id="term-zombie">zombie</dt> <dd><p >Zombies are running jobs that fail authentication when communicating with the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></p> <p><a href="#term-child-command"><em class="xref std std-term">child command</em></a> s like (init, event,meter, label, abort,complete) are placed in the <a href="#term-ecf-script"><em >ecf script</em></a> file and are used to communicate with the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>.</p> <p>The <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> authenticates each connection attempt made by the <a href="#term-child-command"><em class="xref std std-term">child command</em></a>. Authentication can fail for a number of reasons:</p> <blockquote> <div><ul > <li>password(ECF_PASS) supplied with the <a href="#term-child-command"><em >child command</em></a>, does not match the one in the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></li> <li>path name(ECF_NAME) supplied with the <a href="#term-child-command"><em >child command</em></a>, does not locate a <a href="#term-task"><em >task</em></a> in the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></li> <li>process id(ECF_RID) supplied with <a href="#term-child-command"><em >child command</em></a>, does not correspond with the one stored in the <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></li> <li><a href="#term-task"><em class="xref std std-term">task</em></a> is already <a href="#term-active"><em >active</em></a>, but receives another init <a href="#term-child-command"><em >child command</em></a></li> <li><a href="#term-task"><em class="xref std std-term">task</em></a> is already <a href="#term-complete"><em >complete</em></a>, but receives another <a href="#term-child-command"><em >child command</em></a></li> </ul> </div></blockquote> <p>When authentication fails the job is considered to be a zombie. The <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a> will keep a note of the zombie for a period of time, before it is automatically removed. However the removed zombie, may well re-appear. ( this is because each <a href="#term-child-command"><em class="xref std std-term">child command</em></a> will continue attempting to contact the <a href="#term-ecflow-server"><em >ecflow_server</em></a> for 24 hours. This is configurable see ECF_TIMEOUT on <a href="#term-ecflow-client"><em >ecflow_client</em></a>)</p> <p>For python see <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.ZombieAttr" title="ecflow.ZombieAttr"><tt ><span >ecflow.ZombieAttr</span></tt></a>, <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.ZombieUserActionType" title="ecflow.ZombieUserActionType"><tt class="xref py py-class docutils literal"><span >ecflow.ZombieUserActionType</span></tt></a></p> <p >There are several types of zombies see <a class="reference internal" href="#term-zombie-type"><em >zombie type</em></a> and <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.ZombieType" title="ecflow.ZombieType"><tt ><span >ecflow.ZombieType</span></tt></a></p> </dd> <dt id="term-zombie-attribute">zombie attribute</dt> <dd><p >The zombie attribute defines how a <a class="reference internal" href="#term-zombie"><em >zombie</em></a> should be handled in an automated fashion. Very careful consideration should be taken before this attribute is added as it may hide a genuine problem. It can be added to any <a href="#term-node"><em >node</em></a>. But is best defined at the <a href="#term-suite"><em class="xref std std-term">suite</em></a> or <a href="#term-family"><em >family</em></a> level. If there is no zombie attribute the default behaviour is to block the <a href="#term-child-command"><em class="xref std std-term">child command</em></a>.</p> <p >To add a zombie attribute in python, please see: <a href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.ZombieAttr" title="ecflow.ZombieAttr"><tt ><span >ecflow.ZombieAttr</span></tt></a></p> </dd> <dt id="term-zombie-type">zombie type</dt> <dd><p >See <a href="#term-zombie"><em >zombie</em></a> and class <a class="reference internal" href="/wiki/display/ECFLOW/ecFlow+Python+Api#ecflow.ZombieAttr" title="ecflow.ZombieAttr"><tt ><span >ecflow.ZombieAttr</span></tt></a> for further information. There are several types of zombies:</p> <blockquote> <div><ul> <li><dl > <dt>path</dt> <dd><ul > <li>The task path can not be found in the server, because node tree was deleted, replaced,reload, server crashed or backup server does not have node tree.</li> <li>Jobs could have been created, via server <a href="#term-scheduling"><em >scheduling</em></a> or by <a href="#term-user-commands"><em class="xref std std-term">user commands</em></a></li> </ul> </dd> </dl> </li> <li><dl > <dt>user</dt> <dd><p >Job is created by <a href="#term-user-commands"><em >user commands</em></a> like, rerun, re-queue. User zombies are differentiated from server(scheduled) since they are automatically created when the force option is used and we have tasks in an <a href="#term-active"><em >active</em></a> or <a href="#term-submitted"><em class="xref std std-term">submitted</em></a> states.</p> </dd> </dl> </li> <li><dl > <dt>ecf</dt> <dd><p >Jobs are created as part of the normal <a class="reference internal" href="#term-scheduling"><em class="xref std std-term">scheduling</em></a></p> <ul > <li>Server crashed ( or terminated and restarted) and the recovered <a href="#term-check-point"><em class="xref std std-term">check point</em></a> file is out of date.</li> <li>A <a href="#term-task"><em class="xref std std-term">task</em></a> is repeatedly re-run, earlier copies will not be remembered.</li> <li>Job sent by another <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a>, but which can not talk to the original <a href="#term-ecflow-server"><em class="xref std std-term">ecflow_server</em></a></li> <li>Network glitches/network down</li> </ul> </dd> </dl> </li> </ul> </div></blockquote> <p >The type of the zombie is not fixed and may change.</p> </dd> </dl> </div> |