Currently, in ecflow ecFlow, we can have jobs that are identical but vary only in the step.
In these cases, each step carries a latency, i.e. submission cost,(i.e. starting hundreds of new processes)
The Queue attribute was created to address this situation.
...
...
...
...
...
...
...
...
...
...
...
...
|
| Code Block |
|---|
suite test
family |
|
|---|
...
...
...
001 002 003 004 005 006
task t
|
|
|---|
...
| Code Block |
|---|
suite test_queue
family f1
queue q1 001 002 queue003 q2004 1005 2006 3007
task t
endfamily
taskfamily bf2
queue q2 1 2 3 4 5 6 trigger /test_queue/f1:q1 > 18 9 10
task a
task b
# notice that queue name is accessible to the trigger
trigger /test_queue/f1:q1 > 5 task c
task c
trigger /test_queue../f2/a:q2 > 19
endfamily
endsuite |
There is a new child command --queue, that will will signal when a step is active, complete, or has aborted.
| Code Block |
|---|
| language | bash |
|---|
| title | New Child command, for use in .ecf scripts |
|---|
|
# Note: because --queue is treated like a child command(init,complete,event,label,meter,abort,wait), the task path ECF_NAME is read from the environment
# The --queue command will search up the node hierarchy for the queue name. If not found it fails.
step=$(ecflow_client --queue queue_name active) # returns first queued/aborted step from the server and makes it active, Return "NULL" for the last step.
ecflow_client --queue queue_name complete $step # Tell the server that step has completed for the given queue
ecflow_client --queue queue_name aborted $step # Tell the server that step has aborted for the given queue
no_of_aborted=$(ecflow_client --queue queue_name no_of_aborted) # returns as a string the number of aborted steps
ecflow_client --queue queue_name reset # sets the index to the first queued/aborted step. Allows steps to be reprocessed for errors |
Heres how it could be used in a script:
This attribute makes it possible to follow a producer(server)/consumer(tasks) pattern. Note additional task consumers can be added for load balancing.
The queue values can be strings, however, if they are to be used in trigger expressions, they must be convertible to integers.
| Code Block |
|---|
suite test_queue
family f1 |
| Code Block |
|---|
|
step=""
QNAME="my_queue_name"
while [1 == 1 ] ; do
# this return the first queued/aborted step, then increments to next step, return <NULL> when all steps processed
step=$(ecflow_client --queue=$QNAME active) # of the form string i.e \"003\". this step is now active
if [[ $step == "<NULL>" ]] ; then
break; # noqueue moreq1 steps
red orange yellow fi
green blue indigo ...violet
ecflow_client --queue=$QNAME $step complete # tell ecflow this step completed
donetask t
trap() { ecflow_client --queue=$QNAME $step aborted # tell ecflow this step failed } |
...
