You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/SubWorkflows.md
+16-44Lines changed: 16 additions & 44 deletions
Original file line number
Diff line number
Diff line change
@@ -1,29 +1,11 @@
1
-
_For the Doc-A-Thon_
2
-
**Questions to answer and things to consider:**
1
+
In order to support the composition and reuse of workflows, WDL allows the execution of an entire workflow as a step in a larger workflow. When a workflow calls another workflow, that second workflow is referred to as a sub-workflow. Note that sub-workflows can themselves contain sub-workflows and so on and there is no explicit limit as to how deeply workflows can be nested. Cromwell supports execution of such workflows.
3
2
4
-
1. Who is visiting the Subworkflows page?
5
-
6
-
2. What do they need to know first?
7
-
8
-
3. Is all the important information there? If not, add it!
9
-
10
-
4. Are there things that don't need to be there? Remove them.
11
-
12
-
5. Are the code and instructions accurate? Try it!
13
-
14
-
---
15
-
**DELETE ABOVE ONCE COMPLETE**
16
-
17
-
---
18
-
19
-
20
-
WDL allows the execution of an entire workflow as a step in a larger workflow (see WDL SPEC for more details), which is what will be referred to as a sub workflow going forward.
21
-
Cromwell supports execution of such workflows. Note that sub workflows can themselves contain sub workflows, etc... There is no limitation as to how deeply workflows can be nested.
3
+
However, that a single WDL file can contain only a single workflow definition. In order to reference a sub-workflow, the import directive can be used to bring that sub-workflow into existence and referenced by it's alias and name. See the [documentation on imports](Imports.md) for more details of how to declare and reference tasks and workflows via imports.
22
4
23
5
**Execution**
24
6
25
-
Sub workflows are executed exactly as a task would be.
26
-
*This means that if another call depends on an output of a subworkflow, this call will run when the whole subworkflow completes (successfully).*
7
+
A sub-workflows is executed exactly as a task would be.
8
+
*This means that if another call depends on an output of a sub-workflow, this call will run when the whole sub-workflow completes (successfully).*
27
9
For example, in the following case :
28
10
29
11
`main.wdl`
@@ -49,9 +31,6 @@ task hello {
49
31
command {
50
32
echo "Hello ${addressee}!"
51
33
}
52
-
runtime {
53
-
docker: "ubuntu:latest"
54
-
}
55
34
output {
56
35
String salutation = read_string(stdout())
57
36
}
@@ -62,9 +41,6 @@ task goodbye {
62
41
command {
63
42
echo "Goodbye ${addressee}!"
64
43
}
65
-
runtime {
66
-
docker: "ubuntu:latest"
67
-
}
68
44
output {
69
45
String salutation = read_string(stdout())
70
46
}
@@ -83,27 +59,27 @@ workflow hello_and_goodbye {
83
59
}
84
60
```
85
61
86
-
`myTask` will start only when hello_and_goodbye completes (which means all of its calls are done), even though `myTask` only needs the output of hello in the hello_and_goodbye subworkflow.
62
+
`myTask` will start only when hello_and_goodbye completes (which means all of its calls are done), even though `myTask` only needs the output of hello in the hello_and_goodbye sub-workflow.
87
63
If hello_and_goodbye fails, then `myTask` won't be executed.
88
-
Only workflow outputs are visible outside a workflow, which means that references to outputs produced by a subworkflow will only be valid if those outputs are exposed in the workflow output section.
64
+
Only workflow outputs are visible outside a workflow, which means that references to outputs produced by a sub-workflow will only be valid if those outputs are exposed in the workflow output section.
89
65
90
-
Subworkflows are executed in the context of a main workflow, which means that operations that are normally executed once per workflow (set up, clean up, outputs copying, log copying, etc...)
91
-
will NOT be re-executed for each subworkflow. For instance if a resource is created during workflow initialization, subworkflows will need to share this same resource.
92
-
Workflow outputs will be copied for the main root workflow but not for intermediate subworkflows.
66
+
Sub-workflows are executed in the context of a main workflow, which means that operations that are normally executed once per workflow (set up, clean up, outputs copying, log copying, etc...)
67
+
will NOT be re-executed for each sub-workflow. For instance if a resource is created during workflow initialization, sub-workflows will need to share this same resource.
68
+
Workflow outputs will be copied for the main root workflow but not for intermediate sub-workflows.
93
69
94
70
Restarts, aborts, and call-caching work exactly as they would with tasks.
95
-
All tasks run by a subworkflow are eligible for call caching under the same rules as any other task.
71
+
All tasks run by a sub-workflow are eligible for call caching under the same rules as any other task.
96
72
However, workflows themselves are not cached as such. Which means that running the exact same workflow twice with call caching on will trigger each task to cache individually,
97
73
but not the workflow itself.
98
74
99
-
The root path for subworkflow execution files (scripts, output files, logs) will be under the parent workflow call directory.
75
+
The root path for sub-workflow execution files (scripts, output files, logs) will be under the parent workflow call directory.
100
76
For example, the execution directory for the above main workflow would look like the following:
101
77
102
78
```
103
79
cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/ <- main workflow id
104
80
└── call-hello_and_goodbye <- call directory for call hello_and_goodbye in the main workflow
105
-
└── hello_and_goodbye <- name of the subworkflow
106
-
└── a6365f91-c807-465a-9186-a5d3da98fe11 <- subworkflow id
81
+
└── hello_and_goodbye <- name of the sub-workflow
82
+
└── a6365f91-c807-465a-9186-a5d3da98fe11 <- sub-workflow id
107
83
├── call-goodbye
108
84
│ └── execution
109
85
│ ├── rc
@@ -129,7 +105,7 @@ cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/ <- main
129
105
130
106
**Metadata**
131
107
132
-
Each subworkflow will have its own workflow ID. This ID will appear in the metadata of the parent workflow, in the call section corresponding to the subworkflow, under the "subWorkflowId" attribute.
108
+
Each sub-workflow will have its own workflow ID. This ID will appear in the metadata of the parent workflow, in the call section corresponding to the sub-workflow, under the "subWorkflowId" attribute.
133
109
For example, querying the `main_workflow` metadata above (minus the `myTask` call) , could result in something like this:
0 commit comments