You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Superset gives the ability to execute SQL queries and build dashboards. Open the `airflow` endpoint `webserver-airflow`
61
-
in your browser (`http://172.18.0.2:31979` in this case).
61
+
in your browser (`http://172.19.0.5:30913` in this case).
62
62
63
63
image::airflow-scheduled-job/airflow_1.png[]
64
64
65
-
Log in with the username `admin` and password `adminadmin`. The overview screen shows the DAGs mounted during the demo
66
-
setup (`date_demo`).
65
+
Log in with the username `admin` and password `adminadmin`.
66
+
Click in 'Active DAGs' at the top and you will see an overview showing the DAGs mounted during the demo
67
+
setup (`date_demo` and `sparkapp_dag`).
67
68
68
69
image::airflow-scheduled-job/airflow_2.png[]
69
70
70
-
There are two things to notice here. Both DAGs have been enabled, as shown by the slider to the left of the DAG name
71
-
(DAGs are all `paused` initially and can be activated manually in the UI or via a REST call, as done in the setup for
72
-
this demo):
71
+
There are two things to notice here.
72
+
Both DAGs have been enabled, as shown by the slider on the far right of the screen for each DAG
73
+
(DAGs are all `paused` initially and can be activated manually in the UI or via a REST call, as done in the setup for this demo):
73
74
74
75
image::airflow-scheduled-job/airflow_3.png[]
75
76
76
-
Secondly, the `date_demo` job has been busy, with several runs already logged. The `sparkapp_dag` has only been run
77
-
once because they have been defined with different schedules.
77
+
Secondly, the `date_demo` job has been busy, with several runs already logged.
78
+
The `sparkapp_dag` has only been run once because they have been defined with different schedules.
78
79
79
80
image::airflow-scheduled-job/airflow_4.png[]
80
81
81
-
Clicking on the number under `Runs` will display the individual job runs:
82
+
Clicking on the DAG name and then on `Runs` will display the individual job runs:
82
83
83
84
image::airflow-scheduled-job/airflow_5.png[]
84
85
85
-
The `demo_date` job is running every minute. With Airflow, DAGs can be started manually or scheduled to run when certain
86
-
conditions are fulfilled- In this case, the DAG has been set up to run using a cron table, which is part of the DAG
87
-
definition.
86
+
The `demo_date` job is running every minute.
87
+
With Airflow, DAGs can be started manually or scheduled to run when certain conditions are fulfilled - in this case, the DAG has been set up to run using a cron table, which is part of the DAG definition.
88
88
89
89
=== `demo_date` DAG
90
90
91
-
Let's drill down a bit deeper into this DAG. Click on one of the job runs shown in the previous step to display the
92
-
details. The DAG is displayed as a graph (this job is so simple that it only has one step, called `run_every_minute`).
91
+
Let's drill down a bit deeper into this DAG.
92
+
At the top under the DAG name there is some scheduling information, which tells us that this job will run every minute continuously:
93
93
94
94
image::airflow-scheduled-job/airflow_6.png[]
95
95
96
-
In the top right-hand corner there is some scheduling information, which tells us that this job will run every minute
97
-
continuously:
96
+
Click on one of the job runs in the list to display the details for the task instances.
97
+
In the left-side pane the DAG is displayed either as a graph (this job is so simple that it only has one step, called `run_every_minute`), or as a "bar chart" showing each run.
98
98
99
99
image::airflow-scheduled-job/airflow_7.png[]
100
100
101
-
Click on the `run_every_minute` box in the centre of the page and then select `Logs`:
101
+
Click on the `run_every_minute` box in the centre of the page to select the logs:
102
102
103
103
[WARNING]
104
104
====
@@ -108,26 +108,27 @@ See the https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/exec
108
108
If you are interested in persisting the logs, take a look at the xref:logging.adoc[] demo.
109
109
====
110
110
111
-
image::airflow-scheduled-job/airflow_9.png[]
111
+
image::airflow-scheduled-job/airflow_8.png[]
112
112
113
-
To look at the actual DAG code click on `Code`. Here we can see the crontab information used to schedule the job as well
114
-
the `bash` command that provides the output:
113
+
To look at the actual DAG code click on `Code`.
114
+
Here we can see the crontab information used to schedule the job as well the `bash` command that provides the output:
115
115
116
-
image::airflow-scheduled-job/airflow_10.png[]
116
+
image::airflow-scheduled-job/airflow_9.png[]
117
117
118
118
=== `sparkapp_dag` DAG
119
119
120
-
Go back to DAG overview screen. The `sparkapp_dag` job has a scheduled entry of `None` and a last-execution time
121
-
(`2022-09-19, 07:36:55`). This allows a DAG to be executed exactly once, with neither schedule-based runs nor any
122
-
https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dag-run.html#backfill[backfill]. The DAG can
123
-
always be triggered manually again via REST or from within the Webserver UI.
120
+
Go back to DAG overview screen.
121
+
The `sparkapp_dag` job has a scheduled entry of `None` and a last-execution time.
122
+
This allows a DAG to be executed exactly once, with neither schedule-based runs nor any
0 commit comments