Skip to content

Commit 4f56431

Browse files
adwk67adwk67
committed
initial commit: wip
1 parent 8d58660 commit 4f56431

File tree

12 files changed

+347
-79
lines changed

12 files changed

+347
-79
lines changed

‎docs/antora.yml

+1
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,6 @@ name: spark-k8s
33
version: "nightly"
44
title: Stackable Operator for Apache Spark on Kubernetes
55
nav:
6+
- modules/getting_started/nav.adoc
67
- modules/ROOT/nav.adoc
78
prerelease: true

‎docs/modules/ROOT/nav.adoc

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
* xref:installation.adoc[]
1+
* xref:configuration.adoc[]
22
* xref:usage.adoc[]
33
* xref:job_dependencies.adoc[]
44
* xref:rbac.adoc[]

‎docs/modules/ROOT/pages/installation.adoc

-51
This file was deleted.

‎docs/modules/ROOT/pages/usage.adoc

-27
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,5 @@
11
= Usage
22

3-
== Create an Apache Spark job
4-
5-
If you followed the installation instructions, you should now have a Stackable Operator for Apache Spark up and running, and you are ready to create your first Apache Spark kubernetes cluster.
6-
7-
The example below creates a job running on Apache Spark 3.3.0, using the spark-on-kubernetes paradigm described in the spark documentation. The application file is itself part of the spark distribution and `local` refers to the path on the driver/executors; there are no external dependencies.
8-
9-
cat <<EOF | kubectl apply -f -
10-
apiVersion: spark.stackable.tech/v1alpha1
11-
kind: SparkApplication
12-
metadata:
13-
name: spark-clustermode-001
14-
spec:
15-
version: 1.0
16-
mode: cluster
17-
mainClass: org.apache.spark.examples.SparkPi
18-
mainApplicationFile: local:///stackable/spark/examples/jars/spark-examples_2.12-3.3.0.jar
19-
image: 3.3.0-stackable0.1.0
20-
driver:
21-
cores: 1
22-
coreLimit: "1200m"
23-
memory: "512m"
24-
executor:
25-
cores: 1
26-
instances: 3
27-
memory: "512m"
28-
EOF
29-
303
== Examples
314

325
The following examples have the following `spec` fields in common:

‎docs/modules/getting_started/examples/code/getting_started.sh

+81
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
# This script contains all the code snippets from the guide, as well as some assert tests
5+
# to test if the instructions in the guide work. The user *could* use it, but it is intended
6+
# for testing only.
7+
# The script will install the operators, create a superset instance and briefly open a port
8+
# forward and connect to the superset instance to make sure it is up and running.
9+
# No running processes are left behind (i.e. the port-forwarding is closed at the end)
10+
11+
if [ $# -eq 0 ]
12+
then
13+
echo "Installation method argument ('helm' or 'stackablectl') required."
14+
exit 1
15+
fi
16+
17+
case "$1" in
18+
"helm")
19+
echo "Adding 'stackable-dev' Helm Chart repository"
20+
# tag::helm-add-repo[]
21+
helm repo add stackable-dev https://repo.stackable.tech/repository/helm-dev/
22+
# end::helm-add-repo[]
23+
echo "Installing Operators with Helm"
24+
# tag::helm-install-operators[]
25+
helm install --wait commons-operator stackable-dev/commons-operator --version 0.3.0-nightly
26+
helm install --wait secret-operator stackable-dev/secret-operator --version 0.6.0-nightly
27+
helm install --wait spark-k8s-operator stackable-dev/spark-k8s-operator --version 0.5.0-nightly
28+
# end::helm-install-operators[]
29+
;;
30+
"stackablectl")
31+
echo "installing Operators with stackablectl"
32+
# tag::stackablectl-install-operators[]
33+
stackablectl operator install \
34+
commons=0.3.0-nightly \
35+
secret=0.6.0-nightly \
36+
spark-k8s=0.5.0-nightly
37+
# end::stackablectl-install-operators[]
38+
;;
39+
*)
40+
echo "Need to give 'helm' or 'stackablectl' as an argument for which installation method to use!"
41+
exit 1
42+
;;
43+
esac
44+
45+
echo "Creating a Spark Application"
46+
# tag::install-sparkapp[]
47+
kubectl apply -f pyspark-pi.yaml
48+
# end::install-sparkapp[]
49+
50+
echo "Waiting on AirflowDB ..."
51+
# tag::wait-airflowdb[]
52+
kubectl wait airflowdb/airflow \
53+
--for jsonpath='{.status.condition}'=Ready \
54+
--timeout 300s
55+
# end::wait-airflowdb[]
56+
57+
sleep 5
58+
59+
echo "Awaiting Airflow rollout finish"
60+
# tag::watch-airflow-rollout[]
61+
kubectl rollout status --watch statefulset/airflow-webserver-default
62+
kubectl rollout status --watch statefulset/airflow-worker-default
63+
kubectl rollout status --watch statefulset/airflow-scheduler-default
64+
# end::watch-airflow-rollout[]
65+
66+
echo "Starting port-forwarding of port 8080"
67+
# tag::port-forwarding[]
68+
kubectl port-forward svc/airflow-webserver 8080 2>&1 >/dev/null &
69+
# end::port-forwarding[]
70+
PORT_FORWARD_PID=$!
71+
trap "kill $PORT_FORWARD_PID" EXIT
72+
sleep 5
73+
74+
echo "Checking if web interface is reachable ..."
75+
return_code=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/login/)
76+
if [ "$return_code" == 200 ]; then
77+
echo "Webserver UI reachable!"
78+
else
79+
echo "Could not reach Webserver UI."
80+
exit 1
81+
fi

‎docs/modules/getting_started/examples/code/pyspark-pi.yaml

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
apiVersion: spark.stackable.tech/v1alpha1
3+
kind: SparkApplication
4+
metadata:
5+
name: pyspark-pi
6+
namespace: default
7+
spec:
8+
version: "1.0"
9+
sparkImage: docker.stackable.tech/stackable/pyspark-k8s:3.3.0-stackable0.1.0
10+
mode: cluster
11+
mainApplicationFile: local:///stackable/spark/examples/src/main/python/pi.py
12+
driver:
13+
cores: 1
14+
coreLimit: "1200m"
15+
memory: "512m"
16+
executor:
17+
cores: 1
18+
instances: 3
19+
memory: "512m"

‎docs/modules/getting_started/nav.adoc

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
** xref:index.adoc[]
2+
** xref:installation.adoc[]
3+
** xref:first_steps.adoc[]

‎docs/modules/getting_started/pages/first_steps.adoc

+114
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
= First steps
2+
3+
Once you have followed the steps in the xref:installation.adoc[] section to install the Operator and its dependencies, you will now create a Spark job. Afterwards you can <<_verify_that_it_works, verify that it works>> by looking at the logs from the driver pod.
4+
5+
=== Airflow
6+
7+
An Airflow cluster is made of up three components:
8+
9+
- `webserver`: this provides the main UI for user-interaction
10+
- `workers`: the nodes over which the job workload will be distributed by the scheduler
11+
- `scheduler`: responsible for triggering jobs and persisting their metadata to the backend database
12+
13+
Create a file named `pyspark-pi.yaml` with the following contents:
14+
15+
[source,yaml]
16+
----
17+
include::example$code/pyspark-pi.yaml[]
18+
----
19+
20+
And apply it:
21+
22+
----
23+
include::example$code/getting_started.sh[tag=install-sparkapp]
24+
----
25+
26+
Where:
27+
28+
- `metadata.name` contains the name of the SparkApplication
29+
- `spec.version`: the current version is "1.0"
30+
- `spec.sparkImage`: the docker image that will be used by job, driver and executor pods. This can be provided by the user.
31+
- `spec.mode`: only `cluster` is currently supported
32+
- `spec.mainApplicationFile`: the artifact (Java, Scala or Python) that forms the basis of the Spark job.
33+
- `spec.driver`: driver-specific settings.
34+
- `spec.executor`: executor-specific settings.
35+
36+
37+
NOTE: If using Stackable image versions, please note that the version you need to specify for `spec.version` is not only the version of Spark which you want to roll out, but has to be amended with a Stackable version as shown. This Stackable version is the version of the underlying container image which is used to execute the processes. For a list of available versions please check our
38+
https://repo.stackable.tech/#browse/browse:docker:v2%2Fstackable%spark-k8s%2Ftags[image registry].
39+
It should generally be safe to simply use the latest image version that is available.
40+
41+
This will create the SparkApplication that in turn creates the Spark job.
42+
43+
=== Initialization of the Airflow database
44+
45+
When creating an Airflow cluster, a database-initialization job is first started to ensure that the database schema is present and correct (i.e. populated with an admin user). A Kubernetes job is created which starts a pod to initialize the database. This can take a while.
46+
47+
You can use kubectl to wait on the resource, although the cluster itself will not be created until this step is complete.:
48+
49+
[source,bash]
50+
include::example$code/getting_started.sh[tag=wait-airflowdb]
51+
52+
The job status can be inspected and verified like this:
53+
54+
[source,bash]
55+
----
56+
kubectl get jobs
57+
----
58+
59+
which will show something like this:
60+
61+
----
62+
NAME COMPLETIONS DURATION AGE
63+
airflow 1/1 85s 11m
64+
----
65+
66+
Then, make sure that all the Pods in the StatefulSets are ready:
67+
68+
[source,bash]
69+
----
70+
kubectl get statefulset
71+
----
72+
73+
The output should show all pods ready, including the external dependencies:
74+
75+
----
76+
NAME READY AGE
77+
airflow-postgresql 1/1 16m
78+
airflow-redis-master 1/1 16m
79+
airflow-redis-replicas 1/1 16m
80+
airflow-scheduler-default 1/1 11m
81+
airflow-webserver-default 1/1 11m
82+
airflow-worker-default 2/2 11m
83+
----
84+
85+
The completed set of pods for the Airflow cluster will look something like this:
86+
87+
image::airflow_pods.png[Airflow pods]
88+
89+
When the Airflow cluster has been created and the database is initialized, Airflow can be opened in the
90+
browser: the webserver UI port defaults to `8080` can be forwarded to the local host:
91+
92+
----
93+
include::example$code/getting_started.sh[tag=port-forwarding]
94+
----
95+
96+
== Verify that it works
97+
98+
The Webserver UI can now be opened in the browser with `http://localhost:8080`. Enter the admin credentials from the Kubernetes secret:
99+
100+
image::airflow_login.png[Airflow login screen]
101+
102+
Since the examples were loaded in the cluster definition, they will appear under the DAGs tabs:
103+
104+
image::airflow_dags.png[Example Airflow DAGs]
105+
106+
Select one of these DAGs by clicking on the name in the left-hand column e.g. `example_complex`. Click on the arrow in the top right of the screen, select "Trigger DAG" and the DAG nodes will be automatically highlighted as the job works through its phases.
107+
108+
image::airflow_running.png[Airflow DAG in action]
109+
110+
Great! You have set up an Airflow cluster, connected to it and run your first DAG!
111+
112+
== What's next
113+
114+
Look at the xref:ROOT:usage.adoc[Usage page] to find out more about configuring your Airflow cluster and loading your own DAG files.

‎docs/modules/getting_started/pages/index.adoc

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
= Getting started
2+
3+
This guide will get you started with Airflow using the Stackable Operator. It will guide you through the installation of the Operator and its dependencies, setting up your first Airflow cluster and connecting to it, and viewing and running one of the example workflows (called DAGs = Direct Acyclic Graphs).
4+
5+
== Prerequisites
6+
7+
You will need:
8+
9+
* a Kubernetes cluster
10+
* kubectl
11+
* Helm
12+
13+
== What's next
14+
15+
The Guide is divided into two steps:
16+
17+
* xref:installation.adoc[Installing the Operators].
18+
* xref:first_steps.adoc[Setting up the Airflow cluster and running an example DAG].

0 commit comments

Comments
 (0)