On this page you will install the Stackable Spark-on-Kubernetes Operator as well as the commons and secret Operators which are required by all Stackable Operators.
The Spark-k8s Operator does not install a cluster or a set of nodes, but rather a SparkApplication that creates a spark-submit job, delegating this to the Spark driver, which in turn spins up executors for the duration of the job. Spark job dependencies can thus vary in their scope and detail. In this guide we are going to keep things simple and look at executing a Spark job that has a minimum of dependencies.
More information about the different ways to define Spark jobs and their dependencies is given on the following pages:
-
ROOT:usage.adoc
-
ROOT:job_dependencies.adoc
There are 2 ways to run Stackable Operators
-
Using stackablectl::index.adoc
-
Using Helm
stackablectl is the command line tool to interact with Stackable operators and our recommended way to install Operators. Follow the installation steps for your platform.
After you have installed stackablectl run the following command to install the Spark-k8s Operator:
link:example$code/getting_started.sh[role=include]The tool will show
[INFO ] Installing commons operator [INFO ] Installing secret operator [INFO ] Installing spark-k8s operator
|
Tip
|
Consult the stackablectl::quickstart.adoc to learn more about how to use stackablectl. For example, you can use the -k flag to create a Kubernetes cluster with kind.
|
You can also use Helm to install the Operators. Add the Stackable Helm repository:
link:example$code/getting_started.sh[role=include]Then install the Stackable Operators:
link:example$code/getting_started.sh[role=include]Helm will deploy the Operators in a Kubernetes Deployment and apply the CRDs for the SparkApplication (as well as the CRDs for the required operators). You are now ready to create a Spark job in Kubernetes.
Execute a Spark Job and verify that it works by inspecting the pod logs.