Skip to content

Commit 45e0cdc

Browse files
committed
Add note regarding job dependencies. (#250)
Part of #141 Reminder: cherry-pick to `release-23.4` after `main` merge.
1 parent bf2801f commit 45e0cdc

File tree

1 file changed

+4
-0
lines changed

1 file changed

+4
-0
lines changed

docs/modules/spark-k8s/pages/usage-guide/job-dependencies.adoc

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@
33

44
== Overview
55

6+
IMPORTANT: With the platform release 23.4.1 (and all previous releases), dynamic provisioning of dependencies using the Spark `packages` field doesn't work. This is a known problem with Spark and is tracked https://github.com/stackabletech/spark-k8s-operator/issues/141[here].
7+
68
The Stackable Spark-on-Kubernetes operator enables users to run Apache Spark workloads in a Kubernetes cluster easily by eliminating the requirement of having a local Spark installation. For this purpose, Stackble provides ready made Docker images with recent versions of Apache Spark and Python - for PySpark jobs - that provide the basis for running those workloads. Users of the Stackable Spark-on-Kubernetes operator can run their workloads on any recent Kubernetes cluster by applying a `SparkApplication` custom resource in which the job code, job dependencies, input and output data locations can be specified. The Stackable operator translates the user's `SparkApplication` manifest into a Kubernetes `Job` object and handles control to the Apache Spark scheduler for Kubernetes to construct the necessary driver and executor `Pods`.
79

810
image::spark-k8s.png[Job Flow]
@@ -88,6 +90,8 @@ include::example$example-pvc.yaml[]
8890

8991
=== Spark native package coordinates and Python requirements
9092

93+
IMPORTANT: With the platform release 23.4.1 (and all previous releases), dynamic provisioning of dependencies using the Spark `packages` field doesn't work. This is a known problem with Spark and is tracked https://github.com/stackabletech/spark-k8s-operator/issues/141[here].
94+
9195
The last and most flexible way to provision dependencies is to use the built-in `spark-submit` support for Maven package coordinates.
9296

9397
These can be specified by adding the following section to the `SparkApplication` manifest file:

0 commit comments

Comments
 (0)