From 96d6912d9182dcd0294d93cfd448b97d44505b6a Mon Sep 17 00:00:00 2001 From: Razvan-Daniel Mihai <84674+razvan@users.noreply.github.com> Date: Tue, 22 Apr 2025 13:32:17 +0200 Subject: [PATCH 1/2] doc: comment on spark history integration --- CHANGELOG.md | 2 ++ docs/modules/spark-k8s/pages/usage-guide/spark-connect.adoc | 5 ++++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c277b5a..b6c1ad78 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,7 @@ All notable changes to this project will be documented in this file. - Replace stackable-operator `print_startup_string` with `tracing::info!` with fields. - BREAKING: Inject the vector aggregator address into the vector config using the env var `VECTOR_AGGREGATOR_ADDRESS` instead of having the operator write it to the vector config ([#551]). +- Document that Spark Connect doesn't integrate with the history server ([#559]) ### Fixed @@ -28,6 +29,7 @@ All notable changes to this project will be documented in this file. [#551]: https://github.com/stackabletech/spark-k8s-operator/pull/551 [#553]: https://github.com/stackabletech/spark-k8s-operator/pull/553 [#554]: https://github.com/stackabletech/spark-k8s-operator/pull/554 +[#559]: https://github.com/stackabletech/spark-k8s-operator/pull/559 ## [25.3.0] - 2025-03-21 diff --git a/docs/modules/spark-k8s/pages/usage-guide/spark-connect.adoc b/docs/modules/spark-k8s/pages/usage-guide/spark-connect.adoc index d4a0307d..7d365429 100644 --- a/docs/modules/spark-k8s/pages/usage-guide/spark-connect.adoc +++ b/docs/modules/spark-k8s/pages/usage-guide/spark-connect.adoc @@ -45,11 +45,14 @@ spec: The example above adds a new endpoint for application metrics. +== Spark History Server + +Unforunately integration with the Spark History Server is not supported yet. The connect server seems to ignore the `spark.eventLog` properties while also prohibiting clients to set them programatically. + == Notable Omissions The following features are not supported by the Stackable Spark operator yet -* Integration with the Spark History Server. * Authorization and authentication. Currently, anyone with access to the Spark Connect service can run jobs. * Volumes and volume mounts can be added only with pod overrides. * Job dependencies must be provisioned as custom images or via `--packages` or `--jars` arguments. From 26bfdb4f91a4da569b43186d8223925c9da69b79 Mon Sep 17 00:00:00 2001 From: Razvan-Daniel Mihai <84674+razvan@users.noreply.github.com> Date: Tue, 22 Apr 2025 14:46:18 +0200 Subject: [PATCH 2/2] Update docs/modules/spark-k8s/pages/usage-guide/spark-connect.adoc Co-authored-by: Sebastian Bernauer --- docs/modules/spark-k8s/pages/usage-guide/spark-connect.adoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/modules/spark-k8s/pages/usage-guide/spark-connect.adoc b/docs/modules/spark-k8s/pages/usage-guide/spark-connect.adoc index 7d365429..bf9bf1d1 100644 --- a/docs/modules/spark-k8s/pages/usage-guide/spark-connect.adoc +++ b/docs/modules/spark-k8s/pages/usage-guide/spark-connect.adoc @@ -47,7 +47,8 @@ The example above adds a new endpoint for application metrics. == Spark History Server -Unforunately integration with the Spark History Server is not supported yet. The connect server seems to ignore the `spark.eventLog` properties while also prohibiting clients to set them programatically. +Unforunately integration with the Spark History Server is not supported yet. +The connect server seems to ignore the `spark.eventLog` properties while also prohibiting clients to set them programatically. == Notable Omissions