Skip to content

Commit 0a4923c

Browse files
siegfriedweberadwk67
siegfriedweber
and
adwk67
authored
feat: Make spark-env.sh configurable (#473)
* feat: Make spark-env.sh configurable * docs: Document "Configuration & Environment Overrides" * Update changelog * fix: Unit test * docs: Fix LanguageTool linter warning * test: Fix assertion in the overrides test * Improve wording Co-authored-by: Andrew Kenworthy <[email protected]> * Improve wording Co-authored-by: Andrew Kenworthy <[email protected]> * Improve wording Co-authored-by: Andrew Kenworthy <[email protected]> * Improve wording * docs: Describe the env property in the SparkApplication spec * Upgrade futures-util because version 0.3.30 was yanked --------- Co-authored-by: Andrew Kenworthy <[email protected]>
1 parent ef71d7c commit 0a4923c

15 files changed

+313
-58
lines changed

‎CHANGELOG.md

+5
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@ All notable changes to this project will be documented in this file.
44

55
## [Unreleased]
66

7+
### Added
8+
9+
- Make spark-env.sh configurable via `configOverrides` ([#473]).
10+
711
### Changed
812

913
- Reduce CRD size from `1.2MB` to `103KB` by accepting arbitrary YAML input instead of the underlying schema for the following fields ([#450]):
@@ -28,6 +32,7 @@ All notable changes to this project will be documented in this file.
2832
[#459]: https://github.com/stackabletech/spark-k8s-operator/pull/459
2933
[#460]: https://github.com/stackabletech/spark-k8s-operator/pull/460
3034
[#472]: https://github.com/stackabletech/spark-k8s-operator/pull/472
35+
[#473]: https://github.com/stackabletech/spark-k8s-operator/pull/473
3136

3237
## [24.7.0] - 2024-07-24
3338

‎Cargo.lock

+14-14
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/modules/spark-k8s/pages/usage-guide/configuration-environment-overrides.adoc

+150
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
= Configuration & Environment Overrides
2+
3+
The cluster definition also supports overriding configuration properties and environment variables, either per role or per role group, where the more specific override (role group) has precedence over the less specific one (role).
4+
5+
IMPORTANT: Overriding operator-set properties (such as the ports) can interfere with the operator and can lead to problems.
6+
7+
8+
== Configuration Properties
9+
10+
For a role or role group, at the same level of `config`, you can specify `configOverrides` for the following files:
11+
12+
* `spark-env.sh`
13+
* `security.properties`
14+
15+
NOTE: `spark-defaults.conf` is not required here, because the properties defined in {crd-docs}/spark.stackable.tech/sparkhistoryserver/v1alpha1/#spec-sparkConf[`sparkConf` (SparkHistoryServer)] and {crd-docs}/spark.stackable.tech/sparkapplication/v1alpha1/#spec-sparkConf[`sparkConf` (SparkApplication)] are already added to this file.
16+
17+
For example, if you want to set the `networkaddress.cache.ttl`, it can be configured in the SparkHistoryServer resource like so:
18+
19+
[source,yaml]
20+
----
21+
nodes:
22+
roleGroups:
23+
default:
24+
configOverrides:
25+
security.properties:
26+
networkaddress.cache.ttl: "30"
27+
replicas: 1
28+
----
29+
30+
Just as for the `config`, it is possible to specify this at the role level as well:
31+
32+
[source,yaml]
33+
----
34+
nodes:
35+
configOverrides:
36+
security.properties:
37+
networkaddress.cache.ttl: "30"
38+
roleGroups:
39+
default:
40+
replicas: 1
41+
----
42+
43+
All override property values must be strings.
44+
45+
The same applies to the `job`, `driver` and `executor` roles of the SparkApplication.
46+
47+
=== The spark-env.sh file
48+
49+
The `spark-env.sh` file is used to set environment variables.
50+
Usually, environment variables are configured in `envOverrides` or {crd-docs}/spark.stackable.tech/sparkapplication/v1alpha1/#spec-env[`env` (SparkApplication)], but both options only allow static values to be set.
51+
The values in `spark-env.sh` are evaluated by the shell.
52+
For instance, if a SAS token is stored in a Secret and should be used for the Spark History Server, this token could be first stored in an environment variable via `podOverrides` and then added to the `SPARK_HISTORY_OPTS`:
53+
54+
[source,yaml]
55+
----
56+
podOverrides:
57+
spec:
58+
containers:
59+
- name: spark-history
60+
env:
61+
- name: SAS_TOKEN
62+
valueFrom:
63+
secretKeyRef:
64+
name: adls-spark-credentials
65+
key: sas-token
66+
configOverrides:
67+
spark-env.sh:
68+
SPARK_HISTORY_OPTS: >-
69+
$SPARK_HISTORY_OPTS
70+
-Dspark.hadoop.fs.azure.sas.fixed.token.mystorageaccount.dfs.core.windows.net=$SAS_TOKEN
71+
----
72+
73+
NOTE: The given properties are written to `spark-env.sh` in the form `export KEY="VALUE"`.
74+
Make sure to escape the value already in the specification.
75+
Be aware that some environment variables may already be set, so prepend or append a reference to them in the value, as it is done in the example.
76+
77+
=== The security.properties file
78+
79+
The `security.properties` file is used to configure JVM security properties.
80+
It is very seldom that users need to tweak any of these, but there is one use-case that stands out, and that users need to be aware of: the JVM DNS cache.
81+
82+
The JVM manages its own cache of successfully resolved host names as well as a cache of host names that cannot be resolved.
83+
Some products of the Stackable platform are very sensitive to the contents of these caches and their performance is heavily affected by them.
84+
As of version 3.4.0, Apache Spark may perform poorly if the positive cache is disabled.
85+
To cache resolved host names, and thus speed up queries, you can configure the TTL of entries in the positive cache like this:
86+
87+
[source,yaml]
88+
----
89+
spec:
90+
nodes:
91+
configOverrides:
92+
security.properties:
93+
networkaddress.cache.ttl: "30"
94+
networkaddress.cache.negative.ttl: "0"
95+
----
96+
97+
NOTE: The operator configures DNS caching by default as shown in the example above.
98+
99+
For details on the JVM security see https://docs.oracle.com/en/java/javase/11/security/java-security-overview1.html
100+
101+
102+
== Environment Variables
103+
104+
Similarly, environment variables can be (over)written. For example per role group:
105+
106+
[source,yaml]
107+
----
108+
nodes:
109+
roleGroups:
110+
default:
111+
envOverrides:
112+
MY_ENV_VAR: "MY_VALUE"
113+
replicas: 1
114+
----
115+
116+
or per role:
117+
118+
[source,yaml]
119+
----
120+
nodes:
121+
envOverrides:
122+
MY_ENV_VAR: "MY_VALUE"
123+
roleGroups:
124+
default:
125+
replicas: 1
126+
----
127+
128+
In a SparkApplication, environment variables can also be defined with the {crd-docs}/spark.stackable.tech/sparkapplication/v1alpha1/#spec-env[`env`] property for the job, driver and executor pods at once.
129+
The result is basically the same as with `envOverrides`, but `env` also allows to reference Secrets and so on:
130+
131+
[source,yaml]
132+
----
133+
---
134+
apiVersion: spark.stackable.tech/v1alpha1
135+
kind: SparkApplication
136+
spec:
137+
env:
138+
- name: SAS_TOKEN
139+
valueFrom:
140+
secretKeyRef:
141+
name: adls-spark-credentials
142+
key: sas-token
143+
...
144+
----
145+
146+
147+
== Pod overrides
148+
149+
The Spark operator also supports Pod overrides, allowing you to override any property that you can set on a Kubernetes Pod.
150+
Read the xref:concepts:overrides.adoc#pod-overrides[Pod overrides documentation] to learn more about this feature.

‎docs/modules/spark-k8s/pages/usage-guide/history-server.adoc

-29
Original file line numberDiff line numberDiff line change
@@ -74,32 +74,3 @@ spark-history-node-cleaner NodePort 10.96.203.43 <none> 18080:325
7474
By setting up port forwarding on 18080 the UI can be opened by pointing your browser to `http://localhost:18080`:
7575

7676
image::history-server-ui.png[History Server Console]
77-
78-
== Configuration Properties
79-
80-
For a role group of the Spark history server, you can specify: `configOverrides` for the following files:
81-
82-
* `security.properties`
83-
84-
=== The security.properties file
85-
86-
The `security.properties` file is used to configure JVM security properties.
87-
It is very seldom that users need to tweak any of these, but there is one use-case that stands out, and that users need to be aware of: the JVM DNS cache.
88-
89-
The JVM manages its own cache of successfully resolved host names as well as a cache of host names that cannot be resolved.
90-
Some products of the Stackable platform are very sensible to the contents of these caches and their performance is heavily affected by them.
91-
As of version 3.4.0, Apache Spark may perform poorly if the positive cache is disabled.
92-
To cache resolved host names, and thus speeding up queries you can configure the TTL of entries in the positive cache like this:
93-
94-
[source,yaml]
95-
----
96-
nodes:
97-
configOverrides:
98-
security.properties:
99-
networkaddress.cache.ttl: "30"
100-
networkaddress.cache.negative.ttl: "0"
101-
----
102-
103-
NOTE: The operator configures DNS caching by default as shown in the example above.
104-
105-
For details on the JVM security see https://docs.oracle.com/en/java/javase/11/security/java-security-overview1.html

‎docs/modules/spark-k8s/partials/nav.adoc

+1
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
** xref:spark-k8s:usage-guide/logging.adoc[]
1111
** xref:spark-k8s:usage-guide/history-server.adoc[]
1212
** xref:spark-k8s:usage-guide/examples.adoc[]
13+
** xref:spark-k8s:usage-guide/configuration-environment-overrides.adoc[]
1314
** xref:spark-k8s:usage-guide/operations/index.adoc[]
1415
*** xref:spark-k8s:usage-guide/operations/applications.adoc[]
1516
*** xref:spark-k8s:usage-guide/operations/pod-placement.adoc[]

‎rust/crd/src/constants.rs

+1
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ pub const HISTORY_ROLE_NAME: &str = "node";
7474
pub const SPARK_IMAGE_BASE_NAME: &str = "spark-k8s";
7575

7676
pub const SPARK_DEFAULTS_FILE_NAME: &str = "spark-defaults.conf";
77+
pub const SPARK_ENV_SH_FILE_NAME: &str = "spark-env.sh";
7778

7879
pub const SPARK_CLUSTER_ROLE: &str = "spark-k8s-clusterrole";
7980
pub const SPARK_UID: i64 = 1000;

‎rust/crd/src/history.rs

+1
Original file line numberDiff line numberDiff line change
@@ -212,6 +212,7 @@ impl SparkHistoryServer {
212212
(
213213
vec![
214214
PropertyNameKind::File(SPARK_DEFAULTS_FILE_NAME.to_string()),
215+
PropertyNameKind::File(SPARK_ENV_SH_FILE_NAME.to_string()),
215216
PropertyNameKind::File(JVM_SECURITY_PROPERTIES_FILE.to_string()),
216217
],
217218
self.spec.nodes.clone(),

‎rust/crd/src/lib.rs

+21
Original file line numberDiff line numberDiff line change
@@ -819,6 +819,7 @@ impl SparkApplication {
819819
(
820820
vec![
821821
PropertyNameKind::Env,
822+
PropertyNameKind::File(SPARK_ENV_SH_FILE_NAME.to_string()),
822823
PropertyNameKind::File(JVM_SECURITY_PROPERTIES_FILE.to_string()),
823824
],
824825
Role {
@@ -841,6 +842,7 @@ impl SparkApplication {
841842
(
842843
vec![
843844
PropertyNameKind::Env,
845+
PropertyNameKind::File(SPARK_ENV_SH_FILE_NAME.to_string()),
844846
PropertyNameKind::File(JVM_SECURITY_PROPERTIES_FILE.to_string()),
845847
],
846848
Role {
@@ -863,6 +865,7 @@ impl SparkApplication {
863865
(
864866
vec![
865867
PropertyNameKind::Env,
868+
PropertyNameKind::File(SPARK_ENV_SH_FILE_NAME.to_string()),
866869
PropertyNameKind::File(JVM_SECURITY_PROPERTIES_FILE.to_string()),
867870
],
868871
Role {
@@ -1037,6 +1040,20 @@ fn resources_to_executor_props(
10371040
Ok(())
10381041
}
10391042

1043+
/// Create the content of the file spark-env.sh.
1044+
/// The properties are serialized in the form 'export {k}="{v}"',
1045+
/// escaping neither the key nor the value. The user is responsible for
1046+
/// providing escaped values.
1047+
pub fn to_spark_env_sh_string<'a, T>(properties: T) -> String
1048+
where
1049+
T: Iterator<Item = (&'a String, &'a String)>,
1050+
{
1051+
properties
1052+
.map(|(k, v)| format!("export {k}=\"{v}\""))
1053+
.collect::<Vec<String>>()
1054+
.join("\n")
1055+
}
1056+
10401057
#[cfg(test)]
10411058
mod tests {
10421059

@@ -1296,6 +1313,10 @@ mod tests {
12961313
"default".into(),
12971314
vec![
12981315
(PropertyNameKind::Env, BTreeMap::new()),
1316+
(
1317+
PropertyNameKind::File("spark-env.sh".into()),
1318+
BTreeMap::new(),
1319+
),
12991320
(
13001321
PropertyNameKind::File("security.properties".into()),
13011322
vec![

‎rust/operator-binary/src/history/history_controller.rs

+12-2
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ use stackable_operator::{
3636
role_utils::RoleGroupRef,
3737
time::Duration,
3838
};
39-
use stackable_spark_k8s_crd::constants::METRICS_PORT;
39+
use stackable_spark_k8s_crd::constants::{METRICS_PORT, SPARK_ENV_SH_FILE_NAME};
4040
use stackable_spark_k8s_crd::{
4141
constants::{
4242
ACCESS_KEY_ID, APP_NAME, HISTORY_CONTROLLER_NAME, HISTORY_ROLE_NAME,
@@ -49,7 +49,7 @@ use stackable_spark_k8s_crd::{
4949
history,
5050
history::{HistoryConfig, SparkHistoryServer, SparkHistoryServerContainer},
5151
s3logdir::S3LogDir,
52-
tlscerts,
52+
tlscerts, to_spark_env_sh_string,
5353
};
5454
use std::collections::HashMap;
5555
use std::{collections::BTreeMap, sync::Arc};
@@ -382,6 +382,16 @@ fn build_config_map(
382382
.build(),
383383
)
384384
.add_data(SPARK_DEFAULTS_FILE_NAME, spark_defaults)
385+
.add_data(
386+
SPARK_ENV_SH_FILE_NAME,
387+
to_spark_env_sh_string(
388+
config
389+
.get(&PropertyNameKind::File(SPARK_ENV_SH_FILE_NAME.to_string()))
390+
.cloned()
391+
.unwrap_or_default()
392+
.iter(),
393+
),
394+
)
385395
.add_data(
386396
JVM_SECURITY_PROPERTIES_FILE,
387397
to_java_properties_string(jvm_sec_props.iter()).with_context(|_| {

0 commit comments

Comments
 (0)