fix resource limit usage (#166)

# Description

- Restores setting # of executor instances
- Define spark conf settings as well as defining the pod template
  - set kuberenetes.driver/executor.* from CRD directly
  - use cpu limit to set driver/executor.cores
- Integration test to test/illustrate
  - just resource limits
  - just spark.conf settings
  - just defaults
  
Fixes #161.
Jenkins: https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/spark-k8s-operator-it-custom/41/

Author: adwk67

CHANGELOG.md

@@ -9,10 +9,12 @@ All notable changes to this project will be documented in this file.
 - Bumped image to `3.3.0-stackable0.2.0` in tests and docs ([#145])
 - BREAKING: use resource limit struct instead of passing spark configuration arguments ([#147])
 - Fixed resources test ([#151])
+- Fixed inconsistencies with resources usage ([#166])
 
 [#145]: https://github.com/stackabletech/spark-k8s-operator/pull/145
 [#147]: https://github.com/stackabletech/spark-k8s-operator/pull/147
 [#151]: https://github.com/stackabletech/spark-k8s-operator/pull/151
+[#166]: https://github.com/stackabletech/spark-k8s-operator/pull/166
 
 ## [0.5.0] - 2022-09-06
 

Cargo.lock

@@ -0,0 +1,32 @@
+---
+apiVersion: spark.stackable.tech/v1alpha1
+kind: SparkApplication
+metadata:
+  name: pyspark-streaming
+  namespace: default
+spec:
+  version: "1.0"
+  sparkImage: docker.stackable.tech/stackable/pyspark-k8s:3.3.0-stackable0.1.0
+  mode: cluster
+  mainApplicationFile: local:///stackable/spark/examples/src/main/python/streaming/hdfs_wordcount.py
+  args:
+    - "/tmp2"
+  sparkConf:
+    spark.kubernetes.submission.waitAppCompletion: "false"
+    spark.kubernetes.driver.pod.name: "pyspark-streaming-driver"
+    spark.kubernetes.executor.podNamePrefix: "pyspark-streaming"
+  driver:
+    resources:
+      cpu:
+        min: "1"
+        max: "2"
+      memory:
+        limit: "1Gi"
+  executor:
+    instances: 1
+    resources:
+      cpu:
+        min: "1700m"
+        max: "3"
+      memory:
+        limit: "2Gi"

deploy/crd/sparkapplication.crd.yaml

@@ -180,6 +180,11 @@ executor:
 WARNING: The default values are _most likely_ not sufficient to run a proper cluster in production. Please adapt according to your requirements.
 For more details regarding Kubernetes CPU limits see: https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/[Assign CPU Resources to Containers and Pods].
 
+Spark allocates a default amount of non-heap memory based on the type of job (JVM or non-JVM). This is taken into account when defining memory settings based exclusively on the resource limits, so that the "declared" value is the actual total value (i.e. including memory overhead). This may result in minor deviations from the stated resource value due to rounding differences.
+
+NOTE: It is possible to define Spark resources either directly by setting configuration properties listed under `sparkConf`, or by using resource limits. If both are used, then `sparkConf` properties take precedence. It is recommended for the sake of clarity to use *_either_* one *_or_* the other.
+
+
 == CRD argument coverage
 
 Below are listed the CRD fields that can be defined by the user:

deploy/helm/spark-k8s-operator/crds/crds.yaml

@@ -17,3 +17,6 @@ serde_json = "1.0"
 serde_yaml = "0.8"
 snafu = "0.7"
 strum = { version = "0.24", features = ["derive"] }
+
+[dev-dependencies]
+rstest = "0.15.0"

deploy/manifests/crds.yaml

@@ -22,3 +22,6 @@ pub const SECRET_ACCESS_KEY: &str = "secretAccessKey";
 pub const S3_SECRET_DIR_NAME: &str = "/stackable/secrets";
 
 pub const SPARK_UID: i64 = 1000;
+pub const MIN_MEMORY_OVERHEAD: u32 = 384;
+pub const JVM_OVERHEAD_FACTOR: f32 = 0.1;
+pub const NON_JVM_OVERHEAD_FACTOR: f32 = 0.4;