k8s Role setup + Helm

[fhennig]

We want an s3 section in the CRD, for now we can use the same structure as in druid:

s3:
  endpoint: s3-de-central.profitbricks.com
  credentialsSecret: s3-credentials

for the endpoint there is a configoption, the access key and secret key should be mounted from the referenced secret. The secret structure used in druid is:

apiVersion: v1
kind: Secret
metadata:
  name: s3-credentials
stringData:
  accessKeyId: YOURACCESSKEYIDHERE
  secretAccessKey: YOURSECRETACCESSKEYHERE

The secret should be mounted and the env vars for that are AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY. The secret also needs to be mounted in the executors, so they can read from S3 too; so the pod template needs to be adjusted accordingly.

---

Update:

We don't know in which namespaces the SparkApplications will be created, so we will need to create the ServiceAccount and RoleBinding on demand in the namespace of the SparkApplication. Our Kafka Operator is already creating service accounts, we can have a look there.

The role can be a ClusterRole, created in the Helm Chart.

[fhennig]

I think I encountered a bug: https://issues.apache.org/jira/browse/SPARK-28360?jql=text%20~%20%22kubernetes%20serviceaccount%22

[fhennig]

I fixed it by setting the serviceaccount on our job pod