Investigate setting up a history server

[razvan]

Description

Spark monitoring with the history server: https://spark.apache.org/docs/latest/monitoring.html

An example using the Google operator and a shared NFS volume : https://stackoverflow.com/a/58593909

Multiple buckets: https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Configuring_different_S3_buckets_with_Per-Bucket_Configuration

Refinement questions

CRD and instances

• Create a CRD specifically for the history server ?

It should be possible to create multiple instances of history servers. The history server CR can reference an S3Bucket.
Spark applications can reference one of these.
The operator resolves the necessary spark configuration for the job and adds it to the spark-submit command.

1. The job references a history server -> the operator reads the endpoint, bucket, path from there but uses the same credentials as the data access.
2. There are separate s3 connection definitions for reading and writing. The job uses the write connection while the history server uses the read connection.

History event storage

• Support for ReadWriteMany volumes ? Keep as issue for the future.
• Integration with Hadoop ? Keep as issue for the future. Depends on a future HDFSConnection object.

UI usage and security

• Kerberos support ? Keep as issue for the future.
• Will internal links in the web UI use the published k8s service name ? Eventually it's only a problem with the ui of the running applications, not the UI of the history server.

Acceptance criteria:

• we have a plan! An ADR.

[lfrancke]

I'm looking at this