Make volume mounts on the job pod configurable.

[razvan]

Description

The CR allows users to mount volumes on the driver and executor pods, but not on job pods.

But in kerberized environments, users need the possibility to mount key tabs on the pod executing the spark-submit command. This currently only possible with pod overrides which is clumsy.

• Add a SparkApplication.spec.job.config.volumeMounts field
• So that SparkApplication.spec.volumes can be mounted in Jobs just like in the Driver and Executor
• This was an oversight in the initial design because we never needed to integrate with e.g. HDFS so far, now we do

Proposed change:

spec:
    job:
      config:
        volumeMounts:

With volumeMounts be the exact same as for driver and executor.

Acceptance criteria

- [x] Users can mount volumes on the job pod by specifying them directly in the `SparkApplication` resource without the need to create pod overrides.
- [x] Documentation: Check if we have places where podOverrides are used to achieve the same and update this
- [x] Documentation: Update CRD Reference page

[fhennig]

I just wanted to raise awareness that I saw a bunch of different ways of mounting volumes across the platform, and we should keep it similar. But I think spark is the odd one out anyways. Link to relevant ticket: stackabletech/issues#471

[lfrancke]

Good point, thanks!

@razvan et. al. any opinions?
Can someone go through all operators and briefly check how consistent they are?

[razvan]

TL;DR: IMO naming should be consistent but structure should be flexible.

I agree that the property names should be consistent. Druid's choice of clusterConfig.extraVolumes is off considering there seem to be no clusterConfig.volumes to begin with (from the user's perspective).

But there are several constraints to be considered when thinking about the CRD structure in general:

• Does the product even have the concept of clusterConfig ? Not for Spark applications (at least until we add support for spark connector) .
• What volume mounts are supported by different product roles ? Can all volumes be mounted everywhere ? Short answer: no. Example: Druid local cache volumes only make sense on one role.
• operator-rs design: logging, secret volumes, listener volumes

[nightkr]

Do we expect users to mount arbitrary volumes or is this primarily for the Kerberos use case?

Secret-op is implemented using volumes, but I don't think they're the right abstraction to expose unless you actually want to store arbitrary data. For the Kerberos use case I'd assume you'd also need to tell spark-submit about where to find the Kerberos config/keytab, right?

[nightkr]

As for consistency, AFAIK we always treat secrets as separate from storage volumes elsewhere, even for products that use both.

[lfrancke]

Do we expect users to mount arbitrary volumes or is this primarily for the Kerberos use case?

Yes, this came up due to Kerberos.
But it is also an inconsistency as the other two (driver & executor) already allow it
And it is a valid use-case to mount arbitrary stuff you need in your job.

So, I think it's worth having this field in either case but we might also think about whether there is a better way to "support" Kerberos.

We had a chat about that yesterday after the daily and should have added it to the ticket: We decided to go with this simple solution for now, knowing that "Discovery 2.0" is a thing we want to tackle at some point.