Implement resource limits and requests for Spark pods

[fhennig]

In other products we're introducing a common resource limit configuration. This ticket is about evaluating whether it makes sense to use it in this operator too. If it does, this ticket includes the implementation:

Part of this epic stackabletech/issues#241

Update

The spark-k8s operator hands management of driver and executor pods off to spark itself, and there are CRD fields for this purpose that are used to construct the relevant spark-submit arguments. This will remain unchanged. Currently there are no resources specified for the initiating/Job Pod, so this ticket will cover that. Specifically, since the operator does follow the role-pattern for other products, the struct will a top-level (directly under .spec) field: the resources defined here will be passed on to the ContainerBuilder used for the job, here.

Acceptance criteria

• Resource requests and limits are configurable in CRD using the common structs from operator-rs
• Resource requests and limits are configured for the initial Kubernetes Job pod (only)
• Resource requests and limits are configured in the product (e.g. "-Xmx" etc. for Java based images)
• Adapt/Add integration tests to specify and test correct amount of resources
• Adapt/Add examples
• Adapt documentation: New section in usage.adoc with product specific information and link to common shared resources concept
• Optional: Use sensible defaults for each role (if reasonable and applicable) and document accordingly in usage.adoc
• Feature Tracker is updated

[lfrancke]

I left this in the refinement column because I vaguely remember someone saying that this is already implemented for Spark and might just need a different implementation? I'm not sure to be honest and I might be wrong.

[adwk67]

Cores, Core-limit and memory can be specified for driver & executor pods (but not for the initiating job): e.g. https://github.com/stackabletech/spark-k8s-operator/blob/main/examples/ny-tlc-report-image.yaml#L36-L42

[lfrancke]

Thank you! I need some more input though I'm afraid :)

Is there still anything to be done to close this ticket or can it already be closed as done?

1. Do we need this for the initiating job?
2. Because it was implemented earlier than the others: Does it follow the CRD standard of the other operators and does it use the same implementation etc.?

[adwk67]

1. I would say, no, as the initiating pod is only relatively short-lived and gets removed after a few minutes.
2. The implementation for driver/executors is a little different, as we use the resource defs to add spark configs to a spark-submit command rather than building those pods directly. If we were to implement this for the initiating pod then it would be a little different, as we don't have any roles/rolegroups as such (so the resources field in the CRD could be defined the same way but added at a top level directly under .spec).

@razvan what do you think?

[adwk67]

Having thought about this, I think it is worthwhile to implement this for the job Pod (the implementation would follow the standard pattern of introducing a struct that is defined in the CRD and passed through to the ContainerBuilder), but to leave the driver/executor management to Spark (as is currently done).

[lfrancke]

Great, thank you!
In that case I'll leave it in the Refinement column for someone to update the ticket accordingly.

[adwk67]

In that case I'll leave it in the Refinement column for someone to update the ticket accordingly.

Have updated the issue description accordingly.

[razvan]

Sorry for having missed this and yes I agree with @adwk67 that the initiating spark-submit Pod should specify resource limits.

I'm not sure it's a good idea to make these limits configurable by the user.

[lfrancke]

I will update the Feature Tracker, could you please check if all other boxes can be ticked? @razvan @adwk67

[adwk67]

All other boxes have been ticked, except

• "only implement for Job pod" as the common struct has now been used for all pods (Job, driver and executor) to keep things consistent
• "product limits" (Spark sets e.g. overall memory based on memory limit + memoryOverHead-factor which defaults to 10%)