Skip to content

Files

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Latest commit

author
Felix Hennig
Apr 12, 2023
397ae52 · Apr 12, 2023

History

History
36 lines (31 loc) · 1.43 KB

File metadata and controls

36 lines (31 loc) · 1.43 KB

Resource Requests

home:concepts:stackable_resource_requests.adoc

If no resources are configured explicitly, the operator uses the following defaults:

job:
  resources:
    cpu:
      min: '500m'
      max: "1"
    memory:
      limit: '1Gi'
driver:
  resources:
    cpu:
      min: '1'
      max: "2"
    memory:
      limit: '2Gi'
executor:
  resources:
    cpu:
      min: '1'
      max: "4"
    memory:
      limit: '4Gi'
Warning
The default values are most likely not sufficient to run a proper cluster in production. Please adapt according to your requirements. For more details regarding Kubernetes CPU limits see: Assign CPU Resources to Containers and Pods.

Spark allocates a default amount of non-heap memory based on the type of job (JVM or non-JVM). This is taken into account when defining memory settings based exclusively on the resource limits, so that the "declared" value is the actual total value (i.e. including memory overhead). This may result in minor deviations from the stated resource value due to rounding differences.

Note
It is possible to define Spark resources either directly by setting configuration properties listed under sparkConf, or by using resource limits. If both are used, then sparkConf properties take precedence. It is recommended for the sake of clarity to use either one or the other.