Merge branch 'feat/delta' of github.com:stackabletech/spark-k8s-operator into feat/delta

Author: siegfriedweber

CHANGELOG.md

@@ -16,7 +16,7 @@ All notable changes to this project will be documented in this file.
 - [BREAKING] Remove `userClassPathFirst` properties from `spark-submit`. This is an experimental feature that was
   introduced to support logging in XML format. The side effect of this removal is that the vector agent cannot
   aggregate output from the `spark-submit` containers. On the other side, it enables dynamic provisionining of
-  delta.io packages (among others) with Stackable stock images which is much more important. ([#355])
+  java packages (such as Delta Lake) with Stackable stock images which is much more important. ([#355])
 
 ### Fixed
 

deploy/helm/spark-k8s-operator/crds/crds.yaml

@@ -7061,90 +7061,6 @@ spec:
                     config:
                       default: {}
                       properties:
-                        logging:
-                          default:
-                            enableVectorAgent: null
-                            containers: {}
-                          description: Logging configuration, learn more in the [logging concept documentation](https://docs.stackable.tech/home/nightly/concepts/logging).
-                          properties:
-                            containers:
-                              additionalProperties:
-                                anyOf:
-                                  - required:
-                                      - custom
-                                  - {}
-                                description: Log configuration of the container
-                                properties:
-                                  console:
-                                    description: Configuration for the console appender
-                                    nullable: true
-                                    properties:
-                                      level:
-                                        description: The log level threshold. Log events with a lower log level are discarded.
-                                        enum:
-                                          - TRACE
-                                          - DEBUG
-                                          - INFO
-                                          - WARN
-                                          - ERROR
-                                          - FATAL
-                                          - NONE
-                                        nullable: true
-                                        type: string
-                                    type: object
-                                  custom:
-                                    description: Custom log configuration provided in a ConfigMap
-                                    properties:
-                                      configMap:
-                                        description: ConfigMap containing the log configuration files
-                                        nullable: true
-                                        type: string
-                                    type: object
-                                  file:
-                                    description: Configuration for the file appender
-                                    nullable: true
-                                    properties:
-                                      level:
-                                        description: The log level threshold. Log events with a lower log level are discarded.
-                                        enum:
-                                          - TRACE
-                                          - DEBUG
-                                          - INFO
-                                          - WARN
-                                          - ERROR
-                                          - FATAL
-                                          - NONE
-                                        nullable: true
-                                        type: string
-                                    type: object
-                                  loggers:
-                                    additionalProperties:
-                                      description: Configuration of a logger
-                                      properties:
-                                        level:
-                                          description: The log level threshold. Log events with a lower log level are discarded.
-                                          enum:
-                                            - TRACE
-                                            - DEBUG
-                                            - INFO
-                                            - WARN
-                                            - ERROR
-                                            - FATAL
-                                            - NONE
-                                          nullable: true
-                                          type: string
-                                      type: object
-                                    default: {}
-                                    description: Configuration per logger
-                                    type: object
-                                type: object
-                              description: Log configuration per container.
-                              type: object
-                            enableVectorAgent:
-                              description: Wether or not to deploy a container with the Vector log agent.
-                              nullable: true
-                              type: boolean
-                          type: object
                         resources:
                           default:
                             memory:

docs/modules/spark-k8s/pages/usage-guide/logging.adoc

@@ -1,6 +1,6 @@
 = Logging
 
-The Spark operator installs a https://vector.dev/docs/setup/deployment/roles/#agent[vector agent] as a side-car container in every application Pod except the `job` Pod that runs `spark-submit`. It also configures the logging framework to output logs in XML format. This is the same https://logging.apache.org/log4j/2.x/manual/layouts.html#XMLLayout[format] used across all Stackable products and it enables the https://vector.dev/docs/setup/deployment/roles/#aggregator[vector aggregator] to collect  logs across the entire platform.
+The Spark operator installs a https://vector.dev/docs/setup/deployment/roles/#agent[vector agent] as a side-car container in every application Pod except the `job` Pod that runs `spark-submit`. It also configures the logging framework to output logs in XML format. This is the same https://logging.apache.org/log4j/2.x/manual/layouts.html#XMLLayout[format] used across all Stackable products and it enables the https://vector.dev/docs/setup/deployment/roles/#aggregator[vector aggregator] to collect logs across the entire platform.
 
 It is the user's responsibility to install and configure the vector aggregator, but the agents can discover the aggregator automatically using a discovery ConfigMap as described in the xref:concepts:logging.adoc[logging concepts].