fix: Remove userClassPathFirst properties #355

razvan · 2024-02-12T16:23:43Z

Fixes #354

feat(test): addded test for delta.io
fix: remove userClassPathFirst and extraClassPath properties.
feat: remove all refs to the extra-jars folder

@17

This fails with: ``` spark org/apache/spark/sql/delta/stats/StatisticsCollection$SqlParser$$anon$1.visitMultipartIdentifierList(Lorg/apache/spark/sql/catalyst/parser/SqlBaseParser$MultipartIdentifierListContext;)Lscala/collection/Seq; @17: invokevirtual spark Reason: spark Type 'org/apache/spark/sql/catalyst/parser/SqlBaseParser$MultipartIdentifierListContext' (current frame, stack[1]) is not assignable to 'org/antlr/v4/runtime/ParserRuleContext' spark Current Frame: spark bci: @17 spark flags: { } spark locals: { 'org/apache/spark/sql/delta/stats/StatisticsCollection$SqlParser$$anon$1', 'org/apache/spark/sql/catalyst/parser/SqlBaseParser$MultipartIdentifierListContext' } spark stack: { 'org/apache/spark/sql/catalyst/parser/ParserUtils$', 'org/apache/spark/sql/catalyst/parser/SqlBaseParser$MultipartIdentifierListContext', 'scala/Option', 'scala/Function0' } spark Bytecode: spark 0000000: b200 232b b200 23b6 0027 2a2b ba00 3f00 spark 0000010: 00b6 0043 c000 45b0 spark spark at org.apache.spark.sql.delta.stats.StatisticsCollection$SqlParser.<init>(StatisticsCollection.scala:409) spark at org.apache.spark.sql.delta.stats.StatisticsCollection$.<init>(StatisticsCollection.scala:422) spark at org.apache.spark.sql.delta.stats.StatisticsCollection$.<clinit>(StatisticsCollection.scala) spark at org.apache.spark.sql.delta.OptimisticTransactionImpl.updateMetadataInternal(OptimisticTransaction.scala:429) spark at org.apache.spark.sql.delta.OptimisticTransactionImpl.updateMetadataInternal$(OptimisticTransaction.scala:424) spark at org.apache.spark.sql.delta.OptimisticTransaction.updateMetadataInternal(OptimisticTransaction.scala:142) spark at org.apache.spark.sql.delta.OptimisticTransactionImpl.updateMetadata(OptimisticTransaction.scala:400) spark at org.apache.spark.sql.delta.OptimisticTransactionImpl.updateMetadata$(OptimisticTransaction.scala:393) spark at org.apache.spark.sql.delta.OptimisticTransaction.updateMetadata(OptimisticTransaction.scala:142) spark at org.apache.spark.sql.delta.schema.ImplicitMetadataOperation.updateMetadata(ImplicitMetadataOperation.scala:97) spark at org.apache.spark.sql.delta.schema.ImplicitMetadataOperation.updateMetadata$(ImplicitMetadataOperation.scala:56) spark at org.apache.spark.sql.delta.commands.WriteIntoDelta.updateMetadata(WriteIntoDelta.scala:76) spark at org.apache.spark.sql.delta.commands.WriteIntoDelta.write(WriteIntoDelta.scala:162) spark at org.apache.spark.sql.delta.commands.WriteIntoDelta.$anonfun$run$1(WriteIntoDelta.scala:105) ```

razvan · 2024-02-12T16:52:00Z

CI: https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/spark-k8s-operator-it-custom/122/

…ties.

razvan · 2024-02-13T13:44:34Z

CI: https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/spark-k8s-operator-it-custom/123/

razvan · 2024-02-13T15:20:27Z

Tests pass on OpenShift 4.13:

--- PASS: kuttl (1670.68s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-true_spark-3.5.0 (167.41s)
        --- PASS: kuttl/harness/spark-history-server_openshift-true_spark-3.5.0_s3-use-tls-true (409.71s)
        --- PASS: kuttl/harness/smoke_openshift-true_spark-3.5.0_s3-use-tls-true (282.40s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-true_spark-3.5.0_ny-tlc-report-0.1.0 (207.65s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-true_spark-3.5.0 (245.33s)
        --- PASS: kuttl/harness/spark-pi-public-s3_openshift-true_spark-3.5.0 (167.31s)
        --- PASS: kuttl/harness/iceberg_spark-3.5.0 (115.25s)
        --- PASS: kuttl/harness/spark-examples_openshift-true_spark-3.5.0 (102.49s)
        --- PASS: kuttl/harness/pod_overrides_openshift-true_spark-3.5.0 (231.25s)
        --- PASS: kuttl/harness/resources_openshift-true_spark-3.5.0 (131.30s)
        --- PASS: kuttl/harness/logging_openshift-true_spark-3.5.0_ny-tlc-report-0.1.0 (544.06s)
        --- PASS: kuttl/harness/delta_spark-delta-3.5.0_delta-3.1.0 (281.07s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-true_spark-3.5.0_s3-use-tls-true (231.64s)
PASS

soenkeliebau · 2024-02-13T15:24:12Z

Tests pass on OpenShift 4.13:

this is super sexy :)

NickLarsenNZ

LGTM

tests/templates/kuttl/delta/40-spark-app.yaml.j2

NickLarsenNZ · 2024-02-13T18:37:02Z

re-running the failed job (503 from docker login)

sbernauer

I would be ok with merging this, but I would really like us solving the classpath problem with the logging libs instead of ripping out logging :)

CHANGELOG.md

docs/modules/spark-k8s/pages/usage-guide/logging.adoc

tests/test-definition.yaml

tests/templates/kuttl/delta/40-spark-app.yaml.j2

Co-authored-by: Sebastian Bernauer <[email protected]>

razvan · 2024-02-14T15:59:44Z

I would be ok with merging this, but I would really like us solving the classpath problem with the logging libs instead of ripping out logging :)

What do you mean "ripping out logging" ? Logs from applications are not affected by this change.

…tor into feat/delta

razvan · 2024-02-14T16:59:58Z

CI: https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/spark-k8s-operator-it-custom/124/

sbernauer

What do you mean "ripping out logging"

That the logs of the spark-submit job will not end up in you logging sink if I understood correctly. Imagine a job that runs every hours did not properly start this night at 02:00. You don't have any insights what happened, e.g. the Kerberos server not being reachable.

But if you did not get it to work it is what it is I guess

razvan added 5 commits February 12, 2024 11:33

fix: remove userClassPathFirst and extraClassPath properties.

e82bb56

feat: remove all refs to the extra-jars folder

7f65711

fix: pr number

faef52d

fix lints

fa0f40f

razvan mentioned this pull request Feb 12, 2024

Bugfix: remove usage of "userClassPathFirst" properties [was: Investigate delta.io integration] #354

Closed

4 tasks

razvan added 2 commits February 13, 2024 11:34

Revert extra-jars removal but don't set the userClassPathFirst proper…

a93de66

…ties.

fix(tests): update Python script for logging tests

a76d5e1

fix(docs): update changelog and add logging.adoc page

91cf911

razvan requested a review from siegfriedweber February 13, 2024 14:15

razvan mentioned this pull request Feb 13, 2024

Reorganize logging jars [was: experimental: spark with delta] stackabletech/docker-images#556

Closed

fix: language tool lints

8531e85

NickLarsenNZ previously approved these changes Feb 13, 2024

View reviewed changes

tests/templates/kuttl/delta/40-spark-app.yaml.j2 Outdated Show resolved Hide resolved

fix: better logging docs

ccb6672

razvan dismissed NickLarsenNZ’s stale review via ccb6672 February 14, 2024 12:43

Remove the logging configuration for the Spark submit job

eba0795

sbernauer self-requested a review February 14, 2024 15:27

sbernauer requested changes Feb 14, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

docs/modules/spark-k8s/pages/usage-guide/logging.adoc Outdated Show resolved Hide resolved

tests/test-definition.yaml Outdated Show resolved Hide resolved

tests/templates/kuttl/delta/40-spark-app.yaml.j2 Outdated Show resolved Hide resolved

sbernauer changed the title ~~Remove userClassPathFirst properties~~ fix: Remove userClassPathFirst properties Feb 14, 2024

razvan and others added 3 commits February 14, 2024 16:53

Update CHANGELOG.md

6e08536

Co-authored-by: Sebastian Bernauer <[email protected]>

Update docs/modules/spark-k8s/pages/usage-guide/logging.adoc

761f1dc

Co-authored-by: Sebastian Bernauer <[email protected]>

regenerate charts

fc86d7a

siegfriedweber added 2 commits February 14, 2024 17:13

Remove unnecessary volume mount

f4ca6b5

Remove the logging configuration for the submit job in all tests

d78b3d2

siegfriedweber and others added 2 commits February 14, 2024 17:15

Merge branch 'feat/delta' of github.com:stackabletech/spark-k8s-opera…

bd23972

…tor into feat/delta

Implement review feedback

61a82ff

sbernauer previously approved these changes Feb 15, 2024

View reviewed changes

Fix assertions in the resources test

3815236

siegfriedweber dismissed sbernauer’s stale review via 3815236 February 15, 2024 07:52

siegfriedweber approved these changes Feb 15, 2024

View reviewed changes

razvan enabled auto-merge February 15, 2024 12:33

razvan added this pull request to the merge queue Feb 15, 2024

Merged via the queue into main with commit 98299e2 Feb 15, 2024

razvan deleted the feat/delta branch February 15, 2024 12:38

siegfriedweber mentioned this pull request Dec 19, 2024

feat: run containerdebug in the background #508

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix: Remove userClassPathFirst properties #355

fix: Remove userClassPathFirst properties #355

Uh oh!

razvan commented Feb 12, 2024 •

edited

Loading

Uh oh!

razvan commented Feb 12, 2024

Uh oh!

razvan commented Feb 13, 2024

Uh oh!

razvan commented Feb 13, 2024

Uh oh!

soenkeliebau commented Feb 13, 2024

Uh oh!

NickLarsenNZ left a comment

Uh oh!

Uh oh!

NickLarsenNZ commented Feb 13, 2024 •

edited

Loading

Uh oh!

sbernauer left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

razvan commented Feb 14, 2024

Uh oh!

razvan commented Feb 14, 2024

Uh oh!

sbernauer left a comment

Uh oh!

Uh oh!

Uh oh!

fix: Remove userClassPathFirst properties #355

fix: Remove userClassPathFirst properties #355

Uh oh!

Conversation

razvan commented Feb 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

razvan commented Feb 12, 2024

Uh oh!

razvan commented Feb 13, 2024

Uh oh!

razvan commented Feb 13, 2024

Uh oh!

soenkeliebau commented Feb 13, 2024

Uh oh!

NickLarsenNZ left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

NickLarsenNZ commented Feb 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sbernauer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

razvan commented Feb 14, 2024

Uh oh!

razvan commented Feb 14, 2024

Uh oh!

sbernauer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

razvan commented Feb 12, 2024 •

edited

Loading

NickLarsenNZ commented Feb 13, 2024 •

edited

Loading