Skip to content

feat: run containerdebug in the background #508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Jan 13, 2025
Merged

Conversation

razvan
Copy link
Member

@razvan razvan commented Dec 10, 2024

Description

Part of stackabletech/issues#527

The containerdebug binary is executed in the Spark application pods by evaluating the _STACKABLE_PRE_HOOK in the run-spark.sh script. This script is the entrypoint of the Spark product image,

Had to reorganize the smoke test in order to fetch debug information while the Spark application is running. Fetching the containerdebug output after the job succeeds is not possible because the pods are terminated.

Also refactored the container's command and args properties to be more readable and more in line with other operators.

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes
# Author
- [ ] Changes are OpenShift compatible
- [ ] CRD changes approved
- [ ] CRD documentation for all fields, following the [style guide](https://docs.stackable.tech/home/nightly/contributor/docs/style-guide).
- [ ] Helm chart can be installed and deployed operator works
- [ ] Integration tests passed (for non trivial changes)
- [ ] Changes need to be "offline" compatible
# Reviewer
- [ ] Code contains useful comments
- [ ] Code contains useful logging statements
- [ ] (Integration-)Test cases added
- [ ] Documentation added or updated. Follows the [style guide](https://docs.stackable.tech/home/nightly/contributor/docs/style-guide).
- [ ] Changelog updated
- [ ] Cargo.toml only contains references to git tags (not specific commits or branches)
# Acceptance
- [ ] Feature Tracker has been updated
- [ ] Proper release label has been added
- [ ] [Roadmap](https://github.com/orgs/stackabletech/projects/25/views/1) has been updated

@razvan razvan marked this pull request as ready for review December 10, 2024 20:33
@razvan razvan requested a review from a team December 10, 2024 20:33
@razvan
Copy link
Member Author

razvan commented Dec 11, 2024

🟢 Tests

--- FAIL: kuttl (2263.15s)
    --- FAIL: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/spark-examples_openshift-false_spark-3.5.2 (52.55s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-3.5.2_s3-use-tls-true (149.42s)
        --- FAIL: kuttl/harness/logging_openshift-false_spark-3.5.2_ny-tlc-report-0.2.0 (45.57s)
        --- PASS: kuttl/harness/resources_openshift-false_spark-3.5.2 (87.20s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-3.5.2_s3-use-tls-true (141.01s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-false_spark-3.5.2 (131.98s)
        --- PASS: kuttl/harness/overrides_openshift-false_spark-3.5.2 (113.82s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-false_spark-3.5.2 (109.78s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-false_spark-3.5.2_ny-tlc-report-0.2.0 (133.09s)
        --- PASS: kuttl/harness/spark-pi-public-s3_openshift-false_spark-3.5.2 (96.44s)
        --- PASS: kuttl/harness/custom-log-directory_openshift-false_spark-3.5.2_hdfs-latest-3.4.0_zookeeper-latest-3.9.2 (255.88s)
        --- PASS: kuttl/harness/iceberg_openshift-false_spark-3.5.2 (106.98s)
        --- PASS: kuttl/harness/hbase-connector_openshift-false_spark-3.5.2_hbase-2.4.18_hdfs-latest-3.4.0_zookeeper-latest-3.9.2 (353.36s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-3.5.2_s3-use-tls-true (238.10s)
        --- PASS: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-3.5.2_delta-3.1.0 (247.97s)
FAIL

The logging test passed when executed as a single:

--- PASS: kuttl (281.34s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/logging_openshift-false_spark-3.5.2_ny-tlc-report-0.2.0 (281.33s)
PASS

@razvan
Copy link
Member Author

razvan commented Dec 19, 2024

I cleaned it up a bit

--- PASS: kuttl (1425.77s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/spark-pi-public-s3_openshift-false_spark-3.5.2 (119.06s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-3.5.2_s3-use-tls-true (239.11s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-false_spark-3.5.2 (134.04s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-3.5.2_s3-use-tls-true (156.62s)
        --- PASS: kuttl/harness/resources_openshift-false_spark-3.5.2 (88.08s)
        --- PASS: kuttl/harness/spark-examples_openshift-false_spark-3.5.2 (47.55s)
        --- PASS: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-3.5.2_delta-3.1.0 (286.13s)
        --- PASS: kuttl/harness/iceberg_openshift-false_spark-3.5.2 (67.87s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-3.5.2_s3-use-tls-true (148.06s)
        --- PASS: kuttl/harness/custom-log-directory_openshift-false_spark-3.5.2_hdfs-latest-3.4.0_zookeeper-latest-3.9.2 (234.34s)
        --- PASS: kuttl/harness/overrides_openshift-false_spark-3.5.2 (138.04s)
        --- PASS: kuttl/harness/hbase-connector_openshift-false_spark-3.5.2_hbase-2.4.18_hdfs-latest-3.4.0_zookeeper-latest-3.9.2 (583.99s)
        --- PASS: kuttl/harness/logging_openshift-false_spark-3.5.2_ny-tlc-report-0.2.0 (335.81s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-false_spark-3.5.2_ny-tlc-report-0.2.0 (126.25s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-false_spark-3.5.2 (110.46s)
PASS

nightkr
nightkr previously approved these changes Dec 19, 2024
Co-authored-by: Natalie Klestrup Röijezon <[email protected]>
nightkr
nightkr previously approved these changes Jan 13, 2025
@razvan razvan added this pull request to the merge queue Jan 13, 2025
Merged via the queue into main with commit b166061 Jan 13, 2025
17 checks passed
@razvan razvan deleted the feature/containerdebug branch January 13, 2025 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

4 participants