Skip to content

[Merged by Bors] - Logging #226

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 43 commits into from
Closed

[Merged by Bors] - Logging #226

wants to merge 43 commits into from

Conversation

siegfriedweber
Copy link
Member

@siegfriedweber siegfriedweber commented Apr 3, 2023 •
edited
Loading

Description

Add log aggregation

Closes #218

Integration tests

https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/spark-k8s-operator-it-custom/81/

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes
# Author
- [x] Changes are OpenShift compatible
- [x] CRD changes approved
- [x] Helm chart can be installed and deployed operator works
- [x] Integration tests passed (for non trivial changes)

# Reviewer
- [x] Code contains useful comments
- [x] (Integration-)Test cases added
- [x] Documentation added or updated
- [x] Changelog updated
- [x] Cargo.toml only contains references to git tags (not specific commits or branches)

# Acceptance
- [ ] Feature Tracker has been updated
- [ ] Proper release label has been added

Once the review is done, comment bors r+ (or bors merge) to merge. Further information

@siegfriedweber
Allow fragments for job, driver, and executor configurations
631aa0e
@siegfriedweber
Implement log aggregation for the Spark driver
dd33058
@siegfriedweber
Shutdown Vector after completion of the main container
d36a0f8
@siegfriedweber
Use separate config maps for driver and executor
e418781
@siegfriedweber
Add integration test for log aggregation
f993971
@siegfriedweber
Add log aggregation for the init containers and the submit job
c7bebf1
@siegfriedweber
Add support for custom log configurations
5bc8767
@siegfriedweber
Add logging to the history server
eb118ad
@siegfriedweber
Add logging tests for pyspark
fa9bfbe
@siegfriedweber
Fix integration tests
4848b92
@siegfriedweber
Add necessary volume mounts to the Spark Submit job
6b0dfed
@siegfriedweber
Enable log aggregation in all integration tests
45391ff
@siegfriedweber
Reorganize imports
13dff68
@siegfriedweber
Fix bug in configuration merge
fed4572
@siegfriedweber
Make volume mounts and node selector optional
21bef7a
@siegfriedweber
Remove unused error variants
512146a
@siegfriedweber
Upgrade stackable-operator to version 0.38.0
5413cf6
@siegfriedweber
Upgrade clap
6abfa4a
@siegfriedweber
Update changelog
a03f344
@siegfriedweber
Use volume mounts from the merged configuration
b5ccdb7
@siegfriedweber
Merge branch 'fragment-config' into logging
8196a96
@siegfriedweber
Fix Clippy warning
661fc61
@siegfriedweber
Merge branch 'fragment-config' into logging
bc4226a
@siegfriedweber
Add SubmitJobContainer struct
2d1e8eb
@siegfriedweber
Remove unused constants
fb742a1
@siegfriedweber
Adapt spark-submit container in integration tests
9e42564
@siegfriedweber
Use separate indexes for test steps
6a170f2
@siegfriedweber
Wait 10 seconds in all tests steps for MinIO to start up
aaa3cf0
@siegfriedweber
Use distinct names for SecretClasses in the tests
68cfb03
@siegfriedweber
Merge branch 'fragment-config' into logging
d508ffd
@siegfriedweber siegfriedweber self-assigned this Apr 3, 2023
maltesander
maltesander reviewed Apr 3, 2023
View reviewed changes
Copy link
Member

@maltesander maltesander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First batch. LGTM already, just minor things / questions.

@maltesander maltesander mentioned this pull request Apr 3, 2023
Closed
16 tasks
@siegfriedweber siegfriedweber requested a review from adwk67 April 4, 2023 08:29
adwk67
adwk67 requested changes Apr 4, 2023
View reviewed changes
@siegfriedweber siegfriedweber requested a review from adwk67 April 4, 2023 12:16
adwk67
adwk67 approved these changes Apr 4, 2023
View reviewed changes
Copy link
Member

@adwk67 adwk67 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@siegfriedweber
Copy link
Member Author

bors merge

bors bot pushed a commit that referenced this pull request Apr 4, 2023
@siegfriedweber
Logging (#226)
bd19c4a 
# Description

Add log aggregation
@bors
Copy link

bors bot commented Apr 4, 2023

Pull request successfully merged into main.

Build succeeded:

@bors bors bot changed the title Logging [Merged by Bors] - Logging Apr 4, 2023
@bors bors bot closed this Apr 4, 2023
@bors bors bot deleted the logging branch April 4, 2023 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maltesander maltesander maltesander left review comments

@adwk67 adwk67 adwk67 approved these changes

Assignees

@siegfriedweber siegfriedweber

Labels
None yet
Projects
Stackable Engineering
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Enable Log Aggregation for Spark
3 participants
@siegfriedweber @adwk67 @maltesander