Skip to content

Commit 48a499c

Browse files
adwk67adwk67
committed
spark-k8s bundle for 23.4 (#238)
# Description Bundle files for the 23.4 release of the spark-k8s operator. Tests on Openshift 4.11: ``` --- PASS: kuttl (1097.71s) --- PASS: kuttl/harness (0.00s) --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-true_spark-3.3.0-stackable23.4.0_ny-tlc-report-0.1.0 (154.66s) --- PASS: kuttl/harness/resources_openshift-true_spark-3.3.0-stackable23.4.0_examples-3.3.0 (127.55s) --- PASS: kuttl/harness/spark-ny-public-s3_openshift-true_spark-3.3.0-stackable23.4.0_examples-3.3.0 (85.45s) --- PASS: kuttl/harness/logging_openshift-true_spark-3.3.0-stackable23.4.0_ny-tlc-report-0.1.0_examples-3.3.0 (279.31s) --- PASS: kuttl/harness/spark-examples_openshift-true_spark-3.3.0-stackable23.4.0_examples-3.3.0 (40.49s) --- PASS: kuttl/harness/spark-history-server_openshift-true_spark-3.3.0-stackable23.4.0_examples-3.3.0 (153.51s) --- PASS: kuttl/harness/spark-pi-private-s3_openshift-true_spark-3.3.0-stackable23.4.0_examples-3.3.0 (80.40s) --- PASS: kuttl/harness/spark-pi-public-s3_openshift-true_spark-3.3.0-stackable23.4.0_examples-3.3.0 (73.35s) --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-true_spark-3.3.0-stackable23.4.0 (92.54s) ```
1 parent 0584c74 commit 48a499c

10 files changed

+4780
-2
lines changed

‎.gitignore

+4-1
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,7 @@ crate-hashes.json
1414
result
1515
image.tar
1616

17-
tilt_options.json
17+
tilt_options.json
18+
19+
**/bundle/
20+
**/bundle.Dockerfile

‎CHANGELOG.md

+5
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@ All notable changes to this project will be documented in this file.
44

55
## [Unreleased]
66

7+
### Added
8+
9+
- Generate OLM bundle for Release 23.4.0 ([#238]).
10+
711
### Changed
812

913
- `operator-rs` `0.38.0` -> `0.41.0` ([#235]).
@@ -12,6 +16,7 @@ All notable changes to this project will be documented in this file.
1216

1317
[#235]: https://github.com/stackabletech/spark-k8s-operator/pull/235
1418
[#236]: https://github.com/stackabletech/spark-k8s-operator/pull/236
19+
[#238]: https://github.com/stackabletech/spark-k8s-operator/pull/238
1520

1621
## [23.4.0] - 2023-04-17
1722

‎deploy/olm/23.4.0/manifests/configmap.yaml

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
---
2+
apiVersion: v1
3+
data:
4+
properties.yaml: |
5+
---
6+
version: 0.1.0
7+
spec:
8+
units: []
9+
properties: []
10+
kind: ConfigMap
11+
metadata:
12+
name: spark-k8s-operator-configmap
13+
labels:
14+
app.kubernetes.io/name: spark-k8s-operator
15+
app.kubernetes.io/instance: spark-k8s-operator
16+
app.kubernetes.io/version: "23.4.0"

‎deploy/olm/23.4.0/manifests/roles.yaml

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
---
2+
apiVersion: rbac.authorization.k8s.io/v1
3+
kind: ClusterRole
4+
metadata:
5+
name: spark-k8s-clusterrole
6+
rules:
7+
- apiGroups:
8+
- ""
9+
resources:
10+
- configmaps
11+
- persistentvolumeclaims
12+
- pods
13+
- secrets
14+
- serviceaccounts
15+
- services
16+
verbs:
17+
- create
18+
- delete
19+
- get
20+
- list
21+
- patch
22+
- update
23+
- watch
24+
- apiGroups:
25+
- events.k8s.io
26+
resources:
27+
- events
28+
verbs:
29+
- create
30+
- apiGroups:
31+
- security.openshift.io
32+
resources:
33+
- securitycontextconstraints
34+
resourceNames:
35+
- stackable-products-scc
36+
verbs:
37+
- use

‎deploy/olm/23.4.0/manifests/sparkapplication.yaml

+2,909
Large diffs are not rendered by default.

‎deploy/olm/23.4.0/manifests/sparkhistoryserver.yaml

+1,517
Large diffs are not rendered by default.

‎deploy/olm/23.4.0/manifests/stackable-spark-k8s-operator.clusterserviceversion.yaml

+249
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,249 @@
1+
---
2+
apiVersion: operators.coreos.com/v1alpha1
3+
kind: ClusterServiceVersion
4+
metadata:
5+
name: spark-k8s-operator.v23.4.0
6+
spec:
7+
annotations:
8+
support: stackable.tech
9+
olm.properties: '[]'
10+
categories: Storage
11+
capabilities: Full Lifecycle
12+
description: Stackable Operator for Apache Spark
13+
repository: https://github.com/stackabletech/spark-k8s-operator
14+
containerImage: docker.stackable.tech/stackable/spark-k8s-operator:23.4.0
15+
16+
displayName: Stackable Operator for Apache Spark
17+
description: |-
18+
This is a Kubernetes operator to manage [Apache Spark](https://spark.apache.org/) ensembles. The Stackable Apache Spark Operator
19+
is part of the Stackable Data Platform, a curated selection of the best open source data apps like Kafka, Druid, Trino or Spark, all
20+
working together seamlessly. Based on Kubernetes, it runs everywhere – on prem or in the cloud.
21+
22+
You can install the operator using [stackablectl or helm](https://docs.stackable.tech/spark-k8s/stable/getting_started/installation.html).
23+
See it in action in one of our [demos](https://stackable.tech/en/demos/) or follow this
24+
[tutorial](https://docs.stackable.tech/spark-k8s/stable/getting_started/first_steps.html).
25+
26+
N.B. this operator requires the following Stackable internal operators to be installed as well:
27+
28+
- [Commons Operator](https://github.com/stackabletech/commons-operator)
29+
- [Secret Operator](https://github.com/stackabletech/secret-operator)
30+
keywords:
31+
- spark-k8s
32+
maintainers:
33+
- email: [email protected]
34+
name: Stackable GmbH
35+
maturity: stable
36+
provider:
37+
name: Stackable GmbH
38+
url: https://stackable.tech
39+
version: 23.4.0
40+
minKubeVersion: 1.23.0
41+
42+
installModes:
43+
- supported: true
44+
type: OwnNamespace
45+
- supported: true
46+
type: SingleNamespace
47+
- supported: false
48+
type: MultiNamespace
49+
- supported: false
50+
type: AllNamespaces
51+
52+
customresourcedefinitions:
53+
owned:
54+
# a list of CRDs that this operator owns
55+
# name is the metadata.name of the CRD (which is of the form <plural>.<group>)
56+
- name: sparkapplications.spark.stackable.tech
57+
# version is the spec.versions[].name value defined in the CRD
58+
version: v1alpha1
59+
# kind is the CamelCased singular value defined in spec.names.kind of the CRD.
60+
kind: SparkApplication
61+
# human-friendly display name of the CRD for rendering in graphical consoles (optional)
62+
displayName: Apache Spark Application
63+
# a short description of the CRDs purpose for rendering in graphical consoles (optional)
64+
description: Represents a Spark Application
65+
- name: sparkhistoryservers.spark.stackable.tech
66+
version: v1alpha1
67+
kind: SparkHistoryServer
68+
displayName: Apache Spark History Server
69+
description: Represents a Spark History Server
70+
71+
relatedImages:
72+
- name: spark-k8s-operator
73+
image: docker.stackable.tech/stackable/spark-k8s-operator:23.4.0
74+
install:
75+
# strategy indicates what type of deployment artifacts are used
76+
strategy: deployment
77+
# spec for the deployment strategy is a list of deployment specs and required permissions - similar to a pod template used in a deployment
78+
spec:
79+
permissions:
80+
- serviceAccountName: spark-k8s-operator
81+
rules:
82+
- apiGroups:
83+
- ""
84+
resources:
85+
- pods
86+
verbs:
87+
- '*'
88+
# permissions required at the cluster scope
89+
clusterPermissions:
90+
- serviceAccountName: spark-k8s-operator
91+
rules:
92+
- apiGroups:
93+
- ""
94+
resources:
95+
- nodes
96+
verbs:
97+
- list
98+
- watch
99+
- apiGroups:
100+
- ""
101+
resources:
102+
- persistentvolumeclaims
103+
verbs:
104+
- list
105+
- apiGroups:
106+
- ""
107+
resources:
108+
- pods
109+
- configmaps
110+
- secrets
111+
- services
112+
- endpoints
113+
- serviceaccounts
114+
verbs:
115+
- create
116+
- delete
117+
- get
118+
- list
119+
- patch
120+
- update
121+
- watch
122+
- apiGroups:
123+
- rbac.authorization.k8s.io
124+
resources:
125+
- rolebindings
126+
verbs:
127+
- create
128+
- delete
129+
- get
130+
- list
131+
- patch
132+
- update
133+
- watch
134+
- apiGroups:
135+
- apps
136+
resources:
137+
- statefulsets
138+
verbs:
139+
- get
140+
- create
141+
- delete
142+
- list
143+
- patch
144+
- update
145+
- watch
146+
- apiGroups:
147+
- batch
148+
resources:
149+
- jobs
150+
verbs:
151+
- create
152+
- get
153+
- list
154+
- patch
155+
- update
156+
- watch
157+
- apiGroups:
158+
- apiextensions.k8s.io
159+
resources:
160+
- customresourcedefinitions
161+
verbs:
162+
- get
163+
- apiGroups:
164+
- events.k8s.io
165+
resources:
166+
- events
167+
verbs:
168+
- create
169+
- apiGroups:
170+
- spark.stackable.tech
171+
resources:
172+
- sparkapplications
173+
- sparkhistoryservers
174+
verbs:
175+
- get
176+
- list
177+
- patch
178+
- watch
179+
- apiGroups:
180+
- spark.stackable.tech
181+
resources:
182+
- sparkapplications/status
183+
verbs:
184+
- patch
185+
- apiGroups:
186+
- s3.stackable.tech
187+
resources:
188+
- s3connections
189+
- s3buckets
190+
verbs:
191+
- get
192+
- list
193+
- watch
194+
- apiGroups:
195+
- authentication.stackable.tech
196+
resources:
197+
- authenticationclasses
198+
verbs:
199+
- get
200+
- list
201+
- watch
202+
- apiGroups:
203+
- rbac.authorization.k8s.io
204+
resources:
205+
- clusterroles
206+
verbs:
207+
- bind
208+
resourceNames:
209+
- spark-k8s-clusterrole
210+
- apiGroups:
211+
- security.openshift.io
212+
resources:
213+
- securitycontextconstraints
214+
resourceNames:
215+
- hostmount-anyuid
216+
verbs:
217+
- use
218+
219+
deployments:
220+
- name: spark-k8s-operator
221+
spec:
222+
replicas: 1
223+
strategy:
224+
type: Recreate
225+
selector:
226+
matchLabels:
227+
app.kubernetes.io/name: spark-k8s-operator
228+
app.kubernetes.io/instance: spark-k8s-operator
229+
template:
230+
metadata:
231+
labels:
232+
app.kubernetes.io/name: spark-k8s-operator
233+
app.kubernetes.io/instance: spark-k8s-operator
234+
spec:
235+
serviceAccountName: spark-k8s-operator
236+
securityContext: {}
237+
containers:
238+
- name: spark-k8s-operator
239+
securityContext: {}
240+
image: docker.stackable.tech/stackable/spark-k8s-operator:23.4.0
241+
imagePullPolicy: IfNotPresent
242+
resources: {}
243+
volumeMounts:
244+
- mountPath: /etc/stackable/spark-k8s-operator/config-spec
245+
name: config-spec
246+
volumes:
247+
- name: config-spec
248+
configMap:
249+
name: spark-k8s-operator-configmap

‎deploy/olm/23.4.0/metadata/dependencies.yaml

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
dependencies:
3+
- type: olm.package
4+
value:
5+
packageName: commons-operator-package
6+
version: "23.4.0"
7+
- type: olm.package
8+
value:
9+
packageName: secret-operator-package
10+
version: "23.4.0"

‎deploy/olm/bundle.sh

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
#!/usr/bin/env bash
2+
# usage: bundle.sh <release>, called from base folder:
3+
# e.g. ./deploy/olm/bundle.sh 23.1.0
4+
5+
set -euo pipefail
6+
set -x
7+
8+
OPERATOR_NAME="spark-k8s-operator"
9+
10+
bundle-clean() {
11+
rm -rf "deploy/olm/${VERSION}/bundle"
12+
rm -rf "deploy/olm/${VERSION}/bundle.Dockerfile"
13+
}
14+
15+
16+
build-bundle() {
17+
opm alpha bundle generate --directory manifests --package "${OPERATOR_NAME}-package" --output-dir bundle --channels stable --default stable
18+
cp metadata/*.yaml bundle/metadata/
19+
docker build -t "docker.stackable.tech/stackable/${OPERATOR_NAME}-bundle:${VERSION}" -f bundle.Dockerfile .
20+
docker push "docker.stackable.tech/stackable/${OPERATOR_NAME}-bundle:${VERSION}"
21+
opm alpha bundle validate --tag "docker.stackable.tech/stackable/${OPERATOR_NAME}-bundle:${VERSION}" --image-builder docker
22+
}
23+
24+
main() {
25+
VERSION="$1";
26+
27+
pushd "deploy/olm/${VERSION}"
28+
bundle-clean
29+
build-bundle
30+
popd
31+
}
32+
33+
main "$@"

‎tests/test-definition.yaml

-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
#
22
# To run these tests on OpenShift you have to ensure that:
33
# 1. The "openshift" dimension below is set to "true"
4-
# 2. At least one node in the cluster is labeled with "node: 1"
54
#
65
---
76
dimensions:

0 commit comments

Comments
 (0)