You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
:description: Learn how to specify product images for Stackable Data Platform deployments, including using default, custom, or mirrored registries.
5
5
6
-
To run any product on the Stackable Data Platform, you need to specify which version you want to run in the resource definition (i.e. a SparkApplication or DruidCluster).
7
-
The simplest way to do this is this:
6
+
To run any product on the Stackable Data Platform, you need to specify the desired version in the resource definition (e.g., a `SparkApplication` or `DruidCluster`).
7
+
The simplest way to do this is as follows:
8
8
9
9
[source,yaml]
10
10
----
11
11
spec:
12
12
image:
13
13
productVersion: 1.2.3 <.>
14
-
# stackableVersion: 24.3.0 optional <.>
14
+
# stackableVersion: 25.7.0 # optional <.>
15
15
----
16
-
<.> The version of your product - consult the product operator documentation to find out about supported product versions.
17
-
<.> The version of the Stackable Data Platform. If you omit it, the operator will use it's own version together with the product version to select the product image. It is recommended to omit this field, so that you are always using the correct, compatible version of the product image.
18
-
19
-
This page explains the different ways of specifying product images and the components that are involved,
20
-
starting from the simple way of using the default images to explaining how to use custom or mirrored registries as well as custom images.
16
+
<.> The version of your product.
17
+
Consult the product operator documentation to find out about supported product versions.
18
+
<.> The version of the Stackable Data Platform.
19
+
If omitted, the operator will use its own version along with the product version to select the product image.
20
+
It is recommended to omit this field to ensure you are always using the correct, compatible version of the product image.
21
21
22
22
== Product and Stackable version explained
23
23
24
-
All products of the Stackable Data Platform run on Kubernetes and are managed by Stackable Operators. One operator is responsible for exactly one product, for example Apache Spark, but can manage multiple instances (with different versions) of the product at the same time.
25
-
The operators transform a Stacklet definition (for example a SparkApplication object) into Kubernetes native objects. Some of these objects are Pods and they require _container images_ to run.
26
-
These images contain different tools for initialization jobs and/or the actual product itself and the images are version and architecture specific.
27
-
28
-
Stackable uses two separate versions to describe the images that are provided as part of the platform:
24
+
All products of the Stackable Data Platform run on Kubernetes and are managed by Stackable Operators.
25
+
Each operator is responsible for one product, such as Apache Spark, but can manage multiple instances (with different versions) of the product simultaneously.
26
+
The operators transform a Stacklet definition (e.g., a `SparkApplication` object) into Kubernetes-native objects.
27
+
Some of these objects are Pods, which require _container images_ to run.
28
+
These images contain various tools for initialization jobs and/or the actual product itself and are version and architecture-specific.
29
29
30
+
Stackable uses two separate versions to describe the images provided as part of the platform:
30
31
31
32
**Product version** +
32
-
This is the version of the product which this image provides, so this could for example be Kafka 3.3.1.
33
+
This is the version of the product that the image provides, such as Kafka 3.9.0.
33
34
34
35
TIP: You can find all products and their supported versions in the xref:operators:supported_versions.adoc[supported versions overview].
35
-
You can also find the supported versions per product on each operator page, for example for xref:kafka:index.adoc#_supported_versions[Apache Kafka].
36
-
New versions, deprecations and removals are announced in the xref:ROOT:release-notes.adoc[release notes].
36
+
You can also find the supported versions per product on each operator page, for example, for xref:kafka:index.adoc#_supported_versions[Apache Kafka].
37
+
New versions, deprecations, and removals are announced in the xref:ROOT:release-notes.adoc[Stackable Data Platform release notes].
37
38
38
39
**Stackable version** +
39
-
This version is used to track changes to the structure of the image containing the product (in the version specified by _product version_).
40
-
Stackable operators expect to find a very specific structure in the images they use to roll out the products.
41
-
This can be things like startup scripts being present, parameters these startup scripts expect, presence or location of extra libraries and similar things.
42
-
In order for our operators to work as intended every operator has to be used with images from the same release line as this operator.
43
-
44
-
What this means is, that for example the Stackable Operator for Apache HBase will by default try to use images with the same Stackable version, the following table shows a few examples to make this clearer:
40
+
This version tracks changes to the structure of the image containing the product (in the version specified by the _product version_).
41
+
Stackable operators expect to find a specific file and folder structure in the images they use to roll out the products.
42
+
This can include things like startup scripts, parameters these scripts expect, the presence or location of extra libraries, and similar aspects.
43
+
For our operators to work as intended, every operator must be used with images from the same release line as the operator.
45
44
45
+
This means, for example, that the Stackable Operator for Apache HBase will, by default, try to use images with the same Stackable version.
46
+
The following table provides a few examples to clarify this:
46
47
47
48
|===
48
-
|Operator version |HBase version |Image
49
-
50
-
|23.4.0
51
-
|3.3.0
52
-
|hbase:3.3.0-stackable23.4.0
53
-
54
-
|23.4.0
55
-
|3.3.1
56
-
|hbase:3.3.1-stackable23.4.0
57
-
58
-
|23.7.0
59
-
|3.3.0
60
-
|hbase:3.3.0-stackable23.7.0
49
+
| Operator version | HBase version | Image
61
50
62
-
|23.7.0
63
-
|3.3.1
64
-
|hbase:3.3.1-stackable23.7.0
51
+
| 23.4.0 | 3.3.0 | hbase:3.3.0-stackable23.4.0
52
+
| 23.4.0 | 3.3.1 | hbase:3.3.1-stackable23.4.0
53
+
| 23.7.0 | 3.3.0 | hbase:3.3.0-stackable23.7.0
54
+
| 23.7.0 | 3.3.1 | hbase:3.3.1-stackable23.7.0
65
55
|===
66
56
57
+
However, since the last digit of the Stackable version is considered a patch level indicator, operators will be compatible with all images from the same release line.
58
+
So, an operator of version _25.7.x_ will be compatible with all images of version _25.7.y_ release line.
59
+
This allows for shorter update cycles for users when new image versions are released that may contain security fixes.
67
60
68
-
However, since the last digit of the Stackable version is considered to be a patch level indicator, operators will be compatible with all images from the same release line.
69
-
So an operator of version _23.4.x_ will be compatible with all images of version _23.4.y_.
70
-
This is intended to allow shorter update cycles for users, when new image versions are released that may contain security fixes.
71
-
72
-
The following paragraphs explain the available settings and how they work.
73
-
74
-
At the bottom of this page in the <<_common_scenarios, common scenarios>> section some common update scenarios are explained as examples.
61
+
The following sections explain the available settings and how they work.
62
+
At the bottom of this page, in the <<_common_scenarios, common scenarios>> section, some common update scenarios are explained as examples.
75
63
76
64
== Stackable provided images
77
65
78
-
If your Kubernetes cluster has internet access, the easiest way is to use the publicly available images from the https://oci.stackable.tech/[Image registry hosted by Stackable] (as has been noted at the beginning of the page):
66
+
If your Kubernetes cluster has internet access, the easiest way is to use the publicly available images from the https://oci.stackable.tech/[Stackable Image Registry].
67
+
68
+
TIP: All our images are also mirrored to our https://quay.io/organization/stackable[Stackable Quay.io organization].
79
69
80
70
[source,yaml]
81
71
----
82
72
spec:
83
73
image:
84
-
productVersion: 3.3.1 <.>
85
-
# stackableVersion: 23.7.0 # optional <.>
74
+
productVersion: 3.9.0 <.>
75
+
# stackableVersion: 25.7.0 # optional <.>
86
76
----
87
-
<.> The version of your product - consult the product operator documentation to find out about supported product versions.
88
-
<.> The version of the Stackable Data Platform - simply omit this to use the operator version.
77
+
<.> The version of your product.
78
+
Consult the product operator documentation to find out about supported product versions.
79
+
<.> The version of the Stackable Data Platform.
80
+
Simply omit this to use the operator version.
89
81
90
-
NOTE: If the Kubernetes cluster does not have internet access, a xref:_custom_docker_registry[] or xref:_custom_images[] can be used.
82
+
NOTE: If the Kubernetes cluster does not have internet access, you can use a xref:_custom_docker_registry[] or xref:_custom_images[].
91
83
92
-
You only need to specify the product version but _can_ also specify an explicit Stackable version.
93
-
the product version can be found on the xref:operators:supported_versions.adoc[list of supported product versions] or on the product operator documentation page.
84
+
You only need to specify the product version, but you _can_ also specify an explicit Stackable version.
85
+
The product version can be found on the xref:operators:supported_versions.adoc[list of supported product versions] or on the product operator documentation page.
94
86
95
-
As images should be updated from time to time (e.g. new base image, security updates), a Stackable version can be provided.
96
-
An image with the Stackable version `23.7.0` is fixed and will never change.
97
-
Security updates within a release line will result in patch version bumps in the Stackable version to e.g. `23.7.1`.
87
+
As images should be updated from time to time (e.g., new base image, security updates), a Stackable version can be provided.
88
+
An image with the Stackable version `25.7.0` is fixed and will never change.
89
+
Security updates within a release line will result in patch version bumps in the Stackable version, for example, to `25.7.1`.
98
90
99
-
If you don't specify the Stackable version, the operator will use its own version, e.g. `23.7.0`.
91
+
If you don't specify the Stackable version, the operator will use its own version, e.g., `25.7.0`.
100
92
When using a nightly operator or a `pr` version, it will use the nightly `0.0.0-dev` image.
101
93
102
-
All the available images (with their product and stackable version) can be found in our https://oci.stackable.tech/api/v2.0/projects/sdp[OCI registry]. Information on how to browse the registry can be found xref:contributor:project-overview.adoc#docker-images[here]).
94
+
All the available images (with their product and Stackable versions) can be found in our https://oci.stackable.tech/api/v2.0/projects/sdp[Stackable OCI registry].
95
+
Information on how to browse the registry can be found in the xref:contributor:project-overview.adoc#docker-images[Docker images section of the project overview].
103
96
104
97
== Custom docker registry
105
98
106
-
Custom docker registries can be used to fetch the image from a local image registry rather than from the internet.
107
-
The perquisite is that you mirror all the required images the same way (with the same name and tag) as the images provided by Stackable.
99
+
Custom Docker registries can be used to fetch the image from a local image registry rather than from the internet.
100
+
The prerequisite is that you mirror all the required images in the same way (with the same name and tag) as the images provided by Stackable.
108
101
109
-
Afterwards you can use the following snippet to configure your custom docker repo:
102
+
Afterward, you can use the following snippet to configure your custom Docker repo:
110
103
111
104
[source,yaml]
112
105
----
113
106
spec:
114
107
image:
115
108
productVersion: 3.3.1
116
-
stackableVersion: 23.7.0
117
-
repo: my.corp/myteam/stackable
109
+
stackableVersion: 25.7.0 # Optional
110
+
repo: my.corp/myteam/stackable <.>
118
111
----
112
+
<.> We recommend not including a slash at the end while we plan on https://github.com/stackabletech/operator-rs/issues/1020[improving the situation].
119
113
120
114
This will change the image from the default Stackable repository `oci.stackable.tech/sdp/kafka:3.3.1-stackable23.7.0` to `my.corp/myteam/stackable/kafka:3.3.1-stackable23.7.0`.
121
115
122
116
== [[customimages]] Custom images
123
117
124
118
Custom images can be used to fetch arbitrary images from local or public registries.
125
-
In comparison to the xref:_custom_docker_registry[], this allows to provide self-hosted or user-created images (e.g. userextended Stackable images).
126
-
If your image has other tags or names than the ones provided by Stackable you need to use this option.
119
+
Unlike the xref:_custom_docker_registry[], this allows you to provide self-hosted or user-created images (e.g., user-extended Stackable images).
120
+
If your image has different tags or names than those provided by Stackable, you need to use this option.
127
121
128
122
[source,yaml]
129
123
----
@@ -134,32 +128,34 @@ spec:
134
128
----
135
129
136
130
Even though the product version is not used anymore for image selection, you still need to provide it, as the operators configure their respective products based on the product version.
137
-
This affects e.g. configuration properties or available product features.
138
-
Only when the correct product version is given to the operator, the product will work correctly, so you need to provide the product version that you have used in your custom image.
131
+
This affects configuration properties or available product features.
132
+
Only when the correct product version is given to the operator will the product work correctly, so you need to provide the product version that you have used in your custom image.
139
133
140
134
Using custom images has a few limitations that users should be aware of:
141
135
142
-
* The images will *have* to have the same structures that Stackable operators expect.
143
-
This should usually be ensured by specifying a Stackable image in the `FROM` clause of the Dockerfile (all the available images can be found in our https://oci.stackable.tech/api/v2.0/projects/sdp[OCI registry] - the schema is typically: `oci.stackable.tech/sdp/<product>:<product-version>-stackable<stackable-version>`. Information on how to browse the registry can be found xref:contributor:project-overview.adoc#docker-images[here]).
136
+
* The images must have the same structures that Stackable operators expect.
137
+
This should usually be ensured by specifying a Stackable image in the `FROM` clause of the Dockerfile (all the available images can be found in our https://oci.stackable.tech/api/v2.0/projects/sdp[Stackable OCI registry] - the schema is typically: `oci.stackable.tech/sdp/<product>:<product-version>-stackable<stackable-version>`.
138
+
Information on how to browse the registry can be found in the xref:contributor:project-overview.adoc#docker-images[Docker images section of the project overview]).
144
139
145
-
* Images will have to be upgraded for every new Stackable release to follow structural changes that Stackable may have made to their images.
146
-
When deriving images from official Stackable images this will mean updating the version of the image in the `FROM` clause to the correct Stackable release.
140
+
* Images will need to be upgraded for every new Stackable release to follow structural changes that Stackable may have made to their images.
141
+
When deriving images from official Stackable images, this will mean updating the version of the image in the `FROM` clause to the correct Stackable release.
147
142
148
143
* It is not possible to update the Stackable Platform to a new version without changing the deployed cluster definitions when using custom images.
149
144
The recommended process here is:
150
145
151
-
** Set `reconciliationPaused` to `true` in your product cluster (see xref:operations/cluster_operations.adoc[])
152
-
** Update Stackable platform
153
-
** Change custom images in cluster specifications
154
-
** Set `reconciliationPaused` to `false` again to start reconciliation again
146
+
** Set `reconciliationPaused` to `true` in your product cluster (see xref:operations/cluster_operations.adoc[cluster operations documentation]).
147
+
** Update the Stackable platform.
148
+
** Change custom images in cluster specifications.
149
+
** Set `reconciliationPaused` to `false` again to start reconciliation.
155
150
156
-
## [[common_scenarios]] Common Scenarios
151
+
== [[common_scenarios]] Common scenarios
157
152
158
-
### Planned platform updates
159
-
This is probably the most common scenario, users do not specify a Stackable version, and thus the operators always pick the image from their exact release.
160
-
Updates happen by updating Stackable Operators, which will in turn restart the products with the new images.
153
+
=== Planned platform updates
161
154
162
-
#### Config
155
+
This is probably the most common scenario, where users do not specify a Stackable version, and thus the operators always pick the image from their exact release.
156
+
Updates happen by updating Stackable Operators, which will, in turn, restart the products with the new images.
157
+
158
+
==== Config
163
159
164
160
[source,yaml]
165
161
----
@@ -168,13 +164,14 @@ spec:
168
164
productVersion: 3.3.1
169
165
----
170
166
171
-
### Custom images / pinned images
172
-
When a setup requires the utmost stability, and it is preferable for things to break, rather than run with a different image version that for example has not been certified.
173
-
Or when a user requires custom libraries / code in the images they run and build their own images derived from official Stackable images, this is the only possible way to do this.
167
+
=== Custom images / pinned images
168
+
169
+
When a setup requires the utmost stability, and it is preferable for things to break rather than run with a different image version that, for example, has not been certified, or when a user requires custom libraries or code in the images they run and builds their own images derived from official Stackable images, this is the only possible way to do this.
170
+
171
+
Please see the warnings in the <<customimages, custom images section>> above for how to upgrade in this scenario.
174
172
175
-
Please see the warnings in <<customimages, custom images section>> above for how to upgrade in this scenario.
0 commit comments