Skip to content

Commit 7edf6a1

Browse files
authored
Merge pull request #1776 from tormath1/tormath1/sysext
✨ Add `flatcar-sysext` template to use regular Flatcar images
2 parents e590271 + 5a39a20 commit 7edf6a1

File tree

10 files changed

+553
-7
lines changed

10 files changed

+553
-7
lines changed

Makefile

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -158,7 +158,8 @@ e2e-templates: $(addprefix $(E2E_NO_ARTIFACT_TEMPLATES_DIR)/, \
158158
cluster-template-without-lb.yaml \
159159
cluster-template.yaml \
160160
cluster-template-flatcar.yaml \
161-
cluster-template-k8s-upgrade.yaml)
161+
cluster-template-k8s-upgrade.yaml \
162+
cluster-template-flatcar-sysext.yaml)
162163
# Currently no templates that require CI artifacts
163164
# $(addprefix $(E2E_TEMPLATES_DIR)/, add-templates-here.yaml) \
164165
@@ -406,7 +407,8 @@ release-notes: $(RELEASE_NOTES) ## Generate release notes
406407
templates: ## Generate cluster templates
407408
templates: templates/cluster-template.yaml \
408409
templates/cluster-template-without-lb.yaml \
409-
templates/cluster-template-flatcar.yaml
410+
templates/cluster-template-flatcar.yaml \
411+
templates/cluster-template-flatcar-sysext.yaml
410412

411413
templates/cluster-template.yaml: kustomize/v1alpha8/default $(KUSTOMIZE) FORCE
412414
$(KUSTOMIZE) build "$<" > "$@"

docs/book/src/clusteropenstack/configuration.md

Lines changed: 34 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -74,10 +74,33 @@ The image can be referenced by exposing it as an environment variable `OPENSTACK
7474

7575
Some OS like [Fedora CoreOS](https://getfedora.org/en/coreos) or [Flatcar](https://www.flatcar.org/) do not use cloud-init but [Ignition](https://coreos.github.io/ignition/) to provision the instance. You need to enable the [Ignition experimental feature](https://cluster-api.sigs.k8s.io/tasks/experimental-features/ignition.html): `export EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true`
7676

77-
To use Flatcar image:
78-
* Build the image with the [image-builder](https://image-builder.sigs.k8s.io/capi/providers/openstack.html): `make OEM_ID=openstack build-qemu-flatcar`
79-
* Export the name of the uploaded image: `export OPENSTACK_FLATCAR_IMAGE_NAME=flatcar-stable-3374.2.5-kube-v1.25.6`
80-
* When generating the cluster configuration, use the following Cluster API [flavor](https://cluster-api.sigs.k8s.io/clusterctl/commands/generate-cluster.html?#flavors): `--flavor flatcar` (_NOTE_: Don't forget to refer to the [external-cloud-provider](https://cluster-api-openstack.sigs.k8s.io/topics/external-cloud-provider.html) section)
77+
Flatcar comes in two [flavor][flavor] variants:
78+
* `flatcar`
79+
80+
This variant relies on a Flatcar image built using the image-builder project: the Kubernetes version is bound to the Flatcar version and a rebuild of the image is required for each Kubernetes or Flatcar upgrade.
81+
82+
To build and use Flatcar image:
83+
* Build the image with the [image-builder][image-builder]: `make OEM_ID=openstack build-qemu-flatcar`
84+
* Upload the image
85+
* Export the name of the uploaded image: `export OPENSTACK_FLATCAR_IMAGE_NAME=flatcar-stable-3374.2.5-kube-v1.25.6`
86+
* When generating the cluster configuration, use the following Cluster API [flavor][flavor]: `--flavor flatcar` (_NOTE_: Don't forget to refer to the [external-cloud-provider][external-cloud-provider] section)
87+
88+
* `flatcar-sysext`
89+
90+
This variant relies on a plain Flatcar image and it leverages [systemd-sysext][systemd-sysext] feature to install and update Kubernetes components: the Kubernetes version is not bound to the Flatcar version (i.e Flatcar can be independently upgraded from Kubernetes and vice versa).
91+
92+
The template comes with a [systemd-sysupdate][systemd-sysupdate] configuration file that will download each new patch version of Kubernetes (i.e if you start with Kubernetes 1.x.y, systemd-sysupdate will automatically pull 1.x.y+1 but not 1.x+1.y), please note that this behavior is disabled by default. To enable the Kubernetes auto-update you can:
93+
* Update the template to enable the `systemd-sysupdate.timer`
94+
* Or run the following command on the nodes: `sudo systemctl enable --now systemd-sysupdate.timer`
95+
96+
When the Kubernetes release reaches end-of-life it will not receive updates anymore. To switch to a new major version, do a `sudo rm /etc/sysupdate.kubernetes.d/kubernetes-*.conf` and download the new update config into the folder with `cd /etc/sysupdate.kubernetes.d && sudo wget https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${KUBERNETES_VERSION%.*}.conf`.
97+
98+
To coordinate the node reboot, we recommend to use [Kured][kured]. Note that running `kubeadm upgrade apply` on the first controller and `kubeadm upgrade node` on all other nodes is not automated (yet), see the [docs](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/).
99+
100+
To use Flatcar image:
101+
* Upload an image on OpenStack from the Flatcar release servers (e.g for Stable, you might use this image: https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_openstack_image.img)
102+
* Export the name of the uploaded image: `export FLATCAR_IMAGE_NAME=flatcar_production_openstack_image`
103+
* When generating the cluster configuration, use the following Cluster API [flavor][flavor]: `--flavor flatcar-sysext` (_NOTE_: Don't forget to refer to the [external-cloud-provider][external-cloud-provider] section)
81104

82105
## SSH key pair
83106

@@ -640,3 +663,10 @@ $ kubectl get openstackcluster
640663
NAME CLUSTER READY NETWORK SUBNET BASTION
641664
nonha nonha true 2e2a2fad-28c0-4159-8898-c0a2241a86a7 53cb77ab-86a6-4f2c-8d87-24f8411f15de 10.0.0.213
642665
```
666+
667+
[external-cloud-provider]: https://cluster-api-openstack.sigs.k8s.io/topics/external-cloud-provider.html
668+
[flavor]: https://cluster-api.sigs.k8s.io/clusterctl/commands/generate-cluster.html?#flavors
669+
[image-builder]: https://image-builder.sigs.k8s.io/capi/providers/openstack.html
670+
[kured]: https://github.com/kubereboot/kured
671+
[systemd-sysext]: https://www.flatcar.org/docs/latest/provisioning/sysext/
672+
[systemd-sysupdate]: https://www.freedesktop.org/software/systemd/man/latest/sysupdate.d.html

hack/ci/cloud-init/controller.yaml.tpl

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,8 @@
6464
IMAGE_URLS+="https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/cirros/2022-12-05/cirros-0.6.1-x86_64-disk.img,"
6565
IMAGE_URLS+="https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/ubuntu/2023-09-29/ubuntu-2204-kube-v1.27.2.img,"
6666
IMAGE_URLS+="https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/ubuntu/2023-09-29/ubuntu-2204-kube-v1.28.2.img,"
67-
IMAGE_URLS+="https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/flatcar/flatcar-stable-3602.2.0-kube-v1.28.2.img"
67+
IMAGE_URLS+="https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/flatcar/flatcar-stable-3602.2.0-kube-v1.28.2.img,"
68+
IMAGE_URLS+="https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_openstack_image.img"
6869

6970
[[post-config|$NOVA_CONF]]
7071
[DEFAULT]
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
resources:
4+
- ../default
5+
6+
patches:
7+
- path: patch-flatcar.yaml
Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
---
2+
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha7
3+
kind: OpenStackCluster
4+
metadata:
5+
name: ${CLUSTER_NAME}
6+
spec:
7+
apiServerLoadBalancer:
8+
$patch: delete
9+
---
10+
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
11+
kind: KubeadmControlPlane
12+
metadata:
13+
name: "${CLUSTER_NAME}-control-plane"
14+
spec:
15+
replicas: ${CONTROL_PLANE_MACHINE_COUNT}
16+
kubeadmConfigSpec:
17+
joinConfiguration:
18+
nodeRegistration:
19+
name: $${COREOS_OPENSTACK_HOSTNAME}
20+
kubeletExtraArgs:
21+
provider-id: null
22+
initConfiguration:
23+
nodeRegistration:
24+
name: $${COREOS_OPENSTACK_HOSTNAME}
25+
kubeletExtraArgs:
26+
# Fixme(lentzi90): This is here just to override the value set in the default
27+
# kustomization. It will be replaced with a value that works for flatcar in
28+
# https://github.com/kubernetes-sigs/cluster-api-provider-openstack/pull/1564
29+
provider-id: null
30+
format: ignition
31+
ignition:
32+
containerLinuxConfig:
33+
additionalConfig: |
34+
storage:
35+
links:
36+
- path: /etc/extensions/kubernetes.raw
37+
hard: false
38+
target: /opt/extensions/kubernetes/kubernetes-${KUBERNETES_VERSION}-x86-64.raw
39+
files:
40+
- path: /etc/sysupdate.kubernetes.d/kubernetes-${KUBERNETES_VERSION%.*}.conf
41+
mode: 0644
42+
contents:
43+
remote:
44+
url: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${KUBERNETES_VERSION%.*}.conf
45+
- path: /etc/sysupdate.d/noop.conf
46+
mode: 0644
47+
contents:
48+
remote:
49+
url: https://github.com/flatcar/sysext-bakery/releases/download/latest/noop.conf
50+
- path: /opt/extensions/kubernetes/kubernetes-${KUBERNETES_VERSION}-x86-64.raw
51+
contents:
52+
remote:
53+
url: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${KUBERNETES_VERSION}-x86-64.raw
54+
systemd:
55+
units:
56+
- name: systemd-sysupdate.service
57+
dropins:
58+
- name: kubernetes.conf
59+
contents: |
60+
[Service]
61+
ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes"
62+
ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update
63+
ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new"
64+
ExecStartPost=/usr/bin/sh -c "if ! cmp --silent /tmp/kubernetes /tmp/kubernetes-new; then touch /run/reboot-required; fi"
65+
- name: update-engine.service
66+
# Set this to 'false' if you want to enable Flatcar auto-update
67+
mask: ${FLATCAR_DISABLE_AUTO_UPDATE:=true}
68+
- name: locksmithd.service
69+
# NOTE: To coordinate the node reboot in this context, we recommend to use Kured.
70+
mask: true
71+
- name: systemd-sysupdate.timer
72+
# Set this to 'true' if you want to enable the Kubernetes auto-update.
73+
# NOTE: Only patches version will be pulled.
74+
enabled: false
75+
76+
enabled: true
77+
- name: kubeadm.service
78+
enabled: true
79+
dropins:
80+
- name: 10-flatcar.conf
81+
contents: |
82+
[Unit]
83+
Requires=containerd.service coreos-metadata.service
84+
After=containerd.service coreos-metadata.service
85+
[Service]
86+
EnvironmentFile=/run/metadata/flatcar
87+
preKubeadmCommands:
88+
- export COREOS_OPENSTACK_HOSTNAME=$${COREOS_OPENSTACK_HOSTNAME%.*}
89+
- envsubst < /etc/kubeadm.yml > /etc/kubeadm.yml.tmp
90+
- mv /etc/kubeadm.yml.tmp /etc/kubeadm.yml
91+
---
92+
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
93+
kind: KubeadmConfigTemplate
94+
metadata:
95+
name: ${CLUSTER_NAME}-md-0
96+
spec:
97+
template:
98+
spec:
99+
joinConfiguration:
100+
nodeRegistration:
101+
name: $${COREOS_OPENSTACK_HOSTNAME}
102+
kubeletExtraArgs:
103+
# Fixme(lentzi90): This is here just to override the value set in the default
104+
# kustomization. It will be replaced with a value that works for flatcar in
105+
# https://github.com/kubernetes-sigs/cluster-api-provider-openstack/pull/1564
106+
provider-id: null
107+
preKubeadmCommands:
108+
- export COREOS_OPENSTACK_HOSTNAME=$${COREOS_OPENSTACK_HOSTNAME%.*}
109+
- envsubst < /etc/kubeadm.yml > /etc/kubeadm.yml.tmp
110+
- mv /etc/kubeadm.yml.tmp /etc/kubeadm.yml
111+
format: ignition
112+
ignition:
113+
containerLinuxConfig:
114+
additionalConfig: |
115+
storage:
116+
links:
117+
- path: /etc/extensions/kubernetes.raw
118+
hard: false
119+
target: /opt/extensions/kubernetes/kubernetes-${KUBERNETES_VERSION}-x86-64.raw
120+
files:
121+
- path: /etc/sysupdate.kubernetes.d/kubernetes-${KUBERNETES_VERSION%.*}.conf
122+
mode: 0644
123+
contents:
124+
remote:
125+
url: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${KUBERNETES_VERSION%.*}.conf
126+
- path: /etc/sysupdate.d/noop.conf
127+
mode: 0644
128+
contents:
129+
remote:
130+
url: https://github.com/flatcar/sysext-bakery/releases/download/latest/noop.conf
131+
- path: /opt/extensions/kubernetes/kubernetes-${KUBERNETES_VERSION}-x86-64.raw
132+
contents:
133+
remote:
134+
url: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${KUBERNETES_VERSION}-x86-64.raw
135+
systemd:
136+
units:
137+
- name: systemd-sysupdate.service
138+
dropins:
139+
- name: kubernetes.conf
140+
contents: |
141+
[Service]
142+
ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes"
143+
ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update
144+
ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new"
145+
ExecStartPost=/usr/bin/sh -c "if ! cmp --silent /tmp/kubernetes /tmp/kubernetes-new; then touch /run/reboot-required; fi"
146+
- name: update-engine.service
147+
# Set this to 'false' if you want to enable Flatcar auto-update
148+
mask: ${FLATCAR_DISABLE_AUTO_UPDATE:=true}
149+
- name: locksmithd.service
150+
# NOTE: To coordinate the node reboot in this context, we recommend to use Kured.
151+
mask: true
152+
- name: systemd-sysupdate.timer
153+
# Set this to 'true' if you want to enable the Kubernetes auto-update.
154+
# NOTE: Only patches version will be pulled.
155+
enabled: false
156+
157+
enabled: true
158+
- name: kubeadm.service
159+
enabled: true
160+
dropins:
161+
- name: 10-flatcar.conf
162+
contents: |
163+
[Unit]
164+
Requires=containerd.service coreos-metadata.service
165+
After=containerd.service coreos-metadata.service
166+
[Service]
167+
EnvironmentFile=/run/metadata/flatcar
168+
---
169+
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha7
170+
kind: OpenStackMachineTemplate
171+
metadata:
172+
name: ${CLUSTER_NAME}-md-0
173+
spec:
174+
template:
175+
spec:
176+
image: ${FLATCAR_IMAGE_NAME}
177+
---
178+
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha7
179+
kind: OpenStackMachineTemplate
180+
metadata:
181+
name: ${CLUSTER_NAME}-control-plane
182+
spec:
183+
template:
184+
spec:
185+
image: ${FLATCAR_IMAGE_NAME}

0 commit comments

Comments
 (0)