Skip to content

fix: issues with mindthegap deplotment #660

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
May 16, 2024
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ jobs:
name: Run chart-testing (lint)
run: |
devbox run -- \
ct lint --config charts/ct-config.yaml
ct lint --namespace=caren-system --config charts/ct-config.yaml

- if: steps.list-changed.outputs.changed == 'true'
name: Create kind cluster
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ export KUBERNETES_VERSION=v1.28.7
clusterctl generate cluster ${CLUSTER_NAME} \
--from ${CLUSTER_FILE} \
--kubernetes-version ${KUBERNETES_VERSION} \
-n caren-system \
--worker-machine-count 1 | \
kubectl apply --server-side -f -
```
Expand Down
6 changes: 6 additions & 0 deletions charts/cluster-api-runtime-extensions-nutanix/Chart.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
dependencies:
- name: trust-manager
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not keen on trust-manager being a requirement for CAREN - shouldn't this be a user deployment configuration rather than a strict requirement? It could conflict with existing trust-manager deployment for example.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can keep this functionality until we implement #497

We might be able to then set this behind a boolean gate. If that is the case, i'm not sure what the implication for konvoy would be. How should we template it?

On the other hand, CAREN is already a highly opinionated and has a ton of dependencies adding another isn't that big of a deal IMO

repository: https://charts.jetstack.io
version: v0.10.0
digest: sha256:58f4f9fd7f71f972e9344c62c1f61978ab19cc174d8854c9d5de7da83eee2049
generated: "2024-05-13T12:35:00.623992598-06:00"
4 changes: 4 additions & 0 deletions charts/cluster-api-runtime-extensions-nutanix/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ maintainers:
url: https://eng.d2iq.com
sources:
- https://github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix
dependencies:
- name: trust-manager
version: v0.10.0
repository: https://charts.jetstack.io

appVersion: v0.0.0-dev
version: v0.0.0-dev
7 changes: 7 additions & 0 deletions charts/cluster-api-runtime-extensions-nutanix/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@ A Helm chart for cluster-api-runtime-extensions-nutanix

* <https://github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix>

## Requirements

| Repository | Name | Version |
|------------|------|---------|
| https://charts.jetstack.io | trust-manager | v0.10.0 |

## Values

| Key | Type | Default | Description |
Expand Down Expand Up @@ -84,3 +90,4 @@ A Helm chart for cluster-api-runtime-extensions-nutanix
| service.port | int | `443` | |
| service.type | string | `"ClusterIP"` | |
| tolerations | list | `[{"effect":"NoSchedule","key":"node-role.kubernetes.io/master","operator":"Equal"},{"effect":"NoSchedule","key":"node-role.kubernetes.io/control-plane","operator":"Equal"}]` | Kubernetes pod tolerations |
| trust-manager.app.trust.namespace | string | `"caren-system"` | |
Binary file not shown.

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: aws
name: aws-quick-start
namespace: caren-system
spec:
controlPlane:
machineInfrastructure:
Expand Down Expand Up @@ -64,6 +65,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: aws
name: aws-quick-start
namespace: caren-system
spec:
template:
spec: {}
Expand All @@ -74,6 +76,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: aws
name: aws-quick-start-control-plane
namespace: caren-system
spec:
template:
spec:
Expand Down Expand Up @@ -102,6 +105,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: aws
name: aws-quick-start-control-plane
namespace: caren-system
spec:
template:
spec:
Expand All @@ -114,6 +118,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: aws
name: aws-quick-start-worker-machinetemplate
namespace: caren-system
spec:
template:
spec:
Expand All @@ -126,6 +131,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: aws
name: aws-quick-start-worker-bootstraptemplate
namespace: caren-system
spec:
template:
spec:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: docker
name: docker-quick-start
namespace: caren-system
spec:
controlPlane:
machineInfrastructure:
Expand Down Expand Up @@ -50,6 +51,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: docker
name: docker-quick-start-cluster
namespace: caren-system
spec:
template:
spec: {}
Expand All @@ -60,6 +62,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: docker
name: docker-quick-start-control-plane
namespace: caren-system
spec:
template:
spec:
Expand All @@ -76,6 +79,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: docker
name: docker-quick-start-control-plane
namespace: caren-system
spec:
template:
spec:
Expand All @@ -89,6 +93,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: docker
name: docker-quick-start-default-worker-machinetemplate
namespace: caren-system
spec:
template:
spec:
Expand All @@ -102,6 +107,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: docker
name: docker-quick-start-default-worker-bootstraptemplate
namespace: caren-system
spec:
template:
spec:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: nutanix
name: nutanix-quick-start-kcfg-0
namespace: caren-system
spec:
template:
spec:
Expand All @@ -26,6 +27,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: nutanix
name: nutanix-quick-start
namespace: caren-system
spec:
controlPlane:
machineHealthCheck:
Expand Down Expand Up @@ -116,6 +118,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: nutanix
name: nutanix-quick-start-kcpt
namespace: caren-system
spec:
template:
spec:
Expand Down Expand Up @@ -164,6 +167,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: nutanix
name: nutanix-quick-start-nct
namespace: caren-system
spec:
template:
spec:
Expand All @@ -185,6 +189,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: nutanix
name: nutanix-quick-start-cp-nmt
namespace: caren-system
spec:
template:
spec:
Expand All @@ -210,6 +215,7 @@ metadata:
labels:
cluster.x-k8s.io/provider: nutanix
name: nutanix-quick-start-md-nmt
namespace: caren-system
spec:
template:
spec:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Copyright 2024 Nutanix. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
name: caren-bundle
spec:
sources:
- useDefaultCAs: true
- secret:
key: tls.crt
name: mindthegap-tls
target:
configMap:
key: ca-certificates.crt
namespaceSelector:
matchLabels:
clusterctl.cluster.x-k8s.io: ""
Original file line number Diff line number Diff line change
Expand Up @@ -10,36 +10,37 @@ data:
cilium: |
ChartName: cilium
ChartVersion: 1.15.0
RepositoryURL: oci://mindthegap.{{ .Release.Namespace }}.svc
RepositoryURL: oci://mindthegap.caren-system.svc/charts
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't work if deployed in a different namespace?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that is correct. it needs to be deployed in that namespace, this is a limitation of trust-manager needing trust namespace set. this comment explains it

cert-manager/trust-manager#120 (comment)

if helm was able to use templating in values.yaml we could get around this

cluster-autoscaler: |
ChartName: cluster-autoscaler
ChartVersion: 9.35.0
RepositoryURL: oci://mindthegap.{{ .Release.Namespace }}.svc
RepositoryURL: oci://mindthegap.caren-system.svc/charts
metallb: |
ChartName: metallb
ChartVersion: v0.14.5
RepositoryURL: oci://mindthegap.{{ .Release.Namespace }}.svc
RepositoryURL: oci://mindthegap.caren-system.svc/charts
nfd: |
ChartName: node-feature-discovery
ChartVersion: v0.15.2
RepositoryURL: oci://mindthegap.{{ .Release.Namespace }}.svc
ChartVersion: 0.15.2
RepositoryURL: oci://mindthegap.caren-system.svc/charts
nutanix-ccm: |
ChartName: nutanix-cloud-provider
ChartVersion: 0.3.3
RepositoryURL: oci://mindthegap.{{ .Release.Namespace }}.svc
RepositoryURL: oci://mindthegap.caren-system.svc/charts
nutanix-snapshot-csi: |
ChartName: nutanix-csi-snapshot
ChartVersion: 6.3.2
RepositoryURL: oci://mindthegap.{{ .Release.Namespace }}.svc
RepositoryURL: oci://mindthegap.caren-system.svc/charts
nutanix-storage-csi: |
ChartName: nutanix-csi-storage
ChartVersion: v3.0.0-beta.1912
RepositoryURL: oci://mindthegap.{{ .Release.Namespace }}.svc
ChartVersion: 3.0.0-beta.1912
RepositoryURL: oci://mindthegap.caren-system.svc/charts
tigera-operator: |
ChartName: tigera-operator
ChartVersion: v3.26.4
RepositoryURL: oci://mindthegap.{{ .Release.Namespace }}.svc
RepositoryURL: oci://mindthegap.caren-system.svc/charts
kind: ConfigMap
metadata:
creationTimestamp: null
name: {{ .Values.helmAddonsConfigMap }}
namespace: caren-system
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Copyright 2024 Nutanix. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: helm-hook-role
namespace: caaph-system
rules:
- apiGroups:
- apps
resources:
- deployments
verbs:
- get
- list
- patch
- update
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Copyright 2024 Nutanix. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: helm-hook-binding
namespace: caaph-system
subjects:
- kind: ServiceAccount
name: helm-hook-sa
namespace: caaph-system
roleRef:
kind: Role
name: helm-hook-role
apiGroup: rbac.authorization.k8s.io
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Copyright 2024 Nutanix. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

apiVersion: v1
kind: ServiceAccount
metadata:
name: helm-hook-sa
namespace: caaph-system
66 changes: 66 additions & 0 deletions charts/cluster-api-runtime-extensions-nutanix/templates/job.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Copyright 2024 Nutanix. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
apiVersion: batch/v1
kind: Job
metadata:
name: update-trust-manager-namespace
namespace: caaph-system
spec:
template:
spec:
serviceAccountName: helm-hook-sa
containers:
- name: kubectl
image: bitnami/kubectl:latest
command:
- /bin/sh
- -c
- |
set -e
# Check if volume already exists
if kubectl get deployment caaph-controller-manager -n caaph-system -o jsonpath='{.spec.template.spec.volumes[?(@.name=="ca-certificate-only")]}' | grep -q "ca-certificate-only"; then
echo "Volume already exists. Skipping addition."
else
kubectl patch deployment caaph-controller-manager \
-n caaph-system \
--type=json \
-p='[
{
"op": "add",
"path": "/spec/template/spec/volumes/-",
"value": {
"name": "ca-certificate-only",
"configMap": {
"name": "caren-bundle",
"defaultMode": 420,
"optional": false,
"items": [
{
"key": "ca-certificates.crt",
"path": "ca-certificates.crt"
}
]
}
}
}
]'
fi
if kubectl get deployment caaph-controller-manager -n caaph-system -o jsonpath='{.spec.template.spec.containers[0].volumeMounts[?(@.mountPath=="/etc/ssl/certs/")]}' | grep -q "ca-certificate-only"; then
echo "VolumeMount already exists. Skipping addition."
else
kubectl patch deployment caaph-controller-manager \
-n caaph-system \
--type=json \
-p='[
{
"op": "add",
"path": "/spec/template/spec/containers/0/volumeMounts/-",
"value": {
"mountPath": "/etc/ssl/certs/",
"name": "ca-certificate-only",
"readOnly": true
}
}
]'
fi
restartPolicy: Never
Loading
Loading