Skip to content

Commit f9c2f95

Browse files
dkoshkindkoshkin
authored
feat: add registry addon (#1116)
**What problem does this PR solve?**: This PR adds a new addon `registryMirror` that deploys https://github.com/distribution/distribution as a StatefulSet and a [sidecar container to sync images](https://github.com/regclient/regclient/blob/main/docs/regsync.md) across instances. ``` $ kubectl get pods -n registry-system NAME READY STATUS RESTARTS AGE cncf-distribution-registry-docker-registry-0 2/2 Running 0 2m13s cncf-distribution-registry-docker-registry-1 2/2 Running 0 8s ``` This addon is designed to only be a mirror and not used a regular registry, hence the name and the lack of external access to the Service. In a follow up PR, the in-cluster Service will be used a Containerd mirror. In a follow up PR, it will also be deployed with randomly generated credentials to further prevent direct use. **Which issue(s) this PR fixes**: Fixes # **How Has This Been Tested?**: <!-- Please describe the tests that you ran to verify your changes. Provide output from the tests and any manual steps needed to replicate the tests. --> **Special notes for your reviewer**: <!-- Use this to provide any additional information to the reviewers. This may include: - Best way to review the PR. - Where the author wants the most review attention on. - etc. -->
1 parent 1829819 commit f9c2f95

34 files changed

+781
-13
lines changed

‎api/v1alpha1/addon_types.go

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,8 @@ const (
2727

2828
ServiceLoadBalancerProviderMetalLB = "MetalLB"
2929

30+
RegistryProviderCNCFDistribution = "CNCF Distribution"
31+
3032
AddonStrategyClusterResourceSet AddonStrategy = "ClusterResourceSet"
3133
AddonStrategyHelmAddon AddonStrategy = "HelmAddon"
3234

@@ -100,6 +102,9 @@ type GenericAddons struct {
100102

101103
// +kubebuilder:validation:Optional
102104
ServiceLoadBalancer *ServiceLoadBalancer `json:"serviceLoadBalancer,omitempty"`
105+
106+
// +kubebuilder:validation:Optional
107+
Registry *RegistryAddon `json:"registry,omitempty"`
103108
}
104109

105110
type AddonStrategy string
@@ -335,3 +340,10 @@ type AddressRange struct {
335340
// +kubebuilder:validation:Format=ipv4
336341
End string `json:"end"`
337342
}
343+
344+
type RegistryAddon struct {
345+
// The OCI registry provider to deploy.
346+
// +kubebuilder:default="CNCF Distribution"
347+
// +kubebuilder:validation:Enum="CNCF Distribution"
348+
Provider string `json:"provider"`
349+
}

‎api/v1alpha1/constants.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@ const (
2828
ClusterAutoscalerVariableName = "clusterAutoscaler"
2929
// ServiceLoadBalancerVariableName is the Service LoadBalancer config patch variable name.
3030
ServiceLoadBalancerVariableName = "serviceLoadBalancer"
31+
// RegistryAddonVariableName is the OCI registry config patch variable name.
32+
RegistryAddonVariableName = "registry"
3133

3234
// GlobalMirrorVariableName is the global image registry mirror patch variable name.
3335
GlobalMirrorVariableName = "globalImageRegistryMirror"

‎api/v1alpha1/crds/caren.nutanix.com_awsclusterconfigs.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -234,6 +234,17 @@ spec:
234234
- HelmAddon
235235
type: string
236236
type: object
237+
registry:
238+
properties:
239+
provider:
240+
default: CNCF Distribution
241+
description: The OCI registry provider to deploy.
242+
enum:
243+
- CNCF Distribution
244+
type: string
245+
required:
246+
- provider
247+
type: object
237248
serviceLoadBalancer:
238249
properties:
239250
configuration:

‎api/v1alpha1/crds/caren.nutanix.com_dockerclusterconfigs.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -243,6 +243,17 @@ spec:
243243
- HelmAddon
244244
type: string
245245
type: object
246+
registry:
247+
properties:
248+
provider:
249+
default: CNCF Distribution
250+
description: The OCI registry provider to deploy.
251+
enum:
252+
- CNCF Distribution
253+
type: string
254+
required:
255+
- provider
256+
type: object
246257
serviceLoadBalancer:
247258
properties:
248259
configuration:

‎api/v1alpha1/crds/caren.nutanix.com_nutanixclusterconfigs.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -243,6 +243,17 @@ spec:
243243
- HelmAddon
244244
type: string
245245
type: object
246+
registry:
247+
properties:
248+
provider:
249+
default: CNCF Distribution
250+
description: The OCI registry provider to deploy.
251+
enum:
252+
- CNCF Distribution
253+
type: string
254+
required:
255+
- provider
256+
type: object
246257
serviceLoadBalancer:
247258
properties:
248259
configuration:

‎api/v1alpha1/zz_generated.deepcopy.go

Lines changed: 20 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎charts/cluster-api-runtime-extensions-nutanix/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,8 @@ A Helm chart for cluster-api-runtime-extensions-nutanix
8585
| hooks.nfd.crsStrategy.defaultInstallationConfigMap.name | string | `"node-feature-discovery"` | |
8686
| hooks.nfd.helmAddonStrategy.defaultValueTemplateConfigMap.create | bool | `true` | |
8787
| hooks.nfd.helmAddonStrategy.defaultValueTemplateConfigMap.name | string | `"default-nfd-helm-values-template"` | |
88+
| hooks.registry.cncfDistribution.defaultValueTemplateConfigMap.create | bool | `true` | |
89+
| hooks.registry.cncfDistribution.defaultValueTemplateConfigMap.name | string | `"default-cncf-distribution-registry-helm-values-template"` | |
8890
| hooks.serviceLoadBalancer.metalLB.defaultValueTemplateConfigMap.create | bool | `true` | |
8991
| hooks.serviceLoadBalancer.metalLB.defaultValueTemplateConfigMap.name | string | `"default-metallb-helm-values-template"` | |
9092
| hooks.virtualIP.kubeVip.defaultTemplateConfigMap.create | bool | `true` | |

‎charts/cluster-api-runtime-extensions-nutanix/addons/registry/cncf-distribution/values-template.yaml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
replicaCount: 2
2+
persistence:
3+
enabled: true
4+
size: 50Gi
5+
service:
6+
type: ClusterIP
7+
clusterIP: {{ .ServiceIP }}
8+
port: 80
9+
statefulSet:
10+
enabled: true
11+
syncer:
12+
interval: 2m

‎charts/cluster-api-runtime-extensions-nutanix/templates/helm-config.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,10 @@ data:
2323
ChartName: cluster-autoscaler
2424
ChartVersion: 9.46.3
2525
RepositoryURL: '{{ if .Values.helmRepository.enabled }}oci://helm-repository.{{ .Release.Namespace }}.svc/charts{{ else }}https://kubernetes.github.io/autoscaler{{ end }}'
26+
cncf-distribution-registry: |
27+
ChartName: docker-registry
28+
ChartVersion: 2.3.1
29+
RepositoryURL: '{{ if .Values.helmRepository.enabled }}oci://helm-repository.{{ .Release.Namespace }}.svc/charts{{ else }}https://mesosphere.github.io/charts/staging/{{ end }}'
2630
cosi-controller: |
2731
ChartName: cosi
2832
ChartVersion: 0.0.1-alpha.5

‎charts/cluster-api-runtime-extensions-nutanix/templates/registry/distribution/helm-addon-installation.yaml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Copyright 2025 Nutanix. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
{{- if .Values.hooks.registry.cncfDistribution.defaultValueTemplateConfigMap.name }}
5+
apiVersion: v1
6+
kind: ConfigMap
7+
metadata:
8+
name: '{{ .Values.hooks.registry.cncfDistribution.defaultValueTemplateConfigMap.name }}'
9+
data:
10+
values.yaml: |-
11+
{{- .Files.Get "addons/registry/cncf-distribution/values-template.yaml" | nindent 4 }}
12+
{{- end -}}

‎charts/cluster-api-runtime-extensions-nutanix/values.schema.json

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -504,6 +504,27 @@
504504
},
505505
"type": "object"
506506
},
507+
"registry": {
508+
"properties": {
509+
"cncfDistribution": {
510+
"properties": {
511+
"defaultValueTemplateConfigMap": {
512+
"properties": {
513+
"create": {
514+
"type": "boolean"
515+
},
516+
"name": {
517+
"type": "string"
518+
}
519+
},
520+
"type": "object"
521+
}
522+
},
523+
"type": "object"
524+
}
525+
},
526+
"type": "object"
527+
},
507528
"serviceLoadBalancer": {
508529
"properties": {
509530
"metalLB": {

‎charts/cluster-api-runtime-extensions-nutanix/values.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,11 @@ hooks:
111111
defaultValueTemplateConfigMap:
112112
create: true
113113
name: default-cosi-controller-helm-values-template
114+
registry:
115+
cncfDistribution:
116+
defaultValueTemplateConfigMap:
117+
create: true
118+
name: default-cncf-distribution-registry-helm-values-template
114119

115120
helmAddonsConfigMap: default-helm-addons-config
116121

‎docs/content/addons/registry.md

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
+++
2+
title = "Registry"
3+
icon = "fa-solid fa-eye"
4+
+++
5+
6+
By leveraging CAPI cluster lifecycle hooks, this handler deploys an OCI [Distribution] registry,
7+
at the `AfterControlPlaneInitialized` phase and configures it as a mirror on the new cluster.
8+
The registry will be deployed as a StatefulSet with a persistent volume claim for storage
9+
and multiple replicas for high availability.
10+
A sidecar container in each Pod running [Regsync] will periodically sync the OCI artifacts across all replicas.
11+
12+
Deployment of this registry is opt-in via the [provider-specific cluster configuration]({{< ref ".." >}}).
13+
14+
The hook will use the [Cluster API Add-on Provider for Helm] to deploy the registry resources.
15+
16+
## Example
17+
18+
To enable deployment of the registry on a cluster, specify the following values:
19+
20+
```yaml
21+
apiVersion: cluster.x-k8s.io/v1beta1
22+
kind: Cluster
23+
metadata:
24+
name: <NAME>
25+
spec:
26+
topology:
27+
variables:
28+
- name: clusterConfig
29+
value:
30+
addons:
31+
registry: {}
32+
```
33+
34+
[Distribution]: https://github.com/distribution/distribution
35+
[Cluster API Add-on Provider for Helm]: https://github.com/kubernetes-sigs/cluster-api-addon-provider-helm
36+
[Regsync]: https://regclient.org/usage/regsync/

‎examples/capi-quick-start/docker-cluster-calico-crs.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ spec:
4141
strategy: ClusterResourceSet
4242
nfd:
4343
strategy: ClusterResourceSet
44+
registry: {}
4445
serviceLoadBalancer:
4546
configuration:
4647
addressRanges:

‎examples/capi-quick-start/docker-cluster-calico-helm-addon.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ spec:
3636
default: {}
3737
snapshotController: {}
3838
nfd: {}
39+
registry: {}
3940
serviceLoadBalancer:
4041
configuration:
4142
addressRanges:

‎examples/capi-quick-start/docker-cluster-cilium-crs.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ spec:
4141
strategy: ClusterResourceSet
4242
nfd:
4343
strategy: ClusterResourceSet
44+
registry: {}
4445
serviceLoadBalancer:
4546
configuration:
4647
addressRanges:

‎examples/capi-quick-start/docker-cluster-cilium-helm-addon.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ spec:
3636
default: {}
3737
snapshotController: {}
3838
nfd: {}
39+
registry: {}
3940
serviceLoadBalancer:
4041
configuration:
4142
addressRanges:

‎hack/addons/helm-chart-bundler/repos.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,11 @@ repositories:
3131
charts:
3232
cosi:
3333
- 0.0.1-alpha.5
34+
docker-registry:
35+
repoURL: https://mesosphere.github.io/charts/staging/
36+
charts:
37+
docker-registry:
38+
- 2.3.1
3439
local-path-provisioner:
3540
repoURL: https://charts.containeroo.ch
3641
charts:

‎hack/addons/kustomize/cncf-distribution-registry/kustomization.yaml.tmpl

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Copyright 2025 Nutanix. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
# NOTE This file is used by the tool in `hack/tools/helm-cm` to add
5+
# docker-registry chart metadata to the "helm-addons" ConfigMap. The tool takes
6+
# a kustomization as input. We do not use this file with kustomize.
7+
8+
apiVersion: kustomize.config.k8s.io/v1beta1
9+
kind: Kustomization
10+
11+
metadata:
12+
name: registry-distribution
13+
14+
sortOptions:
15+
order: fifo
16+
17+
helmCharts:
18+
- name: docker-registry
19+
repo: https://mesosphere.github.io/charts/staging/
20+
releaseName: cncf-distribution-registry
21+
version: 2.3.1
22+
valuesFile: helm-values.yaml
23+
includeCRDs: true
24+
skipTests: true
25+
namespace: registry-system
26+
27+
namespace: registry-system

‎hack/examples/bases/docker/cluster/kustomization.yaml.tmpl

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,10 @@ patches:
4040
path: ../../../patches/docker/csi.yaml
4141
- target:
4242
kind: Cluster
43-
path: ../../../patches/nutanix/cosi.yaml
43+
path: ../../../patches/docker/cosi.yaml
44+
- target:
45+
kind: Cluster
46+
path: ../../../patches/docker/registry.yaml
4447
- target:
4548
kind: Cluster
4649
path: ../../../patches/encryption.yaml

‎hack/examples/patches/docker/registry.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# Copyright 2025 Nutanix. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
- op: "add"
5+
path: "/spec/topology/variables/0/value/addons/registry"
6+
value: {}

‎pkg/handlers/generic/lifecycle/config/cm.go

Lines changed: 13 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -17,18 +17,19 @@ import (
1717
type Component string
1818

1919
const (
20-
Autoscaler Component = "cluster-autoscaler"
21-
Tigera Component = "tigera-operator"
22-
Cilium Component = "cilium"
23-
NFD Component = "nfd"
24-
NutanixStorageCSI Component = "nutanix-storage-csi"
25-
SnapshotController Component = "snapshot-controller"
26-
NutanixCCM Component = "nutanix-ccm"
27-
MetalLB Component = "metallb"
28-
LocalPathProvisionerCSI Component = "local-path-provisioner-csi"
29-
AWSEBSCSI Component = "aws-ebs-csi"
30-
AWSCCM Component = "aws-ccm"
31-
COSIController Component = "cosi-controller"
20+
Autoscaler Component = "cluster-autoscaler"
21+
Tigera Component = "tigera-operator"
22+
Cilium Component = "cilium"
23+
NFD Component = "nfd"
24+
NutanixStorageCSI Component = "nutanix-storage-csi"
25+
SnapshotController Component = "snapshot-controller"
26+
NutanixCCM Component = "nutanix-ccm"
27+
MetalLB Component = "metallb"
28+
LocalPathProvisionerCSI Component = "local-path-provisioner-csi"
29+
AWSEBSCSI Component = "aws-ebs-csi"
30+
AWSCCM Component = "aws-ccm"
31+
COSIController Component = "cosi-controller"
32+
CNCFDistributionRegistry Component = "cncf-distribution-registry"
3233
)
3334

3435
type HelmChartGetter struct {

0 commit comments

Comments
 (0)