fix: Correctly configure non-mirror registry certificates

[jimmidyson]

This PR updates the runtime extensions to include two mutation handlers for Nutanix clusters:

• nutanixclusterconfigpatch-gp.cluster-api-runtime-extensions-nutanix - original name and original (buggy functionality). This will still be referenced from existing CCs and so must retain the same name.
• nutanixclusterv2configpatch-gp.cluster-api-runtime-extensions-nutanix - new name with fixed functionality. This is now referenced from the example CC in this project and can be used by existing clusters by rebasing on a CC that references this handler name.

The new handler fixes are:

• CA certificates are now written to /etc/containerd/certs.d/<registryHost>/ca.crt
  as per https://github.com/containerd/containerd/blob/main/docs/hosts.md#support-for-dockers-certificate-file-pattern. This does not require a hosts.toml file for configuration, although that explicit configuration may be preferred (can do in future if so).
• Remove non-mirror registry config from /etc/containerd/certs.d/_default/hosts.toml
  which was causing all registries to be configured as mirror registry.

[dlipovetsky]

Thanks!

I think this makes

cluster-api-runtime-extensions-nutanix/pkg/handlers/generic/mutation/mirrors/inject.go

Lines 264 to 266 in 17bc80d

	// This handler reads input from two user provided variables: globalImageRegistryMirror and imageRegistries.
	// The handler will be used to either add configuration for a global mirror or CA certificates for image registries.
	func needContainerdConfiguration(configs []containerdConfig) bool {

unused, and I don't think there's a future need for it.

If that's correct, can I can push a commit that removes it, and related test code?

[jimmidyson]

@dlipovetsky

If that's correct, can I can push a commit that removes it, and related test code?

No this is still used in https://github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/blob/jimmi/fix-registry-ca-certs/pkg/handlers/generic/mutation/mirrors/inject.go#L124-L131

[dkoshkin]

I configured my test docker cluster like:

      value:
        imageRegistries:
          - url: https://172.18.0.10:5000
            credentials:
              secretRef:
                name: local-registry
          - url: https://docker.io
            credentials:
              secretRef:
                name: dockerhub

And the resulting KCP includes an extra file:

      - content: |
          apiVersion: kubelet.config.k8s.io/v1
          kind: CredentialProviderConfig
          providers:
          - name: dynamic-credential-provider
            args:
            - get-credentials
            - -c
            - /etc/kubernetes/dynamic-credential-provider-config.yaml
            matchImages:
            - "172.18.0.10:5000"
            - "docker.io"
            - "*"
            - "*.*"
            - "*.*.*"
            - "*.*.*.*"
            - "*.*.*.*.*"
            - "*.*.*.*.*.*"
            defaultCacheDuration: "0s"
            apiVersion: credentialprovider.kubelet.k8s.io/v1
        path: /etc/kubernetes/image-credential-provider-config.yaml
        permissions: "0600"
      - content: |
          apiVersion: credentialprovider.d2iq.com/v1alpha1
          kind: DynamicCredentialProviderConfig
          credentialProviderPluginBinDir: /etc/kubernetes/image-credential-provider/
          credentialProviders:
            apiVersion: kubelet.config.k8s.io/v1
            kind: CredentialProviderConfig
            providers:
            - name: static-credential-provider
              args:
              - /etc/kubernetes/static-image-credentials.json
              matchImages:
              - "172.18.0.10:5000"
              defaultCacheDuration: "0s"
              apiVersion: credentialprovider.kubelet.k8s.io/v1
            - name: static-credential-provider
              args:
              - /etc/kubernetes/static-image-credentials.json
              matchImages:
              - "docker.io"
              defaultCacheDuration: "0s"
              apiVersion: credentialprovider.kubelet.k8s.io/v1
        path: /etc/kubernetes/dynamic-credential-provider-config.yaml
        permissions: "0600"
      - contentFrom:
          secret:
            key: static-credential-provider
            name: dkoshkin-mutliple-registires-static-credential-provider-response
        path: /etc/kubernetes/static-image-credentials.json
        permissions: "0600"
      - content: |2+

        path: /etc/containerd/certs.d/_default/hosts.toml
        permissions: "0600"
      - contentFrom:
          secret:
            key: ca.crt
            name: local-registry
        path: /etc/containerd/certs.d/172.18.0.10:5000/ca.crt
        permissions: "0600"

(still trying to actually test and pull images)

[dlipovetsky]

This is confirmed to work for a single configured registry. There are issues with two or more configured registries, but we'll address that in a follow-up PR.

[dlipovetsky]

@jimmidyson Can you update the PR description, please? I think we'll have two different patches (nutanixclusterconfigpatch-gp.cluster-api-runtime-extensions-nutanix and nutanixclusterv2configpatch-gp.cluster-api-runtime-extensions-nutanix)?

[jimmidyson]

Updated PR description.

[dlipovetsky]

I verified the configuration, and functionality using a Docker cluster.