Create Helm chart as part of the build process

[rptaylor]

Hello,

Do you plan to provide a Helm chart for HNC installation? This would make it easier for operators to manage HNC installations.

Thanks!

[adrianludwin]

I'm certainly open to it but I don't really use Helm myself. HNC doesn't really have any options - how would a Helm chart be better than a simple manifest?

Would you be willing to make a PR to add one? If so, I think the way to do it would be similar to how we create the Krew manifest:

• Create the chart file in the hack/ directory (like the krew version)
• Add a target to the makefile to generate the chart (like this)
• Add a stage to the Cloud Build process to upload the chart (like this)

[rptaylor]

A helm chart is useful because it bundles all the components together and allows operations (list, upgrade, uninstall etc.) to be performed on them in aggregate. Currently with one big YAML file, you can install all the components at once with kubectl apply, but then you are left with a slew of objects and resources in your cluster that can not be cohesively managed. How would you uninstall HNC? Currently I think there would be no clean way to do this. You could delete the HNC namespace, but you'd still be left with all the un-namespaced objects like CRDs and ClusterRoles.

Also kubectl apply can not apply negative configuration, so if there are some major changes in a new HNC version say from 0.9 to 1.0, and there are old resource definitions in hnc-manager-0.9.yaml that are not used anymore in hnc-manager-1.0.yaml , they will nevertheless remain in your cluster when you kubectl apply -f hnc-manager-1.0.yaml instead of being removed, adding clutter over time. Once you include something in the HNC YAML file, you will never be able to remove it from the cluster deployments (unless you tell cluster operators to manually kubectl delete things when they upgrade HNC). With Helm, the HNC deployment would be managed so the environment would always be clean, with no leftover resources from previous installations of HNC versions.

Even if there are no configurable HNC options ,

• will that really always be the case?
• cluster operators may nevertheless wish to customize some aspects of deployment that are not HNC-specific, but pertain to scaling, production use , or security practices, e.g. set a priorityClass for important cluster apps, increase number of replicas or resource reqs/limits, maybe select their own locally deployed registry mirror to pull images from, apply a PodSecurityPolicy, etc.

Helm provides the flexibility to do that.

I have made a basic Helm chart and might be able to work the HNC YAML into a Helm chart, but I am not sure if it is that simple in this case. It looks like there is a "build" mechanism that uses a makefile to process the YAML with kustomize to produce the output hnc-manager.yaml ?
https://github.com/kubernetes-sigs/hierarchical-namespaces/blob/master/Makefile#L138

Normally with Helm I think you would just maintain the YAML templates and chart directly in git , and publish them as they are to a chart repo server, without needing to process or "build" them. There would be no need to combine all the k8s resources into a single YAML file. What is the role of kustomize in the build process?

[rptaylor]

I'm not sure if it would make sense to add more processing in the Makefile to "unbuild" all the YAML into a Helm chart, when ideally another benefit of Helm would be to simplify the packaging and maintenance of YAML files for the developers and avoid having to do YAML processing. But I don't really know how the build process works, or if you think it would be practical to revamp it.

[rptaylor]

Noting https://github.com/kubernetes-sigs/multi-tenancy/blob/master/incubator/hnc/docs/user-guide/how-to.md#excluding-namespaces-from-hnc
operators might very well want the configurability to exclude additional namespaces:
https://github.com/kubernetes-sigs/multi-tenancy/blob/master/incubator/hnc/docs/user-guide/how-to.md#modify-command-line-arguments
This may remain true after HNC graduates to GA status.

[rptaylor]

It seems that HNC does have configurable options that operators may wish to modify, such as the HNCConfiguration object.
https://github.com/kubernetes-sigs/multi-tenancy/blob/master/incubator/hnc/docs/user-guide/how-to.md#uninstall-hnc-from-a-cluster
Wouldn't any modifications to HNCConfiguration be lost when upgrading HNC with kubectl apply?

With Helm, user-specified properties could easily be integrated together with the essential baseline properties , as values going into the template for the HNCConfiguration object, then applied cohesively in one step.

[adrianludwin]

Excluded namespaces are a good point, I could see you wanting to configure those in Helm. HNCConfiguration isn't really a good point since the default manifest doesn't come with a default copy of HNCConfiguration so it will never be overwritten. You can delete HNC today with kubectl delete -f <manifest> but I agree this is pretty clunky.

None of this is an argument against Helm, btw, just giving you more context so we can solve the problem well.

Kustomize has several roles, only one of which is to collate all the yaml files into one manifest. Some of the yaml files in /config are autogenerated by controller-tools, but they're not complete - they need additional processing, such as adding timeouts to the webhooks. Kustomize lets us apply these in a sane way. It also adds the "hnc-" prefix to all objects (I don't think controller-tools can generate those).

I had a look at what Gatekeeper was doing (like HNC, Gatekeeper uses controller-tools) and it looks like they have a tool to automatically generate their Helm charts from their kustomize output. At a glance, it appears to be a custom-built tool. Maybe we could adapt that? It wouldn't be the first thing we've stolen borrowed from Gatekeeper.

[rptaylor]

Hmm I see, where does the default HNCConfiguration object come from then, is it created by the controller itself if it does not exist already?

more context so we can solve the problem well

Indeed, appreciated.

Hmm I found other cases of people using kubebuilder (controller-tools) wanting to generate helm charts:
kubernetes-sigs/kubebuilder#388
kubernetes-sigs/kubebuilder#387 (comment)
but there did not seem to be a solution, aside from suggesting to use kustomize to create Helm charts (sounds possible but maybe clunky, I did not see any kustomize functionality which could facilitate that as far as I could find).

Interestingly, helm can invoke kustomize and kustomize can invoke Helm, but both cases appear to be ways to modify a deployment ("chart inflation"), not the reverse of turning YAML into a chart.

In principle I could make a Helm chart based on the current monolithic YAML file but I don't think it would be very useful or practical to do as a one-off manual effort. It seems there are broader considerations involved relating to the whole build process.

[adrianludwin]

Hmm I see, where does the default HNCConfiguration object come from then, is it created by the controller itself if it does not exist already?

Exactly - we report status there, so if the cluster doesn't already have an object we create a blank one. This is the same way we create a HierarchyConfiguration object in any namespace with a child, even if the user didn't create one themselves. But this will never overwrite user data.

Hmm I found other cases of people using kubebuilder (controller-tools) wanting to generate helm charts:
kubernetes-sigs/kubebuilder#388
kubernetes-sigs/kubebuilder#387 (comment)
but there did not seem to be a solution, aside from suggesting to use kustomize to create Helm charts (sounds possible but maybe clunky, I did not see any kustomize functionality which could facilitate that as far as I could find).

Yeah, that's what I found too :( Gatekeeper seems to have the only viable solution that I've seen, did you look into that?

In principle I could make a Helm chart based on the current monolithic YAML file but I don't think it would be very useful or practical to do as a one-off manual effort.

Agreed, but maybe the GK tool is worth adapting.

Quick Q on Helm: Gatekeeper on seems to keep one version of its Helm chart at a time. Is this typical for Helm? Or do they usually maintain charts for multiple releases?

[rptaylor]

Agreed, but maybe the GK tool is worth adapting.

Maybe. I had a brief look but I'm not a developer per se and don't know Go, so can't say much about helmify but would be nice if it could be used.

Well you could maintain or produce one version of the chart at a time, but then you would probably put it in a chart repo which can include all the previous charts too, like a yum repo. Setting up a basic chart repo is pretty simple, I just made my own served with GH Pages. But for a project like this one might want to automate it and incorporate it in the CI/CD process.

[adrianludwin]

Ok, I'll mark this as a good issue for new contributors, we've had a few people come through lately with interest in this area (e.g. @RealHarshThakur might be interested).

/good-first-issue

[k8s-ci-robot]

@adrianludwin:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

Ok, I'll mark this as a good issue for new contributors, we've had a few people come through lately with interest in this area (e.g. @RealHarshThakur might be interested).

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.