controller/openstackcluster: reconcileID is re-used 1000+ times

[EmilienM]

/kind bug

What steps did you take and what happened:

Open https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-openstack/1789/pull-cluster-api-provider-openstack-e2e-test/1736818285917245440/artifacts/clusters/bootstrap/logs/capo-system/capo-controller-manager/capo-controller-manager-66896777f9-z6k2q/manager.log

And grep for:

reconcileID="12187d68-f3cf-4596-94e8-83666306d5fc"

2191 uses of the same reconcile ID.

That's wrong and it wasn't the case in late October:
https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-openstack/1696/pull-cluster-api-provider-openstack-e2e-test/1717592179636965376/artifacts/clusters/bootstrap/logs/capo-system/capo-controller-manager/capo-controller-manager-66896777f9-kfgvg/manager.log

I strongly suspect this PR to be the cause, I've bisected CI jobs and this issue started after this PR.

What did you expect to happen:

The reconcile loops should be uniquely used and IDs shouldn't be the same. I found this issue when debugging #1788.

[mdbooth]

It looks like when we create a cached provider scope we put the logger in it. The problem is that this logger is reused across multiple reconciles and has different context annotations added for each. We need to find some way to separate the logger from the scope. Perhaps we should pass it around in the context instead.

The impact of this is that the logs are incorrect. It shouldn't actually be breaking any functionality, but we should fix it asap.

[EmilienM]

@kubernetes-sigs/cluster-api-provider-openstack-maintainers please add v1beta1 to milestone.

[mdbooth]

I'm confident this is not a v1beta1 issue. I'm going to remove the tag.

[EmilienM]

It looks like when we create a cached provider scope we put the logger in it. The problem is that this logger is reused across multiple reconciles and has different context annotations added for each. We need to find some way to separate the logger from the scope. Perhaps we should pass it around in the context instead.

The impact of this is that the logs are incorrect. It shouldn't actually be breaking any functionality, but we should fix it asap.

I don't know if that's what you had in mind, but I started something here: #1861