Refactor metric defer() statements to gRPC metric interceptor

[pwschuurman]

Migrate metric defer() statements to gRPC metric interceptor. This allows for more accurate error code reporting of the internal operation_errors metric when gRPC handling functionality is refactored

/kind api-change
/kind bug

/kind cleanup

/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it: Currently the ControllerServer reports error metrics through a defer() function pattern. This captures the output of captured variables when the function returns. If captured references are not updated (eg: a statement is refactored, and the err variable is not updated), the error code reported may not be accurate. This can result in poor error classification when this metric is used for reporting (as all error codes may be classified as INTERNAL).

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Improved field accuracy for `csidriver/operation_errors` metric.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pwschuurman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [pwschuurman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[pwschuurman]

/retest

[pwschuurman]

/retest

[pwschuurman]

/assign @amacaskill

[amacaskill]

@pwschuurman Can you please add a release note?

The release note in the PR is how changes show up in the release notes/changelog that is generated when you press the generate release notes button when cutting a release. I think this is useful to know what version had what change, especially for SLO changes when we are debugging to see if whatever cluster triggered the alert, has your fix.

[pwschuurman]

/hold Tests added but want to add a regression test for one of the rpc calls that invoke the GetMetadataFromDisk()

[pwschuurman]

/unhold

[amacaskill]

/lgtm

[pwschuurman]

/cherry-pick release-1.14

[k8s-infra-cherrypick-robot]

@pwschuurman: #1876 failed to apply on top of branch "release-1.14":

Applying: Migrate metric defer() statements to gRPC metric interceptor. This allows for more accurate error code reporting if gRPC functionality is refactored
Using index info to reconstruct a base tree...
M	cmd/gce-pd-csi-driver/main.go
M	pkg/gce-pd-csi-driver/controller.go
M	pkg/gce-pd-csi-driver/gce-pd-driver.go
M	pkg/gce-pd-csi-driver/gce-pd-driver_test.go
M	test/sanity/sanity_test.go
Falling back to patching base and 3-way merge...
Auto-merging test/sanity/sanity_test.go
Auto-merging pkg/gce-pd-csi-driver/gce-pd-driver_test.go
CONFLICT (content): Merge conflict in pkg/gce-pd-csi-driver/gce-pd-driver_test.go
Auto-merging pkg/gce-pd-csi-driver/gce-pd-driver.go
Auto-merging pkg/gce-pd-csi-driver/controller.go
CONFLICT (content): Merge conflict in pkg/gce-pd-csi-driver/controller.go
Auto-merging cmd/gce-pd-csi-driver/main.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 Migrate metric defer() statements to gRPC metric interceptor. This allows for more accurate error code reporting if gRPC functionality is refactored

In response to this:

/cherry-pick release-1.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[pwschuurman]

/cherry-pick release-1.15

[k8s-infra-cherrypick-robot]

@pwschuurman: new pull request created: #1886

In response to this:

/cherry-pick release-1.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.