Return Canceled or DeadlineExceeded error code for context errors returned from polling

[amacaskill]

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug

/kind cleanup

/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:
This filters out context errors from the SLO. Right now we are returning Internal error code for context deadline exceeded and context canceled errors which are returned from waitForZonalOp or waitForRegionalOp. We already did a similar thing in the Filestore CSI Driver here.

Here is an example of an error we will now return DeadlineExceeded instead of Internal:

ERROR 2023-05-17T18:24:36.808758095Z WaitForOp(op: operation-xxx, zone: "us-west1-c") failed to poll the operation
ERROR 2023-05-17T18:24:36.808797527Z ControllerExpandVolume failed to resize disk: failed waiting for op for zonal resize for Key{"pvc-xxx", zone: "us-west1-c"}: context deadline exceeded
ERROR 2023-05-17T18:24:36.808807589Z /csi.v1.Controller/ControllerExpandVolume returned with error: rpc error: code = Internal desc = ControllerExpandVolume failed to resize disk: failed waiting for op for zonal resize for Key{"pvc-xxx", zone: "us-west1-c"}: context deadline exceeded

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

None

[mattcary]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amacaskill, mattcary

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [amacaskill,mattcary]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[amacaskill]

/cherry-pick release-1.10

[k8s-infra-cherrypick-robot]

@amacaskill: new pull request created: #1231

In response to this:

/cherry-pick release-1.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[judemars]

/cherry-pick release-1.9

[k8s-infra-cherrypick-robot]

@judemars: #1230 failed to apply on top of branch "release-1.9":

Applying: filter out context errors
Using index info to reconstruct a base tree...
M	pkg/gce-pd-csi-driver/utils.go
M	pkg/gce-pd-csi-driver/utils_test.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/gce-pd-csi-driver/utils_test.go
CONFLICT (content): Merge conflict in pkg/gce-pd-csi-driver/utils_test.go
Auto-merging pkg/gce-pd-csi-driver/utils.go
CONFLICT (content): Merge conflict in pkg/gce-pd-csi-driver/utils.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 filter out context errors
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.