fix leaderelection: release lock when error occour and graceful shotd… #1747

fangfenghuang · 2022-03-15T02:57:10Z

…own #1669

Signed-off-by fangfenghuang [email protected]

What type of PR is this?

/kind bug

What this PR does / why we need it:

release lock when error occour and graceful shotdown

Which issue(s) this PR fixes:

Fixes #1669

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

NONE

k8s-ci-robot · 2022-03-15T02:57:18Z

Welcome @fangfenghuang!

It looks like this is your first PR to kubernetes-client/python 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-client/python has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

fangfenghuang · 2022-03-15T02:58:25Z

/assign @roycaihw

k8s-triage-robot · 2022-06-13T03:57:47Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

roycaihw · 2022-06-13T17:28:02Z

/remove-lifecycle stale

roycaihw

it's unclear to me how the graceful shotdown works. Could you elaborate?

roycaihw · 2022-06-14T01:04:42Z

kubernetes/base/leaderelection/leaderelection.py

+        if lock_status:
+            logging.info("release lock: {}".format(old_election_record.holder_identity))
+            old_election_record.lease_duration = 1
+            old_election_record.holder_identity = None


the lock may have been successfully acquired by another client. Doing so may force that client to lose the leadership. Why do we want to do this?

this happens when lock has been acquired and exception occurs that lock cannot be normally release, if not release the lock here, the next leader election must wait for timeout

i fix it just like client-go did: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go#L294

roycaihw

Thanks. LGTM otherwise

roycaihw · 2022-06-14T22:14:13Z

kubernetes/base/leaderelection/leaderelection.py

+        if lock_status:
+            logging.info("release lock: {}".format(old_election_record.holder_identity))
+            old_election_record.lease_duration = 1
+            old_election_record.holder_identity = None


please also set renew time and acquire time

roycaihw · 2022-06-14T22:17:02Z

kubernetes/base/leaderelection/leaderelection.py

+    def release(self):
+        lock_status, old_election_record = self.election_config.lock.get(self.election_config.lock.name,
+                                                                        self.election_config.lock.namespace)
+        if lock_status:


let's check if the elector is the leader. We can short-circuit the release method if not

roycaihw · 2022-06-14T22:20:10Z

kubernetes/base/leaderelection/electionconfig.py

+            retry_period,
+            onstarted_leading,
+            onstopped_leading,
+            release_on_cancel=False):


please document the caveat

nit: also "cancel" may not be the most accurate word here, since it was from the Golang context idea.

…own #1669 Signed-off-by fangfenghuang <[email protected]>

k8s-ci-robot · 2022-06-16T11:09:20Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: fangfenghuang
To complete the pull request process, please ask for approval from roycaihw after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

kubernetes/base/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-triage-robot · 2022-09-21T03:31:06Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2022-10-21T03:51:53Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2022-11-20T04:23:23Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen
Mark this PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2022-11-20T04:23:25Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen

Mark this PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 15, 2022

k8s-ci-robot requested review from fabianvf and roycaihw March 15, 2022 02:57

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Mar 15, 2022

k8s-ci-robot assigned roycaihw Mar 15, 2022

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 13, 2022

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 13, 2022

roycaihw reviewed Jun 14, 2022

View reviewed changes

fix leaderelection: release lock when error occour and graceful shotd…

f5b53f0

…own #1669 Signed-off-by fangfenghuang <[email protected]>

fangfenghuang requested a review from roycaihw June 23, 2022 02:40

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 21, 2022

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 21, 2022

k8s-ci-robot closed this Nov 20, 2022

fix leaderelection: release lock when error occour and graceful shotd… #1747

fix leaderelection: release lock when error occour and graceful shotd… #1747

Uh oh!

Conversation

fangfenghuang commented Mar 15, 2022

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

k8s-ci-robot commented Mar 15, 2022

Uh oh!

fangfenghuang commented Mar 15, 2022

Uh oh!

k8s-triage-robot commented Jun 13, 2022

Uh oh!

roycaihw commented Jun 13, 2022

Uh oh!

roycaihw left a comment

Choose a reason for hiding this comment

Uh oh!

roycaihw Jun 14, 2022

Choose a reason for hiding this comment

Uh oh!

fangfenghuang Jun 14, 2022

Choose a reason for hiding this comment

Uh oh!

fangfenghuang Jun 14, 2022

Choose a reason for hiding this comment

Uh oh!

roycaihw left a comment

Choose a reason for hiding this comment

Uh oh!

roycaihw Jun 14, 2022

Choose a reason for hiding this comment

Uh oh!

fangfenghuang Jun 16, 2022

Choose a reason for hiding this comment

Uh oh!

roycaihw Jun 14, 2022

Choose a reason for hiding this comment

Uh oh!

roycaihw Jun 14, 2022

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jun 16, 2022

Uh oh!

k8s-triage-robot commented Sep 21, 2022

Uh oh!

k8s-triage-robot commented Oct 21, 2022

Uh oh!

k8s-triage-robot commented Nov 20, 2022

Uh oh!

k8s-ci-robot commented Nov 20, 2022

Uh oh!

Uh oh!