Skip to content

Support revert-pattern in logcounter #847

Closed
@linxiulei

Description

@linxiulei

In addition to --pattern argument supported in logcounter binary, I'd like to have a --revert-pattern argument, which decrease the counter when matched.

The reason I need the this feature is to skip counting restarts initiated deliberately by system administrators (e.g. systemctl restart containerd). Otherwise, NPD would post a node condition that causes noise.

"condition": "FrequentContainerdRestart",
"reason": "FrequentContainerdRestart",
"path": "/home/kubernetes/bin/log-counter",
"args": [
  "--journald-source=systemd",
  "--log-path=/var/log/journal",
  "--lookback=20m",
  "--count=5",
  "--pattern=Starting containerd container runtime...",
  "--revert-pattern=Stopping containerd container runtime..."
],

Notes:

  • systemctl restart containerd results in two logs in journald "Stopping containerd container runtime..." and later "Starting containerd container runtime..." so the counter value should not be changed after matching both pattern and revert-pattern
  • systemctl restart containerd is rather safe to be considered deliberate operations that should be disregarded as availability issues. Health checks to restart containerd are done by executing systemctl kill --kill-who=main containerd, which doesn't result in journald log Stopping containerd container runtime...

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions