Skip to content

🐛 Priorityqueue: Yet another queue_depth metric fix #3085

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 23, 2025
+45 −3
Merged

🐛 Priorityqueue: Yet another queue_depth metric fix #3085

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions pkg/controller/priorityqueue/metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,11 +85,11 @@ func (m *defaultQueueMetrics[T]) get(item T) {
return
}

m.depth.Dec()

m.mapLock.Lock()
defer m.mapLock.Unlock()

m.depth.Dec()

m.processingStartTimes[item] = m.clock.Now()
if startTime, exists := m.addTimes[item]; exists {
m.latency.Observe(m.sinceInSeconds(startTime))
Expand Down
2 changes: 1 addition & 1 deletion pkg/controller/priorityqueue/priorityqueue.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ func (w *priorityqueue[T]) AddWithOpts(o AddOpts, items ...T) {
}

if item.ReadyAt != nil && (readyAt == nil || readyAt.Before(*item.ReadyAt)) {
if readyAt == nil {
if readyAt == nil && !w.becameReady.Has(key) {
w.metrics.add(key)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to be sure.

Do we also have to add it to becameReady here?

w.becameReady.Insert(item.Key)

I'm not sure if we otherwise still have another edge case where we count twice

Should we maybe simply always add it to becameReady when we call metrics.add?

If I see correctly we always remove it from there when we get the item out of the queue, so it should be fine to always add it?

(might also make sense to rename becameReady to something that expresses that an item is currently counted in the queue depth or something)

Copy link
Member Author

@alvaroaleman alvaroaleman Jan 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason that isn't needed is that in spin we only call add if the item has a non-nil readyAt wheras here we only call it here if we are going to set ReadyAt to nil which is how mutual exclusivity this way round is ensured.

I've extended the two last tests that validate that we do call metrics.add here to do a queue.get at the end which ensures the codepath that potentially calls add again in spin gets execercised so that we validate the case you described.

I do agree the current name isn't great but I couldn't come up with a better one - calledAdd would be confusing because this is only relevant for items that have a RequeueAfter and because we remove from it when we do the metrics.get. If you have ideas for a better name here, I am all ears :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also no better ideas for the name :)

}
item.ReadyAt = readyAt
Expand Down
42 changes: 42 additions & 0 deletions pkg/controller/priorityqueue/priorityqueue_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,6 +395,48 @@ var _ = Describe("Controllerworkqueue", func() {
Expect(q.Len()).To(Equal(1))
metrics.mu.Lock()
Expect(metrics.depth["test"]).To(Equal(1))
metrics.mu.Unlock()

// Get the item to ensure the codepath in
// `spin` for the metrics is passed by so
// that this starts failing if it incorrectly
// calls `metrics.add` again.
item, _ := q.Get()
Expect(item).To(Equal("foo"))
Expect(q.Len()).To(Equal(0))
metrics.mu.Lock()
Expect(metrics.depth["test"]).To(Equal(0))
metrics.mu.Unlock()
})

It("Updates metrics correctly for an item whose requeueAfter expired that gets added again without requeueAfter", func() {
q, metrics := newQueue()
defer q.ShutDown()

q.AddWithOpts(AddOpts{After: 50 * time.Millisecond}, "foo")
time.Sleep(100 * time.Millisecond)

Expect(q.Len()).To(Equal(1))
metrics.mu.Lock()
Expect(metrics.depth["test"]).To(Equal(1))
metrics.mu.Unlock()

q.AddWithOpts(AddOpts{}, "foo")
Expect(q.Len()).To(Equal(1))
metrics.mu.Lock()
Expect(metrics.depth["test"]).To(Equal(1))
metrics.mu.Unlock()

// Get the item to ensure the codepath in
// `spin` for the metrics is passed by so
// that this starts failing if it incorrectly
// calls `metrics.add` again.
item, _ := q.Get()
Expect(item).To(Equal("foo"))
Expect(q.Len()).To(Equal(0))
metrics.mu.Lock()
Expect(metrics.depth["test"]).To(Equal(0))
metrics.mu.Unlock()
})
})

Expand Down
Loading