Skip to content

extensive usages of withContext/Undispatched coroutines causes ThreadLocal maps to grow, which causes application to become unresponsive #3669

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
RestfulBlue opened this issue Mar 9, 2023 · 2 comments

Comments

@RestfulBlue
Copy link

RestfulBlue commented Mar 9, 2023

Each undispatched coroutine spawns a separate threadLocal and if there are a lot of undispatched coroutines - threadLocal map on all threads becomes huge and server spend 100% of cpu on traversing threadLocal maps in order to find empty spot, shift entries and so on

Below a attach screenshots of heapdump from production application which have 2gb of heap.
It creates around 2k of UndispatchedCoroutines. In such scenario ThreadLocalMap on each thread is around ~700. ThreadLocalMap is resized when 75% of slots are used, which means there were at least 500 elements used in parallel at some point of time.
Earlier we had constant incidents when threadLocal map was reaching 1500 elements. At that point all cpu is being spent on threadLocal map updates.

Screenshot 2023-03-09 at 14 17 03

Screenshot 2023-03-09 at 14 28 56

0f.png">

Screenshot 2023-03-09 at 14 34 08

Screenshot 2023-03-09 at 18 11 29

issue seems to be quite critical, as far as i understand any highloaded server which relies on coroutines could become unavailable at any point of time

@qwwdfsad
Copy link
Member

qwwdfsad commented May 5, 2023

After #3593 it's not clear whether the evidence for the issue exists, we are waiting for the additional experiments from Space

@qwwdfsad
Copy link
Member

Closing as "not enough data to claim it's a problem".
Feel free to reopen if the problem reproduces

@qwwdfsad qwwdfsad closed this as not planned Won't fix, can't repro, duplicate, stale Jun 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants