Avoid hangs on many-core Windows machines (#7035)

Creating a multiprocessing Pool with too many processes can hit
ValueError exceptions or hangs or both. The number that counts as "too
many" depends on the Python version so this change uses 56 as a
guaranteed safe limit.

Details in #6965

Note that this only avoids the issue if an explicit jobs count is not
passed in.

Co-authored-by: Daniël van Noord <[email protected]>

Author: 2 people

doc/whatsnew/2/2.14/full.rst

@@ -13,6 +13,11 @@ Release date: TBA
 
   Closes #7006
 
+* Fixed an issue where many-core Windows machines (>~60 logical processors) would hang when
+  using the default jobs count.
+
+  Closes #6965
+
 What's New in Pylint 2.14.3?
 ----------------------------
 Release date: 2022-06-18

pylint/lint/base_options.py

@@ -244,7 +244,8 @@ def _make_linter_options(linter: PyLinter) -> Options:
                 "short": "j",
                 "default": 1,
                 "help": "Use multiple processes to speed up Pylint. Specifying 0 will "
-                "auto-detect the number of processors available to use.",
+                "auto-detect the number of processors available to use, and will cap "
+                "the count on Windows to avoid hangs.",
             },
         ),
         (

pylint/lint/run.py

@@ -82,6 +82,9 @@ def _cpu_count() -> int:
         cpu_count = multiprocessing.cpu_count()
     else:
         cpu_count = 1
+    if sys.platform == "win32":
+        # See also https://github.com/python/cpython/issues/94242
+        cpu_count = min(cpu_count, 56)  # pragma: no cover
     if cpu_share is not None:
         return min(cpu_share, cpu_count)
     return cpu_count