CI/TST: Error on PytestUnraisableExceptionWarning instead of using psutil to check open resources #51443

mroeschke · 2023-02-16T23:30:55Z

Using psutil to check for open resources in tests has been a false positive for a while, picking up unrelated connections such as the database services.

Erroring on ResourceWarnings, captured by PytestUnraisableExceptionWarning, should be sufficient for validating that resources are not left open as a result of a test. pytest-dev/pytest#9825

…for open resouces for tests

jbrockmendel · 2023-02-16T23:35:33Z

there have been times in the past where we did leave files open incorrectly, which I think the check this removes would catch. does the check this adds still catch those?

mroeschke · 2023-02-16T23:57:22Z

Correct for example

 % git diff
diff --git a/pandas/tests/frame/test_api.py b/pandas/tests/frame/test_api.py
index e5787a7f16..c280fb9748 100644
--- a/pandas/tests/frame/test_api.py
+++ b/pandas/tests/frame/test_api.py
@@ -379,3 +379,7 @@ class TestDataFrameMisc:
         df = DataFrame()
         with tm.assert_produces_warning(None):
             inspect.getmembers(df)
+
+
+def test_open():
+    open("/dev/null")

% pytest pandas/tests/frame/test_api.py

...

pandas/tests/frame/test_api.py .................................F

...
    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            yield
            if cm.unraisable:
                if cm.unraisable.err_msg is not None:
                    err_msg = cm.unraisable.err_msg
                else:
                    err_msg = "Exception ignored in"
                msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                msg += "".join(
                    traceback.format_exception(
                        cm.unraisable.exc_type,
                        cm.unraisable.exc_value,
                        cm.unraisable.exc_traceback,
                    )
                )
>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
E
E               Traceback (most recent call last):
E                 File "...pandas/tests/frame/test_api.py", line 385, in test_open
E                   open("/dev/null")
E               ResourceWarning: unclosed file <_io.TextIOWrapper name='/dev/null' mode='r' encoding='UTF-8'>

../opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarnin

jbrockmendel · 2023-02-17T01:07:31Z

awesome, thanks for explaining

mroeschke · 2023-02-17T01:07:55Z

Looks like this caught actual issues with open files in ExcelWriter

pandas/io/excel/_base.py

jbrockmendel · 2023-02-19T16:43:09Z

pandas/tests/scalar/timestamp/test_timestamp.py

+            # https://github.com/dateutil/dateutil/issues/197
+            pytest.skip(
+                "tzlocal() on a 32 bit platform causes internal overflow errors"
+            )


are we giving incorrect results in these cases?

Running the 32 bit container locally it appears this still returns the correct result but hits the traceback in the dateutil issue

>>> from pandas import Timestamp >>> import dateutil.tz >>> dt = Timestamp("2100-01-01 00:00:00", tz=dateutil.tz.tzlocal()) Traceback (most recent call last): File "pandas/_libs/tslibs/tzconversion.pyx", line 125, in pandas._libs.tslibs.tzconversion.Localizer.utc_val_to_local_val return utc_val + _tz_localize_using_tzinfo_api( File "pandas/_libs/tslibs/tzconversion.pyx", line 736, in pandas._libs.tslibs.tzconversion._tz_localize_using_tzinfo_api dt = _astimezone(dts, tz) File "pandas/_libs/tslibs/tzconversion.pyx", line 768, in pandas._libs.tslibs.tzconversion._astimezone return tz.fromutc(result) File "/root/virtualenvs/pandas-dev/lib/python3.8/site-packages/dateutil/tz/_common.py", line 144, in fromutc return f(self, dt) File "/root/virtualenvs/pandas-dev/lib/python3.8/site-packages/dateutil/tz/_common.py", line 261, in fromutc _fold = self._fold_status(dt, dt_wall) File "/root/virtualenvs/pandas-dev/lib/python3.8/site-packages/dateutil/tz/_common.py", line 196, in _fold_status if self.is_ambiguous(dt_wall): File "/root/virtualenvs/pandas-dev/lib/python3.8/site-packages/dateutil/tz/tz.py", line 254, in is_ambiguous naive_dst = self._naive_is_dst(dt) File "/root/virtualenvs/pandas-dev/lib/python3.8/site-packages/dateutil/tz/tz.py", line 260, in _naive_is_dst return time.localtime(timestamp + time.timezone).tm_isdst OverflowError: timestamp out of range for platform time_t Exception ignored in: 'pandas._libs.tslibs.conversion._localize_tso' Traceback (most recent call last): File "pandas/_libs/tslibs/tzconversion.pyx", line 125, in pandas._libs.tslibs.tzconversion.Localizer.utc_val_to_local_val return utc_val + _tz_localize_using_tzinfo_api( File "pandas/_libs/tslibs/tzconversion.pyx", line 736, in pandas._libs.tslibs.tzconversion._tz_localize_using_tzinfo_api dt = _astimezone(dts, tz) File "pandas/_libs/tslibs/tzconversion.pyx", line 768, in pandas._libs.tslibs.tzconversion._astimezone return tz.fromutc(result) File "/root/virtualenvs/pandas-dev/lib/python3.8/site-packages/dateutil/tz/_common.py", line 144, in fromutc return f(self, dt) File "/root/virtualenvs/pandas-dev/lib/python3.8/site-packages/dateutil/tz/_common.py", line 261, in fromutc _fold = self._fold_status(dt, dt_wall) File "/root/virtualenvs/pandas-dev/lib/python3.8/site-packages/dateutil/tz/_common.py", line 196, in _fold_status if self.is_ambiguous(dt_wall): File "/root/virtualenvs/pandas-dev/lib/python3.8/site-packages/dateutil/tz/tz.py", line 254, in is_ambiguous naive_dst = self._naive_is_dst(dt) File "/root/virtualenvs/pandas-dev/lib/python3.8/site-packages/dateutil/tz/tz.py", line 260, in _naive_is_dst return time.localtime(timestamp + time.timezone).tm_isdst OverflowError: timestamp out of range for platform time_t >>> dt.is_leap_year False

jbrockmendel · 2023-02-19T16:46:17Z

pandas/tests/io/excel/conftest.py

-        proc = psutil.Process()
-        flist = proc.open_files()
-        yield
-        flist2 = proc.open_files()


documenting in case we wind up keeping this: looks like all of the flaky failures have raddr.port of either 5432 or 3306 (mostly 5432), which correspond to postgres and mysql defaults, respectively. We could filter those out here

FWIW I have seen the port 443 show up a few times too

phofl · 2023-02-20T21:00:54Z

merging, I guess we should backport to get rid of these on the 2.0.x branch?

phofl · 2023-02-20T21:01:11Z

thx @mroeschke

…ionWarning instead of using psutil to check open resources

jbrockmendel · 2023-02-20T21:07:28Z

nice!

…ExceptionWarning instead of using psutil to check open resources) (#51507) Backport PR #51443: CI/TST: Error on PytestUnraisableExceptionWarning instead of using psutil to check open resources Co-authored-by: Matthew Roeschke <[email protected]> Co-authored-by: Patrick Hoefler <[email protected]>

CI/TST: Check for PytestUnraisableExceptionWarning instead of psutil …

8df59ed

…for open resouces for tests

mroeschke added CI Continuous Integration Unreliable Test Unit tests that occasionally fail labels Feb 16, 2023

Make ResourceWarning also error

f903f62

mroeschke added 2 commits February 16, 2023 16:34

Merge remote-tracking branch 'upstream/main' into ci/open_files_rework

72b70cc

Fix ExcelWriter not clossing files during instantiation

5c24bdd

Ignore asyncio ResourceWarnings

a046267

phofl reviewed Feb 17, 2023

View reviewed changes

pandas/io/excel/_base.py Show resolved Hide resolved

mroeschke added 14 commits February 17, 2023 09:35

Merge remote-tracking branch 'upstream/main' into ci/open_files_rework

844f277

Merge remote-tracking branch 'upstream/main' into ci/open_files_rework

cfdd50f

Forgot comma

edaf9db

Merge remote-tracking branch 'upstream/main' into ci/open_files_rework

648e83c

Ignore socketwarnings

d5d269b

Merge remote-tracking branch 'upstream/main' into ci/open_files_rework

bd490be

More permissive mesage match

8401c79

Ignore tzlocal specific 32 bit error

bae2fc0

Merge remote-tracking branch 'upstream/main' into ci/open_files_rework

6b80a8e

Merge remote-tracking branch 'upstream/main' into ci/open_files_rework

4c2ef83

Try these changes

b99158c

Merge remote-tracking branch 'upstream/main' into ci/open_files_rework

8c9cf4e

Try the whole file?

efb667b

Just skip under condidition

1daf6d7

mroeschke added this to the 2.0 milestone Feb 19, 2023

jbrockmendel reviewed Feb 19, 2023

View reviewed changes

phofl mentioned this pull request Feb 20, 2023

RLS: 2.0 #46776

Closed

1 task

phofl approved these changes Feb 20, 2023

View reviewed changes

phofl merged commit cbad73a into pandas-dev:main Feb 20, 2023

meeseeksmachine mentioned this pull request Feb 20, 2023

Backport PR #51443 on branch 2.0.x (CI/TST: Error on PytestUnraisableExceptionWarning instead of using psutil to check open resources) #51507

Merged

meeseeksmachine pushed a commit to meeseeksmachine/pandas that referenced this pull request Feb 20, 2023

Backport PR pandas-dev#51443: CI/TST: Error on PytestUnraisableExcept…

0bca209

…ionWarning instead of using psutil to check open resources

mroeschke deleted the ci/open_files_rework branch February 20, 2023 22:03

mroeschke mentioned this pull request Feb 21, 2023

CI/TST: Enable -W error:::pandas in pyproject.toml #51312

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI/TST: Error on PytestUnraisableExceptionWarning instead of using psutil to check open resources #51443

CI/TST: Error on PytestUnraisableExceptionWarning instead of using psutil to check open resources #51443

mroeschke commented Feb 16, 2023

jbrockmendel commented Feb 16, 2023

mroeschke commented Feb 16, 2023

jbrockmendel commented Feb 17, 2023

mroeschke commented Feb 17, 2023

jbrockmendel Feb 19, 2023

mroeschke Feb 19, 2023 •

edited

Loading

jbrockmendel Feb 19, 2023

mroeschke Feb 19, 2023

phofl commented Feb 20, 2023

phofl commented Feb 20, 2023

jbrockmendel commented Feb 20, 2023

CI/TST: Error on PytestUnraisableExceptionWarning instead of using psutil to check open resources #51443

CI/TST: Error on PytestUnraisableExceptionWarning instead of using psutil to check open resources #51443

Conversation

mroeschke commented Feb 16, 2023

jbrockmendel commented Feb 16, 2023

mroeschke commented Feb 16, 2023

jbrockmendel commented Feb 17, 2023

mroeschke commented Feb 17, 2023

jbrockmendel Feb 19, 2023

Choose a reason for hiding this comment

mroeschke Feb 19, 2023 • edited Loading

Choose a reason for hiding this comment

jbrockmendel Feb 19, 2023

Choose a reason for hiding this comment

mroeschke Feb 19, 2023

Choose a reason for hiding this comment

phofl commented Feb 20, 2023

phofl commented Feb 20, 2023

jbrockmendel commented Feb 20, 2023

mroeschke Feb 19, 2023 •

edited

Loading