From ac871c4560b9fc5b4a2fd360a9c73017ada2109c Mon Sep 17 00:00:00 2001 From: Richard Shadrach Date: Sun, 17 Jan 2021 12:41:09 -0500 Subject: [PATCH 1/5] CI/TST: Disallow usage of pytest.xfail --- .pre-commit-config.yaml | 5 ++++ doc/source/development/code_style.rst | 39 +++++++++++++++++++++++++++ 2 files changed, 44 insertions(+) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 0fc6e61049a44..70ee318677860 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -127,6 +127,11 @@ repos: types: [python] files: ^pandas/tests/ exclude: ^pandas/tests/extension/ + - id: unwanted-patters-pytest-xfail + name: Check for use of pytest.xfail + entry: pytest\.xfail + language: pygrep + types: [python] - id: inconsistent-namespace-usage name: 'Check for inconsistent use of pandas namespace in tests' entry: python scripts/check_for_inconsistent_pandas_namespace.py diff --git a/doc/source/development/code_style.rst b/doc/source/development/code_style.rst index 9477a9ac79dd6..410e53d375547 100644 --- a/doc/source/development/code_style.rst +++ b/doc/source/development/code_style.rst @@ -161,6 +161,45 @@ For example: # wrong from common import test_base +Testing +======= + +xfailing tests +-------------- + +See https://docs.pytest.org/en/latest/skipping.html for background. + +``pytest.xfail`` +---------------- + +Do not use this method. It has the same behavior as ``pytest.skip``, namely +it immediately stops the test and does not check if the test will fail. If +this is the behavior you desire, use ``pytest.skip`` instead. + +``pytest.mark.xfail`` +--------------------- + +Use this method if a test is known to fail but the manner in which it fails +is not meant to be captured. It is common to use this method for a test that +exhibits buggy behavior or a non-implemented feature. If +the failing test has flaky behavior, use the argument ``strict=False``. This +will make it so pytest does not fail if the test happens to pass. + +Prefer the decorator ``@pytest.mark.xfail`` and the argument ``pytest.param`` +over usage within a test so that the test is appropriately marked during the +collection phase of pytest. For xfailing a test that involves multiple +parameters, a fixture, or a combination of these, it is only possible to +xfail during the testing phase. To do so, use the ``request`` fixture: + +.. code-block:: python + import pytest + + def test_xfail(request): + request.node.add_marker(pytest.mark.xfail(reason="Indicate why here")) + +xfail is not to be used for tests involving failure due to invalid user arguments. +For these tests, we need to verify the correct exception type and error message +is being raised, using ``pytest.raises`` instead. Miscellaneous ============= From f859fdd468cf840daac1f771438a70acf2899451 Mon Sep 17 00:00:00 2001 From: Richard Shadrach Date: Sat, 6 Feb 2021 10:36:34 -0500 Subject: [PATCH 2/5] Only check pandas/tests --- .pre-commit-config.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 70ee318677860..fc08a1b164caf 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -132,6 +132,7 @@ repos: entry: pytest\.xfail language: pygrep types: [python] + files: ^pandas/tests/ - id: inconsistent-namespace-usage name: 'Check for inconsistent use of pandas namespace in tests' entry: python scripts/check_for_inconsistent_pandas_namespace.py From de746cc06d17d73e2d23b1bf511ab86b4fc5285d Mon Sep 17 00:00:00 2001 From: Richard Shadrach Date: Sat, 6 Feb 2021 10:43:42 -0500 Subject: [PATCH 3/5] Fix docs --- doc/source/development/code_style.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/source/development/code_style.rst b/doc/source/development/code_style.rst index 410e53d375547..3fa02d4aecf85 100644 --- a/doc/source/development/code_style.rst +++ b/doc/source/development/code_style.rst @@ -192,6 +192,7 @@ parameters, a fixture, or a combination of these, it is only possible to xfail during the testing phase. To do so, use the ``request`` fixture: .. code-block:: python + import pytest def test_xfail(request): From eff2b62314113b2db3919f7360966c96ca7124d7 Mon Sep 17 00:00:00 2001 From: Richard Shadrach Date: Sun, 7 Feb 2021 08:53:21 -0500 Subject: [PATCH 4/5] Change section name to Failing --- doc/source/development/code_style.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/development/code_style.rst b/doc/source/development/code_style.rst index 3fa02d4aecf85..3fd57562a4b8d 100644 --- a/doc/source/development/code_style.rst +++ b/doc/source/development/code_style.rst @@ -164,7 +164,7 @@ For example: Testing ======= -xfailing tests +Failing tests -------------- See https://docs.pytest.org/en/latest/skipping.html for background. From ff2c7c4734878e51152b9d02dc79ddbd46c3ff39 Mon Sep 17 00:00:00 2001 From: Richard Shadrach Date: Sun, 7 Feb 2021 12:23:50 -0500 Subject: [PATCH 5/5] Capitalize titles --- doc/source/development/code_style.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/source/development/code_style.rst b/doc/source/development/code_style.rst index 3fd57562a4b8d..19d83eb75e5bd 100644 --- a/doc/source/development/code_style.rst +++ b/doc/source/development/code_style.rst @@ -169,15 +169,15 @@ Failing tests See https://docs.pytest.org/en/latest/skipping.html for background. -``pytest.xfail`` ----------------- +Do not use ``pytest.xfail`` +--------------------------- Do not use this method. It has the same behavior as ``pytest.skip``, namely it immediately stops the test and does not check if the test will fail. If this is the behavior you desire, use ``pytest.skip`` instead. -``pytest.mark.xfail`` ---------------------- +Using ``pytest.mark.xfail`` +--------------------------- Use this method if a test is known to fail but the manner in which it fails is not meant to be captured. It is common to use this method for a test that