Merge pull request #3289 from Zac-HD/fix-junit-xml

Fix `junit-xml` support

Author: Zac-HD

.github/workflows/main.yml

@@ -43,6 +43,8 @@ jobs:
           - lint-conjecture-rust
           - check-nose
           - check-pytest46
+          - check-pytest54
+          - check-pytest62
           - check-django40
           - check-django32
           - check-django22

AUTHORS.rst

@@ -23,6 +23,7 @@ their individual contributions.
 * `Benjamin Palmer <https://github.com/benjpalmer>`_
 * `Bex Dunn <https://github.com/BexDunn>`_ ([email protected])
 * `Bill Tucker <https://github.com/imbilltucker>`_ ([email protected])
+* `Brandon Chinn <https://github.com/brandonchinn178>`_
 * `Bryant Eisenbach <https://github.com/fubuloubu>`_
 * `Buck Evan, copyright Google LLC <https://github.com/bukzor>`_
 * `Cameron McGill <https://www.github.com/Cameron-JM>`_

CONTRIBUTING.rst

@@ -165,7 +165,7 @@ Some notable commands:
 ``./build.sh check-coverage`` will verify 100% code coverage by running a
 curated subset of the test suite.
 
-``./build.sh check-py36`` (etc.) will run most of the test suite against a
+``./build.sh check-py37`` (etc.) will run most of the test suite against a
 particular python version.
 
 ``./build.sh format`` will reformat your code according to the Hypothesis coding style. You should use this before each
@@ -180,3 +180,53 @@ Run ``./build.sh tasks`` for a list of all supported build task names.
 Note: The build requires a lot of different versions of python, so rather than have you install them yourself,
 the build system will install them itself in a local directory. This means that the first time you run a task you
 may have to wait a while as the build downloads and installs the right version of python for you.
+
+~~~~~~~~~~~~~
+Running Tests
+~~~~~~~~~~~~~
+
+The tasks described above will run all of the tests (e.g. ``check-py37``). But
+the ``tox`` task will give finer-grained control over the test runner. At a
+high level, the task takes the form:
+
+.. code-block::
+
+    ./build.sh tox py37-custom 3.7.13 [tox args] -- [pytest args]
+
+Namely, first provide the tox environment (see ``tox.ini``), then the python
+version to test with, then any ``tox`` or ``pytest`` args as needed. For
+example, to run all of the tests in the file
+``tests/nocover/test_conjecture_engine.py`` with python 3.8:
+
+.. code-block::
+
+    ./build.sh tox py38-custom 3.8.13 -- tests/nocover/test_conjecture_engine.py
+
+See the ``tox`` docs and ``pytest`` docs for more information:
+* https://docs.pytest.org/en/latest/how-to/usage.html
+* https://tox.wiki/en/latest/config.html#cli
+
+^^^^^^^^^^^
+Test Layout
+^^^^^^^^^^^
+
+See ``hypothesis-python/tests/README.rst``
+
+^^^^^^^^^^^^^^^^
+Useful Arguments
+^^^^^^^^^^^^^^^^
+
+Some useful arguments to pytest include:
+
+* You can pass ``-n 0`` to turn off ``pytest-xdist``'s parallel test execution.
+  Sometimes for running just a small number of tests its startup time is longer
+  than the time it saves (this will vary from system to system), so this can
+  be helpful if you find yourself waiting on test runners to start a lot.
+* You can use ``-k`` to select a subset of tests to run. This matches on substrings
+  of the test names. For example ``-kfoo`` will only run tests that have "foo" as
+  a substring of their name. You can also use composite expressions here.
+  e.g. ``-k'foo and not bar'`` will run anything containing foo that doesn't
+  also contain bar.  `More information on how to select tests to run can be found
+  in the pytest documentation <https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`__.
+
+

guides/testing-hypothesis.rst

@@ -37,7 +37,7 @@ a starting point, not the final goal.
   because when it was found and fixed, someone wrote a test to make sure it
   couldn't come back!
 
-The ``tests/`` directory has some notes in the README file on where various
+The ``hypothesis-python/tests/`` directory has some notes in the README file on where various
 kinds of tests can be found or added.  Go there for the practical stuff, or
 just ask one of the maintainers for help on a pull request!
 
@@ -48,122 +48,8 @@ Further reading: How `SQLite is tested <https://sqlite.org/testing.html>`_,
 Dan Luu writes about `fuzz testing <https://danluu.com/testing/>`_ and
 `broken processes <https://danluu.com/wat/>`_, among other things.
 
----------------------------------------
-Setting up a virtualenv to run tests in
----------------------------------------
-
-If you want to run individual tests rather than relying on the make tasks
-(which you probably will), it's easiest to do this in a virtualenv.
-
-The following will give you a working virtualenv for running tests in:
-
-.. code-block:: bash
-
-  pip install virtualenv
-  python -m virtualenv testing-venv
-
-  # On Windows: testing-venv\Scripts\activate
-  source testing-venv/bin/activate
-
-  # Can also use pip install -e .[all] to get
-  # all optional dependencies
-  pip install -e .
-
-  # Test specific dependencies.
-  pip install -r requirements/test.in
-
-Now whenever you want to run tests you can just activate the virtualenv
-using ``source testing-venv/bin/activate`` or ``testing-venv\Scripts\activate``
-and all of the dependencies will be available to you and your local copy
-of Hypothesis will be on the path (so any edits will be picked up automatically
-and you don't need to reinstall it in the local virtualenv).
-
 -------------
 Running Tests
 -------------
 
-In order to run tests outside of the make/tox/etc set up, you'll need an
-environment where Hypothesis is on the path and all of the testing dependencies
-are installed.
-We recommend doing this inside a virtualenv as described in the previous section.
-
-All testing is done using `pytest <https://docs.pytest.org/en/latest/>`_,
-with a couple of plugins installed. For advanced usage we recommend reading the
-pytest documentation, but this section will give you a primer in enough of the
-common commands and arguments to get started.
-
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Selecting Which Files to Run
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The following invocation runs all of the tests in the file
-`tests/cover/test_conjecture_engine.py`:
-
-.. code-block::
-
-    python -m pytest tests/cover/test_conjecture_engine.py
-
-If you want to run multiple files you can pass them all as arguments, and if
-you pass a directory then it will run all files in that directory.
-For example the following runs all the files in `test_conjecture_engine.py`
-and `test_slippage.py`
-
-.. code-block::
-
-    python -m pytest tests/cover/test_conjecture_engine.py tests/cover/test_slippage.py
-
-If you were running this in bash (if you're not sure: if you're not on Windows
-you probably are) you could also use the syntax:
-
-.. code-block::
-
-    python -m pytest tests/cover/test_{conjecture_engine,slippage}.py
-
-And the following would run all tests under `tests/cover`:
-
-.. code-block::
-
-    python -m pytest tests/cover
-
-
-~~~~~~~~~~~
-Test Layout
-~~~~~~~~~~~
-
-The top level structure of the tests in Hypothesis looks as follows:
-
-* ``cover`` contains tests that we measure coverage for. This is intended to
-  be a fairly minimal and fast set of tests that still gives pretty good
-  confidence in the behaviour of the test suite. It is currently failing at
-  both "minimal" and "fast", but we're trying to move it back in that
-  direction. Try not to add tests to this unless they're actually to cover
-  some specific target.
-* ``nocover`` is a general dumping ground for slower tests that aren't needed
-  to achieve coverage.
-* ``quality`` is for expensive tests about the distribution or shrinking of
-  examples. These will only be run on one Python version.
-* The remaining test directories are for testing specific extras modules and
-  should have the same name.
-
-As a rule of thumb when writing new tests, they should go in nocover unless
-they are for a specific extras module or to deliberately target a particular
-line for coverage. In the latter case, prefer fast unit tests over larger and
-slower integration tests (we are not currently very good at this).
-
-
-~~~~~~~~~~~~~~~~
-Useful Arguments
-~~~~~~~~~~~~~~~~
-
-Some useful arguments to pytest include:
-
-* You can pass ``-n 0`` to turn off ``pytest-xdist``'s parallel test execution.
-  Sometimes for running just a small number of tests its startup time is longer
-  than the time it saves (this will vary from system to system), so this can
-  be helpful if you find yourself waiting on test runners to start a lot.
-* You can use ``-k`` to select a subset of tests to run. This matches on substrings
-  of the test names. For example ``-kfoo`` will only run tests that have "foo" as
-  a substring of their name. You can also use composite expressions here.
-  e.g. ``-k'foo and not bar'`` will run anything containing foo that doesn't
-  also contain bar.  `More information on how to select tests to run can be found
-  in the pytest documentation <https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`__.
+Tests are run via ``build.sh``. See ``CONTRIBUTING.rst`` for more details.

hypothesis-python/RELEASE.rst

@@ -0,0 +1,16 @@
+RELEASE_TYPE: patch
+
+The Hypothesis pytest plugin was not outputting valid xunit2 nodes when
+``--junit-xml`` was specified. This has been broken since Pytest 5.4, which
+changed the internal API for adding nodes to the junit report.
+
+This also fixes the issue when using hypothesis with ``--junit-xml`` and
+``pytest-xdist`` where the junit xml report would not be xunit2 compatible.
+Now, when using with ``pytest-xdist``, the junit report will just omit the
+``<properties>`` node.
+
+For more details, see `this pytest issue <https://github.com/pytest-dev/pytest/issues/1126#issuecomment-484581283>`__,
+`this pytest issue <https://github.com/pytest-dev/pytest/issues/7767#issuecomment-1082436256>`__,
+and :issue:`1935`
+
+Thanks to Brandon Chinn for this bug fix!

hypothesis-python/src/_hypothesis_pytestplugin.py

@@ -25,6 +25,11 @@
 
 import pytest
 
+try:
+    from _pytest.junitxml import xml_key
+except ImportError:
+    xml_key = "_xml"  # type: ignore
+
 LOAD_PROFILE_OPTION = "--hypothesis-profile"
 VERBOSITY_OPTION = "--hypothesis-verbosity"
 PRINT_STATISTICS_OPTION = "--hypothesis-show-statistics"
@@ -53,6 +58,8 @@
 check to assure Hypothesis that you understand what you are doing.
 """
 
+STATS_KEY = "_hypothesis_stats"
+
 
 class StoringReporter:
     def __init__(self, config):
@@ -255,9 +262,7 @@ def pytest_runtest_call(item):
 
             def note_statistics(stats):
                 stats["nodeid"] = item.nodeid
-                item.hypothesis_statistics = base64.b64encode(
-                    describe_statistics(stats).encode()
-                ).decode()
+                item.hypothesis_statistics = describe_statistics(stats)
 
             with collector.with_value(note_statistics):
                 with with_reporter(store):
@@ -266,6 +271,16 @@ def note_statistics(stats):
             if store.results:
                 item.hypothesis_report_information = list(store.results)
 
+    def _stash_get(config, key, default):
+        if hasattr(config, "stash"):
+            # pytest 7
+            return config.stash.get(key, default)
+        elif hasattr(config, "_store"):
+            # pytest 5.4
+            return config._store.get(key, default)
+        else:
+            return getattr(config, key, default)
+
     @pytest.hookimpl(hookwrapper=True)
     def pytest_runtest_makereport(item, call):
         report = (yield).get_result()
@@ -274,46 +289,42 @@ def pytest_runtest_makereport(item, call):
                 ("Hypothesis", "\n".join(item.hypothesis_report_information))
             )
         if hasattr(item, "hypothesis_statistics") and report.when == "teardown":
+            stats = item.hypothesis_statistics
+            stats_base64 = base64.b64encode(stats.encode()).decode()
+
             name = "hypothesis-statistics-" + item.nodeid
-            try:
-                item.config._xml.add_global_property(name, item.hypothesis_statistics)
-            except AttributeError:
-                # --junitxml not passed, or Pytest 4.5 (before add_global_property)
-                # We'll fail xunit2 xml schema checks, upgrade pytest if you care.
-                report.user_properties.append((name, item.hypothesis_statistics))
+
+            # Include hypothesis information to the junit XML report.
+            #
+            # Note that when `pytest-xdist` is enabled, `xml_key` is not present in the
+            # stash, so we don't add anything to the junit XML report in that scenario.
+            # https://github.com/pytest-dev/pytest/issues/7767#issuecomment-1082436256
+            xml = _stash_get(item.config, xml_key, None)
+            if xml:
+                xml.add_global_property(name, stats_base64)
+
+            # If there's a terminal report, include our summary stats for each test
+            terminalreporter = item.config.pluginmanager.getplugin("terminalreporter")
+            if terminalreporter is not None:
+                # ideally, we would store this on terminalreporter.config.stash, but
+                # pytest-xdist doesn't copy that back to the controller
+                report.__dict__[STATS_KEY] = stats
+
             # If there's an HTML report, include our summary stats for each test
-            stats = base64.b64decode(item.hypothesis_statistics.encode()).decode()
             pytest_html = item.config.pluginmanager.getplugin("html")
             if pytest_html is not None:  # pragma: no cover
                 report.extra = getattr(report, "extra", []) + [
                     pytest_html.extras.text(stats, name="Hypothesis stats")
                 ]
 
     def pytest_terminal_summary(terminalreporter):
-        if not terminalreporter.config.getoption(PRINT_STATISTICS_OPTION):
-            return
-        terminalreporter.section("Hypothesis Statistics")
-
-        def report(properties):
-            for name, value in properties:
-                if name.startswith("hypothesis-statistics-"):
-                    if hasattr(value, "uniobj"):
-                        # Under old versions of pytest, `value` was a `py.xml.raw`
-                        # rather than a string, so we get the (unicode) string off it.
-                        value = value.uniobj
-                    line = base64.b64decode(value.encode()).decode() + "\n\n"
-                    terminalreporter.write_line(line)
-
-        try:
-            global_properties = terminalreporter.config._xml.global_properties
-        except AttributeError:
-            # terminalreporter.stats is a dict, where the empty string appears to
-            # always be the key for a list of _pytest.reports.TestReport objects
-            for test_report in terminalreporter.stats.get("", []):
-                if test_report.when == "teardown":
-                    report(test_report.user_properties)
-        else:
-            report(global_properties)
+        if terminalreporter.config.getoption(PRINT_STATISTICS_OPTION):
+            terminalreporter.section("Hypothesis Statistics")
+            for reports in terminalreporter.stats.values():
+                for report in reports:
+                    stats = report.__dict__.get(STATS_KEY)
+                    if stats:
+                        terminalreporter.write_line(stats + "\n\n")
 
     def pytest_collection_modifyitems(items):
         if "hypothesis" not in sys.modules: