Investigate using Hypothesis for some tests

[TomAugspurger]

Hypothesis recently added support for generating pandas objects: http://hypothesis.readthedocs.io/en/latest/numpy.html#pandas

This could be useful for testing some of the higher-level methods like melt, merge, etc.

Are there any objections to adding some hypothesis-based tests to our test suite? Is anyone familiar with hypothesis and interested in putting together examples of how we could use it?

[TomAugspurger]

As another example, I could see Hypothesis being very useful for something like #13432 (buggy index set ops for duplicates, non monotonic, categorical)

[shoyer]

This sounds like it's definitely worth trying! Is there a guarantee that the test cases it produces will always be the same? (Non-reproducible tests could be pretty painful.)

[TomAugspurger]

I believe that once a failing test case is found it's cached (somehow, not sure on the details.)

…

On Wed, Oct 25, 2017 at 10:49 AM, Stephan Hoyer ***@***.***> wrote: This sounds like it's definitely worth trying! Is there a guarantee that the test cases it produces will always be the same? (Non-reproducible tests could be pretty painful.) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#17978 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIvYHXq8I4yFc8irTLznyt0jUidU7ks5sv1h7gaJpZM4QF4u7> .

[jbrockmendel]

A test database is kept in a .hypothesis directory. Not sure if that plays nicely with CI.

[jbrockmendel]

@TomAugspurger did this idea ever go anywhere? I'm thinking this might be a useful for testing offsets.

[TomAugspurger]

Don't think so, go for it!

[sushobhit27]

Hi, Can I work on this issue. I have recently implemented some unit test cases using hypothesis and therefore have lil bit of exp which I think can be used here.
Infact I have zeroed in some files, where it can be used easily like pandas/tests/tools/test_numeric.py, e.g

[TomAugspurger]

That seems reasonable. I think the main thing is ensuring this doesn't increase our test times too much.

…

On Tue, Mar 27, 2018 at 12:33 PM, sushobhit27 ***@***.***> wrote: Hi, Can I work on this issue. I have recently implemented some unit test cases using hypothesis and therefore have lil bit of exp which I think can be used here. Infact I have zeroed in some files, where it can be used easily like pandas/tests/tools/test_numeric.py, e.g [image: image] <https://user-images.githubusercontent.com/31987769/37984121-0dc4695e-3213-11e8-973b-6834195bffd2.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17978 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIh1OvooYADMoaghVQl0p3DGKp1iqks5tinf9gaJpZM4QF4u7> .

[sushobhit27]

Hypothesis test run time depends on mainly 3 things:
number of examples to run for a test case. (is configurable per test case)
range of the test data like in aforementioned example (bounded by min_size and max_size, therefore this is also configurable)
how internally hypothesis generates different datasets from the range (called shrinking mechanism AFAIK).
We can play with first two options to have some influence on test run time, however it will surely gonna take more time than current pytest test cases as here test cases are constant.

[sushobhit27]

@TomAugspurger, I created a pull request #20590 for this issue by adding hypothesis based test cases in just one file. However travis CI build failed with error "ImportError: No module named 'hypothesis'". Could you please check how it (hypothesis module) can be available, so that tests can be run.

[Zac-HD]

Hi all - I'm a maintainer of Hypothesis and happy user of Pandas, so hopefully I can help with this 😄

• Hypothesis is not deterministic by default - though you can derandomize it, we don't recommend this.
• For obvious reasons though, we put a lot of effort into ensuring that test failures are always reproducible. Short version: we try to print a call you can copy-paste to get the error plus the seed value if necessary. If it looks like that might fail, we have a decorator to force everything to the right internal state and print that code with the base64 data it needs to reproduce everything.
• Failing examples are also persisted to disk, so the test will fail immediately if you rerun the test suite before fixing the bug no matter how unlikely it was to find.
• Test runtime is a reasonable concern; there's not usually much overhead but running a test tens or hundreds of times is inherently slower than running it once! The usual solution is some combination of "don't use it for everything", "turn down the max_examples setting in CI", and "just deal with it".
• All hypothesis tests automatically apply a pytest marker to themselves too, so you can include or exclude them with -m hypothesis or -m "not hypothesis". So it would be easy to only run them with the slow tests, for example.

And my question to Pandas maintainers: what's the best way to add this as a dependency? Hypothesis can be installed from PyPI or from conda-forge, so it's down to whatever would work best in your CI.

[TomAugspurger]

Thanks @Zac-HD , we have #20590 that's seemingly ready to go. Need to find the time to take a look again.

[TomAugspurger]

Actually, @Zac-HD would you mind taking a look in #20590? One of the issues we're facing is that none of the pandas maintainers have really used hypothesis before. We're trying to define some best practices in that PR, since others will mimic it. You may be the ideal person to set those standards for pandas :)

[Zac-HD]

I have three high-level concerns about #20590:

1. Configuration can be centralised in conftest.py, which makes it easier to edit or just override with a command-line argument than using explicit decorators everywhere.

2. I would ditch the util/_hypothesis.py module, and just the Hypothesis api directly. We've put a lot of thought into making it clear, flexible, and orthogonal - and it has much better performance and error reporting than anything I know of downstream. If there's anything Pandas-specific that would be really helpful, we would be happy to add it to hypothesis.extra.pandas upstream so everyone can use it!

3. Pandas would probably get much more value out of Hypothesis by using it for integration-style tests, i.e. for more complex invariants where there are more interacting parts. In Hypothesis' own test suite we still use traditional unit tests (often with pytest.mark.parametrize) to check input validation, because it's just not worth the runtime.
   [WIP] implement tests using hypothesis #18761 has some interesting stuff for that, though it too could be much simpler with more idiomatic use of Hypothesis!

So rather than tread on toes by suggesting detailed edits to those PRs, I might open a third one this weekend incorporating some of both and some of my own notes.