[WIP] implement tests using hypothesis

[jbrockmendel]

[skipci]

Related: #17978

Exposes a whole bunch of broken cases. Many of them look like they are not currently caught because the large majority of offset tests use tz-naive pydatetime inputs.

Note this does not fix these bugs. The goal here is a Proof Of Concept for using hypothesis and discuss if/how we can make it useful.

It also doesn't add hypothesis to the requirements because I have no idea what the appropriate file is for that.

[jreback]

so you could add this to ci/requirements_dev.txt. but also in this test module you need a pytest.importorskip('hypthosesis')

[jreback]

are these determinstic tests on a single run? e.g. we are not saving state for travis runs

[pganssle]

Why not just datetime(1900, 1, 1)?

[jbrockmendel]

I think this was copy/pasted from somewhere else where it was originally pd.Timestamp.min, had to adjust to avoid OverfowErrors.

[pganssle]

@jreback I don't think hypothesis tests are deterministic, but in my experience they are "deterministic enough" (they will usually find the most significant edge cases very quickly - they don't sample the space randomly). If you find edge cases, rather than trying to save the state of the database between runs, you can add specific examples with the @hypothesis.example decorator.

[pganssle]

I have been briefly in contact with @DRMacIver about adding a hypothesis.extras.dateutil extra to do just this. It may require some additional public interface stuff on the dateutil side.

That said, it seems that all timezones provided by pytz map identically to dateutil zones, for all datetimes less than 2038-01-01. To prove it, here's a hypothesis test I ran:

from hypothesis import given, assume
from hypothesis import strategies as st
from hypothesis.extra import pytz as hepytz

from dateutil import tz
from datetime import datetime

@given(dt=st.datetimes(), tzi=hepytz.timezones())
def test_dateutil_compat(dt, tzi):
    tzi_du = tz.gettz(str(tzi))
    dt_pytz = tzi.localize(dt)
    dt_du = dt.replace(tzinfo=tzi_du)

    assume(dt < datetime(2038, 1, 1))
    assert dt_pytz == dt_du

So you should be able to get dateutil zones from tz.gettz(str(pytz_zone))

See dateutil/dateutil#590 for the reason why the assume(dt < datetime(2038, 1, 1) is in there.

[pganssle]

Also, does this need to be a function? Why not just assign timezone_strategy = st.one_of([None, hepytz.timezones()])?

[jbrockmendel]

Also, does this need to be a function?

Probably not. I'm still getting the hang of @composite and, data, and draw.

[jbrockmendel]

So you should be able to get dateutil zones from tz.gettz(str(pytz_zone))

That looks great, thanks. I'll take a look at dateutil-590 to see if I can be helpful.

[pganssle]

I think you need to add some assumptions here to make sure that your year, month, week, day, etc values are valid. You can do this by drawing them from integers with min/max set, drawing them as you are and then adding assume statements, or (and this might be a bit out there), draw a random datetime() and just use the values from that (won't work for week, but you can calculate the week from the datetime if you want to use it, I imagine).

[jbrockmendel]

Oh most definitely. I just threw those in and figured I'd put in the appropriate bounds one-by-one as errors came up.

[jbrockmendel]

After a couple of days of poking at this, I think it is worth implementing, but it is much too slow to run in the CI.

Running locally has turned up dozens of broken cases. To the extent that we can phrase tests as "for all A such that B, assert C" this can be a useful tool for identifying corner cases. When they are found they should just be made into regular tests (and uh, fixed).

[pganssle]

After a couple of days of poking at this, I think it is worth implementing, but it is much too slow to run in the CI.

You can always tune the hypothesis tests to reduce the amount run on any given CI run. Maybe on each PR you run 50 test cases or something and then set up a master branch build that runs once a week that runs 500 or so.

[jbrockmendel]

I'm still interested in this, but closing because it is not actionable at the moment.