Skip to content

TST/REF: Fixturize constant functions in ConsistencyBase #33943

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
May 10, 2020

Conversation

charlesdong1991
Copy link
Member

@charlesdong1991 charlesdong1991 commented May 2, 2020

This PR fixturizes the last constants (base_functions and no_nan_functions) in ConsistencyBase, and take corresponding tests out of class.

After this one, only constants in Base class needs to be fixed (if Base is also wanted to be cleaned)

cc @jreback

@@ -1477,6 +1476,7 @@ def test_rolling_consistency_std(consistency_data, window, min_periods, center):
)


@pytest.mark.slow
Copy link
Member Author

@charlesdong1991 charlesdong1991 May 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jbrockmendel
thanks for your comment in #33813 (comment) , indeed, i tested the execution time and seems around 40s which is quite long and should mark as slow. And the execution time of expanding_consistency_cov test is okay (~4 to 5 s)

since this change is very tiny, i think it is okay to just change it here although it is out of scope of this PR.

@charlesdong1991 charlesdong1991 marked this pull request as draft May 3, 2020 14:59
@charlesdong1991 charlesdong1991 marked this pull request as ready for review May 3, 2020 16:12
@@ -18,3 +18,51 @@ def binary_ew_data():
@pytest.fixture(params=[0, 1, 2])
def min_periods(request):
return request.param


base_functions_list = [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you document the tuples here (e.g. what they refer to)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, i added in the docs

# GH 8269
if no_nans:
functions = base_functions + no_nan_functions
for (f, require_min_periods, name) in functions:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e.g in an ideal case we don't have this loop here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now it may be that this increases the number of tests by 10x so we don'twant to do this, but have a look see

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, refer to the comment above

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep sounds good

@jreback jreback added Testing pandas testing functions or related to the test suite Window rolling, ewma, expanding labels May 5, 2020
@jreback jreback added this to the 1.1 milestone May 6, 2020
@jreback
Copy link
Contributor

jreback commented May 6, 2020

are tests here there any that need marking as slow?

@charlesdong1991
Copy link
Member Author

charlesdong1991 commented May 6, 2020

if 5s per test function is acceptable, then we are fine for tests in expanding i think

i marked two functions as slow in rolling one, one is above which replied to @jbrockmendel comment, that test takes 40s, and the other is test_rolling_apply_consistency which is involved in the change in this pr, and this takes ~105s.

generally tests in rolling takes longer and for the rest of tests in rolling, basically all <15s, and in total ~250s, so maybe it's okay to leave them not as slow? @jreback

@charlesdong1991
Copy link
Member Author

charlesdong1991 commented May 6, 2020

I will double check a bit, but would be nice to get the ballpark for the threshold for slow?

Update:

other relatively big ones:
test_rolling_consistency_std: ~20s
test_rolling_consistency_var: ~25s

i also mark them as slow

@charlesdong1991
Copy link
Member Author

emm, the error does not seem related.

@charlesdong1991
Copy link
Member Author

gentle ping @jreback

@charlesdong1991 charlesdong1991 requested a review from jreback May 10, 2020 06:58
@jreback jreback merged commit b3f54d7 into pandas-dev:master May 10, 2020
@jreback
Copy link
Contributor

jreback commented May 10, 2020

thanks @charlesdong1991 very nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Testing pandas testing functions or related to the test suite Window rolling, ewma, expanding
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants