TST: GH18498 - Split test_pytables into sub-modules #19735

mikebarkas · 2018-02-17T01:25:10Z

closes TST: split out test_pytables.py to sub-module of tests #18498
tests moved / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

Moved test classes for pytables into sub-modules.
Moved hdf test data files needed into sub-module.

I did not change any tests, just broke up the classes into separate modules.

This commit moves the existing test classes out of the pytables test module into sub-modules.

These tests now live in the pytables sub-module.

pep8speaks · 2018-02-17T01:25:14Z

Hello @mikebarkas! Thanks for updating the PR.

In the file pandas/tests/io/pytables/common.py, following are the PEP8 issues :

Line 67:9: E722 do not use bare except'
Line 75:5: E722 do not use bare except'
Line 84:5: E722 do not use bare except'

Comment last updated on April 29, 2018 at 17:58 Hours UTC

mikebarkas · 2018-02-17T01:30:10Z

PEP8 issue for common.py
do not use bare except This is in existing code.
This issue is for moving the tests and not changing test.

codecov · 2018-02-17T03:02:24Z

Codecov Report

Merging #19735 into master will decrease coverage by 0.24%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #19735      +/-   ##
==========================================
- Coverage   91.82%   91.58%   -0.25%     
==========================================
  Files         152      150       -2     
  Lines       49248    48908     -340     
==========================================
- Hits        45222    44790     -432     
- Misses       4026     4118      +92

Flag	Coverage Δ
#multiple	`91.56% <ø> (+1.35%)`	⬆️
#single	`32.81% <ø> (-9.09%)`	⬇️

Impacted Files	Coverage Δ
pandas/tseries/plotting.py	`0% <0%> (-100%)`	⬇️
pandas/core/dtypes/base.py	`47.61% <0%> (-44.28%)`	⬇️
pandas/plotting/_compat.py	`62% <0%> (-28.91%)`	⬇️
pandas/core/arrays/base.py	`60% <0%> (-24.15%)`	⬇️
pandas/io/s3.py	`72.72% <0%> (-13.64%)`	⬇️
pandas/core/missing.py	`85.9% <0%> (-5.75%)`	⬇️
pandas/plotting/_timeseries.py	`60.82% <0%> (-4.49%)`	⬇️
pandas/io/formats/console.py	`66.66% <0%> (-3.04%)`	⬇️
pandas/io/html.py	`85.98% <0%> (-2.81%)`	⬇️
pandas/io/formats/format.py	`96.25% <0%> (-2%)`	⬇️
... and 58 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cdfce2b...5052897. Read the comment docs.

jreback · 2018-02-18T15:45:56Z

pandas/tests/io/pytables/common.py

+import pandas.util.testing as tm
+
+
+tables = pytest.importorskip('tables')


if things are not used here, remove them

jreback · 2018-02-18T15:47:08Z

pandas/tests/io/pytables/common.py

+# contextmanager to ensure the file cleanup
+
+
+def safe_remove(path):


would like to tun these into fixtures, also this removes the need for the Base class entirely. ok with doing this in another pass though

jreback · 2018-02-18T15:47:31Z

pandas/tests/io/pytables/test_hdf_complex_values.py

@@ -0,0 +1,163 @@
+import pytest
+from warnings import catch_warnings
+


you can just name this test_complex.py

jreback · 2018-02-18T15:52:59Z

pandas/tests/io/pytables/test_hdf_store.py

-    def setup_method(self, method):
-        self.path = 'tmp.__%s__.h5' % tm.rands(10)
-
-    def teardown_method(self, method):


the idea is to split this out into many submodules, e.g.

test_store (construct & instrospect a store)
test_append
test_multiindex
test_remove
test_where
test_select

basically just select similar tests (by name to start is ok) and group them into sub-modules

jreback · 2018-02-18T15:53:21Z

pandas/tests/io/pytables/test_timezones.py

+
+    def test_append_with_timezones_dateutil(self):
+
+        from datetime import timedelta


move any imports to the top

Removed test classes, created test functions Grouped tests into sub-modules Cleaned up imports, comments, and spaces

jreback · 2018-02-22T02:29:58Z

@mikebarkas looks great!

oddly there is 1 less tests here (174 and 1 xfail) vs 175 and 1 xfail on master.

happy to have parameterization of any tests as well.

mikebarkas · 2018-02-22T02:36:52Z

My test build on my fork is failing because ImportError: No module named tables
Looking into it.

I will look for the missing test.

mikebarkas · 2018-02-22T03:27:05Z

This is my missing test

def test_longpanel(self):
    pass

I will add it back into test_panel

jreback · 2018-02-22T03:30:02Z

oh no need to add that back then

jreback

a bit too much splitting. idea is to put common files together. tests might be named inconsitenly, but still to group them together, we should end up with only about 5 ish modules here overall.

jreback · 2018-02-23T11:26:45Z

pandas/tests/io/pytables/test_common.py

+from pandas import (Series, DataFrame, date_range, Index, Timestamp,
+                    DatetimeIndex, compat, concat, isna)
+tables = pytest.importorskip('tables')
+from pandas.tests.io.pytables.common import (ensure_clean_store, safe_close,


you can just do a

from .common import .....

(and do this generally)

jreback · 2018-02-23T11:27:04Z

pandas/tests/io/pytables/test_common.py

+
+import pandas.util.testing as tm
+import pandas.util._test_decorators as td
+from pandas.util.testing import (assert_panel_equal,


prefer not to import these and just reference with tm.

also do this generally across all files

jreback · 2018-02-23T11:28:40Z

pandas/tests/io/pytables/test_date.py

+
+import numpy as np
+import pandas as pd
+from pandas import Series, DataFrame, date_range, timedelta_range, bdate_range


can you rename to test_datetime

jreback · 2018-02-23T11:29:38Z

pandas/tests/io/pytables/test_panel.py

+    with catch_warnings(record=True):
+        wp = tm.makePanel()
+        _check_roundtrip(wp, assert_panel_equal)
+


there are may more panel tests, can you move here

jreback · 2018-02-23T11:30:40Z

pandas/tests/io/pytables/test_path.py

+
+        store.close()
+        assert 'CLOSED' in store.info()
+        assert not store.is_open


these need to go with any other open/close tests, which I think you have in test_common.py now. call this file test_open_close (move this one and other tests)

jreback · 2018-02-23T11:31:06Z

pandas/tests/io/pytables/test_read.py

+import pytest
+from warnings import catch_warnings
+
+import numpy as np


there are many more selection / query type tests that are in test_common. move here

jreback · 2018-02-23T11:31:26Z

pandas/tests/io/pytables/test_read.py

@@ -0,0 +1,270 @@
+import pytest


rename this to test_query

jreback · 2018-02-23T11:31:56Z

pandas/tests/io/pytables/test_select.py

+import numpy as np
+from pandas import (Series, DataFrame, date_range, Index, Timestamp,
+                    concat, MultiIndex, bdate_range, Panel)
+from pandas.tests.io.pytables.common import (ensure_clean_store,


consolidate with test_read (and call test_query)

jreback · 2018-02-23T11:32:33Z

pandas/tests/io/pytables/test_start_stop.py

+from pandas.tests.io.pytables.common import ensure_clean_store
+import pandas.util.testing as tm
+
+


move these to test_select, we don't need a separate file for this

jreback · 2018-02-23T11:32:56Z

pandas/tests/io/pytables/test_store.py

+    index = MultiIndex(levels=[['foo', 'bar', 'baz', 'qux'],
+                               ['one', 'two', 'three']],
+                       labels=[[0, 0, 0, 1, 1, 2, 2, 3, 3, 3],
+                               [0, 1, 2, 0, 1, 1, 2, 0, 1, 2]],


these need to be put in the right place, not a separate file

alsobay · 2018-04-29T10:03:08Z

Hi Mike, are you still working on this? Would be glad to help out or pick it up if you're too busy.

Consolidate functions to reduce amount of module files.

mikebarkas · 2018-04-29T18:18:59Z

@alsobay I broke the build with latest changes, in a different part of the test suite. I have not had time to determine where and to fix it.

I am currently moving and have little free time. I would appreciate the help to get this PR finished.

jreback · 2018-07-28T14:36:46Z

closing as stale, happy to take an updated PR of this

Mike Barkas added 3 commits February 15, 2018 21:23

TST: GH18498: Move existing tests out of test_pytables module

f148d0c

This commit moves the existing test classes out of the pytables test module into sub-modules.

TST: GH18498: Removed io/test_pytables module

f9ecbf3

These tests now live in the pytables sub-module.

TST: GH:18498 Move legacy hdf data files into pytables sub-module

65419ce

mikebarkas changed the title ~~Gh18498~~ Gh18498 - Spit test_pytables into sub-modules Feb 17, 2018

mikebarkas changed the title ~~Gh18498 - Spit test_pytables into sub-modules~~ TST: GH18498 - Spit test_pytables into sub-modules Feb 17, 2018

mikebarkas changed the title ~~TST: GH18498 - Spit test_pytables into sub-modules~~ TST: GH18498 - Split test_pytables into sub-modules Feb 17, 2018

jreback added Testing pandas testing functions or related to the test suite IO HDF5 read_hdf, HDFStore labels Feb 18, 2018

jreback requested changes Feb 18, 2018

View reviewed changes

TST: GH:18498 Move test methods into sub-modules

27e8cc4

Removed test classes, created test functions Grouped tests into sub-modules Cleaned up imports, comments, and spaces

TST: GH:18498 Conditionally test with tables import

bd75829

jreback requested changes Feb 23, 2018

View reviewed changes

TST: GH18498 Requested pull request changes

5052897

Consolidate functions to reduce amount of module files.

jreback closed this Jul 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TST: GH18498 - Split test_pytables into sub-modules #19735

TST: GH18498 - Split test_pytables into sub-modules #19735

mikebarkas commented Feb 17, 2018

pep8speaks commented Feb 17, 2018 •

edited

Loading

mikebarkas commented Feb 17, 2018

codecov bot commented Feb 17, 2018 •

edited

Loading

jreback Feb 18, 2018

jreback Feb 18, 2018

jreback Feb 18, 2018

jreback Feb 18, 2018

jreback Feb 18, 2018

jreback commented Feb 22, 2018

mikebarkas commented Feb 22, 2018

mikebarkas commented Feb 22, 2018

jreback commented Feb 22, 2018

jreback left a comment

jreback Feb 23, 2018

jreback Feb 23, 2018

jreback Feb 23, 2018

jreback Feb 23, 2018

jreback Feb 23, 2018

jreback Feb 23, 2018

jreback Feb 23, 2018

jreback Feb 23, 2018

jreback Feb 23, 2018

jreback Feb 23, 2018

jreback Feb 23, 2018

alsobay commented Apr 29, 2018

mikebarkas commented Apr 29, 2018

jreback commented Jul 28, 2018

		import pandas.util.testing as tm


		tables = pytest.importorskip('tables')

		# contextmanager to ensure the file cleanup


		def safe_remove(path):

		@@ -0,0 +1,163 @@
		import pytest
		from warnings import catch_warnings


		def test_append_with_timezones_dateutil(self):

		from datetime import timedelta

		from pandas.tests.io.pytables.common import ensure_clean_store
		import pandas.util.testing as tm

TST: GH18498 - Split test_pytables into sub-modules #19735

TST: GH18498 - Split test_pytables into sub-modules #19735

Conversation

mikebarkas commented Feb 17, 2018

pep8speaks commented Feb 17, 2018 • edited Loading

Comment last updated on April 29, 2018 at 17:58 Hours UTC

mikebarkas commented Feb 17, 2018

codecov bot commented Feb 17, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Feb 22, 2018

mikebarkas commented Feb 22, 2018

mikebarkas commented Feb 22, 2018

jreback commented Feb 22, 2018

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alsobay commented Apr 29, 2018

mikebarkas commented Apr 29, 2018

jreback commented Jul 28, 2018

pep8speaks commented Feb 17, 2018 •

edited

Loading

codecov bot commented Feb 17, 2018 •

edited

Loading