Skip to content

TST: skip gbq test again / fix parsers/test_network url #15407

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

TST: skip gbq test again / fix parsers/test_network url #15407

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4b97db4
TST: disable gbq tests again
jreback Feb 15, 2017
896812d
TST: fix incorrect url in compressed url network tests in parser
jreback Feb 15, 2017
827ec23
TST: incorrect skip in when --skip-network is run
jreback Feb 15, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pandas/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,5 @@ def pytest_runtest_setup(item):
if 'slow' not in item.keywords and item.config.getoption("--only-slow"):
pytest.skip("skipping due to --only-slow")

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TomAugspurger FYI

if 'skip' in item.keywords and item.config.getoption("--skip-network"):
if 'network' in item.keywords and item.config.getoption("--skip-network"):
pytest.skip("skipping due to --skip-network")
56 changes: 25 additions & 31 deletions pandas/tests/io/parser/test_network.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,42 +15,36 @@
from pandas.io.parsers import read_csv, read_table


class TestCompressedUrl(object):
@pytest.fixture(scope='module')
def salaries_table():
path = os.path.join(tm.get_data_path(), 'salaries.csv')
return read_table(path)

compression_to_extension = {
'gzip': '.gz',
'bz2': '.bz2',
'zip': '.zip',
'xz': '.xz',
}

def setup(self):
path = os.path.join(tm.get_data_path(), 'salaries.csv')
self.local_table = read_table(path)
self.base_url = ('https://github.com/pandas-dev/pandas/raw/master/'
'pandas/io/tests/parser/data/salaries.csv')
@pytest.mark.parametrize(
"compression,extension", [('gzip', '.gz'), ('bz2', '.bz2'),
('zip', '.zip'), ('xz', '.xz')])
def test_compressed_urls(salaries_table, compression, extension):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TomAugspurger I had to do this in an odd way (meaning having the parameterized fixed call another function)

as the @tm.network decorator doesn't play nice in python 2 ONLY. It works in python 3 though.

any idea?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC, decorators need to fully copy the whole argument signature. This is not too bad on Python 3.4+ because functools.wraps does it already, but it's a bit of a pain on 2.7. The easiest is to use the decorator package, though on matplotlib, I just replaced the decorators like that with an automatic fixture and some markers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@QuLogic thanks for the expl.

yeah I think we need to fix tm.network and just make it a marker.. Though if you have suggestions pls speak up! (or just do a PR!).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so you just use decorator.decorator instead of functools.wraps?

if its only for testing purposes, then we could easily do that (though even easier would be to fix our tm.network ....)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The syntax of decorator.decorator is a bit different; it replaces the outer function portion and you only need the wrapper bit, but I didn't pursue it much further.

check_compressed_urls(salaries_table, compression, extension)


@tm.network
def check_compressed_urls(salaries_table, compression, extension):
# test reading compressed urls with various engines and
# extension inference
base_url = ('https://github.com/pandas-dev/pandas/raw/master/'
'pandas/tests/io/parser/data/salaries.csv')

url = base_url + extension

# args is a (compression, engine) tuple
for (c, engine) in product([compression, 'infer'], ['python', 'c']):

@tm.network
def test_compressed_urls(self):
# Test reading compressed tables from URL.
msg = ('Test reading {}-compressed tables from URL: '
'compression="{}", engine="{}"')

for compression, extension in self.compression_to_extension.items():
url = self.base_url + extension
# args is a (compression, engine) tuple
for args in product([compression, 'infer'], ['python', 'c']):
# test_fxn is a workaround for more descriptive nose reporting.
# See http://stackoverflow.com/a/37393684/4651668.
test_fxn = functools.partial(self.check_table)
test_fxn.description = msg.format(compression, *args)
yield (test_fxn, url) + args

def check_table(self, url, compression, engine):
if url.endswith('.xz'):
tm._skip_if_no_lzma()
url_table = read_table(url, compression=compression, engine=engine)
tm.assert_frame_equal(url_table, self.local_table)

url_table = read_table(url, compression=c, engine=engine)
tm.assert_frame_equal(url_table, salaries_table)


class TestS3(tm.TestCase):
Expand Down
16 changes: 8 additions & 8 deletions pandas/tests/io/test_gbq.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ def test_generate_bq_schema_deprecated():
gbq.generate_bq_schema(df)


@pytest.mark.single
@pytest.mark.xfail(run=False, reason="intermittent failures")
class TestGBQConnectorIntegrationWithLocalUserAccountAuth(tm.TestCase):

def setUp(self):
Expand Down Expand Up @@ -299,7 +299,7 @@ def test_get_application_default_credentials_returns_credentials(self):
self.assertTrue(isinstance(credentials, GoogleCredentials))


@pytest.mark.single
@pytest.mark.xfail(run=False, reason="intermittent failures")
class TestGBQConnectorIntegrationWithServiceAccountKeyPath(tm.TestCase):
def setUp(self):
_setup_common()
Expand Down Expand Up @@ -331,7 +331,7 @@ def test_should_be_able_to_get_results_from_query(self):
self.assertTrue(pages is not None)


@pytest.mark.single
@pytest.mark.xfail(run=False, reason="intermittent failures")
class TestGBQConnectorIntegrationWithServiceAccountKeyContents(tm.TestCase):
def setUp(self):
_setup_common()
Expand Down Expand Up @@ -449,7 +449,7 @@ def test_read_gbq_with_corrupted_private_key_json_should_fail(self):
private_key=re.sub('[a-z]', '9', _get_private_key_contents()))


@pytest.mark.single
@pytest.mark.xfail(run=False, reason="intermittent failures")
class TestReadGBQIntegration(tm.TestCase):

@classmethod
Expand Down Expand Up @@ -503,7 +503,7 @@ def test_should_read_as_service_account_with_key_contents(self):
tm.assert_frame_equal(df, DataFrame({'valid_string': ['PI']}))


@pytest.mark.single
@pytest.mark.xfail(run=False, reason="intermittent failures")
class TestReadGBQIntegrationWithServiceAccountKeyPath(tm.TestCase):

@classmethod
Expand Down Expand Up @@ -906,7 +906,7 @@ def test_configuration_without_query(self):
configuration=config)


@pytest.mark.single
@pytest.mark.xfail(run=False, reason="intermittent failures")
class TestToGBQIntegrationWithServiceAccountKeyPath(tm.TestCase):
# Changes to BigQuery table schema may take up to 2 minutes as of May 2015
# As a workaround to this issue, each test should use a unique table name.
Expand Down Expand Up @@ -1219,7 +1219,7 @@ def test_dataset_does_not_exist(self):
DATASET_ID + "_not_found"), 'Expected dataset not to exist')


@pytest.mark.single
@pytest.mark.xfail(run=False, reason="intermittent failures")
class TestToGBQIntegrationWithLocalUserAccountAuth(tm.TestCase):
# Changes to BigQuery table schema may take up to 2 minutes as of May 2015
# As a workaround to this issue, each test should use a unique table name.
Expand Down Expand Up @@ -1277,7 +1277,7 @@ def test_upload_data(self):
self.assertEqual(result['num_rows'][0], test_size)


@pytest.mark.single
@pytest.mark.xfail(run=False, reason="intermittent failures")
class TestToGBQIntegrationWithServiceAccountKeyContents(tm.TestCase):
# Changes to BigQuery table schema may take up to 2 minutes as of May 2015
# As a workaround to this issue, each test should use a unique table name.
Expand Down