Skip to content

test: prevents query cache reuse that seems to trigger dtypes test failure #557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Sep 2, 2022
Merged

test: prevents query cache reuse that seems to trigger dtypes test failure #557

merged 6 commits into from
Sep 2, 2022

Conversation

chalmerlowe
Copy link
Collaborator

@chalmerlowe chalmerlowe commented Aug 17, 2022 •
edited
Loading

This PR is intended to determine whether cached results are causing a test to fail.

NOTE: it includes a fix from another PR that was also causing tests to fail (i.e. the dataset name had changed so we need to point to the correct dataset).

Used to point to: bigquery-public-data.new_york_taxi_trips.tlc_green_trips_2014
Now points to: bigquery-public-data.new_york.tlc_green_trips_2014

Fixes #554 🦕

@product-auto-label product-auto-label bot added size: xs Pull request size is extra small. api: bigquery Issues related to the googleapis/python-bigquery-pandas API. labels Aug 17, 2022
@chalmerlowe chalmerlowe changed the title Dtypes test failure fix: prevents query cache reuse that seems to trigger dtypes test failure Aug 18, 2022
@chalmerlowe chalmerlowe marked this pull request as ready for review August 18, 2022 16:22
@chalmerlowe chalmerlowe requested a review from a team as a code owner August 18, 2022 16:22
@chalmerlowe chalmerlowe requested review from a team and prash-mi August 18, 2022 16:22
@chalmerlowe chalmerlowe added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 18, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 18, 2022
@parthea parthea added the owlbot:run Add this label to trigger the Owlbot post processor. label Aug 23, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Aug 23, 2022
@parthea parthea changed the title fix: prevents query cache reuse that seems to trigger dtypes test failure test: prevents query cache reuse that seems to trigger dtypes test failure Aug 23, 2022
@chalmerlowe
Copy link
Collaborator Author

Re: the Kokoro prerelease failure:

Based on conversation in the pandas github issue tracker, the root cause of these failing tests appears to be that the pandas system is trying to store too large a number in an int32 memory location. The bug has been added to the pandas 1.5 milestone (pandas is currently at 1.5.0 release candidate stage)

See: pandas-dev/pandas#48255

leahecole
leahecole reviewed Sep 1, 2022
View reviewed changes
tests/system/test_read_gbq_with_bqstorage.py
@@ -50,7 +50,7 @@ def test_large_results(random_dataset, method_under_test):
total_amount,
passenger_count,
trip_distance
FROM `bigquery-public-data.new_york_taxi_trips.tlc_green_trips_2014`
FROM `bigquery-public-data.new_york.tlc_green_trips_2014`
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh, I know the pain behind this change 🚕

leahecole
leahecole reviewed Sep 1, 2022
View reviewed changes
Copy link

@leahecole leahecole left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but will switch to approval after confirmation that we don't also need to add a test that has UseQueryCache: True

@chalmerlowe chalmerlowe added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 1, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 1, 2022
@product-auto-label product-auto-label bot added size: s Pull request size is small. and removed size: xs Pull request size is extra small. labels Sep 1, 2022
@chalmerlowe
Copy link
Collaborator Author

@leahecole per our conversation, added a comment to the code to identify why the useQueryCache=False argument was added to this specific test to avoid issues with potentially cached and mismatched data.

Please approve. Thanks.

@chalmerlowe
Copy link
Collaborator Author

@leahecole would it be possible to get an approval on this?
The pre-release failure is due to a known pandas issue, as noted above.

@chalmerlowe chalmerlowe merged commit 5365ccc into main Sep 2, 2022
@chalmerlowe chalmerlowe deleted the dtypes-test-failure branch September 2, 2022 10:56
@chalmerlowe
Copy link
Collaborator Author

Fixes #545

@chalmerlowe chalmerlowe mentioned this pull request Sep 6, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@leahecole leahecole leahecole left review comments

@parthea parthea parthea approved these changes

@prash-mi prash-mi Awaiting requested review from prash-mi

Assignees
No one assigned
Labels
api: bigquery Issues related to the googleapis/python-bigquery-pandas API. size: s Pull request size is small.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

tests.system.test_read_gbq_with_bqstorage: test_large_results failed
4 participants
@chalmerlowe @parthea @leahecole @yoshi-kokoro