Skip to content

Resolve ImportError in read_gbq which appears with oauthclient >= 2.0 #12582

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Resolve ImportError in read_gbq which appears with oauthclient >= 2.0 #12582

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ci/requirements-2.7.pip
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,6 @@ blosc
httplib2
google-api-python-client == 1.2
python-gflags == 2.0
oauth2client == 1.5.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gr8!

pathlib
py
1 change: 1 addition & 0 deletions ci/requirements-3.4.pip
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ python-dateutil==2.2
blosc
httplib2
google-api-python-client
oauth2client
8 changes: 5 additions & 3 deletions doc/source/install.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,9 +267,11 @@ Optional Dependencies
<http://www.vergenet.net/~conrad/software/xsel/>`__, or `xclip
<http://sourceforge.net/projects/xclip/>`__: necessary to use
:func:`~pandas.io.clipboard.read_clipboard`. Most package managers on Linux distributions will have ``xclip`` and/or ``xsel`` immediately available for installation.
* Google's `python-gflags <http://code.google.com/p/python-gflags/>`__
and `google-api-python-client <http://github.com/google/google-api-python-client>`__: Needed for :mod:`~pandas.io.gbq`
* `httplib2 <http://pypi.python.org/pypi/httplib2>`__: Needed for :mod:`~pandas.io.gbq`
* Google's `python-gflags <http://code.google.com/p/python-gflags/>`__ ,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can u add in oauth2client into the appropriate ci/requirements files (eg where the existing reward are)

`oauth2client <https://github.com/google/oauth2client>`__ ,
`httplib2 <http://pypi.python.org/pypi/httplib2>`__
and `google-api-python-client <http://github.com/google/google-api-python-client>`__
: Needed for :mod:`~pandas.io.gbq`
* One of the following combinations of libraries is needed to use the
top-level :func:`~pandas.io.html.read_html` function:

Expand Down
1 change: 1 addition & 0 deletions doc/source/whatsnew/v0.18.0.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1271,3 +1271,4 @@ Bug Fixes
- Bug when specifying a UTC ``DatetimeIndex`` by setting ``utc=True`` in ``.to_datetime`` (:issue:`11934`)
- Bug when increasing the buffer size of CSV reader in ``read_csv`` (:issue:`12494`)
- Bug when setting columns of a ``DataFrame`` with duplicate column names (:issue:`12344`)
- Resolve ImportError in ``read_gbq`` which appears with oauthclient >= 2.0.0 (:issue:`12572`)
41 changes: 34 additions & 7 deletions pandas/io/gbq.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,6 @@ def _test_google_api_imports():
from apiclient.errors import HttpError # noqa
from oauth2client.client import AccessTokenRefreshError # noqa
from oauth2client.client import OAuth2WebServerFlow # noqa
from oauth2client.client import SignedJwtAssertionCredentials # noqa
from oauth2client.file import Storage # noqa
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was added for 0.18.0 (IOW we didn't release a version with this?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, 0.17.x and earlier do not need this fix

from oauth2client.tools import run_flow, argparser # noqa
except ImportError as e:
Expand Down Expand Up @@ -179,7 +178,30 @@ def get_user_account_credentials(self):
return credentials

def get_service_account_credentials(self):
from oauth2client.client import SignedJwtAssertionCredentials
# Bug fix for https://github.com/pydata/pandas/issues/12572
# We need to know that a supported version of oauth2client is installed
# Test that either of the following is installed:
# - SignedJwtAssertionCredentials from oauth2client.client
# - ServiceAccountCredentials from oauth2client.service_account
# SignedJwtAssertionCredentials is available in oauthclient < 2.0.0
# ServiceAccountCredentials is available in oauthclient >= 2.0.0
oauth2client_v1 = True
oauth2client_v2 = True

try:
from oauth2client.client import SignedJwtAssertionCredentials
except ImportError:
oauth2client_v1 = False

try:
from oauth2client.service_account import ServiceAccountCredentials
except ImportError:
oauth2client_v2 = False

if not oauth2client_v1 and not oauth2client_v2:
raise ImportError("Missing oauth2client required for BigQuery "
"service account support")

from os.path import isfile

try:
Expand All @@ -197,11 +219,16 @@ def get_service_account_credentials(self):
json_key['private_key'] = bytes(
json_key['private_key'], 'UTF-8')

return SignedJwtAssertionCredentials(
json_key['client_email'],
json_key['private_key'],
self.scope,
)
if oauth2client_v1:
return SignedJwtAssertionCredentials(
json_key['client_email'],
json_key['private_key'],
self.scope,
)
else:
return ServiceAccountCredentials.from_json_keyfile_dict(
json_key,
self.scope)
except (KeyError, ValueError, TypeError, AttributeError):
raise InvalidPrivateKeyFormat(
"Private key is missing or invalid. It should be service "
Expand Down
25 changes: 24 additions & 1 deletion pandas/io/tests/test_gbq.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,6 @@ def _test_imports():

from oauth2client.client import OAuth2WebServerFlow # noqa
from oauth2client.client import AccessTokenRefreshError # noqa
from oauth2client.client import SignedJwtAssertionCredentials # noqa

from oauth2client.file import Storage # noqa
from oauth2client.tools import run_flow # noqa
Expand Down Expand Up @@ -115,6 +114,30 @@ def _test_imports():
raise ImportError(
"pandas requires httplib2 for Google BigQuery support")

# Bug fix for https://github.com/pydata/pandas/issues/12572
# We need to know that a supported version of oauth2client is installed
# Test that either of the following is installed:
# - SignedJwtAssertionCredentials from oauth2client.client
# - ServiceAccountCredentials from oauth2client.service_account
# SignedJwtAssertionCredentials is available in oauthclient < 2.0.0
# ServiceAccountCredentials is available in oauthclient >= 2.0.0
oauth2client_v1 = True
oauth2client_v2 = True

try:
from oauth2client.client import SignedJwtAssertionCredentials # noqa
except ImportError:
oauth2client_v1 = False

try:
from oauth2client.service_account import ServiceAccountCredentials # noqa
except ImportError:
oauth2client_v2 = False

if not oauth2client_v1 and not oauth2client_v2:
raise ImportError("Missing oauth2client required for BigQuery "
"service account support")


def test_requirements():
try:
Expand Down