Gbq service account #11881
Closed
Gbq service account #11881
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from oauth2client.client import OAuth2WebServerFlow | ||
| from oauth2client.file import Storage | ||
| from oauth2client.tools import run_flow, argparser | ||
|
|
||
| _check_google_client_version() | ||
|
|
||
| flow = OAuth2WebServerFlow(client_id='495642085510-k0tmvj2m941jhre2nbqka17vqpjfddtd.apps.googleusercontent.com', | ||
| client_secret='kOc9wMptUtxkcIFbtZCcrEAc', | ||
| scope='https://www.googleapis.com/auth/bigquery', |
There was a problem hiding this comment.
TODO: use self.scope here
|
the |
|
OK, I've added |
tworec
force-pushed
the
gbq_service_account
branch
from
6b67450 to
8fa76e7
Compare
| @@ -37,6 +39,10 @@ def _check_google_client_version(): | |||
| logger = logging.getLogger('pandas.io.gbq') | |||
| logger.setLevel(logging.ERROR) | |||
|
|
|||
| def _print(msg, end='\n'): | |||
| sys.stdout.write(msg + end) | |||
There was a problem hiding this comment.
you should pass verbose to this (and do the if inside here). That way you can simply:
_print(msg, verbose=verbose) in the code itself.
|
pls add a whatsnew entry (enhancements). |
|
cc @parthea pls have a look |
tworec
force-pushed
the
gbq_service_account
branch
from
8914df3 to
cd45e22
Compare
|
pls rebase / squash |
tworec
force-pushed
the
gbq_service_account
branch
from
cd45e22 to
2d12d6d
Compare
|
ok, rabased and squashed |
|
cc @parthea any comments |
tworec
force-pushed
the
gbq_service_account
branch
from
2d12d6d to
4ba4401
Compare
|
I'm receiving the following exception related to the
I believe Travis will also have this issue. Travis is currently skipping integration tests that require a BigQuery project id, so the issue is not reported by Travis. Should we add a dependency for |
|
we don't need any more deps use time.time() |
tworec
force-pushed
the
gbq_service_account
branch
from
4ba4401 to
0d036cc
Compare
|
Ad FileNotFoundError) I'm running tests on python 3 only, where this exception is built-in, hence I've missed this out. After adding support for json contents handling this exception is not needed any more. See line 168 in gbq.py, where I'm checking file exists and is regular file. So file not found exception can not happend. I'm removing it from except clause. Doing this I've also rethinked, rewritten and tested invalid auth scenarios. Ad json.load) it works for me. I cant reproduce it under python3 :( |
|
|
Sample session |
|
|
||
| def __init__(self, project_id, reauth=False): | ||
| def __init__(self, project_id, reauth=False, verbose=True, private_key=None): |
There was a problem hiding this comment.
let's default to verbose=False. you could have an option do this I suppose, e.g. pd.options.gbq.verbose
|
pls |
|
needs a rebase; most of the codebase underwent a PEP cleanup very recently. |
|
|
||
| You will be authenticated to the specified BigQuery account via Google's Oauth2 mechanism. | ||
|
|
||
| Primary auth method is as simple as following the |
There was a problem hiding this comment.
The primary authentication
|
| ) | ||
| except (KeyError, ValueError, TypeError, AttributeError): | ||
| raise InvalidPrivateKeyFormat("Service account private key should be valid JSON (file path or string contents) " | ||
| "with at least two keys: 'client_email' and 'private_key'. Can be obtained from google developers console. ") |
There was a problem hiding this comment.
Can we make a small change in the error message to indicate that file may be missing?
raise InvalidPrivateKeyFormat("Private key is missing or invalid. Service account private key should be valid JSON (file path or string contents) with at least two keys: 'client_email' and 'private_key'. Can be obtained from google developers console. ")
fefd245 to
2e5b404
Compare
tworec
force-pushed
the
gbq_service_account
branch
from
1917703 to
e035747
Compare
|
|
||
| .. note:: | ||
|
|
||
| The `'private_key'` parameter can be set to either the file path of the service account key in JSON format, or |
There was a problem hiding this comment.
need indenting to the note (same below)
|
@tworec just a couple of more stylistic changes to conform to the rest of the code base. almost there! |
|
gr8! all style fixes done. i'm running local tests and pushing it. please review. |
|
All 73 tests have passed both on py3.5 and py2.7 |
|
thanks! pls review the built docs and if necessary issue a followup for fixes http://pandas-docs.github.io/pandas-docs-travis/ (prob will take a few hours, lots of things in the queue :) |
|
It was my first PR. Thanks for feedback. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds service account authentication while still supports standard web auth method.
It also adds some useful std out messages: progress with elapsed time and percentage
+ price calculation.In RTBHouse we're using this service account auth since may. It works perfectly with remote jupyter server (iPython notebooks).
fixes #8489