Skip to content

project_id as optional #127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 25 commits into from
Apr 25, 2018
Merged

project_id as optional #127

Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
c85a685
project_id is optional
max-sixty Feb 20, 2018
8786e6d
Merge branch 'master' into default-project
max-sixty Mar 31, 2018
17a2ede
gitignore
max-sixty Mar 31, 2018
86b6fc5
docstring
max-sixty Mar 31, 2018
11c146e
don't skip if no project
max-sixty Mar 31, 2018
817c63a
formatting
max-sixty Mar 31, 2018
f0a8d36
add marks to allow test selecting
max-sixty Apr 1, 2018
4ecfcfc
assign mark to test
max-sixty Apr 1, 2018
c402952
explicitly chose with auth to do from travis
max-sixty Apr 1, 2018
476bcf1
too hasty to change the pandas installation
max-sixty Apr 1, 2018
ef59bca
do what needs doing
max-sixty Apr 1, 2018
2112acd
Fixing style errors.
stickler-ci Apr 1, 2018
6cc99f6
docstring & import order
max-sixty Apr 1, 2018
56436ff
correct mark expression
max-sixty Apr 1, 2018
5347858
project not required only if default creds available
max-sixty Apr 1, 2018
cd9b37d
remove any more branching in travis
max-sixty Apr 1, 2018
fa97b0f
Merge branch 'master' of https://github.com/pydata/pandas-gbq into de…
max-sixty Apr 2, 2018
a7f6c43
google import inline
max-sixty Apr 2, 2018
2d50519
Merge remote-tracking branch 'upstream/master' into pr-127-maxim-lian…
tswast Apr 25, 2018
7932c59
Use tuple for credentials & project for default project detection.
tswast Apr 25, 2018
31e001f
lint errors.
tswast Apr 25, 2018
08477cc
Remove extra project detection.
tswast Apr 25, 2018
1f1f2c4
Update bad_project_id test to query actual data.
tswast Apr 25, 2018
d920959
Skip credentials tests if key not present.
tswast Apr 25, 2018
a3e6d2f
DOC: add project_id optional to changelog
tswast Apr 25, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 2 additions & 5 deletions pandas_gbq/gbq.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -761,7 +761,7 @@ def read_gbq(query, project_id=None, index_col=None, col_order=None,
----------
query : str
SQL-Like Query to return data values
project_id : str
project_id : str (optional)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The project_id is still required when using user authentication, so we should clarify that it is only optional if using default credentials.

Or maybe also optional with a service account? I think we might have to parse as JSON and grab the project ID from the service account and use https://google-auth.readthedocs.io/en/latest/reference/google.oauth2.service_account.html#google.oauth2.service_account.Credentials.from_service_account_info for that to work.

Copy link
Contributor Author

@max-sixty max-sixty Mar 3, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for bearing with me on the authentication questions

The project_id is still required when using user authentication

Is this right? If I'm running localling with user auth, google.cloud.bigquery can pick up my default project fine:

In [20]: import google.cloud.bigquery

In [21]: c=google.cloud.bigquery.Client()

In [22]: c
Out[22]: <google.cloud.bigquery.client.Client at 0x1138e44a8>

In [23]: c.project
Out[23]: 'sixty-capital'

Auth:

gcloud auth list                                             
       Credentialed Accounts
ACTIVE  ACCOUNT
*       []@sixtycapital.com

To set the active account, run:
    $ gcloud config set account `ACCOUNT`

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

google.cloud.bigquery.Client() does user auth using the Cloud SDK. I'm referring to when the Cloud SDK is not installed, where pandas-gbq does it's user auth flow.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be more specific the credential creation at

https://github.com/pydata/pandas-gbq/blob/16747609dc58f0b6df2c4e896db5a3f365c4677c/pandas_gbq/gbq.py#L272-L283

does not include a default project.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bear with me once more:
Why don't we add it? It's in the JSON file:

image

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In 3LO there are two projects we are concerned with:

  • The client secrets project. In this case, it is the project for Pandas GBQ. It has the BigQuery API enabled, but users shouldn't have permissions to run queries against this project. If we did use it, it would bill the Pandas team, not the user.
  • The user's project. We know this in the case of a service account or when using the gcloud command, but we don't know it if using client secrets.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK I see now. Because we do the app_flow, those creds don't come with a project.
Though not sure what 3LO is

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3LO - 3-legged OAuth 2. (The protocol that app_flow follows.)

Google BigQuery Account project ID.
index_col : str (optional)
Name of result column to use for index in results DataFrame
Expand Down Expand Up @@ -815,9 +815,6 @@ def read_gbq(query, project_id=None, index_col=None, col_order=None,

_test_google_api_imports()

if not project_id:
raise TypeError("Missing required parameter: project_id")

if dialect not in ('legacy', 'standard'):
raise ValueError("'{0}' is not valid for dialect".format(dialect))

Expand Down Expand Up @@ -897,7 +894,7 @@ def to_gbq(dataframe, destination_table, project_id, chunksize=None,
DataFrame to be written
destination_table : string
Name of table to be written, in the form 'dataset.tablename'
project_id : str
project_id : str (optional)
Google BigQuery Account project ID.
chunksize : int (default None)
Number of rows to be inserted in each chunk from the dataframe. Use
Expand Down