Skip to content

Gbq service account #11881

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 30 additions & 7 deletions doc/source/io.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4093,6 +4093,34 @@ The key functions are:

.. _io.bigquery_reader:


Authentication
''''''''''''''

Authentication is possible with either user account credentials or service account credentials.

Authenticating with user account credentials is as simple as following the prompts in a browser window
which will be automatically opened for you. You will be authenticated to the specified
``BigQuery`` account via Google's ``Oauth2`` mechanism. Additional information on the
authentication mechanism can be found `here <https://developers.google.com/identity/protocols/OAuth2#clientside/>`__.

Authentication with service account credentials is possible via the `'private_key'` parameter. This method
is particularly useful when working on remote servers (eg. jupyter iPython notebook on remote host).
The remote authentication using user account credentials is not currently supported in Pandas.
Additional information on service accounts can be found
`here <https://developers.google.com/identity/protocols/OAuth2#serviceaccount>`__.

.. note::

The `'private_key'` parameter can be set to either the file path of the service account key
in JSON format, or key contents of the service account key in JSON format.

.. note::

A private key can be obtained from the Google developers console by clicking
`here <https://console.developers.google.com/permissions/serviceaccounts>`__. Use JSON key type.


Querying
''''''''

Expand All @@ -4107,12 +4135,6 @@ into a DataFrame using the :func:`~pandas.io.gbq.read_gbq` function.

data_frame = pd.read_gbq('SELECT * FROM test_dataset.test_table', projectid)

You will then be authenticated to the specified BigQuery account
via Google's Oauth2 mechanism. In general, this is as simple as following the
prompts in a browser window which will be opened for you. Should the browser not
be available, or fail to launch, a code will be provided to complete the process
manually. Additional information on the authentication mechanism can be found
`here <https://developers.google.com/accounts/docs/OAuth2#clientside/>`__.

You can define which column from BigQuery to use as an index in the
destination DataFrame as well as a preferred column order as follows:
Expand All @@ -4125,7 +4147,7 @@ destination DataFrame as well as a preferred column order as follows:

.. note::

You can find your project id in the `BigQuery management console <https://code.google.com/apis/console/b/0/?noredirect>`__.
You can find your project id in the `Google developers console <https://console.developers.google.com>`__.


.. note::
Expand All @@ -4134,6 +4156,7 @@ destination DataFrame as well as a preferred column order as follows:

.. _io.bigquery_writer:


Writing DataFrames
''''''''''''''''''

Expand Down
2 changes: 2 additions & 0 deletions doc/source/whatsnew/v0.18.0.txt
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,8 @@ Other enhancements
- ``Series`` gained an ``is_unique`` attribute (:issue:`11946`)
- ``DataFrame.quantile`` and ``Series.quantile`` now accept ``interpolation`` keyword (:issue:`10174`).
- ``DataFrame.select_dtypes`` now allows the ``np.float16`` typecode (:issue:`11990`)
- Added Google ``BigQuery`` service account authentication support, which enables authentication on remote servers. (:issue:`11881`). For further details see :ref:`here <io.bigquery_authentication>`


.. _whatsnew_0180.enhancements.rounding:

Expand Down
8 changes: 6 additions & 2 deletions pandas/core/frame.py
Original file line number Diff line number Diff line change
Expand Up @@ -859,7 +859,7 @@ def to_dict(self, orient='dict'):
raise ValueError("orient '%s' not understood" % orient)

def to_gbq(self, destination_table, project_id, chunksize=10000,
verbose=True, reauth=False, if_exists='fail'):
verbose=True, reauth=False, if_exists='fail', private_key=None):
"""Write a DataFrame to a Google BigQuery table.

THIS IS AN EXPERIMENTAL LIBRARY
Expand All @@ -883,14 +883,18 @@ def to_gbq(self, destination_table, project_id, chunksize=10000,
'fail': If table exists, do nothing.
'replace': If table exists, drop it, recreate it, and insert data.
'append': If table exists, insert data. Create if does not exist.
private_key : str (optional)
Service account private key in JSON format. Can be file path
or string contents. This is useful for remote server
authentication (eg. jupyter iPython notebook on remote host)

.. versionadded:: 0.17.0
"""

from pandas.io import gbq
return gbq.to_gbq(self, destination_table, project_id=project_id,
chunksize=chunksize, verbose=verbose, reauth=reauth,
if_exists=if_exists)
if_exists=if_exists, private_key=private_key)

@classmethod
def from_records(cls, data, index=None, exclude=None, columns=None,
Expand Down
Loading