Merge branch 'main' into jg-proxy

Author: kravets-levko

.github/CODEOWNERS

@@ -2,4 +2,4 @@
 # the repo. Unless a later match takes precedence, these
 # users will be requested for review when someone opens a
 # pull request.
-*       @susodapop @arikfr @yunbodeng-db @andrefurlan-db
+*       @rcypher-databricks @arikfr @yunbodeng-db @andrefurlan-db @jackyhu-db @benc-db @kravets-levko

.github/workflows/code-quality-checks.yml

@@ -1,5 +1,5 @@
 name: Code Quality Checks
-on: 
+on:
   push:
     branches:
       - main
@@ -157,7 +157,7 @@ jobs:
       - name: Install library
         run: poetry install --no-interaction
       #----------------------------------------------
-      #              black the code
+      #              mypy the code
       #----------------------------------------------
       - name: Mypy
         run: poetry run mypy --install-types --non-interactive src

.github/workflows/integration.yml

@@ -0,0 +1,59 @@
+name: Integration Tests
+on:
+  push:
+    paths-ignore:
+      - "**.MD"
+      - "**.md"
+
+jobs:
+  run-e2e-tests:
+    runs-on: ubuntu-latest
+    environment: azure-prod
+    env:
+      DATABRICKS_SERVER_HOSTNAME: ${{ secrets.DATABRICKS_HOST }}
+      DATABRICKS_HTTP_PATH: ${{ secrets.TEST_PECO_WAREHOUSE_HTTP_PATH }}
+      DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}
+      DATABRICKS_CATALOG: peco
+      DATABRICKS_USER: ${{ secrets.TEST_PECO_SP_ID }}
+    steps:
+      #----------------------------------------------
+      #       check-out repo and set-up python
+      #----------------------------------------------
+      - name: Check out repository
+        uses: actions/checkout@v3
+      - name: Set up python
+        id: setup-python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.10"
+      #----------------------------------------------
+      #  -----  install & configure poetry  -----
+      #----------------------------------------------
+      - name: Install Poetry
+        uses: snok/install-poetry@v1
+        with:
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+          installer-parallel: true
+
+      #----------------------------------------------
+      #       load cached venv if cache exists
+      #----------------------------------------------
+      - name: Load cached venv
+        id: cached-poetry-dependencies
+        uses: actions/cache@v2
+        with:
+          path: .venv
+          key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ github.event.repository.name }}-${{ hashFiles('**/poetry.lock') }}
+      #----------------------------------------------
+      # install dependencies if cache does not exist
+      #----------------------------------------------
+      - name: Install dependencies
+        run: poetry install --no-interaction --all-extras
+      #----------------------------------------------
+      #              run test suite
+      #----------------------------------------------
+      - name: Run e2e tests
+        run: poetry run python -m pytest tests/e2e
+      - name: Run SQL Alchemy tests
+        run: poetry run python -m pytest src/databricks/sqlalchemy/test_local

CHANGELOG.md

@@ -1,5 +1,38 @@
 # Release History
 
+# (Unreleased)
+
+- Fix: our logger would raise an uncaught exception under certain oauth error conditions
+
+# 3.1.2 (2024-04-18)
+
+- Remove broken cookie code (#379)
+- Small typing fixes (#382, #384 thanks @wyattscarpenter)
+
+# 3.1.1 (2024-03-19)
+
+- Don't retry requests that fail with code 403 (#373)
+- Assume a default retry-after for 429/503 (#371)
+- Fix boolean literals (#357)
+
+# 3.1.0 (2024-02-16)
+
+- Revert retry-after behavior to be exponential backoff (#349)
+- Support Databricks OAuth on Azure (#351)
+- Support Databricks OAuth on GCP (#338)
+
+# 3.0.3 (2024-02-02)
+
+- Revised docstrings and examples for OAuth (#339)
+- Redact the URL query parameters from the urllib3.connectionpool logs (#341)
+
+# 3.0.2 (2024-01-25)
+
+- SQLAlchemy dialect now supports table and column comments (thanks @cbornet!)
+- Fix: SQLAlchemy dialect now correctly reflects TINYINT types (thanks @TimTheinAtTabs!)
+- Fix: `server_hostname` URIs that included `https://` would raise an exception
+- Other: pinned to `pandas<=2.1` and `urllib3>=1.26` to avoid runtime errors in dbt-databricks (#330)
+
 ## 3.0.1 (2023-12-01)
 
 - Other: updated docstring comment about default parameterization approach (#287)
@@ -37,15 +70,15 @@
 
 ## 2.9.2 (2023-08-17)
 
-__Note: this release was yanked from Pypi on 13 September 2023 due to compatibility issues with environments where `urllib3<=2.0.0` were installed. The log changes are incorporated into version 2.9.3 and greater.__
+**Note: this release was yanked from Pypi on 13 September 2023 due to compatibility issues with environments where `urllib3<=2.0.0` were installed. The log changes are incorporated into version 2.9.3 and greater.**
 
 - Other: Add `examples/v3_retries_query_execute.py` (#199)
 - Other: suppress log message when `_enable_v3_retries` is not `True` (#199)
 - Other: make this connector backwards compatible with `urllib3>=1.0.0` (#197)
 
 ## 2.9.1 (2023-08-11)
 
-__Note: this release was yanked from Pypi on 13 September 2023 due to compatibility issues with environments where `urllib3<=2.0.0` were installed.__
+**Note: this release was yanked from Pypi on 13 September 2023 due to compatibility issues with environments where `urllib3<=2.0.0` were installed.**
 
 - Other: Explicitly pin urllib3 to ^2.0.0 (#191)
 
@@ -98,6 +131,7 @@ __Note: this release was yanked from Pypi on 13 September 2023 due to compatibil
 - Other: Relax sqlalchemy required version as it was unecessarily strict.
 
 ## 2.5.0 (2023-04-14)
+
 - Add support for External Auth providers
 - Fix: Python HTTP proxies were broken
 - Other: All Thrift requests that timeout during connection will be automatically retried
@@ -119,8 +153,8 @@ __Note: this release was yanked from Pypi on 13 September 2023 due to compatibil
 
 ## 2.2.2 (2023-01-03)
 
-- Support custom oauth client id and redirect port 
-- Fix: Add none check on _oauth_persistence in DatabricksOAuthProvider
+- Support custom oauth client id and redirect port
+- Fix: Add none check on \_oauth_persistence in DatabricksOAuthProvider
 
 ## 2.2.1 (2022-11-29)
 
@@ -152,57 +186,71 @@ Huge thanks to @dbaxa for contributing this change!
 
 - Add retry logic for `GetOperationStatus` requests that fail with an `OSError`
 - Reorganised code to use Poetry for dependency management.
+
 ## 2.0.2 (2022-05-04)
+
 - Better exception handling in automatic connection close
 
 ## 2.0.1 (2022-04-21)
+
 - Fixed Pandas dependency in setup.cfg to be >= 1.2.0
 
 ## 2.0.0 (2022-04-19)
+
 - Initial stable release of V2
-- Added better support for complex types, so that in Databricks runtime 10.3+, Arrays, Maps and Structs will get 
+- Added better support for complex types, so that in Databricks runtime 10.3+, Arrays, Maps and Structs will get
   deserialized as lists, lists of tuples and dicts, respectively.
 - Changed the name of the metadata arg to http_headers
 
 ## 2.0.b2 (2022-04-04)
+
 - Change import of collections.Iterable to collections.abc.Iterable to make the library compatible with Python 3.10
 - Fixed bug with .tables method so that .tables works as expected with Unity-Catalog enabled endpoints
 
 ## 2.0.0b1 (2022-03-04)
+
 - Fix packaging issue (dependencies were not being installed properly)
 - Fetching timestamp results will now return aware instead of naive timestamps
 - The client will now default to using simplified error messages
 
 ## 2.0.0b (2022-02-08)
+
 - Initial beta release of V2. V2 is an internal re-write of large parts of the connector to use Databricks edge features. All public APIs from V1 remain.
-- Added Unity Catalog support (pass catalog and / or  schema key word args to the .connect method to select initial schema and catalog)
+- Added Unity Catalog support (pass catalog and / or schema key word args to the .connect method to select initial schema and catalog)
 
 ---
 
 **Note**: The code for versions prior to `v2.0.0b` is not contained in this repository. The below entries are included for reference only.
 
 ---
+
 ## 1.0.0 (2022-01-20)
+
 - Add operations for retrieving metadata
 - Add the ability to access columns by name on result rows
 - Add the ability to provide configuration settings on connect
 
 ## 0.9.4 (2022-01-10)
+
 - Improved logging and error messages.
 
 ## 0.9.3 (2021-12-08)
+
 - Add retries for 429 and 503 HTTP responses.
 
 ## 0.9.2 (2021-12-02)
+
 - (Bug fix) Increased Thrift requirement from 0.10.0 to 0.13.0 as 0.10.0 was in fact incompatible
 - (Bug fix) Fixed error message after query execution failed -SQLSTATE and Error message were misplaced
 
 ## 0.9.1 (2021-09-01)
+
 - Public Preview release, Experimental tag removed
 - minor updates in internal build/packaging
 - no functional changes
 
 ## 0.9.0 (2021-08-04)
+
 - initial (Experimental) release of pyhive-forked connector
 - Python DBAPI 2.0 (PEP-0249), thrift based
 - see docs for more info: https://docs.databricks.com/dev-tools/python-sql-connector.html

conftest.py

@@ -0,0 +1,44 @@
+import os
+import pytest
+
+
+@pytest.fixture(scope="session")
+def host():
+    return os.getenv("DATABRICKS_SERVER_HOSTNAME")
+
+
+@pytest.fixture(scope="session")
+def http_path():
+    return os.getenv("DATABRICKS_HTTP_PATH")
+
+
+@pytest.fixture(scope="session")
+def access_token():
+    return os.getenv("DATABRICKS_TOKEN")
+
+
+@pytest.fixture(scope="session")
+def ingestion_user():
+    return os.getenv("DATABRICKS_USER")
+
+
+@pytest.fixture(scope="session")
+def catalog():
+    return os.getenv("DATABRICKS_CATALOG")
+
+
+@pytest.fixture(scope="session")
+def schema():
+    return os.getenv("DATABRICKS_SCHEMA", "default")
+
+
+@pytest.fixture(scope="session", autouse=True)
+def connection_details(host, http_path, access_token, ingestion_user, catalog, schema):
+    return {
+        "host": host,
+        "http_path": http_path,
+        "access_token": access_token,
+        "ingestion_user": ingestion_user,
+        "catalog": catalog,
+        "schema": schema,
+    }

examples/interactive_oauth.py

@@ -1,29 +1,16 @@
 from databricks import sql
 import os
 
-"""Bring Your Own Identity Provider with fined grained OAuth scopes is currently public preview on
-Databricks in AWS. databricks-sql-connector supports user to machine OAuth login which means the
-end user has to be present to login in a browser which will be popped up by the Python process. You
-must enable OAuth in your Databricks account to run this example. More information on how to enable
-OAuth in your Databricks Account in AWS can be found here:
-
-https://docs.databricks.com/administration-guide/account-settings-e2/single-sign-on.html
+"""databricks-sql-connector supports user to machine OAuth login which means the
+end user has to be present to login in a browser which will be popped up by the Python process.
 
 Pre-requisites:
-- You have a Databricks account in AWS.
-- You have configured OAuth in Databricks account in AWS using the link above.
 - You have installed a browser (Chrome, Firefox, Safari, Internet Explorer, etc) that will be
   accessible on the machine for performing OAuth login.
 
 This code does not persist the auth token. Hence after the Python process terminates the
 end user will have to login again. See examples/persistent_oauth.py to learn about persisting the
 token across script executions.
-
-Bring Your Own Identity Provider is in public preview. The API may change prior to becoming GA. 
-You can monitor these two links to find out when it will become generally available:
-
-  1. https://docs.databricks.com/administration-guide/account-settings-e2/single-sign-on.html 
-  2. https://docs.databricks.com/dev-tools/python-sql-connector.html
 """
 
 with sql.connect(server_hostname = os.getenv("DATABRICKS_SERVER_HOSTNAME"),

examples/persistent_oauth.py

@@ -1,14 +1,7 @@
-"""Bring Your Own Identity Provider with fined grained OAuth scopes is currently public preview on
-Databricks in AWS. databricks-sql-connector supports user to machine OAuth login which means the
-end user has to be present to login in a browser which will be popped up by the Python process. You
-must enable OAuth in your Databricks account to run this example. More information on how to enable
-OAuth in your Databricks Account in AWS can be found here:
-
-https://docs.databricks.com/administration-guide/account-settings-e2/single-sign-on.html
+"""databricks-sql-connector supports user to machine OAuth login which means the
+end user has to be present to login in a browser which will be popped up by the Python process.
 
 Pre-requisites:
-- You have a Databricks account in AWS.
-- You have configured OAuth in Databricks account in AWS using the link above.
 - You have installed a browser (Chrome, Firefox, Safari, Internet Explorer, etc) that will be
   accessible on the machine for performing OAuth login.
 
@@ -18,12 +11,6 @@
 shows which methods you may implement.
 
 For this example, the DevOnlyFilePersistence class is provided. Do not use this in production.
-
-Bring Your Own Identity Provider is in public preview. The API may change prior to becoming GA. 
-You can monitor these two links to find out when it will become generally available:
-
-  1. https://docs.databricks.com/administration-guide/account-settings-e2/single-sign-on.html 
-  2. https://docs.databricks.com/dev-tools/python-sql-connector.html
 """
 
 import os