Introduce SeaDatabricksClient (Session Implementation) #577

varun-edachali-dbx · 2025-05-29T06:43:43Z

What type of PR is this?

Feature

Description

Introduce SEA Backend client as an alternative to the existing Thrift backend
implement open_session() and close_session() functionality, and add an experimental, temporary test script to test them.

How is this tested?

Unit tests
E2E Tests
Manually
N/A

Related Tickets & Documents

https://docs.google.com/document/d/1Y-eXLhNqqhrMVGnOlG8sdFrCxBTN1GdQvuKG4IfHmo0/edit?usp=sharing
https://databricks.atlassian.net/browse/PECOBLR-482?atlOrigin=eyJpIjoiODdkMGQ5NjZhMjY5NDRmZWEwZmIyNGEyNzFlNGQxNzEiLCJwIjoiaiJ9

…544) This PR replaces the previous implementation of convert_decimals_in_arrow_table() with a more efficient approach that uses PyArrow's native casting operation instead of going through pandas conversion and array creation. - Remove conversion to pandas DataFrame via to_pandas() and apply() methods - Remove intermediate steps of creating array from decimal column and setting it back - Replace with direct type casting using PyArrow's cast() method - Build a new table with transformed columns rather than modifying the original table - Create a new schema based on the modified fields The new approach is more performant by avoiding pandas conversion overhead. The table below highlights substantial performance improvements when retrieving all rows from a table containing decimal columns, particularly when compression is disabled. Even greater gains were observed with compression enabled—showing approximately an 84% improvement (6 seconds compared to 39 seconds). Benchmarking was performed against e2-dogfood, with the client located in the us-west-2 region. ![image](https://github.com/user-attachments/assets/5407b651-8ab6-4c13-b525-cf912f503ba0) Signed-off-by: Jayant Singh <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

ensure maintenance of current APIs of Connection while delegating responsibility Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

… through Connection Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

as in CONTRIBUTING.md Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

in case the openSession takes long, the initialisation of the session will not complete immediately. This could make the session attribute inaccessible. If the Connection is deleted in this time, the open() check will throw because the session attribute does not exist. Thus, we default to the Connection being closed in this case. This was not an issue before because open was a direct attribute of the Connection class. Caught in the integration tests. Signed-off-by: varun-edachali-dbx <[email protected]>

WARNING: incomplete - result set still aligned heavily with thrift and the necessity of some defined functions is to be validated Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

…emany test without this the call_args are not logged on the instance and the unit test fails Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

…amples Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

new codeowners Signed-off-by: varun-edachali-dbx <[email protected]>

…t to prevent server side resource leaks (#554) * Enhance Cursor close handling and context manager exception management * tests * fmt * Fix Cursor.close() to properly handle CursorAlreadyClosedError * Remove specific test message from Cursor.close() error handling * Improve error handling in connection and cursor context managers to ensure proper closure during exceptions, including KeyboardInterrupt. Add tests for nested cursor management and verify operation closure on server-side errors. * add * add Signed-off-by: varun-edachali-dbx <[email protected]>

* PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <[email protected]> * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <[email protected]> * fixed format Signed-off-by: Sai Shree Pradhan <[email protected]> * used lazy logging Signed-off-by: Sai Shree Pradhan <[email protected]> * changed debug to error logs Signed-off-by: Sai Shree Pradhan <[email protected]> * used lazy logging Signed-off-by: Sai Shree Pradhan <[email protected]> --------- Signed-off-by: Sai Shree Pradhan <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>

More conditions to run github actions Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

…terface

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-05-29T06:43:57Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-05-29T06:52:18Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-05-29T09:34:53Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-05-30T05:18:26Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

varun-edachali-dbx · 2025-06-03T08:31:09Z

superseded by #582 (cleaner commit history)

jayantsing-db and others added 30 commits May 28, 2025 03:40

[PECOBLR-361] convert column table to arrow if arrow present (#551)

c7ea206

Signed-off-by: varun-edachali-dbx <[email protected]>

decouple session class from existing Connection

a47163e

ensure maintenance of current APIs of Connection while delegating responsibility Signed-off-by: varun-edachali-dbx <[email protected]>

add open property to Connection to ensure maintenance of existing API

6d92c76

Signed-off-by: varun-edachali-dbx <[email protected]>

update unit tests to address ThriftBackend through session instead of…

0c0675f

… through Connection Signed-off-by: varun-edachali-dbx <[email protected]>

chore: move session specific tests from test_client to test_session

c021435

Signed-off-by: varun-edachali-dbx <[email protected]>

formatting (black)

9b82d61

as in CONTRIBUTING.md Signed-off-by: varun-edachali-dbx <[email protected]>

use connection open property instead of long chain through session

8789e32

Signed-off-by: varun-edachali-dbx <[email protected]>

trigger integration workflow

035b5fd

Signed-off-by: varun-edachali-dbx <[email protected]>

introduce databricksClient interface and thrift backend implementation

f9fcb79

WARNING: incomplete - result set still aligned heavily with thrift and the necessity of some defined functions is to be validated Signed-off-by: varun-edachali-dbx <[email protected]>

change names of ThriftBackend -> ThriftDatabricksClient in tests

7925954

Signed-off-by: varun-edachali-dbx <[email protected]>

fix: remove excess debug log

54804b9

Signed-off-by: varun-edachali-dbx <[email protected]>

fix: replace thrift_backend with backend in result set param

cbaa1f3

Signed-off-by: varun-edachali-dbx <[email protected]>

fix: replace module replacement with concrete mock instance in execut…

98b0984

…emany test without this the call_args are not logged on the instance and the unit test fails Signed-off-by: varun-edachali-dbx <[email protected]>

formatting: black + re-organise backend into new dir

882dda7

Signed-off-by: varun-edachali-dbx <[email protected]>

fix: sql.thrift_backend -> sql.backend.thrift_backend in tests and ex…

a57f6c3

…amples Signed-off-by: varun-edachali-dbx <[email protected]>

introduce Result Set interface and concrete thrift implementation

d4a49ca

Signed-off-by: varun-edachali-dbx <[email protected]>

create concrete ThriftResultSet instead of ResultSet type

574df21

Signed-off-by: varun-edachali-dbx <[email protected]>

ensure backend client returns a ResultSet type in backend tests

8ea5cf4

Signed-off-by: varun-edachali-dbx <[email protected]>

formatting (black)

3eba92a

Signed-off-by: varun-edachali-dbx <[email protected]>

fix: correct client tests

6735fd5

Signed-off-by: varun-edachali-dbx <[email protected]>

formatting (black)

12388e8

Signed-off-by: varun-edachali-dbx <[email protected]>

remove excess comments (nit)

853e6c3

Signed-off-by: varun-edachali-dbx <[email protected]>

Update CODEOWNERS (#562)

4f8b54c

new codeowners Signed-off-by: varun-edachali-dbx <[email protected]>

Update github actions run conditions (#569)

2c2984e

More conditions to run github actions Signed-off-by: varun-edachali-dbx <[email protected]>

fix: thrift_backend->backend, ResultSet -> ThriftResultSet

4979422

Signed-off-by: varun-edachali-dbx <[email protected]>

fix: thrift_backend -> backend in e2e test_driver

6361e85

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx added 11 commits May 29, 2025 03:56

newline for cleanliness

d838653

Signed-off-by: varun-edachali-dbx <[email protected]>

fix circular import

6654f06

Signed-off-by: varun-edachali-dbx <[email protected]>

formatting (black)

89425f9

Signed-off-by: varun-edachali-dbx <[email protected]>

to_hex_id -> get_hex_id

93e55e8

Signed-off-by: varun-edachali-dbx <[email protected]>

better comment on protocol version getter

7689d75

Signed-off-by: varun-edachali-dbx <[email protected]>

formatting (black)

1ec8c45

Signed-off-by: varun-edachali-dbx <[email protected]>

Merge remote-tracking branch 'origin/backend-interface' into fetch-in…

80b7bc3

…terface

stricter typing for cursor

904efe7

Signed-off-by: varun-edachali-dbx <[email protected]>

correct typing

c91bc37

Signed-off-by: varun-edachali-dbx <[email protected]>

init sea backend

42a0d08

Signed-off-by: varun-edachali-dbx <[email protected]>

move test script into experimental dir

1056bc2

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod May 29, 2025 06:43 — with GitHub Actions Failure

varun-edachali-dbx changed the title ~~Sess sea~~ Introduce SEADatabricksClient May 29, 2025

formatting (black)

d59880c

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod May 29, 2025 06:52 — with GitHub Actions Failure

varun-edachali-dbx changed the title ~~Introduce SEADatabricksClient~~ Introduce SeaDatabricksClient (Session Implementation) May 29, 2025

cleanup: removed excess comments, validated decisions

16ff4ec

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod May 29, 2025 09:34 — with GitHub Actions Failure

add unit tests for sea backend

95b0781

Signed-off-by: varun-edachali-dbx <[email protected]>

varun-edachali-dbx had a problem deploying to azure-prod May 30, 2025 05:18 — with GitHub Actions Failure

varun-edachali-dbx closed this Jun 3, 2025

varun-edachali-dbx deleted the sess-sea branch June 3, 2025 09:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Introduce SeaDatabricksClient (Session Implementation) #577

Introduce SeaDatabricksClient (Session Implementation) #577

Uh oh!

varun-edachali-dbx commented May 29, 2025 •

edited

Loading

Uh oh!

github-actions bot commented May 29, 2025

Uh oh!

github-actions bot commented May 29, 2025

Uh oh!

github-actions bot commented May 29, 2025

Uh oh!

github-actions bot commented May 30, 2025

Uh oh!

varun-edachali-dbx commented Jun 3, 2025

Uh oh!

Uh oh!

Introduce SeaDatabricksClient (Session Implementation) #577

Introduce SeaDatabricksClient (Session Implementation) #577

Uh oh!

Conversation

varun-edachali-dbx commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

Description

How is this tested?

Related Tickets & Documents

Uh oh!

github-actions bot commented May 29, 2025

Uh oh!

github-actions bot commented May 29, 2025

Uh oh!

github-actions bot commented May 29, 2025

Uh oh!

github-actions bot commented May 30, 2025

Uh oh!

varun-edachali-dbx commented Jun 3, 2025

Uh oh!

Uh oh!

varun-edachali-dbx commented May 29, 2025 •

edited

Loading