Skip to content

Consider making pandas an optional dependency #489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
FBruzzesi opened this issue Jan 17, 2025 · 1 comment
Open

Consider making pandas an optional dependency #489

FBruzzesi opened this issue Jan 17, 2025 · 1 comment

Comments

@FBruzzesi
Copy link

FBruzzesi commented Jan 17, 2025

Description

Currently pandas is a hard requirement for the library, however not necessarily used.

By delaying pandas import in src/databricks/sql/client.py from the top of the file to _convert_arrow_table method after the if self.connection.disable_pandas is True: condition it would be possible to make pandas an optional dependency.

I was able to run all the tests with this change flawlessly, as well as uninstalling pandas and then run the following:

import polars as pl
from databricks.sql.client import Connection

credentials = ...
query = ...

conn = Connection(
    server_hostname=credentials["server_hostname"],
    http_path=credentials["http_path"],
    access_token=credentials["access_token"],
)

dframe = pl.read_database(query, conn)

without any issue

@FBruzzesi FBruzzesi changed the title Make pandas an optional dependency Consider making pandas an optional dependency Jan 17, 2025
@gs11
Copy link

gs11 commented Mar 24, 2025

Adding to this - the size of pandas makes it really hard to run this in e.g. an AWS Lambda environment where total deployment package size is limited.

gs11 pushed a commit to gs11/databricks-sql-python that referenced this issue Mar 24, 2025
gs11 pushed a commit to gs11/databricks-sql-python that referenced this issue Mar 24, 2025
gs11 pushed a commit to gs11/databricks-sql-python that referenced this issue Mar 24, 2025
gs11 pushed a commit to gs11/databricks-sql-python that referenced this issue Mar 24, 2025
gs11 added a commit to gs11/databricks-sql-python that referenced this issue Mar 24, 2025
gs11 added a commit to gs11/databricks-sql-python that referenced this issue Mar 24, 2025
gs11 added a commit to gs11/databricks-sql-python that referenced this issue Mar 24, 2025
gs11 added a commit to gs11/databricks-sql-python that referenced this issue Mar 24, 2025
gs11 added a commit to gs11/databricks-sql-python that referenced this issue Mar 24, 2025
gs11 added a commit to gs11/databricks-sql-python that referenced this issue Mar 24, 2025
gs11 added a commit to gs11/databricks-sql-python that referenced this issue Mar 24, 2025
gs11 added a commit to gs11/databricks-sql-python that referenced this issue Mar 24, 2025
gs11 added a commit to gs11/databricks-sql-python that referenced this issue Apr 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants