Request for Async Support and Dependency Update

[devmessias]

Hi all.

In my work we're using the databricks-sql-connector with sqlalchemy-databricks but it currently lacks support for async operations, which is causing performance issues for us. It would be appreciated if you could consider adding async support in a future release.

We have noticed that one dependency of the sqlalchemy (https://github.com/crflynn/sqlalchemy-databricks) dialect it is pyhive. The problem here is that pyhive is not being maintained anymore by dropbox. This raises security concerns. There are any additional considerations about changing that?

I would appreciate if you could let me know if there is anything I can do to help with these requests.

https://github.com/dropbox/PyHive/issues

[andrefurlan-db]

Hi @devmessias, tudo bem?

Just to clarify a bit, what exactly do you mean by async? Separate thread for parallelism/concurrency, or the ability to save operation handle and come back to it later in a potentially different process?

[susodapop]

Ditto Andre's questions.

Separately:

We have noticed that one dependency of the sqlalchemy (https://github.com/crflynn/sqlalchemy-databricks) dialect it is pyhive. The problem here is that pyhive is not being maintained anymore by dropbox. This raises security concerns. There are any additional considerations about changing that?

sqlalchemy-databricks is a community project. We are building our own SQLAlchemy dialect which does not rely on Pyhive and will be available soon (there is an open pull request that implements it). We don't officially support the community project and once ours is available you will be able to connect with SQLAlchemy without installing any separate packages.

sqlalchemy-databricks but it currently lacks support for async operations

We don't have plans to implement the SQLAlchemy asynchronous API at this time but are open to considering this in the future. SQLAlchemy considers asyncio support to be a beta feature so I don't think it makes sense to prioritise it at this time.

which is causing performance issues for us.

I'm very curious to know what performance issues you're seeing. If you're interested we can share a dev build of databricks-sql-connector with you that includes the first-party SQLAlchemy connector -- this very well may solve the performance issues even without async.

I would appreciate if you could let me know if there is anything I can do to help with these requests.

The best thing we can get is feedback on our new connector :) #57

[susodapop]

SQLAlchemy support is now built-in to databricks-sql-connector as of v2.4.0.