ENH: Delta Lake file format support

[jackwellsxyz]

Is your feature request related to a problem?

No

Describe the solution you'd like

I'd love it if Pandas could support Databricks' Delta Lake file format (https://github.com/delta-io/delta). It's a type of versioned parquet file format that supports updates/inserts/deletions.

API breaking implications

None that I'm aware of

Describe alternatives you've considered

N/A

Additional context

N/A

[yehoshuadimarsky]

I'm pretty sure Delta Lake only supports access via Spark at the moment, not through any other tools or APIs. I'll post the link in the docs where it mentions it when I can find it, but I do remember reading that.

EDIT: Well, their README does say this:

The only stable public APIs, currently provided by Delta Lake, are through the DataFrameReader/Writer (i.e. spark.read, df.write, spark.readStream and df.writeStream). Options to these APIs will remain stable within a major release of Delta Lake (e.g., 1.x.x).

All other interfaces in this library are considered internal, and they are subject to change across minor/patch releases.

[jbrockmendel]

I doubt we would support that directly, but if there is an implementation we could link to it in the ecosystem docs.

[ggGibs]

FYI Koalas reads/writes to delta lake (link), and has easy koalas <--> pandas methods
But it looks like a pandas-like wrapper on top of pyspark - you will need a spark cluster somewhere to handle the I/O. No hope running it standalone

[TomAugspurger]

Happy to take an addition to the ecosystem docs when a reader / writer is available. That won't be implemented in pandas, but perhaps in / using pyarrow.

[houqp]

Happy to share that we now have native support for Delta lakes outside of JVM and Spark. It's a full deltalake implementation in pure Rust. We also provide a thin python wrapper for pandas integration, see https://github.com/delta-io/delta-rs/tree/main/python#usage.

@jbrockmendel perhaps we can link this to the ecosystem docs?

[jbrockmendel]

fine by me

[MrPowers]

There is a delta-rs project that makes it easy to read Delta Lakes into pandas DataFrames as mentioned by @houqp. See this snippet:

from deltalake import DeltaTable

dt = DeltaTable("resources/delta/1")
df = dt.to_pandas()

Python write access doesn't exist yet, but hopefully it'll be added soon cause it'd be an awesome addition for the Pandas community!

[houqp]

Add to @MrPowers 's comment, delta-rs now has write support in the rust core, just waiting for someone to send us a PR to expose that write api to the python shim ;)