Using DeltaLake Table from Datafusion in Python

[gopinathcs]

Description

Using DeltaLake Table from Datafusion in Python. I searched for document and example code , but didn't find one. Can I use this now, can I have sample code.

Use Case

Related Issue(s)

[wjones127]

I haven't touched it recently, but there is a PyArrow dataset integration in Python DataFusion.

from datafusion import SessionContext
from deltalake import DeltaTable

# Create a DataFusion context
ctx = SessionContext()

# resolve a table
table = DeltaTable("path/to/table")

# Register table with context
ctx.register_dataset('my_table', table.to_pyarrow_dataset())

# Execute SQL
df = ctx.sql("select * from my_table")

[gopinathcs]

Thanks. But seems this is really slow, as this materializing entire table into arrow, which defeat the purpose.

[wjones127]

But seems this is really slow, as this materializing entire table into arrow, which defeat the purpose.

Do you mean you think the code looks like it materializes the entire table? Or that you ran it a query and it was slow?