Skip to content

Conversation

hoxbro
Copy link
Member

@hoxbro hoxbro commented Aug 17, 2024

It makes it so that the following will give the same types before it returned (string[pyarrow], object).

import dask
import spatialpandas.io as sio

dask.config.set({"dataframe.convert-string": True})

# http://s3.amazonaws.com/datashader-data/nyc_buildings.parq.zip
ddf = sio.read_parquet_dask("./data/nyc_buildings.parq")
ddf["type"].dtype, ddf["type"].compute().dtype

Together with holoviz/holoviews#6362 should make it possible to run the NYC Buildings example.

@hoxbro hoxbro added the type: compatibility Compability with upstream packages label Aug 17, 2024
Copy link

codecov bot commented Aug 17, 2024

Codecov Report

Attention: Patch coverage is 93.10345% with 2 lines in your changes missing coverage. Please review.

Project coverage is 77.72%. Comparing base (3a1b6eb) to head (6c0f0e1).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
spatialpandas/io/parquet.py 75.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #156      +/-   ##
==========================================
+ Coverage   77.47%   77.72%   +0.24%     
==========================================
  Files          50       50              
  Lines        4844     4871      +27     
==========================================
+ Hits         3753     3786      +33     
+ Misses       1091     1085       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@hoxbro hoxbro requested a review from philippjfr August 18, 2024 14:44
@hoxbro hoxbro changed the title Ensure that pandas dtype matches dask when loading data from parquet compat: Ensure that pandas dtype matches dask when loading data from parquet Oct 6, 2024
Copy link
Member

@philippjfr philippjfr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@hoxbro hoxbro enabled auto-merge (squash) November 12, 2024 16:20
@hoxbro hoxbro merged commit dec5b6a into main Nov 12, 2024
12 checks passed
@hoxbro hoxbro deleted the force_dask_dtype_parquet branch November 12, 2024 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: compatibility Compability with upstream packages

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants