Alternative serialization for text data

[mrocklin]

Pandas DataFrames containing text columns are expensive to serialize. This affects dask.dataframe performance in multiprocessing or distributed settings.

Pickle is expensive

In particular the current solution of using pickle.dumps for object dtype columns can be needlessly expensive when all of the values in the column are text. In this case fairly naive solutions, like msgpack can be much much faster.

Here is an old blogpost on the topic: http://matthewrocklin.com/blog/work/2015/03/16/Fast-Serialization

And an image

Alternatives

There are naive solutions like msgpack (already in pandas) or encoding the text directly.

There are more sophisticated solutions as well that would provide efficient packing, including with repeated elements.

But sometimes objects are objects

One concern here is that sometimes the Python objects aren't text. I propose that Pandas does a check each time or asks for forgiveness on an exception.

Anyway, I would find this valuable. It would help to reduce bottlenecks in dask.dataframe in some situations.

[jreback]

I think its reasonable now that pyarrow 0.2 is out, that we could add convenience functions (marked experimental hah!)
.to_arrow() and .from_arrow, similar to the .to_feather and .from_feather
xref dask/distributed#614
cc @wesm @mrocklin

[wesm]

FYI: I started working on combining the Feather and pyarrow codebases which will also bring new Feather features (e.g. reading and writing to Python file objects).

Another aside: I'm also planning eventually to develop a native C++ JSON table parser using RapidJSON within the Arrow codebase. Schema inference is the most annoying part, but I would want this arrow_json component to pass the pandas JSON test suite. Eventually we can support schemas with unions (where some fields might contain multiple types). pandas 2.0 can acquire union types, too

[jbrockmendel]

has pyarrow solved this to everyone's satisfaction?

[xhochy]

I guess pickle5 / pickle.dump(protocol=5) and a new string type based on Arrow will fully solve this but Feather may already be enough.

[mroeschke]

I think the 'string[pyarrow]' dtype released in 1.2 might have sufficiently addressed this so closing. We can follow up with a new issue if there are additional requirements not covered by that dtype