BUG: to_csv doesn't support file handles from ZipFiles

[twoertwein]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

from zipfile import ZipFile

import numpy as np
import pandas as pd

with ZipFile("plots.zip", mode="w") as file:
    # write some matplotlib plots into the zip file

    # write CSV into zip file
    data = pd.DataFrame(np.random.rand(10, 3))
    with file.open("data.csv", mode="w") as csv_file:
        data.to_csv(csv_file)

Last part of the stack trace:

~/miniforge3/envs/panama/lib/python3.8/site-packages/pandas/io/formats/csvs.py in _save_header(self)
    278         if not has_mi_columns or has_aliases:
    279             encoded_labels += list(write_cols)
--> 280             writer.writerow(encoded_labels)
    281         else:
    282             # write out the mi

~/miniforge3/envs/panama/lib/python3.8/zipfile.py in write(self, data)
   1134         nbytes = len(data)
   1135         self._file_size += nbytes
-> 1136         self._crc = crc32(data, self._crc)
   1137         if self._compressor:
   1138             data = self._compressor.compress(data)

TypeError: a bytes-like object is required, not 'str'

Problem description

I know that to_csv supports compression but I want to write a DataFrame into a zip file with multiple other files. I'm not sure whether the above error is an error in pandas or a missing feature/bug in zipfiles.

Expected Output

to_csv supports writing to a ZipFile's file handle.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.8.3.final.0
python-bits : 64
OS : Linux
OS-release : 3.10.0-693.5.2.el7.x86_64
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.0.4
numpy : 1.18.4
pytz : 2020.1
dateutil : 2.8.1
pip : 20.1.1
setuptools : 47.1.1.post20200529
Cython : None
pytest : 5.4.3
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.15.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.2.1
numexpr : 2.7.1
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 5.4.3
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : None
tables : 3.6.1
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

[gfyoung]

Part of the challenge in supporting file handlers that require bytes and not str is elegantly checking if a file handler requires this. A simpler example of what you showed is this:

from pandas import DataFrame

with open("output.csv", "w+b") as f:
    DataFrame([1, 2, 3]).to_csv(f)

If you can find a way to do that, then we'll be all set!

[twoertwein]

One approach would be to push this responsibility to the user. Would it be reasonable to require to_csv's existing keyword argument mode to contain b when the user provides a binary file handle?

Unfortunately, it seems that to_csv can currently not even cope with a binary mode for a file name.

from pandas import DataFrame

DataFrame([1, 2, 3]).to_csv("output.csv", mode="w+b")

~/miniforge3/envs/panama/lib/python3.8/site-packages/pandas/io/common.py in get_handle(path_or_buf, mode, encoding, compression, memory_map, is_text)
    453         if encoding:
    454             # Encoding
--> 455             f = open(path_or_buf, mode, encoding=encoding, newline="")
    456         elif is_text:
    457             # No explicit encoding

ValueError: binary mode doesn't take an encoding argument

[gfyoung]

At the moment, yes, but you're more than welcome to try modifying that logic to so that the argments to open don't take an encoding if a b is in mode.