Strange behaviour, dbx_file.iter_lines() ends prematurely

[thestick613]

I'm streaming a large CSV file from dropbox, and i'm feeding it into a csv.DictReader(). Well off into the file, iter_lines randomly yields an incomplete line, which causes csv.DictReader to stop processing the file.

This is how my code looks:

def dropbox_yielder(dbx_file):
    for line in dbx_file.iter_lines(chunk_size=1024 * 1024): # also tried with no chunk_size
        encoding = cchardet.detect(line)
        decoded = codecs.decode(line, encoding['encoding'] or 'UTF8', 'ignore')
        logger.info("Yielded %d bytes - < %s ... %s >", len(decoded), decoded[:9], decoded[-9:])
        yield decoded

2019-09-08 09:37:24,364.364 INFO import_service_v2 - dropbox_yielder: Yielded 121 bytes - < asdasdasd ... 11/2004 00:00:00,"", >
2019-09-08 09:37:24,366.366 INFO import_service_v2 - dropbox_yielder: Yielded 122 bytes - < asdasdasd ... 11/2004 00:00:00,"", >
2019-09-08 09:37:24,368.368 INFO import_service_v2 - dropbox_yielder: Yielded 18 bytes - < asdasdasd ... asdasdasd >

The csv lines are supposed to end with a timestamp, which in the last line does not. It's too short - only 18 bytes. Has anyone else ever had similar problems? I'm not sure if this is an issue with dropbox or with requests.

[greg-db]

Can you also share the code that's producing the dbx_file in this case? Thanks in advance!

[thestick613]

dbx = dropbox.Dropbox(dropbox_access_token, timeout=600) # also tried without no timeout defined
metadata, dbx_file = dbx.files_download(path=dropbox_path)

I think this is caused by requests, so there's not much that dropbox can do..

[greg-db]

Thanks! The Dropbox Python SDK does use requests, but for reference, can you let me know what in particular makes you think this is an issue with requests itself? Are you getting any error/output from requests? If so, please do share that as well.

[thestick613]

psf/requests#844

From what i understand, requests couldn't know when an endline characters occurs in the decoded representation of the data, and makes some assumptions.

The iteration sometimes stops at the same spot in the file, and other times it stops nearby (+/- 100.000 lines). If in change the chunk_size it will make the iteration stop at a different place in the file. I have no way to debug this as it takes a while for the problem to appear. I'm downloading the files locally and iterating in them without any problem.

The data i'm reading also not very good, since i'm dealing with mixed encodings in the same file..

[greg-db]

I see, thanks. The files_download method does return the requests.models.Response object directly, so I believe you're correct that this isn't under our control.

Accordingly, I'll close this issue. I recommend you open one for requests if you want help debugging this. (It looks like there's also some documentation on using iter_lines here in case it helps.)