load gzipped models

[kba]

pyrnn models are generally distributed gzipped. If loading gzipped models is inefficient, I can also adapt OCR-D/ocrd_all#103 to gunzip models after download.

[bertsky]

This merge came to fast for me. I suggest reverting.

[bertsky]

I don't think this is necessary. The default setting zip=0 already decompressed files if they are named *.gz. See here:

ocrd_cis/ocrd_cis/ocropy/ocrolib/common.py

Line 447 in b7bba57

if zip==0 and fname.endswith(".gz"):

Ignoring the file name will now make parameter files (or workflow definitions) fail which don't use .gz suffix, for example because of an old workaround to #41

[finkf]

fine by me.

[kba]

I was a bit too eager with the PR, this was not meant to be merged as such, I had a lot of trouble getting ocropy to find the models and this fixed it for me. Ocropy should gunzip even with zip==0 if model ends with .gz and should expand the filename with a .gz suffix when searching. But either there is a flaw in the logic somewhere (proper unit tests in ocropy would help) or in my setup.

[bertsky]

@kba In #41 I wrote this summary of what gets searched.

I agree we should test this, document this, and probably even generalise a little (all ocrolib search directories except OCROPUS_DATA are unrealistic in OCR-D context). Do we have a consensus/spec on where to search relative path names of model files yet? This seems important for deployment...

[kba]

Do we have a consensus/spec on where to search relative path names of model files yet?

You mean like https://ocr-d.de/en/models#ocropy--ocrd_cis ?

[bertsky]

Do we have a consensus/spec on where to search relative path names of model files yet?

You mean like https://ocr-d.de/en/models#ocropy--ocrd_cis ?

No, locally, like $VIRTUAL_ENV/lib/python3.6/site-packages/$MODULE or $VIRTUAL_ENV/lib/python3.6/site-packages/ocrd_models/processors or under a common environment variable. I mean, this is a problem for all processors that need model files. You usually cannot distribute them as part of package_data via PyPI or even in the same repo, because they are too big. But the package can expect some installer (presumably within ocrd_all) to download and copy it somewhere for it to find at runtime under a relative path.

[finkf]

I totally agree with @bertsky. But I guess this should be discussed in a separate issue. Maybe even on ocrd_core?

[bertsky]

But I guess this should be discussed in a separate issue.

Right. See OCR-D/spec#160