Reduce sdist size

[ocefpaf]

At the moment the sdist is +40 MB, which is a lot for PyPI. Most of it are the notebooks in the docs and their example data. We can alleviate that, and still ship the notebooks as part of the docs with:

• Keep only clean notebooks, no outputs, in the source code. We can, and should, run them when building the docs only to have a rendered version. That not only makes the sdist smaller, but also makes the notebook diffs easier to interpret, and allow us to check if they are not stale with each doc build.
• All example data should be downloaded and cached when running the notebook for the first time. We can use pooch for that.

[ChrisBarker-NOAA]

For this release, let's keep it simple, and not include the examples at all. Or if you want, the examples without the data files is OK.

I'm still unclear on how best to deal with "docs and demos" -- I really think they should be separate anyway.

Also unclear on how best to handle large files that need to be downloaded -- where to put them?

But this is all needs a lot of cleanup for this project anyway, so let's just get the package published and installable in as easy a way as possible right now.

Thanks for noticing though!

-CHB

[ChrisBarker-NOAA]

NOTE: I haven't looked, but some of those big data files may have been accidentally included in the repo -- people tend not to be very careful about git add * :-(

[ocefpaf]

I'm still unclear on how best to deal with "docs and demos" -- I really think they should be separate anyway.

I usually work in poor internet connection places/countries. I love when docs and demos are available so I don't have to keep googling for them.

Also unclear on how best to handle large files that need to be downloaded -- where to put them?

IMO pooch is the answer. It downloads them, hashes them, store into a cache directory. Pretty much all libraries that ship example data uses it like metpy, xarray, etc.

[ocefpaf]

NOTE: I haven't looked, but some of those big data files may have been accidentally included in the repo -- people tend not to be very careful about git add * :-(

That's why I love pre-commit checks: https://github.com/ioos/xarray-subset-grid/pull/68/files#diff-63a9c44a44acf85fea213a857769990937107cf072831e1a26808cfde9d096b9R10

[ChrisBarker-NOAA]

IMO pooch is the answer.

Looks handy, yes. We wrote a not-as-easy-to-use-or-as-robust homegrown version for PyGNOME ages ago. Maybe we should start using it..

But you still need to store the files somewhere to be accessed.

Anyway, I'm all for having a good system for this project -- just not sure we've got teh time to set it up right now. But if you can find the time to set it up, great!

-CHB

[ocefpaf]

But you still need to store the files somewhere to be accessed.

You mean remotely or the local cache? The local cache is handled by pooch, remotely I plan to implement the dual tag/release where we add the data files as assets on GH.

[ChrisBarker-NOAA]

You mean remotely or the local cache?

remotely -- pooch isn't helpful if they aren't served up somewhere...

I plan to implement the dual tag/release where we add the data files as assets on GH.

If you can get that set up, great!

[ChrisBarker-NOAA]

That's why I love pre-commit checks:

Is there one that will reject large files? I didn't see it, but that would be nice, yes.

[ChrisBarker-NOAA]

looking a bit more, yes the largest of those should not have been added. But I'll have to poke into it a bit more to be sure what to do -- so for now, excluding the docs from the sdist, as you have, is the way to go.