a way to read .trees metadata without loading the whole file would be useful

[bhaller]

This came up in SLiM (https://github.com/MesserLab/SLiM/discussions/233) but I think it has general utility, as I'll describe. What the user wants is a way to get the metadata from a .trees file without the big overhead of loading the whole .trees into memory (whether as tables or a tree sequence); just the metadata.

In SLiM they want this so they can read in parameters that they previously put into the metadata, before the point in their script where they would actually load in the .trees data.

This seems generally useful to me because someone might wish to, e.g., loop in Python over the .trees files in a directory that contains many of them, and do "something" with each .trees file that has metadata with a certain property. Like: process all the .trees files that come from msprime but not those from SLiM, or copy all those where their parameter XYZZY had a value of 15.5 into a different folder, or whatever.

So for SLiM it'd be great to have C API for this; for other uses Python API seems called for. Would be nice to have it in C API 1.0 – we would use it immediately in SLiM – but it's not a big deal if it isn't.

[grahamgower]

This seems generally useful to me because someone might wish to, e.g., loop in Python over the .trees files in a directory that contains many of them, and do "something" with each .trees file that has metadata with a certain property.

I have done exactly this, but to get the provenance info (I bodged parameters in here before tskit had metadata). The kastore code is fairly minimal. However I agree that it makes sense to have a small kastore wrapper in tskit that that is specifically for metadata.

[jeromekelleher]

I can definitely see the value of adding this to the Python API, and it would be simple enough to implement. Pulling the metadata out of the kastore and decoding it is definitely handy, and not trivial to implement. The only downside is a run-time dependency on the kastore Python package, but that's no big deal. I guess the simplest thing about be to call the function load_metadata or something? (Or maybe peek_metadata?)

I'm less enthusiastic about adding it to the C API, mainly because it really only is a few lines of kastore code to do (and anyone using the tskit C API already has kastore). Since we're not decoding the metadata in C, tskit would just give you back a bunch of bytes. Then, would you want to return just the metadata, or would you want the schema also? The only argument for keeping it in tskit I would imagine is that so client code wouldn't need to know about the internal file format, but it's pretty simple and I would be very surprised if the way of accessing the top level metadata ever changed. We probably haven't made any formal contracts about the file format, but the top level details won't change now.

I could be convinced otherwise though, of course, if anyone wants to suggest a C API.

[bhaller]

I'm less enthusiastic about adding it to the C API, mainly because it really only is a few lines of kastore code to do (and anyone using the tskit C API already has kastore). Since we're not decoding the metadata in C, tskit would just give you back a bunch of bytes. Then, would you want to return just the metadata, or would you want the schema also?

Schema also, I suppose?

The only argument for keeping it in tskit I would imagine is that so client code wouldn't need to know about the internal file format, but it's pretty simple and I would be very surprised if the way of accessing the top level metadata ever changed. We probably haven't made any formal contracts about the file format, but the top level details won't change now.

I could be convinced otherwise though, of course, if anyone wants to suggest a C API.

Well, we can certainly do that "few lines of kastore code" ourselves in SLiM. Seems like a cleaner/better design for those lines to live inside tskit, and I personally would have no idea what those few lines of kastore code would actually look like so there's that :->, but with a helping hand I'm sure we can get it done. As for suggesting a C API, perhaps that would be best done by @petrelharp? I have no idea. :->

[jeromekelleher]

Thinking about this a bit more, I think the right approach is to add a new flag TSK_LOAD_SKIP_TABLES (or something) which is provided to table_collection_load. With this flag, only the top-level data for a tree sequence will be loaded in and set on the table collection instance. The tables will be ignored, and therefore have zero rows in the returned table collection/tree sequence.

This is the simplest approach because:

a) We'll nearly always want to look at things like the file format version anyway if peeking at the metadata
b) We don't need to come up with any new APIs or ways of packaging up just the metadata/schema. It'll just be there on an empty table collection/tree sequence.
c) It'll be easy to implement - just put a conditional around the x_table_load functions here

This will also work in Python, as we can add a boolean flag to the load function and it'll take care of itself.

We'd need some help to get this done if you'd like it before 1.0.0 @bhaller/@grahamgower - we're already snowed under with stuff to get wrapped up.

[bhaller]

I'm happy to take a crack at it, but will probably flail. :-> It looks like tsk_table_collection_load() has an options already, so I can define a new flag for that, and I guess add a bool flag to tsk_table_collection_loadf_inited() that uses that passed in; that's not public API I guess, so I can add a bbol to its interface I suppose? Then obvs easy to if out the x_table_load functions in tsk_table_collection_loadf_inited() based on that flag.

I have absolutely no idea how to do the Python-side work at all, though – you say "add a boolean flag to the load function" but mysteries abound. :-> Wouldn't know how to test the python side (or the C side, for that matter) properly even if I did get it working. Also not sure what the protocol is for all of this – fork tskit, make a local branch in my fork, commit something to that local branch, push the commit to my GitHub fork, turn it into a PR in GitHub? And then do I need to do something to make it a PR for the "real" tskit, not just for my fork of tskit? You probably don't realize how clueless I am in Git, I'm very accustomed to working in my own fiefdom and I never do anything complicated. :-> Best might be if I could watch someone else like @grahamgower or @petrelharp fix this over zoom; I'd certainly be interested in learning how to do this kind of minor fix!

[jeromekelleher]

Thanks @bhaller! We could leave out the Python side of things on the initial pass, it's an easy one to fill in later.

Re the workflow, the dev docs should answer your questions (if they don't, it's a bug). We've just done a format switch, so if you see any issues, please do let us know.

[bhaller]

Wow, you guys are organized – dev docs! OK @jeromekelleher, I will take a crack at it in the next couple days, unless @clwgg (who started this whole thing! :->) wants to have a go. :->

[clwgg]

I'm happy to give it a try!

[jeromekelleher]

Great! Feel free to ping questions here if you hit any issues @clwgg!

[benjeffery]

@clwgg @bhaller this is a great opportunity to debug the dev docs, so please don't hold back on corrections or suggestions.

[clwgg]

@benjeffery thanks, the dev docs are pretty fantastic! They made it much much easier to get into the project from the outside, especially when touching the backend for the first time. I just opened a PR that fixes two broken links I found (#1880) -- otherwise I don't think they missed anything I felt I needed.

[clwgg]

alright, I just opened a (very-much-a-)draft PR for this feature. looking forward to your feedback (: