Slicing chunks

[Stebalien]

In https://github.com/ipfs/unixfs-v2/blob/master/SPEC.md#file-data, it would be nice if it were possible to only include a sub-slice of the chunk. That is,

[
  [start, stop, CHUNK, [offset]]
]

Where offset is an offset into CHUNK.

---

Motivation: ipfs/notes#373

This would make it possible to implement a VCS on top of IPFS without having to re-implement half of git.

[andrew]

Would that also work for streaming video from a certain point?

[mikeal]

@andrew for video streaming you are going to want to use a chunker that can parse the video format and just make the chunks respect the key frame boundaries. If you don’t then you’ll just have to slice a larger chunk in order to serve the range request, which also already works.

This is something else, this let’s you take an arbitrarily large chunk of data where only a slice of the data is relevant file data. It’s only useful when a different program already produced data and somewhere in the data is the file data you want to references.

[mikeal]

I’ve been thinking about this feature a lot and I’m a little conflicted about it.

On the one hand, I like this feature. On the other hand, this part of the spec is the by far the hardest part to implement even without this feature.

When I initially wrote the spec it wasn’t written in a way that was agnostic of block boundaries and the recursion I had in my head was rather simple to implement. After the spec was updated to be property/path based I updated the implementation and when I hit this part of the spec it was much more difficult to implement (it may not even be complete yet). Even after we got the new JS IPLD stack ready this remained problematic while other areas had their code reduced a lot by using shared path libraries.

Adding this feature will further complicate writing implementations of unixfsv2 and we don’t yet have a good tool we can hand implementors to reduce that burden. We don’t yet have a selector that can return you all the chunks of a file, even against the current spec, much less do range queries.

With that in mind, this is the staging I’d like to see for this feature:

1. We implement selectors that can help with the relevant file iteration and range queries of files conforming to the existing spec.
2. We update the spec to note the selector(s) people should be using.
3. We implement selector improvements that could handle this new feature.
4. We update the spec to add this feature and note the new selector(s) implementors can rely on.

[Stebalien]

I agree this complicates the spec a bit but it shouldn't make it any harder to implement as implementations already need to support seeking. Given that this structure is recursive, this is literally just a seek + limit call.

[rvagg]

closing for archival