Possible updates with the use of non-Rabin algorithms

[DonaldTsang]

According to some papers, FastCDC and Asymmetric Extremum has better performance.
It might be a good idea to have algorithms that can achieve better speeds while being secure.
FastCDC: https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
AE reference: http://www.ijarcsms.com/docs/paper/volume4/issue6/V4I6-0021.pdf

[fd0]

Thanks a lot for the pointers! Please be aware that, at least for restic, we don't depend on the CDC algorithm being "secure" (whatever that means for a CDC algorithm) :)

[DonaldTsang]

@fd0 I do have a hint that this project is related to IPFS, so from their perspective it has to be secure.
I do think though that in the experiment stage, security is not a concern at all.

[fd0]

They are using a fork of this package, correct, but as far as I know they are also using it to just cut files into pieces and properly encrypt the pieces afterwards.

[DonaldTsang]

@fd0 And from what I see, they are currently stuck at a stage where the performance is not optimal.
That is why I am purposing something that can help the situation, hoping IPFS can improve.
ipfs-inactive/archives#134 ipfs-inactive/archives#136 ipfs-inactive/archives#137 ipfs-inactive/archives#142

[fd0]

I appreciate the pointers to more recent algorithms for CDC! I also think this repo/package is the wrong place. It is meant to be used by restic, our backup program, and we won't change the chunking algorithm lightly, because that will cause problems for our users. For a backup program, the repository/storage format is the most important thing.

Besides, in our application of Rabin fingerprints, we don't see any performance problems, the algorithm and this implementation performs great for us :)

The nice people from IPFS have forked the package to a new repo, here: https://github.com/whyrusleeping/chunker

I'm therefore closing this issue for now, please feel free to add further comments. Thanks!