Fresh perspective from learning IPFS

[JustinDrake]

I just spent about two weeks intensely studying IPFS (and learning Go) for an OpenBazaar project. (Ping me if you'd like to talk about OpenBazaar.) Going in I had some knowledge of cryptography, networking and decentralised networks. I think I wrapped my head around the "mountain of abstraction". I was prompted by @diasdavid to share some thoughts. Hopefully they will help some.

• Having a concrete project in mind (https://github.com/OpenBazaar/openbazaar-go) motivated me and helped me learn.
• As far as I can tell IPFS development is centralised around GitHub (a bit ironic? :) ). The most relevant GitHub accounts are @ipfs, @libp2p, @ipld and @multiformats.
• There's a deluge of repos, but don't be too intimidated. Most of the repos are small and really focused. I ended embracing the modular approach as it's very much in line with the overall philosophy of being protocol-driven and agnostic of implementation for everything.
• My goal was to build a strong understanding from the foundations up. The place to start for this is libp2p. Be ready to take in a lot of networking knowledge.
• The libp2p specs are a good starting point, but you have to dig into the code or discussions if you want meat.
• A good first-pass filter for repos is GitHub's language filter. There are "only" 34 repos in Go within libp2p.
• gx deps --tree is your friend. Understand how the various modules fit together is critical. The dependency tree is a great starting point. The --highlight= option is especially useful on large repos such as go-ipfs or go-libp2p.
• A good exercise is to try to separate out the repos by type (some are interface specifications, some are interface implementations, some are stubs).
• It's not all that clear which repos are actually used in production, and which are in development or not activated. For example, there are 5 transport implementations for go-ipfs, but as far I as can tell only go-tcp-transport is used in production. Things like the relay protocol or super node routing are in the codebase but not in use. A document summarising the development status of every component would help.
• It wasn't immediately obvious to me that the js-ipfs and go-ipfs are currently disjoint networks because there is no common transport layer and the relay protocol is not yet ready.
• There are various examples scattered around. I wish I knew about them earlier as they made my life easier. See for example the echo example, hosting a website or this js-libp2p benchmark.
• There's an open IPFS FAQ. It's a bit unconventional being built around permanently-open GitHub issues. I've asked 10 questions and got a reply for about half.
• For questions there's also the IRC channel #ipfs. I'm not very comfortable with IRC, but I did get some useful responses out of it. My hit rate was similar to the FAQ (about 50%). My plea to the IPFS team is to move to the 21st century and embrace Slack. 👍
• On IRC there are 5 "operators" (David Dias @diasdavid, Juan Benet @jbenet, Jakub Sztandera @Kubuxu, Lars Gierth @lgierth, Jeromy Johnson @whyrusleeping). I think these five guys do most of the heavy lifting and I look forward to get to know them better.
• The ipfs/notes repo is a treasure trove. Some of the notes I enjoyed include packet switching, pubsub, Signal protocol, petnames, Tor, I2P and chunking.
• The meaty bits are not where you expect. You have to work a bit to find them. The websites ipfs.io, libp2p.io and ipld.io are not yet loaded with docs. Most of the discussions happen in GitHub issues and pull-requests. I found discussion on IPFS paths quite fascinating.
• The IPFS project is moving extremely fast so it's good to bathe yourself in context. The roadmaps (e.g. js-ipfs, libp2p, go-libp2p) help, but I wish there was something even better.
• go-ipfs has a lot of code centred around the IPFS daemon. I think the daemon code should be segregated out into a new repository go-ipfs-daemon, with go-ipfs being as thin and clean as possible for others to reuse.
• A bunch of introductory talks by Juan on YouTube were useful. This talk by David was also useful.
• I found the "multi" nomenclature in multiformat confusing. For example, I initially assumed that a "multihash" was multiple hashes packed in one object. Replace "multi" in your mind with self-described.
• A lot of the development is guided by academic research in decentralised systems/protocols, which is cool. For example, I think this paper inspired the secio handshake.
• Learning about IPFS was a bit like delving into a new mini branch of mathematics, or a technology like Bitcoin. There are layers of abstraction with the main learning bottleneck being communication of concepts sometimes better explained by a human with diagrams and hand-waving. But once you get your ahead round the concepts, you realise they are actually quite natural and simple.

Onwards!

[dignifiedquire]

Thanks a lot @JustinDrake for writing this!

[whyrusleeping]

Hey @JustinDrake Thank you so much for all of this! This is great :)

What are the top three things we could do to make this whole process easier?

[jbenet]

Excellent post @JustinDrake ❤️ 🎉 -- I would love to translate it into an action item list of improvements we can make 😃

[RichardLitt]

I thought this deserved some more comments! :D

The most relevant GitHub accounts are @ipfs, @libp2p, @ipld and @multiformats.

As far as I know (and I should), those are the only ones that we run collaboratively. All of the others are private accounts (excepting, perhaps, https://github.com/ipfsbot).

There's a deluge of repos, but don't be too intimidated. Most of the repos are small and really focused. I ended embracing the modular approach as it's very much in line with the overall philosophy of being protocol-driven and agnostic of implementation for everything.

It is super nice to hear this from someone who hasn't been living in this project for decades. Thank you so much!

The libp2p specs are a good starting point, but you have to dig into the code or discussions if you want meat.

We really ought to have a list of amazing discussions. I know we have a subset of these on ipfs/awesome-ipfs; would be great if you could highlight some specific issues that helped you out so that we can add them there.

A good first-pass filter for repos is GitHub's language filter. There are "only" 34 repos in Go within libp2p.

• gx deps --tree is your friend. Understand how the various modules fit together is critical. The dependency tree is a great starting point. The --highlight= option is especially useful on large repos such as go-ipfs or go-libp2p.

NTS: We need to add GitHub search help to any guide to our GitHub organization. That's exactly what we hope users do, and we don't say it anywhere.

It's not all that clear which repos are actually used in production, and which are in development or not activated. For example, there are 5 transport implementations for go-ipfs, but as far I as can tell only go-tcp-transport is used in production. Things like the relay protocol or super node routing are in the codebase but not in use. A document summarising the development status of every component would help.

Eeesh... yeah, I know. What we could do is have 'date of last commit' on project-repos.ipfs.io. Would that be helpful?

It wasn't immediately obvious to me that the js-ipfs and go-ipfs are currently disjoint networks because there is no common transport layer and the relay protocol is not yet ready.

We should add this to their READMEs, then.

There are various examples scattered around. I wish I knew about them earlier as they made my life easier. See for example the echo example, hosting a website or this js-libp2p benchmark.

@diasdavid, how do you feel about having an 'examples' document that we could link these to? Alternatively, I could port these over to ipfs/examples...

move to the 21st century and embrace Slack. 👍

So muc noise. I wish we could do this, and we may at some point, but I don't think that would actively help us develop faster.

On IRC there are 5 "operators" (David Dias @diasdavid, Juan Benet @jbenet, Jakub Sztandera @Kubuxu, Lars Gierth @lgierth, Jeromy Johnson @whyrusleeping). I think these five guys do most of the heavy lifting and I look forward to get to know them better.

Sounds about right. Those are also the five who know IRC best, really. I think I should be an op, actually. And you can be one too; check out this thread.

The ipfs/notes repo is a treasure trove. Some of the notes I enjoyed include packet switching, pubsub, Signal protocol, petnames, Tor, I2P and chunking.
...
I found discussion on IPFS paths quite fascinating.
...
A bunch of introductory talks by Juan on YouTube were useful. This talk by David was also useful.

Could you PR these to ipfs/awesome-ipfs? I'm happy to do it, too.

The IPFS project is moving extremely fast so it's good to bathe yourself in context. The roadmaps (e.g. js-ipfs, libp2p, go-libp2p) help, but I wish there was something even better.

We're working on it! \o/

go-ipfs has a lot of code centred around the IPFS daemon. I think the daemon code should be segregated out into a new repository go-ipfs-daemon, with go-ipfs being as thin and clean as possible for others to reuse.

Interesting. Might be worth raising in ipfs/go-ipfs as an issue. I have no idea what kind of response this would get, though.

Replace "multi" in your mind with self-described.

I am going to add this to the multiformats/multiformats repo. This makes a lot of sense. Unless you want to? (Yes, I keep asking you to PR; I want to make sure you feel your contributions are properly attributed! I'm happy to do this, too, again.)

Learning about IPFS was a bit like delving into a new mini branch of mathematics, or a technology like Bitcoin. There are layers of abstraction with the main learning bottleneck being communication of concepts sometimes better explained by a human with diagrams and hand-waving. But once you get your ahead round the concepts, you realise they are actually quite natural and simple.

Can we put this on the website? (just kidding (no but really, we should. This is one of the nicest things. Thank you for this whole thread))

I was prompted by @diasdavid to share some thoughts. Hopefully they will help some.

Thank you, @diasdavid! :)

[jbenet]

` 👎 on slack. in its present iteration, it runs counter to so much of what we work for (decentralization, e2ee, open data, ...).

Learning about IPFS was a bit like delving into a new mini branch of mathematics, or a technology like Bitcoin. There are layers of abstraction with the main learning bottleneck being communication of concepts sometimes better explained by a human with diagrams and hand-waving. But once you get your ahead round the concepts, you realise they are actually quite natural and simple.

I really like this, and why i want to start producing some learning videos.

[JustinDrake]

Just for everyone's information, I do intend to make an actionable TODO list out of this. Please bear with me!

[jbshirk]

Probably also of interest
ipfs/notes#200

[jbshirk]

For a long time I had wondered what GX is (google was no help). Today I decided to finally track it down -- I had never considered that it is an IPFS project.

https://github.com/whyrusleeping/gx