Consider libp2p mono repo

[vasco-santos]

Taking into account the overhead that we experience in libp2p development when sometimes for getting a feature in we need to orchestrate several PRs just to land a change, we should consider turning libp2p into a mono repo. This would also help new contributors as they would not need to know where other modules are and that they would need to be updated (like the interop, in case its tests fail in CI).

Some of the repos that should be considered are the daemon, daemon client and libp2p-interfaces.

This issue should be used for this discussion, where we should put together benefits, drawbacks and lessons learned from js-ipfs change into a mono repo.

cc @jacobheun @achingbrain @hugomrdias

[jacobheun]

The other really nice thing we could do with this not listed is releasing preconfigured libp2p nodes more easily with every libp2p release so users don't have to configure anything if they just want something ready to go.

[achingbrain]

Some related modules have been grouped together as monorepos since this was opened:

• https://github.com/libp2p/js-libp2p-daemon
• https://github.com/libp2p/js-libp2p-interfaces
• https://github.com/libp2p/js-libp2p-peer-id
• https://github.com/libp2p/js-libp2p-webrtc-star

The primary piece of learning from js-ipfs is that choosing what to add to a monorepo is very important, and that you should not add everything unless you like very long build times.

Modules should also be homogeneous in terms of technology, scope and testing - if you have one GUI app in a monorepo and ten tiny utils, releasing changes to the utils takes as long as releasing changes to the GUI app which is a bit nuts.

Also flaky tests become that much painful.

Also, also, contributors tend to get scared off by the scope of a monorepo.

I don't think there are other modules that need grouping together in the libp2p ecosystem so I'm going to close this issue but it's always something worth keeping an eye on.

[p-shahi]

The subject of migrating js-libp2p components into a monorepo was brought up by @MarcoPolo and discussed in the 2023-03-07 js-libp2p triage meeting with @achingbrain @wemeetagain @maschad and @p-shahi

The reasons for moving to a monorepo given were:

• It is painful for maintainers to manage separate repos
• Even though its cleaner conceptually to have separate repos, they are harder to track.
  • In practice there's added overhead even with CI bots to make things smoother.

The original proposed solution was to move any potential component a user pulls into a simple js-libp2p app (i.e. TCP, mplex, Yamux, Noise, WebSocket, WebTransport, Interfaces, etc.) into a monorepo.
However, we decided not to do this because:

• Yamux, mplex, Noise, and GossipSub are maintained by ChainSafe and should be separate repos (to have other parties in ecosystem be responsible for libp2p components)
• A simple app or user may not use all transports or components (some transports like TCP are not used by browsers)

So instead of one monorepo, our current thought is to create different groupings of smaller monorepos. These would look like:

• js-libp2p: libp2p/interop, interfaces (consolidate interfaces to add and others to exclude)
• js-libp2p-transports: WebRTC, WebTransport, WebSocket
• js-libp2p-protocols: js-libp2p-kad-dht, identify, ping
• js-libp2p-peer-discovery: js-libp2p-mdns, js-libp2p-bootstrap

@MarcoPolo also brought up the idea of using Git subtrees (where one can make changes to a repo directly and from a monorepo pull those changes in or you can make changes from monorepo and push changes out) as an option - but no consensus here.

POC: https://github.com/marcopolo/js-libp2p-monorepo

[achingbrain]

Having spent more time that I'd like to admit pushing changes around during the IPFS Althing I am becoming more amenable to this as I'd like to spend more time shipping features and less time shipping dependency updates.

After libp2p@0.45.x I'm going to set something up like this here:

• /packages/libp2p - the current repo
• /packages/interface - types from @libp2p/interface-libp2p - this will be the public API and will have no dependencies on anything we consider the internal API, the goal being this should not change often so projects like Helia can depend on it and be free of internal module churn
• /packages/interface-internal - a new module, @libp2p/interface-libp2p-internal that contains interfaces for internal components like ConnectionManager, TransportManager, etc. Modules in the services hash can depend on this to define the components they require to operate. By including this here it can be released alongside libp2p
• /packages/interop - this is for interop tests

Essentially the first bullet point from the comment above.

Docs, examples stay where they are.

[achingbrain]

Another problem that we have is releasing changes to modules ahead of libp2p.

For example the next libp2p release changes the peer store to be atomic, so that means:

1. Update @libp2p/interface-peer-store with the breaking changes and release a major
2. Update @libp2p/peer-store to implement the interface changes and release a major
3. Update consumers of the peer store @libp2p/bootstrap, @chainsafe/libp2p-gossipsub, etc and release majors
4. Update libp2p and release a major

Before 4 is completed there's a window during which js users see an update to a module, upgrade to it and then wonder why stuff doesn't work. Ts users will see a type error, though frequently they will ignore it and also wonder why stuff doesn't work.

We cannot control the release schedule of external modules (e.g. @chainsafe/libp2p-gossipsub) but being able to release 1-4 in one PR would go some way to reducing module churn and user surprise.

This adds a bit of weight to the whole-hog option of adding every supported libp2p module into the js-libp2p monorepo.