RPC over REST

[Stebalien]

Re: #224

I'd like to move in the opposite direction: switch to a transport-agnostic RPC API internally so we can support multiple RPC protocols/libraries and multiple transports. I'm concerned that REST, specifically the use of verbs like PUT, DELETE, etc.) will tightly couple us to HTTP.

Why:

1. Eventually, I'd like to support efficient memory sharing RPC systems using unix domain sockets, memory maps, and file descriptor passing.
2. I'd like to avoid re-implementing bindings for every command when we do this.
3. I'd like multiplex multiple commands over a single connection. HTTP/2 can do this but it requires TLS which requires a CA cert.
4. Progress reported by ipfs get and ipfs cat is currently incorrect because we can't accurately calculate the size of the response before we send it. To fix this, we'll likely need to send chunks of data along with progress information. However, that will break curl -o file .../api/v0/get/ipfs/QmFoo.
5. I'd like to drop everything related to multipart.

---

Thoughts? What about JSON RPC (or some similar protocol, ideally using CBOR)?

[lidel]

I must admit, current HTTP API comes with limitations, even in web context: "realtime" updates are not possible without constant polling over HTTP (ipfs-companion, ipfs-desktop) and REST does not improve that. Interactive web applications would benefit from RPC-based API over websockets, so ideally we need something transport-agnostic.

Some RPC candidates were previously discussed in ipfs/go-ipfs-cmds#37 and other places, recurring ones:

• JSON-RPC / OpenRPC - suggested in JSON-RPC interface kubo#6420, used by Ethereum. It is super simple (curl examples, web3.js is using it as-is). No spec for binary/CBOR version (yet?), no streaming?
• gRPC - language-agnostic RPC with IDL for code generation. Bidirectional streaming RPC. Uses muxing internally (pipes). If we care about maintenance and generating clients instead of crafting them by hand it may be worth evaluating. Seems to have the best tooling?
• capnproto - "fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster." Least popular, no muxing?
• libp2p 👀

Is anything else worth adding to the list? What is IPFS Cluster using for RPC?

[warpfork]

FWIW, I don't think it would be at all out of line to think about specing some sort of RPC-ish system that's codec agnostic and under either the umbrella of either IPLD or libp2p.

Pretty much all the other options out there seem to be tightly bound to a specific single codec, and that strikes me as... unfortunate.

We've already done some protocols that are defacto doing this -- e.g. Graphsync, which is doing something RPC-like (without having an explicit RPC library) over IPLD and incidentally using CBOR. It seems to work fine.

[warpfork]

Also, quick blog post I feel is borderline obligatory to link whenever "RPC" is brought up, either as a thing to use or as a thing to design: https://incubaid.wordpress.com/2012/03/28/the-game-of-distributed-systems-programming-which-level-are-you/

It contains some very useful cautionary tales.

[aschmahmann]

I'd recommend at least considering a strategy that is compatible with libp2p. People frequently want to build applications that use an RPC pattern (even if they are mere Level 1s 😃) and are confused by how to do it via libp2p. Sometimes they literally just use pubsub to transport messages because there is no obvious answer available.

For example, there was some work to make gprc utilize libp2p at libp2p/libp2p#26 and https://github.com/paralin/go-libp2p-grpc

[achingbrain]

My experience of RPC has led me to a few observations.

1. RPC exposes the internal workings of a system to the outside world

By the very nature of RPC, operations, inputs, outputs and general structure are tightly coupled to the remote implementation.

This in turn tightly couples client implementations to the remote implementation since there is no translation layer between them, which means every client feels pain every time you want to refactor something.

You also end up leaking the conventions of the remote implementation into clients which may be in a different language and/or have different expectations and norms.

You could, could, write a translation layer, but no-one does that. They just think 'Oh cool, this gives me everything I want' and start passing data types around. Then you are trapped, doomed to an endless loop of refactors that must be important to someone but seem utterly pointless to you.

2. RPC tends towards a large API surface area

Because RPC takes a granular approach this leads to lots of operations and then lots of code being required to support those operations. A better approach is to have fewer generic resources that the client can orchestrate to perform complex operations. IPFS is in a nice place where most of its operations are idempotent so we should really leverage that.

3. The network is not free, don't pretend that it is

The page @warpfork links to makes this point about Java RMI. Been there, been burnt, not massively keen on going back.

Having RPC be completely transparent is a neat trick but encourages the developer to make lots of RPC calls which will end up with the developer wondering why their application performance is so bad.

4. Optimising your application for the transport layer is a mistake

If HTTP is the thing that's making your application slow, you are doing pretty well.

[warpfork]

A better approach is to have fewer generic resources that the client can orchestrate to perform complex operations. IPFS is in a nice place where most of its operations are idempotent so we should really leverage that.

SUPER well said. 👏 👏 👏

Idempotency is a superpower and we should absolutely lean on it, at all times and as much as possible. It makes better things happen with less work and higher reliability.

(RPC frameworks are concerning because tend to lead people off that path quite easily unless there's diligent awareness of this. I'll agree it's not "inevitable", but yeah... it's a tendency, and a concerning one. It doesn't mean we shouldn't do an RPC system, or at least document patterns around it, but it's a note to keep forefront in mind.)

[hacdias]

We now have defined HTTP specifications at specs.ipfs.tech - the RPC API concerns Kubo only, as an implementation, and no longer belongs in this repository. Therefore, I'm closing this issue. Feel free to move this to Kubo if you feel like it's useful.