Pinning Service API

[lanzafame]

A Pinning Service is a service that accepts hashes from a user and will host the associated hash, i.e. Pinata, Infura, et al.

The rationale behind defining a pinning service API is to have a baseline functionality and interface that can be provided by these services so that tools can be built on top of a common base of functionality.

Draft pinning service api: https://app.swaggerhub.com/apis/lanzafame/ipfs-pinning-service/v0.0.0

Latest draft: https://app.swaggerhub.com/apis/lanzafame/ipfs-pinning-service/0.0.1

//cc @obo20 @MichaelMure

Anyone who knows someone running an IPFS pinning service, please tag them here, thanks. @parkan

[agentofuser]

It would be very helpful for ipfs-deploy if pinning services had a common API.

My impression is that this was going to be sort of standardized by https://github.com/ipfs-shipyard/cube, but I don't know what's the status on that project.

[lanzafame]

@agentofuser Yes, ipfs-deploy is a perfect example of the kind of tooling that would greatly benefit from this common API definition 👍.

The Cube project is building a pinning service that is easy to deploy for you and your friends/family. It has it's own pinning service API definition and is another potential adopter of this common API.

[lanzafame]

Sidenote: Utilizing DID in the security/user-id component of the API would be ideal.

[parkan]

➡️ a huge 👍 to this in general

➡️ a lot of object store services to date have taken to being S3 api compatible, we probably can't follow this 1:1 but I wonder if being nearly-compatible or at least inspired by would help folks migrate? in particular, stuff like legal holds probably encodes enough real-world experience to be worth looking at

➡️ this may be out of scope for this issue, but an alternate "API" I envision is a "re-pin everything I pin" service: one way this could work would be following a pubsub channel that announces new items

[lanzafame]

this may be out of scope for this issue, but an alternate "API" I envision is a "re-pin everything I pin" service: one way this could work would be following a pubsub channel that announces new items

This is something that Cluster is looking to provide but it is beyond what I envision being provided by a straight IPFS pinning service. It also has a lot of technical and security challenges.

a lot of object store services to date have taken to being S3 api compatible, we probably can't follow this 1:1 but I wonder if being nearly-compatible or at least inspired by would help folks migrate? in particular, stuff like legal holds probably encodes enough real-world experience to be worth looking at

Not opposed to this idea, I would still want to define this API as a way of clarifying the language around what pinning is and its different forms/options so that developers in the IPFS world have a common language to talk about it with.

[bonedaddy]

Having started to work with gRPC more, I think if the idea is to have a unified pinning service API that can be implemented across multiple providers, gRPC is a must have for a few reasons:

• The API can be defined in a single language, with reproducible ways of building clients+servers across a wide variety of language, and overall reduced time for developers involved whether they be those that are creating the API, or those that are using the API.
• Documentation can't get out of date due to the self-generating nature of protocol buffers + gRPC. Any update to the design of the API can be reflected in the API documentation automatically, or manually with very few commands. This i think is a major feature to have as across the IPFS ecosystem (official projects, non official projects, etc..) have issues with documentation being up-to-date. This is easily explained by the ecosystem moving so quickly that documentation is hard to keep updated, so anything that makes this easier is a must have imo
• Swagger integration
• Allows room for "implementation specific features" (ie, company A can implement all features of the API, while company B can choose to only implement 50% of the features). Protobuffs and gRPC handle this extremely easy
• I suppose GraphQL could be used, but it's slower than gRPC
• Protocol buffers are already being used in some libp2p/ipfs projects
• Enable service-interoperability between multiple service providers easily, since one's grpc server could also run a small client alongside it to forward pin requests to other providers and not have to worry about missing feature compatibility due to gRPC amazingness

These are my initial thoughts for now, however if I have more i'll update this post.

[lanzafame]

@postables I agree that gRPC would be awesome and it does seem to be coming along in this space, see Improbable's grpc-web blog post and gRPC's web protocol. My biggest issue with gRPC is that they don't allow for custom transports yet™.

Did you have any thoughts on the actual API defined here: https://app.swaggerhub.com/apis/lanzafame/ipfs-pinning-service/v0.0.0? In terms of usability? or how it integrates with your payment system? or the use of JWT to identify who pinned a cid?

[parkan]

my $0.02: broadly speaking, I'm hugely in favor of well-defined, testable, and documented APIs that 3rd parties can implement

however, gRPC/protobuf might be excessive for an API of limited complexity, small message size (aside from files themselves), and relatively low throughput requirements

you can still get the validation upside just by using swagger, and the logic duplication is pretty mild for an API of this cale

[bonedaddy]

@lanzafame Thanks for the links i'll check them out.

As for initial thoughts on the API that's define so far, it looks quite nice, and would be pretty easy to integrate with as is. The JWT in my system uses id instead of user_id so my personal preference would be to have id, however there isn't anything wrong with user_id and represents a fairly trivial switch for my backend to handle id.

Additionally I believe some other blockchain based authentication systems also use JWTs sousing JWT's in general is a good idea, as it helps encourage interoperability with different types of auth systems.

As for the payment system, my only suggestion would be to add 409 as a valid response status code that can be returned for all the POST /pins and POST /pins/{cid} calls, to indicate that a payment is required for this particular action. In the case of my API this is sent as a response code when a user doesn't have enough balance in the account to cover the storage costs. But generally speaking, this might be a good status code to have to allow various pinning service providers to handle payments in their normal ways.

The other suggestions I have are:

• Make 500 internal server error a response code that's part of the API
• POST /pins incorrectly lists 500 as an unauthorized status code.
• Make 401 unauthorized as response code that's part of the API

Those minor nits aside, this is off to a great start already 👌

@parkan That's a valid point and I suppose if the API never goes beyond it's current scope in functionality gRPC/protobuf would definitely be excessive

[lanzafame]

@postables I have updated the API with your corrections and suggestions, thanks.
And if anyone is wondering about the 409 error it is a throwback to RedAlert 2 😄.

    InsufficientFunds:
      description: Insufficient Funds (409)
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'

[bonedaddy]

Excellent 👌 thanks for implementing the suggestions, I think this gives a lot of room for overlaying this API ontop of existing implementations different service providers have.

Grpc using multiple transports is not ready and I suspect will come out around the same time a the final RFC for http/3 which is July. So one month.

I the meantime you can use grpc and convert it forward and back at design time using gnostic.
Google made this and I suspect for the same use cases / architectural reuse use cases.

So you can define thing in grpc and share the API to non golang projects over openapi. Those users can then gen a proxy client from the openapi spec into whatever language they are using.

The last bit is data validation. Open api is rich and uses JSON schema. So the client can do client side validation. I think gnostic supports translation of grpc data validation but not sure since I was last using it.

Grpc over quic ( uni- directional ) and grpc over webrtc ) bi directional ) will make things very easy. Pions have a great webrtc golang stack now with good momentum.

Ultimately grpc generators that support these new transports for other languages will be years away. Just how it is

Quic and Webrtc are both likely to get browser full support do that client will be in good shape.
However your always going to be stuck for other servers, desktops and mobiles that want client proxy code in their languages.
So gnostic and openapi will always be needed. OpenAPI suggests REST and HTTP and so all the push aspects are lost and so hooks need to be designed in to the base architecture ?

[bonedaddy]

@lanzafame As mentioned via email, I'm happy to get started on an open-source implementation of the Pinning Service API, and implement it within Temporal.

However before I get started it's probably best worth deciding on how it gets implemented. Since there isn't any consensus yet on whether or not to implement gRPC, I'm wondering if its best to start implementing the API using go-chi/chi. Although the nice thing with gRPC is we can actually tackle both a gRPC, and HTTP implementation of the Pinning Service API using https://github.com/grpc-ecosystem/grpc-gateway.

[lanzafame]

@postables that is awesome! thanks very much.

I think the interface should be defined in HTTP terms for now as not everyone supports communicating via gRPC on the server side, i.e. HTTP proxy<->gRPC service. So if Temporal supports gRPC, which I believe it does, you are free to use grpc-gateway, as long as the HTTP API that it produces is compatible with the swagger spec. This way we can move forward with this now and then once gRPC gets native support for other transports, we can reassess and go with gRPC in the future.

Also, I was just looking at the openid spec and it has a standard claim for the user id so I will be updating user_id to sub.

[yangwao]

I've just seen your lighting talk from IPFS camp 2019 and you have a perfect idea that could rise more pinning services around! Looking forward :)
🎥 https://www.youtube.com/watch?v=zdAvXTpBIfc&list=PLuhRWgmPaHtQVNQcBaCKg5kKhfOBv45Jb&index=12