Description
The OpenAPI 2.0 Specification states that its goal is “to define a standard, language-agnostic interface to REST APIs”. REST is an extremely popular style for implementing APIs that run over HTTP and related protocols (HTTPS, HTTP/2, QUIC). But other transport mechanisms are common and APIs are often defined in ways that correspond to procedure calls. In some ways, procedure calls are more fundamental, as REST APIs are often implemented by systems that convert REST requests into procedure calls.
At times it is desirable to use remote procedure calls (RPCs) as the primary form of an API. RPC systems allow expression of semantics that go outside the range of the HTTP verbs used by REST. RPC systems map naturally to the function calls that are used to implement most software systems, and RPC provides a design pattern that can be use to guide system implementations. The RPC abstraction is also much older than REST, and new systems such as gRPC and Thrift show its enduring usefulness.
With this in mind, we think it would be beneficial to expand the scope of the OpenAPI specification to include remote procedure call semantics. Here we propose an expanded specification that builds on our experience using RPCs to connect very large distributed systems while being general enough to apply to a broad range of variations.
Background
Protocol Buffers
We give special consideration to the Protocol Buffer message representation. At Google, Protocol Buffers are used to send and receive tens of billions of messages each second. Protocol Buffers are also used for messages in the open-source gRPC framework and other more specialized RPC frameworks [1] [2] [3].
Protocol Buffers are typically represented in a binary form, and the latest version (proto3) also allows mapping to representations in JSON and YAML formats. Messages can be generally treated as collections of typed fields, just as they are in other serialization formats including Thrift, Avro, and Cap’n Proto. Thus message description can be treated similarly for all of these representations, including the JSON and YAML that OpenAPI uses to describe request and response objects.
Protocol Buffer interfaces are defined using a special description language (.proto) that describes messages and the services that use them. This language is different from the JSON and YAML used for OpenAPI specifications. To keep the specification and tooling simple, this proposal replaces this language with extensions to OpenAPI that carry the same meanings. It seeks to express as much as possible of the .proto language within OpenAPI so that if desired, .proto files can be converted into OpenAPI RPC descriptions and vice versa.
RPC Elements
Messages
Remote procedure calls are described in terms of the messages that they send and receive. Messages contain fields that have names, types, and other attributes, often including an integer field number that is used in message serialization.
Services
In RPC terminology, "Services" are collections of remote procedure calls that send and receive messages. Each call has a single input and output message, and some RPC implementations will allow input and output messages to be streamed in a single call. Thus each remote procedure call is described by its name, input type, output type, and whether or not the input and output messages are streamed.
Example
Here we show an example .proto description of an API. For illustration, we've specified the GetBook RPC as a streaming call that accepts a stream of book requests and returns a stream of books. This is indicated with the stream keyword that appears before the request and response types.
syntax = "proto3";
package examples.bookstore;
import "google/protobuf/empty.proto";
import "google/protobuf/struct.proto";
service Bookstore {
rpc ListShelves(google.protobuf.Empty) returns (ListShelvesResponse) {}
rpc CreateShelf(CreateShelfRequest) returns (Shelf) {}
rpc GetShelf(GetShelfRequest) returns (Shelf) {}
rpc DeleteShelf(DeleteShelfRequest) returns (google.protobuf.Value) {}
rpc ListBooks(ListBooksRequest) returns (ListBooksResponse) {}
rpc CreateBook(CreateBookRequest) returns (Book) {}
rpc GetBook(stream GetBookRequest) returns (stream Book) {}
rpc DeleteBook(DeleteBookRequest) returns (google.protobuf.Value) {}
}
message Shelf {
string name = 2;
string theme = 3;
}
message Book {
string author = 2;
string name = 3;
string title = 4;
}
message ListShelvesResponse {
repeated Shelf shelves = 1;
}
message CreateShelfRequest {
Shelf shelf = 1;
}
message GetShelfRequest {
int64 shelf = 1;
}
message DeleteShelfRequest {
int64 shelf = 1;
}
message ListBooksRequest {
int64 shelf = 1;
}
message ListBooksResponse {
repeated Book books = 1;
}
message CreateBookRequest {
int64 shelf = 1;
Book book = 2;
}
message GetBookRequest {
int64 shelf = 1;
int64 book = 2;
}
message DeleteBookRequest {
int64 shelf = 1;
int64 book = 2;
}
Open API Representation
Messages
RPC messages are described in the OpenAPI definitions section. To these we add a few new fields to provide protocol buffer semantics. New fields are prefixed x- but in the accepted proposal, we would expect all x- prefixes to be deleted.
Properties of a message correspond to fields in a protocol buffer message. x-field-number (or fieldNumber) is a required integer property that associates a unique field number with each property. x-repeated is an optional boolean property (by default false) that, when true, indicates that a field may occur more than once.
Here we show an example OpenAPI representation of the messages in the Bookstore API.
definitions:
"Shelf":
type: object
properties:
name:
type: string
x-field-number: 2
theme:
type: string
x-field-number: 3
"Book":
type: object
properties:
author:
type: string
x-field-number: 2
name:
type: string
x-field-number: 3
title:
type: string
x-field-number: 4
"ListShelvesResponse":
type: object
properties:
shelves:
allOf:
- $ref: "#/definitions/Shelf"
- x-repeated: true
- x-field-number: 1
"CreateShelfRequest":
type: object
properties:
shelf:
allOf:
- $ref: "#/definitions/Shelf"
- x-field-number: 1
"GetShelfRequest":
type: object
properties:
shelf:
type: integer
format: int64
x-field-number: 1
"DeleteShelfRequest":
type: object
properties:
shelf:
type: integer
format: int64
x-field-number: 1
"ListBooksRequest":
type: object
properties:
shelf:
type: integer
format: int64
x-field-number: 1
"ListBooksResponse":
type: object
properties:
books:
allOf:
- $ref: "#/definitions/Book"
- x-repeated: true
- x-field-number: 1
"CreateBookRequest":
type: object
properties:
shelf:
type: integer
format: int64
x-field-number: 1
book:
allOf:
- $ref: "#/definitions/Book"
- x-field-number: 2
"GetBookRequest":
type: object
properties:
shelf:
type: integer
format: int64
x-field-number: 1
book:
type: integer
format: int64
x-field-number: 2
"DeleteBookRequest":
type: object
properties:
shelf:
type: integer
format: int64
x-field-number: 1
book:
type: integer
format: int64
x-field-number: 2
Services
Services are a distinct new entity that we represent using the x-services key at the top level of the OpenAPI description. This allows an API to include both RPC and REST representations side-by-side.
Services are described by their name and the procedures they contain. Procedures are described by the objects they accept and return. Streamed values are indicated with x-streaming, an optional boolean property (by default false).
Here is an example OpenAPI representation of the services in the Bookstore API (in this case, a single service named "Bookstore").
x-services:
"Bookstore":
x-procedures:
"ListShelves":
x-accepts:
$ref: "google-protobuf.yaml#/Empty"
x-returns:
$ref: "#/definitions/ListShelvesResponse"
"CreateShelf":
x-accepts:
$ref: "#/definitions/CreateShelfRequest"
x-returns:
$ref: "#/definitions/Shelf"
"GetShelf":
x-accepts:
$ref: "#/definitions/GetShelfRequest"
x-returns:
$ref: "#/definitions/Shelf"
"DeleteShelf":
x-accepts:
$ref: "#/definitions/DeleteShelfRequest"
x-returns:
$ref: "google-protobuf.yaml/#Value"
"ListBooks":
x-accepts:
$ref: "#/definitions/ListBooksRequest"
x-returns:
$ref: "#/definitions/ListBooksResponse"
"CreateBook":
x-accepts:
$ref: "#/definitions/CreateBookRequest"
x-returns:
$ref: "#/definitions/Book"
"GetBook":
x-accepts:
allOf:
- $ref: "#/definitions/GetBookRequest"
- streaming: true
x-returns:
allOf:
- $ref: "#/definitions/Book"
- streaming: true
"DeleteBook":
x-accepts:
$ref: "#/definitions/DeleteBookRequest"
x-returns:
$ref: "#/definitions/Shelf"
Discussion
Our intent is to make it possible to write tools that convert back-and-forth between .proto and OpenAPI representations and to build RPC flows that completely replace .proto inputs with OpenAPI RPC specifications.
OpenAPI representations are more verbose than .proto, but are more amenable to automated processing and custom editors like swagger-editor. Our hope is that this representation will lead more editors and other tools to support RPC APIs.
Types in OpenAPI and Protocol Buffers
Protocol Buffers contain many finely-differentiated scalar types, while the OpenAPI spec uses general types such as number and integer. In OpenAPI, an additional format field supplements the type field with additional representation detail, so our proposal uses this field to include the full name of the corresponding Protocol Buffer type.
| .proto field type | OpenAPI type field value |
OpenAPI format field value |
|---|---|---|
| double | number | double |
| float | number | float |
| int64 | integer | int64 |
| uint64 | integer | uint64 |
| int32 | integer | int32 |
| fixed64 | integer | fixed64 |
| fixed32 | integer | fixed64 |
| bool | boolean | - |
| string | string | - |
| group | - | - |
| message | - | - |
| bytes | binary | - |
| uint32 | integer | uint32 |
| enum | ? | ? |
| sfixed32 | integer | sfixed32 |
| sfixed64 | integer | sfixed64 |
| sint32 | integer | sint32 |
| sint64 | integer | sint64 |
In the above table, the bool, string, and bytes types directly correspond to OpenAPI types and need no additional detail in the format field. The group type in .proto is deprecated and unsupported, and the message type corresponds to inclusion of another message, which is represented in OpenAPI with the $ref property.
The enum type is represented in .proto as an integer and in OpenAPI as a string. This difference is unresolved in this proposal.
Gaps between OpenAPI and Protocol Buffers
There are some Protocol Buffer features that aren’t yet covered by this proposal. Gaps that are significant and unresolved may be addressed in future proposals.
Default values
Default field values could be represented with the existing OpenAPI default property, but in some message representations (such as proto3, default values are specified for each type and are not modifiable.
Enumerations
Enumerations are represented with strings in OpenAPI and integers in .proto. Resolution of this is a high priority future proposal.
Maps
The .proto format allows fields to have map types which include type specifications for map keys and values. When serialized, these maps are represented with special automatically-created messages.
Here we omit further discussion of maps, leaving it as a more general question about the OpenAPI Specification.
Extensions
Extensions are defined in proto2 and allow additional fields to be added to messages and for ranges of field numbers to be reserved for third-party extensions. Extensions are not supported in proto3 and are omitted from this proposal.
Nested messages
The .proto language allows messages to be defined inside other messages. Nested types can’t be written directly in OpenAPI, but we can define messages with hierarchical names similar to the ones that would be implied for nested .proto messages.
Options
In .proto, options are predefined annotations that can be added in various places in a .proto file. Common options configure code generators (by specifying package names or class name prefixes) or map RPC procedures to REST methods. Options are not addressed here but are a priority for inclusion in a future proposal.
Oneof
In .proto, the oneof statement groups fields to indicate that only one of the grouped fields can be included in a message. There is no corresponding concept in OpenAPI 2.0, but this appears to be addressed by a pending pull request.
Packages and API Versions
.proto descriptions optionally include a package name. This has a similar purpose as the basePath field in the OpenAPI root object and we suggest that either a new field named package or the existing basePath be used for this.
API Versions are commonly indicated by the last segment in a package name (segments are separated by periods). When the last segment looks like a version name (beginning with a 'v' and following with a possibly-dotted number), it is used as the version name. We omit this convention from this proposal and leave the representation of API versions to the existing version field of the info object.