proto: make the Message interface behaviorally complete

[dsnet]

Filing on behalf of @alandonovan.

The proto.Message interface is unsatisfactory. A behavioral interface is an abstraction over the underlying data types that exposes just the operations necessary for their correct use. By contrast, proto.Message exposes essentially no useful functionality and serves only as a marker. But a marker of what? If one could assume all its implementations were protoc-generated struct types with field tags, then at least it would be possible to write reflection-based algorithms that do useful things. However, there are many concrete types satisfying Message that are not of this form.

It's not only the set of implementations that is unbounded; the set of useful operations is also large and growing. The two most important, Marshal and Unmarshal, are handled quite cleanly since there are separate behavioral interfaces for Marshaler and Unmarshaler that allow each concrete type to implement these operations. But there are many functions in the proto API, for which no interface exists: proto.Merge, proto.Clone, the extensions API, and so on.

The cross-product of concrete implementations and operations is growing, but the fraction of these combinations that actually work is diminishing.

I think we should assess what it would take to change the proto.Message interface, and all its implementations, so that it is a true behavioral interface. This would require at a minimum that the interface include a new method that provides complete control over the abstract state of a message: accessing and updating its fields, inspecting any extensions or unrecognized fields, and so on, without revealing the concrete representation. It should be possible to implement all the major functions in the proto API, as well as most users' ad hoc functions, in terms of this interface so that they work with any concrete implementation. If an optimized version of a crucial operation is available, the generic implementation should dispatch to it, as it does today for Marshal and Unmarshal.

We can't add methods to proto.Message without breaking backwards compatibility. One approach we can take is to define proto.MessageV2 that is a much more semantically complete interface that provides a form of "protobuf reflection". In Marshal, Unmarshal, Merge, Clone, and so on, we can type assert if it implements proto.MessageV2 and use that interface to implement generic versions of those functions. If proto.Message doesn't satisfy proto.MessageV2, then Merge can just fail (it already does on most third-party implementations of proto.Message).

[jhump]

Interesting proposal. I faced a similar challenge when designing the public API for a dynamic message implementation.

If this goes anywhere, I'll be curious to see how similar they are in terms of shape/surface area. And I'll be excited to see how it can simplify that dynamic message's implementation (particularly around extensions, which I think is the weakest part of the current generated code).