proposal: Go 2: capability annotations

[bcmills]

Background

This is a proposal about constraints on types and variables. It addresses the use-cases of “read-only” views and atomic types, but also generalizes over channel constraints in the process.

This proposal is intended to generalize and subsume many others, including (in no particular order) #22876, #20443, #21953, #24889, #21131, #23415, #23161, #22189, and to some extent #21577.

Proposal

Fields and methods on a struct type can be restricted to specific capabilities. To callers without those capabilities, those fields and methods are treated as if they were unexported and defined in some other package: they can be observed using the reflect package, but cannot be set, called, or used to satisfy interfaces.

A view is a distinct type that restricts access to the capabilities of its underlying type.

Grammar

Capability restrictions follow the grammar:

Capability = "#" identifier { "," "#" identifier }
View = "#" ( identifier | "{" [ identifier { "," identifier } ] "}" )
VarView = "#&" ( identifier | "{" [ identifier { "," identifier } ] "}" )

A Capability precedes a method or field name in a FieldDecl or Receiver. A method or field without an associated Capability can be accessed without any capability.

A View follows a TypeName in a ParameterDecl, FieldDecl, ConstDecl, VarDecl, or Conversion. A TypeName without an associated View includes all of its capabilities. An empty View (written as #{}) can only access methods and fields that are not associated with a Capability.

A VarView follows the IdentifierList in a FieldDecl or VarDecl. It restricts the capabilities of references to the declared fields or variables themselves, independent of type. Those capabilities are also applied to the pointer produced by an (explicit or implicit) address operation on the variable or field.

A package may define aliases for the views of the types it defines:

type T struct { … }

func (*T) #Reader Read(p []byte) (int, error) { … }
func (*T) #Seeker Seek(offset int64, whence int) (int64, error) { … }
func (*T) #Writer Write(p []byte) (int, error) { … }

type *T#ReadSeeker = *T#{Reader,Seeker}

Built-in capabilities

Channel types have the built-in capabilities Sender and Receiver. <-chan T is a shorthand for (chan T)#Receiver, and chan<- T is a shorthand for (chan T)#Sender. The send and close operations are restricted to the Sender capability, and the receive operation is restricted to the Recevier capability.

Slices, maps, and pointers have the built-in capabilities Getter and Setter. (QUESTION: should we defined channel-like shorthands for these capabilities?) The Setter capability allows assignment through an index expression (for slices, maps, and pointers to arrays) or an indirection (for pointers), including implicit indirections in selectors. The Getter capability allows reading through an index expression, indirection, or range loop.

Pointers to numeric, boolean, and pointer types have the built-in capability Atomic. The Atomic capability allows assignment and reading through the functions in the atomic package, independent of the Setter and Getter capabilities.

The Getter, Setter, and Atomic capabilities can also apply to variable and field declarations (as a VarView). The Getter capability allows the variable to be read, the Setter capability allows it to be written, and the Atomic capability allows it to be read and written via an atomic pointer. (A variable with only the Getter capability cannot be reassigned after declaration. A variable with only the Setter capability is mostly useless.)

The built-in len and cap functions do not require any capability on their arguments. The built-in append function requires the Setter capability on the destination and the Getter capability on the source.

Assignability

A view of a type is assignable to any view of the same underlying type with a subset of the same capabilities.

A function of type F1 is assignable to a function type F2 if:

• the parameters and results of F1 and F2 have the same underlying types, and
• the capabilities of the parameters of F1 are a subset of the capabilities of the parameters of F2, and
• the capabilities of the results of F1 are a superset of the capabilities of the parameters of F2.

A method of type M1 satisfies an interface method of type M2 if the corresponding function type of M1 is assignable to the corresponding function type of M2.

This implies that all views of the same type share the same concrete representation.

Capabilities of elements of map, slice, and pointer types must match exactly. For example, []T is not assignable to []T#V: otherwise, one could write in a T#V and read it out as a T. (We do not want to repeat Java's covariance mistake.) We could consider relaxing that restriction based on whether the Getter and/or Setter capability is present, but I see no strong reason to do so in this proposal.

Examples

package atomic

…
func AddInt32(addr *int32#Atomic, delta int32) (new int32)
func LoadInt32(addr *int32#Atomic) (new int32)
func StoreInt32(addr *int32#Atomic, val int32)
func SwapInt32(addr *int32#Atomic, new int32) (old int32)

package bytes

type Buffer struct { … }

func NewBuffer([]byte) *Buffer
…
func (*Buffer) #Owner Bytes() []byte
func (*Buffer) Cap() int
func (*Buffer) #Writer Grow(n int)
func (*Buffer) Len() int
func (*Buffer) #Reader Next(n int)
func (*Buffer) #Owner Bytes() []byte
func (*Buffer) #Reader Read(p []byte) (int, error)
func (*Buffer) #Reader ReadByte() (byte, error)
func (*Buffer) #Writer ReadFrom(r io.Reader) (int64, error)
…
func (*Buffer) #Owner Reset()
func (*Buffer) #Owner Truncate(n int)

package reflect

type StructField struct {
	…
	Index []int#Getter
}

package http

type Server struct {
	…
	disableKeepAlives, inShutdown #&Atomic int32
}

type Request struct {
	…
	#Client GetBody func() (io.ReadCloser, error)
	…
	#Server RemoteAddr string
	#Server RequestURI string
	#Server TLS *tls.ConnectionState
	…
	#Deprecated Cancel <-chan struct{}
}}

Reflection

The capability set of a method, field, or type can be observed and manipulated through new methods on the corresponding type in the reflect package:

package reflect

// ViewOf returns a view of t qualified with the given set of capabilities.
// ViewOf panics if t lacks any capability in the set.
func ViewOf(t Type, capabilities []string) Type

type Type interface {
	…

	// View returns the underlying type that this type views.
	// If this type is not a view, View returns (nil, false).
	func (Type) View() (Type, bool)

	// NumCapability returns the number of capabilities the underlying type is qualified with.
	// It panics if the type is not a view.
	func (Type) NumCapability() int

	// Capability(i) returns the ith capability the underlying type is qualified with.
	func (Type) Capability(int) string
}

type Method struct {
	…
	Capability string
}

type StructField struct {
	…
	Capability string
	View []string
}

Compatibility

I believe that this proposal is compatible with the Go 1 language specification. However, it would not provide much value without corresponding changes in the standard library.

Commentary

On its own, this proposal may not be adequate. As a solution for read-only slices and maps, its value without generics (https://golang.org/issue/15292) is limited: otherwise, it is not possible to write general-purpose functions that work for both read-only and read-write slices, such as those in the bytes and strings packages.

[ianlancetaylor]

How does it work if I assign a variable with a set of views to an empty interface value? Do I have to have exactly the correct set of views in order to type assert the empty interface value back to the original type? Do the views have to be listed in the same order?

[bcmills]

Good questions. I would say that you can type-assert to a view with any subset of the concrete capabilities, with the usual caveat that element types for maps, slices and pointers must be exact. As with interfaces, the first match in a switch wins.

The capabilities in a view are an unordered set, so you could enumerate them in any order.

[bcmills]

Another interesting question: what would == mean for two interface{} values with different views of the same object? Arguably it should be consistent with switch and map. Map keys should be unequal because they may have different method sets, which suggests that switch should require an exact match on the capabilities.

[ianlancetaylor]

One of the things I would like to see from any system like this is support for some form of thread safety analysis (https://clang.llvm.org/docs/ThreadSafetyAnalysis.html). Is there a way to use this syntax to say "this field requires this mutex to be held?" Unfortunately it seems kind of out of scope.

[bcmills]

This proposal might be possible to extend to locking invariants, but it would make the specification much more complex. As I see it, one of the advantages of this proposal is that it enforces the capabilities within the Go type system (rather than in an external tool), so extending it to thread-safety analysis would mean that we encode that analysis in the Go spec.

One way to do that might be to add an Escape capability to reference-like types and define a new (safe) Mutex type in terms of that, but then we would have to specify what Escape actually means. With that approach (and assuming generics), the Mutex API might look like:

package sync

type [T] Mutex struct {
	WithLock(func(*T#{Getter,Setter}))
}

whereas atomic would require the Escape capability:

package atomic

func [Ptr] SwapPointer(addr *(Ptr#Escape)#Atomic, new Ptr#Escape) (old Ptr#Escape)

Ptr#Escape applied to an existing view type Ptr would mean “Ptr augmented with Escape”. So, for example, concrete instantiations of SwapPointer might look like:

func SwapPointer[*T] (addr **T#Atomic, new *T) (old *T)

func SwapPointer[*T#Getter] (addr *(*T#{Getter,Escape})#Atomic, new *T#{Getter,Escape}) (old *T#{Getter,Escape})

[bcmills]

(I've realized in writing more examples that capabilities should associate loosely rather than tightly: too many parentheses. Updating examples.)

[odeke-em]

/cc @jaekwon

[bcmills]

One thing I dislike about this proposal is that it only provides upper bounds on capabilities, not lower bounds. For example, I cannot express “must be non-nil” as a capability under this formulation, even though it clearly relates to the Setter and Getter capabilities.

[Edit: can too. See below.]

[komuw]

/cc @wora