Protected instance members

[lrhn]

This is an attempt at a full feature specification for protected instance members, as another solution to #835.
Edit: And this is version 2 where the protected members can be overridden by subclasses.

Dart protected instance members

Dart has only one notion of declared access restriction: Library privacy.

Having private names solves two problems: Avoiding internal implementations having name clashes with third-party code, and ensuring what third-party code is able, and not able, to access, which makes it easier to ensure invariants.

Since the author controls the size and scope of a library, and is expected to be fully cooperative with all code they have the power to modify anyway, that’s a rather sweet spot of privacy. By scoping a library to a single class, or to what would be an entire directory of files in another language, the author can choose anything between class privacy and “package privacy”. It’s an accessibility where the author can control the scope.

One accessibility feature that other object oriented languages have, and that Dart cannot emulate with library privacy, is protected access: Instance members of a class which can only be accessed from subclasses which extend that class.

Protected members are useful for reusable libraries. A library can expose a base or skeleton class that third-party subclasses can extend to provide customized classes. However, if the subclass needs to interact with the base class, it can currently only do so if the base class exposes public API, which then contaminates the public API of the subclass with operations that end-users shouldn’t be using.

Dart provides the @protected annotation which makes the analyzer warn if a non-subclass invokes the annotated member. That’s not a safe solution, the API is still public and accessible, anyone can choose to ignore the warning.

Instance protected members

We add instance members that can only be accessed and invoked by subclasses. Such a member can be declared in a class, enum or mixin declaration. It’s available to any class which has the class or enum, or an application of the mixin, in its superclass chain. This includes the class itself — and for enums, nothing else.

Syntax

The immediate choice is to make protected a built-in identifier and use it as a modifier on any instance member declaration:

protected int foo = 42;
protected int bar(int x) => x;
protected abstract final int id;

Since protected members cannot be static, we just have to decide where it goes relative to other modifiers. Let’s say after external, before abstract, approximately the same place as static.

Let’s go with this, it can be changed if we have a better idea.

The grammar is updated to allow protected on any non-static, non-constructor member declaration. (Grammar taken from Dart.g, skipping the augment declarations for now. They may or may not need to repeat the protected.):

methodSignature
    :    constructorSignature initializers
    |    factoryConstructorSignature
    |    (STATIC | PROTECTED)? functionSignature
    |    (STATIC | PROTECTED)? getterSignature
    |    (STATIC | PROTECTED)? setterSignature
    |    PROTECTED? operatorSignature
    |    constructorSignature
    ;

declaration
    :    EXTERNAL factoryConstructorSignature
    |    EXTERNAL constantConstructorSignature
    |    EXTERNAL constructorSignature
    |    (EXTERNAL (STATIC | PROTECTED)?)? getterSignature
    |    (EXTERNAL (STATIC | PROTECTED)?)? setterSignature
    |    (EXTERNAL (STATIC | PROTECTED)?)? functionSignature
    |    EXTERNAL ((STATIC | PROTECTED)? finalVarOrType | PROTECTED? COVARIANT varOrType) identifierList
    |    EXTERNAL? PROTECTED? operatorSignature
    |    PROTECTED? ABSTRACT (finalVarOrType | COVARIANT varOrType) identifierList
    ;

PROTECTED
    :    'protected'
    ;

builtInIdentifier
    :    ABSTRACT
    …
    |    TYPEDEF
    |    PROTECTED
    ;

Semantics

We ensure that the modifier is only allowed where it makes sense. We introduce the following new rules and changes, with anything not mentioned assumed to work the same for protected members as for any other instance member.

• It’s a compile-time error if any non-instance member is declared protected. (The grammar doesn’t allow it, but an error-recovering parser will probably read it and then complain later.)
• It’s a compile-time error if an instance member is declared as protected, and the containing declaration is not a class, mixin or enum declaration. (Not allowed inside extension or extension type declarations.)
• It’s a compile-time error if a class, enum or mixin declaration declares a protected member, and any superinterface of the containing declaration has a non-protected member signature with the same name. If a member is already public, the protected declaration is misleading and probably an error.
  • Introducing a protected member through a mixin application to a class that also has a non-protected member with the same name is allowed. The member will be public, but that might be deliberate, and cannot be fixed simply by removing a misleading protected.

The behavior of a correctly declared protected member is:

• A protected member declaration works just like any other instance member declaration, except that the member signature it introduces into the interface is protected, unless any superinterface has a non-protected member with the same name. Protected interface members are special wrt. accessibility and inheritance through implements, but otherwise work as normal. An interface member can only be protected if all super-interface members of the same name are protected, otherwise it falls back to being “public”.

  • As usual, it’s a compile-time error if a non-abstract class does not implement its interface, including protected members.

• The combined super-interface of a class, mixin, enum or extension type declaration is computed almost like today, except:

  • Protected members of a super-interface introduced by an implements clause are omitted. (If the implementation is not inherited, nor is the protected interface member signature.)
  • A resulting interface member signature is protected if all the super-interface members of that name are protected.
  • This implies that a class that only implements classes with a particular protected member, does not itself have that protected member in its interface. The interface of a subtype may not have all the members of its superinterface.

  This computation affects the interface of the declaration itself, if it does not declare any members with the same name. It also affects the members available to a mixin through its on types. A mixin has a protected member if its on type does.

• A protected member is considered inaccessible when accessed through a typed member invocation for any receiver other than this. Doing Foo o = ...; o.id(); where Foo's interface has a protected member named id, is a compile-time error, because Foo does not have an accessible member named id.

• A protected member cannot be invoked through a dynamic invocation either. In a dynamic invocation, if an instance member is found which is protected in the interface of the receiver’s runtime type, dynamic invocation treats it as an inaccessible member and invokes noSuchMethod (it’s a noSuchMethod-thrower).

• Protected members can only be invoked through super or this invocations (including the implicit this. for plain identifiers that are not in the lexical scope).

  • If invoked through super, a protected member works just like any other super member invocation, directly invoking the implementation of the member. Super member invocations do not use the interface of the superclass, but directly invokes the implementation. They ignore whether a member is protected.

    • In a mixin, a super-member invocation can invoke a protected member of an on type. As usual, any super-member invocation in a mixin requires all concrete superclasses the mixin is applied to to have a valid implementation of that member.

  • If invoked through this, a protected member works just like any other instance member invocation, except that it does not consider protected members as inaccessible.

Summary

• One extra modifier, protected, on any instance member of a class, mixin or enum, which marks it as protected in the class interface.
• Can only be invoked through this or super. That’s what the flag is used for, disallowing any other invocations.
• Not inherited along implements, only through superclass relation.
• Can be made public by a non-protected declaration of the same name. Once public, always public.
• Otherwise like any other instance member.

[escamoteur]

Protected members are non-virtual, non-interface methods. They can be shadowed by other protected or non-protected members. They become virtual only when made public by being given a interface signature, without introducing new implementation.

I don't really understand this paragraph. why can they be shadowed? and why aren't they virtual?

class A {
  protected void protectedFunc(){
     print('in A');
  };
  void init(){
    protectedFunc();
  }
}

class B extends A{
  @override
  protected void protectedFunc(){
    print('in B');
  }
}

B b;
b.init(); // -> should call B.protectedFunc() and not the one of A

[lrhn]

A virtual method is late-bound, the actual implementation chosen based on the runtime type of the receiver it's called on. An interface method is basically the same in Dart. (Java has separate JVM instructions for interface invocations and virtual instance member invocations because they have separate interface and class declarations. It's all interfaces in Dart.)

Now, the question is whether this is the correct design. Your example would print in A the way this is specified, because A doesn't know that protectedFunc is overridden in subclasses. But that doesn't seem necessary, or particularly useful.
If a call could be virtual, that is: a this.name invocation would be virtual, and a super.name is still not, then subclasses can override protected members (and must if the protected members is abstract, which now also makes sense).

That's actually much more useful. (Aka: Doh! Why didn't I see that.)

I'll see when I can find time do an update :)

[lrhn]

I agree that it's a good idea if it prints B, but that was not what I had specified. As specified, it would print A.
So I fixed it. 😉 (And now nobody can see the first version any more, and laugh at its naivity. Hah!)

[ghost]

The title still says "non-virtual", but the second edition never mentions the word "virtual" again. In what sense are they non-virtual? It looks like their virtualness is the same as that of normal methods - they are just visible only to subclasses. If that's not the case, some examples could help.

[escamoteur]

virtual doesn't only mean visible into subclasses but also overridable at least for my understanding but the "non" should then be removed from the title.

[mateusfccp]

I know this proposal is not directly related to this, but if we are ever going to have protected members, we should probably want to deal with private.

As you yourself say, similar things should look similar. Having the _ prefix meaning private and an actual protected keyword will probably look too asymmetrical, and also be confusing to people non-familiar with the language (IMO, the _ prefix is already confusing).

[escamoteur]

I would love to see that. Especially as it looks totally ugly to use _properties as constructor and there is even a lint discouraging it so you need initilizers. Am 22. Mai 2024, 20:30 +0200 schrieb Mateus Felipe C. C. Pinto ***@***.***>:

…

I know this proposal is not directly related to this, but if we are ever going to have protected members, we should probably want to deal with private. As you yourself say, similar things should look similar. Having the _ prefix meaning private and an actual protected prefix will probably look too asymmetrical, and also be confusing to people non-familiar with the language (IMO, the _ prefix is already confusing). — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[ghost]

There are different scopes of privacy. "protected" is also kind-of-private, but it's private for the class and all the descendants.
The declaration int _foo in a class is private to the library, not to the class. If private x is introduced, then it must be private to the class (not visible even to descendants), so it doesn't have the same visibility as _x. Both kinds have their uses.

Does dart have a concept of package-private sub-packages (libraries)? The fact that the libraries in /src directory can be accessed from the outside via import 'package:name/src/foo.dart' breaks their assumed "privacy". We can also envision package-private variables, functions, methods, classes etc. Do you want to support all of those? If not, why not?

(should /src directory be treated as a private sub-package? Its ontological status is unclear.)

Regardless, I find _foo syntax for library-private variables very convenient. These variables are so common that special-casing their syntax is well-justified IMO. Dart won't be the same language without this convention.

[escamoteur]

@tatumizer I agree, that it would make sense to have int _foo as well as private int foo where the later one would really mean class private.
For my understanding library private means accessible to everything inside one file/part-of files, but not from out side of that. There is no concept of a package as a library as such.

[lrhn]

Fixed title to not admit first version was non-virtual.

Dart currently has no other privacy than library privacy.
Library privacy can emulate most other kinds of first-party privacy, but not protected, because protected is explicitly about hanging rights to some, but not all, third party code.

By first-party privacy I mean access protection against code not written by the same author, but letting the author themselves ignore it. With library privacy, the author gets to decide the granularity.

Protecting against the same author, who can change the code, it's not as big a priority. Having class private declarations is more about avoiding accidental name clashes between implementation details, than actually protecting against access, file/library protection is sufficient for the latter.

[mateusfccp]

Protecting against the same author, who can change the code, it's not as big a priority. Having class private declarations is more about avoiding accidental name clashes between implementation details, than actually protecting against access, file/library protection is sufficient for the latter.

I was talking more about syntactic similarities. I actually don't dislike the fact that we have library privacy. You can do anything we can do with what a private keyword would be able to do (by using different files) and more (I think what people dislike about it is that you have to put everything in the same file).

I was talking specifically about syntax. Having one as a name prefix (_) and the other as a keyword (protected) feels somewhat asymmetrical to me, considering that both do similar things (protecting code access), although in different levels.

[lrhn]

A prefix private to declare a library private declaration could work for any statically resolved declaration. It would simply not be seen from any other library. It might cause more name collisions to use non-_ names, but you could still use those if you wanted to.

The reason to use a name prefix for library privacy is instance members, and dynamic invocation in particular.

If two classes in a superclass chain both introduce a private name, then the two are considered different names.
If we used private on the declarations instead, then they should still be considered different names, but you cannot see that at the invocation.
If a superclass in the same library has a private int foo; field, and another superclass from another library adds a public void foo() method, then either

• there is a conflict, which there wouldn't be today, making you want to prefix your private instance variables with your library name, or something similarly cumbersome, or
• the subclass now cannot reference both of these, because to it, they have the same name, this.foo can only regret to one. (That's obviously fixed by renaming the library private variable, but again, you start wanting to prefix your library private variables, to avoid such churn.)

For dynamic invocation, it would need to know the library it's called from, to know which foo it should find out skip. That's doable, but also error prone.

With the leading _, one can see at the user-site that the library's private is wanted.

For protected, which is cross library, using a modifier is easier, because it prohibits most of the problem cases.
It could also use %id as name, and I'm sure it would work. There'd be no way to make a protected member public, would just have to forward to it instead.

But all the good name prefixes are taken.

[munificent]

• A protected member is considered inaccessible when accessed through a typed member invocation for any receiver other than this. Doing Foo o = ...; o.id(); where Foo's interface has a protected member named id, is a compile-time error, because Foo does not have an accessible member named id.

This means that you're proposing doing "instance protected" versus "class protected" like other OOP languages I'm familiar with do (C++, C#, Java). Is that a deliberate choice? For what it's worth, I have often found it useful to be able to call protected members on other instances of the surrounding class. We essentially have that capability in Dart with private members because of library privacy (although not up the superclass chain of course). So it feels a little strange that you could call a private method on some other instance of your own class, but not a protected method on that same instance.

(This also raises the question of whether you can have private + protected methods and whether that is a useful combination.)

[lrhn]

"instance protected" ... Is that a deliberate choice?

Yes. I have always believed that to be a better match for Dart because of two things: dynamic invocations and not having interface declarations.

Class based protected makes more sense when you can see at the call point whether you are calling the protected member or not. That works for static members and interface members. You cannot see that for a dynamic invocation.

The alternative would be to just make protected members inaccessible in dynamic invocations, which they are in this design anyway. It just feels like a missing feature then, not a logical design consequence. But possible.

Being "inside a subclass" does not give not protection you might expect if one can simply implements Superclass or an abstract ... extends Superclass and add a static member to it that accesses the protected members of Superclass that your class is extending.
Java protected members are class members, not interface members. Dart does not make a distinction. This feature tries to make a distinction by only allowing accesses that are definitely on the same class, because they are on the same object (this.foo and super.foo). Any other .foo invocation could be an interface invocation.

Another difference is that Dart cannot declare an inherited protected member as final. There is no fix for that, if you inherit a protected member, and your class is extensible, they'll inherit it too.
On the other hand, it feels like that should be the only way to access your protected members.
If someone can implement your superclass and then have acces to all the virtual protected members you inherited, it's not much of a protection.

The way this is specified, access only directly on this or super, virtual members are more like implantation members that are not part of the interface, you can access them on the object itself (like you can super of a member that has a different signature than in the interface), but not through an interface member invocation.
They're the opposite of an abstract member, which adds to the interface, but has no implementation.
(Except that to avoid having two members with the same name, they are in the interface, just marked as inaccessible. If you write, or inherit, a public declaration, then that same virtual member becomes public from there.)

That does mean no protected static members. Again, when you can implement any class, class-based protected statics it's not much of a protection.

You can have private protected members, but it's not really useful. It doesn't make much sense, since outside the library they will just be private and inaccessible. You're only protecting against yourself.