From 1d78df77a165547c4d80cafe346e15d83e2dbe0d Mon Sep 17 00:00:00 2001 From: Dimi Racordon Date: Mon, 25 Nov 2024 14:31:57 +0100 Subject: [PATCH 01/10] Start SIP on existential containers --- content/existential-containers.md | 147 ++++++++++++++++++++++++++++++ 1 file changed, 147 insertions(+) create mode 100644 content/existential-containers.md diff --git a/content/existential-containers.md b/content/existential-containers.md new file mode 100644 index 0000000..83a9bee --- /dev/null +++ b/content/existential-containers.md @@ -0,0 +1,147 @@ +--- +layout: sip +permalink: /sips/:title.html +stage: design +status: submitted +presip-thread: n/a +title: SIP-NN - Existential Containers +--- + +**By: Dimi Racordon and Eugene Flesselle and Matt Bovel** + +## History + +| Date | Version | +|---------------|--------------------| +| Nov 25th 2024 | Initial Draft | + +## Summary + +Type classes have become a well-established feature in the Scala ecosystem to escape some of the shortcomings of subtyping with respect to extensibility. +Unfortunately, type classes do not support run-time polymorphism and dynamic dispatch, two features typically taken for granted in Scala. + +This SIP proposes a feature called *existential containers* to address this problem. +An existential container wraps a value together with a witness of its conformance to one or several type classes into an object exposing the API defined by these type classes. + +## Motivation + +Type classes can address some of the well-known limitations of subtyping with respect to extensibility, such as the ability to extend existing data types with new behaviors [1]. +A type class describes the interface of a generic _concept_ as a set of requirements, expressed in the form of operations and associated types. +These requirements can be implemented for a specific type, thereby specifying how this type _models_ the concept. +The following illustrates: + +```scala +import shapes.{Square, Hexagon} + +trait TypeClass: + type Self + +trait Polygon extends TypeClass: + extension (self: Self) + def area: Double + +given Square is Polygon: ... +given Hexagon is Polygon: ... +``` + +Defining `Polygon` as a type class rather than an abstract class to be inherited allows us to retroactively state that squares are polygons without modifying the definition of `Square`. +Sticking to subtyping only would require the definition of an inneficient and verbose wrapper class. + +Alas, type classes offer limited support for type erasure–the eliding of some type information at compile-time. +Hence, it is difficult to manipulate heterogeneous collections or write procedures returning arbitrary values known to model a particular concept. +The following illustrates: + +```scala +def largest[T: Polygon](xs: Seq[T]): Option[T] = + xs.maxByOption(_.area) + +largest(List(Square(), Hexagon())) +// error: No given instance of type Polygon{type Self = Square | Hex} was found for a context parameter of method largest +``` + +The call to `largest` is illegal because, although there exist witnesses of the `Polygon` and `Hexagon`'s conformance to `Polygon`, no such witness exists for their least common supertype. +In other words, it is impossible to call `largest` with an heterogeneous sequence of polygons. + +## Proposed solution + +The problems raised above can be worked around if, instead of using generic parameters with a context bound, we use pairs bundling a value with its conformance witness. +For example, we can rewrite `largest` as follows: + +```scala +def largest(xs: Seq[(Any, PolygonWitness)]): Option[(Any, PolygonWitness)] = + xs.maxByOption((a) => a(1).area(a(0))) +``` + +A pair `(Any, PolygonWitness)` conceptually represents a type-erased polygon. +We call this pair an _existential container_ and the remainder of this SIP explains how to express this idea in a single, type-safe abstraction by leveraging Scala 3 features. + +### Specification + +As mentioned above, an existential container is merely a pair containing a value and a witness of its conformance to some concept(s). +Expressing such a value in Scala is easy: just write `(Square(1) : Any, summon[Square is Polygon] : Any)`. +This encoding, however, does not allow the selection of any method defined by `Polygon` without an unsafe cast due to the widening applied on the witness. +Fortunately, this issue can be addressed with path dependent types: + +```scala +/** A value together with an evidence of its type conforming to some type class. */ +trait Container[Concept <: TypeClass]: + /** The type of the contained value. */ + type Value : Concept as witness + /** The contained value. */ + val value: Value + +object Container: + /** Wraps a value of type `V` into a `Container[C]` provided a witness that `V is C`. */ + def apply[C <: TypeClass](v: Any)[V >: v.type](using V is C) = + new Container[C]: + type Value >: V <: V + val value: Value = v +``` + +### Compatibility + +A justification of why the proposal will preserve backward binary and TASTy compatibility. Changes are backward binary compatible if the bytecode produced by a newer compiler can link against library bytecode produced by an older compiler. Changes are backward TASTy compatible if the TASTy files produced by older compilers can be read, with equivalent semantics, by the newer compilers. + +If it doesn't do so "by construction", this section should present the ideas of how this could be fixed (through deserialization-time patches and/or alternative binary encodings). It is OK to say here that you don't know how binary and TASTy compatibility will be affected at the time of submitting the proposal. However, by the time it is accepted, those issues will need to be resolved. + +This section should also argue to what extent backward source compatibility is preserved. In particular, it should show that it doesn't alter the semantics of existing valid programs. + +### Feature Interactions + +A discussion of how the proposal interacts with other language features. Think about the following questions: + +- When envisioning the application of your proposal, what features come to mind as most likely to interact with it? +- Can you imagine scenarios where such interactions might go wrong? +- How would you solve such negative scenarios? Any limitations/checks/restrictions on syntax/semantics to prevent them from happening? Include such solutions in your proposal. + +### Other concerns + +If you think of anything else that is worth discussing about the proposal, this is where it should go. Examples include interoperability concerns, cross-platform concerns, implementation challenges. + +### Open questions + +If some design aspects are not settled yet, this section can present the open questions, with possible alternatives. By the time the proposal is accepted, all the open questions will have to be resolved. + +## Alternatives + +This section should present alternative proposals that were considered. It should evaluate the pros and cons of each alternative, and contrast them to the main proposal above. + +Having alternatives is not a strict requirement for a proposal, but having at least one with carefully exposed pros and cons gives much more weight to the proposal as a whole. + +## Related work + +This section should list prior work related to the proposal, notably: + +- A link to the Pre-SIP discussion that led to this proposal, +- Any other previous proposal (accepted or rejected) covering something similar as the current proposal, +- Whether the proposal is similar to something already existing in other languages, +- If there is already a proof-of-concept implementation, a link to it will be welcome here. + +## FAQ + +This section will probably initially be empty. As discussions on the proposal progress, it is likely that some questions will come repeatedly. They should be listed here, with appropriate answers. + +## References + +1. Stefan Wehr and Peter Thiemann. 2011. JavaGI: The Interaction of Type Classes with Interfaces and Inheritance. ACM Transactions on Programming Languages and Systems 33, 4 (2011), 12:1–12:83. https://doi.org/10.1145/1985342.1985343 +2. From 17903034518f3fb1d089f0352a95624950919fa9 Mon Sep 17 00:00:00 2001 From: Dimi Racordon Date: Mon, 2 Dec 2024 14:49:06 +0100 Subject: [PATCH 02/10] Complete SIP on existential containers --- content/existential-containers.md | 135 +++++++++++++++++++++--------- 1 file changed, 95 insertions(+), 40 deletions(-) diff --git a/content/existential-containers.md b/content/existential-containers.md index 83a9bee..9e2bcec 100644 --- a/content/existential-containers.md +++ b/content/existential-containers.md @@ -45,7 +45,7 @@ given Hexagon is Polygon: ... ``` Defining `Polygon` as a type class rather than an abstract class to be inherited allows us to retroactively state that squares are polygons without modifying the definition of `Square`. -Sticking to subtyping only would require the definition of an inneficient and verbose wrapper class. +Sticking to subtyping would require the definition of an inneficient and verbose wrapper class. Alas, type classes offer limited support for type erasure–the eliding of some type information at compile-time. Hence, it is difficult to manipulate heterogeneous collections or write procedures returning arbitrary values known to model a particular concept. @@ -65,83 +65,138 @@ In other words, it is impossible to call `largest` with an heterogeneous sequenc ## Proposed solution The problems raised above can be worked around if, instead of using generic parameters with a context bound, we use pairs bundling a value with its conformance witness. -For example, we can rewrite `largest` as follows: +In broad strokes, our solution generalizes the following possible implementation of `largest`: ```scala -def largest(xs: Seq[(Any, PolygonWitness)]): Option[(Any, PolygonWitness)] = - xs.maxByOption((a) => a(1).area(a(0))) +trait AnyPolygon: + type Value + val value: Value + val witness: Polygon { type Self = Value } + +def largest(xs: Seq[AnyPolygon]): Option[AnyPolygon] = + xs.maxByOption((a) => a.witness.area(a.value)) ``` -A pair `(Any, PolygonWitness)` conceptually represents a type-erased polygon. -We call this pair an _existential container_ and the remainder of this SIP explains how to express this idea in a single, type-safe abstraction by leveraging Scala 3 features. +The type `AnyPolygon` conceptually represents a type-erased polygon. +It consists of a pair containing some arbitrary value as well as a witness of that value's type being a polygon. +We call this pair an _existential container_, as a nod to a similar feature in Swift, and the remainder of this SIP explains how to express this idea in a single, type-safe abstraction. ### Specification -As mentioned above, an existential container is merely a pair containing a value and a witness of its conformance to some concept(s). -Expressing such a value in Scala is easy: just write `(Square(1) : Any, summon[Square is Polygon] : Any)`. -This encoding, however, does not allow the selection of any method defined by `Polygon` without an unsafe cast due to the widening applied on the witness. -Fortunately, this issue can be addressed with path dependent types: +Existential containers are encoded as follows: ```scala +import language.experimental.{clauseInterleaving, modularity} + +/** A type class. */ +trait TypeClass: + type Self + /** A value together with an evidence of its type conforming to some type class. */ -trait Container[Concept <: TypeClass]: +sealed trait Containing[Concept <: TypeClass]: /** The type of the contained value. */ - type Value : Concept as witness + type Value: Concept as witness /** The contained value. */ val value: Value -object Container: - /** Wraps a value of type `V` into a `Container[C]` provided a witness that `V is C`. */ +object Containing: + /** A `Containing[C]` whose value is known to have type `V`. */ + type Precisely[C <: TypeClass, V] = + Containing[C] { type Value >: V <: V } + /** Wraps a value of type `V` into a `Containing[C]` provided a witness that `V is C`. */ def apply[C <: TypeClass](v: Any)[V >: v.type](using V is C) = - new Container[C]: - type Value >: V <: V - val value: Value = v + new Precisely[C, V] { val value: Value = v } + /** An implicit constructor for `Containing.Precisely[C, V]` from `V`. */ + given constructor[C <: TypeClass, V : C]: Conversion[V, Precisely[C, V]] = + apply ``` -### Compatibility +Given a type class `C`, an instance `Containing[C]` is an existential container, similar to `AnyPolygon` shown before. +The context bound on the definition of the `Value` member provides a witness of `Value`'s conformance to `C` during implicit resolution when a method of the `value` field is selected. +The companion object of `Containing` provides basic support to create containers ergonomically. +For instance: -A justification of why the proposal will preserve backward binary and TASTy compatibility. Changes are backward binary compatible if the bytecode produced by a newer compiler can link against library bytecode produced by an older compiler. Changes are backward TASTy compatible if the TASTy files produced by older compilers can be read, with equivalent semantics, by the newer compilers. +```scala +def largest(xs: Seq[Containing[Polygon]]): Option[Containing[Polygon]] = + xs.maxByOption(_.value.area) +``` -If it doesn't do so "by construction", this section should present the ideas of how this could be fixed (through deserialization-time patches and/or alternative binary encodings). It is OK to say here that you don't know how binary and TASTy compatibility will be affected at the time of submitting the proposal. However, by the time it is accepted, those issues will need to be resolved. +To further improve usability, we propose to let the compiler inject the selection of the `value` field implicitly when a method of `Containing[C]` is selected. +That way, one can simply write `xs.maxByOption(_.area)` in the above example, resulting in quite idiomatic scala. -This section should also argue to what extent backward source compatibility is preserved. In particular, it should show that it doesn't alter the semantics of existing valid programs. +```scala +// Version with subtyping: +trait Polygon: + def area: Double +def largest1(xs: Seq[Polygon]): Option[Polygon] = + xs.maxByOption(_.value.area) -### Feature Interactions +// Version with existential containers: +trait Polygon extends TypeClass: + extension (self: Self) def area: Double +def largest2(xs: Seq[Containing[Polygon]]): Option[Containing[Polygon]] = + xs.maxByOption(_.area) +``` -A discussion of how the proposal interacts with other language features. Think about the following questions: +### Compatibility -- When envisioning the application of your proposal, what features come to mind as most likely to interact with it? -- Can you imagine scenarios where such interactions might go wrong? -- How would you solve such negative scenarios? Any limitations/checks/restrictions on syntax/semantics to prevent them from happening? Include such solutions in your proposal. +The change in the syntax does not affect any existing code and therefore this proposal has no impact on source compatibility. -### Other concerns +The semantics of the proposed feature is fully expressible in Scala. +Save for the implicit addition of `.value` on method selection when the receiver is an instance of `Containing[C]`, this proposal requires no change in the language. +As a result, it has no backward binary or TASTy compatibility consequences. -If you think of anything else that is worth discussing about the proposal, this is where it should go. Examples include interoperability concerns, cross-platform concerns, implementation challenges. +### Feature Interactions -### Open questions +The proposed feature is meant to interact with implicit search, as currently implemented by the language. +More specifically, given an existential container `c`, accessing `c.value` _opens_ the existential while retaining its type `c.Value`, effectively keeping an _anchor_ (i.e., the path to the scope of the witness) to the interface of the type class. -If some design aspects are not settled yet, this section can present the open questions, with possible alternatives. By the time the proposal is accepted, all the open questions will have to be resolved. +Since no change in implicit resolution is needed, this proposal cannot create unforeseen negative interactions with existing features. -## Alternatives +### Open questions -This section should present alternative proposals that were considered. It should evaluate the pros and cons of each alternative, and contrast them to the main proposal above. +One problem not addressed by the proposed encoding is the support of multiple type classes to form the interface of a specific container. +For example, one may desire to create a container of values whose types conform to both `Polygon` _and_ `Show`. +We have explored possible encodings for such a feature but decided to remove them from this proposal, as support for multiple type classes can most likely be achieved without any additional language change. -Having alternatives is not a strict requirement for a proposal, but having at least one with carefully exposed pros and cons gives much more weight to the proposal as a whole. +Another open question relates to possible language support for shortening the expression of a container type and/or value. ## Related work -This section should list prior work related to the proposal, notably: +Swift support existential containers. +For instance, `largest` can be written as follows in Swift: + +```swift +func largest(_ xs: [any Polygon]) -> (any Polygon)? { + xs.min { (a, b) in a.area < b.area } +} +``` + +Unlike in this proposal, existential containers in Swift are built-in and have a dedicated syntax (i.e., `any P`). +One advantage of Swift's design is that the type system can treat an existential container as supertype of types conforming to that container's interface. +For example, `any Polygon` is supertype of `Square` (assuming the latter conforms to `Polygon`): + +```swift +print(largest([Square(), Hexagon()])) +``` + +In contrast, to avoid possible undesirable complications, this proposal does not suggest any change to the subtyping relation of Scala. + +Rust also supports existential containers in a similar way, writing `dyn P` to denote a container bundling some value of a type conforming to `P`. +Similar to Swift, existential containers in Rust are considered supertypes of the types conforming to their bound. -- A link to the Pre-SIP discussion that led to this proposal, -- Any other previous proposal (accepted or rejected) covering something similar as the current proposal, -- Whether the proposal is similar to something already existing in other languages, -- If there is already a proof-of-concept implementation, a link to it will be welcome here. + +A more formal exploration of the state of the art as been documented in a research paper presented prior to this SIP [2]. ## FAQ -This section will probably initially be empty. As discussions on the proposal progress, it is likely that some questions will come repeatedly. They should be listed here, with appropriate answers. +#### Is there any significant performance overhead in using existential containers? + +On micro benchmarks testing method dispatch specifcally, we have measured that dispatching through existential containers in Scala was about twice as slow as traditional virtual method dispatch, which is explained by the extra pointer indirection introduced by an existential container. +This overhead drops below 10% on larger, more realistic benchmarks [2]. ## References 1. Stefan Wehr and Peter Thiemann. 2011. JavaGI: The Interaction of Type Classes with Interfaces and Inheritance. ACM Transactions on Programming Languages and Systems 33, 4 (2011), 12:1–12:83. https://doi.org/10.1145/1985342.1985343 -2. +2. Dimi Racordon and Eugene Flesselle and Matt Bovel. 2024. Existential Containers in Scala. ACM SIGPLAN International Conference on Managed Programming Languages and Runtimes, pp. 55-64. https://doi.org/10.1145/3679007.3685056 + From 3e6fc1b107a01baf1564b611d00740f45603f324 Mon Sep 17 00:00:00 2001 From: Dimi Racordon Date: Mon, 2 Dec 2024 15:35:22 +0100 Subject: [PATCH 03/10] Apply Matt's suggestions --- content/existential-containers.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/existential-containers.md b/content/existential-containers.md index 9e2bcec..9793483 100644 --- a/content/existential-containers.md +++ b/content/existential-containers.md @@ -126,15 +126,15 @@ That way, one can simply write `xs.maxByOption(_.area)` in the above example, re ```scala // Version with subtyping: -trait Polygon: +trait Polygon1: def area: Double -def largest1(xs: Seq[Polygon]): Option[Polygon] = - xs.maxByOption(_.value.area) +def largest1(xs: Seq[Polygon1]): Option[Polygon1] = + xs.maxByOption(_.area) // Version with existential containers: -trait Polygon extends TypeClass: +trait Polygon2 extends TypeClass: extension (self: Self) def area: Double -def largest2(xs: Seq[Containing[Polygon]]): Option[Containing[Polygon]] = +def largest2(xs: Seq[Containing[Polygon2]]): Option[Containing[Polygon2]] = xs.maxByOption(_.area) ``` @@ -168,7 +168,7 @@ For instance, `largest` can be written as follows in Swift: ```swift func largest(_ xs: [any Polygon]) -> (any Polygon)? { - xs.min { (a, b) in a.area < b.area } + xs.max { (a, b) in a.area < b.area } } ``` From 4642b49fd777ccbc067410dc39406c58f640143c Mon Sep 17 00:00:00 2001 From: Dimi Racordon Date: Fri, 20 Dec 2024 16:06:54 +0100 Subject: [PATCH 04/10] Submit a SIP on existential containers --- content/existential-containers.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/content/existential-containers.md b/content/existential-containers.md index 9793483..32ce66b 100644 --- a/content/existential-containers.md +++ b/content/existential-containers.md @@ -69,9 +69,8 @@ In broad strokes, our solution generalizes the following possible implementation ```scala trait AnyPolygon: - type Value + type Value: Polygon as witness val value: Value - val witness: Polygon { type Self = Value } def largest(xs: Seq[AnyPolygon]): Option[AnyPolygon] = xs.maxByOption((a) => a.witness.area(a.value)) @@ -153,6 +152,11 @@ More specifically, given an existential container `c`, accessing `c.value` _open Since no change in implicit resolution is needed, this proposal cannot create unforeseen negative interactions with existing features. +### Other concerns + +This document has been written under the experimental modularity improvements for Scala 3. +Although the proposed feature is fully expressible without those changes, the encoding of existential containers can only work with the "old" (i.e., the one currently used in production) or "new" type class style. + ### Open questions One problem not addressed by the proposed encoding is the support of multiple type classes to form the interface of a specific container. From 03c23a598489f852615518cc73cd9c21ac061a21 Mon Sep 17 00:00:00 2001 From: Dimi Racordon Date: Fri, 24 Jan 2025 10:33:52 +0100 Subject: [PATCH 05/10] Update content/existential-containers.md Co-authored-by: Seth Tisue --- content/existential-containers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/existential-containers.md b/content/existential-containers.md index 32ce66b..32aca6c 100644 --- a/content/existential-containers.md +++ b/content/existential-containers.md @@ -45,7 +45,7 @@ given Hexagon is Polygon: ... ``` Defining `Polygon` as a type class rather than an abstract class to be inherited allows us to retroactively state that squares are polygons without modifying the definition of `Square`. -Sticking to subtyping would require the definition of an inneficient and verbose wrapper class. +Sticking to subtyping would require the definition of an inefficient and verbose wrapper class. Alas, type classes offer limited support for type erasure–the eliding of some type information at compile-time. Hence, it is difficult to manipulate heterogeneous collections or write procedures returning arbitrary values known to model a particular concept. From 8b0807c90601c93250d43ce0ee2a17e3cd239a5f Mon Sep 17 00:00:00 2001 From: Dimi Racordon Date: Fri, 24 Jan 2025 10:38:09 +0100 Subject: [PATCH 06/10] Update content/existential-containers.md Co-authored-by: Seth Tisue --- content/existential-containers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/existential-containers.md b/content/existential-containers.md index 32aca6c..02ebd44 100644 --- a/content/existential-containers.md +++ b/content/existential-containers.md @@ -64,7 +64,7 @@ In other words, it is impossible to call `largest` with an heterogeneous sequenc ## Proposed solution -The problems raised above can be worked around if, instead of using generic parameters with a context bound, we use pairs bundling a value with its conformance witness. +The problems raised above can be worked around if, instead of using generic parameters with a context bound, we use pairs bundling each value with its conformance witness. In broad strokes, our solution generalizes the following possible implementation of `largest`: ```scala From 5ac6ae450c204da50ccd934198a8555747459cf3 Mon Sep 17 00:00:00 2001 From: Dimi Racordon Date: Fri, 24 Jan 2025 10:38:55 +0100 Subject: [PATCH 07/10] Update content/existential-containers.md Co-authored-by: Seth Tisue --- content/existential-containers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/existential-containers.md b/content/existential-containers.md index 02ebd44..f2bf1b5 100644 --- a/content/existential-containers.md +++ b/content/existential-containers.md @@ -145,7 +145,7 @@ The semantics of the proposed feature is fully expressible in Scala. Save for the implicit addition of `.value` on method selection when the receiver is an instance of `Containing[C]`, this proposal requires no change in the language. As a result, it has no backward binary or TASTy compatibility consequences. -### Feature Interactions +### Feature interactions The proposed feature is meant to interact with implicit search, as currently implemented by the language. More specifically, given an existential container `c`, accessing `c.value` _opens_ the existential while retaining its type `c.Value`, effectively keeping an _anchor_ (i.e., the path to the scope of the witness) to the interface of the type class. From ef4829ddabe02c436202658500c9834b69688940 Mon Sep 17 00:00:00 2001 From: Dimi Racordon Date: Fri, 24 Jan 2025 10:42:41 +0100 Subject: [PATCH 08/10] Update content/existential-containers.md Co-authored-by: Seth Tisue --- content/existential-containers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/existential-containers.md b/content/existential-containers.md index f2bf1b5..0d08c47 100644 --- a/content/existential-containers.md +++ b/content/existential-containers.md @@ -167,7 +167,7 @@ Another open question relates to possible language support for shortening the ex ## Related work -Swift support existential containers. +Swift supports existential containers. For instance, `largest` can be written as follows in Swift: ```swift From 45f77d9ef958e23830083b2e4ad0eb8587e1967d Mon Sep 17 00:00:00 2001 From: Dimi Racordon Date: Fri, 24 Jan 2025 11:00:46 +0100 Subject: [PATCH 09/10] Avoid mentions of type erasure --- content/existential-containers.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/existential-containers.md b/content/existential-containers.md index 0d08c47..6f39ea1 100644 --- a/content/existential-containers.md +++ b/content/existential-containers.md @@ -47,7 +47,7 @@ given Hexagon is Polygon: ... Defining `Polygon` as a type class rather than an abstract class to be inherited allows us to retroactively state that squares are polygons without modifying the definition of `Square`. Sticking to subtyping would require the definition of an inefficient and verbose wrapper class. -Alas, type classes offer limited support for type erasure–the eliding of some type information at compile-time. +Alas, type classes offer limited support for eliding type information at compile-time. Hence, it is difficult to manipulate heterogeneous collections or write procedures returning arbitrary values known to model a particular concept. The following illustrates: @@ -76,7 +76,7 @@ def largest(xs: Seq[AnyPolygon]): Option[AnyPolygon] = xs.maxByOption((a) => a.witness.area(a.value)) ``` -The type `AnyPolygon` conceptually represents a type-erased polygon. +The type `AnyPolygon` conceptually represents an arbitrary polygon. It consists of a pair containing some arbitrary value as well as a witness of that value's type being a polygon. We call this pair an _existential container_, as a nod to a similar feature in Swift, and the remainder of this SIP explains how to express this idea in a single, type-safe abstraction. From dacee46eac9de369766a3ed0f52c97af3b4f1e55 Mon Sep 17 00:00:00 2001 From: Dimi Racordon Date: Fri, 24 Jan 2025 16:07:44 +0100 Subject: [PATCH 10/10] Apply @lihaoyi's suggestions --- content/existential-containers.md | 150 +++++++++++++++++++++++------- 1 file changed, 116 insertions(+), 34 deletions(-) diff --git a/content/existential-containers.md b/content/existential-containers.md index 6f39ea1..b5f5fc8 100644 --- a/content/existential-containers.md +++ b/content/existential-containers.md @@ -20,7 +20,7 @@ title: SIP-NN - Existential Containers Type classes have become a well-established feature in the Scala ecosystem to escape some of the shortcomings of subtyping with respect to extensibility. Unfortunately, type classes do not support run-time polymorphism and dynamic dispatch, two features typically taken for granted in Scala. -This SIP proposes a feature called *existential containers* to address this problem. +This SIP proposes a minimal change to the language to support *existential containers*, which address this problem. An existential container wraps a value together with a witness of its conformance to one or several type classes into an object exposing the API defined by these type classes. ## Motivation @@ -65,7 +65,7 @@ In other words, it is impossible to call `largest` with an heterogeneous sequenc ## Proposed solution The problems raised above can be worked around if, instead of using generic parameters with a context bound, we use pairs bundling each value with its conformance witness. -In broad strokes, our solution generalizes the following possible implementation of `largest`: +In broad strokes, a solution generalizes the following possible implementation of `largest`: ```scala trait AnyPolygon: @@ -78,50 +78,59 @@ def largest(xs: Seq[AnyPolygon]): Option[AnyPolygon] = The type `AnyPolygon` conceptually represents an arbitrary polygon. It consists of a pair containing some arbitrary value as well as a witness of that value's type being a polygon. -We call this pair an _existential container_, as a nod to a similar feature in Swift, and the remainder of this SIP explains how to express this idea in a single, type-safe abstraction. +We call this pair an _existential container_, as a nod to a similar feature in Swift. -### Specification +While the above example "hardcodes" the type class `Polygon`, Scala's type system is actually rich enough to safely define a generic abstraction represeting existential containers parameterized by a type class. +A possible implementation is presented in the appendix but its details are actually irrelevant to for the proposed change. +The purpose of this SIP is _only_ to support the selection of an existential container's `value` field implicitly. +That way, one could simply write `xs.maxByOption(_.area)` in the above example, resulting in quite idiomatic scala. -Existential containers are encoded as follows: +To illustrate further, assume the existence of an abstraction named `Containing[TC]` for representing containers pairing an arbitrary value with a witness of its conformance to some type class `TC`. +The proposed change would let the compiler accept the following example: ```scala -import language.experimental.{clauseInterleaving, modularity} +trait Polygon extends TypeClass: + extension (self: Self) def area: Double -/** A type class. */ -trait TypeClass: - type Self +def largest(xs: Seq[Containing[Polygon]]): Option[Containing[Polygon]] = + xs.maxByOption(_.area) +``` -/** A value together with an evidence of its type conforming to some type class. */ -sealed trait Containing[Concept <: TypeClass]: - /** The type of the contained value. */ - type Value: Concept as witness - /** The contained value. */ - val value: Value +This implementation of `largest` requires existential containers to take and return arbitrary polygons. +Indeed, we wish to operate on a _heterogeneous_ list of polygons (i.e., types conforming to `Polygon`), not a list of a particular type happing to have an instance of the type class. +On the return side, we wish to return any type known to be a polygon paired with the witness of its conformance. +Again, doing so (conveniently) is not possible without existential containers. -object Containing: - /** A `Containing[C]` whose value is known to have type `V`. */ - type Precisely[C <: TypeClass, V] = - Containing[C] { type Value >: V <: V } - /** Wraps a value of type `V` into a `Containing[C]` provided a witness that `V is C`. */ - def apply[C <: TypeClass](v: Any)[V >: v.type](using V is C) = - new Precisely[C, V] { val value: Value = v } - /** An implicit constructor for `Containing.Precisely[C, V]` from `V`. */ - given constructor[C <: TypeClass, V : C]: Conversion[V, Precisely[C, V]] = - apply +The above example generalizes to any occurrence of heterogeneous collection. +For instance: + +```scala +trait CustomHashable extends TypeClass: + extension (self: Self) def hashInto(hasher: Hasher) + +def customHashValue(xs: List[Containing[CustomHashable]]): Int = + val h = Hasher() + for x <- xs do xs.hashInto(h) + h.finalize() ``` -Given a type class `C`, an instance `Containing[C]` is an existential container, similar to `AnyPolygon` shown before. -The context bound on the definition of the `Value` member provides a witness of `Value`'s conformance to `C` during implicit resolution when a method of the `value` field is selected. -The companion object of `Containing` provides basic support to create containers ergonomically. -For instance: +Returning a value paired with its witness generalizes similarly. ```scala -def largest(xs: Seq[Containing[Polygon]]): Option[Containing[Polygon]] = - xs.maxByOption(_.value.area) +trait Sizeable extends TypeClass: + extension (self: Self) def size: Int + +def shortest[A: Sizeable, B: Sizeable](a: A, b: B): Containing[Sizeable] = + if b.size < a.size then Containing(a) else Containing(b) ``` -To further improve usability, we propose to let the compiler inject the selection of the `value` field implicitly when a method of `Containing[C]` is selected. -That way, one can simply write `xs.maxByOption(_.area)` in the above example, resulting in quite idiomatic scala. +Further motivation for existential containers in Scala have been described in a research paper [2]. + +### Specification + +Assuming the existence of an abstraction named `Containing[TC]` for representing containers pairing an arbitrary value with a witness of its conformance to some type class `TC` in the standard library, the compiler injects the selection of the `value` field implicitly when a method of `Containing[TC]` is selected. + +Illustrating with our running example: ```scala // Version with subtyping: @@ -134,7 +143,7 @@ def largest1(xs: Seq[Polygon1]): Option[Polygon1] = trait Polygon2 extends TypeClass: extension (self: Self) def area: Double def largest2(xs: Seq[Containing[Polygon2]]): Option[Containing[Polygon2]] = - xs.maxByOption(_.area) + xs.maxByOption(_.area) // <- sugared form of `xs.maxByOption(_.value.area)` ``` ### Compatibility @@ -189,9 +198,49 @@ In contrast, to avoid possible undesirable complications, this proposal does not Rust also supports existential containers in a similar way, writing `dyn P` to denote a container bundling some value of a type conforming to `P`. Similar to Swift, existential containers in Rust are considered supertypes of the types conforming to their bound. +Existential contains are also featured in Haskell, under the [`ExistentialQuantification`](https://wiki.haskell.org/Heterogenous_collections) extension. +Unlike in Swift and Rust, packing and unpacking in and out of existential containers requires more boilerplate: + +```haskell +{-# LANGUAGE ExistentialQuantification #-} + +class Polygon a where + area :: a -> Double + +data AnyPolygon = forall a . Polygon a => MakePolygon a +pack :: Polygon a => a -> AnyPolygon +pack = MakePolygon + +instance Polygon Square where + area s = 1.0 +instance Polygon Hexagon where + area h = 1.0 + +largest :: [AnyPolygon] -> Maybe AnyPolygon +largest (x : xs) = case (largest xs) of + Nothing -> Just x + Just(y) -> Just (if (f x) < (f y) then y else x) + where f (MakePolygon a) = area a +largest [] = Nothing +``` A more formal exploration of the state of the art as been documented in a research paper presented prior to this SIP [2]. +## Alternatives considered + +### No change to the language + +As already mentioned, Scala's type system is currently strong enough to support the definition of existential containers. +Hence, no change to the language is strictly necessary to support them. +The gain offered by the proposal is that method selection on an existential container will look more familiar. + +### Converting to and from wrappers + +As mentioned in our motivations, the problem of creating a heterogeneous collections can be addressed by defining custom wrappers. +One can further define implicit conversions to alleviate the syntactic burden. +This approach is nonetheless strictly more verbose since it requires the definition of a specific container for each type class used in conjunction with heterogeneous collections. +In that sense, existential containers can be understood as a generalized wrapper. + ## FAQ #### Is there any significant performance overhead in using existential containers? @@ -204,3 +253,36 @@ This overhead drops below 10% on larger, more realistic benchmarks [2]. 1. Stefan Wehr and Peter Thiemann. 2011. JavaGI: The Interaction of Type Classes with Interfaces and Inheritance. ACM Transactions on Programming Languages and Systems 33, 4 (2011), 12:1–12:83. https://doi.org/10.1145/1985342.1985343 2. Dimi Racordon and Eugene Flesselle and Matt Bovel. 2024. Existential Containers in Scala. ACM SIGPLAN International Conference on Managed Programming Languages and Runtimes, pp. 55-64. https://doi.org/10.1145/3679007.3685056 + +## Appendix + +The following is a possible implementation of existential containers. + +```scala +import language.experimental.{clauseInterleaving, modularity} + +/** A type class. */ +trait TypeClass: + type Self + +/** A value together with an evidence of its type conforming to some type class. */ +sealed trait Containing[Concept <: TypeClass]: + + /** The type of the contained value. */ + type Value: Concept as witness + + /** The contained value. */ + val value: Value + +object Containing: + + /** Wraps a value of type `V` into a `Containing[TC]` provided a witness that `V is TC`. */ + def apply[TC <: TypeClass](v: Any)[V >: v.type](using V is TC) = + new Containing[TC]: + type Value >: V <: V + val value: Value = v +``` + +Given a type class `C`, an instance `Containing[TC]` is an existential container. +The context bound on the definition of the `Value` member provides a witness of `Value`'s conformance to `TC` during implicit resolution when a method of the `value` field is selected. +The companion object of `Containing` provides basic support to create containers ergonomically.