Skip to content

Commit d8d1c9a

Browse files
heathermillerheathermiller
authored
Merge pull request #600 from szeiger/wip/collections-rework
Scala 2.13 Collections Rework blog post
2 parents ad91a96 + 595c1ee commit d8d1c9a

File tree

1 file changed

+219
-0
lines changed

1 file changed

+219
-0
lines changed

‎blog/_posts/2017-02-28-collections-rework.md

Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
---
2+
layout: blog
3+
post-type: blog
4+
by: Stefan Zeiger
5+
title: Scala 2.13 Collections Rework
6+
---
7+
8+
In October of 2015 Martin Odersky [asked for strawman proposals](https://github.com/lampepfl/dotty/issues/818) for a new collections library design for Scala 2.13, which eventually led to the project that we are currently working on, based on his latest proposal. This was not the first redesign for the Scala collections. The [current design](http://docs.scala-lang.org/overviews/core/architecture-of-scala-collections.html) was first implemented in Scala 2.8 along with the required improvements to type inference in the Scala compiler. It can generally be considered a success, providing powerful and flexible abstractions that bring together immutable and mutable collections, both sequential and parallel, with a high amount of shared interfaces and implementations. However, it does exhibit some symptoms of [second-system syndrome](https://en.wikipedia.org/wiki/Second-system_effect) that have been problematic in practice.
9+
10+
## Goals
11+
12+
Before looking at details of these problems and possible solutions in the new design, let's start with the broader goals for the new design:
13+
14+
- Simplify the API for users: This includes doing common operations without `CanBuildFrom`, pruning back the inheritance tree and a better separation of immutable and mutable collection operations.
15+
16+
- Simplify the API for implementors: Implementing a new collection type is [far from trivial](http://docs.scala-lang.org/overviews/core/architecture-of-scala-collections.html) at the moment. Setting up the implicits to get the desired `CanBuildFrom` instance in all cases (both through static implicits lookup and at run-time) is tricky. Inheriting lots of method implementations by default appears like a benefit at first but you still need to manually check (and possibly override) all methods whose default implementation has unsatisfactory performance for a specific collection.
17+
18+
- Design for better performance: The best algorithms are not very useful if you ignore lower-level performance concerns such as specialization for primitive types or method dispatch in the JVM. For example, [replacing Scala collections in Slick's AST](https://github.com/slick/slick/pull/1252/commits/2adb7c36874c41f068176570d3812b674463660e) by a custom collection implementation improved the performance of Slick's query compiler by 25% (from reducing overhead and implementing efficient operations alone, without any use of specialization). While we do not expect the same improvements for a more generic collection library, it does show that there is room for improvement.
19+
20+
- Provide source compatibility with 2.12 where it makes sense but allow breaking changes where required: We expect the majority of collection usage to be compatible between 2.12 and 2.13 and the majority of the remaining incompatibilities to be automatically fixable with [ScalaFix](https://scalacenter.github.io/scalafix/).
21+
22+
## Traversable and Iterable
23+
24+
Leaving the "generic" abstractions (which encompass both sequential and parallel collections) and the implementation traits (like `TraversableLike` and `IterableLike`) aside, we have `Traversable` at the root of the current collections hierarchy. Its only real subtype that is used by collection implementations is `Iterable`. The difference is that `Traversable` only provides *internal iteration*, i.e. a `foreach` method, whereas `Iterable` gives you the more powerful *external iteration* via an `Iterator`. The `Traversable` abstraction has not carried its weight in the current library and will likely not resurface in the new design. Everything we want to do can be expressed with `Iterable`.
25+
26+
We are also looking at other opportunities to remove collection traits. While each one of them is there for a good reason, their interactions and the sheer number create a huge amount of complexity. For example, this is the class declaration of the standard [`List`](http://www.scala-lang.org/api/2.12.1/scala/collection/immutable/List.html) class:
27+
28+
```scala
29+
sealed abstract class List[+A] extends AbstractSeq[A]
30+
with LinearSeq[A]
31+
with Product
32+
with GenericTraversableTemplate[A, List]
33+
with LinearSeqOptimized[A, List[A]]
34+
with scala.Serializable {
35+
...
36+
}
37+
```
38+
39+
If you traverse all these supertypes you find no less than 37 (!) linear supertypes in total.
40+
41+
## CanBuildFrom
42+
43+
One of the most powerful but also most controversial features of the current collections library is `CanBuildFrom`:
44+
45+
```scala
46+
def map[B, That](f: A => B)(implicit bf: CanBuildFrom[Repr, B, That]): That
47+
```
48+
49+
This is the standard `map` method as declared in `TraversableLike`. It is so inscrutable to beginners that simplified *use case signatures* were added to the API documentation in order to better convey the intended meaning:
50+
51+
```
52+
def map[B](f: A => B): $Coll[B]
53+
```
54+
55+
While use case signatures help you when you look at the scaladocs, you still have to deal with the real definitions in IDE code-completion and in compiler errors.
56+
57+
Here is the same method as defined in `IterablePolyTransforms` in the new design:
58+
59+
```scala
60+
def map[B](f: A => B): C[B]
61+
```
62+
63+
The method has the expected signature, there is no compile-time or run-time overhead to find the right `CanBuildFrom`, and the type constructor `C` is refined to a concrete collection type in almost every use case, so you see the expected definitions in scaladocs, code-completion and error messages.
64+
65+
Of course, `CanBuildFrom` was added in Scala 2.8 for a good reason. It allows a single definition of a standard method like `map` to work for regular, unconstrained collections and for constrained collections that require an implicit evidence for their element type. For example, we can naturally define `class BitSet extends Set[Int]`, but what does it mean to call `map` on a `BitSet`?
66+
67+
```scala
68+
val s: BitSet = ...
69+
val c1 = s.map(i => i+1)
70+
val c2 = s.map(i => i.toString)
71+
```
72+
73+
The current Scala collections design makes it possible to not only build a `BitSet` for `c1` and a `HashSet` for `c2` at run-time but also compute these types statically at compile-time. It gets slightly more complicated when you take code such as
74+
75+
```scala
76+
val s: BitSet = ...
77+
val c1 = (s: Set[Int]).map(i => i+1)
78+
```
79+
80+
You wouldn't expect `c1` to be of type `BitSet` but is it still backed by a `BitSet` at run-time or do you get a default `Set` implementation? In this case it is the latter but there are inconsistencies between static and dynamic lookup of `Builder` types in the current library that can lead to unexpected types or implementations.
81+
82+
As shown above, the new design does not have an implicit evidence parameter for `map`, so you are assured that you always get the same implementation no matter if you statically see your `BitSet` as a `BitSet` or a `Set[Int]`. This "same implementation" in the case of `BitSet` means `Set` though. A collection implementation is free to pick any collection type to build (between what its supertype promises and what it implements itself, of course) as long as it is *unconstrained*. In order to call `BitSet.map` and get another constrained `BitSet` out of it, we need to overload the `map` method:
83+
84+
```scala
85+
class BitSet extends Set[Int] {
86+
// inherited:
87+
//def map[B](f: Int => B): Set[B]
88+
89+
def map(f: Int => Int): BitSet
90+
}
91+
```
92+
93+
Thanks to [an improvement to type inference](https://github.com/scala/scala/pull/5307) it is possible to call the overloaded method with a lambda without explicit type annotations in Scala 2.12:
94+
95+
```scala
96+
val s: BitSet = ...
97+
98+
// Scala 2.11 and earlier would have required:
99+
val c1 = s.map((i: Int) => i+1)
100+
101+
// This works in 2.12+:
102+
val c2 = s.map(i => i+1)
103+
```
104+
105+
While `BitSet` is a rather esoteric collection type, the same principle of implicit constraints on element types `T` applies to other collection types as well:
106+
107+
- BitSet: `T <:< Int` ("must be an Int")
108+
- All `Map` types: `T <:< (_, _)` ("must be a Tuple2")
109+
- All sorted collections (like `TreeSet`): `Ordering[T]` ("must have an Ordering")
110+
- String (not a collection type per se but it gets many collection methods as extension methods): `T <:< Char` ("must be a Char")
111+
- Array (same situation as String): `ClassTag[T]` ("must have a ClassTag")
112+
113+
This covers most uses of `CanBuildFrom` in a much simpler way. The most important use cases that cannot be supported directly by the new design are `collection.breakOut` (for building a different collection type directly without an extra conversion step) and `to` (for converting to a different collection type for which a `CanBuildFrom` is available). The new design provides alternative ways to achieve the same effects.
114+
115+
## FromIterable
116+
117+
The `to` method still exists in the new design but it is only a decorator around `FromIterable`, the basic abstraction that is implemented by every unconstrained collection type's companion object:
118+
119+
```scala
120+
trait IterableOps[+A] extends Any {
121+
def to[C[X] <: Iterable[X]](fi: FromIterable[C]): C[A @uncheckedVariance] =
122+
fi.fromIterable(coll)
123+
...
124+
}
125+
126+
trait FromIterable[+C[X] <: Iterable[X]] {
127+
def fromIterable[B](it: Iterable[B]): C[B]
128+
}
129+
```
130+
131+
Note that the `fi` parameter is not implicit (like the `CanBuildFrom` in the current library), so you now call `to` with a value and have the type inferred instead of calling it with a type and having the value filled in by implicit lookup:
132+
133+
```scala
134+
val s: Set[Int] = ...
135+
val v = s.to(Vector) // this used to be to[Vector]
136+
```
137+
138+
This has the advantage that we can overload `to` to cover maps (and maybe also other constrained collection types) which is not possible in the current design (where `to` only works for unary type constructors).
139+
140+
## Views
141+
142+
[Views](http://docs.scala-lang.org/overviews/collections/views) in the current collections library [are one of the lesser-used features](https://github.com/scala/collection-strawman/issues/21#issuecomment-275861238) yet they add a lot of complexity to the implementation. The new design separates immutable views from mutable views (the latter have not yet been implemented), with only two types of immutable views: indexed and non-indexed. This is a huge simplification over the current design where a view type is specific to a its underlying collection type.
143+
144+
Conceptually a `View` is now a reified operation over an `Iterator`. Like a [Java 8 Stream](https://zeroturnaround.com/rebellabs/java-8-streams-cheat-sheet/) it has *terminal* operations which run the operation represented by the current `View` on a fresh `Iterator` (e.g. `foreach` and `foldLeft`) and *intermediate* operations which create a new `View` (e.g. `filter` and `flatMap`). This provides clear semantics even when used together with mutable collections: Composing views with intermediate operations is always independent of concurrent modifications to the underlying collection. Only when you call a terminal operation will the current state of the collection be used (by calling `iterator` on it).
145+
146+
Views are also the recommended replacement for `collection.breakOut`. For example,
147+
148+
```scala
149+
val s: Seq[Int] = ...
150+
val set: Set[String] = s.map(_.toString)(collection.breakOut)
151+
```
152+
153+
can be expressed with the same performance characteristics as:
154+
155+
```scala
156+
val s: Seq[Int] = ...
157+
val set = s.view.map(_.toString).to(Set)
158+
```
159+
160+
If you combine multiple operations they are executed lazily:
161+
162+
```scala
163+
val s: Seq[Int] = ...
164+
165+
// First build a filtered Seq, then a mapped Seq, then a Set
166+
val set1 = s.filter(_ > 0).map(_.toString).to(Set)
167+
168+
// Build a Set directly by evaluating filter and map together:
169+
val set2 = s.view.filter(_ > 0).map(_.toString).to(Set)
170+
```
171+
172+
## Laziness
173+
174+
Aside from views the current collections library also supports lazy collections but they do not get a first class treatment. While we do have `Stream`, it is only lazy in the tail but still strict in the head element and the implementation needs to special-case pretty much everything because the basic abstraction for building new collections (which is used by all of the default implementations that strict collection types can safely inherit) is `CanBuildFrom` / `Builder` which uses strict, push-based collection building.
175+
176+
By combining `FromIterable` with the new `View` design we can turn this around to provide pull-based building which can be used by strict and lazy collections alike. For example, this is the default implementation of `map` in `IterablePolyTransforms` which does not need to be overridden in `LazyList` to provide the expected laziness:
177+
178+
```scala
179+
def map[B](f: A => B): C[B] = fromIterable(View.Map(coll, f))
180+
```
181+
182+
Any operation that is available on `View` can get a default implementation like this based on rebuilding the current collection type from a `View` operation. Naturally a `LazyList` can be "built" from an `Iterable` by getting an `Iterator` and only pulling elements one by one as they are needed, so we automatically get a lazy implementation of `map`.
183+
184+
## Language Integration
185+
186+
You may have wondered about the asymmetry in the default types that are in scope through `Predef` or the `scala` package object:
187+
188+
```
189+
scala> classOf[Set[_]]
190+
res1: Class[Set[_]] = interface scala.collection.immutable.Set
191+
192+
scala> classOf[Map[_, _]]
193+
res2: Class[Map[_, _]] = interface scala.collection.immutable.Map
194+
195+
scala> classOf[Seq[_]]
196+
res3: Class[Seq[_]] = interface scala.collection.Seq
197+
```
198+
199+
Unlike all other collection types, `Seq` is not the immutable version but the generic one that encompasses both, mutable and immutable sequences. The reason for this is that the Scala specification represents varargs as type `scala.Seq` (and it should not have to rely on any type outside the top-level `scala` package). Since varargs in Java (and on the JVM) are really mutable arrays, the default `Seq` has to allow mutable collections.
200+
201+
In practice though, this kind of array can be treated as quasi-immutable, so we plan to add a new `ImmutableArray` wrapper in the new design which can be used for varargs, thereby removing the need for a non-immutable default `Seq` type.
202+
203+
## Outlook
204+
205+
We are currently working on the new design as part of [SCP-007](https://github.com/scalacenter/advisoryboard/blob/master/proposals/007-collections.md) which is funded by [Scala Center](https://scala.epfl.ch/). At this stage all work is happening in the [collection-strawman](https://github.com/scala/collection-strawman) repository with the goal of refining and completing it to the point where we are confident that it can and should become a part of Scala 2.13. The core project team consists of Julien Richard-Foy (from Scala Center), Martin Odersky, Rex Kerr and myself (as a member of the Scala team at Lightbend who maintain the Scala compiler).
206+
207+
If you'd like to get involved, now is the time to weigh in on the discussions that are happening on the pull requests and issues. We also have a [Gitter channel](https://gitter.im/scala/collection-strawman) and a [Scala Contributors](https://contributors.scala-lang.org/t/ongoing-work-on-standard-collections-redesign/293) discourse discussion thread.
208+
209+
A few topics still need further exploration:
210+
211+
- Currently there is no support for [specialization](http://www.scala-notes.org/2011/04/specializing-for-primitive-types/) of collections. It would be nice to allow this in the new design if we can do it without too much of an impact on the majority of non-specialized collections.
212+
213+
- We need a story for parallel collections. They will be [moved into a separate module](https://github.com/scala/scala/pull/5603/) in Scala 2.13 as part of the ongoing modularization of the standard library but it is not clear yet how closely they will be integrated into the new design.
214+
215+
- The [scala-java8-compat](https://github.com/scala/scala-java8-compat) module provides better integration of collections with Java Streams. Some basic parts like the specialized `Stepper` types (which unify Java Iterators, Scala Iterators and Java Spliterators) may find their way into the standard library.
216+
217+
- Apart from [ScalaFix](https://scalacenter.github.io/scalafix/) we should explore other migration options, for example using IntelliJ IDEA's refactoring API or providing compatibility libraries that allow you to cross-build most sources against the old and new collection libraries.
218+
219+
The current roadmap calls for the basic design to be completed in Q1 2017 so we can focus on migration options in Q2 and come to a decision whether to adopt the new design in Scala 2.13. If all goes according to plan, the remaining implementation work should be finished by the end of the year in time for Scala 2.13.0-RC1.

0 commit comments

Comments
 (0)