Add typeclass derivation #5540

odersky · 2018-11-29T12:57:03Z

An implementation of typeclass derivation. Direct link to docs:

https://github.com/dotty-staging/dotty/blob/add-derive-2/docs/docs/reference/derivation.md

Based on #5518.

library/src-scala3/scala/typelevel/Mirror.scala

library/src-scala3/scala/typelevel/Shape.scala

tests/run/typeclass-derivation3.scala

julienrf · 2018-12-03T20:57:14Z

tests/run/typeclass-derivation3.scala

+      inline erasedValue[Elems] match {
+        case _: (elem *: elems1) =>
+          tryPickle[elem](buf, elems(n).asInstanceOf[elem])
+          pickleElems[elems1](buf, elems, n + 1)


I find it a bit fragile that we have to manually manage the n counter. Would it be possible to get the corresponding field mirror as a parameter? Or maybe to extract it?

case _: (mirrorred(elem, label) *: elems1) =>

That means we'd have to handle extractors in the inlining framework, which is not yet possible.

OK, so maybe forget about using an extractor. Still, I think it would be better to provide the typeclass implementers a correctly typed elem value (to avoid the asInstanceOf) along with its label, without requiring them to manage an elements counter.

Yes, I can see that as part of a framework, which could also be built on top of what we have here.

julienrf · 2018-12-03T20:59:32Z

tests/run/typeclass-derivation3.scala

+          if (n == ordinal) unpickleCase[T, elems](r, buf, ordinal)
+          else unpickleCases[T, alts1](r, buf, ordinal, n + 1)
+        case _ =>
+          throw new IndexOutOfBoundsException(s"unexpected ordinal number: $ordinal")


It’s unclear to me at what time is this exception raised. Is it at compile-time or at run-time?

Also, how do we report at compile-time to the user that a typeclass instance for the shape of its type can not be generically derived?

It's at runtime. There's a typelevel.error method that gives a compile-time error message.

odersky · 2018-12-09T16:09:41Z

@OlivierBlanvillain @milessabin @propensive This is more or less baked now. Feedback appreciated!

It's probably best to start with the docs https://github.com/dotty-staging/dotty/blob/add-derive-2/docs/docs/reference/derivation.md.

julienrf · 2018-12-11T13:52:58Z

docs/docs/reference/derivation.md

+Besides for `enums`, typeclasses can also be derived for other sets of classes and objects that form an algebraic data type. These are:
+
+ - individual case classes or case objects
+ - sealed classes or traits that have only case classes and case objects as children.


Can they also have sealed classes as children?

if they are not case classes, no.

julienrf · 2018-12-11T13:57:02Z

docs/docs/reference/derivation.md

+
+A trait or class can appear in a `derives` clause if
+
+ - it has a single type parameter, and


Is there any restriction on the kind of that type parameter? For instance, is it possible to write derives Functor (assuming a trait Functor[F[_]] definition)?

There is no such restriction. But there's a caveat: The Shape type is always first-kinded. If you want to derive a higher-kinded type class instance from that you have to do some lifting of the Shape type to the type constructor level. That's possible using the type match machinery. @OlivierBlanvillain has done a PoC.

julienrf · 2018-12-11T14:09:55Z

docs/docs/reference/derivation.md

+If the ADT in question does not have a `derives` clause, an implicit `Generic` instance
+would still be synthesized by the compiler at the point where `derived` is called.
+This is similar to the situation with type tags or class tags: If no implicit instance
+is found, the compiler will synthesize one.


Could the synthesis of Generic be kept as an implementation detail? Do we need to be aware of where it comes from, if anyway we can always get a Generic[A] when we ask one?

Yes, we could keep it as a detail. It's just that the synthesis does generate a significant amount of code, so it's good to know that this will be done only once per class with a derives clause instead of once per usage.

OlivierBlanvillain · 2018-12-14T11:01:05Z

What would we lose if we stick to the traditional sum-of-product representation instead of Mirror/Case/Cases/GenericClass? I feel that the added complexity needs very good motivation. The documentation claims faster compilation times and better runtime performance, it would be interesting to benchmarks both points.

The generated code for Eq[Tree[T]] is still far away from a handwritten version that doesn't allocate. It's impressive that we can get that far with inlining alone, but I feel that staging would be a more appropriate tool to get to zero allocation.

Here is a proof of concept implementation for typeclass derivation for Eq that uses staging and a Shapeless style sum-of-product representation. The generated code doesn't allocate any intermediate representation: the HList is only materialized at compile time and completely folded away by the staging framework to generate direct field access. The code is also type-safe (no casts), but I didn't measure compilation times.

If staging can be used reliably to do typeclass derivation I feel that Generic interface proposed in this PR sits in an awkward spot: it's both more complicated than shapeless.Generic and generates suboptimal code...

odersky · 2018-12-14T12:10:18Z

What would we lose if we stick to the traditional sum-of-product representation instead of Mirror/Case/Cases/GenericClass? I feel that the added complexity needs very good motivation.

I believe a Sum-Of-Products representation is more complex than what we have! In particularly, representing sums and products as binary constructs obscures things a lot. I feel it's more natural to use tuples and indexing. But I am happy to be proven wrong on this.

We now have test cases (typeclass-derivation2a.scala/typeclass-derivation3.scala) that could be adapted to a different scheme and we could compare.

Here are the criteria by which I would like to measure solutions, in decreasing order of importance:

size of generated code
runtime efficiency of generated instances (both baseline and ability to be optimized)
compile times
ability to support typeclasses of all kinds
difficulty of developing generic typeclasses

`shapeWithClassType` does not return a NoType anymore.

- harden addChild to also work if existing children don't have positions (this can happen if they came from other compilation units) - tweak position of Deriver, so that it always points to the deriving class template.

Since Shape is no longer generated separately we do not need to make an exception for synthesized members.

Children now appear in the order of their definition, where before there was no guaranteed order and the actual order was more likely the reverse of definition order.

Better not to mix different proposals.

The erasure of these classes should be Product not Object. Otherwise we cannot discriminate in pattern matching between pairs and ().

Cases in a match could be empty in case of errors

odersky added the stat:wip label Nov 29, 2018

odersky force-pushed the add-derive-2 branch 5 times, most recently from 9d8f1f3 to 0bf30ce Compare December 3, 2018 13:22

odersky removed the stat:wip label Dec 3, 2018

julienrf reviewed Dec 3, 2018

View reviewed changes

library/src-scala3/scala/typelevel/Mirror.scala Outdated Show resolved Hide resolved

julienrf reviewed Dec 3, 2018

View reviewed changes

library/src-scala3/scala/typelevel/Shape.scala Outdated Show resolved Hide resolved

odersky force-pushed the add-derive-2 branch from 9bceb06 to 7b0698e Compare December 3, 2018 20:53

julienrf reviewed Dec 3, 2018

View reviewed changes

odersky force-pushed the add-derive-2 branch 11 times, most recently from 790779a to 48dcb89 Compare December 9, 2018 14:53

odersky self-assigned this Dec 10, 2018

odersky mentioned this pull request Dec 10, 2018

Allow , in parents of template #5518

Closed

odersky force-pushed the add-derive-2 branch from 1ec1d4d to 57b75d4 Compare December 10, 2018 18:08

julienrf reviewed Dec 11, 2018

View reviewed changes

odersky force-pushed the add-derive-2 branch from ff81f2a to f662c81 Compare December 13, 2018 20:09

odersky and others added 26 commits January 19, 2019 17:38

Improve robustness in the face of errors

5eca939

Drop redundant check

12609e4

`shapeWithClassType` does not return a NoType anymore.

Fixes to positions

4d52930

- harden addChild to also work if existing children don't have positions (this can happen if they came from other compilation units) - tweak position of Deriver, so that it always points to the deriving class template.

Revert changes to CheckRealizable

f3d901e

Since Shape is no longer generated separately we do not need to make an exception for synthesized members.

Fix patmat exhaustivity check files

465ffcc

Children now appear in the order of their definition, where before there was no guaranteed order and the actual order was more likely the reverse of definition order.

Polishings

d4025d6

Tweaks to docs

1a9c5c0

Fix typo

1a58222

Revert to standard implicit syntax in docs

7fccb61

Better not to mix different proposals.

Polishings

46e4b19

Fix erasure of *: and NonEmptyTuple

5182b8b

The erasure of these classes should be Product not Object. Otherwise we cannot discriminate in pattern matching between pairs and ().

Make NonEmptyTuple sealed.

efb3e5f

More tests

4e594ed

Fix typos

8e98772

Add derived field to Tasty reflect

683b93b

Drop failing assertion

569f918

Cases in a match could be empty in case of errors

Add missing comma and update checkfiles

e3f74e3

Update another check file

d9be6e2

Fix rebase breakage

caece0e

Fix test

9d87241

Another check file fix

248971b

Fix typos in documentation

49901fe

Fix spacing and add return types

4cd81ba

Fix failing fuzzy test

296958e

Fix rebase breakage

bbb54cb

Fix indent

5e4fa24

odersky force-pushed the add-derive-2 branch from 13a93c0 to 5e4fa24 Compare January 19, 2019 16:40

odersky merged commit 2d16d9f into scala:master Jan 19, 2019

allanrenucci deleted the add-derive-2 branch January 19, 2019 19:05

OlivierBlanvillain mentioned this pull request Jan 28, 2019

Synthesize Representable type class #3663

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add typeclass derivation #5540

Add typeclass derivation #5540

odersky commented Nov 29, 2018 •

edited

Loading

julienrf Dec 3, 2018 •

edited

Loading

odersky Dec 4, 2018

julienrf Dec 4, 2018 •

edited

Loading

odersky Dec 4, 2018

julienrf Dec 3, 2018

odersky Dec 4, 2018

odersky commented Dec 9, 2018

julienrf Dec 11, 2018

odersky Dec 12, 2018

julienrf Dec 11, 2018

odersky Dec 12, 2018

julienrf Dec 11, 2018 •

edited

Loading

odersky Dec 12, 2018

OlivierBlanvillain commented Dec 14, 2018 •

edited

Loading

odersky commented Dec 14, 2018 •

edited

Loading


		A trait or class can appear in a `derives` clause if

		- it has a single type parameter, and

Add typeclass derivation #5540

Add typeclass derivation #5540

Conversation

odersky commented Nov 29, 2018 • edited Loading

julienrf Dec 3, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

julienrf Dec 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

odersky commented Dec 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

julienrf Dec 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OlivierBlanvillain commented Dec 14, 2018 • edited Loading

odersky commented Dec 14, 2018 • edited Loading

odersky commented Nov 29, 2018 •

edited

Loading

julienrf Dec 3, 2018 •

edited

Loading

julienrf Dec 4, 2018 •

edited

Loading

julienrf Dec 11, 2018 •

edited

Loading

OlivierBlanvillain commented Dec 14, 2018 •

edited

Loading

odersky commented Dec 14, 2018 •

edited

Loading