100 implicits cause OOME #9203
Comments
|
Here's an analysis why that behavior is actually quite plausible:
|
|
Does upickle have crazy implicit searches like that? |
|
Yes, it is a blocker for me. I cannot compile the test suite because of that. Apparently this line is to blame. The way the issue manifests in Upickle is as follows. Upickle generates 44 implicits, readers and writers, for all the 22 tuples, synthetically, via code generation. Next, we have a trait Api which is an entry point to all of the upickle functionality and types. It inherits from I could probably disable the offending tests, just to make it work. However, I would strongly advise against dismissing this issue. I believe there is a regression of implicits performance compared to Scala 2. The following is the heap chart for upickle compiled by Scala 2.13.1, from scratch, up to
Notice how the heap never goes above 400 MB and stays more or less the same. The compilation time is 51 second. In contrast, look at this heap chart of the same compilation run done by Dotty:
Notice how the heap grows up to the upper limit – that's at least 6 times more space consumption than Scala 2. The compilation time (until OOME) is 2 minutes 54 seconds – 3 times more than Scala 2. And that's up to OOME, not up to success. If you remove one of the issue conditions, like double inheritance or the second top-level implicit, the heap doesn't grow. GC is able to keep it in check. The following is the heap chart for the compilation run of the compilation of the issue as reported, but without double inheritance,
Notice how the heap still wants to grow large, but something, I assume GC, keeps it in check regularly. Notice also that I had to terminate this run manually after 2 minutes and a half. I believe there is some memory leak in the compiler involving typechecking of applied types. I believe under certain conditions described in the issue, it is possible to push the compilation times and heap space arbitrarily high with unreasonably small amounts of code. I believe so because it seems that the typer is producing more I believe also this issue may manifest itself in a weaker form even though all of the conditions described are not met. For example, without double inheritance, the heap still jumps up to 1 GB and the times go beyond 2 minutes on the small issue as described above. All in all, I believe there is a performance regression of implicits compared to Scala 2. This kind of issue can be very hard to debug, since the compilation is so slow and you don't get the feedback on the offending code fragment. We do not want people to start getting OOMEs and large compilation times on migration like I did. Even without OOME, some projects may be affected without even knowing it, by getting larger compilation times or larger memory consumption. I believe we should reopen this issue and address it since performance is an issue for Scala users. |
Minimized code
Output (click arrow to expand)
The text was updated successfully, but these errors were encountered: