scala compile fails on simple but lengthy repetitive script (ClassNotFound and/or StackOverflowError)

[scabug]

attached script fails, failure mode depends on scala version:

scala 2.9.2: ClassNotFoundException: Main(...) at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:103)

scala 2.10.0-M7: (notes: delete line #2 of attached script for this version; on linux, hangs, or possibly fails after uncharacteristically long delay):

error: 
     while compiling: /Users/philwalk/Workspaces/juno-workspace/cse.spooler.gui/./listItems.sc
        during phase: typer
     library version: version 2.10.0-M7
    compiler version: version 2.10.0-M7
  reconstructed args: -Xscript Main -d /var/folders/vy/l3zbb71j15lbnmysddxfspz00000gn/T/scalascript6524079972914240372.tmp

  last tree to typer: Select(This(anonymous class $anon), Item)
              symbol: object Item (flags: <module> <synthetic> <triedcooking> private)
   symbol definition: private object Item
                 tpe: this.Item.type
       symbol owners: object Item -> anonymous class $anon -> method main -> object Main -> package <empty>
      context owners: method readResolve -> object Item -> anonymous class $anon -> method main -> object Main -> package <empty>

== Enclosing template or block ==

DefDef( // private def readResolve(): Object in object Item
  <method> private <synthetic>
  "readResolve"
  []
  List(Nil)
  <tpt> // tree.tpe=Object
  $anon.this."Item" // private object Item, tree.tpe=this.Item.type
)

== Expanded type of tree ==

TypeRef(
  pre = ThisType(anonymous class $anon)
  TypeSymbol(
    class Item extends AbstractFunction3[String,String,String,this.Item] with Serializable
    
  )
)

uncaught exception during compilation: java.lang.StackOverflowError

[scabug]

Imported From: https://issues.scala-lang.org/browse/SI-6543?orig=1
Reporter: Phil Walker (philwalk9)
Affected Versions: 2.9.2, 2.10.0, 2.11.0-M1
Attachments:

• bigString.sc (created on Oct 19, 2012 5:15:01 PM UTC, 550594 bytes)
• ChiSquareAlpha.scala (created on May 12, 2013 11:51:42 AM UTC, 1345567 bytes)
• listItems.sc (created on Oct 19, 2012 3:01:54 PM UTC, 713470 bytes)
• showScript.sc (created on Oct 19, 2012 4:18:26 PM UTC, 1323 bytes)
• showScript.sc (created on Oct 19, 2012 4:17:22 PM UTC, 1323 bytes)

[scabug]

@paulp said:
Since this is due to #3295 (the stack overflow is in List#foldRight) perhaps martin will reconsider the wontfix status of that ticket.

[scabug]

@paulp said:
LinearSeqOptimized#foldRight, if that was ambiguous.

[scabug]

@paulp said:
Some points of interest if only to my future self:

1. -XX:MaxJavaStackTraceDepth=10000
2. This code gets 8462 deep at LinearSeqOptimized.scala:119.
3. The origin is Typer#addSynthetics, which is not only ugly as already observed in the comment but appears to be obscenely expensive. It looks like for a template with N statements, it creates at least 3N ListBuffers and performs at least 2N ::: operations on Lists. And calls foldRight.

[scabug]

Phil Walker (philwalk9) said (edited on Oct 19, 2012 4:35:58 PM UTC):
NOTE: this attachment relates to the subsequent comment, not to the prior discussion.
Attachment: example of a scala script that displays its' filesystem path.

[scabug]

Phil Walker (philwalk9) said (edited on Oct 19, 2012 4:31:11 PM UTC):
After reading the discussion on #3295, I'd like to add a bit of context re: what itch this script scratches.

I have been using scala (quite successfully, thank you) as my primary scripting language for the last 2 years.
There are only a couple of remaining significant limitations that force me to solve a few problems without using scala:

1. scala needs the ability to carry its' own data (the role of "here-documents" in most scripting languages).

2. although not related to this ticket, it's still quite difficult for a stand-alone scala script to know where it resides in the filesystem

Both of these features are necessary for a scala script (and/or equivalent executable jar file) to be distributed as a single file, placed anywhere in the filesystem, and executed without a wrapper (e.g., a bash script).

The failing script referred to here was generated via the command line 'find tmp -type f | xargs cksum > listItems.sc', after which I intended to quickly convert the output to a script that permits duplicate files (those with equal cksum in column 1) to be reported. Due to the length of the list and the foldRight limitation, I haven't figured out how I might resolve this (without resorting to ruby).

Feature #2 is a trivial operation in mainstream scripting languages, and although it is possible in scala, the solution is quite ugly: see 'showScript.sc', attached).

[scabug]

@paulp said:
FYI it is easy to modify that bit of Typers not to use foldRight, which will make this issue go away. But I'd like to use the fact that the compiler itself fell prey to the foldRight behavior to also fix #3295. My point though is that this will be fixed regardless.

[scabug]

Phil Walker (philwalk9) said:
I took a different approach to creating my large list, and encountered a different bug (I can file a separate issue, but am still searching to see if this is a known issue).
This time, I tried the following approach (see the subsequent comment attachment for the actual script):

val list = """
493270, 1241, tmp/scala/Function1$mcJD$sp$$anonfun$compose$mcJD$sp$1.class)
604021, 8602, tmp/scala/collection/parallel/ParIterableLike$Partition.class)
1116524, 1245, tmp/scala/collection/immutable/NumericRange$$anon$1$$anonfun$foreach$1.class)
...
4292142009, 859, tmp/scala/Function2$mcJJI$sp.class)
4292278090, 1167, tmp/org/joda/time/tz/CachedDateTimeZone$Info.class)
4294011439, 1427, tmp/scala/collection/immutable/PagedSeq$$anonfun$fromStrings$1.class)

""".trim.split("""[\r\n]+""".toList

I wasn't surprised that it didn't succeed (due to similar failed past attempts), but the error I expected was a "maximum String length exceeded", or similar.
The actual error was:

java.lang.ClassFormatError: Unknown constant tag 47 in class file Main$$anon$1
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at Main$.main(listItems.sc:1)
at Main.main(listItems.sc)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:71)
at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:139)
at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:71)
at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:139)
at scala.tools.nsc.CommonRunner$class.run(ObjectRunner.scala:28)
at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:45)
at scala.tools.nsc.CommonRunner$class.runAndCatch(ObjectRunner.scala:35)
at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:45)
at scala.tools.nsc.ScriptRunner.scala$tools$nsc$ScriptRunner$$runCompiled(ScriptRunner.scala:171)
at scala.tools.nsc.ScriptRunner$$anonfun$runScript$1.apply(ScriptRunner.scala:188)
at scala.tools.nsc.ScriptRunner$$anonfun$runScript$1.apply(ScriptRunner.scala:188)
at scala.tools.nsc.ScriptRunner$$anonfun$withCompiledScript$1$$anonfun$apply$mcZ$sp$1.apply(ScriptRunner.scala:157)
at scala.tools.nsc.ScriptRunner$$anonfun$withCompiledScript$1$$anonfun$apply$mcZ$sp$1.apply(ScriptRunner.scala:157)
at scala.Option.exists(Option.scala:219)
at scala.tools.nsc.ScriptRunner$$anonfun$withCompiledScript$1.apply$mcZ$sp(ScriptRunner.scala:157)
at scala.tools.nsc.ScriptRunner$$anonfun$withCompiledScript$1.apply(ScriptRunner.scala:131)
at scala.tools.nsc.ScriptRunner$$anonfun$withCompiledScript$1.apply(ScriptRunner.scala:131)
at scala.tools.nsc.util.package$.trackingThreads(package.scala:51)
at scala.tools.nsc.util.package$.waitingForThreads(package.scala:35)
at scala.tools.nsc.ScriptRunner.withCompiledScript(ScriptRunner.scala:130)
at scala.tools.nsc.ScriptRunner.runScript(ScriptRunner.scala:188)
at scala.tools.nsc.ScriptRunner.runScriptAndCatch(ScriptRunner.scala:201)
at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:76)
at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:96)
at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:105)
at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)

[scabug]

Phil Walker (philwalk9) said:
attachment: a script that declares a very large string, and fails with a very odd error message

[scabug]

BitMagier (bitmagier) said:
I get a similar error when running a program using a big Map object defined in source code.
Code:

object ChiSquareAlpha
{
  val ChiSquareToAlphaIndexTable = Map(
    1 -> Array(0.00016,0.00063,0.00393,0.01579,0.06418,0.14847,0.45494,1.07419,1.64237,2.70554,3.84146,5.41189,6.63490,10.82757),
    .. 9998 other entries ..
    10000 -> Array(9673.95,9711.71,9768.53,9819.19,9880.79,9925.36,9999.33,10073.68,10118.82,10181.66,10233.75,10292.58,10331.93,10442.73)
  )

  def main(args: Array[String]): Unit = {
    println(ChiSquareToAlphaIndexTable(9999)(0))
  }
}

With Scala 2.10 (target jvm-1.6) and oracle-jre-bin-1.7.0.21 (same result with jre-1.6.0.45) under 64 Bit Linux:

Exception in thread "main" java.lang.NoClassDefFoundError: bm/statistic/ChiSquareAlpha$
	at bm.statistic.ChiSquareAlpha.main(ChiSquareAlpha.scala)
Caused by: java.lang.ClassNotFoundException: bm.statistic.ChiSquareAlpha$
	at java.net.URLClassLoader$1.run(Unknown Source)
	at java.net.URLClassLoader$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(Unknown Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	... 1 more

I'll attach source file ChiSquareAlpha.scala to reproduce it.
Hope that helps to fix the bug.

[SethTisue]

comment/reopen if this still happens with current Scala