StringLike#linesIterator not properly splitting on \r

[lihaoyi]

This fails to split and returns a single string:

@ "hello\rworld".linesIterator.toArray.length
res5: Int = 1

This is inconsistent with the JVM String#lines: java.util.stream.Stream[String] method:

@ "hello\rworld".lines.toArray.length
res4: Int = 2

The standard java.io.BufferedReader#readLines method:

@
val x = new BufferedReader(new InputStreamReader(new ByteArrayInputStream("hello\rworld".getBytes)))
x: BufferedReader = java.io.BufferedReader@5769e7ae

@ x.readLine()
res2: String = "hello"

@ x.readLine()
res3: String = "world"

@ x.readLine()
res4: String = null

Or other languages like Python

>>> "hello\rworld".splitlines()
['hello', 'world']

Shouldn't it do the same thing?

[som-snytt]

scala 2.13.0-M5> "hello\n".stripLineEnd
res0: String = hello

scala 2.13.0-M5> "hello\r".stripLineEnd
"es1: String = "hello

scala 2.13.0-M5> "hello\r".stripLineEnd.length
res2: Int = 6

[szeiger]

Should \r still be recognized as a line separator today? AFAIK it was only used by the old MacOS which got replaced almost 2 decades ago. Not recognizing \r makes the implementation simpler and probably faster, too.

[som-snytt]

Compatibility with String.lines is pretty compelling.

OTOH, a\rb could be understood as an overstrike, and therefore one line.

Padding of \r\n suggests it shouldn't be a heuristic, but a mode, so that in CRNL mode, \r\r\r\n and \r\u0000\n are a single line separator.

When I scalac file.scala > scala.out ; vim scala.out, vim picks a mode and shows ^M where scalac emits mixed output because it doesn't respect the line separator.

Since the future of scala is opinionated, it has the option of ignoring line separator and String.lines and solo CR.

[lihaoyi]

Not just String.lines, but BufferedReader and all that. Surely wrapping a string in a new BufferedReader(new StringReader(s)) is simple enough in terms of implementation?

[som-snytt]

As another data point, CR is a line break for regexes, also matched by \R.

[adriaanm]

Could someone take this one and implement compat with String.lines? WWJD FTW

[som-snytt]

Because Scala recognizes FF, I just juiced the current code.

I would propose that FF be taken only if not preceded by a line terminator, since normally, FF would not introduce a line:

"abc\n\f"   // one line? status quo is two lines
"abc\f\n"   // two lines

but for now I left it as it.

FF affects line numbering in the compiler, IIRC.

[SethTisue]

scala/scala#7699 doesn't recognize FF, since Java's String.lines doesn't either

[som-snytt]

Rust issue 55743 covers the same territory.