Continuation lines confuse the intra-word highlighting

[danielshahaf]

% tests/generate.zsh $'echo foo\\\nbar"baz"' main backslash-continuation2
BUFFER=$'echo foo\\\nbar"baz"'

expected_region_highlight=(
  '1 4 builtin' # echo
  '6 16 default' # foo\\\nbar"ba
  '12 16 double-quoted-argument' # ar"ba
)

What happens is that ${(z)} eats the backslash for us:

% pz $'echo foo\\\nbar"baz"' 
$'echo'
$'foobar"baz"'
% 

… but we don't account for that.

[danielshahaf]

I'm tempted to fix this by adding a new ${(Z+…+)} subflag to zsh itself. Thoughts?

[danielshahaf]

FWIW, the zsh patch would be something like this:

zsh patch: add `${(Z+\+)}` subflag

diff --git a/Src/lex.c b/Src/lex.c
index 1d86da94e..41008409a 100644
--- a/Src/lex.c
+++ b/Src/lex.c
@@ -714,7 +714,7 @@ gettok(void)
     switch (lexact1[STOUC(c)]) {
     case LX1_BKSLASH:
 	d = hgetc();
-	if (d == '\n')
+	if (d == '\n' && !(lexflags & LEXFLAGS_CONTINUATION_LINES))
 	    goto beginning;
 	hungetc(d);
 	lexstop = 0;
@@ -1236,7 +1236,7 @@ gettokstr(int c, int sub)
 	    break;
 	case LX2_BKSLASH:
 	    c = hgetc();
-	    if (c == '\n') {
+	    if (c == '\n' && !(lexflags & LEXFLAGS_CONTINUATION_LINES)) {
 		c = hgetc();
 		if (!lexstop)
 		    continue;
@@ -2140,6 +2140,7 @@ skipcomm(void)
      * across the entire construct, and parse_event() needs embedded
      * newlines to be "real" when looking for the OUTPAR token.
      */
+    /* TODO: should LEXFLAGS_CONTINUATION_LINES be removed here, or not? */
     lexflags &= ~(LEXFLAGS_ZLE|LEXFLAGS_NEWLINE);
     dbparens = 0;	/* restored by zcontext_restore_partial() */
 
diff --git a/Src/subst.c b/Src/subst.c
index 79efc9ad2..be86160ed 100644
--- a/Src/subst.c
+++ b/Src/subst.c
@@ -2204,6 +2204,11 @@ paramsubst(LinkList l, LinkNode n, char **str, int qt, int pf_flags,
 				shsplit |= LEXFLAGS_NEWLINE;
 				break;
 
+			    case Bnull:
+			    case '\\':
+				shsplit |= LEXFLAGS_CONTINUATION_LINES;
+				break;
+
 			    default:
 				*t = sav;
  				goto flagerr;
diff --git a/Src/zsh.h b/Src/zsh.h
index 834142895..d41797dd8 100644
--- a/Src/zsh.h
+++ b/Src/zsh.h
@@ -2271,6 +2271,10 @@ struct histent {
  * Treat newlines as whitespace
  */
 #define LEXFLAGS_NEWLINE	0x0010
+/*
+ * Pass continuation lines unhindered
+ */
+#define LEXFLAGS_CONTINUATION_LINES 0x0020
 
 /*******************************************/
 /* Definitions for programmable completion */

With this patch, I get:

% b/Src/zsh -fc 'print -r -- ${(qqqq)${(z)@}}' . $': foo\\\nbar'
$':' $'foobar'
% b/Src/zsh -fc 'print -r -- ${(qqqq)${(Z+\+)@}}' . $': foo\\\nbar'
$':' $'foo\\\nbar'
% 

But some tests fail (once I patch main-highlighter.zsh to use the new feature):

z-sy-h test failures with the zsh patch and with main highlighter patched to use it

ZSH_PATCHLEVEL=zsh-5.8-57-gda83522
Running test brackets
Running test main

## backslash-continuation
# PREBUFFER='echo \
'
# BUFFER=noglob
not ok 1 - [1,8] «noglob» - expected (1 6 "default"), observed (1 8 "default"). 
ok 2 - cardinality check

## backslash-continuation2
# BUFFER='echo foo\
bar"baz"'
ok 1 - [1,4] «echo»
ok 2 - [6,18] «foo\↵bar"baz"» - # TODO "issue #705"
ok 3 - [14,18] «"baz"» - # TODO "issue #705"
ok 4 - cardinality check # SKIP cardinality check disabled whilst regular test points are expected to fail

## empty-line
# BUFFER='\
; ls'
not ok 1 - [3,4] «; » - expected (3 3 "unknown-token"), observed (3 4 "unknown-token"). 
not ok 2 - [5,5] «l» - expected (5 6 "command"), observed (5 5 "commandseparator"). 
not ok 3 - cardinality check - have 2 expectations and 3 region_highlight entries: «expected_region_highlight=( '3 3 unknown-token' '5 6 command' )» «region_highlight=( '2 4 unknown-token' '4 5 commandseparator' '5 7 command' )»
# expected_region_highlight  region_highlight
# '3 3 unknown-token'        '2 4 unknown-token'
# '5 6 command'              '4 5 commandseparator'
# .                          '5 7 command'

## loop-newline
# BUFFER='for i in \
; do done'
ok 1 - [1,3] «for»
ok 2 - [5,5] «i»
ok 3 - [7,8] «in»
not ok 4 - [12,13] «; » - expected (12 12 "commandseparator"), observed (12 13 "default"). 
not ok 5 - [14,14] «d» - expected (14 15 "reserved-word"), observed (14 14 "commandseparator"). 
not ok 6 - [15,16] «o » - expected (17 20 "reserved-word"), observed (15 16 "reserved-word"). 
not ok 7 - cardinality check - have 6 expectations and 7 region_highlight entries: «expected_region_highlight=( '1 3 reserved-word' '5 5 default' '7 8 default' '12 12 commandseparator' '14 15 reserved-word' '17 20 reserved-word' )» «region_highlight=( '0 3 reserved-word' '4 5 default' '6 8 default' '11 13 default' '13 14 commandseparator' '14 16 reserved-word' '16 20 reserved-word' )»
# expected_region_highlight  region_highlight
# '1 3 reserved-word'        '0 3 reserved-word'
# '5 5 default'              '4 5 default'
# '7 8 default'              '6 8 default'
# '12 12 commandseparator'   '11 13 default'
# '14 15 reserved-word'      '13 14 commandseparator'
# '17 20 reserved-word'      '14 16 reserved-word'
# .                          '16 20 reserved-word'
Running test pattern
Running test regexp

Is this a good direction to pursue?

[danielshahaf]

In not ok 1 - [1,8] «noglob», why is the string between guillemets not of length 8?