From 8fefd4f925d7e0a1871ac499ed7ef62923b6a3cd Mon Sep 17 00:00:00 2001 From: Thomas Lively <7121787+tlively@users.noreply.github.com> Date: Tue, 7 Jun 2022 15:19:40 -0700 Subject: [PATCH] [spec] Clarify that strings can separate tokens Since `"` is not included in `idchar`, it can mark the end of a token. For example, `0"foo"$x` would parse as `0`, `"foo"`, `$x`. --- document/core/text/lexical.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/document/core/text/lexical.rst b/document/core/text/lexical.rst index a3b4529d35..d27f02669d 100644 --- a/document/core/text/lexical.rst +++ b/document/core/text/lexical.rst @@ -63,7 +63,8 @@ The set of *keyword* tokens is defined implicitly, by all occurrences of a :ref: Any token that does not fall into any of the other categories is considered *reserved*, and cannot occur in source text. .. note:: - The effect of defining the set of reserved tokens is that all tokens must be separated by either parentheses or :ref:`white space `. + The effect of defining the set of reserved tokens is that all tokens must be separated by either parentheses, + :ref:`white space `, or :ref:`string ` literals. For example, :math:`\text{0\$x}` is a single reserved token. Consequently, it is not recognized as two separate tokens :math:`\text{0}` and :math:`\text{\$x}`, but instead disallowed. This property of tokenization is not affected by the fact that the definition of reserved tokens overlaps with other token classes.