Skip to content

Commit 98d4d62

Browse files
committed
gh-105390: Correctly raise TokenError instead of SyntaxError for tokenize errors
1 parent 0cb6b9b commit 98d4d62

File tree

7 files changed

+1934
-24
lines changed

7 files changed

+1934
-24
lines changed

Doc/library/tokenize.rst

-5
Original file line numberDiff line numberDiff line change
@@ -139,11 +139,6 @@ function it uses to do this is available:
139139
2,
140140
3
141141

142-
Note that unclosed single-quoted strings do not cause an error to be
143-
raised. They are tokenized as :data:`~token.ERRORTOKEN`, followed by the
144-
tokenization of their contents.
145-
146-
147142
.. _tokenize-cli:
148143

149144
Command-Line Usage

Doc/whatsnew/3.12.rst

+6-5
Original file line numberDiff line numberDiff line change
@@ -1490,14 +1490,15 @@ Changes in the Python API
14901490
Additionally, there may be some minor behavioral changes as a consecuence of the
14911491
changes required to support :pep:`701`. Some of these changes include:
14921492

1493-
* Some final ``DEDENT`` tokens are now emitted within the bounds of the
1494-
input. This means that for a file containing 3 lines, the old version of the
1495-
tokenizer returned a ``DEDENT`` token in line 4 whilst the new version returns
1496-
the token in line 3.
1497-
14981493
* The ``type`` attribute of the tokens emitted when tokenizing some invalid Python
14991494
characters such as ``!`` has changed from ``ERRORTOKEN`` to ``OP``.
15001495

1496+
* Incomplete single-line strings now also raise :exc:`TokenError` as incomplete
1497+
multiline strings do.
1498+
1499+
* Some incomplete or invalid Python code now raises :exc:`TokenError` instead of
1500+
returning arbitrary ``ERRORTOKEN`` tokens when tokenizing it.
1501+
15011502
Build Changes
15021503
=============
15031504

0 commit comments

Comments
 (0)