bpo-38056: overhaul Error Handlers section in codecs documentation

[ghost]

• replace/backslashreplace/surrogateescape were wrongly described as encoding only, in fact they can also be used in decoding.
• clarify the description of surrogatepass.
• add more descriptions to each handler.
• add two REPL examples.
• add indexes for Error Handler's name.
• add default parameter values in codecs.rst
• improve term "text encoding".

I read the source code, hope there is no mistake and misleading.

The indexes were tested in .chm documentation.

The term "text encoding" appeared 38 times in the entire document, the change in glossary.rst should be correct.

Also verified these terms with Wikipedia and Google:

1. XML/HTML numeric character reference
2. Name property from Unicode Character Database

October 2nd effect:

https://bugs.python.org/issue38056

[aeros]

Thanks for the PR @animalize.

As someone who is not particularly experienced with using the codec module (for my purposes, I've never had much of a need for it), these examples are quite helpful for visually representing the purpose of each of the error handlers.

I'm not entirely certain that we need to provide an example for all four of them, but in general I think that providing an example or two would be very helpful to readers.

[aeros]

If these changes are approved, I think it would be appropriate to backport the changes to the 3.7 and 3.8 docs. Also, I've added a skip news label. You can add one if you'd like to, but it shouldn't be required for adding examples to the documentation.

I'll cc the experts for codecs.

/cc @malemburg @doerwalter

[ghost]

I'm not entirely certain that we need to provide an example for all four of them, but in general I think that providing an example or two would be very helpful to readers.

Thanks for your review.
My idea is to demonstrate these things:

• appropriate XML character reference
• backslashed escape sequences
• \N{...} escape sequences

So I would like to include at least these three.

I prefered to use this example, it covers a non-BMP character 🏫.

>>> '42 µ ♪ 🏫'.encode(encoding='ascii', errors='replace')
b'42 ? ? ?'
>>> '42 µ ♪ 🏫'.encode(encoding='ascii', errors='xmlcharrefreplace')
b'42 µ ♪ 🏫'
>>> '42 µ ♪ 🏫'.encode(encoding='ascii', errors='backslashreplace')
b'42 \\xb5 \\u266a \\U0001f3eb'
>>> '42 µ ♪ 🏫'.encode(encoding='ascii', errors='namereplace')
b'42 \\N{MICRO SIGN} \\N{EIGHTH NOTE} \\N{SCHOOL}'

But if paste the statement into the REPL on Windows, the non-BMP character is broken, and get an invalid result \\ufffd. Maybe this will confuse the user, so I abandoned this example.

>>> '42 µ ♪ �'.encode(encoding='ascii', errors='backslashreplace')
b'42 \\xb5 \\u266a \\ufffd'

[aeros]

Not that I'm opposed to the changes, but I think the examples alone would be significantly easier and less controversial to review. Also, it's usually more efficient from a workflow perspective to have more multiple focused PRs than a single large overhaul at once.

[ghost]

Not that I'm opposed to the changes, but I think the examples alone would be significantly easier and less controversial to review. Also, it's usually more efficient from a workflow perspective to have more multiple focused PRs than a single large overhaul at once.

Ah, just like suddenly getting excited for this PR. 😄

There are obvious errors, for example the backslashreplace error handler works with decoding since Python 3.5, but it is still in "encode only table".

replace error handler also works with both encode and decode codecs.

[ghost]

If codec experts want to do this, or who thinks he/she can do better than me, free feel to create a new PR.

[aeros]

@animalize:

If codec experts want to do this, or who thinks he/she can do better than me, free feel to create a new PR.

I'm not recommending for anyone else to do the other parts of the current one, you can still potentially apply the all of the changes. I just mean having only the examples in this PR and then splitting the other sections into their own PRs. As the scope of a PR gets larger, the time required to fully review it prior to a merge significantly increases. From my experience, the core developers tend to prefer smaller and more focused PRs when it's possible.

[JulienPalard]

I'm not sure reorganizing the table is worth it, what did you try to enhance by replacing the separation "all codecs vs text codecs" by the separation "both encode/decode vs encode only"?

[bedevere-bot]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

Author pushed changes.

[JulienPalard]

LGTM with nits, but not codec expert here.

[ghost]

LGTM with nits, but not codec expert here.

I am reading the source code these days, hope there will be no mistakes.

This PR bases on #15135.
We need to wait for it to be backported to 3.7/3.8 branches.

[miss-islington]

Thanks @animalize for the PR, and @JelleZijlstra for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

[miss-islington]

Thanks @animalize for the PR, and @JelleZijlstra for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

[miss-islington]

Thanks @animalize for the PR, and @JelleZijlstra for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

[bedevere-bot]

GH-92526 is a backport of this pull request to the 3.9 branch.

[bedevere-bot]

GH-92527 is a backport of this pull request to the 3.11 branch.

[bedevere-bot]

GH-92528 is a backport of this pull request to the 3.10 branch.

[ghost]

Thanks for your reviews, which has led to these revisions being improved in every aspects.

I just didn't add more examples, but I think too many examples will disturb the readers.