support encoded filename in Content-Disposition for HTTP in cgi.FieldStorage

[MyroslavOpyr]

BPO	23434
Nosy	@warsaw, @ezio-melotti, @bobince, @bitdancer, @vadmium, @serhiy-storchaka, @demianbrecht, @pawciobiel
PRs	bpo-33027: Fix cgi.FieldStorage to handle Content-Disposition filename* with encoding according to RFC5987 #6027
Files	test_cgi.py-v2.7.5-rfc6266_filename.patch: test revealing the issue cgi.py-v2.7.5-rfc6266_filename.patch: rfc6266 powered fix

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-02-10.14:54:47.750>
labels = ['3.7', '3.8', 'expert-email', 'type-feature', 'library', 'expert-unicode']
title = 'support encoded filename in Content-Disposition for HTTP in cgi.FieldStorage'
updated_at = <Date 2020-01-08.10:45:02.342>
user = 'https://bugs.python.org/MyroslavOpyr'

bugs.python.org fields:

activity = <Date 2020-01-08.10:45:02.342>
actor = 'aclover'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Unicode', 'email']
creation = <Date 2015-02-10.14:54:47.750>
creator = 'Myroslav.Opyr'
dependencies = []
files = ['38092', '38094']
hgrepos = []
issue_num = 23434
keywords = ['patch']
message_count = 11.0
messages = ['235688', '235732', '235734', '235909', '236101', '236365', '314265', '314297', '359224', '359533', '359577']
nosy_count = 10.0
nosy_names = ['barry', 'ezio.melotti', 'aclover', 'r.david.murray', 'Myroslav.Opyr', 'martin.panter', 'piotr.dobrogost', 'serhiy.storchaka', 'demian.brecht', 'pawciobiel']
pr_nums = ['6027']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23434'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[MyroslavOpyr]

cgi.FieldStorage has problems parsing the multipart/form-data request with file fields with non-latin filenames. It drops the filename parameter formatted according to RFC6266 [1] (most modern browsers do). There is already python implementation for that RFC in rfc6266 module [2].

Ref:
[1] https://tools.ietf.org/html/rfc6266
[2] https://pypi.python.org/pypi/rfc6266

[MyroslavOpyr]

In test_cgi.py-v2.7.5-rfc6266_filename.patch there is a patch to test_cgi.py (Python 2.7.5) that reveals the issue.

[MyroslavOpyr]

As a proof of concept there is fix for the issue powered by rfc6266 library[1]. See cgi.py-v2.7.5-rfc6266_filename.patch

References:
[1] https://pypi.python.org/pypi/rfc6266

[bitdancer]

Since that library is not part of the stdlib, this is not an appropriate patch for CPython.

Note that this issue is also relevant to the email library, which intends to support RFC2616 header parsing/generation, and therefore should also be enhanced to support RFC 6266.

[MyroslavOpyr]

Hi David,

According to "Test Cases for HTTP Content-Disposition header field" overview [1], this is not about email headers, but only about HTTP headers. It look like email standards and http standars are different in this area.

I do know that my patch is poor. It is just proof of concept, to show that there is an issue in stdlib and one of the possible fast patches to get functionality needed.

Regards,

Myroslav

Ref:
[1] http://greenbytes.de/tech/tc2231/

[bitdancer]

I know it is called the 'email' package, but the intent is to support http header parsing as well (cf email.policy.HTTP).

[pawciobiel]

I didn't find this and created a duplicate
https://bugs.python.org/issue33027

I've added similar/updated changes
#6027

@r.david.murray wouldn't it be wise to do one step at a time rather than implementing full support for RFC6266? Please tell exactly what is your expectations so I can fix the patch if it needs to be fixed.

This is also related to RFC5987
https://tools.ietf.org/html/rfc5987
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition

[bitdancer]

I haven't read the http rfcs, but my understanding is that they follow the MIME standards, and the email library already has code to do proper parsing and decoding of encoded filenames in Content-Disposition headers. It should be possible to call that code for this use case (the http libraries already depend on the email libraries, although I'm not sure if cgi itself does currently). There may be additional considerations involved in fully supporting the http RFCs, but to determine that someone will need to read both and understand them, which is not a small undertaking :)

In the meantime, I'm pretty sure that using the existing mime header parsing code in the email library (see email.headerregistry) will provide better parsing than the only-handles-simple-cases heuristic in your PR. Granted, I don't think you have to deal with multi-part headers in http, but I vaguely remember that there are other subtleties not handled by a simple split on '.

[bobince]

HTTP generally isn't an RFC 822-family standard. Its headers look a lot like it, but they have their own defined syntax that differs in niggling little details. Using mail parsing code for HTTP isn't usually the right thing.

HTTP has always used its own syntax definitions for the headers on the main request/response entities, but it has traditionally partially deferred to RFC 822-family specs for the definitions of structured entity bodies. This is moot, however, as the reality of what browsers support has rarely coincided with those specs.

Nowadays HTML5.2 explicitly defers to RFC 7578 for definition of multipart/form-data headers. (This RFC is a replacement for the vague and broken RFC 2388.) As is to be expected for an HTML5-related spec, RFC 7578 shrugs and documents existing browser behaviour [section 4.2]:

• some browsers do UTF-8
• some browsers do data mangling (IE's %-encoding sadness)
• some browsers might do something else

but it explicitly rules out the solution proposed here:

"The encoding method described in [RFC5987], which would add a 'filename*' parameter to the Content-Disposition header field, MUST NOT be used."

The introductions of both RFC 5987 and RFC 6266 explicitly exclude multipart/form-data headers from their remit.

So in summary:

• we shouldn't do anything
• the situation with submitted filenames will continue to be broken for everyone indefinitely

[bitdancer]

Are you saying there is no (http) RFC compliant way to fix this, or no way to fix it with the email library parsers? If the latter, the library is pretty flexible and for internal stdlib use it would probably be permissible to directly call methods in the internal parsing module, if those would be useful.

I haven't re-read the issue to reload my brain, so this question may be off point (except for the first clause of the question).

[bobince]

Are you saying there is no (http) RFC compliant way to fix this

Sadly, yes.

And though RFCs aren't always a fair representation of real-world use, RFC 7578 is informative as well as normative: at present nothing produces "filename*=" in multipart/form-data.

[hugovk]

Closing, because the cgi module was deprecated in Python 3.11 and will be removed in 3.13:

• PEP 594 – Removing dead batteries from the standard library
• Deprecate modules listed in PEP 594 #91217

It has been moved to a separate package on PyPI, maintained by the community:

• https://discuss.python.org/t/pep-594-take-2-removing-dead-batteries-from-the-standard-library/13508/23?u=hugovk
• https://pypi.org/project/legacy-cgi/