Test failures due to .es file name suffix for text/javascript file

[maxnikulin]

The following tests may fail:

• tests.test_scrapbook_indexer.TestUnSingleHtmlConverter.test_rewrite_svg
• tests.test_scrapbook_indexer.TestUnSingleHtmlConverter.test_rewrite_svg_file

-   <script href="313d6864fa48b411d082f7692efd0c0892788fc4.es"></script>
?                                                          ^
+   <script href="313d6864fa48b411d082f7692efd0c0892788fc4.js"></script>
?                                                          ^

Originally I mentioned it in #34 (comment)

I think the origin of the issue is

In [1]: import mimetypes

In [2]: mimetypes.guess_extension("text/javascript")
Out[2]: '.es'

due to

grep -w es /etc/mime.types 
text/javascript                                 es js mjs

It is specific to Debian Linux and its derivatives
https://salsa.debian.org/debian/media-types/-/commit/f67681aabbce05378e0ac45f92d398f8b7a31d5e
while RedHat-based distributions (Fedora, etc.) are not affected
https://pagure.io/mailcap/blob/master/f/mime.types#_1990

While I believe that it was oversight of the Debian media-types project to put es suffix before js, PyWebScrapBook should either explicitly set text/javascript to .js mapping or should allow .es in tests.

[maxnikulin]

I have filed https://bugs.debian.org/1061477 "Use js as first file name suffix for text/javascript", however Debian 12 bookworm (current stable release) is affected, so a workaround at the side of applications is desired.

[danny0838]

Thank you for the information and report.

During the investigation I found something bizarre. According to Python source code, mimetypes.init should generate an internal database that:

1. loads the default MIME types map defined in the source code
2. loads MIME types map from Windows registry (if available)
3. loads MIME types map from the defined config files (mimetypes.knownfiles) which resides in most *nix devices (whenever available)

When loading any of the above map, mimetypes.MimeTypes.add_type is called to add a new MIME->ext and ext->MIME mapping to the database for each entry. According to the source code, an ext->MIME mapping is last-win, while a MIME->ext mapping is first-win, and therefore the system MIME types definition should not be able to overwrite the default text/javascript->.js mapping in the source code.

This seems to be true for Python 3.12 to 3.7, and I currently cannot figure out why your system doesn't do so...

Can you provide your Python implementation for further investigation? (or better, check the source code)

[maxnikulin]

Prior to Python 3.12 it was application/javascript, not text/javascript and Debian 12 bookworm was released with Python 3.11

• mimetypes.py uses deprecated "application/javascript" python/cpython#97646
• gh-97646: Change .js and .mjs files mimetype to conform to RFC 9239 python/cpython#97934
  Adoption of rfc9239 has different pace in different packages.

I was disappointed by first win vs. last win discrepancy as well. My first idea was to suggest set_default instead of assignment for extension to media type mapping. It gives some consistency, but unfortunately it would be impossible to override mappings by e.g. calling mimetypes.init with an application-specific file.

Perhaps it is possible to extend mimetypes.add_type() and its callers to specify whether provided info should be treated with higher priority or as a fallback. For completeness remove methods for types and extensions should be added as well. It is not clear whether earlier or later files in the list should be treated with higher priority. Likely earlier ones. Some applications have ~/.mime.types before /etc/mime.types.

[danny0838]

Sorry, I did not check versions < 3.12 carefully enough and failed to notice that it's application/javascript.

I wonder the last-win and first-win behavior is implemented as how most mime.types like files work. Unfortunately the native mimetypes module does not provide a way to modify the already added MIME->ext mapping. Although we can subclass the mimetypes.MimeTypes class and generate our own db, it's not safe enough to do so as the underlying packages such as flask and werkzeug may still rely on the native mimetypes.

I implemented a quick patch that may have fixed the issue, and feel free to check if it does the job. Though it's still suboptimal as it accesses the hidden mimetypes._db, which is not part of standard library and is at higher risk of breakage in future Python releases.

[maxnikulin]

Danny, have you considered the following workaround?

• Let's do not care concerning Python built-in mapping. It may be obsolete, but it should not hamper since it is not incorrect.
• Provide a package-specific mime.types file with entries currently present in webscrapbook._polyfill.mimetypes. If this file is prepended to mimetypes.knownfiles (a part of the public API) then, unlike mimetypes.add_type() calls, it gets precedence in comparison to /etc/mime.types.
• Read the file once more after standard knownfiles for the sake of last win overrides.

I have tried the following:

#!/usr/bin/env python3
import mimetypes
import os.path

user_mime_types = os.path.join(os.path.expanduser('~'), '.mime.types')
pkg_mime_types = os.path.join(os.path.dirname(__file__), 'mime.types')


def _init(files):
    mimetypes.knownfiles[:] = files + mimetypes.knownfiles.copy() + list(reversed(files))
    mimetypes.init()


_init([user_mime_types, pkg_mime_types])

if __name__ == '__main__':
    import sys
    for typ in sys.argv[1:]:
        all = mimetypes.guess_all_extensions(typ)
        print(f'{typ}: {all}')
        for ext in all:
            name = 'stub' + ext
            print(f'{ext}: {mimetypes.guess_type(name)}')

It reports

text/javascript: ['.es', '.js', '.mjs']
.es: ('text/javascript', None)
.js: ('text/javascript', None)
.mjs: ('text/javascript', None)

However with mime.types as

text/javascript  js mjs
# Obsolete
text/ecmascript  es
application/x-javascript  js mjs
application/x-ecmascript  es
application/javascript  js mjs
application/ecmascript  es
text/x-javascript  js mjs
text/x-ecmascript  es
# Duplicate primary records.
# Python mimetypes last win idiosyncrasy for extension to type mapping.
text/ecmascript  es
text/javascript  js mjs

it gives desired result

text/javascript: ['.js', '.mjs', '.es']
.js: ('text/javascript', None)
.mjs: ('text/javascript', None)
.es: ('text/ecmascript', None)

So the only complication is shipping and extra resource file.

[danny0838]

There are already known issues in the Python default mapping and Windows registry, which are loaded before the knownfiles. Although they are ext=>MIME cases and can be overwritten, we can't guarantee there won't be a MIME=>exe case in the future. The solution shoild be able to cover such potential cases, I think.

[maxnikulin]

There are already known issues in the Python default mapping and Windows registry, which are loaded before the knownfiles.

Could you, please, be more specific? Do you mean the following?

• 5 tests are failing #53
• mimetypes.guess_type returns deprecated mimetype application/x-javascript python/cpython#90193

I have reread mimetypes docs. There is the types_map dictionary that is applied in the MimeTypes constructor, so before Windows registry. So the following should be reliable enough while using public API only. Perhaps there are a couple of media types with several extensions that may be treated specially, but it is just a several lines of code.

types_map_overrides = {
    '.js': 'text/javascript',
    '.es': 'text/ecmascript'
}


def _init2(files=None, overrides=None):
    # First wins for media type to extension mapping.
    if overrides:
        mimetypes.types_map.update(overrides)
    # Load Windows registry and POSIX /etc/mime.types.
    mimetypes.init()
    if files:
        mimetypes.init(files)
    # Ensure stable mapping of extension to media type.
    if overrides:
        for ext, typ in overrides.items():
            mimetypes.add_type(typ, ext)


_init2([user_mime_types], types_map_overrides)

It seems, first wins for media type to file name suffix is a bug:
mimetypes.init(files=None)

Each file named in files or knownfiles takes precedence over those named before it.

[danny0838]

I'm considering the similar approach. See the latest commits.

[danny0838]

There are already known issues in the Python default mapping and Windows registry, which are loaded before the knownfiles.

Could you, please, be more specific? Do you mean the following?

See the code comments, some patches are explicitly for such issues.

It seems, first wins for media type to file name suffix is a bug: mimetypes.init(files=None)

Each file named in files or knownfiles takes precedence over those named before it.

I'm not very sure about what "those named before it" means. But I think raising an issue to the Python developers for this should be OK.

[maxnikulin]

Danny, f47648f breaks tests because '.js': 'application/javascript' from mimetypes.types_map in Python-3.11 overrides '.js': 'text/javascript' in patch_types_map from _polyfill.

[danny0838]

Danny, f47648f breaks tests because '.js': 'application/javascript' from mimetypes.types_map in Python-3.11 overrides '.js': 'text/javascript' in patch_types_map from _polyfill.

Have you tested it? This shouldn't happen as it will be overwritten again by the add_type.

[danny0838]

Unfortunately, there is an issue for the "patch default types_map" approach: This approach won't work if mimetypes.inited==True, which can happen during the init of a library. E.g. In Python < 3.9, import flask imports http.server, which triggers mimetypes.init() during the initialization of SimpleHTTPRequestHandler.

Another problem is how should we deal with the user mime config. Should we allow it to overwrite our patch for native Python and OS configs? Should we allow it to overwrite MIME=>ext mapping as the last-win manner?

Applying the patch and user config when mimetypes.inited==True may be unsafe, as the super script/project may have already configured and used the mimetypes some other where, and an inconsistency of the behavior of WSB and its super script/project may happen.

[danny0838]

Reworked, see the latest commits. It seems that patching the internal _db is still the easiest and safest way to do the job. Please check if it works on your device.