Build platform-specific wheels containing libmagic #294

ddelange · 2023-09-06T12:59:42Z

This PR builds self-contained wheels as discussed in #233. For Windows users, this renders python-magic-bin from @julian-r obsolete.

pip install these wheels

@fpgmaas published the unofficial python-magic-standalone PyPI package based on this branch.
```
pip install python-magic-standalone
```

Alternatively, pip can install these wheels from the GitHub Release assets from my fork:

# space separated list of URLs for --find-links
export PIP_FIND_LINKS='https://github.com/ddelange/python-magic/releases/expanded_assets/0.4.28.post9'
pip install --force-reinstall python-magic

The wheels diff:

- python-magic-0.4.27.tar.gz
+ python-magic-0.4.28.tar.gz
- python_magic-0.4.27-py2.py3-none-any.whl
+ python_magic-0.4.28-py2.py3-none-macosx_10_9_x86_64.whl
+ python_magic-0.4.28-py2.py3-none-macosx_11_0_arm64.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl
+ python_magic-0.4.28-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_aarch64.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_ppc64le.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_s390x.whl
+ python_magic-0.4.28-py2.py3-none-musllinux_1_1_x86_64.whl
+ python_magic-0.4.28-py2.py3-none-win32.whl
+ python_magic-0.4.28-py2.py3-none-win_amd64.whl

The wheels:

Come bundled with libmagic (.dylib on mac, .so on nix, .ddl on win) - no additional user action needed
- latest libmagic (5.46)
  - building from source on linux and macos with maximum backwards compatibility
  - using MSYS2 built libmagic on windows (mingw32 and mingw64)
- skipping i686 linux wheels as they don't pass pytest ref ca4def3
Are forward compatible, they will install on cpython 2.7 and all current and future cpython 3.5+ distributions
Are built for the major cpu architectures (and also for alpine linux)
- ~~Linux architectures limited by availability: https://pkgs.org/search/?q=file-libs~~ now building from source on linux
- Windows ARM64 is not included currently, but MSYS2 built libmagic is available
Are uploaded as GitHub Release assets, as well as upload to PyPI, whenever a GitHub Release is (pre)released.
- 🚩 @ahupp for this, you still need to add wheels.yml as trusted publisher here

CI/CD

dists build with official cibuildwheel on GitHub Actions, and they build in parallel:

The PyPI publish step is failing here ^ because my fork is not a trusted publisher
For any system not covered by these wheels, pip will fall back to the source distribution, which will check for proper libmagic on the systen at install time.

fix #137, fix #288, fix #225, fix #276, fix #248, fix #87, fix #139, fix #233, fix #73, fix #60, fix #34, fix #293, fix #233, fix #278, fix #262, fix #248, fix #238, fix #145, fix #61, fix #12, fix #295, fix #311, fix #312, fix #313, fix #321, fix #332, fix #249, fix #333, fix #350, fix #351, fix #287, fix #335

apirogov · 2023-09-12T13:49:39Z

This is nice! Hope this will be merged soon!

Just ran into issues with my library being not usable by Mac and Windows users because I rely on python-magic. If there are wheels, I don't need to find a workaround or replace the library :)

python-magic-bin did not work for some of them, by the way.

stumpylog · 2023-09-12T14:40:45Z

.github/workflows/main.yml

+      with:
+        files: dist/*
+
+    - name: Upload to PyPI


A small improvement here might be to use the PyPa Action instead: https://github.com/pypa/gh-action-pypi-publish

The big advantage is trusted publishing, instead of storing a password or token as a secret

that's cool, thanks for sharing!

@ahupp shall I make that change and you set it up on PyPI side?

Sounds like trusted publishing is the way to go. I recently got this email:

Hi ddelange!
Earlier this year, we announced that PyPI would require all users to enable a form of two-factor authentication on their accounts by the end of 2023.

Keeping your PyPI account secure is important to all of us. We encourage you to enable two-factor authentication on your PyPI account as soon as possible.

What forms of 2FA can I use?
We currently offer two main forms of 2FA for your account:

Security device including modern browsers (preferred) (e.g. Yubikey, Google Titan)
Authentication app (e.g. Google Authenticator)
Once one of these secure forms is enabled on your account, you will also need to use either Trusted Publishers (preferred) or API tokens to upload to PyPI.

What do I do if I lose my 2FA device?
As part of 2FA enrollment, you will receive one-time use recovery codes. One of them must be used to confirm receipt before 2FA is fully active. Keep these recovery codes safe - they are equivalent to your 2FA device. Should you lose access > to your 2FA device, use a recovery code to log in and swap your 2FA to a new device.

Read more aboutrecovery codes.

Why is PyPI requiring 2FA?
Keeping all users of PyPI is a shared responsibility we take seriously. Strong passwords combined with 2FA is a recognized secure practice for over a decade.

We are requiring 2FA to protect your account and the packages you upload, and to protect PyPI itself from malicious actors. The most damaging attacks are account takeover and malicious package upload.

To see this and other security events for your account, visit your account security history.

Read more on this blog post.

If you run into problems, read the FAQ page. If the solutions there are unable to resolve the issue, contact us via [email protected].

Thanks,
The PyPI Admins

@ahupp so the last thing for you to do is adding this repo as trusted publisher to https://pypi.org/manage/project/python-magic/settings/publishing/

.github/workflows/main.yml

ahupp · 2023-09-28T17:18:10Z

This is huge, thank you! Apology for the delay I thought I'd commented earlier but guess not. I'll look this over soon; I didn't quite understand how bad the binary dep situation was expecially on windows.

.github/workflows/main.yml

jean-humann · 2023-12-03T17:02:05Z

@ahupp @stumpylog could we have this one merged (and released) by the end of the year please ?

ddelange · 2025-03-18T21:32:55Z

so I pulled master again and now Windows wheels are failing with the following error. I don't know why it now started failing. I don't see changes to https://github.com/julian-r/file-windows, nor relevant changes in master. Anyway, better to get rid of that dependency so I pushed for a more official libmagic wheel build for Windows. Fix incoming, with latest libmagic 4.46 as a bonus! 🎉

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-l6641oui\cp312-win_amd64\venv-test\Lib\site-packages\magic\__init__.py", line 120, in __init__
      magic_load(self.cookie, magic_file)
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-l6641oui\cp312-win_amd64\venv-test\Lib\site-packages\magic\__init__.py", line 384, in magic_load
      return _magic_load(cookie, coerce_filename(filename))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-l6641oui\cp312-win_amd64\venv-test\Lib\site-packages\magic\__init__.py", line 275, in errorcheck_negative_one
      raise MagicException(err)
  magic.MagicException: b"File 5.44 supports only version 18 magic files. `C:\\Users\\runneradmin\\AppData\\Local\\Temp\\cibw-run-l6641oui\\cp312-win_amd64\\venv-test\\Lib\\site-packages\\magic\\magic.mgc' is version 20"

ddelange · 2025-03-18T21:59:44Z

for anyone interested in latest libmagic on windows: I created a new GitHub release on my fork, and updated the pip install instructions in this PR description.

DeflateAwning · 2025-08-11T17:05:55Z

Can this please be merged? Is this library actively maintained? Seems like a serious improvement.

.github/workflows/ci.yml

.github/workflows/wheels.yml

cclauss

It would be good to get this landed.

fpgmaas · 2025-09-04T11:40:13Z

Any updates here @ahupp ? I think many users are eagerly waiting for this to be available on PyPI. Even having a prerelease available on PyPI would be great!

fpgmaas · 2025-09-24T11:41:07Z

Since this PR has been open for quite a while and there are no indications of it being merged anytime soon, I took the liberty of uploading the wheels to PyPI under the name python-magic-standalone, see https://pypi.org/project/python-magic-standalone/. It's sole purpose is to make it possible for users to install @ddelange's work directly from PyPI. I don't intend to maintain this fork, and I added a clear disclaimer to the README. It can be installed with any of the following commands, depending on the tool you use;

# pip
pip install python-magic-standalone

# uv
uv add python-magic-standalone

# poetry
poetry add python-magic-standalone

cclauss · 2025-09-24T11:58:21Z

@fpgmaas, please consider building wheels for Python 3.14, as it is less than two weeks away.

Add Python 3.14 to the testing #355

ddelange · 2025-09-24T11:59:53Z

this branch / these wheels are forward compatible, no need for changes in the build 👍

ddelange · 2025-09-24T15:21:23Z

Updated the PR description. Thanks @fpgmaas 🎉

https://github.blog/changelog/2025-09-19-github-actions-macos-13-runner-image-is-closing-down/

.github/workflows/wheels.yml

CodeXex123

Looks good

ddelange · 2025-10-10T19:12:35Z

magic/__init__.py

49982d2 is a git cherry-pick from https://github.com/thc1102/python-magic to fix #287 / #335

This reverts commit b4c04a3.

ddelange changed the title ~~Build ABI3 wheels containing libmagic~~ Build platform-specific wheels containing libmagic Sep 7, 2023

ddelange force-pushed the abi3-wheels branch 2 times, most recently from a98f13b to dc9c393 Compare September 7, 2023 07:26

ddelange mentioned this pull request Sep 7, 2023

Build platform-specific wheels containing libmagic ddelange/python-magic#1

Closed

ddelange force-pushed the abi3-wheels branch 12 times, most recently from d672b91 to 14f7dbb Compare September 7, 2023 10:03

Build platform-specific wheels containing libmagic

ec952d7

ddelange force-pushed the abi3-wheels branch from 14f7dbb to ec952d7 Compare September 7, 2023 10:09

jspraul mentioned this pull request Sep 9, 2023

Update file to 5.44 hey-red/Mime#49

Closed

apirogov mentioned this pull request Sep 12, 2023

python-magic has no wheels for Windows/Mac Materials-Data-Science-and-Informatics/metador-core#85

Open

stumpylog reviewed Sep 12, 2023

View reviewed changes

.github/workflows/main.yml Outdated Show resolved Hide resolved

Move persmissions into job scope, remove ABI3 reference

a437409

ahupp mentioned this pull request Oct 10, 2023

Binary distribution for libmagic on Windows #293

Open

Switch to PyPI trusted publishing

4a715e2

ddelange mentioned this pull request Nov 23, 2023

bugfix: add path to load libmagic bin on Windows #249

Open

ddelange commented Nov 23, 2023

View reviewed changes

.github/workflows/main.yml Outdated Show resolved Hide resolved

Typo

33c827e

Get latest libmagic from MSYS2 for Windows (#10)

688edf0

ddelange mentioned this pull request Mar 18, 2025

Get latest libmagic from MSYS2 for Windows ddelange/python-magic#10

Merged

Remove dev leftover

9dd2ebc

ddelange added 3 commits August 11, 2025 19:25

Merge branch 'master' into abi3-wheels

57b2559

Update README.md

ed077fc

Mention MSYS2 for Windows in README.md

0571c11

cclauss reviewed Aug 13, 2025

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

ddelange commented Aug 13, 2025

View reviewed changes

.github/workflows/wheels.yml Show resolved Hide resolved

cclauss suggested changes Aug 13, 2025

View reviewed changes

cclauss approved these changes Aug 13, 2025

View reviewed changes

ddelange mentioned this pull request Aug 18, 2025

Tests failing #340

Open

Merge branch 'ahupp:master' into abi3-wheels

ead0595

ddelange mentioned this pull request Sep 28, 2025

Add python-magic-bin as a dependency for Windows users #332

Open

mauritsvanrees mentioned this pull request Sep 29, 2025

Introduce python-magic and modernize various places plone/Products.MimetypesRegistry#20

Open

Bump macos-13 to macos-15-intel

3ebe8bc

https://github.blog/changelog/2025-09-19-github-actions-macos-13-runner-image-is-closing-down/

cclauss reviewed Oct 2, 2025

View reviewed changes

.github/workflows/wheels.yml Show resolved Hide resolved

Build wheels on cp313

b4c04a3

CodeXex123 approved these changes Oct 5, 2025

View reviewed changes

update coerce_filename

49982d2

ddelange commented Oct 10, 2025

View reviewed changes

Revert "Build wheels on cp313"

f99e6de

This reverts commit b4c04a3.

Build platform-specific wheels containing libmagic #294

Are you sure you want to change the base?

Build platform-specific wheels containing libmagic #294

Uh oh!

Conversation

ddelange commented Sep 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pip install these wheels

CI/CD

Uh oh!

apirogov commented Sep 12, 2023

Uh oh!

stumpylog Sep 12, 2023

Choose a reason for hiding this comment

Uh oh!

ddelange Sep 12, 2023

Choose a reason for hiding this comment

Uh oh!

ddelange Oct 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ddelange Nov 13, 2023

Choose a reason for hiding this comment

Uh oh!

ddelange Nov 22, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ahupp commented Sep 28, 2023

Uh oh!

Uh oh!

jean-humann commented Dec 3, 2023

Uh oh!

ddelange commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ddelange commented Mar 18, 2025

Uh oh!

DeflateAwning commented Aug 11, 2025

Uh oh!

Uh oh!

Uh oh!

cclauss left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fpgmaas commented Sep 4, 2025

Uh oh!

fpgmaas commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cclauss commented Sep 24, 2025

Uh oh!

ddelange commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ddelange commented Sep 24, 2025

Uh oh!

Uh oh!

CodeXex123 left a comment

Choose a reason for hiding this comment

Uh oh!

ddelange Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

22 participants

ddelange commented Sep 6, 2023 •

edited

Loading

ddelange Oct 26, 2023 •

edited

Loading

ddelange commented Mar 18, 2025 •

edited

Loading

cclauss left a comment •

edited

Loading

fpgmaas commented Sep 24, 2025 •

edited

Loading

ddelange commented Sep 24, 2025 •

edited

Loading

ddelange Oct 10, 2025 •

edited

Loading