Support Markdown for readmes

[ewdurbin]

Originally reported by: Hickford (Bitbucket: hickford, GitHub: hickford)

---

If you write your readme in Restructured Text, PyPI will render it on your project's page (eg. https://pypi.python.org/pypi/requests ). It should be possible to write your readme in Markdown too. Markdown is a pervasive format used on GitHub and StackOverflow. It shouldn't be necessary to learn a new markup language or rewrite your project's documentation to publish a Python package.

This is common frustration. The Python community suffers because it's a barrier to publishing packages and writing documentation . Please consider supporting Markdown!

• http://stackoverflow.com/questions/10718767/have-the-same-readme-both-in-markdown-and-restructuredtext
• http://i.imgur.com/7NYlimN.png
• https://coderwall.com/p/qawuyq

---

Update 2014: Here's a pull request! https://bitbucket.org/pypa/pypi/pull-request/59/support-markdown-for-readmes-fixes-148/diff

---

• Bitbucket: https://bitbucket.org/pypa/pypi/issue/148

[ewdurbin]

Original comment by Will McKenzie (Bitbucket: kmd_willmckenzie, GitHub: Unknown):

---

I'll second this! Much prefer Markdown syntax and it's a pain having to convert before uploading a package.

[ewdurbin]

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

---

This is unlikely to happen in the near term. Warehouse (PyPI 2.0) is organized so that eventually we can support multiple markdown selections. The primary problem is we don't have any way to know if we should be rendering in markdown or rST. One way around that would be to render Markdown if rST fails, however some markdown is valid rST but it renders incorrectly in that case and looks really bad. It's also not a very particularly elegant solution especially if people want to do something else like asciidoc or some other formatting method. Likely the metadata 2.0 stuff will (eventually maybe?) support saying what markup your description is in and then we can render it using the appropriate renderer. It will also enable us to actually reject and upload if a package doesn't render correctly instead of just falling back to plaintext, because we'll know if it's supposed to be able to render or not.

[ewdurbin]

Original comment by Hickford (Bitbucket: hickford, GitHub: hickford):

---

Really? I understand PyPI's current algorithm to be

If there's a readme.rst, readme.txt or readme (no extension) file, render that as restructured text. Otherwise, nothing.

Here's a backwards compatible algorithm that would elate markdown authors:

If there's a readme.rst, readme.txt or readme (no extension) file render that as restructured text. Else if there's a readme.md or readme.markdown file render that as Markdown. Otherwise, nothing.

If you're concerned about projects with a readme.txt written in Markdown, they are already broken. This change wouldn't fix that, but it would help the majority of Markdown authors who use the .md or .markdown extensions. They do that because GitHub renders files with those extensions.

[ewdurbin]

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

---

No, PyPI doesn't exactly do that. PyPI renders a long_description, if a long_description is not provided it will attempt to discover a README file and will insert that as the long_description.

[ewdurbin]

Original comment by Will McKenzie (Bitbucket: kmd_willmckenzie, GitHub: Unknown):

---

A potential interim solution (until the metadata stuff that will let a package maintainer specify the format) could be to have a developer leave the long_description field blank and have a README.(md|markdown) file to be rendered in it's stead? I know with my packages I'm basically just reading the README.txt file and assigning it to the long description field (I didn't know how the algorithm worked).

[ewdurbin]

Original comment by Hickford (Bitbucket: hickford, GitHub: hickford):

---

I had a go! Here's a pull request https://bitbucket.org/pypa/pypi/pull-request/33/support-markdown-for-readmes/diff

[ewdurbin]

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):

---

I like Will's idea. Many, many Python projects have boilerplate in setup.py to set long_description to the contents of their README. That's because people want to have the best chance of their description getting picked up but almost everyone prefers editing a description in a separate file over something embedded in setup.py, plus the file extension tells you what format it's in.

I think it would be great if PyPI/warehouse scanned for a README.rst, README.md, README.markdown and used that as the priority for the description to show. Perhaps setuptools could be modified to warn you in the unlikely event that your long_description has different data than your README (@jaraco: What do you think?).

I think in the next version of the metadata standard I would drop long_description and instead offer something like long_description_filename.

Maybe not a bad idea to add long_description_filename to setuptools now. Then package maintainers could start setting that and PyPI/Warehouse could use that in preference to long_description. That would let the server instantly know the format language from the extension and package maintainers could set it and remove lines from their setup.py to populate long_description from a file.

[ewdurbin]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

@msabramo I'm not sure your suggestion makes sense in the general case. What you and Will are suggesting is to create a new field in the metadata that references content outside the metadata. In particular, long_description_filename references a file that exists in one of the uploaded distribution packages, but which one? What if the distribution package is a wheel or egg or Windows installer? What if there are multiple distributions (Python version specific)? What if the metadata has been uploaded (e.g. through the register command) but no distribution has been uploaded? I see value in having the long description inline with the metadata in all cases, because that means that all metadata is represented in one entity and doesn't require steps to assemble it (and the complication that entails).

[ewdurbin]

Original comment by Krisztian Fekete (Bitbucket: krisztianfekete, GitHub: Unknown):

---

What about a new field: long_description_format defaulting to reST for backward compatibility, but enabling us to have markdown formatted descriptions?

Lifting a README file into the manifest is a different feature, which can be solved with tooling as demonstrated by pbr.

[ewdurbin]

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):

---

@jaraco: Those are good points - I hadn't considered that there could be multiple distributions or even no distribution if the user registered but did not upload. Maybe the idea from @krisztianfekete is a better one?

[ewdurbin]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

I have no objection to supporting markdown or even moving toward preferring markdown. This poses a larger question of what should the metadata reflect to support this ability? Also, does the tool chain that renders documentation support markdown (namely Sphinx or Warehouse)? I'd recommend getting some feedback from pypa-dev on a change of this scope.

[ewdurbin]

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):

---

@hickford's patch would add Markdown to legacy PyPI. I'm thinking it could be tweaked a bit to look for a long_description_format property in the metadata and a similar change could be made to Warehouse (GitHub).

Then we could make a setuptools extension to add that property to package metadata (use an extension so we don't have to modify setuptools unless it proves to be wildly popular).

[ewdurbin]

Original comment by illume NA (Bitbucket: illume, GitHub: illume):

---

Added an issue to devpi https://bitbucket.org/hpk42/devpi/issue/193/support-markdown-for-readmes

[ewdurbin]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

I'm slighly -1 on adding another metadata field to specify the content type of an existing metadata field (i.e. long_description/long_description_format). If we did adopt that, I would recommend using MIME types (so something like "text/x-markdown; charset=utf-8" for markdown) to encourage alignment with other standards.

Rather than storing the metadata for the long description in another field, perhaps the format could be inferred by a heuristic and specified for cases where the heuristic fails using out-of-band features of the format (i.e. comments). For example, the following two snippets could represent a way to specify the content type of the document:

.. Content-Type: text/x-rst

[//]: # Content-Type: text/x-markdown

In this way, the long_description field becomes content-agnostic, but the tools now can infer the content type and users would have a means to force a content type.

This approach would not require any changes to tooling except for those that render package metadata.

[ewdurbin]

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):

---

@jaraco: Very interesting ideas to use MIME types and to attempt to deduce the format without adding metadata. @hickford: Want to update your PR?

[ewdurbin]

Original comment by Richard Jones (Bitbucket: r1chardj0n3s, GitHub: r1chardj0n3s):

---

Merged in hickford/pypi/markdown-readme (pull request #33)

Support Markdown for readmes (fixes #148)

[ewdurbin]

Original comment by Richard Jones (Bitbucket: r1chardj0n3s, GitHub: r1chardj0n3s):

---

Merged in hickford/pypi/markdown-readme (pull request #33)

Support Markdown for readmes (fixes #148)

[ewdurbin]

Original comment by Eugene Yunak (Bitbucket: yunake, GitHub: yunake):

---

@r1chardj0n3s, can you maybe reopen this issue since you reverted the merge?

[ewdurbin]

Original comment by Richard Jones (Bitbucket: r1chardj0n3s, GitHub: r1chardj0n3s):

---

Needs more thought on the security front.

[ewdurbin]

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

---

I'm of the opinion that a fallback on Markdown isn't very good.

• Some Markdown is also valid ReST, but the symbols mean different things, this means that it's not true that a fallback enables proper Markdown support, it only enables markdown support if that markdown happens to not be renderable as ReST. I think depending on the fact that a particular chunk of text doesn't happen to also be valid for ReST is a pretty poor UX.
• It doesn't scale well beyond Markdown. It might be said that why would anyone choose something else over Markdown and ReST but it seems to me that adopting a solution which makes it harder to add more things in the future is a bad long term solution.
• It relies on the fact that we're shoving rendered HTML into the database, however Warehouse does not use that HTML and I want to get rid of it eventually. Relying on shoving html in the database also means that things that consume the data from PyPI cannot determine what syntax to render a description with, without downloading the project and trying to determine how PyPI got the long_description and if it's from a file that ends with .md or .rst.

Overall I'd really like not to add more implementation defined stuff to the packaging toolchain. The processes for doing things should be documented in a PEP and someone should be able to replace each part of the toolchain by reading those PEPs and implementing them. It shouldn't require having to inspect the various implementations to figure things out.

[ewdurbin]

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):

---

@dstufft: Are you more in favor of new metadata like long_description_content_type with a MIME content type? Or something else?

[ewdurbin]

Original comment by digitalvapor (Bitbucket: digitalvapor, GitHub: digitalvapor):

---

Markdown support would be very appreciated! thanks

[ewdurbin]

Original comment by Will Mayner (Bitbucket: wmayner, GitHub: wmayner):

---

Big +1. This would be a huge boon to many!

[ewdurbin]

Original comment by Femto Trader (Bitbucket: FemtoTrader, GitHub: FemtoTrader):

---

+1 I'm also looking for Markdown support!

[ewdurbin]

Original comment by Marc Abramowitz (Bitbucket: msabramo, GitHub: msabramo):

---

For folks looking for Markdown support, have a look at https://pypi.python.org/pypi/setuptools-markdown which I uploaded to PyPI last night.

This is very, very experimental. Make sure to test thoroughly before you put a package on PyPI that uses it. E.g.: I wouldn't want for folks who install your packages to have to install setuptools-markdown and pandoc; I think it should only be required for the package maintainer. But please check that. You can file GitHub issues, but PRs are much preferred!

[ewdurbin]

Original comment by agh1973 (Bitbucket: agh1973, GitHub: Unknown):

---

+1 coming from http://stackoverflow.com/questions/4384796/readme-extension-for-python-projects I'll give the setuptools-markdown stuff a try.

[ewdurbin]

Original comment by Skuli Arnlaugsson (Bitbucket: arnlaugsson, GitHub: arnlaugsson):

---

This would be a lovely addition, Markdown is a much nicer format imo.

[ewdurbin]

Original comment by laike9m (Bitbucket: laike9m, GitHub: laike9m):

---

It's hard to imagine in 2015 Pypi still doesn't support markdown. Really need this!!

[ewdurbin]

Original comment by Fletcher Tomalty (Bitbucket: fletom, GitHub: fletom):

---

+1 Markdown is standard for READMEs and should be supported by PyPI.

[ewdurbin]

Original comment by Tim Santor (Bitbucket: tsantor, GitHub: tsantor):

---

Support Markdown for READMEs please!

[ewdurbin]

Original comment by Christoph Körner (Bitbucket: chaosmail, GitHub: chaosmail):

---

+1 i would love to see markdown being supported by pypi

[ewdurbin]

Original comment by Naoki INADA (Bitbucket: methane, GitHub: methane):

---

Use vote feature, not +1 comment.

[ewdurbin]

Original comment by anatoly techtonik (Bitbucket: techtonik, GitHub: techtonik):

---

This was an official position on any features a year ago:

Please stop submitting pull requests. Development on the existing codebase is halted except for
critical fixes or security issues. You are making extra work for people on this list and it will not be
tolerated. Please consider this your final warning.

http://comments.gmane.org/gmane.comp.python.distutils.devel/20041

[ewdurbin]

Original comment by Hickford (Bitbucket: hickford, GitHub: hickford):

---

That's unwelcoming :( I am not going to read any further or say any more, because quarrels between people who should be working together make me sad.

Happily I've been fortunate in open source, everyone has been most helpful to me.

Whether Markdown support is added to current PyPI or the new site (https://warehouse.python.org/ ), Markdown support is an important issue to many people. I'm sure we can work it out eventually.

[ewdurbin]

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

---

The largest blocker is figuring out how a setup.py should indicate that a project should use Markdown. This is best done on distutils-sig.

[ewdurbin]

Original comment by Hickford (Bitbucket: hickford, GitHub: hickford):

---

Hi Donald. You are set that readme and format should be specified explicitly in setup.py ? That would work. I would also be happy with GitHub-style autodiscovery (use readme.rst, readme.md, or readme.txt).

[ewdurbin]

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

---

There's no way to do auto discovery like Github does because we're not reading from a file (in some cases we are, but not generally), the description gets passed into the long_description field in the setup.py as a string and that gets sent as a POST argument when the file gets uploaded to PyPI. As a way of trying to find something PyPI will also, in the absence of that field, look for a README file in the archive but I personally feel that shouldn't be encouraged as it lives outside the metadata and instead we should focus on a reasonable way to represent it within the metadata.

[ewdurbin]

Original comment by anatoly techtonik (Bitbucket: techtonik, GitHub: techtonik):

---

setup.py must die, also because it is impossible to parse correctly in all cases. The README.txt or README,md is already in the .zip archive. No need to reference it. The problem with PyPI is that it tries to provide an online editor for the metadata and mixes metadata with data. The README.txt content is pasted into PKG-INFO file, but it is a hack.

The good way would be to render README file from archive if the Description field in PKG-INFO (long_description in setup.py) is absent. The context can be well detected based on extension.

[ewdurbin]

Original comment by Nicholas Chammas (Bitbucket: nickchammas, GitHub: Unknown):

---

For people looking for an interim solution, you can use pandoc to convert your long_description from Markdown to reStructuredText as part of setup.py.

Here's an example:

import setuptools
import pypandoc

long_description = pypandoc.convert('README.md', 'rst')

setuptools.setup(
    name='example',
    version=find_version('example.py'),
    long_description=long_description,
    ...
)

It's straightforward and should work for most projects. You can add a fallback like in the Stack Overflow answer I linked to so that people running python setup.py without pandoc installed will get the plain Markdown README.

[femtotrader]

Some people don't like this solution
See discussion here femtotrader/pandas_talib#8

Pinging also :
@OiNutter
@nchammas
@e3krisztian

[nchammas]

@femtotrader - If by "this solution" you are referring to my suggestion just above, I don't see how it is a problem.

The users in the discussion you linked to are complaining that they have to install Pandoc just to install your library. However, at the end of my comment I noted:

It's straightforward and should work for most projects. You can add a fallback like in the Stack Overflow answer I linked to so that people running python setup.py without pandoc installed will get the plain Markdown README.

In other words, only the maintainer who is uploading a package to PyPI will need Pandoc installed.

Perhaps I should have included that in my code sample, rather than just referring to it on Stack Overflow via a link. But to spell things out, if you have something like this in your setup.py:

try:
    import pypandoc
    long_description = pypandoc.convert('README.md', 'rst')
except ImportError:
    long_description = open('README.md').read()

And then you have Pandoc listed as a separate requirement only for maintainers, perhaps in a maintainer.pip file, then you get what you want. Users aren't burdened installing a package they don't need, and maintainers get their README in RST without having to learn that format just for PyPI.

That's how I do it in my project:

• setup.py
• maintainer.pip

It's not the solution we all ultimately want, but I think it's good enough for the time being.

[femtotrader]

I understand this correctly. Sorry for being not clear enough...
I really think that solution is to render correctly Markdown on server side.

[mec07]

I also think pypi should just support markdown.

[berkerpeksag]

Since this is a new feature and implementing it is not easy as calling markdown.markdown(long_description) in legacy PyPI and there is already an open issue in Warehouse's issue tracker, I'm going to close this one. Thank you all for the discussion!