Add sphinx-codeautolink extension to docs build

[TomNicholas]

I think that sphinx-codeautolink is different from sphinx.ext.linkcode...

• Closes Use sphinx-codeautolink in docs? #7010
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

[TomNicholas]

I don't understand why this failed - it works when I build the docs locally, but the RTD build doesn't give me a useful error message...

[keewis]

did you compare the versions? If it works locally, maybe you have a different version of sphinx or sphinx-codeautolink (or any other dependency?)

[dcherian]

There are a lot of these warnings:

/home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/7011/doc/whats-new.rst:1: WARNING: invalid syntax (<unknown>, line 6) in document "whats-new"
Parsed source in `ipython` block:
block source: In [66]: ds = xray.Dataset({"x": np.arange(1000)})
              
              In [67]: with xray.set_options(display_width=40):
                 ....:     print(ds)
                 ....: 
              <xarray.Dataset>
              Dimensions:  (x: 1000)
              Coordinates:
                * x        (x) int64 0 1 2 ... 998 999
              Data variables:
                  *empty*

[felix-hilden]

Hi, it looks like the print output is recognised as Python code and parsed. This shouldn't be happening since your what's-new correctly contains the Out [...]: prefixes which we use to ignore lines for parsing (although I couldn't find the specific case you presented). I'll test the multiline outputs and provide an update to you.

[felix-hilden]

Yep, a weird interaction between rST processing and the IPython transformer. This class of issues fixed in sphinx-codeautolink==0.12.1!

[dcherian]

@felix-hilden Looks like we still have the same issue with

sphinx-codeautolink       0.12.1                   pypi_0    pypi

[dcherian]

We now have a different error where a error message is being treated as code:

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/site-packages/sphinx_codeautolink/extension/block.py", line 204, in parse_source
    names = parse_names(modified_source, node)
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/site-packages/sphinx_codeautolink/parse.py", line 20, in parse_names
    tree = ast.parse(source)
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 638
    ValueError: arguments without labels along dimension 'x' cannot be aligned because they have different dimension size(s) {2} than the size of the aligned dimension labels: 3
                          ^
SyntaxError: invalid syntax

[felix-hilden]

Oh yikes, that should also be fixed now! I hope the parsing logic is finally sound.

[dcherian]

Ah now a different one

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/site-packages/sphinx_codeautolink/extension/block.py", line 213, in parse_source
    names = parse_names(modified_source, node)
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/site-packages/sphinx_codeautolink/parse.py", line 20, in parse_names
    tree = ast.parse(source)
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 638
    Out[125]: 
              ^
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/site-packages/sphinx_codeautolink/extension/__init__.py", line 43, in wrapper
    return func(*args, **kwargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/site-packages/sphinx_codeautolink/extension/__init__.py", line 135, in parse_blocks
    doctree.walkabout(visitor)
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/site-packages/docutils/nodes.py", line 227, in walkabout
    if child.walkabout(visitor):
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/site-packages/docutils/nodes.py", line 227, in walkabout
    if child.walkabout(visitor):
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/site-packages/docutils/nodes.py", line 227, in walkabout
    if child.walkabout(visitor):
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/site-packages/docutils/nodes.py", line 219, in walkabout
    visitor.dispatch_visit(self)
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/site-packages/docutils/nodes.py", line 2021, in dispatch_visit
    return method(node)
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/site-packages/sphinx_codeautolink/extension/block.py", line 166, in visit_literal_block
    return self.parse_source(node, node.get("language", None))
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/site-packages/sphinx_codeautolink/extension/block.py", line 215, in parse_source
    show_source = self._format_source_for_error(source, prefaces)
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/7011/lib/python3.9/site-packages/sphinx_codeautolink/extension/block.py", line 246, in _format_source_for_error
    guides[ix] = "block source:"
IndexError: list assignment index out of range

Thanks for be patient with us!

[felix-hilden]

That actually might be something for you to change. See the issue and in particular this IPython doc on the lexer implementation. It states that blocks not starting with a console prompt should be considered ordinary (IPython) code. So In/Out won't be expected. Now maybe comments are or should be an exception, but I suspect by moving the comment elsewhere you might get it to pass!

If you refuse, I'll have a look at IPython's implementation and how they handle comments 😄

[dcherian]

It's the second code block here and the source is

.. ipython:: python

    # only one argument has the 'x' coordinate
    arr[0] + 1
    # both arguments have the same 'x' coordinate
    arr[0] - arr[0]

It's rendering properly so it seems like this should just work.

So In/Out won't be expected.

The source block doesn't have In/Out but I guess the extension is receiving the output after IPython has executed the block?

I tried moving the comments around which seems to work, but we do this in a lot of places. Is it easy for you to handle it?

[felix-hilden]

Yep we'll definitely follow what IPython does, now available in sphinx-codeautolink==0.13.2!

[felix-hilden]

Wow, it's green! I still see some matching warnings though 🤔 I can have a closer look later if you care about them.

[dcherian]

It works! so on to the next issue :(

https://xray--7011.org.readthedocs.build/en/7011/user-guide/data-structures.html# has no links in the first cell.

Do you execute ipython cells that are "suppressed" (see https://github.com/pydata/xarray/blob/main/doc/user-guide/data-structures.rst). I think we "suppress" cells with imports at the top of all our doc pages.

Thanks again for all your work!

[felix-hilden]

I'm not familiar with how :suppress: works, other than the fact that the documentation states that warnings and errors are ignored. But I see that the whole block is invisible, so maybe it is executed only internally by IPython and all output is suppressed. If this is the case, consider adding something like:

.. autolink-preface::

    import numpy as np
    import pandas as pd
    import xarray as xr
    ...

or you can also set a global preface in conf.py!