Tests: change-in-output with docutils r10151 (0.22-rc)

[jayaddison]

Describe the bug

Revision r10151 of docutils (aka commit b25db649a5d073d76424e6434e12cdba53ed57e8 in the project's git repository) introduces a change that alters the label identifiers produced in the output of some Sphinx builds.

My understanding is that this should not negatively impact users -- the output documentation is still valid, I believe -- but it does cause a failure in one of our LaTeX test cases, test_latex_labels.

In particular, the expected label identifiers for a figure element and a table element differ; previously they were id1 and id2 respectively - after the docutils revision r10151, they become id2 and id3.

How to Reproduce

$ git clone https://github.com/sphinx-doc/sphinx.git && cd sphinx
$ git checkout 546170754f3f2f96c0d12176b2d2fb5688ca75ab
$ python3 -m venv .venv && source .venv/bin/activate
$ pip install '.[test]'
$ pip install git+https://repo.or.cz/docutils.git@b25db649a5d073d76424e6434e12cdba53ed57e8#subdirectory=docutils
$ pytest tests/test_builders/test_build_latex.py -k test_latex_labels

Environment Information

Platform:              linux; (Linux-6.12.27-rt-amd64-x86_64-with-glibc2.41)
Python version:        3.13.3 (main, Apr 10 2025, 21:38:51) [GCC 14.2.0])
Python implementation: CPython
Sphinx version:        8.3.0+/546170754
Docutils version:      0.22rc3.dev
Jinja2 version:        3.1.6
Pygments version:      2.19.1

Sphinx extensions

N/A

Additional context

• Previously/recently discussed in Tests: unpick patch for Py314a7 no longer required with Py314b2 #13606 (an thread addressing a separate, current unit test failure).

[jayaddison]

An additional, related question: the id1 hyperlink target previously emitted for the figure is now placed into a LaTeX \phantomsection; is that something we should be concerned about?

(both @jfbu and I had noticed that in the #13606 discussion thread, but I neglected to document it during the initial writeup of this issue)

[AA-Turner]

I've merged #13610 to fix master, but will keep this open until we're sure everything is OK.

[jayaddison]

Thanks @AA-Turner - agreed, makes sense; I'm not completely confident about the approach taken in #13610.

[jfbu]

An additional, related question: the id1 hyperlink target previously emitted for the figure is now placed into a LaTeX \phantomsection; is that something we should be concerned about?

I think we should a bit, as the HTML output has somehow gotten rid of this extra target. Admittedly \phantomsection\label{\detokenize{index:id1}} following a (cross-) reference implemented by \hyperref mark-up does not cause changes in the output PDF layout. But if there are hundreds of them it will make the PDF bigger as well has auxiliary files and thus cause some slight slow-down in the build times. Perhaps the HTML builder has a transform getting rid of creating such targets if not needed?

Sorry now for utterly naive question but what would be the reST/Sphinx mark-up to actually create a reference (not artificially, i.e. not knowing in advance which of index:id1, index:id2, index.id3 is going to be used for a source file named index.rst) to these targets so that I could test (either in HTML or LaTeX) that they actually are correctly referred-to. As for now, the used reST source in this test-latex-labels does not generate usage of hyperlinks targeting the id1 associated with the "embedded hyperlink reference" (this was edited to not mention here id2 and id3 associated in this example with some figure and table directives). It is time to admit I have never been able to understand Docutils to start with in these matters, finding the vocabulary already confusing. For example this is the reason some improvements I would like to achieve with footnotes is indefinitely delayed, because I don't understand the Docutils code to start with (the Sphinx LaTeX footnote transform is nicely documented but to make a variation achieving other goals, it is needed to actually understand the Docutils layer, which I have tried from time to time and never could achieve).

[gmilde]

An additional, related question: the id1 hyperlink target previously emitted for the figure is now placed into a LaTeX \phantomsection; is that something we should be concerned about?

This is probably due to another recent commit:
r10152 changes the LaTeX writer to no longer ignore "ids" in inline elements. Inline elements with non-empty "ids" attribute will now generate a label and (unless the parent is a title or caption) a \phantomsection to allow cross references to this point.

Where exactly is the id1 placed? On the <figure>, <image>, or <caption> Doctree element or some other element inside these?

[jfbu]

Off-topic: basic.css makes it so that ¶ headerlinks are made visible on hover for table captions, but not so for figure labels.

Where exactly is the id1 placed? On the <figure>, <image>, or <caption> Doctree element or some other element inside these?

Both HTML and LaTeX have similar mark-up such as

<table class="docutils align-default" id="id2">
<caption><span class="caption-text">table caption</span><a class="headerlink" href="#id2" title="Link to this table">¶</a></caption>

But now regarding the source

Embedded standalone hyperlink reference: `subsection <section1_>`_.

it renders in LaTeX as

Embedded standalone hyperlink reference: {\hyperref[\detokenize{index:section1}]{\sphinxcrossref{subsection}}}\phantomsection\label{\detokenize{index:id1}}.

(and other idN's are shifted) whereas in HTML it still renders as

<p>Embedded standalone hyperlink reference: <a class="reference internal" href="#section1">subsection</a>.</p>

and the other idN's are not modified.

[jfbu]

This is probably due to another recent commit:
r10152 c

We saw the modification in LaTeX output already at revision 10151.

[gmilde]

I think we should a bit, as the HTML output has somehow gotten rid of this extra target. Admittedly \phantomsection\label{\detokenize{index:id1}} following a (cross-) reference implemented by \hyperref mark-up does not cause changes in the output PDF layout. But if there are hundreds of them it will make the PDF bigger as well has auxiliary files and thus cause some slight slow-down in the build times. Perhaps the HTML builder has a transform getting rid of creating such targets if not needed?

In HTML, all elements can act as target, if they have an ID. In LaTeX the cost for potential targets is higher: e.g. <p id="id1">Paragraph 1.</p> corresponds to

\phantomsection\label{id1}
Paragraph 1.

Has HTML actually dropped the identifier id1?

[jfbu]

This is probably due to another recent commit:
r10152 changes the LaTeX writer to no longer ignore "ids" in inline elements. Inline elements with non-empty "ids" attribute will now generate a label and (unless the parent is a title or caption) a \phantomsection to allow cross references to this point.

Are you referring to the Docutils LaTeX writer? Sphinx has its own so the \phantomsection is generated there due to the doctree having some label.

[jfbu]

Has HTML actually dropped the identifier id1?

Yes it has.

[gmilde]

But now regarding the source
...

This makes the issue a lot clearer: Let's assume there is a "section1" somewhere,
then

`subsection <section1_>`_.

is translated to the Doctree (in "pseudoxml" representation):

            <reference name="subsection" refid="section1">
                subsection
            <target ids="subsection" names="subsection" refid="section1">

The target is a an indirect target, it helps resolving references by mapping refname="subsection" to refid="section1"
(see docutils.transforms.references.InternalTargets).

The Docutils writers skip indirect targets (i.e. <target> elements with:

    def visit_target(self, node) -> None:
        # Skip indirect targets:
        if ('refuri' in node        # external hyperlink
            or 'refid' in node      # resolved internal link
            or 'refname' in node):  # unresolved internal link
            return
        ...

It seems the Sphinx LaTeX writer has a different test in its
visit_target() method that fails to correctly filter indirect targets
and the addition of an ids attribute triggered the additional

{\sphinxcrossref{subsection}}}\phantomsection\label{\detokenize{index:id1}}.

It is, however, unclear why it uses an auto-id ("id1") instead of the existing ID ("subsection").
The label is not required (as links to "subsection" are resolved by a transfrom)
and with the auto-id it is useless and confusing.