sphinx 6.2 (and the 6.1's) create enormous files that did not happen in 5.3.0 (single-process mode)

[zzzeek]

Describe the bug

sphinx 6.2 creates gigantic doctree files. as before, for me to move to this new version means I have to upgrade hardware. Can I get a rationale for this change?

Building SQLAlchemy docs with 6.2, small machine with 6g of ram runs out of space at:

(.venv) [classic@ln6 build]$ ls -l output/doctrees/
total 220496
drwxrwxr-x. 2 classic classic     4096 Apr 24 09:08 changelog
-rw-rw-r--. 1 classic classic 21185932 Apr 24 09:08 contents.doctree
-rw-rw-r--. 1 classic classic 21280576 Apr 24 09:08 copyright.doctree
drwxrwxr-x. 2 classic classic     4096 Apr 24 09:12 core
drwxrwxr-x. 2 classic classic     4096 Apr 24 09:13 dialects
-rw-rw-r--. 1 classic classic 45248036 Apr 24 09:14 errors.doctree
drwxrwxr-x. 2 classic classic     4096 Apr 24 09:15 faq
-rw-rw-r--. 1 classic classic 45893107 Apr 24 09:15 glossary.doctree
-rw-rw-r--. 1 classic classic 46006969 Apr 24 09:16 index.doctree
-rw-rw-r--. 1 classic classic 46121841 Apr 24 09:16 intro.doctree
drwxrwxr-x. 4 classic classic     4096 Apr 24 09:48 orm

5.3.0, the same directory as its building looks like:

[classic@ln6 output]$ ls -la doctrees/
total 21808
drwxrwxr-x. 8 classic classic    4096 Apr 24 10:25 .
drwxrwxr-x. 4 classic classic    4096 Apr 24 10:18 ..
drwxrwxr-x. 2 classic classic    4096 Apr 24 10:20 changelog
-rw-rw-r--. 1 classic classic  837532 Apr 24 10:20 contents.doctree
-rw-rw-r--. 1 classic classic  841317 Apr 24 10:20 copyright.doctree
drwxrwxr-x. 2 classic classic    4096 Apr 24 10:22 core
drwxrwxr-x. 2 classic classic    4096 Apr 24 10:23 dialects
-rw-rw-r--. 1 classic classic 6109180 Apr 24 10:25 environment.pickle
-rw-rw-r--. 1 classic classic 3751182 Apr 24 10:23 errors.doctree
drwxrwxr-x. 2 classic classic    4096 Apr 24 10:23 faq
-rw-rw-r--. 1 classic classic 3732915 Apr 24 10:23 glossary.doctree
-rw-rw-r--. 1 classic classic 3486487 Apr 24 10:23 index.doctree
-rw-rw-r--. 1 classic classic 3524307 Apr 24 10:23 intro.doctree
drwxrwxr-x. 4 classic classic    4096 Apr 24 10:25 orm
drwxrwxr-x. 2 classic classic    4096 Apr 24 10:25 tutorial

How to Reproduce

git clone https://github.com/sqlalchemy/sqlalchemy
cd doc/build
python -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/pip install sphinx==6.2.0
source .venv/bin/activate
make html

Environment Information

]$ sphinx-build --bug-report
Please paste all output below into the bug report template



Platform:              linux; (Linux-5.3.11-100.fc29.x86_64-x86_64-with-glibc2.28)
Python version:        3.10.8 (main, Oct 20 2022, 16:09:08) [GCC 8.3.1 20190223 (Red Hat 8.3.1-2)])
Python implementation: CPython
Sphinx version:        6.2.0
Docutils version:      0.19
Jinja2 version:        3.1.2
Pygments version:      2.15.1

Sphinx extensions

No response

Additional context

No response

[zzzeek]

looks like it makes a huge doctree for every file individually, because they all have a TOC sidebar

[root@143-42-122-85 build]# find output/doctrees/ -name "*.doctree" -exec ls -lah {} \;
-rw-r--r--. 1 classic classic 44M Apr 24 16:25 output/doctrees/faq/installation.doctree
-rw-r--r--. 1 classic classic 44M Apr 24 16:25 output/doctrees/faq/sqlexpressions.doctree
-rw-r--r--. 1 classic classic 44M Apr 24 16:25 output/doctrees/faq/connections.doctree
-rw-r--r--. 1 classic classic 44M Apr 24 16:25 output/doctrees/faq/sessions.doctree
-rw-r--r--. 1 classic classic 44M Apr 24 16:25 output/doctrees/faq/index.doctree
-rw-r--r--. 1 classic classic 44M Apr 24 16:25 output/doctrees/faq/performance.doctree
-rw-r--r--. 1 classic classic 44M Apr 24 16:25 output/doctrees/faq/ormconfiguration.doctree
-rw-r--r--. 1 classic classic 44M Apr 24 16:25 output/doctrees/faq/thirdparty.doctree
-rw-r--r--. 1 classic classic 44M Apr 24 16:25 output/doctrees/faq/metadata_schema.doctree
-rw-r--r--. 1 classic classic 43M Apr 24 16:25 output/doctrees/dialects/postgresql.doctree
-rw-r--r--. 1 classic classic 40M Apr 24 16:25 output/doctrees/dialects/mssql.doctree
-rw-r--r--. 1 classic classic 43M Apr 24 16:25 output/doctrees/dialects/sqlite.doctree
-rw-r--r--. 1 classic classic 41M Apr 24 16:25 output/doctrees/dialects/mysql.doctree
-rw-r--r--. 1 classic classic 40M Apr 24 16:25 output/doctrees/dialects/index.doctree
-rw-r--r--. 1 classic classic 41M Apr 24 16:25 output/doctrees/dialects/oracle.doctree
-rw-r--r--. 1 classic classic 44M Apr 24 16:25 output/doctrees/glossary.doctree
...

was it always this way and doctree files are just bigger now ? obviously im just going to expand my disk size since it's too small anyway ...seeing if 6.2 can actually build on a linode 2g with nothing else on it at least

[AA-Turner]

Are you constrained by disk space? Could you delete the doctrees directory when building has finished?

I'll look into if there's any specific issue I can pin down.

A

[zzzeek]

turns out we can't get SQLA's docs to build with 6.2 memory-wise on any linode 2g or less, even though it's much better than 6.1, so we will be looking to offload our doc builds into github actions

[zzzeek]

the 6x series generally seems to have much larger search indexes

[AA-Turner]

I don't know what the resource limits of a 'linode 2g' are, do you have a link please?

Also, do you mean build-time RAM or storage disk space by memory?

A

[AA-Turner]

A thought: does setting toc_object_entries to False help the issue?

https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-toc_object_entries

[zzzeek]

I don't know what the resource limits of a 'linode 2g' are, do you have a link please?

a linode 2G is a shared CPU (virt) instance, 2G of ram, 50G storage 1 CPU

https://www.linode.com/pricing/#compute-shared

im not sure if I tried it on a 4g machine or not (4g of ram)

Also, do you mean build-time RAM or storage disk space by memory?

at this point both....it looks like the doctree files get shrunk down after it builds, but it peaks during the build. I can add filespace. but the OOM killer is hitting it too, which is a bigger problem.

re: toc_object_entries I will try that, though 5.3.0 has this feature also ( see https://alembic.sqlalchemy.org/en/latest/ops.html for example) and we dont run out of memory with that.

we're going to work around all this in any case but just the growth in memory use seems significant. I dont know to what degree the structures are related to how many targets we have, like we have lots of classes and functions, or how many things point to those classes/functions etc. , we have lots of that too , every function signature etc.

looks like it creates intermediary doctree files that are just as big with that setting

[AA-Turner]

I've tried replicating the error on Sphinx 6.2.0 (Ubuntu 22.04 via Windows Subsytem for Linux):

(.venv) adam:/mnt/i/Development/sqlalchemy/doc/build$ ls -Al output-6.2.0/doctrees
total 6872
drwxrwxrwx 1 adam adam     512 Apr 25 12:40 changelog
-rwxrwxrwx 1 adam adam   73262 Apr 25 12:40 contents.doctree
-rwxrwxrwx 1 adam adam   74564 Apr 25 12:40 copyright.doctree
drwxrwxrwx 1 adam adam     512 Apr 25 12:40 core
drwxrwxrwx 1 adam adam     512 Apr 25 12:40 dialects
-rwxrwxrwx 1 adam adam 6115962 Apr 25 12:40 environment.pickle
-rwxrwxrwx 1 adam adam  324323 Apr 25 12:40 errors.doctree
drwxrwxrwx 1 adam adam     512 Apr 25 12:40 faq
-rwxrwxrwx 1 adam adam  346411 Apr 25 12:40 glossary.doctree
-rwxrwxrwx 1 adam adam   47115 Apr 25 12:40 index.doctree
-rwxrwxrwx 1 adam adam   40920 Apr 25 12:40 intro.doctree
drwxrwxrwx 1 adam adam     512 Apr 25 12:40 orm
drwxrwxrwx 1 adam adam     512 Apr 25 12:40 tutorial

and

(.venv) adam:/mnt/i/Development/sqlalchemy/doc/build$ find output-6.2.0/doctrees/ -name "*.doctree" -exec ls -lah {} \;
[snip ...]
-rwxrwxrwx 1 adam adam 38K Apr 25 12:40 output-6.2.0/doctrees/dialects/index.doctree
-rwxrwxrwx 1 adam adam 443K Apr 25 12:40 output-6.2.0/doctrees/dialects/mssql.doctree
-rwxrwxrwx 1 adam adam 619K Apr 25 12:40 output-6.2.0/doctrees/dialects/mysql.doctree
-rwxrwxrwx 1 adam adam 285K Apr 25 12:40 output-6.2.0/doctrees/dialects/oracle.doctree
-rwxrwxrwx 1 adam adam 1.3M Apr 25 12:40 output-6.2.0/doctrees/dialects/postgresql.doctree
-rwxrwxrwx 1 adam adam 347K Apr 25 12:40 output-6.2.0/doctrees/dialects/sqlite.doctree
-rwxrwxrwx 1 adam adam 317K Apr 25 12:40 output-6.2.0/doctrees/errors.doctree
-rwxrwxrwx 1 adam adam 94K Apr 25 12:40 output-6.2.0/doctrees/faq/connections.doctree
-rwxrwxrwx 1 adam adam 3.7K Apr 25 12:40 output-6.2.0/doctrees/faq/index.doctree
-rwxrwxrwx 1 adam adam 7.8K Apr 25 12:40 output-6.2.0/doctrees/faq/installation.doctree
-rwxrwxrwx 1 adam adam 20K Apr 25 12:40 output-6.2.0/doctrees/faq/metadata_schema.doctree
-rwxrwxrwx 1 adam adam 58K Apr 25 12:40 output-6.2.0/doctrees/faq/ormconfiguration.doctree
-rwxrwxrwx 1 adam adam 78K Apr 25 12:40 output-6.2.0/doctrees/faq/performance.doctree
-rwxrwxrwx 1 adam adam 157K Apr 25 12:40 output-6.2.0/doctrees/faq/sessions.doctree
-rwxrwxrwx 1 adam adam 132K Apr 25 12:40 output-6.2.0/doctrees/faq/sqlexpressions.doctree
-rwxrwxrwx 1 adam adam 62K Apr 25 12:40 output-6.2.0/doctrees/faq/thirdparty.doctree
-rwxrwxrwx 1 adam adam 339K Apr 25 12:40 output-6.2.0/doctrees/glossary.doctree
[... snip]

I don't get the super-inflated file sizes you do -- could you do pip freeze please? I get:

(.venv) adam:/mnt/i/Development/sqlalchemy/doc/build$ pip freeze
alabaster==0.7.13
Babel==2.12.1
certifi==2022.12.7
changelog @ git+https://github.com/sqlalchemyorg/changelog.git@66e3e124386550e1d6c5c1b3a7a88b9d3dc2046a
charset-normalizer==3.1.0
colorama==0.4.6
docutils==0.19
idna==3.4
imagesize==1.4.1
Jinja2==3.1.2
libsass==0.22.0
livereload==2.6.3
Mako==1.2.4
MarkupSafe==2.1.2
packaging==23.1
Pygments==2.15.1
requests==2.28.2
six==1.16.0
snowballstemmer==2.2.0
Sphinx==6.2.0
sphinx-autobuild==2021.3.14
sphinx-copybutton==0.5.1
sphinx-paramlinks @ git+https://github.com/sqlalchemyorg/sphinx-paramlinks.git@c43e8f0c0356c5294cbe6025f87a0310bfdbef89
sphinxcontrib-applehelp==1.0.4
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.1
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
tornado==6.3.1
typing_extensions==4.5.0
urllib3==1.26.15
zzzeeksphinx @ git+https://github.com/sqlalchemyorg/zzzeeksphinx.git@c82613b1c491846bf642e178c34bbccc27740134

A

[AA-Turner]

For another point of comparison, I get a total disk usage for the entire output directory of ~100 MB:

(.venv) adam:/mnt/i/Development/sqlalchemy/doc/build$ du -sb output-6.2.0
107597374       output-6.2.0
(.venv) adam:/mnt/i/Development/sqlalchemy/doc/build$ du -sh output-6.2.0
104M    output-6.2.0
(.venv) adam:/mnt/i/Development/sqlalchemy/doc/build$ du -sh output-6.2.0/*
69M     output-6.2.0/doctrees
36M     output-6.2.0/html

[zzzeek]

first off, thanks very much for sticking around and trying to investigate this, I appreciate that very much.

The problem seems to be specific to running with just one process. Interestingly, when we build with one process, all of the huge files stay in place - a total of 7.4G for SQLAlchemy's build:

$ ls -la output/doctrees/
total 299676
drwxr-xr-x. 8 classic classic     4096 Apr 25 08:58 .
drwxr-xr-x. 4 classic classic     4096 Apr 25 08:53 ..
drwxr-xr-x. 2 classic classic     4096 Apr 25 08:54 changelog
-rw-r--r--. 1 classic classic 21185908 Apr 25 08:54 contents.doctree
-rw-r--r--. 1 classic classic 21280552 Apr 25 08:54 copyright.doctree
drwxr-xr-x. 2 classic classic     4096 Apr 25 08:55 core
drwxr-xr-x. 2 classic classic     4096 Apr 25 08:55 dialects
-rw-r--r--. 1 classic classic 81080390 Apr 25 08:59 environment.pickle
-rw-r--r--. 1 classic classic 45248012 Apr 25 08:55 errors.doctree
drwxr-xr-x. 2 classic classic     4096 Apr 25 08:55 faq
-rw-r--r--. 1 classic classic 45893083 Apr 25 08:55 glossary.doctree
-rw-r--r--. 1 classic classic 46006945 Apr 25 08:55 index.doctree
-rw-r--r--. 1 classic classic 46121817 Apr 25 08:55 intro.doctree
drwxr-xr-x. 4 classic classic     4096 Apr 25 08:58 orm
drwxr-xr-x. 2 classic classic     4096 Apr 25 08:58 tutorial

if you like I can put a tar of the above build on a dropbox somewhere.

when I build with -j auto, just using three processes, the output folder stays level at 53M total.

To build with just one process, run without the "-j" option:

$ make html SPHINXOPTS="-T"

to view growth, run a running total of filesize in a different window:

$ while [ 1 ]; do du -hs output; sleep 1; done

by the time it's at about 41% reading files, the "du" window will show:

2.1G	output
2.1G	output
2.1G	output
2.2G	output
2.2G	output
2.2G	output
2.2G	output
2.3G	output
 2.3G	output
2.3G	output
2.3G	output
...

up until about 7.4 G where it stays when the build completes. It's also using a ton of RAM when just one process:

I've seen other issues that imply sphinx uses some different mode of operation with just one process. So there seems like there is definitely an ongoing bug here.

[zzzeek]

pip freeze:

$ pip freeze
alabaster==0.7.13
Babel==2.12.1
certifi==2022.12.7
changelog==0.5.9
charset-normalizer==3.1.0
colorama==0.4.6
docutils==0.19
idna==3.4
imagesize==1.4.1
Jinja2==3.1.2
libsass==0.22.0
livereload==2.6.3
Mako==1.2.4
MarkupSafe==2.1.2
packaging==23.1
Pygments==2.15.1
requests==2.28.2
six==1.16.0
snowballstemmer==2.2.0
Sphinx==6.2.0
sphinx-autobuild==2021.3.14
sphinx-copybutton==0.5.1
sphinx-paramlinks @ git+https://github.com/sqlalchemyorg/sphinx-paramlinks.git@c43e8f0c0356c5294cbe6025f87a0310bfdbef89
sphinxcontrib-applehelp==1.0.4
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.1
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
tornado==6.3.1
typing_extensions==4.5.0
urllib3==1.26.15
zzzeeksphinx @ git+https://github.com/sqlalchemyorg/zzzeeksphinx.git@c82613b1c491846bf642e178c34bbccc27740134

[AA-Turner]

The problem seems to be specific to running with just one process.

I've confirmed this, I'm looking at it now.

A

[zzzeek]

see #11124 , this is the issue that I saw