After a SphinxError incremental behaviour does not work

[twodrops]

Describe the bug

Whenever a "SphinxError" is raised, the next build is always a full build without using the pickles. This causes usability issues in large projects whenever an error is created in a file and fixed.

The expectation is that only the changed file (where the error is fixed) is rebuilt.

How to Reproduce

1. Do a sphinx-build
2. Do a second sphinx-build without making any changes (to make sure that incremental build works)
3. Introduce any error in a single file which raises a SphinxError (not a SphinxWarning) and do a sphinx-build
4. Fix the error in the single file and do a sphinx-build again

All source files are marked as changed and read in again.

Environment Information

Sphinx: 5.3.0

Sphinx extensions

No response

Additional context

No response