[🐛 Bug]: Python API reference doc for 4.20.0 is missing some pages

[emanlove]

What happened?

The v4.20.0 Python API reference doc is missing some pages. Generally it reflects the docstrings for all the Python Modules. Currently though several modules are missing their pages (ie there is no link to their pages).

How can we reproduce the issue?

1. Goto https://www.selenium.dev/selenium/docs/api/py/api.html 

2. Scroll down to the modules of "WebDriver.firefox" or "Webdriver.safari", "Webdriver.webkitgtk" or "Webdriver.ie" as well as "selenium.webdriver.common.html5.application_cache" (near the top under "Webdriver.common").

3. Note several of the modules are not linked. These should be linked.

For example I should be able to click on "selenium.webdriver.firefox.webdriver" and be able to read about the "get_full_page_screenshot_as_file" function. Or as one can click on "selenium.webdriver.edge.webdriver" (just above the "Webdriver.firefox" section) and see the documentation for edge the same should happen for these other browsers.

Relevant log output

N/A

Operating System

Chrome

Selenium version

N/A

What are the browser(s) and version(s) where you see this issue?

Chrome, Firefox, Safari

What are the browser driver(s) and version(s) where you see this issue?

N/A

Are you using Selenium Grid?

N/A

[github-actions]

@emanlove, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[iampopovich]

@diemol hi , Am I correct that we need to update the list of classes in the file /selenium/py/docs/source/api.rst to create documentation for them?

i tried to build documentation locally and got some errors like

ons.rst, webdriver_webkitgtk/selenium.webdriver.webkitgtk.service.rst, webdriver_webkitgtk/selenium.webdriver.webkitgtk.webdriver.rst
WARNING: Failed to import selenium.webdriver.firefox.extension_connection.
Possible hints:

there is no class selenium.webdriver.firefox.extension_connection anymore
so shall we remove all the incorrect imports ?

[titusfortner]

I haven't updated api docs for 4.22,going to do that today. I'll get back to you with results.

[titusfortner]

Yes, something isn't working properly with Python documentation.

Executing tox -c py/tox.ini -e docs throws a bunch of warnings.
https://github.com/SeleniumHQ/selenium/actions/runs/9618006350/job/26530983788#step:7:13

Comparing 4.21 to 4.22 shows a number of files getting deleted that shouldn't be
This is what I'm seeing: 58fa379

If you have ideas for what is wrong and how to fix, please help. :)

[iampopovich]

I haven't fully solved the problem yet, but I have some thoughts on a solution:

1. I searched for similar errors and found that there are issues with Sphinx when adding the source code directory for documentation. I checked conf.py - if it worked before, it shouldn't break now. Example of a similar problem: Sphinx issue.

2. When updating the documentation, warnings are thrown such as the failure to import selenium.webdriver.firefox.extension_connection. There's no such class, but autodoc tries to import it. Removing the .rst file from the documentation directory will make the warning disappear.

3. I deleted all previously generated documentation and ran ./go py:docs. The .rst documents are generated, but I'm getting module import errors like websocket or trio, which suggests it's just an issue with my unconfigured environment. However, I don't receive any errors importing already generated documents.

4. I will open a PR with an updated list of classes in api.rst, leaving only those classes that exist in the project under py/selenium directory. This can be considered a small step towards improving the current documentation.

Based on all of the above, I suggest deleting all current documentation from docs and regenerating it with an up-to-date list of classes. Deleted classes should also be removed from the documentation in the docs directory in the future to avoid generating new warnings.

[emanlove]

@iampopovich I think you are close with the assumption on the list of classes within api.rst. I have long had the feeling that at some level these should be auto generated from the code, that is both docstrings as well as package directory structure. I think to a degree what is within api.rst is generated, possibly using sphinx-autogen [doc]. In addition as they should be auto generated the shouldn't be checked in but regenerated each time. Trying some things out around this thought ..

[emanlove]

Also the sphinx version (currently pinned at 1.8.2) really should be bumped up to something much more recent (latest is 7.3.7) along with Jinja2.