Support Python 3.13

[eoyoa]

When trying to setup a new dev environment in Python 3.13, you encounter a couple issues:

• [email protected] does not work with Python 3.12 or above.
  • asyncpg@^0.30.0 is compatible: see Test and build on Python 3.13 MagicStack/asyncpg#1188
• pandas@^1.0.0 is also not compatible with Python 3.13; we must update to at least ^2.2.3.
• asyncore, imghdr, and cgi were all deprecated in 3.12.
  • You can use python-deadlib as a temporary stand-in.

This PR updates these dependencies to their compatible versions to make rs compatible with Python 3.13, and also bumps versions of affected projects.

Note that currently, you are unable to test build this PR since rsmanage requires the newly bumped 7.7.4 version of runestone, which is not on PyPi until @bnmnetp approves this PR and updates the project. However, it should work in theory, once PyPi is updated.

Alternatively, you can decide to just not support Python 3.12 and above for now; this can be done very simply by limiting the Python version in pyproject.toml.

[bnmnetp]

Thanks for the PR @eoyoa

Just a cursory glance so far, I'll look more carefully in the next couple days. This is nice as 3.13 has been on my list for a while, but never gets far enough up the stack.

And... you very wisely left the web2py server alone. Good job.

[eoyoa]

Just as a heads up for your review @bnmnetp, I'm not sure how the python-deadlib replacements will fare on 3.10-3.12. I'm not at my computer so I can't test it.

However, one suitable alternative for the asyncore python-deadlib polyfill might be pyasyncore: it touts that

The module can be installed for earlier Python versions, but it will have no effect, and the standard library version of asyncore will be used in its place.

If the python-deadlib versions don't do the same thing, we'd have to find similar alternatives for imghdr and cgi.

[bnmnetp]

I'm on 3.11 so I'll see what happens.

I notice that you didn't update the base docker image, so the images are still using 3.11

Also, I'm not very good at all about updating the versions of the wheels. I haven't changed the wheel version for rsmanage since 2023. poetry build-project just keeps rebuilding 2.0.0 in the dist folder with whatever source is there.

[eoyoa]

I notice that you didn't update the base docker image, so the images are still using 3.11

@bnmnetp, I just committed a change that upgrades the affected projects' Dockerfiles. Again, I didn't change the ones that didn't need changing (like web2py).

However, you still can't test build this PR due to the aforementioned issue (that rsmanage needs this PR to be published before it can use [email protected]). I'm thinking it would be nice for a future QOL change to make poetry build-project use local versions of runestone when making the wheel when a certain dev flag is set or whatever, but it seems:

1. Annoyingly hard, and
2. Potentially out of scope for this PR.

Would like your input!

[bnmnetp]

Thanks.

I think if you do poetry env use /path/to/python3.11 from the main rs folder you can then interactively test a bunch of things using dstart to at least start a bunch of services.

alternatively it would not be hard for me to do a new release of the interactives so we can move forward.

[eoyoa]

alternatively it would not be hard for me to do a new release of the interactives so we can move forward.

Perhaps you could release under a pre-release version; i.e. something like 7.7.4-alpha? Then this (hopefully) wouldn't affect people who are currently using Runestone 7.7.3, and I could test build this PR.

[eoyoa]

Nevermind, I found a way to locally test this. I found some more dependencies to upgrade.

For future reference, I essentially just did a hack to locally use the built Runestone wheels by:

1. Pointing pyproject.toml to the local wheel file (i.e. runestone = { path = "./projects/interactives/dist/runestone-7.7.4.tar.gz" }),
2. Copying the interactives folder into the rsmanage Docker image,
3. Updating docker-compose.yml to include the interactives folder in the build context.

[bnmnetp]

Alright, this might be a bit messy for you but hopefully a rebase fix it.

I was going to release 7.7.4 today anyway with a bunch of other bug fixes. So I went ahead and bumped asyncpg as well. Which of course cascaded to bumping asyncpg everywhere.

[eoyoa]

Alright, @bnmnetp, seems to work!

For a summary of updates:

• Sphinx is upgraded to be at least version 6.2.0: see imghdr is deprecated and slated for removal in Python 3.13 sphinx-doc/sphinx#10440 (comment)
• pydal is upgraded to be at least version 20241204.1: see Added legacy-cgi as dependency in order to support Python 3.13 web2py/pydal#717 (comment)
• pandas is upgraded to be at least version 2.2.3: see the relevant changelog
• numpy is upgraded to be at least version 2.1.0: see the relevant changelog
• asyncpg is upgraded to be at least 0.30.0: see Test and build on Python 3.13 MagicStack/asyncpg#1188
• Replaced deprecated libraries with python-deadlib equivalents
• Removed unused asyncore imports in some Runestone components

If you'd like to test it on your end before merging, feel free; you'd have to do the previously mentioned workaround in order to get it to work locally though.

[bnmnetp]

We have a problem with pandas and sqlalchemy.

AttributeError: 'Engine' object has no attribute 'cursor'

You can recreate this by creating an simple assignment, answering a couple of questions in the assignment and then hitting the

http://localhost/assignment/instructor/assignment_summary/87

endpoint. The 87 there is the assignment ID. You can just get there from the assignment page.

If I remember correctly this is related to pandas version and the version of sqlalchemy we use.

[bnmnetp]

OK I've looked at this a bit more. We have been on sqlalchemy 1.4.x for a very long time. pandas >= 2.0 needs sqlalchemy >= 2.0. The move to sqlalchemy 2.0 looks like a lot of work and is outside the scope of this PR.

[eoyoa]

I see. Should we instead try to target Python 3.12 and just say that Python 3.13 is unsupported as of now?

[bnmnetp]

My quick experiments are indicating that this might not be as bad as it was when I first looked into it. I think they have done a lot more work to make the transition easier. The book server needs more testing but my initial build indicates that things are working.

I'm sure there are edge cases to find, but this might only be a days worth of work instead of weeks.

[bnmnetp]

OK, Things are fine with sqlalchemy and pandas now. Hopefully that will pave the way for you to rebase and we can be on 3.13

[eoyoa]

Ok, hopefully that fixed things; the rebase was a bit of a headache.

I will regenerate lock files and try this again.

[bnmnetp]

I merged this locally and pushed to main. Did not automatically close, so closing manually.

[bnmnetp]

Thanks a lot for your work on this!