back up/restore conftest.py

[dimpase]

this will fix #39357

📝 Checklist

• [ x] The title is concise and informative.
• [x ] The description explains in detail what this PR is about.
• [x ] I have linked a relevant issue or discussion.
• I have created tests covering the changes.
• [ x] I have updated the documentation and checked the documentation preview.

⌛ Dependencies

[dimpase]

@vbraun - please have a look whether this doesn't break anything in your machinery

[dimpase]

@orlitzky - can you have a look?

[dimpase]

CI is failing in building docker images with

#23 66.22 cp: cannot stat 'configure_wrapper': No such file or directory

No idea what wrapper this is. @kwankyu - do you know?

[jhpalmieri]

CI is failing in building docker images with

#23 66.22 cp: cannot stat 'configure_wrapper': No such file or directory

No idea what wrapper this is. @kwankyu - do you know?

I may not be understanding this correctly, but isn't it just referring to the file, newly created by this PR, called "configure_wrapper"?

[dimpase]

Perhaps I've broken make configure - something that is used by the docker creation. I'm not sure this make target should be preserved - it's not standard, people can and should instead run ./bootstrap directly.

[kwankyu]

No idea what wrapper this is. @kwankyu - do you know?

No idea either.

Perhaps I've broken make configure - something that is used by the docker creation. I'm not sure this make target should be preserved - it's not standard, people can and should instead run ./bootstrap directly.

If something is broken, the standard way is to fix it (after understanding how it went broken), not to remove it.

[dimpase]

No idea what wrapper this is. @kwankyu - do you know?

No idea either.

Perhaps I've broken make configure - something that is used by the docker creation. I'm not sure this make target should be preserved - it's not standard, people can and should instead run ./bootstrap directly.

If something is broken, the standard way is to fix it (after understanding how it went broken), not to remove it.

Creating configure scripts using make is highly non-standard, and totally unneeded. I don't know why we're stuck with this for so long. It makes the build system even more complicated than necessary - because normally speaking makefiles are built by ./configure, not the other way around. So we're creating a chicken vs egg situation for no reason at all.

Anyhow, my latest attempt shows that it's not the issue. The issue is that the docker does not find configure_wrapper, and for the life of me I don't understand why. This file is a new file present in this branch, the branch we are testing!

https://github.com/sagemath/sage/actions/runs/12916470243/job/36020720010?pr=39363#step:13:517

[orlitzky]

I saw your post on the autoconf list and had been planning to try this. It's worse than you say:

1. The fact that ./configure will remove all conftest* files has been documented for many many years, so it was an interesting choice by pytest to name their file conftest.py.
2. Not only does autoconf remove conftest* files, it removes them with rm -rf, so we can't fix this with e.g. chmod a-w conftest.py.
3. Not only does autoconf rm -rf them, it does so in a trap handler to ensure that they are removed as the very last action taken by ./configure. So we can't e.g. move conftest.py to _conftest.py and then symlink it at the end of ./configure.

[dimpase]

I saw your post on the autoconf list and had been planning to try this. It's worse than you say:
I mentioned somewhere that I got a reply to my post on the autoconf list, too, basically confirming your observations.

it was an interesting choice by pytest to name their file conftest.py.
There aren't many python packages around built with autotools. (cpython itself was like this IIRC, but, well...)

[dimpase]

Are we in one of these situations when our docker-based CI won't work until this PR is merged in develop? @kwankyu ?

[kwankyu]

I have no idea what you are asking... What is "these situations"? I was not following the present issue and have no background understanding.

[kwankyu]

By the way, I do not object to changing "make configure" with "./bootstrap" for now if it helps. But removing the existing feature "make configure" as you seem to advocate is another matter.

[dimpase]

I have no idea what you are asking... What is "these situations"? I was not following the present issue and have no background understanding.

basically, running ./configure removes all the files matching conftest* in Sageroot, but pytest needs a file named conftest.py in there.
There is basically no way to modify ./configure to avoid this, so we need to run it in a wrapper which will restore conftest.py after ./configure is run.

to preserve our workflow, we rename configure to real_configure and install configure_wrapper as ./configure. This is done in ./bootstrap. All works just fine on a local machine.
But the part of CI which creates a docker container breaks down while running ./bootstrap,
as it can't find configure_wrapper - which is a new file introduced in this PR.
IMHO it's a bug in CI, but I don't see where.

[user202729]

Probably #39373 , let's see.

[github-actions]

Documentation preview for this PR (built with commit e08d5cc; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

[dimpase]

@user202729 - CI passes (modulo unrelated doctest failures on some platforms). Can you give this a positive review?

[jhpalmieri]

By the way, I do not object to changing "make configure" with "./bootstrap" for now if it helps. But removing the existing feature "make configure" as you seem to advocate is another matter.

I would not have any objections, either, but it is a publicly advertised way to install Sage — it's in the top-level README.md, for example — and so make configure would need to go through the usual deprecation process, we can't just delete it.

[dimpase]

By the way, I do not object to changing "make configure" with "./bootstrap" for now if it helps. But removing the existing feature "make configure" as you seem to advocate is another matter.

I would not have any objections, either, but it is a publicly advertised way to install Sage — it's in the top-level README.md, for example — and so make configure would need to go through the usual deprecation process, we can't just delete it.

This goes without saying, certainly. The thing is that make configure isn't even an explicit make target, it's using the generic target for spkgs. I can make an explicit target which just runs ./bootstrap.

[user202729]

haven't actually reviewed it carefully, just a few minor changes then should be good

[user202729]

does make configure still work? (if yes maybe leave it alone to be conservative, if not maybe we need to fix it)

[dimpase]

yes, make configure should still work. However, I don't know how, and trying to figure it out gives me a headache. OK, let's revert this change and see.

[jhpalmieri]

As an alternate approach, can we rename conftest.py to something_else.py and use pytest -p something_else? (See first comment at https://stackoverflow.com/questions/63236001/in-pytest-how-can-i-specify-a-different-conftest-py-file-to-use, and also https://discuss.python.org/t/pytests-conftest-py-naming-in-compatibility-with-autoconf/78085/11.) I would rather mess with pytest and leave configure unchanged, given the choice.

[user202729]

Thinking about it, maybe that can be automated with addopts in pytest.ini?

[dimpase]

everybody is welcome to provide a better fix. I'm a bit tired of pytest now.

[user202729]

To be fair, naming it something else might lead to some confusion. If this works we might as well merge it in until some other issue comes up.

[tobiasdiez]

pytest is already used quite a bit in testing; in CI this particular issue has an effect, one has to restore this file one way or another.

Since it's a standard package since a few months, pytest is actually run every time (also locally for devs) as part of sage -t.

Switching to pytest for doctesting of Python is not hard. Cython is harder, but also doable. @tobiasdiez - is this all right?

Doctesting is not hard but still requires a bit of work. Biggest problem, as to be expected, is that sage's doctest runner has a couple of special non-standard "features" that need to be migrated. Current status can be seen at #36981.

This is my question: how close are we to using pytest broadly? Is there any hurry to do these dances with the configure script now, or can it wait while people look into more options?

I hope to announce doctest support with pytest in a month or so (which would then be accompanied with a soft deprecation of sage -t). From my understanding of the configure interaction, it seems likely that time will not miraculously solve the issue.

[jhpalmieri]

I never wrote bash scripts which set atexit traps, and @vbraun requested this to be implemented. (so that the files are consistent even if ./configure has an abnormal exit)

If anyone here knows how to do this, please propose a fix. I am short on time for the coming few days.

In the worst case, if conftest.py is not restored, it can be recovered either using git, or by copying its backup copy.

ATM the wrapper doesn't do this recovery (the latter) automatically on the abnormal exit of ./configure. This has to be coded.

Do these two comments represent work that needs doing on this PR?

[dimpase]

Do these two comments represent work that needs doing on this PR?

well, yes - namely, the last command, mv bak_conftest.py conftest.py in configure_wrapper should be done in an atexit block, rather than just as it is - it would only be run now if the renamed ./configure completes more or less correctly, and doesn't destroy its parent process. I don't know what has to be done to the renamed ./configure so that it manages to exit so badly so that its parent does not execute the following command. Maybe it's all a red herring, and nothing needs to be done? @vbraun @jhpalmieri

[strogdon]

Unless I've done something stupid,

./sage -f "spkg"

will delete conftest.py.

[dimpase]

with this branch, or without?

without this branch, anything that triggers a run of ./configure will delete it.

[strogdon]

It was with the branch which is in 10.6.rc0

[user202729]

Huh, apparently a copy of this was included in #39655 . But different SHA-1.

Also… I notice the comment I mentioned about changing $@ to "$@" went unaddressed.

[dimpase]

there are lots of weird switches for sage, and also triggering runs of ./configure make targets.

[dimpase]

@vbraun - this had been merged, right? could you tag it properly, etc?

[user202729]

That was the problem I pointed out, this somehow get merged without any positive review. Seems nobody object so...

[dimpase]

@user202729 - please feel free to do a follow-up PR. I ran (real)_configure 100s of times, and a couple of times had to restore deleted file by git restore.

[dimpase]

It was with the branch which is in 10.6.rc0

OK, I can see this - our recursive make, self-modifying build system calls ./bootstrap from make build in some cases,
and this is when ./configure does not get restored.

[vbraun]

The configure_wrapper swallows exit codes, so configure failures are not seen by the shell. This is fixed in #40906