bpo-46429: Merge all deepfrozen files into one

[kumaraditya303]

See discussion faster-cpython/ideas#218

https://bugs.python.org/issue46429

[gvanrossum]

In the issue you claimed a 1.2 Mb space savings. How did you measure that?

On MacOS I see ~0.2 Mb space savings (both size and ls show roughly this).

[kumaraditya303]

In the issue you claimed a 1.2 Mb space savings. How did you measure that?

Yes, it was 1.2 megabits not 1.2 megabytes, so around ~0.2 MB is correct.

[kumaraditya303]

@gvanrossum What's next step for this PR, I mean do you think this would be merged or was this just for experimentation ?

[gvanrossum]

I still need to review it. Maybe @tiran could also have a look?

[tiran]

If I understand the change correctly then we are no longer tracking dependencies correctly. Does a change to Lib/codecs.py trigger a rebuild of Python/frozen_modules/codecs.h and Python/deepfreeze/deepfreeze.c?

[kumaraditya303]

If I understand the change correctly then we are no longer tracking dependencies correctly. Does a change to Lib/codecs.py trigger a rebuild of Python/frozen_modules/codecs.h and Python/deepfreeze/deepfreeze.c?

Yes, fixed now.

[gvanrossum]

Basics look good, I have some nits to pick. You can undraft this.

[gvanrossum]

Hey Kumar, I wonder if (in a separate PR?) you'd be interested in implementing another idea to make the generated C code more understandable. For string constants, we could easily generate names that are more readable than zipimport_toplevel_consts_21_varnames_21, by taking the first N characters, replacing non-alphanumeric characters with '', and seeing if the result doesn't contain too many '' characters, and adding some hash to ensure uniqueness. So then that variable could become e.g. const_str_12aafd40_comment_size. We could even do similar things for tuples and numbers. What do you think? This wouldn't change the performance of the code, but it would make things a little more readable when idly browsing the generated file.

[gvanrossum]

Can you also open a bpo issue and add a news blurb?

[kumaraditya303]

Hey Kumar, I wonder if (in a separate PR?) you'd be interested in implementing another ...

@gvanrossum I am interested, will create another PR.

[gvanrossum]

Just one request, fold the long line in the Windows project file.

[gvanrossum]

Oh, there was one more request from the Faster CPython team. Could you run the PyPerformance benchmark (in PGO mode, on as quiet a machine as possible) before and after this to see if there's a net perf benefit? (We'd expect a small improvement due to the memory savings, but there could be a slight negative effect due to less memory locality.)

[kumaraditya303]

With patch:

0.011 ./python

Without patch:

0.011 ./python

[gvanrossum]

What benchmark did you run? Definitely not PyPerformance!

[kumaraditya303]

Nope https://github.com/python/cpython/blob/main/Tools/scripts/startuptime.py

I didn't see you asked for pyperformance (sorry).

[gvanrossum]

Yeah, I trust that the startup time is the same or better (actually the precision of that number is insufficient, we should print another digit), but we were wondering about effects on other benchmarks due to locality in deep-frozen modules. It's the only thing left to do here.

[kumaraditya303]

Yeah, I trust that the startup time is the same or better (actually the precision of that number is insufficient, we should print another digit), but we were wondering about effects on other benchmarks due to locality in deep-frozen modules. It's the only thing left to do here.

Perhaps, you can verify it if you have a benchmark setup ? I don't expect any significant change but less memory use.

[gvanrossum]

Okay, I'm just going to merge. We can watch speed.python.org tomorrow.