v2.4.0rc1

[abhinavsingh]

PR to merge develop into master. This is in preparation for v2.4.0rc1 release and our first automated release via GitHub workflows. Here is the planned steps for release:

1. Merge this PR into master branch (no squash merge)
2. Publish v2.4.0rc1 release candidate on PyPi using GitHub workflows
   • In case of problems, repeat above steps.
   • Once after a good rcN release:
   • During this workflow, test.pypi.org can be used to install dev builds, which will resemble the stable builds, because we'll likely freeze new merge into develop until a stable release is made.
3. Publish next stable v2.4.0 release on PyPi using GitHub workflows

[abhinavsingh]

Back in the day, projects didn't have a CI on each change but now you don't do manual testing of every aspect of the project before releasing, you have a confirmation that tests pass at all points in time. The only reason to have multiple branches today is if you want to maintain several version streams simultaneously. Like if you had 2.4.x, 2.3.x, and 2.2.x with their own branches, then you'd do the development and add the features to the main branch while only cherry-picking/backporting the bugfixes (including security) to the stable branches; in such a scenario you'd publish patch-releases from those branches occasionally and you'd create a new branch when making a release with new features + deprecate the oldest stable. But this requires more effort and is only truly sustainable if the project is sponsored and there are resources to do all of these things. Without that, small projects don't tend to go through all of the trouble because the manual release management may be thankless at times. I personally try to automate all of the important bits where possible and where this process is present: I wrote a backporting bot that helps with the processes in aiohttp and many projects in the Ansible ecosystem, for example; plus many projects have changelog fragment based release notes managements (I have a bot and a Sphinx integration for this as well). But if you look even at the fundamental projects in the ecosystem (like setuptools or pip, or most of the web frameworks), all of them only have one main branch and the only supported release is the latest version.
This is why I'm trying to understand why you need the workflow to be this complex.

Thank you. Much of it totally makes sense and it applies for all libraries in general.

Reason why I want to keep a separate release branch is simple.

• There are users who use proxy.py in production to serve real traffic. So they use it as a server, sometimes directly facing the user
• During development, a lot of things can happen (as you may have observed in my PRs), that our tests simply cannot catch just yet.
• Example, is proxy.py leaking any socket file descriptors? You can see I added diff (check) within benchmark.sh for starters. Afaik, due to refactoring, our current --threaded implementation leaves connection in TIME_WAIT state for sometime. OS finally closes them after 2-10s. IIRC, this was not the case in v2.3.1. I am not too worried about it much, as OS does close the descriptors eventually, and anyways --threaded will get deprecated eventually. But my point is, such cases are not covered in our tests currently.
• FWIW, our current test_e2e if run locally, simply leaves 16 hanging processes on my Macbook, because proxy.py isn't shutdown properly. This is not a problem in test_e2e technically because ideally proxy.py must be able to handle signals and shutdown. Then we can send it a signal for clean shutdown.
• Similarly, there can be other issues, which at-least for now, are not handled/covered by our tests.

Development branch always work well for embedded test case scenario. There developers are not worried about open file descriptors / memory leak and other aspects of proxy.py which are only surfaced after 1 day-week of runtime. For that specific reason, we have deployed development branch in one of our clusters. Soon, I'll also add a link to README.md where visitors can try proxy.py immediately by visiting a Url and grabbing ephemeral auth parameters (hosted by @jaxl-innovations-private-limited)

In face of lack of test coverage for such system level issues, I am reluctant to keep pushing our new version every day. May be once we have enough coverage for such aspects, we can start to rollout every day. Such tests will consume too much CPU/RAM and hence I haven't added them to GHA. Most likely will channel them via Jaxl hosted runners in the cloud.

[webknjaz]

• There are users who use proxy.py in production to serve real traffic. So they use it as a server, sometimes directly facing the user

Sounds like instead of merging things back and forth you could just fast-forward master to a tag (or state of devel) that you consider stable. Have you considered this? Or are you adding extra commits/hotfixes on master that don't go to devel first? If yes, then maybe the model described with stable stream branches I mentioned above would fit the requirements better. You could have a branch called stable/2.4 (or release/2.4) with all the features present at the moment of releasing v2.4.0 and you'd add fixes that are not features there (possibly they'd get into devel and get backported, for example).