Synchronize 3.0.4, 3.1.1, and 3.2.0

[handrews]

At the moment, we are working towards releasing three releases, off three branches (v3.0.4-dev, v3.1.1-dev, v3.2.0-dev).

In order to release consistent documents (please let's not face-plant with our first releases in three years), we need to synchronize these three branches (and also apply the GitHub markdown link fixes that were made to main, but not yet to pending release branches).

This is made difficult by our policy of using separate files on each branch, making it impossible to manage such things with github in a normal way. But there are some less-normal ways git can help us as seen in the scripts linked below.

I previously said to a few people that I'd sort this out, but I am overloaded with other OpenAPI tasks and do not expect to be able to do this anytime soon. I did learn that we have two scripts designed to help out with this, although they aren't really robust in the face of the current rather muddled situation. But they help and could probably be improved:

• scripts/fwdport.sh
• scripts/fwdabort.sh

What I'd expect from PRs here is that:

• Each ported commit is still a separate commit (although merge commits need not be ported - it's fine to merge or rebase things differently)
• Commits should remain credited to their original authors, or use the GitHub-recognized Co-Author line (GitHub has documentation about this somewhere)
• It's fine to make one PR for many commits since they've been reviewed before, and GitHub allows examining each commit separately as needed

The main trick is figuring out which changes are applicable or not (for example, fixes to the reference to RFC 3339 in 3.0.4 do not need to be forward-ported, because from 3.1.0 onwards the OAS doesn't have that reference - it's handled by JSON Schema compatiblity). A related thing to watch out for is that sometimes the same change is applied to different branches with differently-titled commits.

We really need to fix this before we can move forward with finalizing these releases. Once we do this, we can decide whether we will continue parallel releases or not (in a recent Thursday call, we tentatively agreed that 3.0.4 would be the last 3.0.y barring a really compelling need; we did not make a decision on whether we would continue 3.1.y patch releases).

[lornajane]

Thanks! I think we're only forward-porting, is that right? Would it make sense to release 3.0.4, do the necessary forward ports for 3.1.1, and then roll 3.2.0 after that? It might make the comms a bit more difficult, but we'd be focusing our available efforts on one thing at a time, which has appeal!

[handrews]

@lornajane there are a lot (like ~10?) of minor fixes to 3.1.1 that would be really great to have in 3.0.4, but people didn't realize there would be a 3.0.4 and put them in the wrong branch/file. Personally, I think it looks awful if we ship 3.0.4 and 3.1.1 out of sync with each other. And it's not that hard to port the fixes. I almost had it sorted out and then carelessly deleted my workspace 🤦

I'm not trying to offload this because it's too much work, it's just that I have too many things right now - it should be easy enough for anyone who wants to help as you don't need specialized knowledge.

My preference would be:

• Backport the relevant things that need backporting
• Release 3.0.4 and 3.1.1 simultaneously as soon as we're happy enough with what's in them (and assume we can do a 3.1.2 if we want, even if we'd rather not do a 3.0.5)
• Make sure 3.1.2 and 3.2.0 are in-sync, and decide how to manage those

While users often don't update their openapi field from 3.0.x to 3.0.x+1, folks reading the spec docs will often read the latest just to get the clarifications. The patch release is supposed to be irrelevant to tools, after all. They're really just for humans (the nullable thing notwithstanding).

Releasing 3.0.4 and 3.1.1 that are consistent with each other will make it look like we know what we're doing and we still care about the majority of people who are on 3.0. If I don't have time to do that, I can't force anyone else to, but I hope we can at least try to recruit someone to do it. It's been so long since we released anything, and several folks I've spoken with have expressed frustration at the project running ahead when users are on 3.0 and only starting to move to 3.1.

I really think the PR value of putting out the best 3.0.4 that we can outweighs the irritation of fiddling with git (and maybe we should really consider @baywet 's suggestion of using git like it was meant to be used? If we did, we wouldn't even be debating this, it would just work and I'd have done it weeks ago. I confess to feeling considerable frustration over this. We are very much in our own way here.

[lornajane]

This is good context, I hadn't dug into the 3.1 fixes yet. I don't feel strongly about synchronising the releases but @handrews if you do, that's a perfectly good reason to do it!

I'm also very open to updating the process to something less confusing, for both contributors and maintainers! Actually, at release time is probably the right moment to make that sort of change. Do we already have a proposal for it?

[handrews]

I'm also very open to updating the process to something less confusing, for both contributors and maintainers! Actually, at release time is probably the right moment to make that sort of change. Do we already have a proposal for it?

I'd consider it to be part of the following, one of which has some comments specifically about the branching / git usage concerns:

• 3.x.y patch release approach #3528
  • 3.x.y patch release approach #3528 (comment)
  • 3.x.y patch release approach #3528 (comment)
• 3.x minor release approach #3529

While some of those branch management concerns are tied up in release proposals that are probably not the right scale for a volunteer project with a small number of regular contributors, there are relevant bits and pieces that I think might suit.

[handrews]

@lornajane

I don't feel strongly about synchronising the releases but @handrews if you do, that's a perfectly good reason to do it!

I appreciate that! This could also be something the reconstituted TSC decides. I can't (and would not attempt to) block the releases if we don't synchronize them. But I think that should be an intentional decision, and not just due to needing to find someone to do a not-that-large-but-somewhat-annoying task.

[handrews]

We also need to apply the changes from

• fix: GitHub changed markdown rendering for headings/anchors #3548

to the pending release branches, as it only fixed things on main.

[miqui]

@handrews - So, the effort is to apply #3548 to a 3.0.4 corresponding md file and

there are a lot (like ~10?) of minor fixes to 3.1.1 that would be really great to have in 3.0.4,

and these as well to 3.0.4? I need to see which of these do indeed apply. If you can guide me that would be great.

[handrews]

@miqui I'll tell you how far I got and how I was going about it:

1. I tagged where each release started in my fork so I could compare the starting points to the current branch status - you'll probably want to do this in your own fork in case I don't keep my branches in sync properly:
   • https://github.com/handrews/OpenAPI-Specification/releases/tag/v3.0.4-init
   • https://github.com/handrews/OpenAPI-Specification/releases/tag/v3.1.1-init
   • (I never got around to making a v3.2.0-init tag, but you get the idea)
2. I used GitHub to show a comparison history between each current branch and the corresponding init tag, all in my own fork
   • like making a PR, but I didn't actually make PRs...
   • ...which is why I can't link to those diffs (maybe saving them as local PRs – not PRs to the main repo – would be better)
3. I started manually going through the commits, and making lists of which ones were present in which releases (I do not seem to have saved the lists, sadly- no idea how that happened)
   • a lot of them are easy as they're obviously the same
   • a few are the same but don't have the same title and description, so you have to look at the changes
   • you'll see some weird patterns where commits were added to PRs by maintainers – I had not yet quite unraveled that, but I'm pretty sure you can look at what got grouped in a merge commit and treat that as one thing
   • I don't think we need to bother porting merge commits, but TBH I did not get far enough to be sure
   • keep in mind that some of the same commits got applied to both branches but in different order – I don't know how or why that happened
   • sometimes its easier to go look at the current files on each branch when you start getting lost in minute formatting changes to table rows - if they've ended up the same it doesn't matter how
   • usually it's pretty clear if something in 3.1.1 doesn't apply to 3.0.4 or vice versa - for example 3.0.4 defines more format values but in 3.1.1 the JSON Schema spec defines a lot of them; conversely anything involving contentEncoding and contentMediaType is 3.1.1-only
4. The fwdport.sh script tries to figure all of this out and do some interactive thing to let you pick, but I found it confusing
   • it seems to figure out the patches and put them in an .mbox file, so I'd started editing the script to just do that and stop so I could go through and choose which patches to apply manually
   • this is where I accidentally deleted my changes to the script and had to stop
   • another option would be to figure out how the script is applying patches across both branches and files so you can do them manually, or re-structure the script. I considered that but never really dug into it.

So I'd say give this a try and see if you can start figuring out which commits are definitely in both or definitely not in both, and then ask about the ones where you're not sure. If this isn't clear enough to make those PR-like comparison things, let me know and I'll try to figure out exactly what I did for that.

If you get far enough that you just need advice on specific commits, feel free to message me on Slack. If you need clarification on the steps above, it's probably better to ask here in case anyone else is following along.

[lornajane]

I took a different approach and looked at the diff between 3.1.0 and 3.1.1 and I think we can make some general assumptions about what should and should not be back-ported.

• Formatting differences: probably yes, although I could argue for just autoformatting both files with the same tool and ruleset (and perhaps adding it to CI). It's mostly indentation, and a small amount of formatting in yaml examples.
• Prose fixing: yes, we added/removed commas and things, let's repeat that.
• Clarifications: assemble a list and accept or deny case-by-case, probably in a TSC meeting.
• Actual changes: separate list, probably a series of individual issues to work through each one.

I also note that 3.1.0 contains a section "Mutual TLS Sample" which doesn't exist in 3.1.1, so we probably should fix that.

[lornajane]

Edited to add PR reference links for things that have been done/proposed

OK, digging through in more detail, this is a list of what I see in 3.1.1 that was not in 3.1.0. The idea is to check if anything we did for 3.1.1 should be back-ported to 3.0.4. Here's what I have, a couple of things should not move, some things are there already, and I've also appended a list of other actions.

Already applied

• clarification that Path Item server settings override the top level ones
• using "entry" rather than "root", reference to multi-document descriptions (there are potentially other follow-through changes possible here, but none that are already only applied to one release)
  • 3.1.1 Describe OpenAPI entry point in OpenAPI description docs #3387
  • 3.0.4 Backport improved OpenAPI specification intro paragraph from 3.1.1 #3697
• replace specific status codes with a reference to the IANA status code registry
  • 3.1.1 Fix: #2471. Clarify status code text. #2630
  • 3.0.4 Apply improved wording around status codes from 3.1.1 #3699
• update wording for the password string type
  • 3.1.1 Generalize description of password data type #3413
  • 3.0.4 Backport password field instruction improvements #3698
• note that security scheme scheme is case-insensitive
  • 3.1.1 Explicitly state that scheme is case-insensitive #2644
  • 3.0.4 Backport case insensitivity note for security schemes to 3.0.4 #3765
• a better description of deepObject as a style value
  • 3.1.1 Clarify deepObject applicability #2846
  • 3.0.4 Backport a clarification about deepObject to 3.0.4 #3766

Consider applying

• updated link to the rfc for oidc
  • 3.0.4 Fix incorrect OpenIDConnect discovery url #3607 (I think that's the right PR? – comment by @handrews) (that one replaces some related earlier changes, so it is the right one now - comment from @lornajane)
• oidc url description
  • 3.1.1 Clarify openIdConnectUrl #3630 #3718
• discriminator wording improvements _(@handrews will do this after finishining merging it with other changes in Clarify discriminator + oneOf/anyOf/allOf usage (3.1.1 modified port of #3822) #3846)
  • 3.1.1 3.1.1 discriminator improvements #2618

Don't backport: Path Items and References

These changes should remain as 3.1 only

• webhooks are a map of Path Item objects, but not Reference Objects.
• in the components section, pathItems can contain Path Item objects, but not Reference objects.
• it is deprecated to use $ref in a PathItem object alongside properties other than summary and description
• callback objects can contain Path Item objects, but not references.

Also don't backport

• explanation of using true/false schemas to make everything/nothing validate

Other actions

My feeling is that we might want to reconcile the items above and get each spec version to contain the correct content before we make any bigger diffs by reformatting.

• apply github heading fixes (they're on main, but need applying to next-release versions)
• 3.1.1-dev branch has a MutualTLS Sample in the 3.1.0 file but NOT the 3.1.1 file Backport improved OpenAPI specification intro paragraph from 3.1.1 #3697
• 3.0.4-dev branch has base64 encoding example for uploads in 3.0.3 but not in 3.0.4
  • The example is still there, but it's been corrected to be format: byte instead of format: base64 which was an error – @handrews
• Do we need to update the JSON schema link on line 2307 of 3.1.1?
• markdown formatting (and automate) as the last thing so it doesn't cloud the rest of the diff - captured as issue Add whitespace/formatting automation to the build process #3596

I have not considered forward porting anything that's in 3.0.4 that should also be in 3.1.1.