Add license parsing code to the Registry pipeline

[thomashoneyman]

The Legacy Import script contains code that checks a package's various manifest files to verify their licenses. In the registry we only accept packages with valid SPDX licenses.

When publishing or updating a package in the Registry pipeline we should verify its license. In other words, we should:

1. Verify that the manifest contains a valid SPDX license string, AND
2. If the repository contains LICENSE files or other manifest types (bower.json, package.json), we should check them as well and ensure they are the same as the manifest SPDX license string

I'm not completely sold on step 2 there, but it seems like we ought to try and do some verification that the manifest license does match with the other license files in the repository, so that our pledge that "packages in the registry have valid SPDX licenses" is true, and that the SPDX license in the package is actually the one in the repository more generally.

...on the other hand, I really don't know the legal ramifications of all this. It brings to mind other questions:

1. What happens when the LICENSE file says BSD-3-Clause but the manifest says Apache-2.0? Should we reject this?
2. What happens if there are other manifest files in the repository -- for example, alternate backends like pureerl?

[mikesol]

Hey! Some opinions:

1. Yes, we should reject packages with a mismatch between the actual license and the license as reported in the manifest (with a courtesy note to the maintainer about the rejection).
2. We should include in the documentation a list of manifest files we will parse, where we will look, and steps that will entail rejection.

Writing this, I'm actually not sure where the CI pipeline lives in this repo. I see SPDX license parsing happening in the legacy bower import, but where is the CI pipeline that will be used for new submissions? Has it been started yet?

Thanks!

[thomashoneyman]

@mikesol the "registry pipeline" is the Registry.API module (it's a bit messy!)

[mikesol]

@thomashoneyman if no one has started on this yet, I can do it.

[f-f]

@mikesol the Manifest type requires a SPDX identifier, so we parse the license when reading the package manifest in (which includes the case when we are not converting from a Bowerfile)

[mikesol]

I can take this on if no one else has grabbed it, just let me know 👍

[thomashoneyman]

No one has grabbed it -- perhaps we can talk about it on the registry call today?

[thomashoneyman]

To summarize where we left off in the call, we're going to split this work into two pull requests.

1. The first PR will pull the license detection code and the license fixup code (rewriting typos, for example) out of the LegacyImport module and into a module in the main source tree. This is just a refactor and there should be no functional difference between the two.
2. The second PR will reuse this logic by adding it to the API pipeline for when a package is added or updated (as part of the runChecks process).

[thomashoneyman]

I've confirmed with our lawyer (informally) that the following are acceptable:

1. We can import legacy packages by inserting a new package manifest into them and then hold on to the tarball
2. We can assign packages a license when only one license type is discovered
3. We can assign packages a license, joining with SPDX's AND conjunctive, when multiple are discovered (ie. MIT AND APACHE-2.0). We don't have to do this, but he said this is the safe option and allows us to make the best claim for preserving the author's intent.
4. We can even rewrite package license identifiers that appear to be typos, so long as we feel comfortable arguing that we are representing author intent -- if someone writes APACHE 2 then we can rewrite it to APACHE-2.0, for example.

Basically, all the steps we take in our package importing process right now are a-ok. Furthermore:

• The gist of it is that everything is about what the author intended and our ability to argue that we are respecting it.
• Since this area of law is a bit ambiguous we're given some leeway by the legal system (but we have to ensure package authors have a clear, documented way to get in touch with us to have packages removed if they have an issue)
• If we do innocently mess something up and someone has a problem with it, then we just need to remove the package from the registry ASAP

As far as policies go, we need to have at least two of them. The first one needs to be for handling DMCA complaints. The second one needs to be a policy specifying what can be in a package, most critically that "what you state in your license field is what you are asserting this source code is being distributed under."

Finally, he recommended that -- even though it isn't strictly necessary -- when new packages are submitted to the registry in the future, we should reject packages that specify multiple licenses (MIT in package.json, BSD-3-Clause in purs.json) and ask them to join those licenses with a proper SPDX conjunctive like AND or OR.

We should also document this so that folks using our provided license scanning tooling (ie. Spago) know that we have verified the license field is accurate to the best of our ability.