Proposal for packages.json file

Author: lrhn

accepted/future-releases/language-versioning/package-config-file-v2.md

@@ -0,0 +1,130 @@
+# Dart Package Configuration File v2.0
+Author: [email protected]<br>
+Version: 0.1
+
+Dart currently uses a `.packages` file to configure *package URI resolution* (converting a package URI to a file name). The file uses a `.ini` file-like format with package names as keys and URI references as values. The URI references generated by Pub are always absolute or relative paths, so third-party users of the file tend to see them as paths, not URI.
+
+For the *language versioning* feature, we have proposed extending the format by adding fragments to the URI references and an entry with the empty string as key. This is a breaking change (the empty key was not previously accepted) and even more breaking for tools which assume the values are paths.
+
+Instead of breaking the existing file users, we will instead introduce a *new* file containing the information that we intended to store in the `.packages` file. This fill will use a (more) structured format which allows extending it with further information in the future. We will still generate the `.packages` file for now, allowing users of the file to continue working until they can migrate to using the new file.
+
+## New File Contents
+
+The file will contain the information that was planned for the `.packages` file:
+
+- A *default package* specified by its package name.
+- A list of *packages*, each with:
+  - A *package name*.
+  - A *path* or *location URI*.
+  - A *language version* (major/minor version number)
+
+On top of that, we also add extra metadata which allows us to version the format and attach information about when and how the file was created.
+
+We use JSON for the format. JSON has parsers available for most languages (not all Dart tools are written in Dart) and it is standardized and efficiently parsable.  JSON is *not* easy to write by hand (but doable), and it does not allow comments, but this is a machine generated file that will hardly ever be edited by hand.
+
+## New File Format
+
+The new file will be called `.packages.json` and stored in the same directory as the `.packages` file. The format is JSON text describing is a single JSON object.
+
+That object has, at least, the following properties:
+
+- `apiVersion`: A single integer marking the version of this format, currently `2` (counting `.packages` as the first). Tools should refuse to work on files with a larger version number than the ones they are aware of. Incremental changes will not require increasing the version, so new versions are not expected any time soon, but code should still prepare for it.
+
+- `defaultPackage`: A string containing a package name.
+- `packages`: A list of packages, each with the following properties:
+  - `name`: A string containing a package name.
+  - either `path`: A string containing a relative or absolute path using `/` as separator. Every character other than `/` is used verbatim. If the path starts with a `/` or with a DOS/Windows drive letter (a single letter followed by `:/`), then it is considered absolute. If not, the path is relative to the directory of the `.packages.json` file. 
+  - or `uri`: A URI reference. This is resolved relative to the (likely `file:`) URI referencing the `.package.json` file. If the URI reference has no scheme, or if it has a `file:` scheme, it is recommended to use the `path` instead.
+  - `languageVersion`: Optional. A string containing a language version (major.minor).
+
+Tools should ignore properties that they don't recognize. This will allow us to add more properties in the future, but not to remove properties or change the meaning of existing properties. 
+
+If we remove a property that tools rely on, change the meaning of an existing property, or add more information that is *required* for resolving and understanding Dart source files, it is a breaking change&mdash;tools expecting the previously available data will no longer understand the whole picture. Such changes will require increasing the `apiVersion` number.
+
+The following properties allows the generator to specify metadata about the file:
+
+- `generated`: A string containing an ISO-8601 timestamp for when the file was generated.
+- `generator`: An object containing at last the following properties:
+  - `name`: Name of the generator. Likely `pub`.
+  - `version`: The version of the generator. A Semantic Version, or a prefix of such (major version only, or major/minor version only are also accepted). Pub can probably use the SDK version.
+
+These are all optional, but if they are present, they should have the expected format.
+
+If a tool wants to store extra information in the `.packages.json` file, they can do so under a key prefixed by `toolname:`, so if Pub wanted to store more information on a package, say the package version, it could do so as `"pub:pkgVersion": "1.16.0"` in the package entry. Keys containing `:` are reserved for custom tool-specific properties. This feature can only really be used by the tool generating the file (as a generated file, it may be overwritten at any time), and it should be used sparingly since unknown properties are just overhead for all other users of the file.
+
+### Package Names
+
+Some properties are specified as being "package names".
+
+A package name is used as both a URI path segment and a directory name. To ensure this is possible, we define a package name as a string containing:
+
+- Only [URI path characters](https://www.ietf.org/rfc/rfc3986.txt) (RFC 3986 "pchar"),
+- but no `\`, `:` or `%`,
+- and containing at least one non-`.` character.
+
+This avoids the characters `/`, `\` and `:` which are meaningful in paths on various operating systems, as well as the strings `.` and `..`, and it ensures that the remaining characters are valid in a URI path and not the empty string.
+
+## Possible Extensions
+
+We can consider already adding more features to the format. These features should be ones that are of relevance to compilers reading source files, since that is the goal of the file.
+
+Any such extensions would need to be added by the generator, so in practice they must come from `pubspec.yaml` to begin with. If the Pub tool wants to support these, it is possible. 
+
+### Experiments
+
+Maybe you can enable experiments for a specific package by adding, for example:
+
+```json
+"experiments": ["non-nullabe"]
+```
+
+to a single package entry, or add it in the outer object to enable it for all compatible packages.
+
+### Environment Variables
+
+Maybe you can hard-code some environment variables by adding, for example:
+
+```json
+"environment": {
+  "myFlag": "true",
+  "logLevel": "verbose"
+}
+```
+
+in the outer object.
+
+## Summary
+
+A file, `.packages.json` will coexist with `.packages` until `.packages` can be phased out.
+
+The format will look like:
+
+```json
+{
+  "apiVersion": 2,
+  "defaultPackage": "myPackage",
+  "packages": [
+    {
+      "name": "myPackage",
+      "path": "lib/",
+      "languageVersion": "2.6"
+    },
+    {
+      "name": "myHelperPackage",
+      "path": "../myHelperPackage/lib/",
+      "languageVesion": "2.5"
+    },
+    {
+      "name": "test",
+      "path": "/users/myself/.pubcache/test-1.16.0/lib/",
+      "languageVersion": "2.5"
+    },
+    ... 
+  ],
+  "generated": "2019-09-12T12:13:14",
+  "generator": {
+    "name": "pub",
+    "version": "2.6.0-dev.0.2"
+  }
+}
+```