go/importer: implement support for importer.Lookup

[griesemer]

Missing functionality. Required for clients that store packages outside the implied file system structure.

[griesemer]

Started.

[griesemer]

We may have to leave this one for 1.7. The existing API based on the importer.Lookup type for flexibility is unfortunately inadequate, and at the same time unimplemented (a non-nil Lookup function was not supported). That is, it's not possible to have any users yet.

This opens the door to a backward-incompatible change. At the same time we don't want to design in a vacuum (as we did on short-notice before 1.5 in this case). Probably better to leave this for early 1.7 when we also get the new import data which should clear things up.

[griesemer]

We have x/tools/gcimporter15 as a stop gap for now.

Due to the additional srdDir argument required for lookup because of vendoring, this API may have to change unfortunately.

[creachadair]

We had to work around the lack of support for custom lookup functions for http://kythe.io/. I did this by the gross but expedient hack of depending on the internal package, but that prevents building with the go tool. Obviously we'd prefer to get rid of the hack.

What can I offer by way of assistance to get this working?

[griesemer]

1. The API - as it was designed for 1.5 - didn't take into account vendoring (at that point considered an experiment introduced late). As a consequence, the current API (go/importer.Lookup function) may have to change. Since it cannot (Go 1 backward-compatibility promise) we need to come up with a new API. We didn't hurry this for 1.6 because we got to it too late and didn't want to prototype yet another possibly incomplete API.

2. The export data format is going to change for 1.7 (the new format is already implemented but for inlined exported functions). It shouldn't affect the API, but it might.

In terms of help: It would be good to have examples of concrete use cases, ideally with suggested minimal API. This would help clarify the final solution.

For what it's worth, we have x/tools/go/gcimporter15 which provides (I hope) all the individual pieces to import from custom locations and it is based on the 1.5/1.6 std repo go/types. Does that help?

[creachadair]

Thank you for the information about gcimporter15. If that works as I think, that might even let me get rid of the hacky dependency I took.

For Kythe, our goal is to be able to capture the files involved in building a Go package and to (later) re-run the type checker from those. An ideal API for our use case would be something like a function that takes a compiled package (say, a .a or .gcgox file) and returns whatever the go/types package needs to use it for type resolution—presumably a package name and some other internal state. We don't care at all about the internal format of this information as long as there's a way to get go/types to use it.

Having the API move around is fine, as long as it's a public API.

[griesemer]

This is not going to happen for 1.7 since it does require additional API work. We have x/tools/go/gcimporter15 as a stop-gap, it is compatible with the latest import/export work.

[rsc]

What is the status here? Is this important for Go 1.8?