cmd/go: go install cmd@version errors out when module with main package has replace directive

[james-lawrence]

What version of Go are you using (go version)?

go version
go version go1.16 linux/amd64

Does this issue reproduce with the latest release?

yes

What did you do?

cd /tmp; go install bitbucket.org/jatone/genieql@latest

What did you expect to see?

go install finishing successfully

What did you see instead?

go install bitbucket.org/jatone/genieql@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

given that genieql is the main module this error appears to be erroneous.

other attempts I made all of which i would have expected to work properly (and use to):

cd /tmp; go install bitbucket.org/jatone/genieql/...@latest
go install bitbucket.org/jatone/genieql/...@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

cd /tmp; go install bitbucket.org/jatone/genieql/cmd/...@latest
go install bitbucket.org/jatone/genieql/cmd/...@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

cd /tmp; go install bitbucket.org/jatone/genieql/cmd/genieql@latest
go install bitbucket.org/jatone/genieql/cmd/genieql@latest: bitbucket.org/jatone/genieql@v0.0.0-20210306180040-be9c7b630c18
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

[seankhliao]

L28 contains a replace directive

replace github.com/containous/yaegi => github.com/james-lawrence/yaegi v0.8.8-modules-enh

Which is not valid for go install

No module is considered the "main" module. If the module containing packages named on the command line has a go.mod file, it must not contain directives (replace and exclude) that would cause it to be interpreted
differently than if it were the main module

Closing as working as intended

[james-lawrence]

yes obviously line 28 contains the directive. see the title. and then see this thread where I was told to open this issue.

[james-lawrence]

that would cause it to be interpreted differently than if it were the main module

is the relevant line of that statement here. the fact is this directive wouldn't cause it to be interpreted differently than if it was the main module. and to any outside viewer, there is a main module, its the one being installed.

[seankhliao]

reopening, I would recommend rephrasing the issue in terms of a proposal to relax the behaviour rather than as a bug report

[james-lawrence]

@seankhliao I'm just not sure if it is a bug or a proposal, the discussion around go installs (new) behavior was long, varied, and confused. =) this particular problem was pointed out during that discussion and seemingly ignored/disregarded/left open to be revisited. so here we are.

[myitcv]

@james-lawrence

this particular problem was pointed out during that discussion and seemingly ignored/disregarded/left open to be revisited.

#40276 was indeed discussed at length. Indeed, over the years even prior to that specific proposal, this issue has been discussed in various forms, many times. To say that the question of replace directives was ignored/disregarded is incorrect. Indeed I specifically linked to a quote that pointed out the option for that particular point to be revisited. It suffices therefore to open an issue, as suggested, to start that discussion.

Deferring to @jayconrod on how best to take the discussion forward.

[james-lawrence]

I meant it was at one of those 3 at various points in that (and the related) thread. I wasn't calling out any particular stage, sorry for that confusion.

[ianthehat]

I think there may be confusion around the concepts of a main module and the package main.
The main module is normally the one in which the go command was invoked (no matter what packages are being built or installed). Modules cannot be installed, only packages. When building in package@version mode there is no main module.

In the case in this issue genieql is the package main, and the way in which it would be built changes if the module of which it is a part is the main module because of the replace statement.
For example, if you did go install bitbucket.org/jatone/genieql in another module that had bitbucket.org/jatone as a dependency it would not obey the replace directive. If you checkout bitbucket.org/jatone and do go install ./genieql then it will obey the replace directives and produce a different result. It would also obey the versions specified in the main module, which would change the results even further.

Also note that it is not something that used to work, it is a brand new feature that limits the number of cases in which it is enabled.

Maybe we should change the terminology for main module to a different word?

[james-lawrence]

Also note that it is not something that used to work

while technically true go get -u did work because it respected the vendor directory: go114 get -u bitbucket.org/jatone/genieql-go114/cmd/... see related issue #44841 and #40053 + the google groups thread in that issue.

cat ~/go/src/bitbucket.org/jatone/genieql-go114/vendor/github.com/containous/yaegi/interp/src.go | grep -i 'func pkgDir(ctx \*build.Context'

returns the patched function from the replace + vendor workflow, and if you started with a clean environment there would be no other sources of the library in question.

now did it work if trying to use a package as a dependency? no, and that was fine because it wasn't about using it as a library. many go programs are never intended to be used that way.

now the rest of your argument is attempting to explain away for current limitations after the fact that go get / go install commands were fundamentally changed to support go modules while simultaneously ignoring vendor workflows of the tools and making that argument with a distinction that never existed until the tools were changed.

now all that being said: installing a binary is different from managing dependencies. I believe this is an accepted fact and a reason go get was changed to be about managing dependencies and go install is being shift towards building and installing binaries. this fundamentally is a good change and I'm wholeheartedly behind it.

I think the question we need to be asking is if the distinction between main package of the main module during installation matter? I'm going to assert it does not and that this error is erroneous, but during dependency management where someone tried to add a dependency that had a replace directive this error would be 100% correct though I'd start out with a warning in that case vs a hard error.

[james-lawrence]

Ah I believe I have a good explanation about why this should be supported.

the replace directive is fundamentally about compilation, its meant to inform the build system how to patch dependencies. this is true both during development where you may need to patch them while debugging a problem which is when you compile your program. and during installation of a main module (previously the main package) onto a system.

so during compilation there is no distinction between the main package and the main module, however, during dependency management replace directives have always been ignored and any software that was relying on a dependency where there was a replace directive was broken (conceptually if not in practice). and this is where we should be erroring out (i.e. in go get in this new world) when we see a replace directive. not during compilation (go install).

[jayconrod]

Closing this issue since it's working as described in #40276.

That being said, at some point in the future when we have more information, it will be reasonable to open a proposal to either ignore or apply replace and exclude directives. That proposal should be based on the discussion in #40276 (and previously #30515), it must include data backing up whichever position is supported, and it must consider tradeoffs for the other side.

I know this is a lot to ask (those issues are very long), but the main point of disagreement was what to do with replace directives, and we debated that for a year and a half. The outcome was a compromise: replace directives are not allowed for now, and we have the option of changing that in the future without making an incompatible change.

To restart that discussion, we need new information: how is the new go install cmd@version form being used, and would replace directives be helpful or harmful?