Defend against .gitattributes-induced working tree modifications

[sdboyer]

I've had this one stewing on my backburner for a while, as it's an edge case with the potential to be deep rabbit hole full of sad.

In Masterminds/glide#635 (comment), some glide users noticed an issue where cloning/checking out a repository containing .gitattributes files could result in a dirty working copy. glide blew up in some ugly ways. The issue hasn't been raised in dep-land yet, but we should get out ahead of this case if we can. Our code definitely assumes right now that the working copy of [git] repositories in the source cache is always clean.

I'd strongly prefer we find some reasonable approximation of correctness that's not unduly punitive towards the use of .gitattributes files, but keeps us from having to dive down the rabbit hole. To get a sense of what that might look like, though, we probably need a bit of research on exactly what circumstances (combinations of .gitattrifbutes and platform) can result in insta-dirty working copies.

I'm hoping somebody wants to do the research here, as this is outside my bandwidth to tackle right now 😄

[carolynvs]

I don't believe it's enough to just have line ending scrambling (either via AutoCRLF or .gitattributes) to trigger this situation. I have seen this occur:

• when there is either a mismatch between what's in the git cache and in the working copy, often seen when in the middle of changing the .giattributes file. This is fixed by running git rm --cached -r . && git reset --hard.
• when a file has mixed line endings due to contributors all using different autocrlf settings, and no unifying .gitattributes to keep them in sync.

Not quite sure off the top of my head how to reproduce in a test though.

[mikijov]

I thought that the goal here is to defend against any kind of dirtyness or corruption in the vendored repo. Why not simply use a combination of git clean -fxd :/ and git checkout -f, and if those fail destroy and re-clone? Am I missing something that requires us to know how the repo was dirtied?

[sdboyer]

The concern here isn't with what's in the vendor/ directory, but what's in the source storage area, $GOPATH/pkg/dep/sources. For repositories that have this problem, if it even really is a problem, we'd end up constantly deleting and recloning them (and I'm not sure we'd actually make progress) simply in order to progress through solving, as they would (presumably) check out dirty at every revision.

[dmitris]

hit a similar error when running dep init in a project with existing/working glide.yaml:

/var/folders/d9/6nm5dx2j7vz3hv0kw1m2mdrw0000gp/T/dep929098858/vendor/golang.org/x/crypto/.gitignore already exists, no checkout

Full output in the gist: https://gist.github.com/dmitris/7216fb8eeeb419166bfbd5966472d2ac

glide.yaml contains the following spec for golang.org/x/crypto:

- package: golang.org/x/crypto
  version: 594708b89f21ece706681be23d04a6513a22de6e
  repo: git@opensource.git.corp.<company>.com:golang/crypto.git
  subpackages:
  - ssh

[sdboyer]

@dmitris can you regenerate that output with dep init -v?

[dmitris]

@sdboyer - done, https://gist.github.com/dmitris/6dc0a0fe71f5f8360628af3e7e08f19a (with a few internal references redacted). Let me know if there's anything I can do to help tracing/debugging it.