Pkg.add: provide a way to download over https instead of git

[crayxt]

Hello,
I am using Windows 7 64-bit, Julia-0.3.0-prerelease, version latest as of today.
I am trying to install packages.
The thing is I am sitting behind the proxy and could not connect to git:// type of links (https:// works fine).
For Pkg.init() there is a workaround by passing full URL of METADATA github repo. I mean below command works just fine
Pkg.init("https://github.com/JuliaLang/METADATA.jl")

But I couldn't find a way to tell Pkg.add to use https instead of git for downloading packages (other than manually replace links in package description files, of course).

[ihnorton]

Please see https://github.com/JuliaLang/julia/blob/master/README.md#source-download-and-compilation

[ihnorton]

(sorry, stupid button. please re-open if this does not solve the problem)

[crayxt]

@ihnorton that helped, thanks!

[ivarne]

This seems to be a common problem. Would it make sense to make Pkg aware of the problem, so that it can retry over https:// if git:// fails?

[nalimilan]

One could also imagine making https:// the default for Pkg.add(), and git:// only for Pkg.checkout(). After all, people who do not use Pkg.checkout() are probably not going to use git to make pull requests (they'll be out of sync with master anyway).

[isislovecruft]

Is there any issue to making https:// the default for fetches and clones? The git:// protocol is neither encrypted nor authenticated, making Man-In-The-Middle attacks trivial to conduct. For developers downloading and executing code from the internet (as is currently the case for Julia's Pkg manager), these attacks are absolutely disastrous, leading to complete compromise of the developer's system.

Please, I don't want to watch Julia make CPAN's and pip's mistakes all over again. Please fix this issue!

[tkelman]

We could potentially modify the gitconfig in the copy of git that we bundle with the Mac and Windows binaries to always prefer https. For tagged package versions we're typically fetching a known sha, so it seems like a mitm attack would be quickly apparent? Though I suppose the content of metadata itself could be spoofed? Could be worth changing

julia/base/pkg.jl

Line 8 in 444c60e

const DEFAULT_META = "git://github.com/JuliaLang/METADATA.jl"

to https.

[isislovecruft]

See also similar issues with go get: golang/go#9637, and Python's pip: pypa/pip#425 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-1629 pypa/pip#1167.

Tangentially related is an issue in Perl's Net::HTTPS API (CVE-2011-0633) which regards TLS certificate verification, and which was assigned some time after they enabled TLS for downloads during module installations with CPAN.pm. (I can't find the original CVE I was remembering regarding CPAN.pm switching to TLS/SSL by default, so I may be misremembering.)

[tkelman]

@StefanKarpinski, @staticfloat, thoughts?

[isislovecruft]

For tagged package versions we're typically fetching a known sha, so it seems like a mitm attack would be quickly apparent? Though I suppose the content of metadata itself could be spoofed?

I'm not sure I fully understand what you meant by "spoofed". Please let me try to explain.

Yes, you are correct; the primary issue is that the list of "known" SHA digests could be altered/updated/filtered by an attacker because the connection is not authenticated. There is also the secondary issue that the objects themselves are not authenticated, but rather than the SHA-1 digests of the contents of those objects merely constitutes a checksum.

To elucidate on the primary issue, the original fetch of the METADATA.jl repo could be MITMed (possibly even MOTSs), or any subsequent fetch/pull of it, to allow arbitrary code execution. An example would be: Alice has already installed the Julia REPL and some Julia packages installed. She does Pkg.update() and Mallory MITMs Alice's subsequent connection to git://github.com/JuliaLang/METADATA.jl. Because the protocol is not authenticated, Alice cannot tell the difference between Mallory and the real Github. Mallory then sends Alice a maliciously updated HEAD for METADATA.jl, which tells Alice that the url for Gadfly.jl is now https://github.com/mallory/Gadfly.jl, and that there is a "new tagged version" of IJulia.jl (which Mallory could then MITM Alice's connection to git://github.com/JuliaLang/IJulia.jl to give Alice this "new version"), etc. Now Alice cannot trust any of the sources of code any time she updates any already installed packages, or installs any new ones, since at any point (starting from the very first time that Alice used Pkg) her connection could have gone somewhere other than git://github.com/JuliaLang/METADATA.jl.

For the second issue, the problem is that — even if https://github.com… is used by default — the SHA-1 sums on the git objects merely constitutes a checksum of the contents of the object, not an authentication of the origin of the object. This attack is less feasible in the real world, however it is often a design concern for package managers; for example, apt provides some protection against this by providing OpenPGP signatures of the updated lists of package digests. And, for an example of this being used in Julia's Pkg case, Mallory breaks into Github and alters/adds objects to any JuliaLang repos she wishes. Hopefully, Github eventually detects this compromise, but if they don't, or before they are able to, Alice installs Gadfly.jl and receives Mallory's malicious version. To return to the ways in which this problem is normally solved, the SHA-1 digests also need to be autheticated in some manner. In Julia's Pkg case, perhaps a simple solution would be to have client's verify OpenPGP signatures on the git tag they are trying to install.

Hopefully these explanations help to clarify the problems. If I failed to explain something, or you have more questions, please let me know!

[IainNZ]

@isislovecruft it seems like that second issue is a really tough nut to crack, I'd almost say its in the "too hard" pile actually, at least for Julia right now. Obviously its also a tougher attack to execute. What in your opinion are the most bang-for-buck changes we can make - like would default https address the 99.9% of scenarios?

[StefanKarpinski]

Using HTTPS is a good idea. Doing signed git tags for package versions may also be a good idea. To some extent, having signed git tags mitigates the need for HTTPS, but doing both doesn't hurt. However, the deeper issue is how one establishes trust in open source packages in the first place. If there are going to be signed versions, we need to know who can be trusted to sign versions. How do I know if I should trust someone who wrote a package or not? Is this going to be an individual decision or will there be an official list? Who decides who to trust? If we're going to just trust anyone, authentication is useless.

[StefanKarpinski]

That being said, changing to HTTPS is a significant improvement since then only people who have had changes to METADATA accepted are being trusted, whereas without HTTPS anyone on any network between you and GitHub is being trusted.

[LachlanGunn]

After the briefest of discussions on julia-dev, I have submitted pull request #11312 to use HTTPS for all Github URLs. This applies to both METADATA.jl and to the individual package URLs, whose protocols are automatically changed to https://.

[erfangc]

👍

[tkelman]

This was implemented by #11312

edit: at least for github-hosted packages - we may want to add an API for accepting different url normalization regexes but I think that can be a new issue