Skip to content

ability to fetch dependencies via git+ssh protocol #14295

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Tracked by #14265
andrewrk opened this issue Jan 13, 2023 · 1 comment
Open
Tracked by #14265

ability to fetch dependencies via git+ssh protocol #14295

andrewrk opened this issue Jan 13, 2023 · 1 comment
Labels
contributor friendly This issue is limited in scope and/or knowledge of Zig internals. enhancement Solving this issue will likely involve adding new logic or components to the codebase. zig build system std.Build, the build runner, `zig build` subcommand, package management
Milestone
0.15.0

Comments

@andrewrk
Copy link
Member

andrewrk commented Jan 13, 2023 •
edited
Loading

Extracted from #14265.

zig build should support fetching via an URL like this:

    .url = "git+ssh://[email protected]:ziglang/zig.git#8b8090d7fad3e444784bc52db6a80188a9dbd3c0",

Note that the fragment is used to fetch a particular commit. I suppose the fragment could be omitted, meaning to fetch the latest HEAD of the default branch, however, this would be not advised since the hash would be wrong as soon as another commit is pushed to that branch. Ideally, if the fragment is omitted then an error would be emitted telling the user to add the fragment, giving them a copy+pasteable snippet, or perhaps even editing the manifest file on the user's behalf.

Open question: should it be built-in? or is this issue a request for a third party contributor to make a fetch plugin (#14294)?

I think the first step would be to implement this as a third-party fetch plugin, and then we can evaluate whether it can be upstreamed and become a builtin.

Related:
@andrewrk andrewrk added enhancement Solving this issue will likely involve adding new logic or components to the codebase. contributor friendly This issue is limited in scope and/or knowledge of Zig internals. standard library This issue involves writing Zig code for the standard library. zig build system std.Build, the build runner, `zig build` subcommand, package management labels Jan 13, 2023
@andrewrk andrewrk added this to the 0.11.0 milestone Jan 13, 2023
@andrewrk andrewrk mentioned this issue Jan 13, 2023
Open
@andrewrk andrewrk removed the standard library This issue involves writing Zig code for the standard library. label Jan 13, 2023
@andrewrk andrewrk modified the milestones: 0.11.0, 0.12.0 Jan 13, 2023
This was referenced Jan 13, 2023
Open
Open
Closed
Merged
Closed
@andrewrk andrewrk modified the milestones: 0.13.0, 0.12.0 Jan 16, 2023
ianprime0509 added a commit to ianprime0509/zig that referenced this issue Sep 23, 2023
@ianprime0509
Support fetching dependencies over git+http(s)
9c8baf6 
Closes ziglang#14298

This commit adds support for fetching dependencies over git+http(s)
using a minimal implementation of the Git protocols and formats relevant
to fetching repository data. It currently supports only version 2 of Git's
wire protocol (documented in
[protocol-v2](https://git-scm.com/docs/protocol-v2)), which was
first introduced in Git 2.19 (2018) and made the default in 2.26 (2020).

The wire protocol behaves similarly when used over other transports,
such as SSH and the "Git protocol" (git:// URLs), so it should be
reasonably straightforward to support fetching dependencies from such
URLs if the necessary transports are implemented (e.g. ziglang#14295).
ianprime0509 added a commit to ianprime0509/zig that referenced this issue Sep 23, 2023
@ianprime0509
Support fetching dependencies over git+http(s)
a9dbe9b 
Closes ziglang#14298

This commit adds support for fetching dependencies over git+http(s)
using a minimal implementation of the Git protocols and formats relevant
to fetching repository data. It currently supports only version 2 of Git's
wire protocol (documented in
[protocol-v2](https://git-scm.com/docs/protocol-v2)), which was
first introduced in Git 2.19 (2018) and made the default in 2.26 (2020).

The wire protocol behaves similarly when used over other transports,
such as SSH and the "Git protocol" (git:// URLs), so it should be
reasonably straightforward to support fetching dependencies from such
URLs if the necessary transports are implemented (e.g. ziglang#14295).
ianprime0509 added a commit to ianprime0509/zig that referenced this issue Sep 23, 2023
@ianprime0509
Support fetching dependencies over git+http(s)
8867df5 
Closes ziglang#14298

This commit adds support for fetching dependencies over git+http(s)
using a minimal implementation of the Git protocols and formats relevant
to fetching repository data. It currently supports only version 2 of Git's
wire protocol (documented in
[protocol-v2](https://git-scm.com/docs/protocol-v2)), which was
first introduced in Git 2.19 (2018) and made the default in 2.26 (2020).

The wire protocol behaves similarly when used over other transports,
such as SSH and the "Git protocol" (git:// URLs), so it should be
reasonably straightforward to support fetching dependencies from such
URLs if the necessary transports are implemented (e.g. ziglang#14295).
ianprime0509 added a commit to ianprime0509/zig that referenced this issue Sep 24, 2023
@ianprime0509
Support fetching dependencies over git+http(s)
63f4dcf 
Closes ziglang#14298

This commit adds support for fetching dependencies over git+http(s)
using a minimal implementation of the Git protocols and formats relevant
to fetching repository data. It currently supports only version 2 of Git's
wire protocol (documented in
[protocol-v2](https://git-scm.com/docs/protocol-v2)), which was
first introduced in Git 2.19 (2018) and made the default in 2.26 (2020).

The wire protocol behaves similarly when used over other transports,
such as SSH and the "Git protocol" (git:// URLs), so it should be
reasonably straightforward to support fetching dependencies from such
URLs if the necessary transports are implemented (e.g. ziglang#14295).
ianprime0509 added a commit to ianprime0509/zig that referenced this issue Sep 24, 2023
@ianprime0509
Support fetching dependencies over git+http(s)
0ec8a01 
Closes ziglang#14298

This commit adds support for fetching dependencies over git+http(s)
using a minimal implementation of the Git protocols and formats relevant
to fetching repository data. It currently supports only version 2 of Git's
wire protocol (documented in
[protocol-v2](https://git-scm.com/docs/protocol-v2)), which was
first introduced in Git 2.19 (2018) and made the default in 2.26 (2020).

The wire protocol behaves similarly when used over other transports,
such as SSH and the "Git protocol" (git:// URLs), so it should be
reasonably straightforward to support fetching dependencies from such
URLs if the necessary transports are implemented (e.g. ziglang#14295).
ianprime0509 added a commit to ianprime0509/zig that referenced this issue Sep 25, 2023
@ianprime0509
Support fetching dependencies over git+http(s)
e2b64ae 
Closes ziglang#14298

This commit adds support for fetching dependencies over git+http(s)
using a minimal implementation of the Git protocols and formats relevant
to fetching repository data. It currently supports only version 2 of Git's
wire protocol (documented in
[protocol-v2](https://git-scm.com/docs/protocol-v2)), which was
first introduced in Git 2.19 (2018) and made the default in 2.26 (2020).

The wire protocol behaves similarly when used over other transports,
such as SSH and the "Git protocol" (git:// URLs), so it should be
reasonably straightforward to support fetching dependencies from such
URLs if the necessary transports are implemented (e.g. ziglang#14295).
ianprime0509 added a commit to ianprime0509/zig that referenced this issue Sep 25, 2023
@ianprime0509
Support fetching dependencies over git+http(s)
9426a24 
Closes ziglang#14298

This commit adds support for fetching dependencies over git+http(s)
using a minimal implementation of the Git protocols and formats relevant
to fetching repository data.

Git URLs can be specified in `build.zig.zon` as follows:

```zig
.xml = .{
    .url = "git+https://github.com/ianprime0509/zig-xml#7380d59d50f1cd8460fd748b5f6f179306679e2f",
    .hash = "122085c1e4045fa9cb69632ff771c56acdb6760f34ca5177e80f70b0b92cd80da3e9",
},
```

The fragment part of the URL may specify a commit ID (SHA1 hash), branch
name, or tag. It is an error to omit the fragment: if this happens, the
compiler will prompt the user to add it, using the commit ID of the HEAD
commit of the repository (that is, the latest commit of the default
branch):

```
Fetch Packages... xml... /var/home/ian/src/zig-gobject/build.zig.zon:6:20: error: url field is missing an explicit ref
            .url = "git+https://github.com/ianprime0509/zig-xml",
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
note: try .url = "git+https://github.com/ianprime0509/zig-xml#dfdc044f3271641c7d428dc8ec8cd46423d8b8b6",
```

This implementation currently supports only version 2 of Git's wire
protocol (documented in
[protocol-v2](https://git-scm.com/docs/protocol-v2)), which was first
introduced in Git 2.19 (2018) and made the default in 2.26 (2020).

The wire protocol behaves similarly when used over other transports,
such as SSH and the "Git protocol" (git:// URLs), so it should be
reasonably straightforward to support fetching dependencies from such
URLs if the necessary transports are implemented (e.g. ziglang#14295).
@ianprime0509 ianprime0509 mentioned this issue Sep 25, 2023
Merged
ianprime0509 added a commit to ianprime0509/zig that referenced this issue Sep 29, 2023
@ianprime0509
Support fetching dependencies over git+http(s)
52a0f49 
Closes ziglang#14298

This commit adds support for fetching dependencies over git+http(s)
using a minimal implementation of the Git protocols and formats relevant
to fetching repository data.

Git URLs can be specified in `build.zig.zon` as follows:

```zig
.xml = .{
    .url = "git+https://github.com/ianprime0509/zig-xml#7380d59d50f1cd8460fd748b5f6f179306679e2f",
    .hash = "122085c1e4045fa9cb69632ff771c56acdb6760f34ca5177e80f70b0b92cd80da3e9",
},
```

The fragment part of the URL may specify a commit ID (SHA1 hash), branch
name, or tag. It is an error to omit the fragment: if this happens, the
compiler will prompt the user to add it, using the commit ID of the HEAD
commit of the repository (that is, the latest commit of the default
branch):

```
Fetch Packages... xml... /var/home/ian/src/zig-gobject/build.zig.zon:6:20: error: url field is missing an explicit ref
            .url = "git+https://github.com/ianprime0509/zig-xml",
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
note: try .url = "git+https://github.com/ianprime0509/zig-xml#dfdc044f3271641c7d428dc8ec8cd46423d8b8b6",
```

This implementation currently supports only version 2 of Git's wire
protocol (documented in
[protocol-v2](https://git-scm.com/docs/protocol-v2)), which was first
introduced in Git 2.19 (2018) and made the default in 2.26 (2020).

The wire protocol behaves similarly when used over other transports,
such as SSH and the "Git protocol" (git:// URLs), so it should be
reasonably straightforward to support fetching dependencies from such
URLs if the necessary transports are implemented (e.g. ziglang#14295).
ianprime0509 added a commit to ianprime0509/zig that referenced this issue Sep 30, 2023
@ianprime0509
Support fetching dependencies over git+http(s)
fa945d9 
Closes ziglang#14298

This commit adds support for fetching dependencies over git+http(s)
using a minimal implementation of the Git protocols and formats relevant
to fetching repository data.

Git URLs can be specified in `build.zig.zon` as follows:

```zig
.xml = .{
    .url = "git+https://github.com/ianprime0509/zig-xml#7380d59d50f1cd8460fd748b5f6f179306679e2f",
    .hash = "122085c1e4045fa9cb69632ff771c56acdb6760f34ca5177e80f70b0b92cd80da3e9",
},
```

The fragment part of the URL may specify a commit ID (SHA1 hash), branch
name, or tag. It is an error to omit the fragment: if this happens, the
compiler will prompt the user to add it, using the commit ID of the HEAD
commit of the repository (that is, the latest commit of the default
branch):

```
Fetch Packages... xml... /var/home/ian/src/zig-gobject/build.zig.zon:6:20: error: url field is missing an explicit ref
            .url = "git+https://github.com/ianprime0509/zig-xml",
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
note: try .url = "git+https://github.com/ianprime0509/zig-xml#dfdc044f3271641c7d428dc8ec8cd46423d8b8b6",
```

This implementation currently supports only version 2 of Git's wire
protocol (documented in
[protocol-v2](https://git-scm.com/docs/protocol-v2)), which was first
introduced in Git 2.19 (2018) and made the default in 2.26 (2020).

The wire protocol behaves similarly when used over other transports,
such as SSH and the "Git protocol" (git:// URLs), so it should be
reasonably straightforward to support fetching dependencies from such
URLs if the necessary transports are implemented (e.g. ziglang#14295).
andrewrk pushed a commit that referenced this issue Oct 1, 2023
@ianprime0509 @andrewrk
Support fetching dependencies over git+http(s)
9a001e1 
Closes #14298

This commit adds support for fetching dependencies over git+http(s)
using a minimal implementation of the Git protocols and formats relevant
to fetching repository data.

Git URLs can be specified in `build.zig.zon` as follows:

```zig
.xml = .{
    .url = "git+https://github.com/ianprime0509/zig-xml#7380d59d50f1cd8460fd748b5f6f179306679e2f",
    .hash = "122085c1e4045fa9cb69632ff771c56acdb6760f34ca5177e80f70b0b92cd80da3e9",
},
```

The fragment part of the URL may specify a commit ID (SHA1 hash), branch
name, or tag. It is an error to omit the fragment: if this happens, the
compiler will prompt the user to add it, using the commit ID of the HEAD
commit of the repository (that is, the latest commit of the default
branch):

```
Fetch Packages... xml... /var/home/ian/src/zig-gobject/build.zig.zon:6:20: error: url field is missing an explicit ref
            .url = "git+https://github.com/ianprime0509/zig-xml",
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
note: try .url = "git+https://github.com/ianprime0509/zig-xml#dfdc044f3271641c7d428dc8ec8cd46423d8b8b6",
```

This implementation currently supports only version 2 of Git's wire
protocol (documented in
[protocol-v2](https://git-scm.com/docs/protocol-v2)), which was first
introduced in Git 2.19 (2018) and made the default in 2.26 (2020).

The wire protocol behaves similarly when used over other transports,
such as SSH and the "Git protocol" (git:// URLs), so it should be
reasonably straightforward to support fetching dependencies from such
URLs if the necessary transports are implemented (e.g. #14295).
@KilianHanich
Copy link

Considering how security sensitive ssh is, I am not sure it would be a good idea to implement the protocol ourselves (be it builtin or plugin). Just calling the git executable instead and let it deal with it, sounds (at least to me) like a more sane solution even if it means that it needs to have Git in it's $PATH.

There are a few reasons for this:

  • There will be requests about "support typing in the password manually", "support ssh keyfiles", "support getting the keyfile via " and possibly more.
  • Some Linux distributions use SELinux to only allow certain executables to access ~/.ssh (and quite frankly for good reason).
  • It's also not like the protocol is small/easy: https://datatracker.ietf.org/doc/html/rfc4254 sure, chapter 6 (interactive sessions) and 7 (TCP/IP forwarding) aren't needed, but just this RFC forwards to other RFCs for details.
  • Git over SSH works by creating a full-blown ssh session (in non-interactive mode, like when you for example say ssh user@server 'ps -A') and then running a few commands (on sites like GitHub restricted by using git-shell as login shell). Because of that not even the Git executable itself implements SSH itself but uses the one it finds in PATH (most often OpenSSH).

So with that in mind, I would argue that implementing this by calling Git (or even ssh) would be preferable since I doubt anybody here wants to essentially maintain a SSH implementation.

@KilianHanich KilianHanich mentioned this issue Jul 28, 2024
Open
@mlugg mlugg moved this to Fetching in Package Manager Aug 22, 2024
@zaddok zaddok mentioned this issue Feb 11, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
contributor friendly This issue is limited in scope and/or knowledge of Zig internals. enhancement Solving this issue will likely involve adding new logic or components to the codebase. zig build system std.Build, the build runner, `zig build` subcommand, package management
Projects
Package Manager
Status: Fetching
Milestone
0.15.0
Development

No branches or pull requests
2 participants
@andrewrk @KilianHanich