Add `Exec` to the gateway API.

[tonistiigi]

gw.Exec should invoke a command similarily to the frontend.Exec does atm. The main difference from Solve is that Exec is never cached. Should be able to take the same arguments that llb.Exec takes (including the special mounts). More complex cases for things like ReadFile could be implemented with Exec.

The mounts of Exec should take References from completed sources as inputs.

[sipsma]

To clarify the intent here, if you also added a feature to connect the Exec'd container's stdio to a pty, would this issue enable an "interactive" container? I.e. a container specified by LLB that spawns a usable shell/editor/other TTY-based program?

[tonistiigi]

Yes, I think there is a need to make stdio accessible through Exec. Not 100% sure about the pty as it somewhat of a trap.

For stdio in regular LLB the current proposal is to use #1337 , llb.Expose(llb.Image(""), llb.Pipe(1, 0))

@hinshun

[tonistiigi]

Thinking about it more, I think we definitely need the ability for the user to interact with a build result like a regular container. This is needed for debug scenarios. Maybe we don't necessarily need the frontend to have this access? And if frontend does use Exec it should show up as a line item in build output (probably with studio logs).

[hinshun]

We previously started the conversation on Slack, but I'm paraphrasing it here to save the ideas somewhere. cc @coryb

• At a high-level we want clients to be able to Exec into a container, outside of LLB and not cached.
• Using this new primitive, you can then build a debugging experience on top, by Execing the result of a failed solve.
• Let's begin by brainstorming about the UX around Exec, consider the following scenarios and the possible usage.

Scenario 1: Exec from regular LLB

execState := llb.Image("busybox").Run(llb.Shlex("/bin/sh"))

// Approach #1a
p := execState.ToExecProcess()
p.Run(ctx, io, tty)

// Appraoch #1b
execState.Run(ctx, io, tty)

Scenario 2: Exec into failed solve result

var st llb.State
def, _ := st.Marshal(ctx)

res, err := c.Solve(ctx, client.SolveRequest{Definition: def.ToPB()})
if err != nil {
	var er *ExecError
	if errors.As(err, &er) {
        	// Approach #2a
		c.Exec(ctx, er.Meta(), er.ErrorStateMounts(), io, tty)

		// Approach #2b
		p := ToExecProcess(er)
		p.Run(ctx, io, tty)
	}
}

Note that Approach #2b still needs Approach #2a to be implemented, #2b is an object-oriented UX slapped ontop of #2a.

How can we make this happen? (Ongoing list)

• Add Exec to the LLBBridge in gateway.proto:

  buildkit/frontend/gateway/pb/gateway.proto

  Line 16 in 17c11d9

  service LLBBridge {

• Modify Executor and ExecOp to not release mounts after error, and return them inside ExecError.
  • If mounts are unhandled, they are released with the gateway api release

[coryb]

I am hoping we can agree on a proto definition for this feature, once we have general agreement on this I should be able to work out a rough implementation.

As I see it we will need 2 grpc components:

1. the Exec rpc call to the LLBBridge service
2. IO bridge streaming service to facilitate stdio and possibly resize events

1 - Exec RPC

I think something like this will work:

service LLBBridge {
	rpc Exec(ExecRequest) returns (ExecResponse)
}

message ExecRequest {
	pb.ExecOp op = 1;
	repeated uint32 openFds = 2;
}

message ExecResponse {
    // What goes here?  Anything?  Task Exit Status?
}

I think the pb.ExecOp already has all the information we would need to implement the exec operation via the gateway. The openFds would be a list of fds to to open/close for the task. Open Fds will be connected to the IO bridge streaming service. We will need also need a tty flag, and for that I suggest we modify the pb.Meta to include that flag as:

message Meta {
	repeated string args = 1;
	repeated string env = 2;
	string cwd = 3;
	string user = 4;
	ProxyEnv proxy_env = 5;
	repeated HostIP extraHosts = 6;
	bool tty = 7;
}

since that matches the structure already available in the executor.Meta.

2 - IO bridge service

We need some way to stream input/output to the container task. I suggest creating a grpc service on the client side that can be accessed from the server via the session manager, similar to the Attachable SSHForwarder implementation. This is the proto I was thinking of:

service IOBridge {
	rpc IO(stream FdMessage) returns (stream IOInput)
}

message IOInput {
	 oneof input {
		FdMessage file = 1;
		ResizeMessage resize = 2;
	}
}

message FdMessage{
	uint32 fd = 1; // what fd the data was from
	bool eof = 2;  // true if eof was reached
	bytes data = 3;
}

message ResizeMessage{
	uint32 rows = 1;
	uint32 cols = 2;
	uint32 xpixel = 3;
	uint32 ypixel = 4;
}

Thoughts? Adjustments? Counter proposals?

[tonistiigi]

how about

service LLBBridge {
  rpc NewContainer(NewContainerRequest) returns (NewContainerResponse);
  rpc ReleaseContainer(ReleaseContainerRequest) returns (ReleaseContainerResponse);
  rpc ExecProcess(stream ExecMessage) returns (stream ExecMessage);
}

message NewContainerRequest {
	string Ref = 1;
        // For mount input values we can use random identifiers passed with ref
	repeated pb.Mount mounts = 2;
	pb.NetMode network = 3;
	pb.SecurityMode security = 4
}

message ReleaseContainerRequest {
	string Ref = 1;
}


message ExecMessage {
	 oneof input {
                InitMessage init = 1;
		FdMessage file = 2;
		ResizeMessage resize = 3;
		StartedMessage started = 4;
		ExitMessage exit = 5;
	}
}

message InitMessage{
  pb.Meta Meta = 1;
  repeated uint32 fds = 2;
  bool tty = 3;
  // ?? way to control if this is PID1? probably not needed
}

message ExitMessage {
  uint32 Code = 1;
}

message FdMessage{
	uint32 fd = 1; // what fd the data was from
	bool eof = 2;  // true if eof was reached
	bytes data = 3;
}

message ResizeMessage{
	uint32 rows = 1;
	uint32 cols = 2;
	uint32 xpixel = 3;
	uint32 ypixel = 4;
}

I added "container" concept atm to support multiple exec processes. Not sure if needed initially but probably better to be safe for future ideas. The complex part of this is that it does not allow reusing the current executor directly, eg. runc needs to be invoked with create/start/exec calls instead of a single run. Or we mark one process as pid1 and then I think we can use run+exec.

Sending pb.Meta inside the giant one-off object is objectively ugly but this is grpc limitation that we can't pass the initial message on streamable endpoints (unless with unsafe context metadata).

[tonistiigi]

These are the changes in the Executor go API proposed in slack

type ProcessInfo {
  Meta Meta
  Stdin io.ReadCloser
  Stdout, Stderr io.WriteCloser
  // tty, resize
}
type Executor interface {
  Run(ctx context.Context, ref string, rootfs cache.Mountable, mounts []Mount, process *ProcessInfo, allowExec true) error
  Exec(ctx, ref string, process ProcessInfo) error
}

Some investigation is needed if both runs and contained support exec without setting the pid1 process. If not then process info passed to Run is not a pointer or first exec becomes pid1.

Outline:

# Approach 1
1. (LLBBridge).NewContainer
2. (Executor).Run(process=nil, allowExec=true)
3. (LLBBridge).ExecProcess()
4. Send InitMessage(meta, fd, tty)
	4.a. Wait for StartedMessage
5. (Executor).Exec(process{meta, fd, tty})
	5.a. Send StartedMessage
6. Send/recv FdMessage/ResizeMessage
7. (LLBBridge).ReleaseContainer
	7.a. Wait for ExitMessage
8. (Executor) - cancels context
	8.a. Sends ExitMessage

# Approach 2
1. (LLBBridge).NewContainer
2. (LLBBridge).ExecProcess()
3. Send InitMessage()
	3.a. Wait for StartedMessage
4. If first InitMessage: (Executor).Run()
   Else: (Executor).Exec()
	4.a. Send StartedMessage
5. Send/recv FdMessage/ResizeMessage
6. (LLBBridge).ReleaseContainer
	6.a. Wait for ExitMessage
7. (Executor) - cancels context
	7.a. Sends ExitMessage

I think the executor changes can be done in a separate PR before the changes in proto API with some unit tests to ease the review.

[coryb]

Runc vs Containerd questions

These seems to be the relevant questions that might influence implementation:
a) Can we create a container without defining pid1?
b) Can we exec into container wihtout pid1 running?
c) Can we attach IO to pid1 after started?

Runc

a) No, config.json must have args where the first arg is validated to be in PATH when container is created (not started)
b) Yes, runc exec will automatically create a pid1 of runc init if nothing else is running
c) No, you cannot create then start a container and then attach IO to pid1. Further if terminal=true in config.json you cannot call runc create, only runc run.

Containerd

a) Yes, you do not have to explicitly define a pid1 (will assume defaults from image)
b) No, container must be running before exec, pid1 is required
c) Yes, IO for pid1 can be attached to after started

Proposal

So it seems for a common implementation we will have to assume "No" to all these questions.

After some discussion with @hinshun this seems like a reasonable approach:

=> NewContainer(ref, rootfs, mounts, network, security) message

• server records this data (in memory, bolt?), containerd or runc is not called at this point

=> ExecProcess.InitMessage(ref, meta, IO) message

• containerd:
  • If first:
    • (containerd.Store).Create(container={meta + recorded data)
    • (containerd.Container).NewTask(IO)
  • If !first:
    • (containerd.Task).Exec(meta, IO)
• runc:
  • If first:
    • (runc).Run(meta + recorded data, IO)
  • If !first:
    • (runc).Exec(meta, IO)

[tonistiigi]

@coryb LGTM