os, internal/poll, runtime: how to use /dev/net/tun on Linux

[zx2c4]

Go 1.12 brought Sysconn() for os.File. In theory that should let us OpenFile on /dev/net/tun, and then use Sysconn() to do all of the TUN-specific ioctls for setting up the device and giving it a name and setting some properties and such. From then out, it's supposed to be a matter of Read, Write, and Close. Since we don't need to call Fd() on the os.File at any point, we gain the benefits of using netpoll (which is epoll behind the scenes).

In addition to allowing the scheduler to make better decisions and not allocating an OS thread for every IO operation, netpoll also lets us call Read in one Go routine and Close in another, and the currently running Read will return immediately with an error saying that it's been closed. This is terrific for shutting down gracefully. To illustrate here's something that does not work as a consequence of using Fd:

        fd, err := os.OpenFile("/dev/net/tun", os.O_RDWR, 0)
        if err != nil {
                log.Fatal(err)
        }
        
        var ifr [unix.IFNAMSIZ + 64]byte
        copy(ifr[:], []byte("cheese"))
        *(*uint16)(unsafe.Pointer(&ifr[unix.IFNAMSIZ])) = unix.IFF_TUN
        
        _, _, errno := unix.Syscall(
                unix.SYS_IOCTL,
                uintptr(fd.Fd()),
                uintptr(unix.TUNSETIFF),
                uintptr(unsafe.Pointer(&ifr[0])),
        )
        if errno != 0 {
                log.Fatal(errno)
        }

        wait := sync.WaitGroup{}
        wait.Add(1)
        go func() {
                var err error
                for {   
                        _, err := fd.Read(b[:])
                        if err != nil {
                                break
                        }
                }
                log.Print("Read errored: ", err)
                wait.Done()
        }()
        time.Sleep(time.Second * 3)
        log.Print("Closing")
        err = fd.Close()
        if err != nil {
                log.Print("Close errored: " , err)
        }
        wait.Wait()
        log.Print("Exiting")

The problem with the above code is that fd.Read(b[:]) never returns after fd.Close() executes, and so the program hangs forever. Thanks to Sysconn in Go 1.12, we can fix that problem like this:

        fd, err := os.OpenFile("/dev/net/tun", os.O_RDWR, 0)
        if err != nil {
                log.Fatal(err)
        }
        
        var ifr [unix.IFNAMSIZ + 64]byte
        copy(ifr[:], []byte("cheese"))
        *(*uint16)(unsafe.Pointer(&ifr[unix.IFNAMSIZ])) = unix.IFF_TUN
        
        var errno syscall.Errno
        s, _ := fd.SyscallConn()
        s.Control(func(fd uintptr) {
                _, _, errno = unix.Syscall(
                        unix.SYS_IOCTL,
                        fd,
                        uintptr(unix.TUNSETIFF),
                        uintptr(unsafe.Pointer(&ifr[0])),
                )
        })
        if errno != 0 {
                log.Fatal(errno)
        }

        wait := sync.WaitGroup{}
        wait.Add(1)
        go func() {
                var err error
                for {   
                        _, err := fd.Read(b[:])
                        if err != nil {
                                break
                        }
                }
                log.Print("Read errored: ", err)
                wait.Done()
        }()
        time.Sleep(time.Second * 3)
        log.Print("Closing")
        err = fd.Close()
        if err != nil {
                log.Print("Close errored: " , err)
        }
        wait.Wait()
        log.Print("Exiting")

This works as expected with regards to that fd.Read(b[:]) getting cancelled. (In Go 1.11, I previously worked around this by manually polling on a cancellation pipe and the tun fd with some pretty gnarly ugliness. I've been eagerly awaiting the Go 1.12 release to stop having to play those games.)

There's a big problem, however: netpoll's use of epoll doesn't seem to agree with the the Linux tun driver's tun_chr_poll. Consider the following program:

package main

import "log"
import "os"
import "unsafe"
import "time"
import "syscall"
import "os/exec"
import "sync"
import "golang.org/x/sys/unix"

func main() {
	fd, err := os.OpenFile("/dev/net/tun", os.O_RDWR, 0)
	if err != nil {
		log.Fatal(err)
	}

	var ifr [unix.IFNAMSIZ + 64]byte
	copy(ifr[:], []byte("cheese"))
	*(*uint16)(unsafe.Pointer(&ifr[unix.IFNAMSIZ])) = unix.IFF_TUN

	var errno syscall.Errno
	s, _ := fd.SyscallConn()
	s.Control(func(fd uintptr) {
		_, _, errno = unix.Syscall(
			unix.SYS_IOCTL,
			fd,
			uintptr(unix.TUNSETIFF),
			uintptr(unsafe.Pointer(&ifr[0])),
		)
	})
	if errno != 0 {
		log.Fatal(errno)
	}

	wait := sync.WaitGroup{}
	wait.Add(1)
	go func() {
		var err error
		c := exec.Command("sh", "-c", "ip link set up cheese && ip a a 192.168.9.2/24 dev cheese")
		c.Start()
		c.Wait()
		exec.Command("sh", "-c", "ping -c 4 -f 192.168.9.1; ip link set down cheese; ip a f dev cheese").Start()
		b := [2000]byte{}
		for {
			n, err := fd.Read(b[:])
			if err != nil {
				break
			}
			log.Printf("Read %d bytes", n)
		}
		log.Print("Read errored: ", err)
		wait.Done()
	}()
	time.Sleep(time.Second * 15)
	log.Print("Closing")
	err = fd.Close()
	if err != nil {
		log.Print("Close errored: ", err)
	}
	wait.Wait()
	log.Print("Exiting")
}

This is supposed to work, but actually the call to Read winds up blocking and not returning any data, and only ever returns upon the call to Close. The above program can be "fixed" by adding fd.Fd() just above the go func() { line, in order to remove fd from netpoll. This, however, incurs the pre-Sysconn-era problem of Close not being cancelable and loosing the nice other benefits of netpoll.

Anybody familiar with netpoll's particular use of epoll interested in taking a look under the hood?

[ianlancetaylor]

The way that netpoll uses epoll is straightforward. It adds descriptors to the epoll descriptor using EPOLL_CTL_ADD with EPOLLIN | EPOLLOUT | EPOLLRDHUP | EPOLLET. It shouldn't be hard to try writing a C program to see how epoll behaves with /dev/net/tun.

[mikioh]

Just to be sure, can you please confirm that:

• "ip tuntap add $devname tun user $username" works well on your node under the test,
• "ip link show" displays the tun interface configured by your program during the test.

[zx2c4]

Yes.

thinkpad ~ # ip tuntap add mode tun name cheese user zx2c4
thinkpad ~ # ip link show dev cheese
150: cheese: <POINTOPOINT,MULTICAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 500
    link/none 

[zx2c4]

Using the same flags as Go's usage as epoll, I'm able to reproduce this in C. Here's the working blocking case as a baseline:

#include <sys/types.h>
#include <sys/epoll.h>
#include <sys/stat.h>
#include <sys/ioctl.h>
#include <net/if.h>
#include <linux/if_tun.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])
{
	char buf[2000];
	ssize_t len;
	int tunfd, ret;
	struct ifreq ifreq = {
		.ifr_name = "cheese",
		.ifr_flags = IFF_TUN
	};
	
	tunfd = open("/dev/net/tun", O_RDWR);
	if (tunfd < 0) {
		perror("open(/dev/net/tun");
		return 1;
	}
	ret = ioctl(tunfd, TUNSETIFF, &ifreq);
	if (ret < 0) {
		perror("ioctl(IFF_TUN)");
		return 1;
	}
	system("ip link set up cheese && ip a a 192.168.9.2/24 dev cheese");
	popen("ping -c 4 -f 192.168.9.1; ip link set down cheese; ip a f dev cheese", "r");
	while ((len = read(tunfd, buf, sizeof(buf))) >= 0)
		printf("Read %ld bytes\n", len);
	return 0;
}

Here's the broken epoll case:

#include <sys/types.h>
#include <sys/epoll.h>
#include <sys/stat.h>
#include <sys/ioctl.h>
#include <net/if.h>
#include <linux/if_tun.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])
{
	char buf[2000];
	ssize_t len;
	int tunfd, efd, ret;
	struct ifreq ifreq = {
		.ifr_name = "cheese",
		.ifr_flags = IFF_TUN
	};
	struct epoll_event event = {
		.events = EPOLLIN | EPOLLOUT | EPOLLRDHUP | EPOLLET
	};
	
	tunfd = open("/dev/net/tun", O_RDWR);
	if (tunfd < 0) {
		perror("open(/dev/net/tun");
		return 1;
	}
	ret = fcntl(tunfd, F_GETFL);
	if (ret < 0) {
		perror("F_GETFL");
		return 1;
	}
	ret = fcntl(tunfd, F_SETFL, ret | O_NONBLOCK);
	if (ret < 0) {
		perror("F_SETFL");
		return 1;
	}
	efd = epoll_create1(0);
	if (efd < 0) {
		perror("epoll_create1");
		return 1;
	}
	ret = epoll_ctl(efd, EPOLL_CTL_ADD, tunfd, &event);
	if (ret < 0) {
		perror("epoll_ctl");
		return 1;
	}
	ret = ioctl(tunfd, TUNSETIFF, &ifreq);
	if (ret < 0) {
		perror("ioctl(IFF_TUN)");
		return 1;
	}
	system("ip link set up cheese && ip a a 192.168.9.2/24 dev cheese");
	popen("ping -c 4 -f 192.168.9.1; ip link set down cheese; ip a f dev cheese", "r");
	for (;;) {
		len = read(tunfd, buf, sizeof(buf));
		if (len < 0 && errno == EAGAIN) {
			ret = epoll_wait(efd, &event, 1, -1);
			if (ret < 0) {
				perror("epoll_wait");
				return 1;
			}
			continue;
		}
		if (len < 0)
			break;
		printf("Read %ld bytes\n", len);
	}
	return 0;
}

[zx2c4]

Interestingly, it appears that removing EPOLLET fixes things. That's not surprising as level triggering is basically the same as ordinary poll.

[mikioh]

it appears that removing EPOLLET fixes things

Sounds like you need to find out some good way to accommodate either a) blocking I/O w/ level-triggered notification, or b) non-blocking I/O w/ edge-triggered notification; the current runtime-integrated network poller is designed for just the latter.

If marking a tun/tap device file with non-blocking does make it possible to work together with the current runtime-integrated network poller, well, it's unlikely, tun_ring_recv in drivers/net/tun.c always returns EAGAIN when the argument noblock is true.

A naive fix might be to make the epoll registration adaptive by referring to the target file capability for non-blocking I/O.

[zx2c4]

tun_ring_recv in drivers/net/tun.c always returns EAGAIN when the argument noblock is true.

Are we reading the same source? It returns 0 and with the buffer if noblock is true and a buffer is available. Otherwise it returns EAGAIN:

static void *tun_ring_recv(struct tun_file *tfile, int noblock, int *err)
{       
        DECLARE_WAITQUEUE(wait, current);
        void *ptr = NULL;
        int error = 0;
        
        ptr = ptr_ring_consume(&tfile->tx_ring);
        if (ptr)
                goto out;
        if (noblock) {
                error = -EAGAIN;
                goto out;
        }
        
        //[...]

out:    
        *err = error;
        return ptr;
}

[ianlancetaylor]

If /dev/net/tun doesn't support EPOLLET, then I don't see a reasonable way to make it work with Go's poller.

[mikioh]

@zx2c4,

A workaround: https://play.golang.org/p/iu6ayVT3Yfe