cmd/trace: visualize time taken by syscall

[aktau]

Currently (Go 1.20-dev), syscalls as processed by go tool trace from a runtime/trace profile visualize as points in time, being about 1 pixel wide no matter how long they took (see reproducer below). It would be nicer if they were slices, to visualize how much time is spent in (conceptually blocking) syscalls for a given goroutine.

Somewhere in the last release cycle (1.19), I tried making the obvious patch to cmd/trace:

                case trace.EvGoSysCall:
-                       ctx.emitInstant(ev, "syscall", "")
+                       if ev.Link != nil {
+                               ctx.emitSlice(ev, "syscall")
+                       } else {
+                               ctx.emitInstant(ev, "syscall", "")
+                       }

And I remembered it didn't work: most (all?) EvGoSysCall events don't have a link to the finish event. But before filing this proposal I tried patching up cmd/trace from 1.20 and to my surprise it did work. I assume something changed in Go 1.20 and now more (most?) EvGoSysCall events have a link, which is great. Debugging output of my patched go tool trace:

...
2022/12/14 05:42:10 syscall event with link (6/12):
    101597987   GoSysCall       p=1     g=1     off=2876        stkID=22        seq=0
 -> 101613258   GoSysExit       p=1000003       g=1     off=264 stkID=0 seq=0 { g=1 seq=17 ts=0 }
 -> 101613376   GoStart p=0     g=1     off=270 stkID=0 seq=0 { g=1 seq=0 }
 -> 101619219   GoSysBlock      p=0     g=1     off=283 stkID=0 seq=0
2022/12/14 05:42:10 syscall event with link (6/13):
    101618055   GoSysCall       p=0     g=1     off=279 stkID=22        seq=0
 -> 101633491   GoSysExit       p=1000003       g=1     off=297 stkID=0 seq=0 { g=1 seq=19 ts=0 }
 -> 101633634   GoStart p=0     g=1     off=303 stkID=0 seq=0 { g=1 seq=0 }
 -> 101637671   GoSysBlock      p=0     g=1     off=315 stkID=0 seq=0
...

Resulting trace view comparison:

Reproducer:

The program that generated this trace was:

package main

import (
        "fmt"
        "os"
        "runtime/trace"
        "time"
)

func main() {
        if err := rmain(); err != nil {
                fmt.Fprintf(os.Stderr, "error: %v", err)
                os.Exit(1)
        }
}

func rmain() error {
        if err := trace.Start(os.Stdout); err != nil {
                return err
        }
        defer trace.Stop()
        f, err := os.Open("/dev/urandom")
        if err != nil {
                return err
        }
        var buf [1024 * 1024 * 1024]byte
        for i, last := 0, time.Now(); i < 5; i++ {
                fmt.Fprintf(os.Stderr, "start...")
                n, err := f.Read(buf[:])
                if err != nil {
                        return err
                }
                if n == 0 {
                        return fmt.Errorf("read: returned empty slice (%d, %v)", n, err)
                }
                now := time.Now()
                fmt.Fprintf(os.Stderr, "took %v\n", now.Sub(last))
                last = now
        }

        return nil
}

Running it:

$ go run longsyscall.go > longsyscall.trace
start...took 2.203378744s
start...took 2.207477581s
start...took 2.211924562s
start...took 2.202174787s
start...took 2.219617454s

[dominikh]

Here are the notes I took on Go runtime tracing and syscall behavior. At the time I looked at Go 1.19's runtime, but I don't think anything of importance has changed in 1.20. I hope this helps.

When user code invokes a syscall, the runtime emits an EvGoSysCall event. If the syscall returns before sysmon finds
the P that's blocked in the syscall, we'll not see any other events. This is a fast path, avoiding a bunch of
scheduling, allowing the goroutine to resume immediately after the syscall returns, having retained its P and M.

If sysmon does find the P, it retakes the P from the M and the runtime emits GoSysBlock, followed by ProcStop. When a
blocking syscall returns, we get a EvGoSysExit event. The EvGoSysExit event gets emitted right before the G resumes
running. However, to maintain accurate timing, the event carries the original timestamp of when the syscall returned,
and the trace parser uses this to reorder the event. [...]

One implication of sysmon detecting blocked syscalls is that the duration between EvGoSysCall and EvGoSysBlock should
be attributes to the syscall, too. This is the time between starting the syscall and Go figuring out that it's
blocking. [...]

Sysmon runs at most every 20 μs, but it will run considerably less often if it thinks there's nothing to do, up to 10
ms. In testing, even in busy programs (with GOMAXPROCS restricted to low values), sysmon sometimes decides to sleep
for 5 ms. A loop calling syscall.Nanosleep, sleeping 10 ms each time, will have EvGoSysBlock spans that last from ~10
ms to 5 ms, with the other 5 ms being the time between EvGoSysCall and EvGoSysBlock.

Another issue is that when a syscall returns fast enough we'll not be provided with information on how long the
syscall took. In the worst case, a fast syscall could've run for 10 ms, and to us it'll look like 10 ms of user code.

One special case is syscall.RawSyscall, which doesn't emit any tracing events whatsoever, and AFAIU isn't handled by
sysmon. That means these syscalls can block Ps and Ms, but we have no insight into it happening.

[prattmic]

cc @golang/runtime @mknyszek

[ianlancetaylor]

I'm not sure this has to be a proposal. Are there any API changes here, or cases where this might potentially break an existing program? Thanks.

[aktau]

There aren't. But I didn't know which template would be appropriate. I doubted between "proposal" and "bug". While this is actually more of a feature request, between proposal and bug I'd choose the former. Which one should I have chosen?

[ianlancetaylor]

The "Bugs" template I guess. I considered changing it to "Bugs or feature requests", but that would lead too many people away from the "Proposal" template appropriate for API changes. So not sure how to make this better.