Skip to content

context: ease debugging of where a context was canceled? #26356

Closed
@matthewceravolo

Description

@matthewceravolo

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/matthew/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/matthew/work"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build356255000=/tmp/go-build -gno-record-gcc-switches"

What did you do?

used context.WithTimeout() to make requests to google calendar api and outlook calendar api

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

What did you expect to see?

Making requests using contexts with timeouts should cancel when the timeout is reached

What did you see instead?

Contexts with timeouts are instantly failing with "context canceled" even though the timeout is set to time.Minute. The error goes away if I remove the timeout context and use one without any limit. It also seems to be transient to some extent

Activity

added this to the Unreleased milestone on Jul 12, 2018
changed the title [-]x/net context canceled[/-] [+]context: context canceled[/+] on Jul 12, 2018
ianlancetaylor

ianlancetaylor commented on Jul 12, 2018

@ianlancetaylor
Contributor

Sorry, but there isn't enough information here to say anything useful. Please show us your code, or tell us how to recreate the problem. Thanks.

added
WaitingForInfoIssue is not actionable because of missing required information, which needs to be provided.
NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.
on Jul 12, 2018
matthewceravolo

matthewceravolo commented on Jul 12, 2018

@matthewceravolo
Author
import (
	"net/http"
	"os"
	"time"

	"golang.org/x/net/context"
	"golang.org/x/oauth2"
	"golang.org/x/oauth2/google"
)

var (
	backgroundContext = context.Background()
	oauthConfig       = &oauth2.Config{
		ClientID:     os.Getenv("GOOGLE_PROJECT_CLIENT_ID"),
		ClientSecret: os.Getenv("GOOGLE_PROJECT_SECRET_KEY"),
		Endpoint:     google.Endpoint,
	}
)

type calendarClient struct {
	*calendar.Service
	requestTimeout time.Duration
}

// NewCalendarClient creates a new authenticated Google Calendar client.
func NewCalendarClient(refreshToken string) (wf.Client, error) {
	client, err := newClient(refreshToken)
	if err != nil {
		return nil, err
	}

	service, err := calendar.New(client)
	if err != nil {
		return nil, err
	}

	return &calendarClient{Service: service, requestTimeout: time.Minute}, nil
}

func newClient(refreshToken string) (*http.Client, error) {
	contextWithTimeout, cancel := context.WithTimeout(backgroundContext, time.Minute)
	defer cancel()
	return oauthConfig.Client(contextWithTimeout, &oauth2.Token{RefreshToken: refreshToken}), nil
}

func (client *calendarClient) GetCalendars() (*calendar.CalendarList, error) {
	contextWithTimeout, cancel := context.WithTimeout(backgroundContext, time.Minute)
	defer cancel()
	return client.CalendarList.List().Context(contextWithTimeout).Do()
}

The issue is that when we call GetCalendars for example, it will instantly return "context canceled". When I use an empty context with no timeout, everything works as expected. This all began failing in the past couple of days since updates were made in the /x/net package

meirf

meirf commented on Jul 13, 2018

@meirf
Contributor

I think the behavior you're seeing is expected. I don't see a Go bug.

NewClient (called by oauthConfig.Client):

The returned client is not valid beyond the lifetime of the context.

Your code has:

func newClient(refreshToken string) (*http.Client, error) {
    contextWithTimeout, cancel := context.WithTimeout(backgroundContext, time.Minute)
    defer cancel()
    return oauthConfig.Client(contextWithTimeout, &oauth2.Token{RefreshToken: refreshToken}), nil
}

The defer cancel() causes the context to be cancelled before the client even gets to calendar.New.

"When I use an empty context with no timeout, everything works as expected. This all began failing in the past couple of days since updates were made in the /x/net package." You are describing to things that might have changed in your code (no longer using empty context, changes in x/net). If my theory above is correct: if you undo the x/net changes (to only change one variable) you'll see that any non-empty context will still have the problem you're seeing.

bcmills

bcmills commented on Jul 13, 2018

@bcmills
Contributor

I've seen this sort of issue crop up several times now.

I wonder if context.Context should record a bit of caller information (say, the PC of the immediate calling function) in context.WithTimeout and in calls to the CancelFunc returned by context.WithCancel. Then we could add a debugging hook to interrogate why a particular context.Context was cancelled.

Perhaps something along the lines of:

package context

// A DoneReasoner describes the reason a Context is done.
// The Context implementations returned by this package implement DoneReasoner.
type DoneReasoner interface {
	DoneReason() string
}

// DoneReason returns a human-readable description of the reason that ctx.Done() is closed,
// or the empty string if ctx.Done() is not closed.
func DoneReason(ctx context.Context) string {
	select {
	case <-ctx.Done():
	default:
		return ""
	}
	if r, ok := ctx.(DoneReasoner); ok {
			return r.DoneReason()
	}
	return ctx.Err().Error()
}
bcmills

bcmills commented on Jul 13, 2018

@bcmills
Contributor

(CC @Sajmani @bradfitz for Context debuggability.)

changed the title [-]context: context canceled[/-] [+]context: ease debugging of where a context was canceled?[/+] on Jul 13, 2018
matthewceravolo

matthewceravolo commented on Jul 13, 2018

@matthewceravolo
Author

@meirf I think there was a misunderstanding in how I described the problem. This code has always been using two contexts like I described but only in the past two days since we pulled updates to the vendor package did it start failing. There were not any changes whatsoever related to any of this code for several months besides pulling in the package update.

Only after we pulled in the update and everything started failing, we switched to contexts without timeouts (still multiple) and started seeing things succeed. If your explanation is correct, the defer cancel was not previously working as expected

However, I believe your explanation is incorrect because the error message is returned when GetCalendars is called, not on client retrieval. We always receive a calendarClient back, so calendar.New is being called correctly

changed the title [-]context: ease debugging of where a context was canceled?[/-] [+]context.WithTimeout is returning context canceled immediately[/+] on Jul 13, 2018
ianlancetaylor

ianlancetaylor commented on Jul 13, 2018

@ianlancetaylor
Contributor

Do you happen to have a record of which revision of the context package you were using previously, and which one you updated to?

86 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

FrozenDueToAgeNeedsFixThe path to resolution is known, but the work has not been done.

Type

No type

Projects

No projects

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @bradfitz@rsc@jtolio@nya3jp@mikeschinkel

      Issue actions

        context: ease debugging of where a context was canceled? · Issue #26356 · golang/go