proposal: encoding/json, encoding/xml: support zero values of structs with omitempty

[joeshaw]

Support zero values of structs with omitempty in encoding/json and encoding/xml.

This bites people a lot, especially with time.Time. Open bugs include #4357 (which has many dups) and #10648. There may be others.

Proposal

Check for zero struct values by adding an additional case to the isEmptyValue function:

case reflect.Struct:
        return reflect.Zero(v.Type()).Interface() == v.Interface()

This will solve the vast majority of cases.

(Optional) Introduce a new encoding.IsZeroer interface, and use this to check for emptiness:

Update: I am dropping this part of the proposal, see below.

type IsZeroer interface {
        IsZero() bool
}

Visit this playground link and note that the unmarshaled time.Time value does not have a nil Location field. This prevents the reflection-based emptiness check from working. IsZero() already exists on time.Time, has the correct semantics, and has been adopted as a convention by Go code outside the standard library.

An additional check can be added to the isEmptyValue() functions before checking the value's Kind:

if z, ok := v.Interface().(encoding.IsZeroer); ok {
        return z.IsZero()
}

Compatibility

The encoding.IsZeroer interface could introduce issues with existing non-struct types that may have implemented IsZero() without consideration of omitempty. If this is undesirable, the encoding.IsZeroer interface check could be moved only within the struct case:

case reflect.Struct:
        val := v.Interface()
        if z, ok := val.(encoding.IsZeroer); ok {
                return z.IsZero()
        }
        return reflect.Zero(v.Type()).Interface() == val

Otherwise, this change is backward-compatible with existing valid uses of omitempty. Users who have applied omitempty to struct fields incorrectly will get their originally intended behavior for free.

Implementation

I (@joeshaw) have implemented and tested this change locally, and will send the CL when the Go 1.6 tree opens.

[gopherbot]

CL https://golang.org/cl/13914 mentions this issue.

[gopherbot]

CL https://golang.org/cl/13977 mentions this issue.

[joeshaw]

The empty struct approach is implemented in CL 13914 and the IsZeroer interface is implemented in CL 13977.

In order for them to be reviewable separately they conflict a bit -- mostly in the documentation -- but I will fix for one if the other is merged.

[joeshaw]

In the CLs @rsc said,

I'd really like to stop adding to these packages. I think we need to leave well enough alone at some point.

I see what he's getting at. CL 13977, which implements the IsZeroer interface is clearly an enhancement and adds API to the standard library that needs to be maintained forever. So, I am abandoning that CL and that part of the proposal.

However, I still feel strongly about omitempty with empty structs, and I want to push for CL 13914 to land for Go 1.6.

I use the JSON encoding in Go a lot, as my work is mostly writing services that communicate with other services, in multiple languages, over HTTP. The fact that structs don't obey omitempty is a frequent source of confusion (see #4357 and its many dups and references, and #10648) and working around it is really annoying. Other programming languages do not conform to Go's ideal "zero value" idea, and as a result encoding a zero value is semantically different in JSON than omitting it or encoding it as null. People run into this most commonly with time.Time. (There is also the issue that decoding a zero time.Time does not result in an empty struct, see #4357 (comment) for background on that.)

I think it should be considered a bug that Go does not support omitempty for these types, and although it adds a small amount of additional code, it fixes a bug.

[jeromenerf]

This proposal is marked as unplanned, yet the related bug report #10648 is marked as go1.7.
Is it still being worked /thought on?

[rsc]

To my knowledge, it is not being worked on. Honestly this seems fairly low
priority and will likely miss Go 1.7.

On Sun, Mar 27, 2016 at 12:22 PM Jérôme Andrieux notifications@github.com
wrote:

This proposal is marked as unplanned, yet the related bug report #10648
#10648 is marked as go1.7.
Is it still being worked /thought on?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#11939 (comment)

[jeromenerf]

OK.

This is more of a convenience than a priority indeed.

It can be a pain point when dealing with libs that don't support "embedded structs" as pointer though.

[Perelandric]

I wonder if a low-impact alternative to the IsZeroer interface would be to allow one to return an error called json.CanOmit (or similar) from an implementation of the Marshaler interface. That way the dev is in control of determining what constitutes a zero value, and it doesn't impact other code.

It's not a perfect solution, since one can't add methods to types defined in another package, but this can be worked around to a degree.

Taking the time.Time example:

type MyTime struct {
  time.Time
}

// Implement the Marshaler interface
func (mt MyTime) MarshalJSON() ([]byte, error) {
  res, err := json.Marshal(mt.Time)

  if err == nil && mt.IsZero() {
    return res, json.CanOmit // Exclude zero value from fields with `omitempty`
  }
  return res, err
}

I haven't looked into implementation, but on the surface it would seem like a low-overhead solution, assuming the only work would be to check if an error returned equals json.CanOmit on fields where omitempty was included.

Using errors as a flag is not without precedent in the standard library, e.g. filepath#WalkFunc allows one to return filepath.SkipDir to skip recursion into a directory.