proposal: net: implement fmt.Formatter on net.IP to expand IPv6 addresses

[nemith]

net.IP types implement a fmt.Stringer interface which will output the human representation for IPv4 or IPv6 addresses. In particular the IPv6 address follows RFC5952 which will compress/shorten IPv6 addresses as much as possible.

However this isn't always the desired output for an IPv6 address. Sometimes you want a fully expanded IPv6 address for comparability with other systems/languages.

This proposal is to add fmt.Formatter implementation to the IPv6 address so that when the IP address is formatted with the v verb with a + flag that the expanded IPv6 address is outputted instead of the compressed version.

Examples:

ip := net.ParseIP("2001:db8::1")
fmt.Printf("%s", ip)  // prints "2001:db8::1"
fmt.Printf("%v", ip)  // prints "2001:db8::1" (same as %s)
fmt.Printf("%+v", ip)  // prints "2001:0db8:0000:0000:0000:0000:0000:0001"

For IPv4 %s, %v, %+v will all be the same.

[mikioh]

[...] RFC5952 which will compress/shorten IPv6 addresses as much as possible.

Well, the heart of RFC 5952 is "canonicalization" or "avoiding ambiguities on the visual perception of IP literals" for "human beings"; at that point frustrated (mostly network ops) people made a uniform of literal IP address to drop a bit troublesome flexibilities from RFC 4291 and they were happy to make their own scripts for their business systems. Therefore ParseIP accepts any form described in RFC 4291 and RFC 5952 and IP.String returns only the form in RFC 5952.

[mikioh]

Sometimes you want a fully expanded IPv6 address for comparability with other systems/languages.

I'm still not sure whether picking up one form in RFC 4291 really makes people happy. For example, what happens when the systems/languages require "every single alpha-hex digit in the IP literal must be upper-case" and vice versa?

[nemith]

Although I agree with RFC5952 the particular system I am working with using using the fully expanded form of the IP address instead of the compressed version.

This is why I am suggesting using the Formatter interface to control the output.

For lower case and uppercase a simple call to strings.ToUpper() would work just fine. Expanding an IP address requires parsing it first.

[mikioh]

For lower case and uppercase a simple call to strings.ToUpper() would work just fine. Expanding an IP address requires parsing it first.

I disagree. If we really need a new knob to control the output form of literal IP address, it should be a custom formatter, never squats in the existing verbs in the package fmt, and never requires additional use of the package strings. Otherwise, I don't see any good reason to introduce a new API instead of using fmt and strings packages.

[nemith]

@mikioh I am not sure what you are disagreeing with. What you are describing seems to be exactly what I am proposing.

never squats in the existing verbs in the package fmt

What is a squats or a never squats?

and never requires additional use of the package strings

My only point is that uppercasing and lowercasing an IPv6 address today is simple since it's just string manipulation. Fully expanding an IPv6 address is not available in using the stdlib and is what this proposal is trying to address. If you can find a nice way of using the existing verbs to do upper or lowercase as an addition to this proposal I am sure it would be welcomed, but there is nothing clear to me.

I don't see any good reason to introduce a new API instead of using fmt and strings packages.

No new API has been proposed here unless you meaning something else?

[mikioh]

@nemith,

What you are describing seems to be exactly what I am proposing.

Well, nope, I'm still doing the assessment of API, though I don't disagree with using a custom formatter might make sense.

What is a squats or a never squats?

If we really need to control the output form of IP literal, we should have new verbs for it for distinction.

No new API has been proposed here unless you meaning something else?

In other words, what's wrong with the following:

return strings.ToUpper(fmt.Sprintf("%02x%02x:%02x%02x:%02x%02x:%02x%02x:%02x%02x:%02x%02x:%02x%02x:%02x%02x",
                ip[0], ip[1], ip[2], ip[3], ip[4], ip[5], ip[6], ip[7], ip[8], ip[9], ip[10], ip[11], ip[12], ip[13], ip[14], ip[15]))

If we introdce and keep maintaining a new API, there needs to be a rationale behind the API. I still see your proposal is just for some specific application by using one of the multiple forms in RFC 4291.

[nemith]

In other words, what's wrong with the following.

I am doing something like this today however there is more to it. You need to check the IP version and deal with IPv4 vs IPv6 differently. It is somewhat trivial to implement so there is a point there.

I still see your proposal is just for some specific application by using one of the multiple forms in RFC 4291.

Although there are a lot of valid representations of a IPv6 address nothing really makes sense besides fully expanded and fully compressed. You can argue that you don't want to support an expanded IP address, but painting it as just one of "multiple forms" is a bit dismissive when RFC4291 is defining the rules for how IP address are parsed and not really a description of valid output forms or their actual use in network code.

To state it in another way. No one in thier right mind would store an IPv6 address as a string in anything but it's compressed form or fully expanded form.

To look at other implementations:

Python 3 has the ipaddress module which parses IP addresses. There are two properties exposed for both IPv4Addrss and IPv6Address: compressed and exploded. For IPv4 they do the same thing and compressed is similar to the behavior of net.IP.String() and exploded is the version I am proposing.

Ruby has the ipaddress module with both compress and expand methods on the IPv6 object. https://www.rubydoc.info/gems/ipaddress/0.8.0/IPAddress/IPv6#compress-class_method

C++ Folly has a method called toFullyQualified() https://github.com/facebook/folly/blob/master/folly/IPAddress.h#L414

All these implementations define similar verbs for IPv6 and IPv4 for both compressed and expanded versions. None of them define any other format than compressed or expanded.

[mikioh]

All these implementations define similar verbs for IPv6 and IPv4 [...] None of them define any other format than compressed or expanded.

Thanks for the investigation. However, I'm not sure "other languages only do this" is enough for a rationale behind the API.

I am doing something like this today however there is more to it. You need to check the IP version and deal with IPv4 vs IPv6 differently. It is somewhat trivial to implement so there is a point there.

I don't agree with the part "somewhat trivial to implement." The IP address identification is actually not handy. It has several pitfalls and really not good for people who just need an opaque (also well-cooked) network-layer identifier for their business. The complexity of IP address identification could be a rationale for introducing a new API using a custom formatter.

My suggestion as follows:

/*
The verbs for printing IP address literal:

        %ip     a default representation format,
                with lower-case letters
        %ip4    an IPv4 dotted-decimal format
        %ip6    an IPv6 colon-separated hexadecimal format
        %ip4e   an IPv4-embedded IPv6 address representation format,
                with the 96 bits prefix when no decimal suffix follows;
                the suffix indicates the variable length of the prefix
                in bits, and 32, 40, 48, 56, 64 and 96 are operational,
                for example, %ip4e32, %ip4e64

        %IP, %IP4, %IP6, %IP4e
                a default representation format, with upper-case letters

        +       the plus flag (%+ip) makes each colon-separated hexadecimal
                digits in IPv6 address to be complete four letters

Examples:

With the IPv4 address 192.0.2.1
        Printf("%ip", v):       192.0.2.1
        Printf("%ip4", v):      192.0.2.1
        Printf("%ip6", v):      !ip6(BADADDR)
        Printf("%ip4e",v):      !ip4e(BADADDR)
        Printf("%+ip", v):      192.0.2.1
        Printf("%+IP4", v):     192.0.2.1
        Printf("%+IP6", v):     !+IP6(BADADDR)

With the IPv6 address 2001:db8::1
        Printf("%ip", v):       2001:db8::1
        Printf("%ip4", v):      !ip4(BADADDR)
        Printf("%ip6", v):      2001:db8::1
        Printf("%ip4e",v):      2001:db8::0.0.0.1
        Printf("%+ip", v):      2001:0db8:0000:0000:0000:0000:0000:0001
        Printf("%+IP4", v):     !+IP4(BADADDR)
        Printf("%+IP6", v):     2001:0DB8:0000:0000:0000:0000:0000:0001

With the IPv4-mapped IPv6 address ::ffff:192.0.2.1
        Printf("%ip", v):       192.0.2.1
        Printf("%ip4", v):      192.0.2.1
        Printf("%ip6", v):      ::ffff:192.0.2.1
        Printf("%ip4e",v):      !ip4e(BADADDR)
        Printf("%+ip", v):      192.0.2.1
        Printf("%+IP4", v):     192.0.2.1
        Printf("%+IP6", v):     0000:0000:0000:0000:0000:ffff:C000:0201

With the IPv4-embedded IPv6 address 2001:db8:122:344::192.0.2.33
        Printf("%ip", v):       2001:db8:122:344::c00:221
        Printf("%ip4", v):      !ip4(BADADDR)
        Printf("%ip6", v):      2001:db8:122:344::c00:221
        Printf("%ip4e",v):      2001:db8:122:344::192.0.2.33
        Printf("%+ip", v):      2001:0db8:0122:0344:0000:0000:0c00:0221
        Printf("%+IP4", v):     !+IP4(BADADDR)
        Printf("%+IP6", v):     2001:0DB8:0122:0344:0000:0000:0C00:0221

See RFC 4291, RFC 5952 and RFC 6052 for further information
*/

Not necessary to implement all verbs at the same time but the IP-literal verbs should work well together with the existing verbs in the package fmt because the type IP is "a slice of bytes"; interfering in user code that scrapes the underlying type is not a good idea.

[nemith]

Thanks for the investigation. However, I'm not sure "other languages only do this" is enough for a rationale behind the API.

It wasn't meant to be but just contrast with what is found out there in other languages/packages.

I like your suggestions for the %ip* verb. I am on the fence on if this should be in the stdlib or provided by an external package which is probably the key part of the discussion.

[rsc]

net can't import fmt, sorry - we want to keep net as a much lower level in the dependency graph - so nothing in net can implement fmt.Formatter (which requires reference to fmt.State).

Also formatter can't do '%ip', because fmt stops parsing a %anything at the first letter (so %anything is %a followed by literal nything). Even %ip vs %ip4 is ambiguous - is %ip4 just %ip followed by "4"?

[mikioh]

net can't import fmt, sorry

Yup, even though the package context imports "fmt" and the package net depends on the package context.

Also formatter can't do '%ip', because fmt stops parsing

Yup, the type of the format verb in the package fmt is rune and we cannot change it; so my suggestion above is also a message for someone who wants to implement own printer for net.IP (like golang.org/x/text/message) in an external package.