fmt: support bounded buffers (Snprint, Snprintf, etc.)

[therc]

Right now, printing large compound objects with %v or %#v can be very expensive, in terms of both CPU and RAM. It would be great to have a way for callers to tell formatting code to stop after N runes have been output. The current alternative is formatting everything anyway, only to discard most of the work after the call returns.

[josharian]

Haven't confirmed, but I'd guess that you can accomplish this now by using fmt.Fprintf into a byte slice with a Write method that rejects data after the buffer is full.

[martisch]

@josharian:
afaik that wont work since fmt only ever writes once https://github.com/golang/go/blob/master/src/fmt/print.go#L182 and that i think was an intentional design decision.

[josharian]

Ah. Bummer. This is not the first time I've bumped up against that particular design decision and been frustrated by it.

[quentinmit]

Can't you specify a precision to limit the output? e.g. %.80v to limit the output to 80 characters? I don't know if this is actually smart about stopping early, though.

[therc]

I think the precision gets applied to the individual fields...

[therc]

See https://play.golang.org/p/2AAboYpJt0

[jech]

Couldn't this be solved by defining a LimitedWriter interface with a Remains() int64 method? (Perhaps LimitedWriter is not the best name for this, due to the asymmetry with LimitedReader.) I'd personally find it jarring if there were an Snprintf but no efficient way to do Fnprintf.

[martisch]

Regardless of the design of the writer the current design of fmt would first fill a buffer with the to be written output. Which does not seem to be a solution to having less ram and CPU use. For fmt to stop generating output after x runes directly the package would need to be redesigned to keep track of how much space is left and then stop in the respective format methods or calling them. Even then fmt has no influence of how e.g. string methods would consume cpu and ram resources and how much output they generate before trying to write it to the internal buffer.

It is currently possible to implement the https://golang.org/pkg/fmt/#GoStringer or https://golang.org/pkg/fmt/#Formatter interface for custom types that could keep track of generated runes and stop generating output themselves if a limit is reached. Formatter can use Precision() to see what precision was set.

To implement Snprintf or Fnprintf in a way that limits also cpu and ram resources used would seems to me to be a larger change and would also have a performance impact for non limited operations.

Just my observations. If there is a good design and consensus as well as proved need to have a limiting print function i am happy to work or help implementing them.

[jech]

I was thinking of fmt doing something like

wl, ok := w.(LimitedWriter)
if ok {
    n := wl.Remains()
    // do something smart since we know that we'll only be outputting n bytes/runes/whatever
}

[martisch]

@jech i though that this would be the intended use of the LimitedWriter. And smart is likely to mean passing an extra parameter down to every format function, check it, and stop writing to the internal buffer if limit reached. fmt can not limit what handlemethods (Stringer,Formatter, ... ) will do and there is no way to communicate the limit to e.g. a String method. Also not sure what should happen if fmt than tries to write to the writer but someone else has taken up some of the remaining capacity. I guess Fnprintf would be fine without introducing LimitedWriter. The question i had is more about how the functionality would be designed to work with all the internal formatting functions.

[minux]

Actually truncating the output is not very useful either,
I'd suggest a go-gettable package for configurable formatting
that ignores or abbreviate a field if it's too large or nesting
too deep, to make the output look like this:

{s: "ABC ... (2343 chars omitted)", ptr: &D{ /* omitted */ }, ...}

I don't think such functionality belong in fmt though.

[bradfitz]

@jech, I just came to propose the same thing as your io.LimitedWriter check. fmt already depends on io, so that's fine.

Want to send a change for Go 1.10?