std.fmt: Improve numeric options, simplify custom formatters, reduce complexity and more

[castholm]

I have had some ideas for std.fmt for a while now but I've been having trouble figuring out how to present them as concrete proposals. The following is an attempt at summarizing a few of them:

Summary of current problems

• Numeric formatting options are too limited and conflated with generic alignment options.
• It is difficult for users to implement custom formatters correctly (in fact, most of the formatters in std fail at this task).
• Rarely used features like named placeholder options increase the overall complexity of std.fmt.
• std.fmt should deal in raw bytes only but some parts of it currently deal in Unicode scalar values/UTF-8 sequences.

Summary of proposed solutions

• Break out numeric formatting options from generic alignment options and instead make them part of relevant base specifiers like d or e.
• Add more numeric formatting options such as for controlling positive signs, zero-padding or 0x prefixes (among other), to make it easier to format numbers correctly for tasks such as pretty-printing or code generation.
• Remove the options: std.fmt.FormatOptions parameter from custom format formatter functions and instead process alignment generically, separately from formatting through clever use of std.io.countingWriter, to make it much easier for users to correctly implement custom formatters.
• Remove named placeholder options to reduce complexity (this use case is better handled by custom formatters).
• Remove the u specifier (better handled by a formatter from the std.unicode namespace), clarify that the s and c specifiers output bytes verbatim, redefine fill to be a literal byte and redefine width to be in bytes.

In more detail

Numeric formatting options are too limited

(Related: #14436, #19488)

Currently, the only available numeric formatting option is precision, which controls the precision of and number of digits in the fractional part of a floating-point number. We can quickly think of a few other properties a user might want to control when formatting a number:

• The minimum number of digits the number should be zero-padded to.

  There is currently no way to correctly pad a number with leading zeroes such that the sign is written in the correct place. Code like std.debug.print("{:0>5}\n", .{-123}) prints 0-123 instead of the expected -0123 or -00123.

• Whether a positive number should have a plus sign.

  It is currently possible to format a positive integer with a plus sign, but how to do this is very obscure: You need to specify a width of 1 or greater, and the integer type must be signed. std.debug.print("{d:1}\n", .{@as(i32, 123)}) prints +123. Unsigned integers and floating-point numbers, however, can currently not be formatted with a plus sign.

• Whether a hexadecimal number should be prefixed with 0x.

  It is currently not possible to format hexadecimal (or binary or octal) numbers with a prefix. In some situations you may be able to get around this by using a placeholder string like 0x{x}, but this will not work for negative values (-77 would produce 0x-4d) or when left-padding is involved.

There may be other options worth considering, but this should hopefully demonstrate that just precision is probably not enough and that width is not a suitable substitute for zero-padding.

Which leads us to the next point...

Numeric formatting options are conflated with alignment options

The four std.fmt.FormatOptions are fill, alignment, width and precision. This is a bit strange, because the first three deal with alignment and are always applicable no matter which type of value is being formatted, while precision is only relevant for numbers.

Numeric formatting (positive sign, leading zeroes, etc.) is a different concern from alignment; "zero-pad a number to a minimum number of digits" is different from "right-align a string to a minimum width by left-padding with the character 0" and there is currently no way for users to simultaneously zero-pad and right-align a number.

It is also a bit funny that a nonsensical precision option used with a non-numeric specifier like in {s:.3} is not an error.

I think it would make sense to break out precision and other future numeric formatting options from the generic alignment options specified after the : and instead make them part of the base specifiers (e.g. d or e) themselves. In other words, today's placeholder {d: >10.3} might become {d.3: >10}.

This also opens up the door for specifier-specific options; for example, an option specifying whether to prefix the number with 0x makes sense for x (similarly for b or o) but not for d and should be a compile error for the latter.

With std.fmt.FormatOptions reduced to only the three alignment-related options, we can move on to the third point...

It is difficult to implement custom format formatter functions correctly

Custom format formatter function currently have the following signature:

pub fn format(
    self: ?,
    comptime fmt: []const u8,
    options: std.fmt.FormatOptions,
    writer: anytype,
) !void

The options parameter of type std.fmt.FormatOptions specifies the fill character, alignment, minimum width and numeric precision, corresponding to the options passed after the colon in the placeholder string. {:_>9.3} is parsed as .{ .fill = '_', .alignment = .right, .width = 9, .precision = 3 }.

The problem is, most custom formatters (both in std and in external packages) completely ignore these options:

std.debug.print("{s:_>20}\n", .{"hello"});
std.debug.print("{:_>20}\n", .{std.fmt.fmtSliceHexLower("hello")});
std.debug.print("{:_>20}\n", .{std.SemanticVersion.parse("1.2.3") catch unreachable});

_______________hello
68656c6c6f
1.2.3

One could argue that the onus is on the custom formatters to correctly implement padding and that it is a bug that formatters like fmtSliceHexLower or SemanticVersion.format don't handle padding.

I will instead point out that padding could be trivially handled in the main std.fmt.format function, without burdening custom formatters with the task of implementing it, simply by writing in two passes; first to a std.io.countingWriter(std.io.null_writer) to determine the width of the unpadded string, then again to the real writer, padding the difference on either side as needed. Left-alignment only requires a single pass to a std.io.countingWriter(writer).

With fill, alignment and width handled generically, the remaining option would be precision. But with that one removed by the above sub-proposal, we are left with no options and can remove the std.fmt.FormatOptions parameter, simplify the format signature to

pub fn format(
    self: ?,
    comptime fmt: []const u8,
    writer: anytype,
) !void

which makes it much easier for users to implement correctly.

(As a side note, the fmt argument here should really be renamed specifier or spec so that it doesn't get mixed up with the fmt string itself.)

Remove named placeholder options

Did you know that the following is possible?

var width: usize = 10;
var precision: usize = 3;
std.debug.print("{d:_>[1].[2]}\n", .{ @as(f32, 1.23456789), width, precision });

That's correct; certain placeholder options like width and precision don't have to be specified literally but can also be resolved at runtime by specifying the name of a field of args.

This is a fairly obscure feature which increases the overall complexity of std.fmt. It is also limited to only width and precision; other options like fill or alignment must be specified literally and can not be resolved at runtime.

Instead of putting all of this complexity in the parsing and handling of the placeholder string itself, runtime control of formatting options is probably better handled by custom formatters, which are not only more flexible but also make the intent of such code more immediately visible and explicit to readers. To help users with the task of runtime-controlled aligned formatting, the std.fmt namespace could expose a formatter function for this purpose.

Remove any notion of Unicode-awareness from std.fmt

(Related: #18536 (comment), 2d9c479, #234)

Simple: std.fmt should not be Unicode-aware and should deal in raw bytes only, for simplicity. Therefore,

• the u "format u21 as UTF-8 sequence" specifier should be removed (better handled by a formatter from std.unicode),
• the s and c specifiers should clarify that they output (sequences of) bytes verbatim, without any sort of replacement or transformation,
• the width placeholder option should clarify that it controls the minimum width in bytes (not code points, grapheme clusters or some other unit of measure), and
• the fill placeholder option should clarify that it is a literal byte repeated verbatim to pad out the string.

Applications that need powerful Unicode-aware formatting should use a different third-party package.

Other considerations

std.fmt currently generates a lot of code which is undesirable and can be problematic for constrained embedded targets. These problems are described in great detail in #9635. It's important that the above suggestions, if applied, do not negatively affect code size, compile times or runtime performance.

[Paul-Dempsey]

Something to consider: If alignment is removed from numeric formatters, how does one achieve decimal alignment?

   123.456
    23.000
    -1.251

Decimal alignment is common in monetary domains of application.

[mnemnion]

I think it would make sense to break out precision and other future numeric formatting options from the generic alignment options specified after the : and instead make them part of the base specifiers (e.g. d or e) themselves. In other words, today's placeholder {d: >10.3} might become {d.3: >10}.

This is a good proposal. The right side of the : is hard to learn because of how much goes on there, and precision applies to only a few format types.

This also opens up the door for specifier-specific options; for example, an option specifying whether to prefix the number with 0x makes sense for x (similarly for b or o) but not for d and should be a compile error for the latter.

Rather than special-casing a few prefixes, a general solution would be to allow one level of nesting. So for e.g. a right-aligned hex column of width sixteen, {0x{x} >16}. This would also work for {${d.2} >6}: a six column, right-aligned decimal, with two places of precision, preceded by $. As long as nesting wasn't to arbitrary depth, this would be fine, I think, since nestedness would be known before the second } was found, so the material to the left of the inner braces would be interpreted as a prefix even if it otherwise looked like a format string.

It's better to keep {u}, Zig has Unicode character literals and therefore std.fmt should provide a way to turn them into codepoints. The clarification in the documentation that strings are byte-addressed... can't hurt, but I don't know who is going to be surprised that a []u8 is indexed per byte.

fill should be read as any Unicode scalar value. There's no reason to limit fills to ASCII (which is the de facto result of limiting it to a single byte), that's a bit chauvinistic and I like to think we've moved past that. An author who wishes to pad with U+3000, "Ideographic Space", should be empowered to do so by the standard library.

It doesn't make sense for a UTF-8 formatting library to not be UTF-8 aware. It's producing a UTF-8 string. "digit: {d}" would produce very different output for something which encodes in UTF-16 or UTF-32. "It's just bytes" isn't going to work here, that's a de facto proposal to limit formatting to the ASCII character subset of UTF-8. Zig isn't ASCII encoded, there's no excuse for impoverishing std.fmt that way.

std.fmt currently generates a lot of code which is undesirable and can be problematic for constrained embedded targets.

Constrained embedded targets are the ones who should be using a powerful embedded-aware formatting library. Limiting the Zig standard library to what can fit on an ESP32 is Procrustean. Code in standard should be as small as it can be while providing its function, but no smaller. Embedded targets have always called for distinct libraries which fit their resource-constrained nature.

[Khitiara]

Constrained embedded targets are the ones who should be using a powerful embedded-aware formatting library. Limiting the Zig standard library to what can fit on an ESP32 is Procrustean. Code in standard should be as small as it can be while providing its function, but no smaller. Embedded targets have always called for distinct libraries which fit their resource-constrained nature.

While this is true, std.fmt has the room to get significantly smaller code generation without compromising features, see #9635 for that discussion.

It doesn't make sense for a UTF-8 formatting library to not be UTF-8 aware. It's producing a UTF-8 string. "digit: {d}" would produce very different output for something which encodes in UTF-16 or UTF-32. "It's just bytes" isn't going to work here, that's a de facto proposal to limit formatting to the ASCII character subset of UTF-8. Zig isn't ASCII encoded, there's no excuse for impoverishing std.fmt that way.

Now im wondering if fmt couldnt somehow be made generic over the character type, with options for utf16 or ascii etc, as long as each type can provide some form of character iteration to find the format specifiers etc. I definitely agree that the default utf8 fmt should allow a utf8 fill character though.

[mnemnion]

I think it's most reasonable for Zig std to focus on UTF-8 as the "blessed" encoding. The std.unicode namespace has basic tools for handling the other encodings, including conversions, it isn't unreasonable to ask users who need e.g. UTF-16LE to format in UTF-8 and convert it to the latter. ASCII is a strict subset of UTF-8, so it's very easy to get ASCII formatting: just don't use any codepoints higher than U+7e. "We don't have strings, just bytes, but if you're using them as a string, those bytes are probably UTF-8" is the right balance to strike IMHO.

Sentiment is very clear that plain text data should be UTF-8, and other encodings should be treated as legacy, and using them as a transfer format, or for data at rest, should be deprecated. HMTL illustrates that clearly. I'd much rather have more and better UTF-8 features, over something agnostic about encoding, which duplicates everything, giving a chance for bugs in each separate implementation (which must then be tested, and maintained). Converting where necessary is a simple algorithm, and a solved problem.

I firmly agree that, all else equal, tighter codegen is better than the alternative. But I don't think that "no features in this proposal can lead to larger object code" is a good precept. Features should be considered on a bang-for-buck basis, making them optimal is a separate concern. The heavy use of comptime should mean that function specializations don't pay for features they don't use, where that's not true, it's something to work on for sure.

[mnemnion]

width and precision [...] can also be resolved at runtime

This is a fairly obscure feature which increases the overall complexity of std.fmt. It is also limited to only width and precision; other options like fill or alignment must be specified literally and can not be resolved at runtime.

I missed this the first time through, and wanted to say that using a runtime width is not an especially obscure feature. Calculating the maximum width of a column, and aligning every printed value to that width, is pretty basic stuff. I wouldn't expect precision to get as much use as width, but it's imaginable for e.g. significant figures to be determined at runtime. I've done that once, actually, but have lost count of the number of times I've calculated a column width on the basis of data.

I see no advantage to having runtime fill or alignment though, those aren't really decisions which are plausibly made on the basis of data.

[castholm]

Replying to some comments in bulk:

Related: #19381

I don't think that issue is related as it concerns zig fmt. This one is about std.fmt.

Something to consider: If alignment is removed from numeric formatters, how does one achieve decimal alignment?

For your example you would use {d.3: >10}, essentially the same way you would today except with the precision option moved. Now, if you want very fancy alignment like

+ 55.555
-123.0
-  0.66667

you will probably need to bring your own formatter.

Rather than special-casing a few prefixes, a general solution would be to allow one level of nesting. So for e.g. a right-aligned hex column of width sixteen, {0x{x} >16}.

Nested placeholders might be interesting to explore (though personally I think this is another use case better handled by custom formatters instead of being built-in into the format placeholder syntax) but how do you envision this solving the problem with signed prefixed numbers, where the sign should go before the prefix? If "{0x{x}: >16}" is as generic as you suggest then it looks like it would still format negative numbers as 0x-4d.

I might write up a sub-proposal for formatting options for the built-in numeric types later, but I believe something fairly simple like {d+3.3} for "decimal, positive sign, integer part zero-padded to a minimum of 3 digits, fractional part rounded/zero-padded to 3 digits" or {x+x4} for "hexadecimal, positive sign, 0x prefix, zero-padded to a minimum of 4 digits" should cover all common use cases while keeping the placeholder syntax itself simple.

It doesn't make sense for a UTF-8 formatting library to not be UTF-8 aware. It's producing a UTF-8 string. "digit: {d}" would produce very different output for something which encodes in UTF-16 or UTF-32. "It's just bytes" isn't going to work here, that's a de facto proposal to limit formatting to the ASCII character subset of UTF-8. Zig isn't ASCII encoded, there's no excuse for impoverishing std.fmt that way.

I encourage you to read the commit message for 2d9c479.

I don't personally think it would be some sort of great sin if std.fmt only understands raw bytes and if built-in format specifiers only output byte values in the ASCII range. std.fmt is mainly used for simple technical tasks like logging, debugging and code generation which normally don't have a need for Unicode awareness. Once your needs go beyond that set of simple tasks, std.fmt is almost certainly the wrong tool for the job and you should consider some external library that is Unicode-aware and can deal with rich text.

And just to clarify, as for the {u} specifier I'm mainly suggesting changing std.log.debug("{u}", .{'⚡'}) into something like std.log.debug("{}", .{std.unicode.fmtUtf8Scalar('⚡')}), to get the Unicode/UTF-8-awareness away from the core std.fmt implementation.

I missed this the first time through, and wanted to say that using a runtime width is not an especially obscure feature. Calculating the maximum width of a column, and aligning every printed value to that width, is pretty basic stuff.

I'm not suggesting removing runtime options, rather I'm suggesting removing runtime options from the format placeholder syntax itself, to keep the syntax and core std.fmt implementation simple. The std.fmt namespace could still expose some std.fmt.fmtAligned formatter which would enable a migration path from std.log.debug("{d: >[1]}", .{ value, width }) to std.log.debug("{d}", .{std.fmt.fmtAligned(value, .{ .fill = ' ', .alignment = .right, .width = width })}).

As an aside, if something like std.fmt.fmtAligned exists, one might also begin to ask why the alignment/padding options should be part of the syntax. I don't really have a good argument for why, other than that alignment/padding might be a common enough task be worthy of syntax sugar. If we take that "everything can be solved using formatter functions" line of thinking even further we could make the argument that there should be no specifiers and that all arguments should be passed as formatters. But I don't think that would make for a very ergonomic or user-friendly API and everyone would probably hate needing to do things like

std.log.debug("{}, your new score is {}!", .{
    std.fmt.fmtString(name),
    std.fmt.fmtNumber(score, .{ .base = .decimal, .precision = 3 }),
});

just to format numbers and strings. std.fmt needs to strike a good balance between shorthands for common use cases and implementational purity/simplicity.

[mnemnion]

how do you envision this solving the problem with signed prefixed numbers, where the sign should go before the prefix?

Something like this should work, although it doesn't:

test "optional signed hex" {
    const d = -5;
    const sign = if (@abs(d) == d) null else '-';
    std.debug.print("optionally-signed hex: {?u}0x{x}\n", .{ sign, @abs(d) });
}

You can't provide a type specifier after ?, only a format specifier. That's worth changing, this is a pretty clear oversight imho.

There's a semantic wrinkle here, because {?} by itself will print the string null for null values. That's a useful debugging tool, but it isn't the same as "encode this value if it exists, and do nothing if it doesn't", which is also a common pattern which should be supported. It might not be so bad (although not ideal) if {?d} means "format a digit, or format 'null'" and {d?} means "format a digit, or do nothing". It's broadly cognate with value.? to unwrap an optional, and ?Type specifying a type which can include null.

I also think that signed 0(x|o|b) numbers aren't worth supporting with special syntax, that's an extremely uncommon thing to do and you probably shouldn't. Those literal syntaxes are best used for literal memory values, or what's the point of using a log₂ radix? If anything, I would argue that the bug is applying a sign to {x} formatted negative numbers, which should instead provide the literal byte value. That would be more useful.

Not that it matters if we can provide a general mechanism which supports it.

I don't personally think it would be some sort of great sin if std.fmt only understands raw bytes and if built-in format specifiers only output byte values in the ASCII range.

I firmly disagree. This kind of chauvinism has no place in a modern language. You're only able to hold this opinion because a) you're a monoglot English speaker and b) have decided that you don't care about the needs of the vast majority of people on the planet.

std.fmt is mainly used for simple technical tasks like logging, debugging and code generation which normally don't have a need for Unicode awareness.

You want to support negative 0x numbers, which are a technically absurd thing which has no place in a codebase, and not this: ±. This is "works on my machine syndrome".

My position is simple: Zig has Unicode literals, like '✓', and as long as that's true, it should be possible to format them into a string. Supporting the first of those things, and not the second, is just crippling the standard library, for... why exactly? Your reasoning makes no sense to me, have you ever written a function which encodes a Unicode codepoint as UTF-8 bytes? I have, in several languages, including Zig. The implementation is sixteen lines long. Here's the standard library version in all its glory. This is hardly code bloat!

I have in fact read the comments in #18536, including this part:

Now we have to import poorly named format helper functions and track down where a format() function is implemented, which starts to approach the annoyance of finding a function definition in C++.

Which is what you want to do to anyone who doesn't share your fixation on the printable one byte UTF-8 characters! It's an absurdity. As an aside, I do agree that measuring width in codepoints is a mistake, for a number of reasons. But that's a bit off topic.

Read this part more carefully, while you're at it:

It was a mistake to deviate from dealing purely in encoded bytes.

Encoded bytes, not the small handful of raw bytes which happen to be mostly adequate to expressing your language and your language alone! I happen to agree that there's room for improvement in how Zig handles literal Unicode characters, but that's off topic for this proposal. I'm actively working on one which will cover it, but I want that to be supported with an implementation and some benchmarks, so it isn't ready yet.

This is the general problem with opening a poorly focused issue like this. It implies that every one of your ideas have to be implemented, or none of them. The numeric options idea is a good one. Your crusade against Unicode is deeply misguided.

I think you should remove the entire Unicode question from this issue, and focus on alignment and improving numeric formatting. If you want to make a case for removing {u}, separate that case from your good ideas.

[castholm]

@mnemnion I appreciate that you are participating in the discussion, offering differing views and raising good points.

I don't appreciate that you are implying that my point of view is misguided by suggesting that I lack technical experience with character encodings or by falsely stating that I'm a monolingual anglophone, and such comments are greatly diminishing my willingness to engage with your replies. English is not even my native language. I read and write multiple different languages at varying levels of proficiency, some of which don't even use the Latin alphabet at all. For as long as I can remember I've regularly run into limitations (or worse, bugs) in systems because my full legal name, which is sometimes used as the basis for usernames, contains non-ASCII characters. Despite having experienced such problems first-hand, I still personally lean on the side of the belief that the core of std.fmt, as a utility mainly used for technical diagnostics, debugging and code generation and not rich text formatting, does not need to be Unicode-aware if such features make std.fmt as a whole less clear-cut and more complex.

[peppergrayxyz]

I think Unicode support should be built in: Grapheme Cluster is the proper unit of working with strings for anyone who wants to do anything with a UI, e.g.:

• how long is a string (display characters on the screen)
• where is the pointer on screen
• use format strings with Unicode strings

Example:

const std = @import("std");

pub fn main() !void {
    const examples = [_][]const u8{
        "abcde",
        "\u{006E}\u{0303}",
        "\u{0001F3F3}\u{FE0F}\u{200D}\u{0001F308}",
        "ห์",
        "ปีเตอร์",
        "fghij",
        "klmno",
    };

    const stdout = std.io.getStdOut().writer();

    try stdout.print("0123456789\n", .{});

    for (examples) |example| {
        try stdout.print("{s: >10}\n", .{example});
    }
}

Expected Output:

0123456789
     abcde
         ñ
         🏳️‍🌈
         ห์
     ปีเตอร์
     fghij
     klmno

Actual Output:

0123456789
     abcde
        ñ
      🏳️‍🌈
        ห์
   ปีเตอร์
     fghij
     klmno

While I do agree that not adding Unicode support makes stdout.print simpler, it makes everyone elses life much harder.

To produce the expected output the code must be modified like this:

const std = @import("std");
const grapheme = @import("grapheme");

pub fn main() !void {
    const examples = [_][]const u8{
        "abcde",
        "\u{006E}\u{0303}",
        "\u{0001F3F3}\u{FE0F}\u{200D}\u{0001F308}",
        "ห์",
        "ปีเตอร์",
        "fghij",
        "klmno",
    };

    const stdout = std.io.getStdOut().writer();

    try stdout.print("0123456789\n", .{});

    for (examples) |example| {
        var gpa = std.heap.GeneralPurposeAllocator(.{}){};
        const allocator = gpa.allocator();

        const gd = try grapheme.GraphemeData.init(allocator);
        defer gd.deinit();
        var iter = grapheme.Iterator.init(example, &gd);
        var len: usize = 0;
        while (iter.next()) |_| : (len += 1) {}

        //try stdout.print("len: {}\n", .{len});

        const num_padding = 10;
        if (len < num_padding) {
            const padding = num_padding - len;

            //try stdout.print("pad: {}\n ", .{padding});
            var i: usize = 0;
            while (i < padding) : (i += 1) {
                try stdout.print(" ", .{});
            }
        }

        try stdout.print("{s}\n", .{example});
    }
}

Output:

0123456789
     abcde
         ñ
         🏳️‍🌈
         ห์
     ปีเตอร์
     fghij
     klmno

Basically, everything is Unicode. I don't understand why you would not give people the power to process Unicode strings in a way where they can easily work with grapheme clusters (units of display width one). I strongly believe working with Unicode should be as simple as possible, because otherwise people just won't (properly) support it in their apps, because they don't bother dealing with its complexity. So yes, I agree that complexity should be reduced, but for the people working with these things, not those providing them.

[daurnimator]

Grapheme Cluster is the proper unit of working with strings for anyone who wants to do anything with a UI,

Grapheme clusters change with each release of unicode, and are useless for practical purposes

how long is a string (display characters on the screen)

Not all characters are the same width; and not all fonts pick the same width.
Also ligatures exist.

where is the pointer on screen

In addition to the above reasons, you have no way to know this with a non-fixed width font.

use format strings with Unicode strings

This is unrelated.