make `packed struct` always use a single backing integer, inferring it if not explicitly provided

[ikskuh]

So, everyone knows the problems we have with packed structs. Some are implementation problems, but we also have a huge problem with not having a proper specification on how they should work at all, except for "zero padding".

So i want to propose a definition for packed structs that is easy to understand and implement.

Let's consider this code example:

const T = packed struct {
    a: u4,
    b: u4,
};

var bits = [_]u8 { 0x01 };
var t = @bitCast(T, bits);

std.debug.print("{}\n", .{ t });

What do you expect it to print?
Is it T{ .a = 1, .b = 0 }? Then you have assumed a little-endian platform, as on big endian, it will print T{ .a = 0, .b = 1 }.
Check it out on Compiler Explorer.

Personally, i find this confusing, thus i propose:

A packed struct will have a similar semantic as a unsigned integer with the same amount of bits. Each field is considered an unsigned integer of the same amount of bits.

To explain the idea, we have this struct:

const T1 = packed struct { // u8
  a: u4, // occupies bits 0..3
  b: u4, // occupies bits 4..7
};

const T2 = packed struct { // u32
  a: u4, // occupies bits 0..3 in our u32
  b: u3, // occupies bits 4..6 in our u32
  c: u25, // occupies bits 7..31 in our u32
};

So the packed struct implements exactly what we would do in a language without packed structs:

var value: T2 = …;
var a = @truncate( u4, 0x0000000F & (value >> 0));
var b = @truncate( u3, 0x00000007 & (value >> 4));
var c = @truncate(u25, 0x01FFFFFF & (value >> 7));

This means that we have a predictable model for backed structs and in case of T2, we can reason about it:

const integer: u32 = 0x165652B6;
const value = @bitCast(T2, integer);
std.debug.assert(value.a == 6);
std.debug.assert(value.b == 3);
std.debug.assert(value.c == 0x2CACA5);

This will be true for both little and big endian hardware, so it will do what we expect, even if the bytes on disk have a different order.
If a known byte order is required, we can use std.mem.writeIntLittle, @byteSwap and others.

In addition, we can declare a struct as packed struct(u32), which will enforce the struct size to 32 bit, and we will get a compiler error if it isn't the case.

This will also guarantee that this struct is handled as a u32 and can be used as such, similar to enums. This also guarantees that loads and stores will not be sliced into more than one access if possible and allows atomic load/store for such packed structs. (See #5049).

What needs to be clarified:

• How to handle floats (float endianess is a thing)
• How to handle pointers in a packed struct
• How to handle pointers to struct fields

Related issues:

• make packed structs versatile #3133
• Incorrect byte offset and struct size for packed structs #2627
• Proposal: Integer-backed packed struct #5049

Regards

• xq

[praschke]

as a weak usecase (weak because this could just be a quirk of X11), the X protocol embeds bitmasks in u16 and u32 fields whose layout is dependent on the byte-order specified by the client in the protocol setup. i don't know if the preferred representation for bitmasks is defining a bunch of 1 << n constants, or packed structs, but to express these elements with packed structs i would have to define two different sets.

it's not super terrible, but having packed structs act however shifts act on the target architecture seems more correct to me.

The packed struct { a: u4, b: u4 } example could be a compiler bug. Endianness determines the byte order only, while the bit order within a single byte is entirely up to the compiler and can be made consistent across all platforms.

That said, it's true that endianness is a problem for packed structs. However, enforcing a single consistent endianness is not a costless action. For example, if you have

const Vec3 = packed struct {
    x: i32,
    y: i32,
    z: i32,
};

then with this proposal every field access would require byte order conversion on big-endian platforms.

Ultimately, the problem is that packed structs are used for two different purposes:

1. As an ordinary struct, but with defined field order and no extra padding.
2. As a bit-for-bit data serialization format.

Ideally, we would have two different struct types to match the different use cases. Maybe something like serialized struct {}. Or even struct {} vs struct(.Packed) {} vs struct(.Serialized) {}.

[ikskuh]

then with this proposal every field access would require byte order conversion on big-endian platforms.

No, it would not. y would still allocate bits 32…64 in the packed struct, which can be extracted by "bitshifting". There is no endianess encoding in that, but only bit offsets in a larger "integer". This means that reading z on a big-endian platform will read from byte offset 0, while reading z on a little-endian platfrom will read from byte-offset 8. They are not layed out consecutively in memory as it is now.

So As an ordinary struct, but with defined field order and no extra padding. will be true, but the field order will be swapped in memory for big-endian platforms.

Imho, the use cases are:

• Reflection of hardware registers (where we need packed struct(u32))
• Compression of in-memory structures (where the memory layout is completly irrelevant)
• Serialization data (where one might still apply manual byte order swapping)

Only the last use case requires the use of extra work for handling endianess, but this is relevant in any case, as file formats, network protocols and such all need to define a byte order and using packed structs aren't the right tool in most cases here anyways

[SpexGuy]

What happens with a very large packed struct? (Over 65536 bits). Is it still built like one giant integer, laid out in reverse field order on big endian? Also what about npot sizes (like 24 bits)? Does that round up to 32 or end at 24?

@MasterQ32,

This means that reading z on a big-endian platform will read from byte offset 0, while reading z on a little-endian platfrom will read from byte-offset 8. They are not layed out consecutively in memory as it is now.

Sorry, I had misunderstood that part. Maybe because I wouldn't ever expect a "packed" struct to have its field order reversed on some platforms. 😄

Also, correct me if I'm wrong, but wouldn't this convention make packed structs useless for parsing binary data? Or at least you would have to always define two structs with fields A, B, C and C, B, A respectively, and then use the appropriate struct depending on platform endianness?

[ikskuh]

What happens with a very large packed struct? (Over 65536 bits). Is it still built like one giant integer, laid out in reverse field order on big endian?

The integer layout is only a mental layout, how access is implemented for non-integer backed structs isn't defined. So yeah, for huge big-endian structs, the order would still be "reverse" to allow efficient access of parts.

Also what about npot sizes (like 24 bits)? Does that round up to 32 or end at 24?

It will be the same as u24, so in a normal struct, it will take up 3 byte aligned to 4, and in a packed struct, it will take exactly 24 bit

[N00byEdge]

There are many more things you have to consider when working with packed structs. When you care about the bit order, bitfields are probably used together with mmio. It has concepts of such things like access/register sizes. I'm not sure if packed structs should have all of these covered. I have written a little library for working with mmio registers which specify bit offsets of fields, while still encompassing them within a register with a certain access/register size something like this:

    command_status: extern union {
        raw: u32,

        start: bf.Boolean(u32, 0),
        recv_enable: bf.Boolean(u32, 4),
        fis_recv_running: bf.Boolean(u32, 14),
        command_list_running: bf.Boolean(u32, 15),
    },

Here each bit offset is explicitly noted, which I believe is going to be desired when you want to deal with these kinds of layouts anyways. The only thing I'm missing is some kind of way of getting rid of the repeated register type/size.

I think the goal for MMIO related structs should either be provided as a library solution like above, or at least something with a similar syntax (but hopefully without the repeated type) register struct?

[ikskuh]

There you have more issues than just the bit order. It has concepts of such things like access/registers sizes.

I know, this is covered by the integer-backed struct, that will be handled like the integer you base it on. See #5049 for a wider discussion of this topic

[N00byEdge]

Oh, I see. Yes that encompasses exactly what I was trying to say here. Nevermind my comment.

[DanB91]

To build on this, I currently (and very carefully) use packed structs to represent bit fields in MMIO hardware registers. It'd be nice if we can actually specify the backing register size. so something like this:

const T1 = packed struct(Type) {
  a: u4, // occupies bits 0..3
  b: u4, // occupies bits 4..7
};

Where Type must be a standard unsigned type that is a byte multiple (e.g. u8, u16, etc). In this case it would be u8 And on top of that, if the sum of the bits don't equal the backing type, you get a compile error (not sure if this would be too much).

This doesn't clarify the 3 points (handling floats, etc) in the original post, so not sure if this would conflict with them. Maybe I'm asking for a completely different structure type here? Like instead of packed struct what I'm asking for should be called bitfield?

[SpexGuy]

@DanB91 you might be interested in #5049 😉

[DanB91]

@DanB91 you might be interested in #5049 😉

Oh man, this proposal is exactly what I was thinking, thanks!