[vm/ffi] Unwrap extension types to their representation types

[halildurmus]

TL;DR: I would like to use extension types like typedef for native types in function signatures to provide users with clearer information and thus enable easier consumption of Win32 APIs within Dart.

---

I'm currently working on v6 of the package:win32 with a focus on enhancing the user experience when using the Win32 APIs.

Let's take a specific example with the CoGetApartmentType function, which is defined in C as follows:

HRESULT CoGetApartmentType(APTTYPE *pAptType, APTTYPEQUALIFIER *pAptQualifier);

typedef enum _APTTYPE {
  APTTYPE_CURRENT = -1,
  APTTYPE_STA = 0,
  APTTYPE_MTA = 1,
  APTTYPE_NA = 2,
  APTTYPE_MAINSTA = 3
} APTTYPE;

typedef enum _APTTYPEQUALIFIER {
  APTTYPEQUALIFIER_NONE = 0,
  APTTYPEQUALIFIER_IMPLICIT_MTA = 1,
  APTTYPEQUALIFIER_NA_ON_MTA = 2,
  APTTYPEQUALIFIER_NA_ON_STA = 3,
  APTTYPEQUALIFIER_NA_ON_IMPLICIT_MTA = 4,
  APTTYPEQUALIFIER_NA_ON_MAINSTA = 5,
  APTTYPEQUALIFIER_APPLICATION_STA = 6,
  APTTYPEQUALIFIER_RESERVED_1 = 7
} APTTYPEQUALIFIER;

In package:win32, this function is projected into Dart as follows:

int CoGetApartmentType(Pointer<Int32> pAptType, Pointer<Int32> pAptQualifier) =>
    _CoGetApartmentType(pAptType, pAptQualifier);

final _CoGetApartmentType = _ole32.lookupFunction<
    Int32 Function(Pointer<Int32> pAptType, Pointer<Int32> pAptQualifier),
    int Function(Pointer<Int32> pAptType,
        Pointer<Int32> pAptQualifier)>('CoGetApartmentType');

However, it's not immediately clear to the user what the pAptType and pAptQualifier parameters represent. Users need to refer back to the Microsoft documentation to understand that they are meant to be APTTYPE and APTTYPEQUALIFIER enums. Even if I chose to represent these enums as typedefs, they would still see Pointer<Int32> type when they hover over the function (issue #39332) and I wouldn't be able to define enum values.

(I should also note that currently, the projected APIs in package:win32 are manually defined in a JSON file, including the C function definition as documentation, which is extracted from the Microsoft documentation website. However, in v6, instead of manually defining the projected APIs in a JSON file, I plan to project the entire Win32 API set (excluding a few ancient APIs) using Microsoft's metadata. Consequently, users won't see these manually added C definitions when they hover over the function in their IDE. Instead, they'll just see the function signature int CoGetApartmentType(Pointer<Int32> pAptType, Pointer<Int32> pAptQualifier), which isn't very helpful.)

To address this, I would like to project Win32 enums like APTTYPE and APTTYPEQUALIFIER as Dart extension types, as shown below:

extension type const APTTYPE(Int32 _) implements Int32 {
  static const APTTYPE_CURRENT = 0;
  static const APTTYPE_STA = 1;
  static const APTTYPE_MTA = 2;
  static const APTTYPE_NA = 3;
  static const APTTYPE_MAINSTA = 4;
}

extension type const APTTYPEQUALIFIER(Int32 _) implements Int32 {
  static const APTTYPEQUALIFIER_NONE = 0;
  static const APTTYPEQUALIFIER_IMPLICIT_MTA = 1;
  static const APTTYPEQUALIFIER_NA_ON_MTA = 2;
  static const APTTYPEQUALIFIER_NA_ON_STA = 3;
  static const APTTYPEQUALIFIER_NA_ON_IMPLICIT_MTA = 4;
  static const APTTYPEQUALIFIER_NA_ON_MAINSTA = 5;
  static const APTTYPEQUALIFIER_APPLICATION_STA = 6;
  static const APTTYPEQUALIFIER_RESERVED_1 = 7;
}

With this approach, users will see the exact enum types (e.g., Pointer<APTTYPE>) when hovering over the function, instead of seeing Pointer<Int32>, and they'll be able to allocate memory for the enums using calloc, as demonstrated below:

void main() {
  final pAptType = calloc<APTTYPE>();
  final pAptQualifier = calloc<APTTYPEQUALIFIER>();

  try {
    final hr = CoGetApartmentType(pAptType, pAptQualifier);
    if (hr == S_OK) {
      print('COM is initialized on the current thread.');

      if (pAptType.value == APTTYPE.APTTYPE_STA) {
        print('The current thread is in a single-threaded apartment.');
      } else if (pAptType.value == APTTYPE.APTTYPE_MTA) {
        print('The current thread is in a multi-threaded apartment.');
      }

      if (pAptQualifier.value == APTTYPEQUALIFIER.APTTYPEQUALIFIER_NONE) {
        print('No qualifier information for the current COM apartment type is '
            'available.');
      } else if (pAptQualifier.value ==
          APTTYPEQUALIFIER.APTTYPEQUALIFIER_IMPLICIT_MTA) {
        print('The current thread is in an implicit multi-threaded apartment.');
      }
    } else {
      print('COM is not initialized on the current thread.');
    }
  } finally {
    calloc
      ..free(pAptType)
      ..free(pAptQualifier);
  }
}

cc @dcharkes

[dcharkes]

@halildurmus Can you detail a bit more about the use case where this is useful?

[dcharkes]

Thanks for the clear explanation! ❤️ (And thanks for continuing development on package:win32!)

So if I understand you correctly, you only want to use extension types as type markers in C function signatures and as type arguments to Pointer for type safety. Do you also have a need for passing values around or only types? (Well you cant have values of Int32 anyway, but you could of structs.)

As for the question of marker types, I wonder if extension types are the right solution yes or no.

One alternative is to generate class APTTYPE implements AbiSpecificInteger. This will also work for if you have APTTYPE directly in function signatures.

Both of these solutions will work today. Though, there might be downsides to doing this that I haven't thought of.

WDYT? Any thoughts, ideas?

(Side note: We have class Foo extends Opaque as a marker type. If we go the direction of extension types, we should probably have people migrate to having an extension type on opaque, rather than extending it. Because you want to use the integer-size with calloc you can't use Opaque for your use case.)

[halildurmus]

Thanks for the clear explanation! ❤️ (And thanks for continuing development on package:win32!)

Thanks!

So if I understand you correctly, you only want to use extension types as type markers in C function signatures and as type arguments to Pointer for type safety.

Exactly.

Do you also have a need for passing values around or only types? (Well you cant have values of Int32 anyway, but you could of structs.)

Yes. Some Win32 APIs directly accept native integer types and structs by value. As you said, given that we can't create values of Int32 anyway, I'll keep using the Dart counterparts (e.g., int, double) in those cases -- and that's fine.

As for the question of marker types, I wonder if extension types are the right solution yes or no.

One alternative is to generate class APTTYPE implements AbiSpecificInteger. This will also work for if you have APTTYPE directly in function signatures.

I couldn't understand this option. Let's say the base modifier on AbiSpecificInteger is removed and one can declare a class like class APTTYPE implements AbiSpecificInteger. How would this function exactly? How do I specify which native type (Int32, Uint32) this class represents so that when I allocate memory with calloc<APTTYPE>(), the size of that native type is used?

[dcharkes]

I meant to say extends not implements, my bad! See the documentation here: https://api.dart.dev/stable/3.3.0/dart-ffi/AbiSpecificInteger-class.html You'd just add the same integer size for all windows targets:

@AbiSpecificIntegerMapping({
  Abi.windowsArm64: Int32(),
  Abi.windowsIA32: Int32(),
  Abi.windowsX64: Int32(),
})
final class APTTYPE extends AbiSpecificInteger {
  const APTTYPE();
}

[halildurmus]

That's nice, thanks for the clarification.

However, there is an important downside to using this option. We can't seem to define members for this class 😢:

Can this restriction be removed?

[dcharkes]

Right, static members should be fine. I can make a fix for that.

Are all the types that you want to wrap with documentation integer types?
Do you have any structs that are identical but need a different documentation? (You could generate two identical structs in that case.)

[halildurmus]

Are all the types that you want to wrap with documentation integer types?

No. I also want to wrap native string types PSTR (Pointer<Utf8>), PWSTR (Pointer<Utf16>), and BSTR (Pointer<Utf16>).

There is an important difference between PWSTR and BSTR types. BSTR has a four-byte integer prefix that contains the number of bytes in the following data string. This means one can't use the toNativeUtf16() method to create a BSTR type. Instead, they need to use the SysAllocString function to create a BSTR. That's why I think it is important that I wrap these types as well. However, this does not seem possible with the AbiSpecificInteger.

I came up with this example using extension types:

import 'dart:ffi';

import 'package:ffi/ffi.dart';
import 'package:win32/win32.dart' hide BSTR;

extension type const PWSTR(Pointer<Utf16> _) implements Pointer<Utf16> {
  PWSTR.fromString(String string) : this(string.toNativeUtf16());

  void free() => calloc.free(this);
}

extension type const BSTR(Pointer<Utf16> _) implements Pointer<Utf16> {
  factory BSTR.fromString(String string) {
    final psz = string.toNativeUtf16();
    final pbstr = SysAllocString(psz);
    calloc.free(psz);
    return BSTR(pbstr);
  }

  void free() => SysFreeString(this);

  BSTR clone() => BSTR(SysAllocString(this));

  int get byteLength => SysStringByteLen(this);

  int get length => SysStringLen(this);

  BSTR operator +(BSTR other) {
    final pbstrResult = calloc<Pointer<Utf16>>().cast<BSTR>();
    VarBstrCat(this, other, pbstrResult);
    final result = BSTR(pbstrResult.value);
    calloc.free(pbstrResult);
    return result;
  }
}

void foo(PWSTR pwstr) {
  print(pwstr.toDartString());
}

void bar(BSTR bstr) {
  print(bstr.toDartString());
}

void baz(Pointer<PWSTR> pwstr) {
  print(pwstr.value.toDartString());
}

void main() {
  final pwstr = PWSTR.fromString('I am a happy PWSTR');
  foo(pwstr);
  pwstr.free();

  final bstr = BSTR.fromString('I am a happy BSTR');
  bar(bstr);
  bstr.free();

  final pwstr2 = PWSTR.fromString('I am a happy PWSTR2');
  final ppwstr = calloc<Pointer<Utf16>>().cast<PWSTR>()..value = pwstr2;
  baz(ppwstr);
  pwstr2.free();
  calloc.free(ppwstr);
}

I must say, I love this! I think using extension types to wrap these types is the way to go. It's much more powerful compared to the AbiSpecificIntegers. WDYT?

Do you have any structs that are identical but need a different documentation? (You could generate two identical structs in that case.)

No, I don't think so.

[dcharkes]

There is an important difference between PWSTR and BSTR types. BSTR has a four-byte integer prefix that contains the number of bytes in the following data string. This means one can't use the toNativeUtf16() method to create a BSTR type. Instead, they need to use the SysAllocString function to create a BSTR. That's why I think it is important that I wrap these types as well. However, this does not seem possible with the AbiSpecificInteger.

You could define a new Opaque type for those, and not relate it at all to Utf16. Since they are not compatible with the Pointer<Utf16> at all, your users should never get the base-type out of the extension type.

Your API would be centered around Pointer<PWSTR> and have extension methods on that type (and on String) rather than PWSTR itself representing a Pointer<something>. This is also how the Pointer<Utf16> API is structured.

Would this work?

I kind of like the extension types as a general wrapper method, because if we allow it, it would work for any native type anywhere in the FFI.

I am slightly worried that it might conflict with us wanting to migrate the dart:ffi API to extension types (for example Int8 being an extension type on int).

• [vm/ffi] ☂️ Exploration of dart:ffi2 using new language features #54504

@lrhn @mkustermann Any thoughts about allowing users to wrap any NativeType in extension types and automatically unwrapping/wrapping?

• Automatic unwrapping on FFI call arguments / FFI callback return value / Pointer store operations
• Automatic wrapping on FFI call return value / FFI callback arguments / Pointer load operations

The only use case I can think of is custom code generators. Users using package:ffigen will not use this feature. Unless we start adding logic to FFIgen to target these. We have a work-around in FFIgen to match certain types and generate another type that is being used to turn char* into Pointer<Utf8>. If we introduce extension type wrapping in dart:ffi, we should probably generalize that mechanism in package:ffi. Issue tracking this for strings:

• Better string support if all strings in a C API follow the same encoding native#315

[mkustermann]

We have 2 different kinds of types:

• C types that specify what C types parameters, return values or fields in composites use. It can influence memory size, layout, alignment, etc (The purpose of AbiSpecificIntger is to allow us to specify a C type that may vary across abis.)
• Dart types specify the types Dart users use to interact with native functions, composites, members of composites, etc.

For some kinds they differ: e.g. integer & float types
For some kinds they are the same: e.g. MyStruct, Pointer<Int8>

The fundamental reason we have this distinction is because we could not represent the C types in dart (e.g. differentiate between integer types). We may be able to bridge this gap in a future redesign of the FFI (e.g. using extension types).

Extension types are a way to add type-safety (separate type, mostly explicit conversion to/from representatation type) and behavior (methods on the extension types) to a dart representation type. Both of these features are much more relevant to the Dart objects that code interacts with and not much relevant for C type annotations (see distinction above).

So I'm not sure whether it makes any sense to use them on the C-only types (as the title of this issue says: "Allow extension types that implement NativeTypes in FFI type arguments"). To me it makes much more sense to use them to represent the Dart objects the programmer interacts with.

In terms of future-proofing the FFI: One can have extension types on other extension types. That means if we changed Pointer from a class to an extension types, we wouldn't break the mechanism, because construction of extension types and conversions to/from the underlying type would be the same (IIUC).

The extension types do have the implicit constructor which isn't allowed to have any user-written logic in it. Which makes it fit into FFI, as we would "construct" those objects (not really, as they are desugared) internally in the C interop.

Representing enums

extension type const AptType(int value) {
  static const AptType current = AptType(0);
  ...
}

@Native<Void Function(Int16)>()
external void foo(AptType arg);

// If we ever make `Int16` an extension type on `int`, `AptType` could become extension type on `Int16`
@Native<Void Function(AptType)>()
external void foo(AptType arg);

// ... or the shorter form
@Native()
external void foo(AptType arg);

class MyStruct extends Struct {
  @Int16()
  external AptType value;

  // If we ever make `Int16` an extension type on `int`, `AptType` could become extension type on `Int16`
  external AptType value;
}

Representing special strings (which are pointers)

extension type const PWSTR(Pointer<Utf16> _) {
  PWSTR.fromString(String string) ...;
  void free() => calloc.free(this);
}

@Native<Pointer<Utf16> Function(Pointer<Utf16>, Pointer<Utf16>)>()
external PWSTR concat(PWSTR, PWSTR);

// But we could allow
@Native<PWSTR Function(PWSTR, PWSTR)>()
external PWSTR concat(PWSTR, PWSTR);

// ... which would allow
@Native()
external PWSTR concat(PWSTR, PWSTR);

class MyStruct extends Struct {
  external PWSTR value;  // Reason to allow PWSTR in native & dart function types (see above)
}

The difficulty starts when one would like to use a pointer to a type that
is different in C and in Dart, if we mirrored our existing API of dereferencing pointer, one would be tempted to write this:

extension on Pointer<AptType> {
  AptType get value = AptType(this.cast<Int16>().value);
  void set value(AptType value) => this.cast<Int16>().value = (value as int);
}

@Native<Void Function(Pointer<AptType>)>()
void foo(Pointer<AptType> arg);

// Or simply
@Native()
void foo(Pointer<AptType> arg);

class MyStruct extends Struct {
  external Pointer<AptType> value;
}

This doesn't work as AptType is extension on int and not Int16 (which would be solved if we made Int16 extension type on int).

Now, strictly speaking we could remove the bound on Pointers type parameter: Any dereferencing access on pointers (for reading/writing the thing being pointed to) is done via special extensions on Pointer<X> where X is a subtype of native type, e.g. extension on Pointer<Int8> { int get value; ... }.

If we made Int32 an extension on int, then this problem would go away (as we'd have AptType extension type on Int32 an extension type on int -- therefore making the native and dart types be equal). Though that does introduce some user issues, as extension types require explicit casts, so users may need to e.g. foo(AptType(Int16(1))).

An alternative to the last point above would be an extension type AptPointer(Pointer<Int16>).

This ability to use typed objects to represent enums without runtime cost (i.e. extensions on integers) seem also very appealing to use by package:ffigen.

So overall I think making the FFI static analysis / FFI kernel transformer simply unwrap all extension types to their representation types before doing the checking would seem fine to me.

[eernstg]

Interesting!!

I'm just wondering, would these ideas have some synergy (or the opposite) with dart-lang/language#3614? That's a proposal to allow some extension types to be assignable from their representation type:

extension type Int32(int _) implements int {}
extension type AptType(Int32 _) implements Int32 {}

Int32 i32 = 42; // OK.
AptType aptType = i32; // Could be OK if the proposal is generalized slightly.
AptType aptType2 = 42; // Perhaps --- that would be yet another generalization.

I think making the FFI static analysis / FFI kernel transformer simply unwrap all extension types to their representation types before doing the checking would seem fine to me.

To me this sounds like it could erase some typing distinctions that we'd want to maintain. Would it be sufficient to erase the extension type when it's used to select a C type?

[lrhn]

The original request looks like a non-issue to allow.

If you do extension type MyInt32(Int32 _) implements Int32 { ... }, then there is no doubt that you're just adding members to Int32, or removing some. The operative type is still clearly Int32.

If you do extension type MyInt32(Int32 _) { ... } with no supertype, then I wouldn't let the type count as Int32.

If you do extension type MyInt32(Int32 _) implements NativeType { ... }... I'd still probably not let it work like Int32, and then it's limited where it can be used.

So, to allow it, the extension type must implement a concrete native type that could be used somewhere, and must also have a representation type that is a subtype of that type. I don't think there are many useful type hierarchies where among native types, so that probably just means implementing the representation type.
(You can also choose to make that the criterion for when you can use an extension type as a type argument where you'd normally require a native type.)

Wrapping pointers is probably where the fun is.

extension type BStr(Pointer<Uint16> _) implements Pointer<Uint16> {
  int get length => _.cast<Uint32>().value;
  Uint16 operator[](int index) => _[2 + IndexError.check(index, length)];
}

I'm not sure I'd make that type implement Pointer<Uint16> though. It feels like an abstraction that would be better served by having a Pointer<Uint16> get address than being the pointer. (Favor composition over inheritance.)

(Can you wrap an Array<Uint16>, which means creating an Array<Uint16> value, or is that only a marker type?
In C, I'd make it a struct like struct {uint32_t length, uint16_t[0] chars}, then access theStruct.chars[i], but not sure is a good idea for FFI, since it doesn't actually have a size.)

About making the native types themselves be extensions, that's precisely the kind of thing extension types are made for, so it's no surprise it seems like a good fit. The places where it might fall short is that extension types are not good at being abstract supertypes. I'd probably use real classes for the Uint32-like "not real types", and extension types for the types that users can interact with.

sealed class NativeType {}
sealed class NativeIntegerType extends NativeType< {}
final class Uint32 extends NativeIntegerType {
  static const int size = 4;
}
final class Uint16 extends NativeIntegerType {}
// ...

extension type Pointer<T extends NativeType>(int address) {
   Pointer<R> cast<R extends NativeType>() => Pointer<R>(address);
   bool operator <(Pointer<NativeType> other) => address < other.address;
   bool operator <=(Pointer<NativeType> other) => address <= other.address;
   bool operator >(Pointer<NativeType> other) => address > other.address;
   bool operator >=(Pointer<NativeType> other) => address >= other.address;
}
extension Uint32Pointer on Pointer<Uint32> {
  int get value => _readUint32(address);
  set value(int value) { _writeUint32(address, value); }
  Pointer<Uint32> operator +(int offset) => Pointer<Uint32>(address + offset * Uint32.size);
  Pointer<Uint32> operator -(int offset) => Pointer<Uint32>(address - offset * Uint32.size);
  int distance(Pointer<Uint32> other) => (other.address - this.address) ~/ Uint32.size;
  int operator [](int offset) => (this + offset).value;
  operator []=(int offset, int value) { (this + offset).value = value; }
}

(The Pointer class has very little functionality itself. Which is not surprising, there is little you can do with a C pointer if you don't know its type, so nothing you can't do with a NULL void pointer. But == works.)

[halildurmus]

Wrapping pointers is probably where the fun is.

extension type BStr(Pointer<Uint16> _) implements Pointer<Uint16> {
  int get length => _.cast<Uint32>().value;
  Uint16 operator[](int index) => _[2 + IndexError.check(index, length)];
}

Just to clarify, the pointer for the BSTR type points to the first character of the data string, not to the length prefix.

So your example should be modified as follows:

import 'dart:ffi';

import 'package:ffi/ffi.dart';
import 'package:win32/win32.dart';

extension type BStr(Pointer<Uint16> _) implements Pointer<Uint16> {
  int get length =>
      Pointer<Uint32>.fromAddress(address - sizeOf<Uint32>()).value;
}

void main() {
  final bstr = BStr(SysAllocString('Hello, world!'.toNativeUtf16()).cast());
  print(bstr.length); // 26 = 13 * sizeOf<Uint16>() (excluding the terminator)
  print(bstr[0]); // 72 (H)
  print(bstr[12]); // 33 (!)
  print(bstr[13]); // 0 (NUL)
}

Also, Win32 API provides String Manipulation Functions for working with BSTR types.

Here's a simple example that uses these APIs:

import 'dart:ffi';

import 'package:ffi/ffi.dart';
import 'package:win32/win32.dart';

extension type BStr(Pointer<Uint16> _) implements Pointer<Uint16> {
  factory BStr.fromString(String string) {
    final psz = string.toNativeUtf16();
    final pbstr = SysAllocString(psz);
    calloc.free(psz);
    return BStr(pbstr.cast());
  }

  int get length => SysStringLen(this.cast());

  void free() => SysFreeString(this.cast());
}