String parameter support

[erjo1776]

From what I can gather, strings are not directly supported as parameters. Of course being C and not C++ makes this is a messy topic (char * is not always a string).

Is there anything that can help here? Are there plans for supporting strings?

If strings are somehow supported, then please change the simple example to use them (the other two examples are a bit heavy to teach a conceptually basic type).

[dcharkes]

dart:ffi exposes char* as Pointer<Int8> and package:ffi exposes it as Pointer<Utf8>.

This package could take either of those two options (or have it configurable). Currently it exposes it as Pointer<Int8>, see https://github.com/dart-lang/ffigen/blob/master/test/large_integration_tests/_expected_sqlite_bindings.dart#L2410-L2416.

If strings are somehow supported, then please change the simple example to use them

👍

[mannprerak2]

We should also add this to the readme's FAQ section(we don't have one currently).

[erjo1776]

OK, the following works for me:
myLib.say(Utf8.toUtf8('plain old C string').cast<Int8>());
This converts Dart string to Utf8 and then recasts it to Int8 which my 'char *' C function accepts.

This page helped me for the conversion:
https://pub.dev/documentation/ffi/latest/ffi/Utf8-class.html

And then IntelliSense (like) feature showed me 'cast<>()'. I am new to Dart but a long time veteran of C, C++, Java (and variants), and Python.

Btw, I tried Gluecodium for generating bindings between C++ & Dart, but it tries to also handle Java JNI and Swift and I never got a full build to work although C++ & Dart appeared to complete. Their Hello World is really Hello The Entire World. Aside from strings and forcing myself to use C instead of C++, FFIgen is working well so far (very easy to understand in goes Header and out comes DartBinding).

Thanks!

[listepo]

As an option, add extensions to package:ffi

extensions.dart

import 'dart:ffi';

import 'package:ffi/ffi.dart';

extension StringExtensions on String {
  Pointer<Int8> toInt8() {
    return Utf8.toUtf8(this).cast<Int8>();
  }
}

extension PointerExtensions<T extends NativeType> on Pointer<T> {
  String toStr() {
    if (T == Int8) {
      return Utf8.fromUtf8(cast<Utf8>());
    }

    throw UnsupportedError('${T} unsupported');
  }
}

Utf8.fromUtf8(lib.hello(Utf8.toUtf8('World').cast()).cast());

vs

lib.hello('World'.toInt8()).toStr();

@dcharkes @mannprerak2 what do you think?

[dcharkes]

The actual API for strings is still up in the air, awaiting dart-lang/sdk#39787.

However, it would be fine to add some extension methods in the mean time.
I would prefer the extension methods to be on Utf8 though.

extension Utf8PointerString on Pointer<Utf8> {

Even better would be to have package:ffigen do the conversion.
@mannprerak2 can we add an option (or possible a filter based on function/argument names) to ffigen to treat char* as null-terminated Utf8? And alternatively as null-terminated ASCII? (Those are probably the two most common encodings used.) Can you make a design for that?

strings:
  - encoding: utf8
    functions:
      include: sqlite.*
  - encoding: ascii
    functions:
      include: my_function
    arguments:
      include: my_string

This would allocate the right memory, encode/decode the Dart string into that memory, do the ffi call, and free the memory.

lib.hello('World'.toInt8()).toStr();

@listepo You're leaking the memory backing World here (unless the native hello frees the memory).

[listepo]

@dcharkes thanks

[listepo]

extension Utf8PointerString on Pointer<Utf8> {
  String toStr() {
    final res = Utf8.fromUtf8(this);
    free(this);
    
    return res;
  }
}

@dcharkes did you talk about something like that?

[dcharkes]

No.

lib.hello('World'.toInt8()).toStr();

Should be:

final mallocedString = 'World'.toInt8();
final returnValue = lib.hello(mallocedString).toStr();
free(mallocedString);
return returnValue;

See https://github.com/dart-lang/sdk/blob/a3b15aefcaed10e62444d449394d561b3b12dcb4/samples/ffi/sqlite/lib/src/database.dart#L31-L36.

[listepo]

@dcharkes understood thanks

This would allocate the right memory, encode/decode the Dart string into that memory, do the ffi call, and free the memory.

now it sounds even more important for me :)

[mannprerak2]

If we were to add string support to ffigen we could generate something like

// Input is string instead of pointer
void print(String s) {
  _print ??= _dylib.lookupFunction<_c_print, _dart_print>('print');
  
  // convert string to pointer.
  final ptr = Utf8.toUtf8(s);

  final result = _print(ptr.cast());
  
  // free the pointer.
  free(ptr);
  return result;
}

All conversion logic should still be in package:ffi and then ffigen can simply call those.
We would also need to handle Utf16.
@dcharkes Are ASCII strings not handled by UTF8?

We can also do this for the return type of the function (But I am not sure if we can differentiate between const char* and char* in libclang, so automatically freeing that pointer can throw an error)

I am a little skeptical to add this as converting to and from strings is not always desired (say someone wants to pass string returned by one function to another or say the function modifies the string and returns it, etc) so basically user ends up having String parameters in some and Pointer<Int8> in others.

And I feel this is something simple enough to be added by the user itself, For ffigen we use extension methods for conversion like this and also handle how to dispose things by using libclang functions.

[dcharkes]

I am a little skeptical to add this as converting to and from strings is not always desired (say someone wants to pass string returned by one function to another or say the function modifies the string and returns it, etc) so basically user ends up having String parameters in some and Pointer in others.

Maybe it's worth investigating some common patterns, and then decide if they are regular enough to be part of ffigen or not.

For SQLite it is always the pattern that the caller keeps ownership of the memory for arguments (so the caller has to free the strings) and that all strings are encoded in utf8. The callee is responsible for the memory backing return values, so you never have to free those.

** ^The pointers returned are valid until a type conversion occurs as
** described above, or until [sqlite3_step()] or [sqlite3_reset()] or
** [sqlite3_finalize()] is called.  ^The memory space used to hold strings
** and BLOBs is freed automatically.  Do not pass the pointers returned
** from [sqlite3_column_blob()], [sqlite3_column_text()], etc. into
** [sqlite3_free()].

https://sqlite.org/src/file?name=src/sqlite.h.in&ci=tip

This pattern might be different for other libraries.

However, if the patterns are very predictable, we could generate something.

[mannprerak2]

I think this sums up the way char* can be used anywhere-

1. char* which the user must free (commonly used in SQLite)
2. const char* strings which must not be freed by the user (commonly used in LibClang)
3. In-place C string modification by a function.
4. Passing char* returned by a function to another function as it is.
5. dynamic array of char* (char**) (commonly used for passing cmd args, freeing this also requires the user to know the length)

The only benefit I see is that If ffigen converts char* to String in function parameters, 1 is handled internally(which the user had to do themselves previously).

[dcharkes]

We need an include/exclude structure (in yaml) to make sure that users can include case 1 (for parameters) and 2 (for return values) and users can exclude the last three cases.

For case 1 we can automatically allocate the memory and free it after the call. For return values the callee is responsible for memory management (either it's const memory, or there is some other API call for cleaning up resources).