String parameter support

[erjo1776]

From what I can gather, strings are not directly supported as parameters. Of course being C and not C++ makes this is a messy topic (char * is not always a string).

Is there anything that can help here? Are there plans for supporting strings?

If strings are somehow supported, then please change the simple example to use them (the other two examples are a bit heavy to teach a conceptually basic type).

[dcharkes]

dart:ffi exposes char* as Pointer<Int8> and package:ffi exposes it as Pointer<Utf8>.

This package could take either of those two options (or have it configurable). Currently it exposes it as Pointer<Int8>, see https://github.com/dart-lang/ffigen/blob/master/test/large_integration_tests/_expected_sqlite_bindings.dart#L2410-L2416.

If strings are somehow supported, then please change the simple example to use them

👍

[mannprerak2]

We should also add this to the readme's FAQ section(we don't have one currently).

[erjo1776]

OK, the following works for me:
myLib.say(Utf8.toUtf8('plain old C string').cast<Int8>());
This converts Dart string to Utf8 and then recasts it to Int8 which my 'char *' C function accepts.

This page helped me for the conversion:
https://pub.dev/documentation/ffi/latest/ffi/Utf8-class.html

And then IntelliSense (like) feature showed me 'cast<>()'. I am new to Dart but a long time veteran of C, C++, Java (and variants), and Python.

Btw, I tried Gluecodium for generating bindings between C++ & Dart, but it tries to also handle Java JNI and Swift and I never got a full build to work although C++ & Dart appeared to complete. Their Hello World is really Hello The Entire World. Aside from strings and forcing myself to use C instead of C++, FFIgen is working well so far (very easy to understand in goes Header and out comes DartBinding).

Thanks!

[listepo]

As an option, add extensions to package:ffi

extensions.dart

import 'dart:ffi';

import 'package:ffi/ffi.dart';

extension StringExtensions on String {
  Pointer<Int8> toInt8() {
    return Utf8.toUtf8(this).cast<Int8>();
  }
}

extension PointerExtensions<T extends NativeType> on Pointer<T> {
  String toStr() {
    if (T == Int8) {
      return Utf8.fromUtf8(cast<Utf8>());
    }

    throw UnsupportedError('${T} unsupported');
  }
}

Utf8.fromUtf8(lib.hello(Utf8.toUtf8('World').cast()).cast());

vs

lib.hello('World'.toInt8()).toStr();

@dcharkes @mannprerak2 what do you think?

[dcharkes]

The actual API for strings is still up in the air, awaiting dart-lang/sdk#39787.

However, it would be fine to add some extension methods in the mean time.
I would prefer the extension methods to be on Utf8 though.

extension Utf8PointerString on Pointer<Utf8> {

Even better would be to have package:ffigen do the conversion.
@mannprerak2 can we add an option (or possible a filter based on function/argument names) to ffigen to treat char* as null-terminated Utf8? And alternatively as null-terminated ASCII? (Those are probably the two most common encodings used.) Can you make a design for that?

strings:
  - encoding: utf8
    functions:
      include: sqlite.*
  - encoding: ascii
    functions:
      include: my_function
    arguments:
      include: my_string

This would allocate the right memory, encode/decode the Dart string into that memory, do the ffi call, and free the memory.

lib.hello('World'.toInt8()).toStr();

@listepo You're leaking the memory backing World here (unless the native hello frees the memory).

[listepo]

@dcharkes thanks

[listepo]

extension Utf8PointerString on Pointer<Utf8> {
  String toStr() {
    final res = Utf8.fromUtf8(this);
    free(this);
    
    return res;
  }
}

@dcharkes did you talk about something like that?

[dcharkes]

No.

lib.hello('World'.toInt8()).toStr();

Should be:

final mallocedString = 'World'.toInt8();
final returnValue = lib.hello(mallocedString).toStr();
free(mallocedString);
return returnValue;

See https://github.com/dart-lang/sdk/blob/a3b15aefcaed10e62444d449394d561b3b12dcb4/samples/ffi/sqlite/lib/src/database.dart#L31-L36.

[listepo]

@dcharkes understood thanks

This would allocate the right memory, encode/decode the Dart string into that memory, do the ffi call, and free the memory.

now it sounds even more important for me :)

[mannprerak2]

If we were to add string support to ffigen we could generate something like

// Input is string instead of pointer
void print(String s) {
  _print ??= _dylib.lookupFunction<_c_print, _dart_print>('print');
  
  // convert string to pointer.
  final ptr = Utf8.toUtf8(s);

  final result = _print(ptr.cast());
  
  // free the pointer.
  free(ptr);
  return result;
}

All conversion logic should still be in package:ffi and then ffigen can simply call those.
We would also need to handle Utf16.
@dcharkes Are ASCII strings not handled by UTF8?

We can also do this for the return type of the function (But I am not sure if we can differentiate between const char* and char* in libclang, so automatically freeing that pointer can throw an error)

I am a little skeptical to add this as converting to and from strings is not always desired (say someone wants to pass string returned by one function to another or say the function modifies the string and returns it, etc) so basically user ends up having String parameters in some and Pointer<Int8> in others.

And I feel this is something simple enough to be added by the user itself, For ffigen we use extension methods for conversion like this and also handle how to dispose things by using libclang functions.

[dcharkes]

I am a little skeptical to add this as converting to and from strings is not always desired (say someone wants to pass string returned by one function to another or say the function modifies the string and returns it, etc) so basically user ends up having String parameters in some and Pointer in others.

Maybe it's worth investigating some common patterns, and then decide if they are regular enough to be part of ffigen or not.

For SQLite it is always the pattern that the caller keeps ownership of the memory for arguments (so the caller has to free the strings) and that all strings are encoded in utf8. The callee is responsible for the memory backing return values, so you never have to free those.

** ^The pointers returned are valid until a type conversion occurs as
** described above, or until [sqlite3_step()] or [sqlite3_reset()] or
** [sqlite3_finalize()] is called.  ^The memory space used to hold strings
** and BLOBs is freed automatically.  Do not pass the pointers returned
** from [sqlite3_column_blob()], [sqlite3_column_text()], etc. into
** [sqlite3_free()].

https://sqlite.org/src/file?name=src/sqlite.h.in&ci=tip

This pattern might be different for other libraries.

However, if the patterns are very predictable, we could generate something.

[mannprerak2]

I think this sums up the way char* can be used anywhere-

1. char* which the user must free (commonly used in SQLite)
2. const char* strings which must not be freed by the user (commonly used in LibClang)
3. In-place C string modification by a function.
4. Passing char* returned by a function to another function as it is.
5. dynamic array of char* (char**) (commonly used for passing cmd args, freeing this also requires the user to know the length)

The only benefit I see is that If ffigen converts char* to String in function parameters, 1 is handled internally(which the user had to do themselves previously).

[dcharkes]

We need an include/exclude structure (in yaml) to make sure that users can include case 1 (for parameters) and 2 (for return values) and users can exclude the last three cases.

For case 1 we can automatically allocate the memory and free it after the call. For return values the callee is responsible for memory management (either it's const memory, or there is some other API call for cleaning up resources).

[dcharkes]

Another alternative is to provide some kind of API that has a function that is called for each parameter and the return value both before and after the call. That way users could implement this API and use other native calls to do string resource management (for example the way that clang requires you to use api calls to look into and free strings).

[listepo]

@dcharkes @mannprerak2 is it possible to check that the Pointer returns NULL?

[mannprerak2]

@listepo Not sure if your query is related to this issue, but you can do ptr==nullptr to check, this works because null pointers are simply Pointer.fromAddress(0)

[listepo]

@mannprerak2 thanks

[vaind]

Now with the Dart SDK 2.12 and its new FFI methods, maybe it makes sense to revisit this issue

extension StringUtf8Pointer on String {
  Pointer<Utf8> toNativeUtf8({Allocator allocator = malloc})
}

extension Utf8Pointer on Pointer<Utf8> {
  String toDartString({int? length})
}

[dcharkes]

@vaind can you point us to an API where you'd want to have this? Also, how would like to identify which char* arguments to have generated as String on the Dart side?

[unsuitable001]

As of Dart SDK version: 2.12.2
I can't find toUtf8 method in Utf8 class. Instead, used this:
'plain old C string'.toNativeUtf8().cast<Int8>()
So, we're using a method from Dart string for the conversion instead of from Utf8 class. This turns the Dart string into Pointer<Utf8> that is then cased to Pointer<Int8>.

OK, the following works for me:
myLib.say(Utf8.toUtf8('plain old C string').cast<Int8>());
This converts Dart string to Utf8 and then recasts it to Int8 which my 'char *' C function accepts.

[smakarov]

This is not working
remoteFileName.toNativeUtf8().cast<ffi.Int8>(),
What i have to do to pass dart String into the native char * (which becomes after ffigen ffi.Pointer<ffi.Int8>)?

[mannprerak2]

@smakarov did you import package:ffi? since this method is an extension on string.

Edit: https://pub.dev/packages/ffi

[smakarov]

@smakarov did you import package:ffi? since this method is an extension on string.

Edit: https://pub.dev/packages/ffi

Yes, I don't have any compile errors. Only runtime.

[mannprerak2]

Could you share the error message?

[smakarov]

Could you share the error message?

It's an internal native library error about wrong file handler.

[mannprerak2]

@smakarov Okay, since its not related to ffigen, consider asking your question on StackOverflow.

[smakarov]

@smakarov Okay, since its not related to ffigen, consider asking your question on StackOverflow.

I don't know where to look in now :( I use the same library with C# wrapper and it works fine. Also it works in Flutter but with local file where I pass String into file parameter, but char *(for playing stream audio file form url) is just not working.

[smakarov]

I've found some strange behavior.
I'm using this project https://github.com/JimTompkins/flutter_bass which trying to adapt BASS audio library to Flutter.
With local file it works fine and plays the track, but with internet stream file it didn't work.
My observation is that it's not working in emulator or when i build debug to real device and leave it plugged into Android Studio. But if close the app and open it again the problem is gone!
It also works when building a release version while leave it plugged into Android Studio.
It works with release version in emulator too, but with great lags.
What could this be related to?