How to pass huge objects across isolates and/or ffi, without huge memory and cpu footprint?

[fzyzcjy]

Hi thanks for wonderful dart and flutter!

Scenario: My app have some image processing features in C++, which takes 1s (for example) to compute the output image from the input image. Since I do not want to block the main isolate (otherwise the ui will freeze), I do the following:

Main Isolate <-----SendPort----> Worker Isolate <----FFI----> C++ Code

However, you know images are quite huge, say, 30MB per image. By doing this, one image will have at least 3 copies! The source image is in main isolate, then using SendPort, it is copied (!) to the worker isolate, then again copied (!) to c++ native code. Thus, we use at least 90MB memory when we can simply use 30MB memory, let alone the wasted time for memory copying.

Is there any ways? Thank you so much!

[munificent]

Is there a reason the image needs to be in memory in main isolate at all? Why not have the worker isolate load the image itself directly from disk or wherever it's coming from?

[fzyzcjy]

@munificent Oh actually it is loaded directly from disk by C++ and my description was just for a brief example. For the real case please consider another situation: C++ loads image from disk and process it. Now the manipulated image bytes are in C++ memory. Then c++ must use ffi to pass to worker isolate (copy #1). Then worker isolate must pass to main isolate (copy #2). After that main isolate can finally show to users by using a Image widget. Thus, two copied are needed.

[mraleph]

@fzyzcjy if the image resides in the native memory then you don't need to copy to pass it around between isolates, instead you just pass around a pointer to the image just like you would do that in C/C++. no need to copy actual bytes.

[fzyzcjy]

@mraleph Thanks for the reply! But I want to display the image, so imho it has to be passed to Flutter Image widget (i.e. the main isolate).

[mraleph]

Then you should probably be able to instantiate the image using an so called external typed data (which is a variant of typed data that simply refers to a bytes in the native heap) rather than copying your bytes into a "internal" typed data. When you do ptr.asTypedList() you will get an external typed data - no copy occurs, so Image.memory(ptr.asTypedList()) should do the trick and give your bytes to the Image without actually copying them on the way.

The only missing piece here is memory management. You would want to attach a finalizer to the external typed data to know when you can free the memory - we are working on the feature that will enable you to do that. (see https://github.com/dart-lang/language/blob/master/working/1847%20-%20FinalizationRegistry/proposal.md )

[fzyzcjy]

@mraleph Aha thank you so much! So I guess I cannot use this solution, until the proposal of "finalizer" is implemented?

In addition, I wonder whether the performance will be good or bad? Because for each access to the typed list, dart have to access native heap instead of the managed heap.

[mraleph]

@fzyzcjy if your application contains native code then you can attach the finalizer by writing a small C/C++ wrapper using dynamically linked C API (https://github.com/dart-lang/sdk/blob/main/runtime/include/dart_api_dl.h).

[fzyzcjy]

@mraleph Thank you! To be clear, I guess the procedure is:

Step 1: in native C++ code: uint8_t* bytes = malloc(...); and fill in those bytes.

By the way, should I use malloc or new here?

Step 1b: As your comment and https://github.com/dart-lang/language/blob/master/working/1847%20-%20FinalizationRegistry/proposal.md#dart-vm-embedding-api suggests, I should attach finalizer to a Dart_Handle that actually points to the uint8_t* pointer.

Step 2: Worker isolate talks with C++ code via dart ffi. Suppose my C++/C api is Dart_Handle manipulate_the_image();, then my dart code will call it and get a Dart_Handle or something like that.

Step 3: Worker isolate transfer this pointer to the main isolate. Only a pointer, no actual data.

Step 4: Main isolate gets Pointer<Uint8> ptr. Then it show UI by Image.memory(ptr.asTypedList()).

This sounds like a quite dangerous operation... I have used dart ffi many times before, and saw that a small mistake in memory operation can let the whole app crash... On the other hand, no matter what pure dart code you write, the app will still work happily except an exception - but will never crash (thanks, dart!). So, for example, I wonder is there any examples or boilerplates.

---

Question: I heard that new/delete/malloc/free are quite dangerous and not very encouraged in modern C++. Therefore, I wonder whether I can use something less dangerous here? For example, smart pointers? Or I have to use raw new/delete?

Thanks!

[mraleph]

Step 1: in native C++ code: uint8_t* bytes = malloc(...); and fill in those bytes.

Correct. You need to allocate memory using a native allocation mechanism.

By the way, should I use malloc or new here?

Does not matter. You choose what is most suitable for your use case.

Step 1b: As your comment and https://github.com/dart-lang/language/blob/master/working/1847%20-%20FinalizationRegistry/proposal.md#dart-vm-embedding-api suggests, I should attach finalizer to a Dart_Handle that actually points to the uint8_t* pointer.

No, you need to attach finalizer to some real Dart object which is going to "own" the life-time of the pointer, e.g. if you create external typed data from a pointer then you attach finalizer to that external typed data.

I'd recommend to not attach any finalizer until the pointer reaches you main isolate. If you want to attach finalizer in all isolates that the pointer is passing through then you need to use some sort of reference counting, because it should not be destroyed if any isolates still use it.

Step 2: Worker isolate talks with C++ code via dart ffi. Suppose my C++/C api is Dart_Handle manipulate_the_image();, then my dart code will call it and get a Dart_Handle or something like that.
Step 3: Worker isolate transfer this pointer to the main isolate. Only a pointer, no actual data.

Yes, with a minor correction that manipulate_the_image should probably return just uint8_t* not a handle.

Step 4: Main isolate gets Pointer ptr. Then it show UI by Image.memory(ptr.asTypedList()).

Yes, with a minor correction: you should do Image.memory(asTypedListWithFinalizer(ptr)) - where you create typedlist and attach finalizer to it.

An alternative implementation could be to use native Dart_PostCObject which allows you to send a piece of native memory from one isolate to another and attach finalizer to it at the same time. You can do something like this:

Dart_CObject message;
message.type = Dart_CObject_kExternalTypedData;
message.value.as_external_typed_data.type = Dart_TypedData_kUint8;
message.value.as_external_typed_data.length = length;
message.value.as_external_typed_data.data = reinterpret_cast<uint8_t*>(image);
message.value.as_external_typed_data.peer = image;
message.value.as_external_typed_data.callback = [](void* isolate_callback_data, void* peer) {
 free(peer);  // peer points to image buffer, see above.
};
result = Dart_PostCObject(port, &message);

ReceivePort will get a Uint8List (no copying) and when that list dies the memory will be freed. This might be a better approach then sending pointers.

[mraleph]

I have filed dart-lang/sdk#47270 and I am going to close this request because it does not directly related to language evolution (which is what language repo is about).

[fzyzcjy]

@mraleph Thank you so much for your detailed reply! That Dart_PostCObject approach seems to be much easier to use and I will choose it.

A quick question: Where should I get the port?

Another quick question: Is there suggested ways to test flutter code with such ffi code? IMHO I cannot use valgrind or santizers :/ So memory leak problem, double free or other memory problems are quite hard to detect. I remembered once when the app crash directly after some time, and I finally realized it is due to a double free (by dart gc and by myself to a native pointer).

[mraleph]

@fzyzcjy you can get port id of a SendPort through this method https://api.dart.dev/dev/2.15.0-139.0.dev/dart-ffi/NativePort/nativePort.html

As for testing then indeed there is no good approach right now.

[fzyzcjy]

@mraleph

you can get port id of a SendPort through this method https://api.dart.dev/dev/2.15.0-139.0.dev/dart-ffi/NativePort/nativePort.html

thanks!

As for testing then indeed there is no good approach right now.

Ah... :/

[fzyzcjy]

@mraleph Hi, I need to do the counterpart as well now... My isolate (which calls the ffi) wants to pass a big Uint8List to the C++/Rust code via FFI. Can I do this without copying?

This big Uint8List is created by my isolate, or created by the C++/Rust code. Both are possible.

If it is created by C++/Rust code, I know I can only pass a pointer from C++ to Flutter. Then later I can reuse the pointer from Flutter to C++. This does have the problem of memory leaking, though. Hope there is a better solution!

If it it created by my isolate, currently I do not know how to pass it as a pointer (or something that I do not need to copy) from Flutter to C++.

Thanks!

[mraleph]

If it is created by C++/Rust code, I know I can only pass a pointer from C++ to Flutter. Then later I can reuse the pointer from Flutter to C++. This does have the problem of memory leaking, though. Hope there is a better solution!

You should be able to use finalisers for this and track whether ownership is on the Dart side (in which case finalizer deletes memory) or already on the C++ side (in which case Dart does nothing in finalizer).

If it it created by my isolate, currently I do not know how to pass it as a pointer (or something that I do not need to copy) from Flutter to C++.

You can't really pass a pointer to the inner contents of Uint8List without copying, because GC can move the list an pointer will become invalid.

So if you want to pass data without copying you must allocate it yourself on the native heap.

[fzyzcjy]

@mraleph Thanks for the information!

So if you want to pass data without copying you must allocate it yourself on the native heap.

Is the performance of Uint8List and a "external typed data" (just like what you suggested above and sent by Dart_PostCObject) the same or not? For example, passing it to a Image widget, or directly access its index like for(var i = 0; i < arr.length; ++i) arr[i] = something * arr[i];

[mraleph]

Performance should be roughly the same in Dart code (I would like to avoid overloading you with some subtleties) and should be completely the same when you give it to Flutter's low level APIs because they have a way to get low-level view on the bytes.

[fzyzcjy]

@mraleph Thank you very much!

[xinyu391]

resides
Is it possible?
Dart isolates has differenet memory space, pass one native memory address from one isolate to another isolate, the memyory should be different on the same address.

[mraleph]

@xinyu391 Dart isolates all exist within the same process and thus within the same virtual address space.

[toasdn]

An alternative implementation could be to use native Dart_PostCObject which allows you to send a piece of native memory from one isolate to another and attach finalizer to it at the same time. You can do something like this:

Dart_CObject message;
message.type = Dart_CObject_kExternalTypedData;
message.value.as_external_typed_data.type = Dart_TypedData_kUint8;
message.value.as_external_typed_data.length = length;
message.value.as_external_typed_data.data = reinterpret_cast<uint8_t*>(image);
message.value.as_external_typed_data.peer = image;
message.value.as_external_typed_data.callback = [](void* isolate_callback_data, void* peer) {
free(peer); // peer points to image buffer, see above.
};
result = Dart_PostCObject(port, &message);
ReceivePort will get a Uint8List (no copying) and when that list dies the memory will be freed. This might be a better approach then sending pointers.

Now with Dart 3, can one use NativeCallable.listerner to get that same job done?