Sending data to an isolate locks the main thread for some time

[Hixie]

The following little flutter program creates an isolate, and times how long the main thread locks while sending a 100MB of zero bytes to an isolate. It does this by having a timer run continuously on the main thread, timing how long since it last ran, and printing any time that it took more than 5ms between invocations ("since last checkin"). If the main thread is never locked, this should never print anything. At the same time, in a Future-mediated loop on the main thread, it sends a 100MB ByteData buffer of zeros to the isolate, then awaits a response, and prints the total round-trip time.

The isolate merely runs a Future-mediated loop that waits for a message, then sends a single byte back.

For flutter, what matters is primarily that the total overhead of sending something to a thread is small (small single-digits of milliseconds at most, ideally microseconds). Total round-trip time is only of academic interest so long as it's not measured in minutes.

import 'dart:async';
import 'dart:isolate';
import 'dart:typed_data';

// true = time the in-thread overhead for sending a big message out of the main thread
// false = time	the in-thread overhead for receiving a big message into the main thread
const bool benchmarkSend = true;

const int toIsolateSize = benchmarkSend ? 100 * 1024 * 1024 : 1;
const int fromIsolateSize = benchmarkSend ? 1 : 100 * 1024 * 1024;

Future<Null> main() async {
  Timer.run(idleTimer);
  ReceivePort port = new ReceivePort();
  StreamIterator<dynamic> inbox = new StreamIterator<dynamic>(port);
  Isolate.spawn(isolateMain, port.sendPort);
  await inbox.moveNext();
  SendPort outbox = inbox.current;
  Stopwatch workWatch = new Stopwatch();
  ByteData data = new ByteData(toIsolateSize);
  while (true) {
    print('sending...');
    workWatch.start();
    outbox.send(data);
    await inbox.moveNext();
    workWatch.stop();
    int	time = workWatch.elapsedMilliseconds;
    print('${time}ms for round-trip');
    workWatch.reset();
  }
}
 
Future<Null> isolateMain(SendPort outbox) async {
  ReceivePort port = new ReceivePort();
  StreamIterator<dynamic> inbox = new StreamIterator<dynamic>(port);
  outbox.send(port.sendPort);
  ByteData data = new ByteData(fromIsolateSize);
  while (true) {
    await inbox.moveNext();
    outbox.send(data);
  }
}

Stopwatch idleWatch = new Stopwatch();

void idleTimer() {
  idleWatch.stop();
  int time = idleWatch.elapsedMilliseconds;
  if (time > 5)
    print('${time}ms since last checkin');
  idleWatch.reset();
  idleWatch.start();
  Timer.run(idleTimer);
}

Here is some representative output for this script running on a Pixel XL 2. The total overhead for sending 100MB to another isolate appears to be in the double-digit millisecond range, which means that sending 100MB to another isolate guarantees that the application will miss a frame.

I/flutter ( 3337): sending...
I/flutter ( 3337): 16ms since last checkin
I/flutter ( 3337): 127ms for round-trip
I/flutter ( 3337): sending...
I/flutter ( 3337): 15ms since last checkin
I/flutter ( 3337): 126ms for round-trip
I/flutter ( 3337): sending...
I/flutter ( 3337): 16ms since last checkin
I/flutter ( 3337): 130ms for round-trip
I/flutter ( 3337): sending...
I/flutter ( 3337): 16ms since last checkin
I/flutter ( 3337): 126ms for round-trip
I/flutter ( 3337): sending...
I/flutter ( 3337): 15ms since last checkin
I/flutter ( 3337): 124ms for round-trip
I/flutter ( 3337): sending...
I/flutter ( 3337): 16ms since last checkin
I/flutter ( 3337): 124ms for round-trip
I/flutter ( 3337): sending...
I/flutter ( 3337): 16ms since last checkin
I/flutter ( 3337): 122ms for round-trip
I/flutter ( 3337): sending...

See also #31959, which is a much more serious problem with receiving data taking double-digit milliseconds.

[mraleph]

@Hixie

Currently there is almost 0 data sharing between isolates. The data is sent by copying - which means you need to copy the whole 100MB between isolates.

Dart VM has support for external typed data - it can be passed between isolates without copying, but APIs for allocating such typed data are not exposed to the Dart code.

[Hixie]

One thing we could do is mark the buffer as frozen as soon as it is sent, then copy it on a different thread, and on the main thread, if we ever try to edit a frozen buffer, then do the copy (maybe only copying one page, and using a page table to indirect accesses into the buffer, if we find that this is common). So long as we document that once you've sent a buffer, you should never mutate it again or you'll pay a high copy cost, that would be fine. Most buffers aren't likely to be edited anyway, they'll be obtained from data and then processed into other data.

In any case, sending immutable data has the same problem. Change data to be a String created as follows:

String data = ''.padLeft(toIsolateSize, '\0');

...and the numbers are actually worse:

I/flutter ( 8055): sending...
I/flutter ( 8055): 16ms since last checkin
I/flutter ( 8055): 371ms for round-trip
I/flutter ( 8055): sending...
I/flutter ( 8055): 16ms since last checkin
I/flutter ( 8055): 370ms for round-trip
I/flutter ( 8055): sending...
I/flutter ( 8055): 16ms since last checkin
I/flutter ( 8055): 374ms for round-trip
I/flutter ( 8055): sending...
I/flutter ( 8055): 78ms since last checkin
I/flutter ( 8055): 435ms for round-trip
I/flutter ( 8055): sending...
I/flutter ( 8055): 16ms since last checkin
I/flutter ( 8055): 375ms for round-trip
I/flutter ( 8055): sending...
I/flutter ( 8055): 16ms since last checkin
I/flutter ( 8055): 376ms for round-trip
I/flutter ( 8055): sending...

Notice the occasional ~80ms delays, and the increased round-trip time, even though the data is theoretically the same. It's probably twice as big since Dart uses UTF-16 strings. Halving the size of the data being sent brings the overall overhead down to the 8ms range on my device (still too high, you'd skip frames at 120Hz) and the round-trip time is still higher, ~200ms instead of ~120ms. Also the occasional 80ms delays becomes 36ms delays.

In this example, though, the data is immutable so the overhead should be zero (mark the string as busy, send a pointer to another thread, have it copy the data, receive a control message back saying the copy is done, mark the string as no longer busy so it can be GC'ed, moved, or whatever).

[mraleph]

It's probably twice as big since Dart uses UTF-16 strings.

Under the hood VM uses one-byte storage for Latin1 strings (which LICENSE is) and two-byte storage for everything else that does not fit into Latin1.

mark the string as busy, send a pointer to another thread, have it copy the data, receive a control message back saying the copy is done, mark the string as no longer busy so it can be GC'ed, moved, or whatever

Well, that's not that trivial. If string was allocated in young space you can't just pin it - young space in the current GC design must be fully evacuated by the GC, so you'll have to pause young space GCs until all pinned data is moved out which will introduce some jank.

Though if large objects are allocated on it's own pages then things become easier because those we can certainly pin.

[a-siva]

seems to be related to #31959

[aam]

cc @mkustermann

[a-siva]

@lrhn should we consider surfacing APIs in Dart to allow allocation of external typed data supported by the VM in Dart code.

[mraleph]

@a-siva fwiw as a workaround you already can allocate external typed data through FFI - you can allocate memory and then use asTypedList on a pointer.