[vm/ffi] Investigate potential incompatibilities from statically linking libc++

[sigalor]

Operating System: Ubuntu 16.04
Dart version: 2.4.1
Used g++ version: 9.1.0
Related to #34452

To reproduce this issue, please create the following four files in the same directory and run make:

Makefile

all:
	g++ -c foo.cpp -fPIC -o foo.o
	g++ -shared -o libfoo.so foo.o
	g++ main.cpp -o main -L. -lfoo
	LD_LIBRARY_PATH=. ./main
	LD_LIBRARY_PATH=. dart main.dart

main.cpp

extern "C" {
  void foo();
}

int main() {
  foo();
  return 0;
}

foo.cpp

#include <iostream>
#include <stdexcept>

extern "C" {
  void foo() {
    try {
      throw std::runtime_error("Hello exception!");
    } catch(...) {
      std::exception_ptr ex = std::current_exception();
      if(!ex) {
        std::cout << "Missing exception info!" << std::endl;
        return;
      }
    
      try {
        std::rethrow_exception(ex);
      } catch(const std::runtime_error& e) {
        std::cout << e.what() << std::endl;
      }
    }
  }
}

main.dart

import "dart:ffi";

main() {
  var lib = DynamicLibrary.open("libfoo.so");
  void Function() foo = lib.lookup<NativeFunction<Void Function()>>("foo").asFunction();
  foo();
}

Expected behavior: Both ./main and dart main.dart output Hello exception!.

Actual behavior: Dart outputs Missing exception info! instead, because std::current_exception returns a null pointer in foo.cpp.

Although the Dart FFI does not support handling exceptions itself (e.g. passing them back to Dart) right now, the underlying C++ code should be able to process them internally nonetheless. Currently, existing libraries which use std::current_exception might not work anymore or at least debugging becomes considerably harder.

Namely, for debugging, the .so library needs to be compiled with debug information, so that e.g. with GDB one can set a breakpoint before the code that throws the exception, so its message can be inspected there. Of course, in a large code base, where the exact location of the executed throw statement is not known, this requires significantly more effort.

[mit-mit]

cc @sjindel-google @dcharkes

[sjindel-google]

This is reproduced even when the executable is C:

Makefile:

all:
	g++ -c foo.cpp -fPIC -o foo.o
	g++ -shared -o libfoo.so foo.o
	g++ -fno-rtti -fno-exceptions main.cpp -o main -L. -lfoo -ldl
	LD_LIBRARY_PATH=. ./main
	LD_LIBRARY_PATH=. ../out/ReleaseX64/dart main.dart

main.c:

#include <dlfcn.h>

int main() {
  void (*foo)() = (void (*)())dlsym(dlopen("./foo.so", RTLD_LAZY), "foo");
  foo();
  return 0;
}

[sjindel-google]

It appears to be an issue with the manner in which the Dart binary statically links in libstdc++, but exposes some of its symbols.

Your library is allocating the exception via __cxa_allocate_exception from the dart executable, but it's using std::rethrow_exception from the system's libstdc++, which are not compatible.

You can see that GCC's implementation of libstdc++ ignores "foreign" exceptions, which is how your exception is being regarded: https://github.com/gcc-mirror/gcc/blob/41d6b10e96a1de98e90a7c0378437c3255814b16/libstdc%2B%2B-v3/libsupc%2B%2B/eh_ptr.cc#L186

[sjindel-google]

We should not symbols from our statically-linked version of libstdc++ from the dart binary.

We also need to think deeply about other ways this issue can surface. For example, does Pointer.allocate using the same allocator as a loaded library (libc)?

There also may be TLS issues with multiple copies of libstdc++ in the process.

[sjindel-google]

/cc @mkustermann

[greenrobot]

That... sounds scary. Nothing you can fix quickly, I suppose?

[sjindel-google]

I'm not sure whether we can fix it quickly, but you can work around it by using our toolchain.

A full checkout of the Dart SDK with gclient sync (see https://github.com/dart-lang/sdk/wiki/Building) will have our toolchain in buildtools/linux-x64. If you compile your code with our toolchain it will work.

[sjindel-google]

/cc @rmacnak-google @mraleph

[greenrobot]

@sjindel-google Thanks. Long term, ofc, we'd prefer to use our standard dynamic libs from our distribution channel. However, if we'd make a special version for Dart, it might also be an option to link the std lib statically into our lib, right?

[sjindel-google]

In principle yes, however even with -static-libstdc++, both GCC and Clang both link __cxa_allocate_exception (and others) weakly, so they will prefer the version defined in the executable even if the library has it's own copy.

[sjindel-google]

Flutter may not have this issue, but if they do, they (and other embedders) need to address it separately.

[sjindel-google]

@sigalor @greenrobot

The issue you encountered should be fixed now.

We need to do some additional investigation to ensure there are no similar issues.

[greenrobot]

@sigalor Great, happy to hear that! 🎉 Do you have a version number of an upcoming release we will be able to verify with?

[sjindel-google]

The next dev release should contain it. You can see when it's released here: https://github.com/dart-lang/sdk/commits/dev

[mkustermann]

@greenrobot If you are very eager to try it out now, you can download the bleeding-edge build, which should be available here gs://dart-archive/channels/be/raw/166289/sdk/dartsdk-linux-x64-release.zip

[sjindel-google]

It appears that there's not an issue with malloc/free. On Linux we statically link in tcmalloc, but expose the malloc etc. symbols so libc routines and others can use it. On Mac we take malloc etc. from libSystem.dylib dynamically.

I'm not sure that the behavior should be different between OSes, but I can't find a way that it would break.

[blaugold]

OS: Ubuntu 20.04.2 LTS
SDK: Dart SDK version: 2.12.0-133.7.beta (beta) (Tue Jan 12 09:25:38 2021 +0100) on "linux_x64"

I'm having an issue where a dynamic_cast of an exception causes a segmentation fault. This is happening with standalone dart as opposed to flutter/flutter#66575.

To replicate call this function from Dart through ffi:

void throwException()
{
    try
    {
        throw std::runtime_error("Hello exception!");
    }
    catch (const std::exception &e)
    {
        auto cast_e = dynamic_cast<const std::runtime_error *>(&e);
        // std::cout << e.what() << std::endl;
    }
}

The uncommented line also leads to a segmentation fault.

This is not an issue on macOS.

[mkustermann]

@blaugold The Dart C FFI interacts with C code. There is currently no C++ specific support. Throwing C++ exceptions across Dart frames is therefore not supported - and there are also no plans on supporting it in the near future.

The FFI calls to C code expect that the C code returns normally (neither exceptions nor longjmp are supported).

[blaugold]

@mkustermann Thanks for the prompt reply. The issue I'm having surfaces in a C API that wraps a C++ API. This wrapper catches a C++ exception and turns it into an error code. The problem is that examining the exception to create the error code causes a crash.

[mkustermann]

@blaugold Appologies, I misunderstood your problem.

It's unclear to me to what extend the runtime system of libc++ and libstdc++ are compatible. They might not be in certain areas, in which case we cannot do much about it - the shared libraries have to use the same C++ library as our distributed SDK.

Though when trying your example it seems to work for me:

% cat foobar.cc
#include <iostream>

void throwExceptionCpp() {
  try {
    throw std::runtime_error("Hello exception!");
  } catch (const std::exception &e) {
    auto cast_e = dynamic_cast<const std::runtime_error *>(&e);
    std::cout << e.what() << std::endl;
  }
}

extern "C" void throwException() {
  throwExceptionCpp();
}

% clang++ -O0 -stdlib=libstdc++ -fPIC -shared  -o foobar1.so foobar.cc
% clang++ -O0 -stdlib=libc++ -fPIC -shared  -o foobar2.so foobar.cc 

% cat test.dart
import 'dart:ffi';

main() {
  final fun1 = DynamicLibrary.open('foobar1.so')
      .lookupFunction<Void Function(), void Function()>('throwException');
  fun1();

  final fun2 = DynamicLibrary.open('foobar2.so')
      .lookupFunction<Void Function(), void Function()>('throwException');
  fun2();
}

% dart test.dart
Hello exception!
Hello exception!

@blaugold Could you provide the entire example?

[blaugold]

@mkustermann I have created a repo with the example. There is a file run.sh, which builds the example and calls throwException from an executable and then from Dart.

I've only tested the example on macOS and Linux and the issue only comes up on Linux.

[blaugold]

@mkustermann Your right. Using libc++ solved it.

[mkustermann]

@blaugold Glad to hear!

Just out of curiosity it works for me on Linux:

% git clone https://github.com/blaugold/dynamic_cast_dart_issue
% cd dynamic_cast_dart_issue
dynamic_cast_dart_issue %  ./run.sh
> Building example
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /usr/.../repositories/dynamic_cast_dart_issue/build
Scanning dependencies of target Lib
[ 25%] Building CXX object CMakeFiles/Lib.dir/Lib.cpp.o
[ 50%] Linking CXX shared library libLib.so
[ 50%] Built target Lib
Scanning dependencies of target Main
[ 75%] Building CXX object CMakeFiles/Main.dir/Main.cpp.o
[100%] Linking CXX executable Main
[100%] Built target Main


> Call throwException from executable
Hello exception!


> Call throwException from Dart
Hello exception!

[blaugold]

@mkustermann

I noticed that I am using a different version of gcc (GNU 9.3.0).

> Building example
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/gabriel/dev/dynamic_cast_dart_issue/build
Scanning dependencies of target Lib
[ 25%] Building CXX object CMakeFiles/Lib.dir/Lib.cpp.o
[ 50%] Linking CXX shared library libLib.so
[ 50%] Built target Lib
Scanning dependencies of target Main
[ 75%] Building CXX object CMakeFiles/Main.dir/Main.cpp.o
[100%] Linking CXX executable Main
[100%] Built target Main


> Call throwException from executable
Hello exception!


> Call throwException from Dart

===== CRASH =====
si_signo=Segmentation fault(11), si_code=128, si_addr=(nil)
version=2.12.0-133.7.beta (beta) (Tue Jan 12 09:25:38 2021 +0100) on "linux_x64"
pid=21245, thread=21251, isolate_group=main(0x25d19f0), isolate=main(0x25d1170)
isolate_instructions=17e7d20, vm_instructions=17e7d20
  pc 0x0000000001cb461e fp 0x00007f44d76fe720 __dynamic_cast+0x5e
  pc 0x00007f44ee3855bb fp 0x00007f44d76fe748 Unknown symbol
  pc 0x00007f44dcc25154 fp 0x00007f44d76fe770 Unknown symbol
  pc 0x00007f44dcc23fc2 fp 0x00007f44d76fe7c8 Unknown symbol
  pc 0x00007f44dcc23c4f fp 0x00007f44d76fe7f0 Unknown symbol
  pc 0x00007f44dcc23b73 fp 0x00007f44d76fe848 Unknown symbol
  pc 0x00007f44dcc22a9e fp 0x00007f44d76fe878 Unknown symbol
  pc 0x00007f44dcc22806 fp 0x00007f44d76fe8d8 Unknown symbol
  pc 0x00007f44dcc221ce fp 0x00007f44d76fe910 Unknown symbol
  pc 0x00007f44ee38265f fp 0x00007f44d76fe988 Unknown symbol
-- End of DumpStackTrace
[exit     : sp(0) fp(0x7f44d76fe748) pc(0)]
[dart     : sp(0x7f44d76fe758) fp(0x7f44d76fe770) pc(0x7f44dcc25154) *dart:ffi_::_FfiTrampoline ]
[dart     : sp(0x7f44d76fe780) fp(0x7f44d76fe7c8) pc(0x7f44dcc23fc2) file:///home/gabriel/dev/dynamic_cast_dart_issue/main.dart_::_main ]
[dart     : sp(0x7f44d76fe7d8) fp(0x7f44d76fe7f0) pc(0x7f44dcc23c4f) file:///home/gabriel/dev/dynamic_cast_dart_issue/main.dart_::_main_main ]
[dart     : sp(0x7f44d76fe800) fp(0x7f44d76fe848) pc(0x7f44dcc23b73) dart:core__Closure@0150898_dyn_call ]
[dart     : sp(0x7f44d76fe858) fp(0x7f44d76fe878) pc(0x7f44dcc22a9e) dart:isolate_::__delayEntrypointInvocation@1026248_<anonymous closure> ]
[dart     : sp(0x7f44d76fe888) fp(0x7f44d76fe8d8) pc(0x7f44dcc22806) dart:core__Closure@0150898_dyn_call ]
[dart     : sp(0x7f44d76fe8e8) fp(0x7f44d76fe910) pc(0x7f44dcc221ce) dart:isolate__RawReceivePortImpl@1026248__handleMessage@1026248 ]
[entry    : sp(0x7f44d76fe920) fp(0x7f44d76fe988) pc(0x7f44ee38265f)]
/home/gabriel/lib/flutter/bin/internal/shared.sh: line 224: 21245 Aborted                 (core dumped) "$DART" "$@"

[blaugold]

@mkustermann FYI the Dart SDK included in the Flutter SDK still exposes symbols from the c++ abi. The example I provided now works for me, when I use the Dart SDK from https://dart.dev/get-dart. When using dart from the Flutter SDK I still get the crash with the seg fault.

objdump -T ~/lib/flutter/stable/bin/cache/dart-sdk/bin/dart | grep __dynamic_cast
0000000001cc8f00 g    DF .text  00000000000000f4  Base        __dynamic_cast

~/lib/flutter/beta/bin/cache/dart-sdk/bin/dart --version            
Dart SDK version: 2.12.0 (stable) (Thu Feb 25 19:50:53 2021 +0100) on "linux_x64"