mimalloc + address sanitizer + std::map(w/length > 1) = error

[mighty1231]

Issue for -sMALLOC=mimalloc + -fsanitize=address

file main.cc, or here

#include <iostream>

#include <unordered_map>

std::unordered_map<std::string, std::pair<bool, bool>> NameToArgUsageMap{
    {"relu", {false, false}},
    {"leakyrelu", {true, false}}};

int main() {
    for (const auto& pair : NameToArgUsageMap) {
        const std::string& key = pair.first;
        const std::pair<bool, bool>& value = pair.second;
        std::cout << key << ": {" << value.first << ", " << value.second << "}" << std::endl;
    }

    return 0;
}

Version of emscripten/emsdk:

docker image emscripten/emsdk:3.1.59, emscripten/emsdk:3.1.74

Failing command line in full:

docker run --rm --user=$(id -u):$(id -g) -t -v $(pwd):/src --workdir /src \
  emscripten/emsdk:3.1.59 em++ \
  -sINITIAL_MEMORY=200mb \
  -sMALLOC=mimalloc \
  -fsanitize=address \
  -o out-3.1.59-false-10-mimalloc-address.js main.cc
node out-3.1.59-false-10-mimalloc-address.js

I tried various test cases for this bug, as in here

• emsdkVersion: 3.1.59, 3.1.74
• map type: unordered_map, map
• size of std::map(or unordered): 1, 2, 5, 10
• mallocType: emmalloc, dlmalloc, mimalloc
• sanitizer: (null), undefined, address

Total 2 * 2 * 4 * 3 * 3 = 144 cases in here.

What kind of error?

• type 1. RuntimeError: remainder by zero (5)

  • 3.1.74, unordered_map, size=2, mimalloc, address
  • 3.1.74, unordered_map, size=5, mimalloc, address
  • 3.1.74, map, size=10, mimalloc, address
  • 3.1.74, map, size=2, mimalloc, address
  • 3.1.74, map, size=5, mimalloc, address

• type 2. ==42==ERROR: LeakSanitizer: detected memory leaks (5)

  • 3.1.59, unordered_map, size=10, mimalloc, address
  • 3.1.59, unordered_map, size=2, mimalloc, address
  • 3.1.59, unordered_map, size=5, mimalloc, address
  • 3.1.59, map, size=10, mimalloc, address
  • 3.1.59, map, size=5, mimalloc, address

• type 3. ==42==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x04d00000 at pc 0x00057169 bp 0x0288c790 sp 0x0288c794 (1)

  • 3.1.74, unordered_map, size=10, mimalloc, address

Discussion

In summary, only just in case, any emsdk ver+any maptype +sanitizer=address+mallocType=mimalloc+size>1(not 1. it works properly), generates error, with exceptional case, 3.1.59+map+size2, (it seems flakyEDIT: I have run 2 times more, it seems to be the unique option that runs properly)

[mighty1231]

I tried -sINITIAL_MEMORY as 1000mb instead 200mb, but results have not changed.

[kripken]

When I do

./emcc a.cpp -sMALLOC=mimalloc -fsanitize=address -g

I get a type 1 result on latest git:

leakyrelu: {1, 0}
relu: {0, 0}
wasm://wasm/a.out.wasm-01464922:1


RuntimeError: remainder by zero
    at a.out.wasm.mi_free_generic_mt (wasm://wasm/a.out.wasm-01464922:wasm-function[343]:0x1344d)
    at a.out.wasm.mi_free_size (wasm://wasm/a.out.wasm-01464922:wasm-function[346]:0x13635)
    at a.out.wasm.void std::__2::__libcpp_operator_delete[abi:ne180100]<void*, unsigned long>(void*, unsigned long) (wasm://wasm/a.out.wasm-01464922:wasm-function[234]:0xf3ee)
    at a.out.wasm.void std::__2::__do_deallocate_handle_size[abi:ne180100]<>(void*, unsigned long) (wasm://wasm/a.out.wasm-01464922:wasm-function[232]:0xf32a)
    at a.out.wasm.std::__2::__libcpp_deallocate[abi:ne180100](void*, unsigned long, unsigned long) (wasm://wasm/a.out.wasm-01464922:wasm-function[230]:0xf265)
    at a.out.wasm.std::__2::allocator<std::__2::__hash_node<std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>, void*>>::deallocate[abi:ne180100](std::__2::__hash_node<std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>, void*>*, unsigned long) (wasm://wasm/a.out.wasm-01464922:wasm-function[245]:0xfb01)
    at a.out.wasm.std::__2::allocator_traits<std::__2::allocator<std::__2::__hash_node<std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>, void*>>>::deallocate[abi:ne180100](std::__2::allocator<std::__2::__hash_node<std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>, void*>>&, std::__2::__hash_node<std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>, void*>*, unsigned long) (wasm://wasm/a.out.wasm-01464922:wasm-function[242]:0xfa1b)
    at a.out.wasm.std::__2::__hash_table<std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>, std::__2::__unordered_map_hasher<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>, std::__2::hash<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>>, std::__2::equal_to<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>>, true>, std::__2::__unordered_map_equal<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>, std::__2::equal_to<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>>, std::__2::hash<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>>, true>, std::__2::allocator<std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>>>::__deallocate_node(std::__2::__hash_node_base<std::__2::__hash_node<std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>, void*>*>*) (wasm://wasm/a.out.wasm-01464922:wasm-function[246]:0xfcaa)
    at a.out.wasm.std::__2::__hash_table<std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>, std::__2::__unordered_map_hasher<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>, std::__2::hash<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>>, std::__2::equal_to<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>>, true>, std::__2::__unordered_map_equal<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>, std::__2::equal_to<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>>, std::__2::hash<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>>, true>, std::__2::allocator<std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>>>::~__hash_table() (wasm://wasm/a.out.wasm-01464922:wasm-function[50]:0x3c16)
    at a.out.wasm.std::__2::unordered_map<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>, std::__2::hash<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>>, std::__2::equal_to<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>>, std::__2::allocator<std::__2::pair<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>> const, std::__2::pair<bool, bool>>>>::~unordered_map[abi:ne180100]() (wasm://wasm/a.out.wasm-01464922:wasm-function[49]:0x3b08)

I then enabled all assertions:

diff --git a/tools/system_libs.py b/tools/system_libs.py
index ef896f6ec..a672580e9 100644
--- a/tools/system_libs.py
+++ b/tools/system_libs.py
@@ -1806,9 +1806,9 @@ class libmimalloc(MTLibrary):
     # build mimalloc with an override of malloc/free
     '-DMI_MALLOC_OVERRIDE',
     # TODO: add build modes that include debug checks 1,2,3
-    '-DMI_DEBUG=0',
+    '-DMI_DEBUG=3',
     # disable `assert()` in the underlying emmalloc allocator
-    '-DNDEBUG',
+    #'-DNDEBUG',
     # avoid use of `__builtin_thread_pointer()`
     '-DMI_LIBC_MUSL',
   ]

Then this happens:

leakyrelu: {1, 0}
relu: {0, 0}
mimalloc: assertion failed: at "../../../system/lib/mimalloc/src/free.c":426, mi_page_usable_size_of
  assertion: "ok"

Aborted(native code called abort())
 a.out.js:622
  /** @suppress {checkTypes} */ var e = new WebAssembly.RuntimeError(what);
                                        ^

RuntimeError: Aborted(native code called abort())
    at abort ( a.out.js:622:41)
    at __abort_js ( a.out.js:4076:24)
    at a.out.wasm.abort (wasm://wasm/a.out.wasm-015b3b7e:wasm-function[392]:0x270f4)
    at a.out.wasm._mi_assert_fail (wasm://wasm/a.out.wasm-015b3b7e:wasm-function[549]:0x30094)
    at a.out.wasm.mi_page_usable_size_of (wasm://wasm/a.out.wasm-015b3b7e:wasm-function[420]:0x2813d)
    at a.out.wasm.mi_free_size (wasm://wasm/a.out.wasm-015b3b7e:wasm-function[419]:0x2806c)
    at a.out.wasm.void std::__2::__libcpp_operator_delete[abi:ne180100]<void*, unsigned long>(void*, unsigned long) (wasm://wasm/a.out.wasm-015b3b7e:wasm-function[234]:0x1bde7)
    at a.out.wasm.void std::__2::__do_deallocate_handle_size[abi:ne180100]<>(void*, unsigned long) (wasm://wasm/a.out.wasm-015b3b7e:wasm-function[232]:0x1bd23)
    at a.out.wasm.std::__2::__libcpp_deallocate[abi:ne180100](void*, unsigned long, unsigned long) (wasm://wasm/a.out.wasm-015b3b7e:wasm-function[230]:0x1bc5e)
    at a.out.wasm.std::__2::allocator<std::__2::__hash_node<std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>, void*>>::deallocate[abi:ne180100](std::__2::__hash_node<std::__2::__hash_value_type<std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>>, std::__2::pair<bool, bool>>, void*>*, unsigned long) (wasm://wasm/a.out.wasm-015b3b7e:wasm-function[245]:0x1c93c)

So this looks like an internal error in mimalloc. In theory it could be memory corruption that is not mimalloc's fault, but we are using asan here, and it finds nothing, so I lean towards that.

It is worth trying to reduce this to a testcase that reproduces without Emscripten (using a native build of mimalloc), that we can file an issue for.

[mighty1231]

It is worth trying to reduce this to a testcase that reproduces without Emscripten (using a native build of mimalloc), that we can file an issue for.

This comment claims mimalloc should not be used when using the address sanitizer.

This issue may be closed, since the issue may not related directly to emscripten.

[sbc100]

We should probably add an error in that case then when those flags are used it combination.