[asan] Re-exec without ASLR if needed on 32-bit Linux

[thurstond]

High-entropy ASLR allows up to 16-bits of entropy (2**16 4KB pages == 256MB; a bit more in practice because of implementation details), which is a significant chunk of the user address space on 32-bit systems (4GB or less). This, combined with ASan's shadow (512MB) and ASan's fixed shadow offset (512MB), makes it possible for large binaries to fail to map the shadow.

This patch changes ASan to do a one-time re-exec without ASLR if it cannot map the shadow, thus reclaiming the ~256MB of address space.

Alternatives considered:

1. We don't lower ASan's fixed shadow offset, because that would limit non-PIE binaries.
2. We don't switch to a dynamic shadow offset, because ASan for 32-bit Linux relies on the compile-time constant offset to optimize its instrumentation and compiler-rt.

This is loosely inspired by #78351, #85142, and #85674, though those were required because there were no static shadow mappings that could fully shadow the range of user mappings; this is not the case for ASan.

[llvmbot]

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: Thurston Dang (thurstond)

Changes

High-entropy ASLR allows up to 16-bits of entropy (256MB), which is a significant chunk of the 32-bit address space (4GB, less if running with a 32-bit kernel). This, combined with ASan's shadow (512MB) and ASan's fixed shadow offset (512MB), makes it possible for large binaries to fail to map the shadow.

This patch will re-exec without ASLR if it cannot map the shadow, thus reclaiming the 256MB of address space.

Alternatives considered:

1. We don't attempt to lower ASan's fixed shadow offset, because that
   would limit non-PIE binaries.
2. We don't switch to a dynamic shadow offset, because ASan on 32-bit
   Linux relies on the constant offset to optimize its instrumentation
   and compiler-rt.

This is loosely inspired by
#78351,
#85142, and #85674, though those were required because there were no static mappings that could fully shadow the range of user mappings; this is not the case for ASan.

---

Full diff: https://github.com/llvm/llvm-project/pull/131975.diff

5 Files Affected:

• (modified) compiler-rt/lib/asan/asan_internal.h (+1)
• (modified) compiler-rt/lib/asan/asan_linux.cpp (+28)
• (modified) compiler-rt/lib/asan/asan_mac.cpp (+3)
• (modified) compiler-rt/lib/asan/asan_shadow_setup.cpp (+9)
• (modified) compiler-rt/lib/asan/asan_win.cpp (+4)

diff --git a/compiler-rt/lib/asan/asan_internal.h b/compiler-rt/lib/asan/asan_internal.h
index 06dfc4b177339..8205e19fdab7f 100644
--- a/compiler-rt/lib/asan/asan_internal.h
+++ b/compiler-rt/lib/asan/asan_internal.h
@@ -82,6 +82,7 @@ void ReplaceSystemMalloc();
 uptr FindDynamicShadowStart();
 void AsanCheckDynamicRTPrereqs();
 void AsanCheckIncompatibleRT();
+void TryReExecWithoutASLR();
 
 // Unpoisons platform-specific stacks.
 // Returns true if all stacks have been unpoisoned.
diff --git a/compiler-rt/lib/asan/asan_linux.cpp b/compiler-rt/lib/asan/asan_linux.cpp
index 4cabca388ca9a..7f502df3cf1b4 100644
--- a/compiler-rt/lib/asan/asan_linux.cpp
+++ b/compiler-rt/lib/asan/asan_linux.cpp
@@ -21,6 +21,7 @@
 #  include <pthread.h>
 #  include <stdio.h>
 #  include <sys/mman.h>
+#  include <sys/personality.h>
 #  include <sys/resource.h>
 #  include <sys/syscall.h>
 #  include <sys/time.h>
@@ -107,6 +108,33 @@ void FlushUnneededASanShadowMemory(uptr p, uptr size) {
   ReleaseMemoryPagesToOS(MemToShadow(p), MemToShadow(p + size));
 }
 
+void ReExecWithoutASLR() {
+  // ASLR personality check.
+  // Caution: 'personality' is sometimes forbidden by sandboxes, so only call
+  // this function as a last resort (when the memory mapping is incompatible
+  // and ASan would fail anyway).
+  int old_personality = personality(0xffffffff);
+  bool aslr_on =
+      (old_personality != -1) && ((old_personality & ADDR_NO_RANDOMIZE) == 0);
+
+  if (aslr_on) {
+    // Disable ASLR if the memory layout was incompatible.
+    // Alternatively, we could just keep re-execing until we get lucky
+    // with a compatible randomized layout, but the risk is that if it's
+    // not an ASLR-related issue, we will be stuck in an infinite loop of
+    // re-execing (unless we change ReExec to pass a parameter of the
+    // number of retries allowed.)
+    VReport(1,
+            "WARNING: AddressSanitizer: memory layout is incompatible, "
+            "possibly due to high-entropy ASLR.\n"
+            "Re-execing with fixed virtual address space.\n"
+            "N.B. reducing ASLR entropy is preferable.\n");
+    CHECK_NE(personality(old_personality | ADDR_NO_RANDOMIZE), -1);
+
+    ReExec();
+  }
+}
+
 #  if SANITIZER_ANDROID
 // FIXME: should we do anything for Android?
 void AsanCheckDynamicRTPrereqs() {}
diff --git a/compiler-rt/lib/asan/asan_mac.cpp b/compiler-rt/lib/asan/asan_mac.cpp
index bfc349223258b..be513a03ed5cd 100644
--- a/compiler-rt/lib/asan/asan_mac.cpp
+++ b/compiler-rt/lib/asan/asan_mac.cpp
@@ -55,6 +55,9 @@ uptr FindDynamicShadowStart() {
                           GetMmapGranularity());
 }
 
+// Not used.
+void TryReExecWithoutASLR() {}
+
 // No-op. Mac does not support static linkage anyway.
 void AsanCheckDynamicRTPrereqs() {}
 
diff --git a/compiler-rt/lib/asan/asan_shadow_setup.cpp b/compiler-rt/lib/asan/asan_shadow_setup.cpp
index fc6de39622b51..ad24f63985f8a 100644
--- a/compiler-rt/lib/asan/asan_shadow_setup.cpp
+++ b/compiler-rt/lib/asan/asan_shadow_setup.cpp
@@ -109,6 +109,15 @@ void InitializeShadowMemory() {
     ProtectGap(kShadowGap2Beg, kShadowGap2End - kShadowGap2Beg + 1);
     ProtectGap(kShadowGap3Beg, kShadowGap3End - kShadowGap3Beg + 1);
   } else {
+#  if SANITIZER_LINUX
+    // The shadow mappings can shadow the entire user address space. However,
+    // on 32-bit systems, the maximum ASLR entropy (currently up to 16-bits
+    // == 256MB) is a significant chunk of the address space; reclaiming it by
+    // disabling ASLR might allow chonky binaries to run.
+    if (sizeof(uptr) == 32)
+      TryReExecWithoutASLR();
+#  endif
+
     Report(
         "Shadow memory range interleaves with an existing memory mapping. "
         "ASan cannot proceed correctly. ABORTING.\n");
diff --git a/compiler-rt/lib/asan/asan_win.cpp b/compiler-rt/lib/asan/asan_win.cpp
index 027340280e068..7e6792c78d5e6 100644
--- a/compiler-rt/lib/asan/asan_win.cpp
+++ b/compiler-rt/lib/asan/asan_win.cpp
@@ -43,6 +43,7 @@ uptr __asan_get_shadow_memory_dynamic_address() {
   __asan_init();
   return __asan_shadow_memory_dynamic_address;
 }
+
 }  // extern "C"
 
 // ---------------------- Windows-specific interceptors ---------------- {{{
@@ -279,6 +280,9 @@ uptr FindDynamicShadowStart() {
                           GetMmapGranularity());
 }
 
+// Not used
+void TryReExecWithoutASLR() {}
+
 void AsanCheckDynamicRTPrereqs() {}
 
 void AsanCheckIncompatibleRT() {}

[fmayer]

I would have

if (old_personality == -1) {
  [log error]
  return;
}

[thurstond]

Done

[fmayer]

if the call is within this guard, why do we need a dummy for macos and windows?

[thurstond]

Done

[fmayer]

revert unrelated change

[thurstond]

Done

[rorth]

This patch broke (at least) the Solaris/sparcv9 and Solaris/amd64 bots: <sys/personality.h> is a Linux-only feature, while asan_linux.cpp, despite its name, is used by serveral Unix OSes.

Please fix.

[thurstond]

This patch broke (at least) the Solaris/sparcv9 and Solaris/amd64 bots: <sys/personality.h> is a Linux-only feature, while asan_linux.cpp, despite its name, is used by serveral Unix OSes.

Please fix.

Sorry, my bad. Fixed in 89a1197c1f4f and adb57757b964

[Alcaro]

Is this correct? From what I can see, uptr is an alias of uintptr_t, meaning sizeof is 4 on 32bit systems and this branch is never taken on any platform.

(Yes, I'm late.)

[Alcaro]

...oh, #132682 deleted that check. This question no longer matters.

[thurstond]

From what I can see, uptr is an alias of uintptr_t, meaning sizeof is 4 on 32bit systems and this branch is never taken on any platform.

You're right, that was a bug. Nice catch :-)