Fix deadlock/timeout in thread unwinding.

When malloc debug is enabled, using libbacktrace to unwind can
result in a deadlock. This happens when an unwind of a thread
is occuring which triggers a signal to be sent to that thread. If
that thread is interrupted while a malloc debug function is
executing and owns a lock, that thread is then stuck in the signal
handler. Then the original unwinding thread attempts to do an
allocation and gets stuck waiting for the same malloc debug lock.

This is not a complete deadlock since the unwinder has timeouts,
but it results in truncated unwinds that take at least five
seconds to complete.

Only the backtrace signals needs to be blocked because it is the only
known signal that will result in a thread being paused in a signal
handler.

Also, added a named signal in the reserved signal list for the
special bionic backtrace signal.

Bug: 150833265

Test: New unit tests pass with fix, fail without fix.
Change-Id: If3e41f092ebd40ce62a59ef51d636a91bc31ed80

1 files changed, 47 insertions, 0 deletions

diff --git a/libc/malloc_debug/tests/malloc_debug_system_tests.cpp b/libc/malloc_debug/tests/malloc_debug_system_tests.cpp
index ac19cf99c..1bfe61e77 100644
--- a/libc/malloc_debug/tests/malloc_debug_system_tests.cpp
+++ b/libc/malloc_debug/tests/malloc_debug_system_tests.cpp
@@ -42,12 +42,20 @@
 #include <gtest/gtest.h>
 #include <log/log_read.h>
 
+#include <atomic>
 #include <string>
 #include <thread>
 #include <vector>
 
+#include <backtrace/Backtrace.h>
+#include <backtrace/BacktraceMap.h>
+
 #include <bionic/malloc.h>
 
+// All DISABLED_ tests are designed to be executed after malloc debug
+// is enabled. These tests don't run be default, and are executed
+// by wrappers that will enable various malloc debug features.
+
 static constexpr time_t kTimeoutSeconds = 10;
 
 extern "C" bool GetInitialArgs(const char*** args, size_t* num_args) {
@@ -525,3 +533,42 @@ TEST(MallocDebugSystemTest, write_leak_info_header) {
   ASSERT_NO_FATAL_FAILURE(FindStrings(pid, std::vector<const char*>{"malloc debug enabled"},
                           std::vector<const char*>{" HAS INVALID TAG ", "USED AFTER FREE ", "UNKNOWN POINTER "}));
 }
+
+TEST(MallocTests, DISABLED_malloc_and_backtrace_deadlock) {
+  std::atomic_bool running(false);
+  pid_t tid;
+  std::thread thread([&tid, &running] {
+    tid = gettid();
+    running = true;
+    while (running) {
+      void* ptr = malloc(200);
+      if (ptr == nullptr) {
+        return;
+      }
+      free(ptr);
+    }
+  });
+
+  while (!running) {
+  }
+
+  static constexpr size_t kNumUnwinds = 1000;
+  for (size_t i = 0; i < kNumUnwinds; i++) {
+    std::unique_ptr<Backtrace> backtrace(Backtrace::Create(getpid(), tid));
+    ASSERT_TRUE(backtrace->Unwind(0)) << "Failed on unwind " << i;
+    ASSERT_LT(1, backtrace->NumFrames()) << "Failed on unwind " << i;
+  }
+  running = false;
+  thread.join();
+}
+
+TEST(MallocDebugSystemTest, malloc_and_backtrace_deadlock) {
+  pid_t pid;
+  ASSERT_NO_FATAL_FAILURE(Exec("MallocTests.DISABLED_malloc_and_backtrace_deadlock",
+                               "verbose verify_pointers", &pid, 0, 180));
+
+  // Make sure that malloc debug is enabled and that no timeouts occur during
+  // unwinds.
+  ASSERT_NO_FATAL_FAILURE(FindStrings(pid, std::vector<const char*>{"malloc debug enabled"},
+                                      std::vector<const char*>{"Timed out waiting for "}));
+}