Investigate why `InferenceBenchmarkerCache` is getting pinned.

[qinsoon]

d-netto/MMTk-Julia-Fragmentation#1 (comment) showed that with our implementation of address-based hashing for object id (we are still missing #240), InferenceBenchmarkerCache is the major reason we have pinned objects.

We need to investigate why the type is still getting pinned. We could see if those objects are pinned due to conservative stack scanning. If that's the case, there is nothing we can do.

[d-netto]

Will try to take a stab at this.

[d-netto]

Methodology

I implemented a profiler in mmtk/julia#88 to track the pinning sites. At the end of every GC, it prints a JSON dictionary with information about object address, type and pinning sites. Something like this:

  {
    "pinned_object": "0x201084e2b70",
    "type": "InferenceBenchmarkerCache",
    "pinning_sites": [
      {
        "filename": "/home/diogo/Code/MMTk/julia/src/julia.h",
        "lineno": 1399,
        "count": 2
      }
    ]
  },

I instrumented the Julia runtime to log OBJHASH_PIN and OBJ_PIN calls. I instrumented the binding to track objects pinned during conservative register and stack scanning.

• MMTk/julia PR: Pinning Profiler julia#88.

• MMTk/julia Make.user:

export MMTK_JULIA_DIR=/home/diogo/Code/MMTk/mmtk-julia
USE_BINARYBUILDER_MMTK_JULIA=0
WITH_THIRD_PARTY_GC=mmtk
MMTK_PLAN=Immix
MMTK_MOVING=1
MMTK_ALWAYS_MOVING=1
MMTK_MAX_MOVING=1

• Diff I applied to the binding:

git diff 7c7c0fb7bfceae44c294d61365b5c784cb6b7f50 mmtk/src/* Makefile
diff --git a/Makefile b/Makefile
index 3fa6746..8ee9803 100644
--- a/Makefile
+++ b/Makefile
@@ -1,5 +1,7 @@
 # By default do a release build with moving immix
 MMTK_MOVING ?= 1
+MMTK_ALWAYS_MOVING ?= 1
+MMTK_MAX_MOVING ?= 1
 MMTK_PLAN ?= Immix
 CURR_PATH := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
 
diff --git a/mmtk/src/api.rs b/mmtk/src/api.rs
index edf4e82..63761b9 100644
--- a/mmtk/src/api.rs
+++ b/mmtk/src/api.rs
@@ -250,6 +250,11 @@ pub extern "C" fn mmtk_total_bytes() -> usize {
     memory_manager::total_bytes(&SINGLETON)
 }
 
+#[no_mangle]
+pub extern "C" fn mmtk_is_reachable_object(object: ObjectReference) -> bool {
+    object.is_reachable()
+}
+
 #[no_mangle]
 pub extern "C" fn mmtk_is_live_object(object: ObjectReference) -> bool {
     object.is_live()
diff --git a/mmtk/src/collection.rs b/mmtk/src/collection.rs
index f6561ac..4cb7961 100644
--- a/mmtk/src/collection.rs
+++ b/mmtk/src/collection.rs
@@ -4,6 +4,7 @@ use crate::{
     jl_hrtime, jl_throw_out_of_memory_error,
 };
 use crate::{JuliaVM, USER_TRIGGERED_GC};
+use crate::jl_gc_log;
 use log::{debug, trace};
 use mmtk::util::alloc::AllocationError;
 use mmtk::util::heap::GCTriggerPolicy;
@@ -84,6 +85,9 @@ impl Collection<JuliaVM> for VMCollection {
     }
 
     fn resume_mutators(_tls: VMWorkerThread) {
+        unsafe {
+            jl_gc_log();
+        }
         // unpin conservative roots
         crate::conservative::unpin_conservative_roots();
 
diff --git a/mmtk/src/conservative.rs b/mmtk/src/conservative.rs
index 0ce6c80..11d6a38 100644
--- a/mmtk/src/conservative.rs
+++ b/mmtk/src/conservative.rs
@@ -1,4 +1,5 @@
 use crate::julia_types::*;
+use crate::jl_log_pinning_event;
 use mmtk::memory_manager;
 use mmtk::util::constants::BYTES_IN_ADDRESS;
 use mmtk::util::{Address, ObjectReference};
@@ -13,7 +14,16 @@ pub fn pin_conservative_roots() {
 
     let mut roots = CONSERVATIVE_ROOTS.lock().unwrap();
     let n_roots = roots.len();
-    roots.retain(|obj| mmtk::memory_manager::pin_object(*obj));
+    roots.retain(|obj| {
+        // Call C function 'jl_log_pinning_event' to log an event
+        // I.e. something like jl_log_pinning_event(obj, NULL, 4242)
+        unsafe {
+            // Convert ObjectReference to void*
+            let obj = (*obj).to_raw_address();
+            jl_log_pinning_event(obj, std::ptr::null_mut(), 4242);
+        }
+        mmtk::memory_manager::pin_object(*obj)
+    });
     let n_pinned = roots.len();
     log::debug!("Conservative roots: {}, pinned: {}", n_roots, n_pinned);
 }
diff --git a/mmtk/src/lib.rs b/mmtk/src/lib.rs
index a04ddaf..7069485 100644
--- a/mmtk/src/lib.rs
+++ b/mmtk/src/lib.rs
@@ -132,6 +132,10 @@ extern "C" {
         total_end: *mut Address,
     );
     pub static jl_true: *mut crate::julia_types::jl_value_t;
+    // Original C signature: JL_DLLEXPORT void jl_log_pinning_event(void *pinned_object, const char *filename, int lineno)
+    pub fn jl_log_pinning_event(pinned_object: Address, filename: *const i8, lineno: i32);
+    // Original C signature: JL_DLLEXPORT void jl_gc_log(void)
+    pub fn jl_gc_log();
 }
 
 #[macro_export]

Limitations

Transitively pinned objects are not tracked ATM.

Preliminary Analysis

I ran the inference benchmark with this instrumentation enabled to see if we pin too many InferenceBenchmarkerCache during conservative scanning. The answer seems to be no.

The vast majority of InferenceBenchmarkerCache entries look like this:

  {
    "pinned_object": "...",
    "type": "InferenceBenchmarkerCache",
    "pinning_sites": [
      {
        "filename": "/home/diogo/Code/MMTk/julia/src/julia.h",
        "lineno": 1399,
        "count": 2
      }
    ]
  },

but I'll get some numbers on what fraction of InferenceBenchmarkerCache are pinned during conservative scanning and get back here.

[d-netto]

The pinning profile is telling us that a lot of pinning is coming from pinned_ref::create, which is not that helpful.

Added this patch, locally, to the HEAD of mmtk-support-moving-upstream, so that we can track the pinning sites more accurately:

diff --git a/src/julia.h b/src/julia.h
index 011c6c1780..0591fd4908 100644
--- a/src/julia.h
+++ b/src/julia.h
@@ -1396,14 +1396,14 @@ public:
     explicit pinned_ref(void* p) : ptr(static_cast<T*>(p)) {}
     operator void*() const { return ptr; }
     T* get() const { return ptr; }
-    static pinned_ref create(void* p) { OBJ_PIN(p); return pinned_ref(p); }
+    static pinned_ref create(void* p, const char *file, int line) { jl_log_pinning_event(p, file, line); jl_gc_pin_object(p); return pinned_ref(p); }
     static pinned_ref assume(void* p) { return pinned_ref(p); }
 };
 
 // Redefine macros for C++ to use the template version
 #define jl_pinned_ref(T) pinned_ref<T>
 #define jl_pinned_ref_assume(T, ptr) pinned_ref<T>::assume(ptr)
-#define jl_pinned_ref_create(T, ptr) pinned_ref<T>::create(ptr)
+#define jl_pinned_ref_create(T, ptr) pinned_ref<T>::create(ptr, __FILE__, __LINE__)
 #define jl_pinned_ref_get(ref) (ref).get()
 
 #else

We have a lot of entries like this in the pinning profile:

  {
    "pinned_object": "0x200ffd5bce0",
    "type": "InferenceBenchmarkerCache",
    "pinning_sites": [
      {
        "filename": "/home/diogo/Code/MMTk/julia/src/engine.cpp",
        "lineno": 69,
        "count": 1
      },
      {
        "filename": "/home/diogo/Code/MMTk/julia/src/engine.cpp",
        "lineno": 142,
        "count": 1
      }
    ]
  },

[qinsoon]

This is the reason why we used pinning for those two places in engine.cpp.
https://github.com/mmtk/julia/blob/fb84d8f9dfe54506d61055cb23692234bb4881bd/src/engine.cpp#L21-L22

Those are stored in
https://github.com/mmtk/julia/blob/fb84d8f9dfe54506d61055cb23692234bb4881bd/src/engine.cpp#L50-L53

We have a lot of entries like this in the pinning profile:

If this is the main reason why InferenceBenchmarkerCache gets pinned, we can just try not pin those objects and let GC trace the type in some way (or use handles).

[d-netto]

The sources of pinning have been identified.

mmtk/julia#89 refactored the inference engine to reduce the number of pinned InferenceBenchmarkerCache.