munmap is not being intercepted for cache refresh

[jsquyres]

Per the thread starting here:

http://www.open-mpi.org/community/lists/users/2014/11/25730.php

munmap is not being intercepted properly. Late in the thread, it appears that this is happening because the wrapper compiler is only -lmpi, and not explicitly bringing in libopen-pal (where the munmap intercept lives). This means that the user is bringing in munmap from libc, and not seeing the OMPI munmap.

Hence, we're not intercepting munmap, and Badness occurs.

One thought on how to fix this is to re-introduce linking to libopen-rte and libopen-pal in the wrapper compilers. We distinctly took this behavior out at one point (and deliberately just linked against libmpi), but I confess to not remembering the exact reason why. It may have been taken out just because "it's the right thing -- we can have implicit dependencies pull in the rest", or it may have been so that we could support external ORTE and OPAL installations. Not sure. Someone will need to spelunk into the history to find out why. This may give insight into whether we can put the -lopen-rte -lopen-pal back in the wrappers.

[emmanuelthome]

the rationale for why the direct dependency on open-rte and open-pal was dropped is there

https://svn.open-mpi.org/trac/ompi/ticket/2092

E.

[jsquyres]

@emmanuelthome Excellent spelunking! Many thanks for reminding me that Past Jeff wrote out the whole reason why we no longer explicitly -lopen-rte -lopen-pal.

@markalle @gpaulsen This might be an excellent argument to bring in the memory hook replacements we discussed this past week (but we'll need to solve the UCX-has-the-same-memory-hooks issue).

[emmanuelthome]

@jsquyres You're welcome ;-)

Speaking of this, I got bitten again this week by some bug which smells like a similar "hook does get called" issue (not 100% sure yet. But at least an openib-specific segfault which disappears with mpi_leave_pinned 0...). I briefly considered trying out ummunotify. Seems to be dead and buried, unfortunately. #429 hints in that direction, at least. Do you confirm ?

[jsquyres]

@emmanuelthome #429 has been on my to-do list for quite a while, and I haven't gotten to it. :-( Mellanox saw some failures that looked like the Open MPI ummunotify code paths were broken, but they didn't investigate deeply -- that's what #429 is about.

There was a conversation at the Open MPI dev meeting this past week (https://github.com/open-mpi/ompi/wiki/Meeting-2016-02) about using the Platform MPI method of memory hooks. That might end up moving forward, which could solve both this issue and (maybe?) obviate the need for ummunotify support...?

That being said, the Platform MPI method is apparently identical to the UCX method, and therefore they can conflict with each other in userspace. Hence, ummunotify may well still be the One True Answer (i.e., kernel-level support). If you have a little time, if you could verify if #429 is actually due to ummunotify code paths in OMPI being broken, that would be most helpful.

[gpaulsen]

It SHOULD be possible that while we're setting up the memory hooks at the symbol level, to determine if a hook is ALREADY installed, and chain the hooks. If ALL of the hooks do this, (i.e. UCX and us) then this could work (analogous to how signal handlers chain together). Of course the signal handler approach is well documented by the OS, and this is not documented and pretty hacky.

I feel like this is edging out further on a thin tree limb that should have broken long ago.

[markalle]

Looking at the code it's "probably" possible to detect a prior usage of this trick, and even save the function pointer that the previous product had registered.  Then inside our interception we could either

1. call syscall(__SYS_munmap/etc, ...) if there was no previous instance of the trick in place2. call their function pointer if there was one
    
   But that seems fragile, both in detection and in the case where multiple products use the same trick (including saving the previously registered product's function pointer). They could get set up into an infinite loop of calling each other back and forth.
    
    
   I'm more inclined to say we look for a small coordination between the two projects:
   Open MPI could offer an
       ompi_memory_register(functionptr)
   where we'd make a weak guarantee of service that IF our product manages to detect memory going away, we'll call functionptr(addr,len).  UCX could offer a similar function guaranteeing the same.
    
   Then we each register our deregistration callback with each other, and we each use the patch_symbol() trick. We wouldn't necessarily know which product would win, but whichever it is, that product calls the other product's deregistration for them.
    
   Mark
    
    
   ----- Original message -----From: Geoff Paulsen [email protected]: open-mpi/ompi [email protected]: Mark Allen/Dallas/IBM@IBMUSSubject: Re: [ompi] munmap is not being intercepted for cache refresh (munmap is not being intercepted for cache refresh #299)Date: Wed, Mar 2, 2016 10:40 PM 
   It SHOULD be possible that while we're setting up the memory hooks at the symbol level, to determine if a hook is ALREADY installed, and chain the hooks. If ALL of the hooks do this, (i.e. UCX and us) then this could work (analogous to how signal handlers chain together). Of course the signal handler approach is well documented by the OS, and this is not documented and pretty hacky.
   I feel like this is edging out further on a thin tree limb that should have broken long ago.
   —Reply to this email directly or view it on GitHub.

 

[gpaulsen]

I will ask to get Mark to contribute the Platform MPI solution for this fix ASAP, possibly with the hook to also allow other hooks to fire during any hook event (similar to signal handlers).

[jsquyres]

@hjelmn has posted a PR with this functionality (after a bunch of off-list discussions with @markalle and @gpaulsen). See #1495.

[jsquyres]

This was closed via #1495.