v4.0.x: RDMA OSC: initialize segment memory before registering the segment

[hjelmn]

Signed-off-by: Joseph Schuchart [email protected]
(cherry picked from commit d9dcdfd)
Signed-off-by: Nathan Hjelm [email protected]

[hjelmn]

@bwbarrett Don't hold up v4.0.0 for this. Can wait for v4.0.1 if v4.0.0 is imminent.

@jsquyres Looks good to me. This just ensures the memory is touched by the owning MPI process before its registered.

[jsquyres]

According to #5921, the goal for this PR is to achieve NUMA locality. Do we really need to memset() the entire window to achieve NUMA locality? Couldn't we do something a bit lighter-weight? I ask because I'm concerned about the performance for large windows.

I know you can use first touch, but it's possible that a naieve approach (e.g., touch the first byte in each page) would have the same end effect of setting locality, yet have less of a performance impact. I'm not 100% sure of that, though, since touching the first byte of a page may cause a paging action, anyway...?

Additionally, hwloc may be able to set the locality -- perhaps before you allocate. Check out https://www.open-mpi.org/projects/hwloc/doc/v2.0.2/a00155.php.

[devreal]

@jsquyres Good point. I meant to perform some measurements on the window allocation performance this week (our system is currently in maintenance though). I would expect that the page allocation is more expensive than the actual memset (I might wrong) but touching the first bytes of the page is certainly an option. I will look into it.

[jsquyres]

@devreal I don't know if the page allocation actually brings the pages in to real memory. It may also be suitable to use hwloc to alloc the pages exactly where you want them...?

[devreal]

The way I understand the documentation you linked, hwloc_set_area_membind is the only viable option since the memory is already allocated as shared memory segment. I will give that a try and come back with some measurements on the allocation overhead in #5915.

[jsquyres]

Ah, good point that this is allocated as shared memory. Duh.

Ok, I guess it's just testing / reading code to see if there's any meaningful difference between touching 1 byte on each page vs. memset() of the entire area.

[gpaulsen]

Moving to v4.0.1

[jsquyres]

See #5969 -- there may be a problem with this PR...?

[jsquyres]

Per #5969, it looks like this PR isn't ready yet.

[devreal]

@jsquyres Yes, I think we can close this PR, I haven't had time to work on this again but I think there might be a better solution.

[jsquyres]

@devreal Cool. I'll close this PR. Feel free to re-open / open a new one with whatever comes out of #5969. Thanks!