hook/prot: Connectivity Map #2825

jjhursey · 2017-01-25T17:10:06Z

This PR requires that PR Add a 'hook' framework #2773 be merged in - since this is a component of that framework.
- This PR includes the hook framework PR.
- Only the top-most commit should be reviewed for discussion
The goal of this PR is to expose this mechanism for discussion
See Issue Show MPI connectivity map during MPI_INIT #30 for more discussion

artpol84 · 2017-01-25T23:50:35Z

bot:mellanox:retest

jjhursey · 2017-01-26T16:42:05Z

Some notes on the tunable options and output.

-mca hook_prot_verbose VALUE
- General component verbosity
-mca hook_prot_enable_mpi_init BOOL
- Enable map display at the bottom of MPI_Init
-mca hook_prot_enable_mpi_finalize BOOL
- Enable map display at the top of MPI_Finalize
-mca hook_prot_platform_prot VALUE
- Alias environment variable: MPI_PROT
- 1 : Same as -mca hook_prot_enable_mpi_init t
- 2 : Same as -mca hook_prot_enable_mpi_finalize t
Environment variable: MPI_PROT_MAX
- When to stop printing the prot table

Output lower than or equal to MPI_PROT_MAX

     host | 0    1    2    3    4    5    6    7    8
    ======|==============================================
        0 : shm  ib   ib   ib   ib   ib   ib   ib   ib
        1 : ib   shm  ib   ib   ib   ib   ib   ib   ib
        2 : ib   ib   self ib   ib   ib   ib   ib   ib
        3 : ib   ib   ib   self ib   ib   ib   ib   ib
        4 : ib   ib   ib   ib   self ib   ib   ib   ib
        5 : ib   ib   ib   ib   ib   self ib   ib   ib
        6 : ib   ib   ib   ib   ib   ib   self ib   ib
        7 : ib   ib   ib   ib   ib   ib   ib   self ib
        8 : ib   ib   ib   ib   ib   ib   ib   ib   self

Output above MPI_PROT_MAX to 3*MMPI_PROT_MAX:

     host | 0 1 2 3 4       8
    ======|====================
        0 : A C C C C C C C C
        1 : C A C C C C C C C
        2 : C C B C C C C C C
        3 : C C C B C C C C C
        4 : C C C C B C C C C
        5 : C C C C C B C C C
        6 : C C C C C C B C C
        7 : C C C C C C C B C
        8 : C C C C C C C C B
    key: A == shm
    key: B == self
    key: C == ib

jjhursey · 2017-01-26T16:44:12Z

ompi/mca/hook/prot/hook_prot_fns.c

+        }
+
+        hostidprotptr = getenv("MPI_PROT_BRIEF");
+        if (hostidprotptr) { hostidprotbrief = atoi(hostidprotptr); }


@markalle It looks like MPI_PROT_BRIEF value is not used. Was the intention to force the "brief" output for all -prot output?

jjhursey · 2017-01-26T18:31:40Z

@markalle @gpaulsen We need to review PR #1974

During Open MPI face-to-face
- Try changing the discovery mechanism to use the model seen in PR MCA component filter #1974
- Once that change is in then then community would like to review it again, but is likely to accept for the v3.x timeframe if it is ready in time.

gpaulsen · 2017-01-26T19:10:05Z

Also, there are some more details from face to face at the bottom of the face to face minutes

markalle · 2017-01-26T19:23:59Z

For what it's worth the part they really want is 9a908f5b9bddd6675621c784c832441533cb4f73 that changes "unic" to "usnic" Then the extra change 8f775b1f3a1ca856791164f5b971a2de274f4733was just for fun, because I thought it was neat to be able to fit larger tables on the screen. Mark ----- Original message -----From: Geoff Paulsen <[email protected]>To: open-mpi/ompi <[email protected]>Cc: Mark Allen/Dallas/IBM@IBMUS, Mention <[email protected]>Subject: Re: [open-mpi/ompi] hook/prot: Connectivity Map (#2825)Date: Thu, Jan 26, 2017 1:10 PM Also, there are some more details from face to face at the bottom of the face to face minutes —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

jjhursey · 2017-01-26T20:38:31Z

I think we might be able to use something like PR #1974 to get the string associated with the component. However, I think we might need some discussion about how the community feels about the section of this PR where we get the list of components actually being used between rank pairs.

Currently, we added a call to pml/ob1 and pml/cm to get that information. Do folks think that is fine, or are there alternative suggestions?

hppritcha · 2017-02-01T03:59:26Z

bot:lanl:retest

hppritcha · 2017-02-01T10:48:53Z

the LANL dlopen-disable seems to have found a legit problem with this PR:

make[2]: Entering directory `/home/hppritcha/jenkins/workspace/ompi_master_pr_disable_dlopen/ompi/tools/ompi_info'
  CC       ompi_info.o
  CC       param.o
  GENERATE ompi_info.1
  CCLD     ompi_info
../../../ompi/.libs/libmpi.so: undefined reference to `dlsym'
collect2: error: ld returned 1 exit status
make[2]: *** [ompi_info] Error 1
make[2]: Leaving directory `/home/hppritcha/jenkins/workspace/ompi_master_pr_disable_dlopen/ompi/tools/ompi_info'
make[1]: *** [install-recursive] Error 1
make[1]: Leaving directory `/home/hppritcha/jenkins/workspace/ompi_master_pr_disable_dlopen/ompi'
make: *** [install-recursive] Error 1

jjhursey · 2017-02-01T13:59:58Z

@hppritcha Yeah, you are right. We need to do some work on this PR anyway before it's ready to review. We'll note that error for fixing during the next revision.

gpaulsen

reports are that --disable-dlopen fails with this PR, but upon code inspection, it's unclear how the two can be related.

hppritcha · 2017-02-05T16:26:05Z

bot:lanl:retest

jjhursey · 2017-02-06T13:39:37Z

This PR needs some significant work before it's ready to merge. So I wouldn't worry too much about the CI until that work has started.

hppritcha · 2017-03-07T16:58:58Z

bot:lanl:retest

jjhursey · 2017-03-07T17:29:43Z

Now that the hook framework has been committed to master I need to update this branch. I'll do that today.

* `-mca hook_prot_verbose VALUE` * General component vebosity * `-mca hook_prot_enable_mpi_init BOOL` * Enable map display at the bottom of `MPI_Init` * `-mca hook_prot_enable_mpi_finalize BOOL` * Enable map display at the top of `MPI_Finalize` * `-mca hook_prot_platform_prot VALUE` * Alias environment variable: `MPI_PROT` * `1 : Same as -mca hook_prot_enable_mpi_init t` * `2 : Same as -mca hook_prot_enable_mpi_finalize t` Signed-off-by: Joshua Hursey <[email protected]>

Normally we print a -prot table up to 16 hosts that looks like this, where 16 can be changed via MPI_PROT_MAX: ``` host | 0 1 2 3 4 5 6 7 8 ======|============================================== 0 : shm ib ib ib ib ib ib ib ib 1 : ib shm ib ib ib ib ib ib ib 2 : ib ib self ib ib ib ib ib ib 3 : ib ib ib self ib ib ib ib ib 4 : ib ib ib ib self ib ib ib ib 5 : ib ib ib ib ib self ib ib ib 6 : ib ib ib ib ib ib self ib ib 7 : ib ib ib ib ib ib ib self ib 8 : ib ib ib ib ib ib ib ib self ``` This checkin reduces MPI_PROT_MAX to 12 but adds a shorter table output that looks like this: ``` host | 0 1 2 3 4 8 ======|==================== 0 : A C C C C C C C C 1 : C A C C C C C C C 2 : C C B C C C C C C 3 : C C C B C C C C C 4 : C C C C B C C C C 5 : C C C C C B C C C 6 : C C C C C C B C C 7 : C C C C C C C B C 8 : C C C C C C C C B key: A == shm key: B == self key: C == ib ``` That is used from 13 up to 36 ranks (or 3*MPI_PROT_MAX). Signed-off-by: Joshua Hursey <[email protected]>

jjhursey · 2017-03-07T17:34:42Z

The branch has been rebased onto master. The two commits represent the prot component.

There is still work to do on this feature - so I'm keeping the WIP label on it.

gpaulsen · 2017-03-09T18:47:56Z

:retest:

bwbarrett · 2017-03-20T18:16:26Z

@jjhursey are you looking to add this to the 3.0 release?

jjhursey · 2017-03-20T18:19:56Z

It still needs some work. I don't think it'll make it for v3.0, but should be ready for the release that follows.

gpaulsen · 2017-03-24T18:35:15Z

The community decided on a call 2 weeks ago, to NOT to take this for v3.0, but to aim for v3.1.

bwbarrett · 2017-03-27T02:51:07Z

Thanks, @gpaulsen. Apparently, I have no short term memory, hence writing everything down...

ibm-ompi · 2017-04-11T08:02:18Z

The IBM CI (PGI Compiler) build failed! Please review the log, linked below.

Gist: https://gist.github.com/73ca9ce0ceed7b37f7d60da3250d7259

jjhursey · 2017-08-31T15:08:42Z

I'm going to close this PR for now. @gpaulsen @markalle Let's re-open a new one once we have finished refining. We should restart that conversation as I forget what is left to do.

gpaulsen · 2020-08-05T14:39:41Z

This Functionality was later re-implemented for the v5.0 timeframe in #5507

jjhursey added enhancement ⚠️ WIP-DNM! labels Jan 25, 2017

jjhursey assigned gpaulsen and markalle Jan 25, 2017

jjhursey force-pushed the topic/hook-fwk-w-prot branch from 198137b to 40346fb Compare January 25, 2017 18:57

jjhursey commented Jan 26, 2017

View reviewed changes

jjhursey mentioned this pull request Jan 26, 2017

Show MPI connectivity map during MPI_INIT #30

Closed

gpaulsen approved these changes Feb 5, 2017

View reviewed changes

gpaulsen added the Target: v2.x label Feb 14, 2017

jsquyres added Target: v3.0.x and removed Target: v2.x labels Feb 21, 2017

markalle added 2 commits March 7, 2017 11:32

jjhursey force-pushed the topic/hook-fwk-w-prot branch from 685176a to a760be1 Compare March 7, 2017 17:33

hppritcha removed the Target: v3.0.x label Mar 21, 2017

hppritcha added this to the Future milestone Mar 21, 2017

jjhursey closed this Aug 31, 2017

jjhursey deleted the topic/hook-fwk-w-prot branch August 31, 2017 15:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

hook/prot: Connectivity Map #2825

hook/prot: Connectivity Map #2825

jjhursey commented Jan 25, 2017 •

edited

Loading

artpol84 commented Jan 25, 2017

jjhursey commented Jan 26, 2017

jjhursey Jan 26, 2017

jjhursey commented Jan 26, 2017 •

edited

Loading

gpaulsen commented Jan 26, 2017

markalle commented Jan 26, 2017 via email

jjhursey commented Jan 26, 2017

hppritcha commented Feb 1, 2017

hppritcha commented Feb 1, 2017

jjhursey commented Feb 1, 2017

gpaulsen left a comment

hppritcha commented Feb 5, 2017

jjhursey commented Feb 6, 2017

hppritcha commented Mar 7, 2017

jjhursey commented Mar 7, 2017

jjhursey commented Mar 7, 2017

gpaulsen commented Mar 9, 2017

bwbarrett commented Mar 20, 2017

jjhursey commented Mar 20, 2017

gpaulsen commented Mar 24, 2017

bwbarrett commented Mar 27, 2017

ibm-ompi commented Apr 11, 2017

jjhursey commented Aug 31, 2017

gpaulsen commented Aug 5, 2020

hook/prot: Connectivity Map #2825

hook/prot: Connectivity Map #2825

Conversation

jjhursey commented Jan 25, 2017 • edited Loading

artpol84 commented Jan 25, 2017

jjhursey commented Jan 26, 2017

jjhursey Jan 26, 2017

Choose a reason for hiding this comment

jjhursey commented Jan 26, 2017 • edited Loading

gpaulsen commented Jan 26, 2017

markalle commented Jan 26, 2017 via email

jjhursey commented Jan 26, 2017

hppritcha commented Feb 1, 2017

hppritcha commented Feb 1, 2017

jjhursey commented Feb 1, 2017

gpaulsen left a comment

Choose a reason for hiding this comment

hppritcha commented Feb 5, 2017

jjhursey commented Feb 6, 2017

hppritcha commented Mar 7, 2017

jjhursey commented Mar 7, 2017

jjhursey commented Mar 7, 2017

gpaulsen commented Mar 9, 2017

bwbarrett commented Mar 20, 2017

jjhursey commented Mar 20, 2017

gpaulsen commented Mar 24, 2017

bwbarrett commented Mar 27, 2017

ibm-ompi commented Apr 11, 2017

jjhursey commented Aug 31, 2017

gpaulsen commented Aug 5, 2020

jjhursey commented Jan 25, 2017 •

edited

Loading

jjhursey commented Jan 26, 2017 •

edited

Loading