Failure to stop at MPIR_Breakpoint with OpenMPI v3.1.x and v4.0.x

[louisespellacy-arm]

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

v3.1.6 and v4.0.3

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

from source tarball with PGI 19.5 and 20.1

Please describe the system on which you are running

• Operating system/version: RHEL7
• Computer hardware: x86_64
• Network type: self

---

Details of the problem

GDB unable to stop at MPIR_Breakpoint when debugging mpirun process with the MPIR interface with out-of-the-box OpenMPI installations. MPIR_Breakpoint is being optimized out.
#5501 relates to this issue and the reproducer can be re-used. This issue is not occurring with PGI 18.7.

However, providing CFLAGS=-O1, FCFLAGS=-O1 and CXXFLAGS=-O1 when building OpenMPI allows GDB to stop at MPIR_Breakpoint.

It does not work if you use -O2.

Upon looking at the building of orted/orted_submit.c, it is being built with CFLAGS when it shouldn't be.

The compile and link lines being generated by automake in orte/Makefile are:

liborted_mpir_la_LINK = $(LIBTOOL) $(AM_V_lt) --tag=CC \
      $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=link $(CCLD) \
      $(liborted_mpir_la_CFLAGS) $(CFLAGS) $(AM_LDFLAGS) $(LDFLAGS) \
      -o $@

$(AM_V_CC_no)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) \
--mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(AM_CPPFLAGS) \
$(CPPFLAGS) $(liborted_mpir_la_CFLAGS) $(CFLAGS) -c -o orted/liborted_mpir_la-orted_submit.lo \
`test -f 'orted/orted_submit.c' || echo '$(srcdir)/'`orted/orted_submit.c

In orte/orted/Makefile.am, the CFLAGS are defined to remove the optimzations, but $(CFLAGS) is being added to the compile line, adding the optimzations anyway.

liborted_mpir_la_CFLAGS = $(CFLAGS_WITHOUT_OPTFLAGS) $(DEBUGGER_CFLAGS)

By manually removing CFLAGS from the compile and link line for orted_submit.c, there would be no optimization when building orted/orted_submit.c, as described in the comments.

[awlauria]

What version of automake are you using?

It was found that older versions of automake are adding the -O3, and newer versions are doing the correct thing in not adding it.

See this comment/discussion:

#6828 (comment)

[awlauria]

it's also documented in the code of orted_submit.c, for reference:

https://github.com/awlauria/ompi/blob/00106f5ac96a3d9e6288ec07dc47e325897cd5f8/orte/orted/orted_submit.c#L181

[jsquyres]

I think @awlauria is asking: are you running autogen.pl during the build process? Or just ./configure / make?

[louisespellacy-arm]

I was building out-of-the-box openmpi-3.1.6.tar.gz and openmpi-4.0.3.tar.gz when first testing. Using
the configure script that it ships with.
./configure / make
Unless modified by the user, the resulting CFLAGS is:

$ ompi_info -a | grep CFLAGS
Build CFLAGS: -O3 -DNDEBUG -finline-functions -fno-strict-aliasing

By just running ./configure / make you get the following line in the orte Makefile:

$(AM_V_CC_no)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS) --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(AM_CPPFLAGS) $(CPPFLAGS) $(liborted_mpir_la_CFLAGS) $(CFLAGS) -c -o orted/liborted_mpir_la-orted_submit.lo `test -f 'orted/orted_submit.c' || echo '$(srcdir)/'`orted/orted_submit.c

Later I pulled from the most recent branch of v4.0.x and used automake/configure/make etc. This results in the makeline for orted_submit.c as seen in my comment.

 $ automake --version
automake (GNU automake) 1.15.1

To ensure that $(CFLAGS) was not added, I needed to make changes similar to those found in https://github.com/awlauria/ompi/blob/00106f5ac96a3d9e6288ec07dc47e325897cd5f8/ompi/debuggers/Makefile.am#L28

[louisespellacy-arm]

Is there any update on this issue? This issue persists with OpenMPI 4.1.0 recently released and PGI 20.1.

orted_submit.c is still being compiled with -O3.

[awlauria]

@louisespellacy-arm sorry, this slipped through the cracks. I'll take a look.

[louisespellacy-arm]

Attempted to fix the above issue with the following changes but unsure if its the best approach:

diff --git a/orte/orted/Makefile.am b/orte/orted/Makefile.am
index 1235e51e69..7523cc5336 100644
--- a/orte/orted/Makefile.am
+++ b/orte/orted/Makefile.am
@@ -19,6 +19,8 @@
 #
 # $HEADER$
 #
+CFLAGS = $(CFLAGS_WITHOUT_OPTFLAGS) $(DEBUGGER_CFLAGS)
+#
 
 # This makefile.am does not stand on its own - it is included from orte/Makefile.am
 
@@ -38,7 +40,7 @@ lib@ORTE_LIB_PREFIX@open_rte_la_SOURCES += \
 noinst_LTLIBRARIES += liborted_mpir.la
 liborted_mpir_la_SOURCES = \
        orted/orted_submit.c
-liborted_mpir_la_CFLAGS = $(CFLAGS_WITHOUT_OPTFLAGS) $(DEBUGGER_CFLAGS)
+#liborted_mpir_la_CFLAGS = $(CFLAGS_WITHOUT_OPTFLAGS) $(DEBUGGER_CFLAGS)
 
 lib@ORTE_LIB_PREFIX@open_rte_la_LIBADD += liborted_mpir.la

[awlauria]

@louisespellacy-arm while that will work in a pinch, I don't think that is the best solution. Doing that will propagate the debugger CFLAGS for everything below the orte tree (losing -O3) for basically all of orte as far as I can tell.

Unfortunately, it seems that libtool is blasting every file with -O3 again. The intent was (as far as I remember) to not compile this file with -O3, but I don't see an easy/clean way of doing that for this one library.

One way that I can think of is to have something like an OPAL_C/CPPFLAGS variable that propagates down to every Makefile.am. It's a tedious change, but will give more control to every library component on what CFLAGS to build with.

[awlauria]

I'm no configure expert, so maybe there's a better way. I have a branch where this is partially implemented, and it seems to as well as before. Basically I just replaced CFLAGS/CPPFLAGS with OPAL_CFLAGS/OPAL_CPPFLAGS, and set CFLAGS/CPPFLAGS to be empty, preventing libtool from tacking them on. Then I've gone across a subset of makefiles and set them for each library. They will also have to be passed down to the likes of hwloc/prrte and other 3rd party packages, but that should be just as easy.

This approach is in line with what gnu suggests in the automake docs:

Using variables like this gives you full control over the ordering of the flags. For instance, 
if there is a flag in $(WARNINGCFLAGS) that you want to negate for a particular target,
you can use something like 'prog1_CFLAGS = $(AM_CFLAGS) -no-flag'. If all of 
 these flags had been forcefully appended to CFLAGS, there would be no way to disable one flag.
 Yet another reason to leave user variables to users. C

https://www.gnu.org/software/automake/manual/html_node/Flag-Variables-Ordering.html

I can continue this work and post a patch if the community is interested.

[jjhursey]

The OPAL_{CFLAGS|CPPFLAGS} method looks like it would work, but is pretty time-intensive to implement. Before going too much further down that road the community (at least those familiar with the build system - @jsquyres @bwbarrett @ggouaillardet mayeb others) should weigh in so Austen doesn't waste effort here if there is another way.

The only other way I thought of was to inject a script at the end of configure that would modify the generated Makefile (or at the end of autogen.pl if we can find a way) to strip out the CFLAGS for the orte_submit.c compilation step. It's pretty hacky, but should do the trick.

@gpaulsen if we don't reach a decision beforehand, can you add this to next week's agenda?

[jsquyres]

It looks like we did this differently in https://github.com/open-mpi/ompi/blob/master/ompi/debuggers/Makefile.am -- we overrode CFLAGS for the entire Makefile.am. It looks like we did that because the _LINK line for the .lo files are like this (from the rendered orte/orted/Makefile -- showing line numbers and line breaks from my text editor, sorry):

2461 orted/liborted_mpir_la-orted_submit.lo: orted/orted_submit.c
2462         $(AM_V_CC)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) $(LIBTOOLFLAGS)\
      --mode=compile $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(AM_CPPFLAGS) $(CPPF\
     LAGS) $(liborted_mpir_la_CFLAGS) $(CFLAGS) -MT orted/liborted_mpir_la-orted_submit.\
     lo -MD -MP -MF orted/$(DEPDIR)/liborted_mpir_la-orted_submit.Tpo -c -o orted/libort\
     ed_mpir_la-orted_submit.lo `test -f 'orted/orted_submit.c' || echo '$(srcdir)/'`ort\
     ed/orted_submit.c

Notice how it still has $(CFLAGS) in there (which contains things like -O3). ☹️

We can override CFLAGS for the entire ompi/debuggers/Makefile.am because that's a small Makefile.am.

This is not the case for orte/orted/Makefile.am -- that file is included in the upper-level orte/Makefile.am rather than traversing down into orte/orted and invoking make. We can't override CFLAGS for all of ORTE, because that will have larger side effects (e.g., remove opt flags like -O3 from all ORTE objects).

It may well be necessary to make orte/orted/Makefile.am be a standalone Makefile.am (where you can override CFLAGS for the whole file)...?

[awlauria]

It may well be necessary to make orte/orted/Makefile.am be a standalone Makefile.am (where you can override CFLAGS for the whole file)...?

That would work to limit the scope some, but all files beneath orte/orted would also be affected (there's a pmix dir there), right?

The global CFLAGS hammer is...burdensome to work-around. If we were to remove it, the added flexibility it will give developers/users to mix and match flags between components is a nice bonus as well.