MPIR_proctable external variable not accessible in gdb in v4.0.6rc

[louisespellacy-arm]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

v4.0.6rc2

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

From openmpi-v4.0.6rc2 tar gz with GCC 8.3.0 or 10.2.0 or PGI 20.1.

Please describe the system on which you are running

• Operating system/version: Ubuntu 18.04
• Computer hardware: x86_64
• Network type: self

---

Details of the problem

Following changes made in #7757, MPIR_proctable is not accessible via gdb.

gdb --quiet --ex start --ex "set MPIR_being_debugged=1" --ex "break MPIR_Breakpoint" --ex "continue" --ex "print MPIR_proctable_size" --ex "print MPIR_proctable" --ex "output (*MPIR_proctable)@4" --args mpirun -n 4 ./a.out

// with openmpi 4.0.5
(gdb) print MPIR_proctable_size
$1 = 4
(gdb) print MPIR_proctable
$2 = (struct MPIR_PROCDESC *) 0x721ca0
(gdb) output (*MPIR_proctable)@4
{{host_name = 0x71f7c0 "mycomputer", executable_name = 0x72aa50 "/home/louspe01/test_dir/./a.out", 
    pid = 13804}, {host_name = 0x720fc0 "mycomputer", 
    executable_name = 0x72aa80 "/home/louspe01/test_dir/./a.out", pid = 13805}, {
    host_name = 0x72bf70 "mycomputer", executable_name = 0x721d10 "/home/louspe01/test_dir/./a.out", 
    pid = 13806}, {host_name = 0x72bea0 "mycomputer", 
    executable_name = 0x72b850 "/home/louspe01/test_dir/./a.out", pid = 13807}}

//with openmpi 4.0.6rc2
(gdb) print MPIR_proctable_size
$1 = 4
(gdb) print MPIR_proctable
$2 = (struct MPIR_PROCDESC *) 0x754690
(gdb) output (*MPIR_proctable)@4
0x754690

When comparing the DWARF output, the symbols for MPIR_proctable were previously found in libopen-rte.so with the following entries:

 <1><697c>: Abbrev Number: 53 (DW_TAG_variable)
    <697d>   DW_AT_name        : (indirect string, offset: 0x40ed): MPIR_proctable
    <6981>   DW_AT_decl_file   : 1
    <6982>   DW_AT_decl_line   : 162
    <6983>   DW_AT_decl_column : 23
    <6984>   DW_AT_type        : <0x6992>
    <6988>   DW_AT_external    : 1
    <6988>   DW_AT_location    : 9 byte block: 3 90 79 2b 0 0 0 0 0     (DW_OP_addr: 2b7990)

However, the symbols are now in libopen-orted-mpir.so as expected from change #7757 but they are all marked as external:

 <1><2f0e>: Abbrev Number: 24 (DW_TAG_variable)
    <2f0f>   DW_AT_name        : (indirect string, offset: 0x2791): MPIR_proctable
    <2f13>   DW_AT_decl_file   : 63
    <2f14>   DW_AT_decl_line   : 21
    <2f15>   DW_AT_decl_column : 30
    <2f16>   DW_AT_type        : <0x2f1a>
    <2f1a>   DW_AT_external    : 1
    <2f1a>   DW_AT_declaration : 1

The resulting behaviour is that the values in the MPIR_proctable are not queriable in gdb.

[awlauria]

@louisespellacy-arm can you try this change to see if it resolves this issue:

awlauria@6911750

It did for me testing locally.

[louisespellacy-arm]

@awlauria Works for me! Thanks for the quick response. Updated dwarf looks like:

 <1><6f0>: Abbrev Number: 23 (DW_TAG_variable)
    <6f1>   DW_AT_name        : (indirect string, offset: 0x6a8): MPIR_proctable
    <6f5>   DW_AT_decl_file   : 1
    <6f6>   DW_AT_decl_line   : 16
    <6f7>   DW_AT_decl_column : 23
    <6f8>   DW_AT_type        : <0x706>
    <6fc>   DW_AT_external    : 1
    <6fc>   DW_AT_location    : 9 byte block: 3 60 10 20 0 0 0 0 0      (DW_OP_addr: 201060)

[awlauria]

@louisespellacy-arm No problem. is this also needed on the v3.* branches?

[louisespellacy-arm]

IIRC the original fix was merged into v.3.1.x branch so it will need to go where the last fix was put.

[awlauria]

Thanks, opened on v3.1.x as well.

[jsquyres]

@louisespellacy-arm Austen and I have been discussing this and trying to understand both the issue and his proposed fix. I'm concerned that we don't understand why his change fixes the issue for you.

I am admittedly not a DWARF expert, so there's probably a subtlety being lost on me.

I see the dwarf change, but I don't see how it affects anything in gdb. I constructed a small case which should be similar to Open MPI:

ompi-mpir-proctable-issue.tar.gz

It has:

• foo.h which declares extern struct foo *the_foo and the foo_func() function
• foo.c which defines struct foo and instantiates struct foo *the_foo, and defines the foo_func() function
• main.c, which includes foo.h and calls foo_func()
• a Makefile with all and clean targets

If I build this, I can still print *the_foo from gdb, and I see all the members of the foo struct.

If I move the definition of struct foo to foo.h, same result.
If I remove the declaration of extern struct foo *the_foo from foo.h, same result.

I do see some changes from dwarfdump output between these changes, but the gdb behavior is the same.

So I guess I'm wondering: exactly what is going on here? What is the change that you need, and why? This is apparently a very subtle issue, and I'd like to understand and document it properly.

[awlauria]

I also see the same behavior as @jsquyres described on Power9 + rhel8.2. The dwarf output is different, but the gdb behavior remains intact. gcc version 8.3.1

[louisespellacy-arm]

Hi both - I am unclear from @jsquyres comment above - were you not able to reproduce the issue in gdb relating to MPIR_proctable?

(gdb) output (*MPIR_proctable)@4
0x754690

[awlauria]

@louisespellacy-arm we are able to reproduce in OMPI. But we are trying to understand why DW_AT_location matters, and why shuffling the declaration of the MPIR_proctable from a header file to a .c file changes it.

Also, we are unable to reproduce similar behavior in a stand-alone toy program. We're just looking for an explanation of why this works, so we can be sure the problem is actually fixed and not just masked somehow. Any input you could provide would be appreciated - for our own understanding and so we can be sure it is correctly fixed.