Patch for linking libfabric

[amckinstry]

Linking libfabric breaks on Debian/Ubuntu systems (at least) without the following patch:

Author: Gianfranco Costamagna <locutusofborg@debian.org>

--- openmpi-2.0.1.orig/ompi/mca/mtl/ofi/Makefile.am
+++ openmpi-2.0.1/ompi/mca/mtl/ofi/Makefile.am
@@ -43,7 +43,7 @@ mca_mtl_ofi_la_SOURCES = $(mtl_ofi_sourc
mca_mtl_ofi_la_LDFLAGS = \
$(ompi_mtl_ofi_LDFLAGS) \
-module -avoid-version
-mca_mtl_ofi_la_LIBADD = $(ompi_mtl_ofi_LIBS) \
+mca_mtl_ofi_la_LIBADD = $(ompi_mtl_ofi_LIBS) $(opal_common_libfabric_LIBS) \
$(OPAL_TOP_BUILDDIR)/opal/mca/common/libfabric/lib@OPAL_LIB_PREFIX@mca_common_libfabric.la

noinst_LTLIBRARIES = $(component_noinst)

[ggouaillardet]

@amckinstry thanks for the patch

@rhc54 @jsquyres that is an intersting one ...

we link libmca_common_libfabric.la with -lfabric and naively hope libfabric.so is a dependency.
as shown by ldd, that is true on Centos 7, but not on debian (i tested ubuntu 14.04.3-LTS). if i understand correctly, the reason is libmca_common_libfabric.la does not need libfabric.so at all (!)

indeed, opal/mca/common/libfabric/common_libfabric.s only contains

int mca_common_libfabric_register_mca_variables(void)
{
    return OPAL_SUCCESS;
}

a possible fix is the suggested patch.
an other one is to really make libmca_common_libfabric.la depend on libfabric.so
for example

int mca_common_libfabric_register_mca_variables(void)
{
    if (fi_version() >= FI_VERSION(1,0)) {
        return OPAL_SUCCESS;
    } else {
        return OPAL_ERROR;
   }
}

and an other one is to simple remove opal/mca/common/libfabric (it does not do much today)

any thoughts ?

[rhc54]

There was some logic behind that component, but I honestly don't recall. I'd just use your patch for now.

[jsquyres]

@ggouaillardet Good call -- yes, calling fi_version() should do the trick.

[rashikakheria]

I am able to reproduce this issue using the OMPI v4.0.x on Ubuntu 16.04. I can see that the libmpi.so is not picking libfabric as its dynamic link.

ubuntu@ip-10-0-1-177:~/aws-ofi-nccl$ ldd /home/ubuntu/mpi-install/lib/libmpi.so
	linux-vdso.so.1 =>  (0x00007ffcd41e9000)
	libopen-rte.so.40 => /home/ubuntu/mpi-install/lib/libopen-rte.so.40 (0x00007ff897ea5000)
	libopen-pal.so.40 => /home/ubuntu/mpi-install/lib/libopen-pal.so.40 (0x00007ff897b91000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff897989000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff897680000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff897463000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff897099000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff896e95000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007ff896c92000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ff898475000)

This causes basic MPI tests like ring to fail with the following error:

/home/ubuntu/ompi/examples/ring_c: symbol lookup error: /home/ubuntu/mpi-install/lib/openmpi/mca_mtl_ofi.so: undefined symbol: fi_dupinfo

Command used to configure OMPI build:

./configure --with-ofi=<absolute-path to libfbaric installation>

Any suggestions on fixing this?

[rashikakheria]

I am able to reproduce this issue using the OMPI v4.0.x on Ubuntu 16.04. I can see that the libmpi.so is not picking libfabric as its dynamic link.

ubuntu@ip-10-0-1-177:~/aws-ofi-nccl$ ldd /home/ubuntu/mpi-install/lib/libmpi.so
	linux-vdso.so.1 =>  (0x00007ffcd41e9000)
	libopen-rte.so.40 => /home/ubuntu/mpi-install/lib/libopen-rte.so.40 (0x00007ff897ea5000)
	libopen-pal.so.40 => /home/ubuntu/mpi-install/lib/libopen-pal.so.40 (0x00007ff897b91000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff897989000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff897680000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff897463000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff897099000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff896e95000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007ff896c92000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ff898475000)

This causes basic MPI tests like ring to fail with the following error:

/home/ubuntu/ompi/examples/ring_c: symbol lookup error: /home/ubuntu/mpi-install/lib/openmpi/mca_mtl_ofi.so: undefined symbol: fi_dupinfo

Command used to configure OMPI build:

./configure --with-ofi=<absolute-path to libfbaric installation>

Any suggestions on fixing this?

[ggouaillardet]

Sounds like the fix never landed the repository ! Will do tomorrow.

[jsquyres]

libmpi.so is not supposed to link against libfabric.so -- mca_mtl_ofi.so is supposed to link against libfabric.so.

Specifically, if you ldd $libdir/openmpi/mca_mtl_ofi.so, I think you'll see that it links against some libfabric.so on your system. It likely would not have linked/installed, otherwise.

I suspect that the issue you're seeing here is that you are linking against and older version of libfabric that does not have the fi_dupinfo() call. If that really is the case, we should ameliorate this in the Open MPI code somehow (i.e., have configure check libfabric for fi_dupinfo() and if it doesn't have it, either code around it or disqualify that installation of libfabric).

What version of libfabric are you linking against?

You should open a new issue for this -- this existing issue was a different problem that was already resolved / closed.

[rashikakheria]

@jsquyres I checked output for ldd mca_mtl_ofi.so and can see it is missing libfabric linking

ubuntu@ip-10-0-1-177:~/mpi-install/lib/openmpi$ ldd mca_mtl_ofi.so
	linux-vdso.so.1 =>  (0x00007ffd6fad2000)
	libmpi.so.40 => /home/ubuntu/mpi-install/lib/libmpi.so.40 (0x00007fbce6098000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fbce5e7b000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbce5ab1000)
	libopen-rte.so.40 => /home/ubuntu/mpi-install/lib/libopen-rte.so.40 (0x00007fbce57fa000)
	libopen-pal.so.40 => /home/ubuntu/mpi-install/lib/libopen-pal.so.40 (0x00007fbce54f1000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fbce52e9000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fbce4fe0000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fbce659a000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fbce4ddc000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fbce4bd9000)

I have confirmed in the past that my libfabric build does have a fi_dupinfo() call. See below:

ubuntu@ip-10-0-1-177:~$ objdump -T ofi-install/lib/libfabric.so | grep fi_dupinfo
00000000000121aa g    DF .text	00000000000004b3  FABRIC_1.2  fi_dupinfo
0000000000017b77 g    DF .text	000000000000046d (FABRIC_1.0) fi_dupinfo
0000000000018100 g    DF .text	000000000000007c (FABRIC_1.1) fi_dupinfo

Also, I am using OFI v1.7.x

I saw that this issue was still open and hence wanted to confirm if it was ever merged. I can open a new issue for this.

[jsquyres]

That is pretty weird to me -- I have no idea how you would have an mca_mtl_ofi.so that does not link against libfabric. For example, here's my build from 4.0.0:

$ ldd ~/bogus/lib/openmpi/mca_mtl_ofi.so | grep fabric
        libfabric.so.1 => /home/jsquyres/libfabric-1.6.1/install/lib/libfabric.so.1 (0x00002aaaab9bd000)

Can you open a new issue and include the stdout from your configure, your config.log, and the stdout from make? (you might need to paste those large files into a gist or something)

[ggouaillardet]

@jsquyres per my previous analysis, libfabric.so is pulled indirectly from the common lib ... expect on ubuntu.

[matcabral]

Hi @rashikakheria, would you please share the details of your setup? I have Ubuntu 16.04.5 LTS (in KVM) but cannot reproduce the issue you are seeing. I'm building from master and I have libfabric v1.7.0.

thanks,

macabral@ubuntu16:/tmp/ompi$ git log -1 |head -1
commit ea40d48

macabral@ubuntu16:/tmp/ompi$ head config.log |grep -e "./configure"
$ ./configure --with-ofi --prefix=/tmp/ompi-master-git

macabral@ubuntu16:/tmp$ ldd /tmp/ompi-master-git/lib/openmpi/mca_mtl_ofi.so
linux-vdso.so.1 => (0x00007ffd0c3ba000)
libmpi.so.0 => /tmp/ompi-master-git/lib/libmpi.so.0 (0x00007f446bc19000)
libopen-rte.so.0 => /tmp/ompi-master-git/lib/libopen-rte.so.0 (0x00007f446b960000)
libopen-pal.so.0 => /tmp/ompi-master-git/lib/libopen-pal.so.0 (0x00007f446b64d000)
libfabric.so.1 => /usr/local/lib/libfabric.so.1 (0x00007f446b3b8000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f446b19b000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f446add1000)
/lib64/ld-linux-x86-64.so.2 (0x00007f446c141000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f446abc9000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f446a8c0000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f446a6bc000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f446a4b9000)

macabral@ubuntu16:/tmp/mpi_helloworld$ mpirun -np 2 -mca pml cm -mca mtl ofi -mca mtl_ofi_provider_include sockets ./mpi_helloworld
Hello world from host ubuntu16 processor ubuntu16, rank 0 out of 2 processors
Hello world from host ubuntu16 processor ubuntu16, rank 1 out of 2 processors