PGI Fortran: runtime error when using using mpi_f08 module on OpenPOWER

[hppritcha]

@PHHargrove has reported a runtime problem when using the PGI 16.10 fortran compiler on OpenPOWER.

This posting on devel describes the problem in some more detail:

https://mail-archive.com/devel@lists.open-mpi.org/msg19836.html

[PHHargrove]

@jjhursey
Believe it or not, I see the same failure with the XLC community edition compilers on the same system!
No issues with gcc/gfortran.

[xiachsh]

@PHHargrove

which version of IBM Compiler did you used ?

[PHHargrove]

@xiachsh Here is the full version info for xlc and xlf:

phargrov@openpower-6:~$ xlc -qversion
IBM XL C/C++ for Linux, V13.1.4 (Community Edition)
Version: 13.01.0004.0001
/opt/ibm/xlC/13.1.4/bin/.orig/xlc: note: XL C/C++ Community Edition is a no-charge product and does not include official IBM support. You can provide feedback at the XL on POWER C/C++ Community Edition forum (http://ibm.biz/xlcpp-linux-ce). For information about a fully supported XL C/C++ compiler with advanced optimization features, visit XL C/C++ for Linux (http://ibm.biz/xlcpp-linux).
phargrov@openpower-6:~$ xlf -qversion
IBM XL Fortran for Linux, V15.1.5 (Community Edition)
Version: 15.01.0005.0001
/opt/ibm/xlf/15.1.5/bin/.orig/xlf: 1501-303 (I) XL Fortran Community Edition is a no-charge product and does not include official IBM support. You can provide feedback at the XL on POWER Fortran Community Edition forum (http://ibm.biz/xlfortran-linux-ce). For information about a fully supported XL Fortran compiler, visit XL Fortran for Linux (http://ibm.biz/xlfortran-linux).

[DanielCChen]

@PHHargrove.
Are you still experiencing problems building openmpi with IBM's community edition compilers?

I tried it using the same IBM XLC and XLF compilers as you mentioned to build Openmpi 2.0.2, and it compiled and ran ring_openmpif08.f90 just fine for me.

I built Openmpi 2.0.2 on a LE Power8 machine that runs Ubuntu 14.04.1. The only difference in comparing to your config file is that I also added --enable-mpi-fortran=usempif08, which is required to build mpi_f08.mod.

Here is the run result:
mpirun -mca btl sm,self -np 2 a.out
Process 0 sending 10 to 1 tag 201 ( 2 processes in ring)
Process 0 sent to 1
Process 0 decremented value: 9
Process 0 decremented value: 8
Process 0 decremented value: 7
Process 0 decremented value: 6
Process 0 decremented value: 5
Process 0 decremented value: 4
Process 0 decremented value: 3
Process 0 decremented value: 2
Process 0 decremented value: 1
Process 0 decremented value: 0
Process 0 exiting
Process 1 exiting

[PHHargrove]

@DanielCChen

If I recall correctly, I had omitted --enable-mpi-fortan=usempif08 because configure had determined that F08 support was present even without passing that flag. Regardless, I have retested today with that flag passed explicitly.

I am using:

$ xlc -qversion 2>/dev/null
IBM XL C/C++ for Linux, V13.1.4 (Community Edition)
Version: 13.01.0004.0001

on

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.2 LTS
Release:        16.04
Codename:       xenial
$ uname -p
ppc64le

The system is a KVM-based VM on Power8E h/w.

Using Open MPI 2.0.2 I still see the error:

mpirun -mca btl sm,self -np 2 examples/ring_usempif08
[openpower-6:14143] *** An error occurred in MPI_Recv
[openpower-6:14143] *** reported by process [645332993,1]
[openpower-6:14143] *** on communicator MPI_COMM_WORLD
[openpower-6:14143] *** MPI_ERR_TYPE: invalid datatype
[openpower-6:14143] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[openpower-6:14143] ***    and potentially your MPI job)
Process 0 sending 10 to  1 tag 201 ( 2 processes in ring)
[openpower-6:14137] 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[openpower-6:14137] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

Meanwhile, ring_usempi and ring_mpifh ran just fine.

[DanielCChen]

@PHHargrove
Hmm, for me --enable-mpi-fortan=usempif08 is mandatory to build openmpi 2.0.2 in order to compile ring_usempif08.f90. Otherwise, the compiler will complain at "use mpi_f08" because the mpi_f08.mod does not exist.

My guess is that your mpi_f08.mod might be compiled using a different version of the compiler that may have different module format as your compilation didn't seem fail complaining "use mpi_f08".

Can you please try to

1. "make clean" first so that all the .mod (s) and .o (s) are deleted (and will be rebuilt by xlc and xlf)
2. The following is my config file:
   #!/bin/bash

export PREFIX=/path_to_openmpi202

./configure --prefix=$PREFIX
CC=/path_to_xlc/xlc
CXX=/path_to_xlC/xlC
FC=/path_to_xlf/xlf
--enable-mpi-fortran=usempif08

3. Then make && make install.

If you still run into the same error, can you please send me your configuration line and the command line you compile ring_usempif08.f90?

[PHHargrove]

@DanielCChen

My build scripts start with downloading the release tarball and unpacking.
So there is nothing to "make clean".

There are also no MPI installs (Open MPI or otherwise) in $PATH, $LD_LIBRARY_PATH, etc.

The full configure command:

[pathto]/configure --prefix=[...] CC=xlc CXX=xlC FC=xlf --disable-oshmem-fortran --enable-mpi-fortran=usempif08

The full "make -V" is here. This should let you determine if the .mod was compiled with the proper compiler.

There is only one xlc/xlf installation on this system.

The ring_* tests are build by running "make" in the examples directory with the freshly built mpicc/mpifort as the only MPI in $PATH:

mpifort -g ring_usempif08.f90 -o ring_usempif08
** ring   === End of Compilation 1 ===
1501-510  Compilation successful for file ring_usempif08.f90.

[DanielCChen]

@PHHargrove
I have another guess. In your log file, when .F90 files got compiled by xlf, -D (e.g. -DHAVE_CONFIG_H ) didn't not get passed to CPP as an option the same way as gfortran, so it is possible the *.F90 files didn't get preprocessed correctly. We have fixed this recently but it is not in the 15.1.5 Community Edition.
If you could send me an email or let me know your email, I can pass on some more information of how to get access to the newer XL compilers.

[DanielCChen]

@PHHargrove
Just to add, in order to pass -D* to CPP with xlf, you would need "-WF,-D*" for each -D*.

[PHHargrove]

@DanielCChen

I am aware that Open MPI incorrectly passes -DVAR=VAL to xlf90, and have reported it as an Open MPI bug in the past.

Are you saying that your results and mine differ because you are using a newer xlf?

My email is the same as my github user id plus AT lbl.gov.

[PHHargrove]

@DanielCChen

I have access to the following two non-community-edition XL compiler installations:

-bash-4.2$ xlc -qversion
IBM XL C/C++ for Linux, V13.1.5 (5725-C73, 5765-J08)
Version: 13.01.0005.0001
-bash-4.2$ xlf -qversion
IBM XL Fortran for Linux, V15.1.5 (5725-C75, 5765-J10)
Version: 15.01.0005.0001

and

-bash-4.2$ xlf -qversion 2>/dev/null
IBM XL Fortran for Linux, V16.1 (Beta 2)
Version: 16.01.0000.0000
-bash-4.2$ xlc -qversion 2>/dev/null
IBM XL C/C++ for Linux, V14.1 (Beta 2)
Version: 14.01.0000.0000

Both are running on

-bash-4.2$ lsb_release -a
LSB Version:    :core-4.1-noarch:core-4.1-ppc64le
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 7.3 (Maipo)
Release:        7.3
Codename:       Maipo

I do not see the error on that system with either pair of compilers.

[DanielCChen]

@PHHargrove

1. I see you already have access to our Beta compiler as I was going to send you an email about it.
2. -DVAR=VAL is not passed as an option to CPP with the XLF compiler you have. XLF has been using -D as the shortcut of -qdlines that specifies whether the compiler compiles fixed source form lines with a D in column1 or treats them as comments. We changed it recently, so you should get updated version via the Beta program.
3. I can now reproduce the error on ubuntu 16.04. It seems it only fails if MPI libs are dynamically linked in.