OPAL assembly handling on multiple archs now fails on 2.1.0 #3442

amckinstry · 2017-05-03T15:29:21Z

Hi.

Openmpi 2.0.2 builds on multiple architectures within Debian:
https://buildd.debian.org/status/package.php?p=openmpi

However changes in 2.1.0 means many are now broken:
https://buildd.debian.org/status/package.php?p=openmpi&suite=experimental

(Ignore the "2.1.0rc2"; thats a Debian-specific version name ; the codebase in 2.1.0 without arch changes)
eg. Arm64 no longer compiles. mips64el compiles but not mipsel.

rhc54 · 2017-05-03T15:49:22Z

@hjelmn Looking thru the output, it appears the problem is a failure to identify the environment so the variadic macros can complete:

In file included from ../../../opal/class/opal_object.h:128:0,
                 from ../../../ompi/errhandler/errhandler.h:36,
                 from cxx_glue.h:18,
                 from intercepts.cc:29:
../../../opal/threads/thread_usage.h: In function 'int32_t opal_thread_add_32(volatile int32_t*, int32_t)':
../../../opal/threads/thread_usage.h:109:55: error: 'opal_atomic_add_32' was not declared in this scope
         return opal_atomic_add_ ## suffix (addr, delta);                \
                                                       ^
../../../opal/threads/thread_usage.h:143:1: note: in expansion of macro 'OPAL_THREAD_DEFINE_ATOMIC_ADD'
 OPAL_THREAD_DEFINE_ATOMIC_ADD(int32_t, 32)
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../../../opal/threads/thread_usage.h:28:0,
                 from ../../../opal/class/opal_object.h:128,
                 from ../../../ompi/errhandler/errhandler.h:36,
                 from cxx_glue.h:18,
                 from intercepts.cc:29:
../../../opal/threads/thread_usage.h: In function 'size_t opal_thread_add_size_t(volatile size_t*, size_t)':
../../../opal/include/opal/sys/atomic.h:484:96: error: 'opal_atomic_add_64' was not declared in this scope
 #define opal_atomic_add_size_t(addr, delta) ((size_t) opal_atomic_add_64((int64_t*) addr, delta))
                                                                                                ^
../../../opal/threads/thread_usage.h:109:16: note: in expansion of macro 'opal_atomic_add_size_t'
         return opal_atomic_add_ ## suffix (addr, delta);                \
                ^~~~~~~~~~~~~~~~
../../../opal/threads/thread_usage.h:144:1: note: in expansion of macro 'OPAL_THREAD_DEFINE_ATOMIC_ADD'
 OPAL_THREAD_DEFINE_ATOMIC_ADD(size_t, size_t)
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../../../opal/class/opal_object.h:128:0,
                 from ../../../ompi/errhandler/errhandler.h:36,
                 from cxx_glue.h:18,
                 from intercepts.cc:29:
../../../opal/threads/thread_usage.h: In function 'bool opal_thread_cmpset_bool_32(volatile int32_t*, int32_t, int32_t)':
../../../opal/threads/thread_usage.h:119:85: error: 'opal_atomic_cmpset_32' was not declared in this scope
         return opal_atomic_cmpset_ ## suffix ((volatile type *) addr, compare, value); \
                                                                                     ^
../../../opal/threads/thread_usage.h:145:1: note: in expansion of macro 'OPAL_THREAD_DEFINE_ATOMIC_CMPSET'
 OPAL_THREAD_DEFINE_ATOMIC_CMPSET(int32_t, int32_t, 32)
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../opal/threads/thread_usage.h: In function 'bool opal_thread_cmpset_bool_ptr(volatile intptr_t*, void*, void*)':
../../../opal/threads/thread_usage.h:119:85: error: 'opal_atomic_cmpset_ptr' was not declared in this scope
         return opal_atomic_cmpset_ ## suffix ((volatile type *) addr, compare, value); \
                                                                                     ^
../../../opal/threads/thread_usage.h:146:1: note: in expansion of macro 'OPAL_THREAD_DEFINE_ATOMIC_CMPSET'
 OPAL_THREAD_DEFINE_ATOMIC_CMPSET(void *, intptr_t, ptr)
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../opal/threads/thread_usage.h: In function 'int32_t opal_thread_swap_32(volatile int32_t*, int32_t)':
../../../opal/threads/thread_usage.h:134:76: error: 'opal_atomic_swap_32' was not declared in this scope
         return opal_atomic_swap_ ## suffix ((volatile type *) ptr, newvalue); \
                                                                            ^
../../../opal/threads/thread_usage.h:147:1: note: in expansion of macro 'OPAL_THREAD_DEFINE_ATOMIC_SWAP'
 OPAL_THREAD_DEFINE_ATOMIC_SWAP(int32_t, int32_t, 32)
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../opal/threads/thread_usage.h: In function 'void* opal_thread_swap_ptr(volatile intptr_t*, void*)':
../../../opal/threads/thread_usage.h:134:76: error: 'opal_atomic_swap_ptr' was not declared in this scope
         return opal_atomic_swap_ ## suffix ((volatile type *) ptr, newvalue); \
                                                                            ^
../../../opal/threads/thread_usage.h:148:1: note: in expansion of macro 'OPAL_THREAD_DEFINE_ATOMIC_SWAP'
 OPAL_THREAD_DEFINE_ATOMIC_SWAP(void *, intptr_t, ptr)
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Makefile:1867: recipe for target 'intercepts.lo' failed
make[3]: *** [intercepts.lo] Error 1
make[3]: *** Waiting for unfinished jobs....
In file included from ../../../opal/class/opal_object.h:128:0,
                 from ../../../ompi/errhandler/errhandler.h:36,
                 from cxx_glue.h:18,
                 from file.cc:19:
../../../opal/threads/thread_usage.h: In function 'int32_t opal_thread_add_32(volatile int32_t*, int32_t)':
../../../opal/threads/thread_usage.h:109:55: error: 'opal_atomic_add_32' was not declared in this scope
         return opal_atomic_add_ ## suffix (addr, delta);                \
                                                       ^
../../../opal/threads/thread_usage.h:143:1: note: in expansion of macro 'OPAL_THREAD_DEFINE_ATOMIC_ADD'
 OPAL_THREAD_DEFINE_ATOMIC_ADD(int32_t, 32)
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../../../opal/threads/thread_usage.h:28:0,
                 from ../../../opal/class/opal_object.h:128,
                 from ../../../ompi/errhandler/errhandler.h:36,
                 from cxx_glue.h:18,
                 from file.cc:19:
../../../opal/threads/thread_usage.h: In function 'size_t opal_thread_add_size_t(volatile size_t*, size_t)':
../../../opal/include/opal/sys/atomic.h:484:96: error: 'opal_atomic_add_64' was not declared in this scope
 #define opal_atomic_add_size_t(addr, delta) ((size_t) opal_atomic_add_64((int64_t*) addr, delta))
                                                                                                ^
../../../opal/threads/thread_usage.h:109:16: note: in expansion of macro 'opal_atomic_add_size_t'
         return opal_atomic_add_ ## suffix (addr, delta);                \
                ^~~~~~~~~~~~~~~~
../../../opal/threads/thread_usage.h:144:1: note: in expansion of macro 'OPAL_THREAD_DEFINE_ATOMIC_ADD'
 OPAL_THREAD_DEFINE_ATOMIC_ADD(size_t, size_t)
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../../../opal/class/opal_object.h:128:0,
                 from ../../../ompi/errhandler/errhandler.h:36,
                 from cxx_glue.h:18,
                 from file.cc:19:
../../../opal/threads/thread_usage.h: In function 'bool opal_thread_cmpset_bool_32(volatile int32_t*, int32_t, int32_t)':
../../../opal/threads/thread_usage.h:119:85: error: 'opal_atomic_cmpset_32' was not declared in this scope
         return opal_atomic_cmpset_ ## suffix ((volatile type *) addr, compare, value); \
                                                                                     ^
../../../opal/threads/thread_usage.h:145:1: note: in expansion of macro 'OPAL_THREAD_DEFINE_ATOMIC_CMPSET'
 OPAL_THREAD_DEFINE_ATOMIC_CMPSET(int32_t, int32_t, 32)
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../opal/threads/thread_usage.h: In function 'bool opal_thread_cmpset_bool_ptr(volatile intptr_t*, void*, void*)':
../../../opal/threads/thread_usage.h:119:85: error: 'opal_atomic_cmpset_ptr' was not declared in this scope
         return opal_atomic_cmpset_ ## suffix ((volatile type *) addr, compare, value); \
                                                                                     ^
../../../opal/threads/thread_usage.h:146:1: note: in expansion of macro 'OPAL_THREAD_DEFINE_ATOMIC_CMPSET'
 OPAL_THREAD_DEFINE_ATOMIC_CMPSET(void *, intptr_t, ptr)
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../opal/threads/thread_usage.h: In function 'int32_t opal_thread_swap_32(volatile int32_t*, int32_t)':
../../../opal/threads/thread_usage.h:134:76: error: 'opal_atomic_swap_32' was not declared in this scope
         return opal_atomic_swap_ ## suffix ((volatile type *) ptr, newvalue); \

bosilca · 2017-05-03T16:45:53Z

On the ARM64 the architecture was correctly detected. The configure complains about missing support for ARM64, because we ARM64.asm file and the compiler apparently does not support atomic builtins, which suggests we are expected to use opal/include/opal/sys/arm64/atomic.h. However, all the functions defined in this file seems to be missing, which suggest that somehow OPAL_GCC_INLINE_ASSEMBLY has not been correctly set.

This is indeed correct as in opal/include/opal/sys/atomic.h we force OPAL_GCC_INLINE_ASSEMBLY for C++ compilers. We could try to fix the assembly inclusion from C++ compilers, or we can drop support for the non-standardized C++ API.

amckinstry · 2017-05-04T09:36:58Z

Asn an experiment I set OPAL_GCC_INLINE_ASSEMBLY in opal/include/opal/sys/arm64/atomic.h and it builds. (Similar patch already present for powerpc)

bosilca · 2017-05-04T14:23:11Z

There is no guarantee that the C++ compiler supports the same type of assembly as the C compiler. It is happening for you, but to be on the safe side we should add a check during configure time.

amckinstry · 2017-05-04T14:29:15Z

Agreed; to be clear, Debian is only using GCC/G++ at this time (though I did test with g++ 7).

hjelmn · 2017-05-04T15:11:41Z

We decided to isolate the internals of Open MPI from C++. I don't remember whether this change made it onto master but I am sure it isn't on v2.x or v2.0.x.

hjelmn · 2017-05-04T15:12:52Z

Hmm, or I am wrong. I did PR it to v2.0.x and v2.x. :D #2163 #2162

hjelmn · 2017-05-04T15:15:00Z

This should have clued me in to the isolation code being in there:

                 from cxx_glue.h:18,

Hmm, I wonder why it isn't working as expected on aarch64.

bosilca · 2017-05-04T15:35:36Z

I don't understand how the isolation layer is supposed to work when cxx_glue.h is included from all the CXX files and it pulls the errorhandler.h (this pull the entire spaghetti header files down to our opal_object.h where we need the atomics).

jsquyres · 2017-05-04T15:48:01Z

Per fc0eeb4, should ompi_cxx_dummy_fn_t be used in cxx_glue.h instead of ompi_errhandler_generic_handler_fn_t? (assumedly with appropriate casting in cxx_glue.cc)

hjelmn · 2017-05-04T15:56:58Z

@bosilca I thought I had prevented pulling everything. It was looking ok on my Mac (it didn't compile before the glue). Looks like we just need to do a little more to make it work in this case. Trying to de the minimum necessary since @jsquyres will git rm ompi/mpi/cxx at some point.

jsquyres · 2017-05-04T15:59:28Z

Yeah, we'll remove the C++ bindings... someday. Probably not soon, though. 😈

jjhursey · 2017-05-04T16:29:09Z

@hjelmn Yeah I noticed this a couple days ago too and added a comment here to one of your PRs. I haven't had a chance to look into it any further though. A user was trying to use the C++ bindings in 2.1 and contacted me about the state of the patches.

amckinstry · 2017-05-08T09:27:45Z

Ok, 2.1.1rc1 builds fine on all archs except x32 when --without-cma is used where appropriate. Work is needed on the x32 patch (its using 64-bit quantities ? (x32 should use 32-bit quantities but the larger register set from x86_64).
I think this should not be a blocker for the 2.1.1 release.

jsquyres · 2017-05-22T19:17:52Z

@amckinstry Am I reading your comment correctly: you applied --without-cma to all the builds, but you're still seeing a failure on the 32 bit machines (i.e., s390x). Is that right?

If so, where do we find the full build output from the s390x? I see on https://buildd.debian.org/status/package.php?p=openmpi&suite=experimental the last 10-20 lines of the build that shows that there's some linker error when compiling opal_wrapper. But seeing the rest of the build log would be helpful.

hjelmn · 2017-05-23T16:31:20Z

Looking into this now.

This commit removes a nonexistent function that was causing build problems under certain environments. Reference open-mpi#3442 Signed-off-by: Nathan Hjelm <[email protected]>

hjelmn · 2017-05-26T16:42:28Z

Does #3589 help?

This commit removes a nonexistent function that was causing build problems under certain environments. Reference open-mpi#3442 Signed-off-by: Nathan Hjelm <[email protected]> (cherry picked from commit ee9093c) Signed-off-by: Nathan Hjelm <[email protected]>

jsquyres · 2017-06-01T13:19:30Z

@amckinstry Have you had a chance to check out Nathan's reply #3442 (comment)?

amckinstry · 2017-06-01T14:00:34Z

I haven't. I've just kicked off a build with the commit ee9093c which I'll push to experimental (assuming it builds on x86)

amckinstry · 2017-06-07T12:43:57Z

i've tested the commit in #3442 (log: https://buildd.debian.org/status/fetch.php?pkg=openmpi&arch=s390x&ver=2.1.1-3&stamp=1496331745&raw=0) Unfortunately s390 is still failing.

This commit removes a nonexistent function that was causing build problems under certain environments. Reference open-mpi#3442 Signed-off-by: Nathan Hjelm <[email protected]> (cherry picked from commit ee9093c) Signed-off-by: Nathan Hjelm <[email protected]>

hjelmn · 2017-06-22T20:48:17Z

We don't have atomics for s390 and it looks like the gcc builtins are not enabled in your build. Try adding --enable-builtin-atomics to your configure line.

amckinstry · 2017-09-01T11:10:42Z

--enable-builtin-atomics works fine for s390. Thanks.

jsquyres added this to the v2.1.1 milestone May 4, 2017

jsquyres added bug question labels May 4, 2017

jsquyres mentioned this issue May 4, 2017

openmpi 2.1.0 fails to build on s390x #3443

Closed

hppritcha modified the milestones: v2.1.2, v2.1.1 May 10, 2017

hjelmn mentioned this issue May 26, 2017

mpi/cxx: remove nonexistent function from cxx glue #3589

Merged

hjelmn mentioned this issue May 26, 2017

v2.1.x: mpi/cxx: remove nonexistent function from cxx glue #3590

Merged

hjelmn mentioned this issue May 26, 2017

v2.0.x: mpi/cxx: remove nonexistent function from cxx glue #3591

Merged

hjelmn mentioned this issue May 26, 2017

v3.0.x: mpi/cxx: remove nonexistent function from cxx glue #3592

Merged

amckinstry closed this as completed Sep 1, 2017

OPAL assembly handling on multiple archs now fails on 2.1.0 #3442

OPAL assembly handling on multiple archs now fails on 2.1.0 #3442

Comments

amckinstry commented May 3, 2017

rhc54 commented May 3, 2017

Uh oh!

bosilca commented May 3, 2017

Uh oh!

amckinstry commented May 4, 2017

Uh oh!

bosilca commented May 4, 2017

Uh oh!

amckinstry commented May 4, 2017

Uh oh!

hjelmn commented May 4, 2017

Uh oh!

hjelmn commented May 4, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hjelmn commented May 4, 2017

Uh oh!

bosilca commented May 4, 2017

Uh oh!

jsquyres commented May 4, 2017

Uh oh!

hjelmn commented May 4, 2017

Uh oh!

jsquyres commented May 4, 2017

Uh oh!

jjhursey commented May 4, 2017

Uh oh!

amckinstry commented May 8, 2017

Uh oh!

jsquyres commented May 22, 2017

Uh oh!

hjelmn commented May 23, 2017

Uh oh!

hjelmn commented May 26, 2017

Uh oh!

jsquyres commented Jun 1, 2017

Uh oh!

amckinstry commented Jun 1, 2017

Uh oh!

amckinstry commented Jun 7, 2017

Uh oh!

hjelmn commented Jun 22, 2017

Uh oh!

amckinstry commented Sep 1, 2017

Uh oh!

hjelmn commented May 4, 2017 •

edited

Loading