Build failure when compiling w/ patched build system in parallel

[amckinstry]

This is seen in Debian (stretch, unstable):

make[3]: Entering directory '/«PKGBUILDDIR»/obj-arm-linux-gnueabi'
[  4%] Building Fortran object src/mpi/CMakeFiles/caf_mpi.dir/__/extensions/opencoarrays.F90.o
[  4%] Building Fortran object src/mpi/CMakeFiles/caf_mpi_static.dir/__/extensions/opencoarrays.F90.o
cd /«PKGBUILDDIR»/obj-arm-linux-gnueabi/src/mpi && /usr/bin/mpifort  -DMPI_WORKING_MODULE -DPREFIX_NAME=_gfortran_caf_ -I/usr/lib/arm-linux-gnueabi/openmpi/lib -I/usr/lib/arm-linux-gnueabi/openmpi/include -I/usr/lib/arm-linux-gnueabi/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include -I/usr/lib/arm-linux-gnueabi/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent -I/usr/lib/arm-linux-gnueabi/openmpi/include/openmpi -I/«PKGBUILDDIR»/src -I/«PKGBUILDDIR»/obj-arm-linux-gnueabi/mod  -g -O2 -fdebug-prefix-map=/«PKGBUILDDIR»=. -fstack-protector-strong -J../../mod   -c /«PKGBUILDDIR»/src/extensions/opencoarrays.F90 -o CMakeFiles/caf_mpi_static.dir/__/extensions/opencoarrays.F90.o
cd /«PKGBUILDDIR»/obj-arm-linux-gnueabi/src/mpi && /usr/bin/mpifort  -DMPI_WORKING_MODULE -DPREFIX_NAME=_gfortran_caf_ -Dcaf_mpi_EXPORTS -I/usr/lib/arm-linux-gnueabi/openmpi/lib -I/usr/lib/arm-linux-gnueabi/openmpi/include -I/usr/lib/arm-linux-gnueabi/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include -I/usr/lib/arm-linux-gnueabi/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent -I/usr/lib/arm-linux-gnueabi/openmpi/include/openmpi -I/«PKGBUILDDIR»/src -I/«PKGBUILDDIR»/obj-arm-linux-gnueabi/mod  -g -O2 -fdebug-prefix-map=/«PKGBUILDDIR»=. -fstack-protector-strong -J../../mod -fPIC   -c /«PKGBUILDDIR»/src/extensions/opencoarrays.F90 -o CMakeFiles/caf_mpi.dir/__/extensions/opencoarrays.F90.o
f951: Fatal Error: Can't rename module file '../../mod/opencoarrays.mod0' to '../../mod/opencoarrays.mod': No such file or directory
compilation terminated.
src/mpi/CMakeFiles/caf_mpi.dir/build.make:113: recipe for target 'src/mpi/CMakeFiles/caf_mpi.dir/__/extensions/opencoarrays.F90.o' failed
make[3]: *** [src/mpi/CMakeFiles/caf_mpi.dir/__/extensions/opencoarrays.F90.o] Error 1
make[3]: Leaving directory '/«PKGBUILDDIR»/obj-arm-linux-gnueabi'
CMakeFiles/Makefile2:178: recipe for target 'src/mpi/CMakeFiles/caf_mpi.dir/all' failed
make[2]: *** [src/mpi/CMakeFiles/caf_mpi.dir/all] Error 2

This appears (randomly?) when make -j2 or -j4 used, but not when -j1. A race condition?

This with openmpi-2.0.2

[zbeekman]

This appears to be a CMake bug, although I need to confirm this to ensure we're setting up the dependencies correctly. Which version of CMake are you using?

[zbeekman]

@amckinstry I suspect that your patch to src/mpi/CMakeLists.txt is to blame for this race condition. IMO, this IS still a CMake bug. CMake should be able to handle building a static and shared Fortran library in parallel that uses module files. Due to the way CMake handle's mod files, building two targets producing the same mod file appears to cause a race condition. I suspect that if we serialize the build of the shared and static libs that this problem will go away.

Knowing which version of CMake you're using would still be helpful so I can test/confirm locally and then submit a bug report to CMake.

[amckinstry]

3.7.2
It might also be triggered by my patch:
#365 (comment)
but I can't tell why.

[zbeekman]

It might also be triggered by my patch:
#365 (comment)
but I can't tell why.

Yes, it is triggered by your patch. It's because we're compiling and adding the coarrays extension module with the library... If we were to make that it's own target this would go away... I'll look into this.

[zbeekman]

Moving patch issue from #365 here because implementing the patch goes hand in hand with fixing the parallel build issue...

Index: ./src/mpi/CMakeLists.txt
===================================================================
--- ./src/mpi/CMakeLists.txt
+++ ./src/mpi/CMakeLists.txt
@@ -36,7 +36,8 @@ if (MPI_Fortran_MODULE_COMPILES)
   set(MPI_CAF_FORTRAN_FILES ../extensions/opencoarrays.F90)
 endif()

-add_library(caf_mpi mpi_caf.c ../common/caf_auxiliary.c ${MPI_CAF_FORTRAN_FILES})
+add_library(caf_mpi SHARED mpi_caf.c ../common/caf_auxiliary.c ${MPI_CAF_FORTRAN_FILES})
+add_library(caf_mpi_static STATIC  mpi_caf.c ../common/caf_auxiliary.c ${MPI_CAF_FORTRAN_FILES})
 target_link_libraries(caf_mpi PRIVATE ${MPI_C_LIBRARIES} ${MPI_Fortran_LIBRARIES})

 set_target_properties ( caf_mpi
@@ -53,9 +54,14 @@ endif()
 include_directories(${CMAKE_BINARY_DIR}/mod)

 install(TARGETS caf_mpi EXPORT OpenCoarraysTargets
+  DESTINATION "${CMAKE_INSTALL_LIBDIR}"
+  LIBRARY DESTINATION "${CMAKE_INSTALL_LIBDIR}"
+)
+install(TARGETS caf_mpi_static EXPORT OpenCoarraysTargets
   ARCHIVE DESTINATION "${CMAKE_INSTALL_LIBDIR}"
   LIBRARY DESTINATION "${CMAKE_INSTALL_LIBDIR}"
 )
+set_target_properties(caf_mpi PROPERTIES SOVERSION 1 SONAME "libcaf_mpi.so.${OpenCoarraysVersion}")

 # Install modules to standard include dir, but namespace them with compiler/version
 set (mod_install "OpenCoarrays/${CMAKE_Fortran_COMPILER_ID}/${CMAKE_Fortran_COMPILER_VERSION}")

dylib-targets.patch.txt

[adamryczkowski]

Hello. I come from completely different Fortran project. We also have hit (I believe) the same race condition-during-compilation bug. Unlike you, we don't build shared libraries, though. Error is non-deterministic and hard to reliably reproduce. I'd like to file a bug report against (CMake? gfortran?), but I can't really nail it.

I understand from this topic that you also didn't find the root cause of the bug. The bug went gone accidentally with the fix for another issue, #365. Am I correct?

[zbeekman]

@adamryczkowski

I understand from this topic that you also didn't find the root cause of the bug. The bug went gone accidentally with the fix for another issue, #365. Am I correct?

No, I know the root cause of this behavior. It is due to CMake's handling of .mod files. The "bug" arises when one or more Fortran source files containing modules is used in the compilation of multiple targets. CMake manipulates the .mod file output by the compiler, so make (or what ever "generator" you're using) might launch parallel compilation tasks for each target that has the Fortran source with the module, but then one of those jobs ends up overwriting/moving/renaming etc. the output .mod file while the other one needs it.

CMake/Kitware is aware of this behavior, and it is intentional (not a bug). In fact, under some circumstances the same problem may be encountered using make directly. The CMake way to handle this is to put the Fortran source file(s) with the module into their own library. If you don't want to explicitly build an additional library you can create an "object library" which is a sort of alias to a collection of object files. You can't install or export this "object library" target, but you CAN pass it, via a generator expression, in the sources list to add_library and add_executable:

add_library(my_obj_lib OBJECT ftn_mod1.f90 ftn_mod2.f90 ftn_mod3.f90)
add_library(lib_using_ftn_mods STATIC $<TARGET_OBJECTS:my_obj_lib> lib_src.f90)
add_executable(exe_using_ftn_mods $<TARGET_OBJECTS:my_obj_lib> a.out)
add_library(other_ftn_lib STATIC $<TARGET_OBJECTS:my_obj_lib> other_lib_src.f90)

Now CMake compiles object files (and corresponding .mod files) into the target my_obj_lib first and then is able to safely use them across multiple targets compiled in parallel.

Hope this helps!