Defect: master, src/mpi/CMakeLists.txt 3rd argument of CHECK_SYMBOL_EXISTS

[jbmaggard]

Defect/Bug Report

• OpenCoarrays Version: master and 1.9.2
• Version of CMake: 3.9.2 (on MSYS) and 3.9.3 (on Linux)

Observed Behavior

Line 52 of src/mpi/CMakeLists.txt: Unless the 3rd argument of CHECK_SYMBOL_EXISTS is a new, unused variable, the function does not seem to be called (or perhaps is called, but does not produce any output or change the value of the 3rd argument).

Expected Behavior

Verify SIGKILL is defined, with -D_POSIX required for GCC, MinGW-w64 on Windows (including MSYS).

Steps to Reproduce

A small test example:

$ pacman -Ss mingw-w64-x86_64-gcc-fortran
mingw64/mingw-w64-x86_64-gcc-fortran 7.2.0-1 (mingw-w64-x86_64-toolchain) [installed]
    GNU Compiler Collection (Fortran) for MinGW-w64

$ rm -rf build
$ cat src/CMakeLists.txt
cmake_minimum_required(VERSION 3.9 FATAL_ERROR)
project(sigkill NONE)
enable_language(C)
include(CheckIncludeFile)
CHECK_INCLUDE_FILE("signal.h" HAVE_SIGNAL_H)
#add_executable(h h.c)
include(CheckSymbolExists)
CHECK_SYMBOL_EXISTS(SIGINT "signal.h" HAVE_SIGINT)
CHECK_SYMBOL_EXISTS(SIGKILL "signal.h" HAVE_SIGKILL)
CHECK_SYMBOL_EXISTS(SIGILL "signal.h" HAVE_SIGILL)
if(NOT HAVE_SIGKILL)
  message("Trying -D_POSIX")
  list( APPEND CMAKE_REQUIRED_DEFINITIONS -D_POSIX)
  CHECK_SYMBOL_EXISTS(SIGKILL "signal.h" HAVE_SIGKILL_NEW)
endif()

$ cmake -Bbuild -Hsrc -G "MSYS Makefiles"
-- The C compiler identification is GNU 7.2.0
-- Check for working C compiler: C:/msys64/mingw64/bin/gcc.exe
-- Check for working C compiler: C:/msys64/mingw64/bin/gcc.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Looking for signal.h
-- Looking for signal.h - found
-- Looking for SIGINT
-- Looking for SIGINT - found
-- Looking for SIGKILL
-- Looking for SIGKILL - not found
-- Looking for SIGILL
-- Looking for SIGILL - found
Trying -D_POSIX
-- Looking for SIGKILL
-- Looking for SIGKILL - found
-- Configuring done
-- Generating done
-- Build files have been written to: C:/msys64/home/bmaggard/temp/sigkill/build

SIGKILL is found after the Trying -D_POSIX message. Switching HAVE_SIGKILL_NEW to the previously used variable HAVE_SIGKILL results in the following output:

$ cmake -Bbuild -Hsrc -G "MSYS Makefiles"
-- The C compiler identification is GNU 7.2.0
-- Check for working C compiler: C:/msys64/mingw64/bin/gcc.exe
-- Check for working C compiler: C:/msys64/mingw64/bin/gcc.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Looking for signal.h
-- Looking for signal.h - found
-- Looking for SIGINT
-- Looking for SIGINT - found
-- Looking for SIGKILL
-- Looking for SIGKILL - not found
-- Looking for SIGILL
-- Looking for SIGILL - found
Trying -D_POSIX
-- Configuring done
-- Generating done
-- Build files have been written to: C:/msys64/home/bmaggard/temp/sigkill/build

With SIGKILL not found after the Trying -D_POSIX message.

Similar behavior with 3rd argument was reproduced on Linux (-G "Unix Makefiles") with cmake 3.9.3; for example, attempting to reuse the HAVE_SIGINT variable with one of the other calls to CHECK_SYMBOL_EXISTS.

The simple work-around for src/mpi/CMakeLists.txt would be to use a new variable for each call to CHECK_SYMBOL_EXISTS.

I didn't see anything about this cmake feature in the documentation or during a brief web search. However, I am a very much a novice with cmake.

@zbeekman
I am continuing to progress on implementation of CAF using the native Windows MSYS gfortran, the Intel-MPI runtime, and building OpenCoarrays.

Adding a couple of symlinks and the previously discussed import library to the MPI runtime installation (#435), I have implemented bash scripts for mpirun (which simply executes the Intel mpiexec.exe), mpicc, and mpifort (mpicc and mpifort are taken from MPICH 3.2, with only the path variables near the top changed) all working together in a fashion similar to MPICH on Linux.

Minor changes to (cmake command line near end of) prerequisites/install-functions/build_opencoarrays.sh, allows install.sh to make it to the first cmake command, which is where I'm at right now. I'm currently learning about MPI builds with cmake. I'll keep you updated if/when I get to the point where it is more feasible to consider adding to install.sh.

I'm considering a way that does not require use of the header files (mpi.h, mpif.h, mpio.h, and mpiof.h) from the Intel-MPI SDK. Since Intel-MPI is ABI compatible with MPICH, it may be possible to use the MPICH header files (with MPI_Aint of type long long on Windows). With ABI compatibility, it may even be possible to build and use the mpi.mod from MPICH source code, to interface with the Intel-MPI runtime.

I recall an article at intel.com discussing using code compiled for Intel-MPI but executing under MSMPI (and/or vice versa); implying that if the runtime has the necessary MPI functions for OpenCoarrays (MSMPI currently does not, but Intel-MPI does), then the MPI functions should be accessible through the same ABI (and therefore, theoretically by using the same .h files {and/or same source code for mpi.mod}) as the MPICH runtime.

[jbmaggard]

@zbeekman

Here is an update. I had some time available over the weekend, so I went forward with the implementation with Intel-MPI working the same way as MPICH on Linux (using mpirun, mpicc, and mpifort bash scripts).

Here is my current configuration for native Win64/MSYS/GCC/Intel-MPI/OpenCoarrays-1.9.2:

$ uname -a
MINGW64_NT-6.1 PE-MGR-LAPTOP 2.9.0(0.318/5/3) 2017-09-13 23:16 x86_64 Msys

$ mpirun --version
Intel(R) MPI Library for Windows* OS, Version 2018 Build 20170713

$ mpicc --version
gcc.exe (Rev1, Built by MSYS2 project) 7.2.0

$ mpifort --version
GNU Fortran (Rev1, Built by MSYS2 project) 7.2.0

$ mpicc -show
gcc -I/c/Intel-MPI/include -L/c/Intel-MPI/lib -Wl,-rpath -Wl,/c/Intel-MPI/lib -lmpi

$ mpifort -show
gfortran -I/c/Intel-MPI/include -I/c/Intel-MPI/include -L/c/Intel-MPI/lib -Wl,-rpath -Wl,/c/Intel-MPI/lib -lmpi

$ echo $PATH
/c/Intel-MPI/bin:/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/c/Windows/System32:/c/Windows

$ echo $MPI_HOME
/c/Intel-MPI

Here are my changes to 1.9.2:

1. Fixed the OpenCoarrays-1.9.2/CMakeLists.txt per Issue 455 install.sh fix for CentOS 6 #456

$ diff CMakeLists.txt ../CMakeLists.txt.orig
219c219
<     OUTPUT_STRIP_TRAILING_WHITESPACE
---
>     OUTPUT_STRIP_TRAILING_WHITES_SPACE

2. MSYS has GCC 7.2.0, so adjusted acceptable_compiler.f90.

$ diff prerequisites/acceptable_compiler.f90 ../acceptable_compiler.f90.orig
38c38
<   print *,(compiler_version() >= "GCC version 6.1.0 ") .and. (compiler_version() < "GCC version 7.9.0 ")
---
>   print *,(compiler_version() >= "GCC version 6.1.0 ") .and. (compiler_version() < "GCC version 7.0.0 ")

3. Fixed src/mpi/CMakeLists.txt per this defect report (see HAVE_SIGKILL2 variable), and also typo fixed, per 1.9.2 typo: "caorrays" instead of "coarrays" #452 . Set CMAKE_REQUIRED_INCLUDES before checking mpi.h for I_MPI_VERSION (introspection for Intel-MPI, to set -DUSE_GCC for long long typedef of MPI_Aint, and long long typedef of MPI_Offset).

$ diff src/mpi/CMakeLists.txt ../CMakeLists.txt.mpi.orig
52,54c52,53
<   CHECK_SYMBOL_EXISTS(SIGKILL "signal.h" HAVE_SIGKILL2)
<   if(HAVE_SIGKILL2)
<     set(HAVE_SIGKILL ${HAVE_SIGKILL2})
---
>   CHECK_SYMBOL_EXISTS(SIGKILL "signal.h" HAVE_SIGKILL)
>   if(HAVE_SIGKILL)
119,128c118
< #JBM Set CMAKE_REQUIRED_INCLUDES before checking for I_MPI_VERSION
< set(old_cmake_required_includes "${CMAKE_REQUIRED_INCLUDES}")
< if(CMAKE_REQUIRED_INCLUDES)
<   set(CMAKE_REQUIRED_INCLUDES ${CMAKE_REQUIRED_INCLUDES};${MPI_C_INCLUDE_PATH})
< else()
<   set(CMAKE_REQUIRED_INCLUDES ${MPI_C_INCLUDE_PATH})
< endif()
< CHECK_INCLUDE_FILES("mpi.h" HAVE_MPI_H)
< CHECK_SYMBOL_EXISTS(I_MPI_VERSION "mpi.h" HAVE_Intel_MPI)
< message("-- WIN32=" ${WIN32})
---
> CHECK_SYMBOL_EXISTS(I_MPI_VERSION mpi.h HAVE_Intel_MPI)
163,164c153,154
<   COMMAND ${CMAKE_COMMAND} -E create_symlink "./${mod_dir_tail}/opencoarrays.mod" "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_INCLUDEDIR}/opencoarrays.mod"
<   COMMENT "Creating symlink ${CMAKE_INSTALL_INCLUDEDIR}/opencoarrays.mod --> ${CMAKE_INSTALL_INCLUDEDIR}/${mod_dir_tail}/opencoarrays.mod")
---
>   COMMAND ${CMAKE_COMMAND} -E create_symlink "./${mod_dir_tail}/opencoarrays.mod" "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_INCLUDEDIR}/opencaorrays.mod"
>   COMMENT "Creating symlink ${CMAKE_INSTALL_INCLUDEDIR}/opencaorrays.mod --> ${CMAKE_INSTALL_INCLUDEDIR}/${mod_dir_tail}/opencoarrays.mod")

4. It is necessary to use -G "MSYS Makefiles" for cmake to find gcc and gfortran, and while I reviewed gfortran version detection in install script fails #305 , help build oc on FreeBSD #338 , and Defect: CMake can't find mpiexec if not in PATH #359 , and tried both -M option of install.sh and setting $MPI_HOME, cmake was stopping with error about not finding MPI. So, I added -DMPI_C_LIBRARIES, -DMPI_C_INCLUDE_PATH, -DMPI_Fortran_LIBRARIES, and -DMPI_Fortran_INCLUDE_PATH.

With a little work, these changes to build_opencoarrays.sh may be suitable for implementation in master. For example, using uname or cmake to set a variable for the -G option for all systems, and also the items with $MPI_HOME as shown.

$ diff prerequisites/install-functions/build_opencoarrays.sh  ../build_opencoarrays.sh.orig
34,35c34,35
<   info "CC=\"${CC}\" FC=\"${FC}\" $CMAKE \"${opencoarrays_src_dir}\" -G \"MSYS Makefiles\" -DCMAKE_INSTALL_PREFIX=\"${install_path}\" -DMPIEXEC=\"${MPIEXEC}\" -DMPI_C_COMPILER=\"${MPICC}\" -DMPI_C_LIBRARIES=\"${MPI_HOME}/lib/libmpi.a\" -DMPI_C_INCLUDE_PATH=\"${MPI_HOME}/include\" -DMPI_Fortran_COMPILER=\"${MPIFC}\" -DMPI_Fortran_LIBRARIES=\"${MPI_HOME}/lib/libmpi.a\" -DMPI_Fortran_INCLUDE_PATH=\"${MPI_HOME}/include\""
<   CC="${CC}" FC="${FC}" $CMAKE "${opencoarrays_src_dir}" -G "MSYS Makefiles" -DCMAKE_INSTALL_PREFIX="${install_path}" -DMPIEXEC="${MPIEXEC}" -DMPI_C_COMPILER="${MPICC}" -DMPI_C_LIBRARIES="${MPI_HOME}/lib/libmpi.a" -DMPI_C_INCLUDE_PATH="${MPI_HOME}/include" -DMPI_Fortran_COMPILER="${MPIFC}" -DMPI_Fortran_LIBRARIES="${MPI_HOME}/lib/libmpi.a" -DMPI_Fortran_INCLUDE_PATH="${MPI_HOME}/include"
---
>   info "CC=\"${CC}\" FC=\"${FC}\" $CMAKE \"${opencoarrays_src_dir}\" -DCMAKE_INSTALL_PREFIX=\"${install_path}\" -DMPIEXEC=\"${MPIEXEC}\" -DMPI_C_COMPILER=\"${MPICC}\" -DMPI_Fortran_COMPILER=\"${MPIFC}\""
>   CC="${CC}" FC="${FC}" $CMAKE "${opencoarrays_src_dir}" -DCMAKE_INSTALL_PREFIX="${install_path}" -DMPIEXEC="${MPIEXEC}" -DMPI_C_COMPILER="${MPICC}" -DMPI_Fortran_COMPILER="${MPIFC}"

Here is output from install.sh run from the OpenCoarrays-1.9.2 directory. Note the mpi.mod in the $MPI_HOME/include directory was built with ifort and not gfortran (and Intel does not seem to provide the .f90 source to build mpi.mod), so the test for MPI_Fortran_MODULE_COMPILES fails. Also note that the second attempt at finding SIGKILL works, as discussed above in item 3 src/mpi/CMakeLists.txt changes.

$ rm -rf build/ prerequisites/builds/opencoarrays/

$ ./install.sh -i /home/bmaggard/oca/OpenCoarrays-1.9.2/build
2017-10-16 06:17:34 UTC [     info] __file: /home/bmaggard/oca/OpenCoarrays-1.9.2/install.sh
2017-10-16 06:17:34 UTC [     info] __dir: /home/bmaggard/oca/OpenCoarrays-1.9.2
2017-10-16 06:17:34 UTC [     info] __base: install
2017-10-16 06:17:34 UTC [     info] __os: Linux
2017-10-16 06:17:34 UTC [     info] __usage: /home/bmaggard/oca/OpenCoarrays-1.9.2/install.sh-usage
2017-10-16 06:17:34 UTC [     info] LOG_LEVEL: 6
2017-10-16 06:17:34 UTC [     info] -b (--install-branch):
2017-10-16 06:17:34 UTC [     info] -B (--list-branches):
2017-10-16 06:17:34 UTC [     info] -c (--with-c):
2017-10-16 06:17:34 UTC [     info] -C (--with-cxx):
2017-10-16 06:17:34 UTC [     info] -d (--debug):            0
2017-10-16 06:17:34 UTC [     info] -D (--print-downloader):
2017-10-16 06:17:34 UTC [     info] -e (--verbose):          0
2017-10-16 06:17:34 UTC [     info] -f (--with-fortran):
2017-10-16 06:17:34 UTC [     info] -h (--help):             0
2017-10-16 06:17:34 UTC [     info] -i (--install-prefix):   /home/bmaggard/oca/OpenCoarrays-1.9.2/build
2017-10-16 06:17:34 UTC [     info] -I (--install-version):
2017-10-16 06:17:34 UTC [     info] -j (--num-threads):      1
2017-10-16 06:17:34 UTC [     info] -l (--list-packages):    0
2017-10-16 06:17:35 UTC [     info] -m (--with-cmake):
2017-10-16 06:17:35 UTC [     info] -M (--with-mpi):
2017-10-16 06:17:35 UTC [     info] -n (--no-color):         0
2017-10-16 06:17:35 UTC [     info] -o (--only-download):    0
2017-10-16 06:17:35 UTC [     info] -p (--package):          opencoarrays
2017-10-16 06:17:35 UTC [     info] -P (--print-path):
2017-10-16 06:17:35 UTC [     info] -u (--from-url):
2017-10-16 06:17:35 UTC [     info] -U (--print-url):
2017-10-16 06:17:35 UTC [     info] -v (--version):          0
2017-10-16 06:17:35 UTC [     info] -V (--print-version):
2017-10-16 06:17:35 UTC [     info] -y (--yes-to-all):       0
2017-10-16 06:17:35 UTC [     info] -z (--disable-bootstrap):0
2017-10-16 06:17:35 UTC [     info] install_path="/home/bmaggard/oca/OpenCoarrays-1.9.2/build"
2017-10-16 06:17:35 UTC [     info] num_threads="1"
2017-10-16 06:17:35 UTC [     info] opencoarrays_src_dir=/home/bmaggard/oca/OpenCoarrays-1.9.2
2017-10-16 06:17:35 UTC [     info] build_path="/home/bmaggard/oca/OpenCoarrays-1.9.2"/prerequisites/builds
2017-10-16 06:17:35 UTC [     info] build_script="/home/bmaggard/oca/OpenCoarrays-1.9.2"/prerequisites/build.sh
2017-10-16 06:17:38 UTC [     info] ${LD_LIBRARY_PATH} is empty. Try setting it if the compiler encounters linking problems.
2017-10-16 06:17:38 UTC [     info] Checking whether the directory /home/bmaggard/oca/OpenCoarrays-1.9.2/build exists...
2017-10-16 06:17:38 UTC [     info] no
2017-10-16 06:17:38 UTC [     info] Checking whether I can create /home/bmaggard/oca/OpenCoarrays-1.9.2/build ...
2017-10-16 06:17:38 UTC [     info] yes.

*** By default, building OpenCoarrays requires CMake 3.4.0 or later,      ***
*** MPICH 3.2, and GCC Fortran (gfortran) 6.1.0 or later.  To see         ***
*** options for forcing the use of older or alternative packages, execute ***
*** this script with the -h flag.  This script will recursively traverse  ***
*** the following dependency tree, asking permission to download, build,  ***
*** and install any packages that are required for building another       ***
*** package and are neither in your PATH nor in                           ***
*** opencoarrays/prerequisites/installations:                             ***

    opencoarrays
    ├── cmake-3.4.0
    └── mpich-3.2
        └── gcc-6.1.0
            ├── flex-2.6.0
            │   └── bison-3.0.4
            │       └── m4-1.4.17
            ├── gmp
            ├── mpc
            └── mpfr


opencoarrays will be installed in /home/bmaggard/oca/OpenCoarrays-1.9.2/build

Ready to rock and roll? (Y/n)

2017-10-16 06:18:24 UTC [     info] Invoking find_or_install mpich
install.sh: Checking whether mpich executable mpifort is in the PATH...yes.
Checking whether mpifort is in /home/bmaggard/oca/OpenCoarrays-1.9.2/prerequisites/installations/mpich/3.2...no.
install.sh: Checking whether mpifort in PATH wraps gfortran...
yes.
2017-10-16 06:18:34 UTC [     info] install.sh: Checking whether mpifort in PATH wraps gfortran version >= 6.1.0 and < 7.0.0 ...
yes.
 install.sh: Using the mpifort found in the PATH.
install.sh: Updated dependency stack (top to bottom = left to right):
( none )
install.sh: Remaining mpich dependency stack (top to bottom = left to right):
( none )
2017-10-16 06:18:35 UTC [     info] Invoking find_or_install cmake
install.sh: Checking whether cmake is in the PATH...yes.
Checking whether cmake is in /home/bmaggard/oca/OpenCoarrays-1.9.2/prerequisites/installations/cmake/3.4.0...no.
install.sh: Checking whether cmake in PATH is version < 3.4.0...
no.
install.sh: Using the cmake found in the PATH.

install.sh: Updated dependency stack (top to bottom = left to right):
( none )
install.sh: Remaining cmake dependency stack (top to bottom = left to right):
( none )
~/oca/OpenCoarrays-1.9.2/prerequisites/builds/opencoarrays/1.9.2 ~/oca/OpenCoarrays-1.9.2/prerequisites
2017-10-16 06:18:48 UTC [     info] Configuring OpenCoarrays in /home/bmaggard/oca/OpenCoarrays-1.9.2/prerequisites/builds/opencoarrays/1.9.2 with the command:
2017-10-16 06:18:48 UTC [     info] CC="gcc" FC="gfortran" cmake "/home/bmaggard/oca/OpenCoarrays-1.9.2" -G "MSYS Makefiles" -DCMAKE_INSTALL_PREFIX="/home/bmaggard/oca/OpenCoarrays-1.9.2/build" -DMPIEXEC="/c/Intel-MPI/bin/mpirun" -DMPI_C_COMPILER="mpicc" -DMPI_C_LIBRARIES="/c/Intel-MPI/lib/libmpi.a" -DMPI_C_INCLUDE_PATH="/c/Intel-MPI/include" -DMPI_Fortran_COMPILER="mpifort" -DMPI_Fortran_LIBRARIES="/c/Intel-MPI/lib/libmpi.a" -DMPI_Fortran_INCLUDE_PATH="/c/Intel-MPI/include"
-- The C compiler identification is GNU 7.2.0
-- The Fortran compiler identification is GNU 7.2.0
-- Check for working C compiler: C:/msys64/mingw64/bin/gcc.exe
-- Check for working C compiler: C:/msys64/mingw64/bin/gcc.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working Fortran compiler: C:/msys64/mingw64/bin/gfortran.exe
-- Check for working Fortran compiler: C:/msys64/mingw64/bin/gfortran.exe  -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether C:/msys64/mingw64/bin/gfortran.exe supports Fortran 90
-- Checking whether C:/msys64/mingw64/bin/gfortran.exe supports Fortran 90 -- yes
-- Building OpenCoarrays version: 1.9.2
-- Performing Test Check_Simple_Coarray_Fortran_Source_Compiles
-- Performing Test Check_Simple_Coarray_Fortran_Source_Compiles - Success
-- Found MPI_C: C:/Intel-MPI/lib/libmpi.a
-- Found MPI_Fortran: C:/Intel-MPI/lib/libmpi.a
-- MPI runtime and compile time environments appear to be consistent
-- Performing Test MPI_C_COMPILES
-- Performing Test MPI_C_COMPILES - Success
-- Performing Test MPI_Fortran_MODULE_COMPILES
-- Performing Test MPI_Fortran_MODULE_COMPILES - Failed
-- Performing Test MPI_Fortran_INCLUDE_COMPILES
-- Performing Test MPI_Fortran_INCLUDE_COMPILES - Success
CMake Warning at CMakeLists.txt:410 (message):
  It appears that MPI was built with a different Fortran compiler.  It is
  possible that this may cause unpredictable behavior.  The build will
  continue using `mpif.h` BUT please report any suspicious behavior to the
  OpenCoarrays developers.


-- Looking for alloca.h
-- Looking for alloca.h - not found
CMake Warning at src/mpi/CMakeLists.txt:33 (message):
  Could not find <alloca.h>.  Assuming functionality is provided elsewhere.


-- Looking for signal.h
-- Looking for signal.h - found
-- Looking for SIGKILL
-- Looking for SIGKILL - not found
-- Looking for SIGKILL
-- Looking for SIGKILL - found
-- Looking for include files mpi.h, mpi-ext.h
-- Looking for include files mpi.h, mpi-ext.h - not found
-- Looking for MPIX_ERR_PROC_FAILED
-- Looking for MPIX_ERR_PROC_FAILED - found
-- Looking for MPIX_ERR_REVOKED
-- Looking for MPIX_ERR_REVOKED - found
-- Looking for MPIX_Comm_failure_ack
-- Looking for MPIX_Comm_failure_ack - found
-- Looking for MPIX_Comm_failure_get_acked
-- Looking for MPIX_Comm_failure_get_acked - found
-- Looking for MPIX_Comm_shrink
-- Looking for MPIX_Comm_shrink - found
-- Looking for MPIX_Comm_agree
-- Looking for MPIX_Comm_agree - found
-- Looking for include file mpi.h
-- Looking for include file mpi.h - found
-- Looking for I_MPI_VERSION
-- Looking for I_MPI_VERSION - found
-- WIN32=1
-- Allocatable components of coarray derived types only supported in GFortran >= 7 with OpenCoarrays > 1.8.4
-- (but full support not anticipated until OpenCoarrays 2.0 release)
-- Configuring done
-- Generating done
-- Build files have been written to: C:/msys64/home/bmaggard/oca/OpenCoarrays-1.9.2/prerequisites/builds/opencoarrays/1.9.2
2017-10-16 06:19:31 UTC [     info] Building OpenCoarrays in /home/bmaggard/oca/OpenCoarrays-1.9.2/prerequisites/builds/opencoarrays/1.9.2 with the command make -j1
Scanning dependencies of target opencoarrays_mod
[  1%] Building Fortran object src/mpi/CMakeFiles/opencoarrays_mod.dir/__/extensions/opencoarrays.F90.obj
[  1%] Built target opencoarrays_mod
Scanning dependencies of target caf_mpi
[  2%] Building C object src/mpi/CMakeFiles/caf_mpi.dir/mpi_caf.c.obj
[  4%] Building C object src/mpi/CMakeFiles/caf_mpi.dir/__/common/caf_auxiliary.c.obj
[  5%] Linking Fortran shared library ../../bin/OpenCoarrays-1.9.2-tests/libcaf_mpi.dll
Creating symlink include/opencoarrays.mod --> include/OpenCoarrays-1.9.2_GNU-7.2.0/opencoarrays.mod
failed to create symbolic link 'C:/msys64/home/bmaggard/oca/OpenCoarrays-1.9.2/prerequisites/builds/opencoarrays/1.9.2/include/opencoarrays.mod': Result too large
make[2]: *** [src/mpi/CMakeFiles/caf_mpi.dir/build.make:128: bin/OpenCoarrays-1.9.2-tests/libcaf_mpi.dll] Error 1
make[2]: *** Deleting file 'bin/OpenCoarrays-1.9.2-tests/libcaf_mpi.dll'
make[1]: *** [CMakeFiles/Makefile2:176: src/mpi/CMakeFiles/caf_mpi.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
2017-10-16 06:19:34 UTC [     info] Cleaning up. Done

Next, I'll take a look at the symlink creation error, and let you know if I make any additional progress.

[zbeekman]

With SIGKILL not found after the Trying -D_POSIX message.

Similar behavior with 3rd argument was reproduced on Linux (-G "Unix Makefiles") with cmake 3.9.3; for example, attempting to reuse the HAVE_SIGINT variable with one of the other calls to CHECK_SYMBOL_EXISTS.

The simple work-around for src/mpi/CMakeLists.txt would be to use a new variable for each call to CHECK_SYMBOL_EXISTS.

I didn't see anything about this cmake feature in the documentation or during a brief web search. However, I am a very much a novice with cmake.

Ah yes, this is indeed a bug thanks to yours truly. (i.e. me!) The problem is that the CHECK_SYMBOL_EXISTS macros create a cache variable. This is the biggest source of frustration and confusion to (mostly new, but sometimes even veteran) CMake users. Once a cache variable is set, it can be shadowed by another variable of the same name, but its actual value cannot be altered (without passing the FORCE argument to set() at least. This is because they are meant to be manually adjustable by the user, although in practice the user generally just reports a build error to the developer of the software package in question. These variables persist between builds and runs of CMake because, presumably, a user pointed them to the right place if the build system didn't, and we don't want the user having to do this more than once. (Once is still a best case scenario...) So what's happening is exactly as you described: The variable does not get updated in the case that the initial call to CHECK_SYMBOL_EXISTS produces a ...not-found. The fix is to either force the update or do some other trickery. I'll try to get to this soon.

I'll keep you updated if/when I get to the point where it is more feasible to consider adding to install.sh.

👍 If we can get CMake to handle it, then maybe we won't need any major changes to install.sh... Ideally the build logic for locating extant prerequisites and building the package can all be encapsulated with CMake.

I recall an article at intel.com discussing using code compiled for Intel-MPI but executing under MSMPI (and/or vice versa); implying that if the runtime has the necessary MPI functions for OpenCoarrays (MSMPI currently does not, but Intel-MPI does), then the MPI functions should be accessible through the same ABI (and therefore, theoretically by using the same .h files {and/or same source code for mpi.mod}) as the MPICH runtime.

Very interesting, although it sounds like it may be a bit of a yack shave...

2. MSYS has GCC 7.2.0, so adjusted acceptable_compiler.f90.

The point limiting to GCC < 7 (via acceptable_compiler.f90) is that there are some regressions in GCC >=7 w/ OpenCoarrays (the API and ABI changes with GFortran). The point of this is to steer novice users towards the more stable 6.x library/compiler combo. Passing explicitly the compilers you wish to use to install.sh with the -f, -c, -C flags or their longer counterparts overrides the version check.

It is necessary to use -G "MSYS Makefiles" for cmake to find gcc and gfortran

Seems odd, but I might chalk that up to a peculiarity of MSYS/Windows

tried both -M option of install.sh and setting $MPI_HOME, cmake was stopping with error about not finding MPI. So, I added -DMPI_C_LIBRARIES, -DMPI_C_INCLUDE_PATH, -DMPI_Fortran_LIBRARIES, and -DMPI_Fortran_INCLUDE_PATH.

This concerns me. I wonder if there's a bug (on our end) somewhere.

With a little work, these changes to build_opencoarrays.sh may be suitable for implementation in master. For example, using uname or cmake to set a variable for the -G option for all systems, and also the items with $MPI_HOME as shown.

🎉 💯 🥇 Nice work! This seems like a great starting point, and like there are a number of your changes that we can incorporate. Feel free to open a Pull Request if I don't get to this soon.

[jbmaggard]

Update on the symlink creation error. The cmake documentation clearly states that -E create_symlink is "available only on UNIX". The following example shows a way to create a symlink using the Windows cmd.exe command, mklink. The example creates a symlink, build/foo -> src/bar, using the same CMakeLists.txt for both MSYS and Linux.

src/CMakeLists.txt:

$ cat src/CMakeLists.txt
cmake_minimum_required(VERSION 3.9 FATAL_ERROR)
project(symlink NONE)
enable_language(C)

message("---- Begin: CMAKE Variables")
message("---- SYSTEM=" ${CMAKE_SYSTEM} ", SYSTEM_NAME=" ${CMAKE_SYSTEM_NAME} ", SYSTEM_VERSION=" ${CMAKE_SYSTEM})
message("---- UNIX=" ${UNIX} ", WIN32=" ${WIN32} ", APPLE=" ${APPLE} ", MINGW=" ${MINGW} ", MSYS=" ${MSYS} ", CYGWIN=" ${CYGWIN})
message("---- BORLAND=" ${BORLAND} ", WATCOM=" ${WATCOM} ", MSVC=" ${MSVC})
message("---- C_COMPILER_ID=" ${CMAKE_C_COMPILER_ID} ", COMPILER_IS_GNUCC=" ${CMAKE_COMPILER_IS_GNUCC})
message("---- End: CMAKE Variables")

message("----CMAKE_CURRENT_BINARY_DIR=" ${CMAKE_CURRENT_BINARY_DIR})
message("----CMAKE_CURRENT_SOURCE_DIR=" ${CMAKE_CURRENT_SOURCE_DIR})

if(WIN32)
  if(MSYS)
    set(my_link "build\\foo")
    set(my_target "c:\\msys64\\home\\bmaggard\\temp\\symlink\\src\\bar")
    message("my_link=" ${my_link})
    message("my_target=" ${my_target})
    execute_process(
      COMMAND cmd.exe /c "mklink ${my_link} ${my_target}"
      RESULT_VARIABLE rc
      OUTPUT_VARIABLE output
      ERROR_VARIABLE error
    )
  endif()
endif()
if(UNIX)
  execute_process(
    COMMAND ${CMAKE_COMMAND} -E create_symlink ${CMAKE_CURRENT_SOURCE_DIR}/bar ${CMAKE_CURRENT_BINARY_DIR}/foo
    RESULT_VARIABLE rc
    OUTPUT_VARIABLE output
    ERROR_VARIABLE error
  )
endif()
message(STATUS "RESULT_VARIABLE ${rc}")
message(STATUS "OUTPUT_VARIABLE ${output}")
message(STATUS "ERROR_VARIABLE ${error}")

On Windows/MSYS, cmake 3.9.2:

$ cat 1_cmake.sh
#!/bin/bash
rm -rf build
echo "foobar" > src/bar
cmake -Bbuild -Hsrc -G "MSYS Makefiles"
cat build/foo

PE-MGR-LAPTOP+maggard.admin@PE-MGR-LAPTOP MINGW64 /home/bmaggard/temp/symlink
$ . 1_cmake.sh
-- The C compiler identification is GNU 7.2.0
-- Check for working C compiler: C:/msys64/mingw64/bin/gcc.exe
-- Check for working C compiler: C:/msys64/mingw64/bin/gcc.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
---- Begin: CMAKE Variables
---- SYSTEM=Windows-6.1.7601, SYSTEM_NAME=Windows, SYSTEM_VERSION=Windows-6.1.7601
---- UNIX=, WIN32=1, APPLE=, MINGW=1, MSYS=1, CYGWIN=
---- BORLAND=, WATCOM=, MSVC=
---- C_COMPILER_ID=GNU, COMPILER_IS_GNUCC=1
---- End: CMAKE Variables
----CMAKE_CURRENT_BINARY_DIR=C:/msys64/home/bmaggard/temp/symlink/build
----CMAKE_CURRENT_SOURCE_DIR=C:/msys64/home/bmaggard/temp/symlink/src
my_link=build\foo
my_target=c:\msys64\home\bmaggard\temp\symlink\src\bar
-- RESULT_VARIABLE 0
-- OUTPUT_VARIABLE symbolic link created for build\foo <<===>> c:\msys64\home\bmaggard\temp\symlink\src\bar

-- ERROR_VARIABLE
-- Configuring done
-- Generating done
-- Build files have been written to: C:/msys64/home/bmaggard/temp/symlink/build
foobar

On Linux, cmake 3.9.4:

$ cat 1_cmake.sh
#!/bin/bash
rm -rf build
echo "foobar" > src/bar
cmake -Bbuild -Hsrc
cat build/foo
$ . 1_cmake.sh
-- The C compiler identification is GNU 7.2.0
-- Check for working C compiler: /scratch/user/bmaggard/.linuxbrew/bin/cc
-- Check for working C compiler: /scratch/user/bmaggard/.linuxbrew/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
---- Begin: CMAKE Variables
---- SYSTEM=Linux-2.6.32-696.13.2.el6.x86_64, SYSTEM_NAME=Linux, SYSTEM_VERSION=Linux-2.6.32-696.13.2.el6.x86_64
---- UNIX=1, WIN32=, APPLE=, MINGW=, MSYS=, CYGWIN=
---- BORLAND=, WATCOM=, MSVC=
---- C_COMPILER_ID=GNU, COMPILER_IS_GNUCC=1
---- End: CMAKE Variables
----CMAKE_CURRENT_BINARY_DIR=/general/home/bmaggard/temp/symlink/build
----CMAKE_CURRENT_SOURCE_DIR=/general/home/bmaggard/temp/symlink/src
-- RESULT_VARIABLE 0
-- OUTPUT_VARIABLE
-- ERROR_VARIABLE
-- Configuring done
-- Generating done
-- Build files have been written to: /general/home/bmaggard/temp/symlink/build
foobar

Next, figure out how to get CMAKE_CURRENT_BINARY_DIR and CMAKE_CURRENT_SOURCE_DIR formatted using Windows backslash in path, instead of MSYS/Linux foward slash; as was done manually for the my_link and my_target variables used with mklink in the example CMakeLists.txt.

It is necessary to use -G "MSYS Makefiles" for cmake to find gcc and gfortran

Seems odd, but I might chalk that up to a peculiarity of MSYS/Windows

I'd say needing -G is more of a cmake on Windows peculiarity, where cmake defaults to the generator for visual studio.

Passing explicitly the compilers you wish to use to install.sh with the -f, -c, -C flags or their longer counterparts overrides the version check.

Thanks for the comment, as I didn't pick up on the override of the version check.

[jbmaggard]

Here is the updated CMakeLists.txt that uses STRING(REPLACE to replace "/" with "\" in my_link and my_target before using mklink to create a Windows symlink (note, must have admin for mklink).

cmake_minimum_required(VERSION 3.9 FATAL_ERROR)
project(symlink NONE)
enable_language(C)

message("---- Begin: CMAKE Variables")
message("---- SYSTEM=" ${CMAKE_SYSTEM} ", SYSTEM_NAME=" ${CMAKE_SYSTEM_NAME} ", SYSTEM_VERSION=" ${CMAKE_SYSTEM})
message("---- UNIX=" ${UNIX} ", WIN32=" ${WIN32} ", APPLE=" ${APPLE} ", MINGW=" ${MINGW} ", MSYS=" ${MSYS} ", CYGWIN=" ${CYGWIN})
message("---- BORLAND=" ${BORLAND} ", WATCOM=" ${WATCOM} ", MSVC=" ${MSVC})
message("---- C_COMPILER_ID=" ${CMAKE_C_COMPILER_ID} ", COMPILER_IS_GNUCC=" ${CMAKE_COMPILER_IS_GNUCC})
message("---- End: CMAKE Variables")

message("----CMAKE_CURRENT_BINARY_DIR=" ${CMAKE_CURRENT_BINARY_DIR})
message("----CMAKE_CURRENT_SOURCE_DIR=" ${CMAKE_CURRENT_SOURCE_DIR})

if(WIN32)
  if(MSYS)
    string(REPLACE "/" "\\" my_link "${CMAKE_CURRENT_BINARY_DIR}/foo")
    string(REPLACE "/" "\\" my_target "${CMAKE_CURRENT_SOURCE_DIR}/bar")
    message("my_link=" ${my_link})
    message("my_target=" ${my_target})
    execute_process(
      COMMAND cmd.exe /c "mklink ${my_link} ${my_target}"
      RESULT_VARIABLE rc
      OUTPUT_VARIABLE output
      ERROR_VARIABLE error
    )
  endif()
endif()
if(UNIX)
  execute_process(
    COMMAND ${CMAKE_COMMAND} -E create_symlink ${CMAKE_CURRENT_SOURCE_DIR}/bar ${CMAKE_CURRENT_BINARY_DIR}/foo
    RESULT_VARIABLE rc
    OUTPUT_VARIABLE output
    ERROR_VARIABLE error
  )
endif()
message(STATUS "RESULT_VARIABLE ${rc}")
message(STATUS "OUTPUT_VARIABLE ${output}")
message(STATUS "ERROR_VARIABLE ${error}")

With output.

$ cat 1_cmake.sh
#!/bin/bash
rm -rf build
echo "foobar" > src/bar
cmake -Bbuild -Hsrc -G "MSYS Makefiles"
cat build/foo

$ . 1_cmake.sh
-- The C compiler identification is GNU 7.2.0
-- Check for working C compiler: C:/msys64/mingw64/bin/gcc.exe
-- Check for working C compiler: C:/msys64/mingw64/bin/gcc.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
---- Begin: CMAKE Variables
---- SYSTEM=Windows-6.1.7601, SYSTEM_NAME=Windows, SYSTEM_VERSION=Windows-6.1.7601
---- UNIX=, WIN32=1, APPLE=, MINGW=1, MSYS=1, CYGWIN=
---- BORLAND=, WATCOM=, MSVC=
---- C_COMPILER_ID=GNU, COMPILER_IS_GNUCC=1
---- End: CMAKE Variables
----CMAKE_CURRENT_BINARY_DIR=C:/msys64/home/bmaggard/temp/symlink/build
----CMAKE_CURRENT_SOURCE_DIR=C:/msys64/home/bmaggard/temp/symlink/src
my_link=C:\msys64\home\bmaggard\temp\symlink\build\foo
my_target=C:\msys64\home\bmaggard\temp\symlink\src\bar
-- RESULT_VARIABLE 0
-- OUTPUT_VARIABLE symbolic link created for C:\msys64\home\bmaggard\temp\symlink\build\foo <<===>> C:\msys64\home\bmaggard\temp\symlink\src\bar

-- ERROR_VARIABLE
-- Configuring done
-- Generating done
-- Build files have been written to: C:/msys64/home/bmaggard/temp/symlink/build
foobar

The web indicates that FILE(TO_NATIVE_PATH does not replace "/" with "\" with the -G "MSYS Makefiles" generator (although it may for the default Windows generator for Visual Studio).

[zbeekman]

mklink to create a Windows symlink (note, must have admin for mklink).

We will probably will just skip creating the link on windows. It's not critical and the wrapper script points to the actual install location. Admin privs is usually a deal breaker. The whole point of the directory name spacing is to try to allow parallel installs with different versions/compilers (thanks to the extreme lack of portability of .mod files).

[jbmaggard]

In addition to the previously discussed changes I skipped the symlink creation per the suggestion of @zbeekman, by adding an if(UNIX) / endif() block to src/mpi/CMakeLists.txt.

$ diff src/mpi/CMakeLists.txt ../CMakeLists.txt.mpi.orig
52,54c52,53
<   CHECK_SYMBOL_EXISTS(SIGKILL "signal.h" HAVE_SIGKILL2)
<   if(HAVE_SIGKILL2)
<     set(HAVE_SIGKILL ${HAVE_SIGKILL2})
---
>   CHECK_SYMBOL_EXISTS(SIGKILL "signal.h" HAVE_SIGKILL)
>   if(HAVE_SIGKILL)
119,128c118
< #JBM Set CMAKE_REQUIRED_INCLUDES before checking for I_MPI_VERSION
< set(old_cmake_required_includes "${CMAKE_REQUIRED_INCLUDES}")
< if(CMAKE_REQUIRED_INCLUDES)
<   set(CMAKE_REQUIRED_INCLUDES ${CMAKE_REQUIRED_INCLUDES};${MPI_C_INCLUDE_PATH})
< else()
<   set(CMAKE_REQUIRED_INCLUDES ${MPI_C_INCLUDE_PATH})
< endif()
< CHECK_INCLUDE_FILES("mpi.h" HAVE_MPI_H)
< CHECK_SYMBOL_EXISTS(I_MPI_VERSION "mpi.h" HAVE_Intel_MPI)
< message("-- WIN32=" ${WIN32})
---
> CHECK_SYMBOL_EXISTS(I_MPI_VERSION mpi.h HAVE_Intel_MPI)
161d150
< if(UNIX)
164,167c153,154
<   COMMAND ${CMAKE_COMMAND} -E create_symlink "./${mod_dir_tail}/opencoarrays.mod" "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_INCLUDEDIR}/opencoarrays.mod"
<   COMMENT "Creating symlink ${CMAKE_INSTALL_INCLUDEDIR}/opencoarrays.mod --> ${CMAKE_INSTALL_INCLUDEDIR}/${mod_dir_tail}/opencoarrays.mod"
< )
< endif()
---
>   COMMAND ${CMAKE_COMMAND} -E create_symlink "./${mod_dir_tail}/opencoarrays.mod" "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_INCLUDEDIR}/opencaorrays.mod"
>   COMMENT "Creating symlink ${CMAKE_INSTALL_INCLUDEDIR}/opencaorrays.mod --> ${CMAKE_INSTALL_INCLUDEDIR}/${mod_dir_tail}/opencoarrays.mod")

And voila/eureka/yippee, with the changes documented above, install.sh runs cleanly from the Win7 / MSYS2 / mingw64 / bash prompt.

After adding the installation bin folder to the path, the caf and cafrun scripts work as expected, and multi-image CAF runs perform as expected.

Now for some more serious testing of this CAF implementation for native Win7 / MSYS2 / Intel-MPI runtime / OpenCoarrays-1.9.2.

Other than testing CAF results and performance, I see the next item to work on as the issue of cmake not finding MPI through the install.sh -M option and/or through setting the $MPI_HOME environment variable (currently cmake requires -DMPI_C_LIBRARIES, etc. to be set in prerequisites/install-functions/build_opencoarrays.sh). One idea would be to figure out what it is that cmake is looking for in the mpicc and mpifort scripts. However, I'm already using the same mpicc and mpifort scripts as MPICH 3.2 (with the paths near the beginning as the only changes).

I think the key was symlink of the MPI runtime to allow using mpirun/mpicc/mpifort bash scripts. However, if the need for administrator to enable user symlinks is a major problem, adding some double quotes in a few places can probably get everything to work with the default install path of Intel-MPI (under the "Program Files (x86)" folder).

@zbeekman

If you are willing to assist me with some brief instructions on the right way to perform testing, I will compare results from this build to Linux on the same system (dual boot) and document.

@rouson

In #435 you mentioned needing a Windows CAF solution for your "PDE solver project starting soon"... I'm feeling pretty optimistic about gfortran/OpenCoarrays/Intel-MPI ("free" as in beer MPI runtime) for native Windows builds, based on recent progress.

[rouson]

@jbmaggard Thanks for all of your tireless efforts on this!

[zbeekman]

@jbmaggard can you send me a patch with your changes, or submit a pull request? I can manually go through your comments, but I'm worried I'll miss something or that one file will be out of sync with another.

Thanks!

[jbmaggard]

Win7/MSYS2/mingw64/GCC-7.2.0/Intel-MPI/OpenCoarrays-1.9.2. OpenCoarrays-1.9.2 was built as described above.

ctest works correctly with a small change to OpenCoarrays-1.9.2/CMakeLists.txt. Note mingw64/ctest is a native windows application. The change was needed so that ctest would execute the cafrun bash script under the MSYS/bash shell. This change will be documented along with the others when the suggested pull request is performed.

Tests 1-46 Passed.

84% tests passed, 9 tests failed out of 55

Total Test time (real) =  65.46 sec

The following tests FAILED:
         47 - image_fail_test_1 (Failed)
         48 - image_fail_and_sync_test_1 (Failed)
         49 - image_fail_and_sync_test_2 (Failed)
         50 - image_fail_and_sync_test_3 (Failed)
         51 - image_fail_and_status_test_1 (Failed)
         52 - image_fail_and_failed_images_test_1 (Failed)
         53 - image_fail_and_stopped_images_test_1 (Failed)
         54 - image_fail_and_get_test_1 (Failed)
         55 - test-installation-scripts.sh (BAD_COMMAND)
Errors while running CTest

The image_fail... group of tests did not pass, most likely because Intel-MPI does not support needed MPI features (as @zbeekman discussed in #435). I would be interested in checking further into this, when I learn which MPI procedures and compile defines are required (the current libcaf_mpi.dll was compiled with -DUSE_FAILED_IMAGES).

The test-installation-scripts.sh test did not pass because modifications to OpenCoarrays-1.9.2/CMakeLists.txt would be required to execute this script under the bash shell.

[jbmaggard]

@zbeekman Here is the patch file you requested. You may decide that most of this can be done in a way that is benign to the Linux build. For parts that are not, the UNIX, WIN32, and MSYS cmake variables are available. I installed with -i, -c -C -f and -M options of install.sh, with $MPI_HOME environment variable set, and $MPI_HOME/bin in the path (for mpirun/mpicc/mpifort bash scripts).

I'm expect you'll be able to so something better than using $MPI_HOME when executing cmake in build_opencoarrays.sh.

As an example, using cmake option -G "Unix Makefiles" for Linux (UNIX) versus -G "MSYS Makefiles" (WIN32 and MSYS) might be considered benign.

Let me know if I can be of assistance with the concern you mentioned regarding cmake find_package(MPI) not working (which is why I added -DMPI_C_LIBRARIES, etc.).

--- ./OpenCoarrays-1.9.2/CMakeLists.txt 2017-09-22 16:24:21 -0500
+++ ./OpenCoarrays-1.9.2.MSYS/CMakeLists.txt    2017-10-20 14:56:55 -0500
@@ -216,7 +216,7 @@
     WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
     OUTPUT_VARIABLE DEFAULT_MPICH_INSTALL_LOC
     OUTPUT_QUIET
-    OUTPUT_STRIP_TRAILING_WHITES_SPACE
+    OUTPUT_STRIP_TRAILING_WHITESPACE
   )
   find_program (MY_MPI_EXEC NAMES mpirun mpiexec lamexec srun
     PATHS "${DEFAULT_MPICH_INSTALL_LOC}" ENV PATH
@@ -578,7 +578,7 @@
      endif()
    endif()
    set(test_parameters -np ${num_caf_img} ${test_parameters})
-   add_test(NAME ${name} COMMAND "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_BINDIR}/cafrun" ${test_parameters} "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${test_target}")
+   add_test(NAME ${name} COMMAND "bash" "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_BINDIR}/cafrun" ${test_parameters} "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${test_target}")
    set_property(TEST ${name} PROPERTY PASS_REGULAR_EXPRESSION "Test passed.")
 endfunction(add_caf_test)

--- ./OpenCoarrays-1.9.2/src/mpi/CMakeLists.txt 2017-09-22 16:24:21 -0500
+++ ./OpenCoarrays-1.9.2.MSYS/src/mpi/CMakeLists.txt    2017-10-18 19:55:45 -0500
@@ -49,8 +49,9 @@
 if(NOT HAVE_SIGKILL) # try -D_POSIX, needed for mingw-w64, maybe others, see #435
                      # https://github.com/sourceryinstitute/OpenCoarrays/issues/435#issuecomment-323592433
   list( APPEND CMAKE_REQUIRED_DEFINITIONS -D_POSIX)
-  CHECK_SYMBOL_EXISTS(SIGKILL "signal.h" HAVE_SIGKILL)
-  if(HAVE_SIGKILL)
+  CHECK_SYMBOL_EXISTS(SIGKILL "signal.h" HAVE_SIGKILL2)
+  if(HAVE_SIGKILL2)
+    set(HAVE_SIGKILL ${HAVE_SIGKILL2})
     add_definitions(-D_POSIX)
   endif()
 endif()
@@ -115,7 +116,16 @@
 #---------------------------------------------------
 # Windows Intel MPI compatibility, see GH issue #435
 #---------------------------------------------------
-CHECK_SYMBOL_EXISTS(I_MPI_VERSION mpi.h HAVE_Intel_MPI)
+#JBM Set CMAKE_REQUIRED_INCLUDES before checking for I_MPI_VERSION
+set(old_cmake_required_includes "${CMAKE_REQUIRED_INCLUDES}")
+if(CMAKE_REQUIRED_INCLUDES)
+  set(CMAKE_REQUIRED_INCLUDES ${CMAKE_REQUIRED_INCLUDES};${MPI_C_INCLUDE_PATH})
+else()
+  set(CMAKE_REQUIRED_INCLUDES ${MPI_C_INCLUDE_PATH})
+endif()
+CHECK_INCLUDE_FILES("mpi.h" HAVE_MPI_H)
+CHECK_SYMBOL_EXISTS(I_MPI_VERSION "mpi.h" HAVE_Intel_MPI)
+message("-- WIN32=" ${WIN32})
 if(HAVE_Intel_MPI AND WIN32)
   add_definitions(-DUSE_GCC)
 endif()
@@ -148,10 +158,13 @@
 )

 # Create a symlink in the include dir
+if(UNIX)
 add_custom_command(TARGET caf_mpi
   POST_BUILD
-  COMMAND ${CMAKE_COMMAND} -E create_symlink "./${mod_dir_tail}/opencoarrays.mod" "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_INCLUDEDIR}/opencaorrays.mod"
-  COMMENT "Creating symlink ${CMAKE_INSTALL_INCLUDEDIR}/opencaorrays.mod --> ${CMAKE_INSTALL_INCLUDEDIR}/${mod_dir_tail}/opencoarrays.mod")
+  COMMAND ${CMAKE_COMMAND} -E create_symlink "./${mod_dir_tail}/opencoarrays.mod" "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_INCLUDEDIR}/opencoarrays.mod"
+  COMMENT "Creating symlink ${CMAKE_INSTALL_INCLUDEDIR}/opencoarrays.mod --> ${CMAKE_INSTALL_INCLUDEDIR}/${mod_dir_tail}/opencoarrays.mod"
+)
+endif()

 install(DIRECTORY "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_INCLUDEDIR}/" DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}"
   FILES_MATCHING PATTERN "*.mod")
--- ./OpenCoarrays-1.9.2/prerequisites/install-functions/build_opencoarrays.sh 2017-09-22 16:24:21 -0500
+++ ./OpenCoarrays-1.9.2.MSYS/prerequisites/install-functions/build_opencoarrays.sh     2017-10-19 12:07:33 -0500
@@ -31,8 +31,8 @@
     MPIEXEC="${MPIEXEC_CANDIDATES[0]}"
   fi
   info "Configuring OpenCoarrays in ${PWD} with the command:"
-  info "CC=\"${CC}\" FC=\"${FC}\" $CMAKE \"${opencoarrays_src_dir}\" -DCMAKE_INSTALL_PREFIX=\"${install_path}\" -DMPIEXEC=\"${MPIEXEC}\" -DMPI_C_COMPILER=\"${MPICC}\" -DMPI_Fortran_COMPILER=\"${MPIFC}\""
-  CC="${CC}" FC="${FC}" $CMAKE "${opencoarrays_src_dir}" -DCMAKE_INSTALL_PREFIX="${install_path}" -DMPIEXEC="${MPIEXEC}" -DMPI_C_COMPILER="${MPICC}" -DMPI_Fortran_COMPILER="${MPIFC}"
+  info "CC=\"${CC}\" FC=\"${FC}\" $CMAKE \"${opencoarrays_src_dir}\" -G \"MSYS Makefiles\" -DCMAKE_INSTALL_PREFIX=\"${install_path}\" -DMPIEXEC=\"${MPIEXEC}\" -DMPI_C_COMPILER=\"${MPICC}\" -DMPI_C_LIBRARIES=\"${MPI_HOME}/lib/libmpi.a\" -DMPI_C_INCLUDE_PATH=\"${MPI_HOME}/include\" -DMPI_Fortran_COMPILER=\"${MPIFC}\" -DMPI_Fortran_LIBRARIES=\"${MPI_HOME}/lib/libmpi.a\" -DMPI_Fortran_INCLUDE_PATH=\"${MPI_HOME}/include\""
+  CC="${CC}" FC="${FC}" $CMAKE "${opencoarrays_src_dir}" -G "MSYS Makefiles" -DCMAKE_INSTALL_PREFIX="${install_path}" -DMPIEXEC="${MPIEXEC}" -DMPI_C_COMPILER="${MPICC}" -DMPI_C_LIBRARIES="${MPI_HOME}/lib/libmpi.a" -DMPI_C_INCLUDE_PATH="${MPI_HOME}/include" -DMPI_Fortran_COMPILER="${MPIFC}" -DMPI_Fortran_LIBRARIES="${MPI_HOME}/lib/libmpi.a" -DMPI_Fortran_INCLUDE_PATH="${MPI_HOME}/include"
   info "Building OpenCoarrays in ${PWD} with the command make -j${num_threads}"
   make "-j${num_threads}"
   if [[ ! -z ${SUDO:-} ]]; then

[jbmaggard]

Based on comments from @zbeekman, I moved ahead with an MSYS2 implementation that can work with the default installation location for Intel-MPI (which includes spaces in the path), and does not require admin permissions to implement native Win7-64 CAF using MSYS2/mingw64/GCC (7.2.0), Intel-MPI (2018.0), and OpenCoarrays-1.9.2. In addition to the previously discussed changes, double quotes were added in a few spots to allow for spaces in the default installation path for Intel-MPI (a complete set of patch files from 1.9.2 is provided below).

$MPI_HOME points to the location of Intel-MPI.

$HOME/mpi points to the bin folder for bash scripts (mpirun, mpicc, and mpifort), lib folder for the import library to impi.dll (called libmpi.a), and the include folder for the needed header files (mpi.h, mpif.h, mpio.h, and mpiof.h) from the Intel-MPI SDK.

With both $MPI_HOME/bin and $HOME/mpi/bin in $PATH, here is how I ran install.sh for a clean build.

$ ./install.sh -i "$HOME"/OpenCoarrays-1.9.2.MSYS/build -c gcc -C g++ -f gfortran -M "$HOME"/mpi

After adding the bin folder from the OpenCoarrays install to the path (and editing cafrun, see after 5. below), ctest gives the same results reported previously (with only the image_fail tests and installation-scripts test not passing).

Here are patches for the modified files:

1. OpenCoarrays-1.9.2/CMakeLists.txt

$ diff -Nu ~/OpenCoarrays-1.9.2/CMakeLists.txt ~/OpenCoarrays-1.9.2.MSYS/CMakeLists.txt
--- /home/bmaggard/OpenCoarrays-1.9.2/CMakeLists.txt    2017-09-22 16:24:21.000000000 -0500
+++ /home/bmaggard/OpenCoarrays-1.9.2.MSYS/CMakeLists.txt       2017-11-02 14:30:20.937504200 -0500
@@ -216,7 +216,7 @@
     WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
     OUTPUT_VARIABLE DEFAULT_MPICH_INSTALL_LOC
     OUTPUT_QUIET
-    OUTPUT_STRIP_TRAILING_WHITES_SPACE
+    OUTPUT_STRIP_TRAILING_WHITESPACE
   )
   find_program (MY_MPI_EXEC NAMES mpirun mpiexec lamexec srun
     PATHS "${DEFAULT_MPICH_INSTALL_LOC}" ENV PATH
@@ -578,7 +578,7 @@
      endif()
    endif()
    set(test_parameters -np ${num_caf_img} ${test_parameters})
-   add_test(NAME ${name} COMMAND "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_BINDIR}/cafrun" ${test_parameters} "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${test_target}")
+   add_test(NAME ${name} COMMAND "bash" "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_BINDIR}/cafrun" ${test_parameters} "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${test_target}")
    set_property(TEST ${name} PROPERTY PASS_REGULAR_EXPRESSION "Test passed.")
 endfunction(add_caf_test)

2. OpenCoarrays-1.9.2/src/mpi/CMakeLists.txt

$ diff -Nu ~/OpenCoarrays-1.9.2/src/mpi/CMakeLists.txt ~/OpenCoarrays-1.9.2.MSYS/src/mpi/CMakeLists.txt
--- /home/bmaggard/OpenCoarrays-1.9.2/src/mpi/CMakeLists.txt    2017-09-22 16:24:21.000000000 -0500
+++ /home/bmaggard/OpenCoarrays-1.9.2.MSYS/src/mpi/CMakeLists.txt       2017-11-02 14:30:42.606043100 -0500
@@ -49,8 +49,9 @@
 if(NOT HAVE_SIGKILL) # try -D_POSIX, needed for mingw-w64, maybe others, see #435
                      # https://github.com/sourceryinstitute/OpenCoarrays/issues/435#issuecomment-323592433
   list( APPEND CMAKE_REQUIRED_DEFINITIONS -D_POSIX)
-  CHECK_SYMBOL_EXISTS(SIGKILL "signal.h" HAVE_SIGKILL)
-  if(HAVE_SIGKILL)
+  CHECK_SYMBOL_EXISTS(SIGKILL "signal.h" HAVE_SIGKILL2)
+  if(HAVE_SIGKILL2)
+    set(HAVE_SIGKILL ${HAVE_SIGKILL2})
     add_definitions(-D_POSIX)
   endif()
 endif()
@@ -115,7 +116,16 @@
 #---------------------------------------------------
 # Windows Intel MPI compatibility, see GH issue #435
 #---------------------------------------------------
-CHECK_SYMBOL_EXISTS(I_MPI_VERSION mpi.h HAVE_Intel_MPI)
+#JBM Set CMAKE_REQUIRED_INCLUDES before checking for I_MPI_VERSION
+set(old_cmake_required_includes "${CMAKE_REQUIRED_INCLUDES}")
+if(CMAKE_REQUIRED_INCLUDES)
+  set(CMAKE_REQUIRED_INCLUDES ${CMAKE_REQUIRED_INCLUDES};${MPI_C_INCLUDE_PATH})
+else()
+  set(CMAKE_REQUIRED_INCLUDES ${MPI_C_INCLUDE_PATH})
+endif()
+CHECK_INCLUDE_FILES("mpi.h" HAVE_MPI_H)
+CHECK_SYMBOL_EXISTS(I_MPI_VERSION "mpi.h" HAVE_Intel_MPI)
+message("-- WIN32=" ${WIN32})
 if(HAVE_Intel_MPI AND WIN32)
   add_definitions(-DUSE_GCC)
 endif()
@@ -148,10 +158,13 @@
 )

 # Create a symlink in the include dir
+if(UNIX)
 add_custom_command(TARGET caf_mpi
   POST_BUILD
-  COMMAND ${CMAKE_COMMAND} -E create_symlink "./${mod_dir_tail}/opencoarrays.mod" "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_INCLUDEDIR}/opencaorrays.mod"
-  COMMENT "Creating symlink ${CMAKE_INSTALL_INCLUDEDIR}/opencaorrays.mod --> ${CMAKE_INSTALL_INCLUDEDIR}/${mod_dir_tail}/opencoarrays.mod")
+  COMMAND ${CMAKE_COMMAND} -E create_symlink "./${mod_dir_tail}/opencoarrays.mod" "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_INCLUDEDIR}/opencoarrays.mod"
+  COMMENT "Creating symlink ${CMAKE_INSTALL_INCLUDEDIR}/opencoarrays.mod --> ${CMAKE_INSTALL_INCLUDEDIR}/${mod_dir_tail}/opencoarrays.mod"
+)
+endif()

 install(DIRECTORY "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_INCLUDEDIR}/" DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}"
   FILES_MATCHING PATTERN "*.mod")

3. OpenCoarrays-1.9.2/prerequisites/install-functions/build_opencoarrays.sh

$ diff -Nu ~/OpenCoarrays-1.9.2/prerequisites/install-functions/build_opencoarrays.sh ~/OpenCoarrays-1.9.2.MSYS/prerequisites/install-functions/build_opencoarrays.sh
--- /home/bmaggard/OpenCoarrays-1.9.2/prerequisites/install-functions/build_opencoarrays.sh     2017-09-22 16:24:21.000000000 -0500
+++ /home/bmaggard/OpenCoarrays-1.9.2.MSYS/prerequisites/install-functions/build_opencoarrays.sh        2017-11-02 15:14:41.335761700 -0500
@@ -13,8 +13,8 @@
   if [[ -z ${MPIFC:-} || -z ${MPICC:-} ]]; then
     emergency "build_opencoarrays.sh: empty \${MPIFC}=${MPIFC:-} or \${MPICC}=${MPICC:-}"
   fi
-  MPIFC_show=($($MPIFC -show))
-  MPICC_show=($($MPICC -show))
+  MPIFC_show=($("$MPIFC" -show))
+  MPICC_show=($("$MPICC" -show))
   if [[ "${MPIFC_show[0]}" != *gfortran* || "${MPICC_show[0]}" != *gcc* ]]; then
     emergency "build_opencoarrays.sh: MPI doesn't wrap gfortran/gcc: \${MPIFC_show}=${MPIFC_show[*]}, \${MPICC_show}=${MPICC_show[*]}"
   fi
@@ -31,8 +31,8 @@
     MPIEXEC="${MPIEXEC_CANDIDATES[0]}"
   fi
   info "Configuring OpenCoarrays in ${PWD} with the command:"
-  info "CC=\"${CC}\" FC=\"${FC}\" $CMAKE \"${opencoarrays_src_dir}\" -DCMAKE_INSTALL_PREFIX=\"${install_path}\" -DMPIEXEC=\"${MPIEXEC}\" -DMPI_C_COMPILER=\"${MPICC}\" -DMPI_Fortran_COMPILER=\"${MPIFC}\""
-  CC="${CC}" FC="${FC}" $CMAKE "${opencoarrays_src_dir}" -DCMAKE_INSTALL_PREFIX="${install_path}" -DMPIEXEC="${MPIEXEC}" -DMPI_C_COMPILER="${MPICC}" -DMPI_Fortran_COMPILER="${MPIFC}"
+  info "CC=\"${CC}\" FC=\"${FC}\" $CMAKE \"${opencoarrays_src_dir}\" -G \"MSYS Makefiles\" -DCMAKE_INSTALL_PREFIX=\"${install_path}\" -DMPIEXEC=\"${MPIEXEC}\" -DMPI_C_COMPILER=\"${MPICC}\" -DMPI_C_LIBRARIES=\"${HOME}/mpi/lib/libmpi.a\" -DMPI_C_INCLUDE_PATH=\"${HOME}/mpi/include\" -DMPI_Fortran_COMPILER=\"${MPIFC}\" -DMPI_Fortran_LIBRARIES=\"${HOME}/mpi/lib/libmpi.a\" -DMPI_Fortran_INCLUDE_PATH=\"${HOME}/mpi/include\""
+  CC="${CC}" FC="${FC}" $CMAKE "${opencoarrays_src_dir}" -G "MSYS Makefiles" -DCMAKE_INSTALL_PREFIX="${install_path}" -DMPIEXEC="${MPIEXEC}" -DMPI_C_COMPILER="${MPICC}" -DMPI_C_LIBRARIES="${HOME}/mpi/lib/libmpi.a" -DMPI_C_INCLUDE_PATH="${HOME}/mpi/include" -DMPI_Fortran_COMPILER="${MPIFC}" -DMPI_Fortran_LIBRARIES="${HOME}/mpi/lib/libmpi.a" -DMPI_Fortran_INCLUDE_PATH="${HOME}/mpi/include"
   info "Building OpenCoarrays in ${PWD} with the command make -j${num_threads}"
   make "-j${num_threads}"
   if [[ ! -z ${SUDO:-} ]]; then

4. OpenCoarrays-1.9.2/prerequisites/install-functions/find_or_install.sh

$ diff -Nu ~/OpenCoarrays-1.9.2/prerequisites/install-functions/find_or_install.sh ~/OpenCoarrays-1.9.2.MSYS/prerequisites/install-functions/find_or_install.sh 
--- /home/bmaggard/OpenCoarrays-1.9.2/prerequisites/install-functions/find_or_install.sh        2017-09-22 16:24:21.000000000 -0500
+++ /home/bmaggard/OpenCoarrays-1.9.2.MSYS/prerequisites/install-functions/find_or_install.sh   2017-11-02 14:31:36.629189400 -0500
@@ -212,7 +212,7 @@

     # Check consistency of MPIFC, if set, and user-specified Fortran compiler
     if [[ ! -z ${MPIFC:-} && ! -z "${arg_f:-}" ]]; then
-      MPIFC_wraps=$(${MPIFC} --version)
+      MPIFC_wraps=$("${MPIFC}" --version)
       compiler=$(${arg_f} --version)
       if [[ "${MPIFC_wraps}" != "${compiler}"   ]]; then
         emergency "Specified MPI ${MPIFC_wraps} wraps a compiler other than the specified Fortran compiler ${compiler}"

5. OpenCoarrays-1.9.2/src/extensions/cafrun.in

$ diff -Nu ~/OpenCoarrays-1.9.2/src/extensions/cafrun.in ~/OpenCoarrays-1.9.2.MSYS/src/extensions/cafrun.in
--- /home/bmaggard/OpenCoarrays-1.9.2/src/extensions/cafrun.in  2017-09-22 16:24:21.000000000 -0500
+++ /home/bmaggard/OpenCoarrays-1.9.2.MSYS/src/extensions/cafrun.in     2017-11-02 14:31:06.957799200 -0500
@@ -44,7 +44,8 @@
 #

 caf_version='@CAF_VERSION@'
-CAFRUN='@MPIEXEC@'
+#CAFRUN='@MPIEXEC@'
+CAFRUN='@"$MPIEXEC"@'
 if [[ ${CAFRUN} == @*@ ]]; then
   CAFRUN=mpiexec
 fi

This last one, on cafrun.in isn't really a patch, but it shows where the problem is. CAFRUN='@MPIEXEC@' needs to behave like CAFRUN="${MPIEXEC}".

If I manually edit (after install.sh completes) both the install bin/cafrun script and the prerequisites/builds/opencoarrays/1.9.2/bin/cafrun script to say CAFRUN="$HOME"/mpi/bin/mpirun then everything works correctly, including ctest.

Any advice on fixing CAFRUN='@MPIEXEC@' to allow spaces in the path?

@zbeekman also previously mentioned concerns about having to use -DMPI_C_COMPILER, etc. in call to cmake in build_opencoarrays.sh.

Updates:

1. I found documentation on @MPIEXEC@, which is for cmake, configure_file, which I found in src/mpi/CMakeLists.txt. I'll do some reading on this and figure out how to allow mpirun (or mpiexec) in cafrun to have spaces in the path.

2. One problem remaining is that if someone has the Intel-MPI runtime (only) installed, we need the $HOME/mpi/bin, $HOME/mpi/lib, and $HOME/mpi/include directories, but if they have the full Intel-MPI installed with SDK, then we could use the folders installed under $I_MPI_ROOT (or $MPI_HOME). Do we need two variables (which might be the same), or would it be better to always assume the folders under $HOME/mpi will be used (with copies or symlinks for the .h files needed in $HOME/mpi/include).