Skip to content

Add openmp support to System z #66081

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Nov 3, 2023
Merged

Add openmp support to System z #66081

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fc32ba5
Add openmp support to System z
nealef Sep 12, 2023
3d78e8d
Fix typo
uweigand Nov 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions openmp/README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ Options for all Libraries
Options for ``libomp``
----------------------

**LIBOMP_ARCH** = ``aarch64|arm|i386|loongarch64|mic|mips|mips64|ppc64|ppc64le|x86_64|riscv64``
**LIBOMP_ARCH** = ``aarch64|arm|i386|loongarch64|mic|mips|mips64|ppc64|ppc64le|x86_64|riscv64|s390x``
The default value for this option is chosen based on probing the compiler for
architecture macros (e.g., is ``__x86_64__`` predefined by compiler?).

Expand Down Expand Up @@ -198,7 +198,7 @@ Optional Features
**LIBOMP_OMPT_SUPPORT** = ``ON|OFF``
Include support for the OpenMP Tools Interface (OMPT).
This option is supported and ``ON`` by default for x86, x86_64, AArch64,
PPC64, RISCV64 and LoongArch64 on Linux* and macOS*.
PPC64, RISCV64, LoongArch64, and s390x on Linux* and macOS*.
This option is ``OFF`` if this feature is not supported for the platform.

**LIBOMP_OMPT_OPTIONAL** = ``ON|OFF``
Expand Down
1 change: 1 addition & 0 deletions openmp/libomptarget/plugins-nextgen/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ add_subdirectory(cuda)
add_subdirectory(ppc64)
add_subdirectory(ppc64le)
add_subdirectory(x86_64)
add_subdirectory(s390x)

# Make sure the parent scope can see the plugins that will be created.
set(LIBOMPTARGET_SYSTEM_TARGETS "${LIBOMPTARGET_SYSTEM_TARGETS}" PARENT_SCOPE)
Expand Down
17 changes: 17 additions & 0 deletions openmp/libomptarget/plugins-nextgen/s390x/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
##===----------------------------------------------------------------------===##
#
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#
##===----------------------------------------------------------------------===##
#
# Build a plugin for a s390x machine if available.
#
##===----------------------------------------------------------------------===##

if(CMAKE_SYSTEM_NAME MATCHES "Linux")
build_generic_elf64("SystemZ" "S390X" "s390x" "s390x-ibm-linux-gnu" "22")
Copy link
Collaborator

@tstellar tstellar Feb 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to build this on Fedora and CMAKE_SYSTEM_PROCESSOR is set to s390x and not SystemZ, so the plugin fails to build. What operating system were you testing this on?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alma Linux 8 I believe.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also see

-- LIBOMPTARGET: Not building S390X NextGen offloading plugin: machine not found in the system.

@nealef can you see what we need to do to fix this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does CMAKE_SYSTEM_NAME get set to anything but Linux for a z build?

if(CMAKE_SYSTEM_NAME MATCHES "Linux")
 build_generic_elf64("SystemZ" "S390X" "s390x" "s390x-ibm-linux-gnu" "22")
else()
 libomptarget_say("Not building s390x NextGen offloading plugin: machine not found in the system.")
endif()

That's the only way we can get this message.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it's a different message (note the "S390X" instead of "s390x" in the message text). This message seems to originate from the implementation of build_generic_elf64:

macro(build_generic_elf64 tmachine tmachine_name tmachine_libname tmachine_triple elf_machine_id)
if(CMAKE_SYSTEM_PROCESSOR MATCHES "${tmachine}$")
[...]
else()
  libomptarget_say("Not building ${tmachine_name} NextGen offloading plugin: machine not found in the system.")
endif()

This is testing CMAKE_SYSTEM_PROCESSOR against SystemZ, but the macro is set to s390x as Tom said.

If I change the build_generic_elf64 invocation to use s390x, an attempt to build the plugin is made, but it fails:

-- LIBOMPTARGET: Building s390x plugin linked with libffi
[...]
/home/uweigand/llvm/llvm-head/openmp/libomptarget/plugins-nextgen/generic-elf-64bit/src/rtl.cpp:409:20: error: no member named 's390x' in 'llvm::Triple'
  409 |     return Triple::LIBOMPTARGET_NEXTGEN_GENERIC_PLUGIN_TRIPLE;
      |            ~~~~~~~~^
<command line>:2:52: note: expanded from macro 'LIBOMPTARGET_NEXTGEN_GENERIC_PLUGIN_TRIPLE'
    2 | #define LIBOMPTARGET_NEXTGEN_GENERIC_PLUGIN_TRIPLE s390x
      |                                                    ^

So there appears to be some mismatch here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turns out actually enabling this uncovers a whole lot of additional problems. I now seem to have it all working; PRs submitted as:
#83978
#83976
#83975

else()
libomptarget_say("Not building s390x NextGen offloading plugin: machine not found in the system.")
endif()
9 changes: 7 additions & 2 deletions openmp/runtime/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ if(${OPENMP_STANDALONE_BUILD})
# If adding a new architecture, take a look at cmake/LibompGetArchitecture.cmake
libomp_get_architecture(LIBOMP_DETECTED_ARCH)
set(LIBOMP_ARCH ${LIBOMP_DETECTED_ARCH} CACHE STRING
"The architecture to build for (x86_64/i386/arm/ppc64/ppc64le/aarch64/mic/mips/mips64/riscv64/loongarch64/ve).")
"The architecture to build for (x86_64/i386/arm/ppc64/ppc64le/aarch64/mic/mips/mips64/riscv64/loongarch64/ve/s390x).")
# Should assertions be enabled? They are on by default.
set(LIBOMP_ENABLE_ASSERTIONS TRUE CACHE BOOL
"enable assertions?")
Expand Down Expand Up @@ -65,6 +65,8 @@ else() # Part of LLVM build
set(LIBOMP_ARCH loongarch64)
elseif(LIBOMP_NATIVE_ARCH MATCHES "ve")
set(LIBOMP_ARCH ve)
elseif(LIBOMP_NATIVE_ARCH MATCHES "s390x")
set(LIBOMP_ARCH s390x)
else()
# last ditch effort
libomp_get_architecture(LIBOMP_ARCH)
Expand All @@ -85,7 +87,7 @@ if(LIBOMP_ARCH STREQUAL "aarch64")
endif()
endif()

libomp_check_variable(LIBOMP_ARCH 32e x86_64 32 i386 arm ppc64 ppc64le aarch64 aarch64_a64fx mic mips mips64 riscv64 loongarch64 ve)
libomp_check_variable(LIBOMP_ARCH 32e x86_64 32 i386 arm ppc64 ppc64le aarch64 aarch64_a64fx mic mips mips64 riscv64 loongarch64 ve s390x)

set(LIBOMP_LIB_TYPE normal CACHE STRING
"Performance,Profiling,Stubs library (normal/profile/stubs)")
Expand Down Expand Up @@ -165,6 +167,7 @@ set(MIPS FALSE)
set(RISCV64 FALSE)
set(LOONGARCH64 FALSE)
set(VE FALSE)
set(S390X FALSE)
if("${LIBOMP_ARCH}" STREQUAL "i386" OR "${LIBOMP_ARCH}" STREQUAL "32") # IA-32 architecture
set(IA32 TRUE)
elseif("${LIBOMP_ARCH}" STREQUAL "x86_64" OR "${LIBOMP_ARCH}" STREQUAL "32e") # Intel(R) 64 architecture
Expand Down Expand Up @@ -193,6 +196,8 @@ elseif("${LIBOMP_ARCH}" STREQUAL "loongarch64") # LoongArch64 architecture
set(LOONGARCH64 TRUE)
elseif("${LIBOMP_ARCH}" STREQUAL "ve") # VE architecture
set(VE TRUE)
elseif("${LIBOMP_ARCH}" STREQUAL "s390x") # S390x (Z) architecture
set(S390X TRUE)
endif()

# Set some flags based on build_type
Expand Down
2 changes: 2 additions & 0 deletions openmp/runtime/cmake/LibompGetArchitecture.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ function(libomp_get_architecture return_arch)
#error ARCHITECTURE=loongarch64
#elif defined(__ve__)
#error ARCHITECTURE=ve
#elif defined(__s390x__)
#error ARCHITECTURE=s390x
#else
#error ARCHITECTURE=UnknownArchitecture
#endif
Expand Down
3 changes: 3 additions & 0 deletions openmp/runtime/cmake/LibompMicroTests.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,9 @@ else()
elseif(${LOONGARCH64})
libomp_append(libomp_expected_library_deps libc.so.6)
libomp_append(libomp_expected_library_deps ld.so.1)
elseif(${S390X})
libomp_append(libomp_expected_library_deps libc.so.6)
libomp_append(libomp_expected_library_deps ld.so.1)
endif()
libomp_append(libomp_expected_library_deps libpthread.so.0 IF_FALSE STUBS_LIBRARY)
libomp_append(libomp_expected_library_deps libhwloc.so.5 LIBOMP_USE_HWLOC)
Expand Down
2 changes: 2 additions & 0 deletions openmp/runtime/cmake/LibompUtils.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,8 @@ function(libomp_get_legal_arch return_arch_string)
set(${return_arch_string} "LOONGARCH64" PARENT_SCOPE)
elseif(${VE})
set(${return_arch_string} "VE" PARENT_SCOPE)
elseif(${S390X})
set(${return_arch_string} "S390X" PARENT_SCOPE)
else()
set(${return_arch_string} "${LIBOMP_ARCH}" PARENT_SCOPE)
libomp_warning_say("libomp_get_legal_arch(): Warning: Unknown architecture: Using ${LIBOMP_ARCH}")
Expand Down
3 changes: 2 additions & 1 deletion openmp/runtime/cmake/config-ix.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,8 @@ else()
(LIBOMP_ARCH STREQUAL ppc64le) OR
(LIBOMP_ARCH STREQUAL ppc64) OR
(LIBOMP_ARCH STREQUAL riscv64) OR
(LIBOMP_ARCH STREQUAL loongarch64))
(LIBOMP_ARCH STREQUAL loongarch64) OR
(LIBOMP_ARCH STREQUAL s390x))
AND # OS supported?
((WIN32 AND LIBOMP_HAVE_PSAPI) OR APPLE OR (NOT WIN32 AND LIBOMP_HAVE_WEAK_ATTRIBUTE)))
set(LIBOMP_HAVE_OMPT_SUPPORT TRUE)
Expand Down
33 changes: 33 additions & 0 deletions openmp/runtime/src/kmp_affinity.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2990,6 +2990,9 @@ static bool __kmp_affinity_create_cpuinfo_map(int *line,

unsigned num_avail = 0;
*line = 0;
#if KMP_ARCH_S390X
bool reading_s390x_sys_info = true;
#endif
while (!feof(f)) {
// Create an inner scoping level, so that all the goto targets at the end of
// the loop appear in an outer scoping level. This avoids warnings about
Expand Down Expand Up @@ -3035,8 +3038,21 @@ static bool __kmp_affinity_create_cpuinfo_map(int *line,
if (*buf == '\n' && *line == 2)
continue;
#endif
#if KMP_ARCH_S390X
// s390x /proc/cpuinfo starts with a variable number of lines containing
// the overall system information. Skip them.
if (reading_s390x_sys_info) {
if (*buf == '\n')
reading_s390x_sys_info = false;
continue;
}
#endif

#if KMP_ARCH_S390X
char s1[] = "cpu number";
#else
char s1[] = "processor";
#endif
if (strncmp(buf, s1, sizeof(s1) - 1) == 0) {
CHECK_LINE;
char *p = strchr(buf + sizeof(s1) - 1, ':');
Expand All @@ -3062,6 +3078,23 @@ static bool __kmp_affinity_create_cpuinfo_map(int *line,
threadInfo[num_avail][osIdIndex]);
__kmp_read_from_file(path, "%u", &threadInfo[num_avail][pkgIdIndex]);

#if KMP_ARCH_S390X
// Disambiguate physical_package_id.
unsigned book_id;
KMP_SNPRINTF(path, sizeof(path),
"/sys/devices/system/cpu/cpu%u/topology/book_id",
threadInfo[num_avail][osIdIndex]);
__kmp_read_from_file(path, "%u", &book_id);
threadInfo[num_avail][pkgIdIndex] |= (book_id << 8);

unsigned drawer_id;
KMP_SNPRINTF(path, sizeof(path),
"/sys/devices/system/cpu/cpu%u/topology/drawer_id",
threadInfo[num_avail][osIdIndex]);
__kmp_read_from_file(path, "%u", &drawer_id);
threadInfo[num_avail][pkgIdIndex] |= (drawer_id << 16);
#endif

KMP_SNPRINTF(path, sizeof(path),
"/sys/devices/system/cpu/cpu%u/topology/core_id",
threadInfo[num_avail][osIdIndex]);
Expand Down
11 changes: 11 additions & 0 deletions openmp/runtime/src/kmp_affinity.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,17 @@ class KMPHwlocAffinity : public KMPAffinity {
#elif __NR_sched_getaffinity != 204
#error Wrong code for getaffinity system call.
#endif /* __NR_sched_getaffinity */
#elif KMP_ARCH_S390X
#ifndef __NR_sched_setaffinity
#define __NR_sched_setaffinity 239
#elif __NR_sched_setaffinity != 239
#error Wrong code for setaffinity system call.
#endif /* __NR_sched_setaffinity */
#ifndef __NR_sched_getaffinity
#define __NR_sched_getaffinity 240
#elif __NR_sched_getaffinity != 240
#error Wrong code for getaffinity system call.
#endif /* __NR_sched_getaffinity */
#else
#error Unknown or unsupported architecture
#endif /* KMP_ARCH_* */
Expand Down
2 changes: 2 additions & 0 deletions openmp/runtime/src/kmp_config.h.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,8 @@
# define CACHE_LINE 128
#elif KMP_ARCH_AARCH64_A64FX
# define CACHE_LINE 256
#elif KMP_ARCH_S390X
# define CACHE_LINE 256
#else
# define CACHE_LINE 64
#endif
Expand Down
6 changes: 4 additions & 2 deletions openmp/runtime/src/kmp_os.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,8 @@ typedef unsigned long long kmp_uint64;
#if KMP_ARCH_X86 || KMP_ARCH_ARM || KMP_ARCH_MIPS
#define KMP_SIZE_T_SPEC KMP_UINT32_SPEC
#elif KMP_ARCH_X86_64 || KMP_ARCH_PPC64 || KMP_ARCH_AARCH64 || \
KMP_ARCH_MIPS64 || KMP_ARCH_RISCV64 || KMP_ARCH_LOONGARCH64 || KMP_ARCH_VE
KMP_ARCH_MIPS64 || KMP_ARCH_RISCV64 || KMP_ARCH_LOONGARCH64 || \
KMP_ARCH_VE || KMP_ARCH_S390X
#define KMP_SIZE_T_SPEC KMP_UINT64_SPEC
#else
#error "Can't determine size_t printf format specifier."
Expand Down Expand Up @@ -1043,7 +1044,8 @@ extern kmp_real64 __kmp_xchg_real64(volatile kmp_real64 *p, kmp_real64 v);
#endif /* KMP_OS_WINDOWS */

#if KMP_ARCH_PPC64 || KMP_ARCH_ARM || KMP_ARCH_AARCH64 || KMP_ARCH_MIPS || \
KMP_ARCH_MIPS64 || KMP_ARCH_RISCV64 || KMP_ARCH_LOONGARCH64 || KMP_ARCH_VE
KMP_ARCH_MIPS64 || KMP_ARCH_RISCV64 || KMP_ARCH_LOONGARCH64 || \
KMP_ARCH_VE || KMP_ARCH_S390X
#if KMP_OS_WINDOWS
#undef KMP_MB
#define KMP_MB() std::atomic_thread_fence(std::memory_order_seq_cst)
Expand Down
7 changes: 6 additions & 1 deletion openmp/runtime/src/kmp_platform.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@
#define KMP_ARCH_RISCV64 0
#define KMP_ARCH_LOONGARCH64 0
#define KMP_ARCH_VE 0
#define KMP_ARCH_S390X 0

#if KMP_OS_WINDOWS
#if defined(_M_AMD64) || defined(__x86_64)
Expand Down Expand Up @@ -146,6 +147,9 @@
#elif defined __ve__
#undef KMP_ARCH_VE
#define KMP_ARCH_VE 1
#elif defined __s390x__
#undef KMP_ARCH_S390X
#define KMP_ARCH_S390X 1
#endif
#endif

Expand Down Expand Up @@ -210,7 +214,8 @@
// TODO: Fixme - This is clever, but really fugly
#if (1 != KMP_ARCH_X86 + KMP_ARCH_X86_64 + KMP_ARCH_ARM + KMP_ARCH_PPC64 + \
KMP_ARCH_AARCH64 + KMP_ARCH_MIPS + KMP_ARCH_MIPS64 + \
KMP_ARCH_RISCV64 + KMP_ARCH_LOONGARCH64 + KMP_ARCH_VE)
KMP_ARCH_RISCV64 + KMP_ARCH_LOONGARCH64 + KMP_ARCH_VE + \
KMP_ARCH_S390X)
#error Unknown or unsupported architecture
#endif

Expand Down
3 changes: 2 additions & 1 deletion openmp/runtime/src/kmp_runtime.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8894,7 +8894,8 @@ __kmp_determine_reduction_method(
int atomic_available = FAST_REDUCTION_ATOMIC_METHOD_GENERATED;

#if KMP_ARCH_X86_64 || KMP_ARCH_PPC64 || KMP_ARCH_AARCH64 || \
KMP_ARCH_MIPS64 || KMP_ARCH_RISCV64 || KMP_ARCH_LOONGARCH64 || KMP_ARCH_VE
KMP_ARCH_MIPS64 || KMP_ARCH_RISCV64 || KMP_ARCH_LOONGARCH64 || \
KMP_ARCH_VE || KMP_ARCH_S390X

#if KMP_OS_LINUX || KMP_OS_DRAGONFLY || KMP_OS_FREEBSD || KMP_OS_NETBSD || \
KMP_OS_OPENBSD || KMP_OS_WINDOWS || KMP_OS_DARWIN || KMP_OS_HURD
Expand Down
10 changes: 7 additions & 3 deletions openmp/runtime/src/kmp_tasking.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1554,7 +1554,7 @@ kmp_task_t *__kmp_task_alloc(ident_t *loc_ref, kmp_int32 gtid,
task = KMP_TASKDATA_TO_TASK(taskdata);

// Make sure task & taskdata are aligned appropriately
#if KMP_ARCH_X86 || KMP_ARCH_PPC64 || !KMP_HAVE_QUAD
#if KMP_ARCH_X86 || KMP_ARCH_PPC64 || KMP_ARCH_S390X || !KMP_HAVE_QUAD
KMP_DEBUG_ASSERT((((kmp_uintptr_t)taskdata) & (sizeof(double) - 1)) == 0);
KMP_DEBUG_ASSERT((((kmp_uintptr_t)task) & (sizeof(double) - 1)) == 0);
#else
Expand Down Expand Up @@ -1737,8 +1737,12 @@ __kmpc_omp_reg_task_with_affinity(ident_t *loc_ref, kmp_int32 gtid,
// gtid: global thread ID of caller
// task: the task to invoke
// current_task: the task to resume after task invocation
static void __kmp_invoke_task(kmp_int32 gtid, kmp_task_t *task,
kmp_taskdata_t *current_task) {
#ifdef __s390x__
__attribute__((target("backchain")))
#endif
static void
__kmp_invoke_task(kmp_int32 gtid, kmp_task_t *task,
kmp_taskdata_t *current_task) {
kmp_taskdata_t *taskdata = KMP_TASK_TO_TASKDATA(task);
kmp_info_t *thread;
int discard = 0 /* false */;
Expand Down
6 changes: 6 additions & 0 deletions openmp/runtime/src/thirdparty/ittnotify/ittnotify_config.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,10 @@
#define ITT_ARCH_VE 8
#endif /* ITT_ARCH_VE */

#ifndef ITT_ARCH_S390X
#define ITT_ARCH_S390X 8
#endif /* ITT_ARCH_S390X */

#ifndef ITT_ARCH
#if defined _M_IX86 || defined __i386__
#define ITT_ARCH ITT_ARCH_IA32
Expand All @@ -181,6 +185,8 @@
#define ITT_ARCH ITT_ARCH_PPC64
#elif defined __ve__
#define ITT_ARCH ITT_ARCH_VE
#elif defined __s390x__
#define ITT_ARCH ITT_ARCH_S390X
#endif
#endif

Expand Down
Loading