Update compiler options (#656)

* Better options for aocc, clarify aocc is in clang module

* Due to bug in OpenBLAS remove thread binding from defaults

The bug OpenMathLib/OpenBLAS#2238 is fixed in the
development version

* REmove fixme

* Fix table formatting

* Improve AMD naming

* Typo fix

* Add example with thread binding (with Openblas caveat)

* Specify full module name

* typo

Author: jussienko

docs/computing/compiling-mahti.md

@@ -13,12 +13,12 @@
 C/C++ and Fortran applications can be built with
 [GNU](https://gcc.gnu.org), [AMD](https://developer.amd.com/amd-aocc/), 
 or [Intel](https://software.intel.com/en-us/parallel-studio-xe/documentation/get-started)
-compiler suites. The GNU suite is loaded by default, AMD or Intel can
-be selected via the [Modules](modules.md) system, i.e.
+compiler suites. The GNU compilers are loaded by default. AMD compilers can be
+loaded using the [Modules](modules.md) system with the command:
 ```
 module load clang
 ```
-or
+and Intel compilers with the command:
 ```
 module load intel
 ```
@@ -37,13 +37,12 @@ the safe level and then move up to intermediate or even aggressive,
 while making sure the results are  correct and the program's
 performance has improved. 
 
-<!-- FIXME: clang options, what are recommend Intel options? -->
 
-| Optimisation level | GNU               | Intel                        | AMD          |
+| Optimisation level | GNU               | Intel                        | AMD (clang) |
 | :----------------- | :---------------- | :--------------------------- | :----------- |
-| **Safe**           | -O2 -march=native | -O2 -fp-model precise | -O2          |
-| **Intermediate**   | -O3 -march=native | -O2                    | -O3          |
-| **Aggressive**     | -O3 -march=native -ffast-math -funroll-loops | -O3 -fp-model fast=2 -no-prec-div -fimf-use-svml=true | -O3 |
+| **Safe**           | -O2 -march=native | -O2 -fp-model precise | -O2 -march=native  |
+| **Intermediate**   | -O3 -march=native | -O2                    | -O3 -march=native |
+| **Aggressive**     | -O3 -march=native -ffast-math -funroll-loops | -O3 -fp-model fast=2 -no-prec-div -fimf-use-svml=true | -O3 -march=native -ffast-math -funroll-loops |
 
 
 A detailed list of options for the Intel and GNU compilers can be found on the _man_

docs/computing/running/creating-job-scripts-mahti.md

@@ -10,7 +10,6 @@ environment. The rest of this page focuses on Mahti specific topics.
     with the exception of interactive jobs (to be added). Many options also work
     differently in Puhti and Mahti, so it is not advisable to copy scripts from Puhti
     to Mahti.
-<!-- FIXME interactive jobs -->
 
 [TOC]
 
@@ -96,9 +95,9 @@ export OMP_NUM_THREADS=1
 srun myprog -i input -o output
 ```
 
-For hybrid applications, one should use `OMP_PLACES` and
-`OMP_PROC_BIND` OpenMP runtime environment variables for obtaining
-optimum placement of OpenMP threads. As an example, in order to run
+For hybrid applications, one should use 
+`OMP_PROC_BIND` OpenMP runtime environment variable for 
+placing the OpenMP threads. As an example, in order to run
 one MPI tasks per NUMA domain and one OpenMP thread per L3cache one
 can set
 
@@ -107,7 +106,6 @@ can set
 #SBATCH --cpus-per-task=16
 
 export OMP_NUM_THREADS=4
-export OMP_PLACES=cores
 export OMP_PROC_BIND=spread
 
 module load myprog/1.2.3

docs/computing/running/example-job-scripts-mahti.md

@@ -52,12 +52,32 @@ srun myprog <options>
 
 # Set the number of threads based on --cpus-per-task
 export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
-# Bind OpenMP threads to cores
+
+srun myprog <options>
+```
+
+## MPI + OpenMP with thread binding
+
+Note! Due to bug in OpenBLAS, thread binding should not be used in applications
+utilizing threaded OpenBLAS (openblas/0.3.10-omp module)
+```
+#!/bin/bash
+#SBATCH --job-name=example
+#SBATCH --account=<project>
+#SBATCH --partition=large
+#SBATCH --time=02:00:00
+#SBATCH --nodes=100
+#SBATCH --ntasks-per-node=16
+#SBATCH --cpus-per-task=8
+
+# Set the number of threads based on --cpus-per-task
+export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
 export OMP_PLACES=cores
 
 srun myprog <options>
 ```
 
+
 ## MPI + OpenMP with simultaneous multithreading
 
 ```
@@ -75,8 +95,6 @@ srun myprog <options>
 
 # Set the number of threads based on --cpus-per-task
 export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
-# Bind OpenMP threads to hardware threads
-export OMP_PLACES=threads
 
 srun myprog <options>
 ```

docs/computing/running/performance-checklist.md

@@ -54,7 +54,10 @@ higher and poor load balancing gets more likely.
 
 Many HPC applications benefit from binding OpenMP threads to CPU cores
 which can be achieved by setting `export OMP_PLACES=cores` in the
-batch job script. When starting new production runs it is also good
+batch job script. Note! Due to bug in OpenBLAS thread binding should not be
+specified when using threaded OpenBLAS (openblas/0.3.10-omp module). 
+
+When starting new production runs it is also good
 practice to ensure correct thread affinity by adding to batch job
 script
 ```