TO BE SQUASHED

Suggestions above Charles' original commit.

Signed-off-by: Jeff Squyres <[email protected]>

Author: jsquyres

faq/openfabrics.inc

@@ -2343,50 +2343,76 @@ shell$ mpirun --mca pml ucx --mca osc ucx --mca scoll ucx --mca atomic ucx ...
 
 /////////////////////////////////////////////////////////////////////////
 
-$q[] = "I'm getting errors about \"initializing an OpenFabrics device\" when running
-4.0.0 with UCX support enabled.  What should I do?";
+$q[] = "I'm getting errors about \"initializing an OpenFabrics device\" when running v4.0.0 with UCX support enabled.  What should I do?";
 $anchor[] = "ofa-device-error";
-$a[] = "Disable ib verbs support (--without-verbs).
 
-The messages below were observed by at least one site where OpenMPI 4.0.0
-was built with support for InfiniBand verbs (--with-verbs), OFA UCX (--with-ucx), 
-and CUDA (--with-cuda) with applications running on gpu-enabled hosts.
+$a[] = "The short answer is that you should probably just disable
+verbs support in Open MPI.
 
-Since the openib BTL (configured via --with-verbs) is deprecated in favor
-of UCX and the UCX PML includes support for the OpenFabrics devices in question 
-(Mellanox HCAs), the openib BTL is not needed.
+The messages below were observed by at least one site where Open MPI v4.0.0
+was built with support for InfiniBand verbs ([--with-verbs]), OFA UCX ([--with-ucx]),
+and CUDA ([--with-cuda]) with applications running on GPU-enabled hosts.
 
-<pre>
+<geshi>
+-----------------------------------------------------------------------------
 WARNING: There was an error initializing an OpenFabrics device.
 
-                Local host: c36a-s39
-                Local device: mlx4_0
-</pre>
+    Local host: c36a-s39
+    Local device: mlx4_0
+-----------------------------------------------------------------------------
+</geshi>
 
 and
 
-<pre>
+<geshi>
 -----------------------------------------------------------------------------
 By default, for Open MPI 4.0 and later, infiniband ports on a device
 are not used by default.  The intent is to use UCX for these devices.
 You can override this policy by setting the btl_openib_allow_ib MCA parameter
 to true.
 
-                    Local host:              c36a-s39
-                    Local adapter:           mlx4_0
-                    Local port:              1
+    Local host:              c36a-s39
+    Local adapter:           mlx4_0
+    Local port:              1
 -----------------------------------------------------------------------------
-</pre>";
+</geshi>
+
+As noted in the messages above, Open MPI deprecated the openib BTL
+(enabled when Open MPI is configured [--with-verbs]) is deprecated in
+favor of the UCX PML, which includes support for OpenFabrics devices.
+The [openib] BTL is therefore not needed.
+
+You can disable the [openib] BTL in a few different ways:
+
+<ul>
+<li> Configure Open MPI [--without-verbs].  This will prevent building
+the [openib] BTL in the first place.</li>
+<li> Disable the [openib] BTL via the [btl] MCA param (see <a
+href=\"?category=tuning#setting-mca-params\">this FAQ item</a> for
+information on how to set MCA params).  For example,
+<geshi bash>
+shell$ mpirun --mca btl '^openib' ...
+</geshi></li>
+<li> Force the use of the UCX PML via the [pml] MCA param.  For example:
+<geshi bash>
+shell$ mpirun --mca pml ucx ...
+</geshi></li>
+</ul>";
+
 /////////////////////////////////////////////////////////////////////////
 
 $q[] = "How can I find out what devices and transports are supported by UCX on my system?";
 $anchor[] = "ucx-supported-devices";
-$a[] = "The <code>ucx_info</code> command can be used.
 
-For example,
-<pre>
+$a[] = "Check out the <a
+href=\"http://www.openucx.org/documentation/\">UCX documentation</a>
+for more information, but you can use the [ucx_info] command.  For
+example:
+
+<gesh bash>
 shell$ ucx_info -d
-</pre>";
+</geshi>";
+
 /////////////////////////////////////////////////////////////////////////
 
 $q[] = "What is <code>cpu-set</code>?";