Merge pull request #21 from jhendersonHDF/subf_user_guide

Update Subfiling User's Guide with notes about stripe count

Author: derobins

RFCs/HDF5_Library/VFD_Subfiling/user_guide/HDF5_Subfiling_VFD_User_s_Guide.pdf

@@ -299,11 +299,15 @@ \subsection{H5FD\_subfiling\_params\_t}
 
 \texttt{int64\_t stripe\_size} - This field contains the size (in bytes) of data stripes
 written to subfiles. The default setting for this parameter is a 32MiB stripe size. The value
-must be $> 0$.
+set must either be the value defined by the \texttt{H5FD\_SUBFILING\_DEFAULT\_STRIPE\_SIZE}
+macro, or an integer value $> 0$.
 
 \texttt{int32\_t stripe\_count} - This field contains the target number of subfiles to use.
-The default setting for this parameter is set such that one subfile is used per machine node.
-The value must be $> 0$.
+It is mostly useful when pre-creating an HDF5 file in a parallel application within some
+subset of \texttt{MPI\_COMM\_WORLD} (see section \ref{sec:tips}). The default setting for
+this parameter is set such that one subfile is used per machine node. The value set must
+either be the value defined by the \texttt{H5FD\_SUBFILING\_DEFAULT\_STRIPE\_COUNT} macro,
+or an integer value $> 0$.
 \end{flushleft}%
 
 \newpage
@@ -392,7 +396,8 @@ \subsection{H5FD\_ioc\_config\_t}
 
 \texttt{int32\_t thread\_pool\_size} - This field contains the number of worker threads to
 use per I/O Concentrator. The default setting for this parameter is 4 worker threads per I/O
-Concentrator. The value must be $> 0$.
+Concentrator. The value set must either be the value defined by the \texttt{H5FD\_IOC\_DEFAULT\_THREAD\_POOL\_SIZE}
+macro, or an integer value $> 0$.
 \end{flushleft}%
 
 \newpage

RFCs/HDF5_Library/VFD_Subfiling/user_guide/appendices/ref_manual.tex

@@ -35,10 +35,10 @@
 % Document metadata: Title, author(s), etc
 \title{HDF5 Subfiling VFD User's Guide}
 \author{Jordan Henderson}
-\date{Jan. 25, 2023}
+\date{Jun. 19, 2023}
 \organization{THG}
 \orglogo{THG_LOGO}
-\rfcversion{v1}
+\rfcversion{v2}
 
 % Revision History
 \input{support/revisions}

RFCs/HDF5_Library/VFD_Subfiling/user_guide/main.tex

@@ -9,6 +9,6 @@ \section*{Acknowledgments}
 
 % Include this section in the table of contents
 \addcontentsline{toc}{section}{Acknowledgements}
-This work was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration.
+This work was supported by the Exascale Computing Project (17-SC-20-SC), a joint project of the U.S. Department of Energy’s Office of Science and National Nuclear Security Administration, responsible for delivering a capable exascale ecosystem, including software, applications, and hardware technology, to support the nation’s exascale computing imperative.
 
 \end{document}

RFCs/HDF5_Library/VFD_Subfiling/user_guide/sections/acknowledgements.tex

@@ -5,6 +5,7 @@
 \newpage
 
 \section{Tips and best practices}
+\label{sec:tips}
 
 \begin{itemize}
 \item While the default configuration for the Subfiling VFD has been found
@@ -23,6 +24,26 @@ \section{Tips and best practices}
 space and vastly increase the size of the resulting file, so caution may be
 needed when trying to find a good balance with this suggestion.
 
+\item If using the Subfiling VFD with a parallel HDF5 application that
+pre-creates an HDF5 file on a single MPI rank or some other subset of
+\texttt{MPI\_COMM\_WORLD}, it is necessary to adjust the "stripe count" (or
+number of subfiles) value that the Subfiling VFD uses internally in order to
+ensure that the proper number of subfiles gets generated when the logical HDF5
+file is created. Without setting this value, the VFD will default to creating
+one subfile per machine node, which will cause it to create only one subfile
+in the case of pre-creating an HDF5 file on a single MPI rank. Note that as
+pre-creating an HDF5 file on a subset of \texttt{MPI\_COMM\_WORLD} is expected
+to be a less common operation when using the Subfiling VFD, an environment
+variable for modifying this value has not been provided at this time and the
+VFD will need to be configured on a File Access Property List for this use case.
+Using the example in section \ref{ex:ex2} as a guide, one can configure the
+stripe count value by setting the \texttt{stripe\_count} field in the
+\texttt{H5FD\_subfiling\_params\_t} structure, which is part of the
+\texttt{H5FD\_subfiling\_config\_t} structure that is passed to 
+\texttt{H5Pset\_fapl\_subfiling}. For that particular example, one would
+set the \texttt{subf\_config.shared\_cfg.stripe\_count} field to the desired
+total number of subfiles.
+
 \item While not always leading to an improvement in performance, it can
 sometimes be useful to enable collective metadata writes and, if the application's
 access pattern allows for it, collective metadata reads in a parallel HDF5

RFCs/HDF5_Library/VFD_Subfiling/user_guide/sections/tips.tex

@@ -194,6 +194,14 @@ \subsubsection{By environment variables}
 worker threads per I/O Concentrator. It is interpreted as an \texttt{int} value and must be $> 0$.
 \end{itemize}
 
+Note that modifying the stripe count from its default setting is expected to be a less common operation
+with the Subfiling VFD. Therefore, an environment variable for changing the value has not currently
+been provided at this time. If an HDF5 application needs to modify the stripe count value (refer to
+section \ref{sec:tips} for information on when modifying the stripe count value is necessary), the
+Subfiling VFD will need to be configured by a File Access Property List. However, note that any
+settings in environment variables will still take precedence, even when the VFD is configured by
+a File Access Property List.
+
 \subsection{Minimal example}
 
 The following is a minimal example of creating a Subfiling-based HDF5 file

RFCs/HDF5_Library/VFD_Subfiling/user_guide/sections/usage.tex

@@ -1,2 +1,3 @@
 %  This section is required and it should be updated with each version change. Add a new revision in the `support/revisions.tex' file, they will be added to this table automatically.  Note the version number, date, and say what it did / what changed. If this is the first version that was published for public comment, tell the reader where to send comments, as in the Version 3 example in the table below.
 \revision{v1}{Jan. 25, 2023}{Version 1 drafted.}
+\revision{v2}{Jun. 19, 2023}{Added notes about Subfiling VFD's stripe count setting}