Skip to content

Commit 30cbe0c

Browse files
jsquyresjsquyres
committed
PROOF OF CONCEPT -- comments welcome
Per discussion from https://github.com/open-mpi/ompi/wiki/Meeting-2025-03-14, we would like to make relocating an Open MPI installation easier. Specifically, if we can avoid the need to propagate various *_PREFIX environment variables / command line options down through multiple layers and subsystems (OMPI, PRRTE, PMIx), that would eliminate/simplify a bunch of our code. Text help files are something that Open MPI currently has to find in the install tree filesystem at run-time. This commit is a proposal: 1. Developers still maintain help messages in the various help_*.txt files around the code base. Maintaining descriptive, user-friendly help messages in text files (instead of manually hand-coding long strings in C) has proven to be quite useful. We do not want to lose this capability. 2. During "make", those help_*.txt files are encoded into .c and .h files where the individual messages are assigned to unique C variables (properly prefixed per the prefix rule). 3. These strings can then be passed to a slightly modified opal_show_help*() function for rendering and deduplication **NOTE:** In this commit, updating opal_show_help() is simulated by creating a new opal_showhelp2() function just so that we can update one component without yet disturbing the rest of the code base. This commit converts the TCP BTL to the proposed system so that the developer community can comment and discuss. Once a final system is decided upon, it can be propagated to the rest of the code base. Changes that will need to be made wherever a help_*.txt file is used (e.g., in components, but also in various base directories): * Update Makefile.am: * Remove the help_*.txt file from dist_opaldata_DATA, add to EXTRA_DIST * Invoke $(OMPI_HELP_GEN) (i.e., a Python script) to generate the .c and .h files containing the help strings / variable declarations * Add the generated .c and .h files to the list of source files * Add the generated .c and .h files to MAINTAINERCLEANFILES * At existing call sites to opal_show_help*(): * #include the relevant .h file(s) declaring the help file strings * Remove the filename from the call to opal_show_help() * Add a prefix string to the topic name (e.g., "btl_tcp:") to guarantee uniqueness * Add the C variable containing the help message as the 2nd param **NOTE:** If this proposal is accepted, *all* help-*.txt files and calls to opal_show_help*() will be converted to the new system. The intent is that *no* text help files will be installed; *all* function calls to display help messages will use the new encoded-in-C-variables-at-"make"-time scheme. This work is intended for main / v6.0.x -- not for v5.0.x. Signed-off-by: Jeff Squyres <[email protected]>
1 parent 9fbd849 commit 30cbe0c

11 files changed

+276
-29
lines changed

‎.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -316,6 +316,10 @@ opal/tools/wrappers/opal.pc
316316
opal/util/show_help_lex.c
317317
opal/util/keyval/keyval_lex.c
318318

319+
# In general, these files are generated
320+
*_help.c
321+
*_help.h
322+
319323
test/monitoring/aggregate_profile.pl
320324
test/monitoring/profile2mat.pl
321325

‎Makefile.ompi-rules

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# Copyright (c) 2008-2022 Cisco Systems, Inc. All rights reserved.
33
# Copyright (c) 2008 Sun Microsystems, Inc. All rights reserved.
44
# Copyright (c) 2020 Intel, Inc. All rights reserved.
5-
# Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
5+
# Copyright (c) 2023-2025 Jeffrey M. Squyres. All rights reserved.
66
# $COPYRIGHT$
77
#
88
# Additional copyrights may follow
@@ -42,3 +42,5 @@ ompi__v_SPHINX_HTML_0 = @echo " GENERATE HTML docs";
4242
OMPI_V_SPHINX_MAN = $(ompi__v_SPHINX_MAN_$V)
4343
ompi__v_SPHINX_MAN_ = $(ompi__v_SPHINX_MAN_$AM_DEFAULT_VERBOSITY)
4444
ompi__v_SPHINX_MAN_0 = @echo " GENERATE man pages";
45+
46+
OMPI_HELP_GEN = $(OMPI_V_GEN) $(top_srcdir)/config/ini-to-c.py

‎config/Makefile.am

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
# Copyright (c) 2016 Research Organization for Information Science
1717
# and Technology (RIST). All rights reserved.
1818
# Copyright (c) 2022 Amazon.com, Inc. or its affiliates. All Rights reserved.
19+
# Copyright (c) 2025 Jeffrey M. Squyres. All Rights reserved.
1920
# $COPYRIGHT$
2021
#
2122
# Additional copyrights may follow
@@ -46,7 +47,8 @@ EXTRA_DIST = \
4647
getdate.sh \
4748
from-savannah/upstream-config.guess \
4849
from-savannah/upstream-config.sub \
49-
extract-3rd-party-configure.pl
50+
extract-3rd-party-configure.pl \
51+
ini-to-c.py
5052

5153
maintainer-clean-local:
5254
rm -f opal_get_version.sh

‎config/ini-to-c.py

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
#!/usr/bin/env python3
2+
#
3+
# Copyright (c) 2025 Jeffrey M. Squyres. All rights reserved.
4+
#
5+
# $COPYRIGHT$
6+
#
7+
# Additional copyrights may follow
8+
#
9+
# $HEADER$
10+
#
11+
12+
# Use minimal modules (e.g., don't use a module for INI) so that we
13+
# can avoid needing to PIP-install anything.
14+
15+
import re
16+
import argparse
17+
import os
18+
19+
def ini_section_to_c_name(section: str, prefix: str) -> str:
20+
# Convert an INI section name to a valid C variable name with a
21+
# prefix
22+
c_name = re.sub(r'[^a-zA-Z0-9_]', '_', section)
23+
24+
# Enforce C11 identifier length limit (63 characters for internal
25+
# identifiers).
26+
return f"{prefix}{c_name}"[:63]
27+
28+
def ini_content_to_c_string(content: str) -> str:
29+
# Convert an INI section content into a valid C string
30+
return content.replace("\n", "\\n").replace("\"", "\\\"")
31+
32+
def parse_ini_file(file_path: str):
33+
# Parse the INI-style text file, excluding comment lines.
34+
sections = {}
35+
current_section = None
36+
37+
with open(file_path, 'r', encoding='utf-8') as file:
38+
for line in file:
39+
line = line.strip()
40+
if line.startswith('#') or not line:
41+
continue
42+
if line.startswith('[') and line.endswith(']'):
43+
current_section = line[1:-1]
44+
sections[current_section] = ""
45+
elif current_section is not None:
46+
sections[current_section] += line + "\n"
47+
return sections
48+
49+
def generate_c_files(sections, input_filename, base_filename: str,
50+
prefix: str):
51+
# Generate a C file and a header file with the parsed INI data
52+
c_filename = f"{base_filename}.c"
53+
h_filename = f"{base_filename}.h"
54+
55+
macro_name = re.sub(r'[^a-zA-Z0-9_]', '_', base_filename.upper())
56+
57+
with open(c_filename, 'w', encoding='utf-8') as c_file, \
58+
open(h_filename, 'w', encoding='utf-8') as h_file:
59+
60+
header = f"""// THIS IS AN AUTOMATICALLY-GENERATED FILE!
61+
// It was generated from {input_filename}
62+
// Edits will be lost"""
63+
64+
h_file.write(f"""{header}
65+
66+
#ifndef {macro_name}_H
67+
#define {macro_name}_H
68+
69+
""")
70+
c_file.write(f"""{header}
71+
72+
#include "{base_filename}.h"
73+
""")
74+
75+
for section, content in sections.items():
76+
c_var_name = ini_section_to_c_name(section, prefix)
77+
c_string = ini_content_to_c_string(content)
78+
79+
h_file.write(f'extern const char {c_var_name}[];\n')
80+
c_file.write(f'\nconst char {c_var_name}[] = "{c_string}";\n')
81+
82+
h_file.write(f"\n#endif // {macro_name}_H\n")
83+
84+
def main():
85+
parser = argparse.ArgumentParser(description="Convert an INI-style text file to C variable definitions.")
86+
parser.add_argument("--in",
87+
dest='input_file',
88+
required=True,
89+
help="Path to the input INI-style file.")
90+
parser.add_argument("--out",
91+
required=True,
92+
help="Basename for output .c and .h files.")
93+
parser.add_argument("--prefix",
94+
required=True,
95+
help="Prefix for all C macro names and variable names.")
96+
parser.add_argument("--verbose",
97+
action=argparse.BooleanOptionalAction,
98+
help="Show output or not")
99+
100+
args = parser.parse_args()
101+
102+
sections = parse_ini_file(args.input_file)
103+
generate_c_files(sections, args.input_file, args.out, args.prefix)
104+
if args.verbose:
105+
print(f"Generated {args.out}.c and {args.out}.h.")
106+
107+
if __name__ == "__main__":
108+
main()

‎opal/mca/btl/tcp/Makefile.am

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,16 +13,30 @@
1313
# Copyright (c) 2013 NVIDIA Corporation. All rights reserved.
1414
# Copyright (c) 2017 IBM Corporation. All rights reserved.
1515
# Copyright (c) 2022 Amazon.com, Inc. or its affiliates. All Rights reserved.
16+
# Copyright (c) 2025 Jeffrey M. Squyres. All Rights reserved.
1617
# $COPYRIGHT$
1718
#
1819
# Additional copyrights may follow
1920
#
2021
# $HEADER$
2122
#
2223

23-
dist_opaldata_DATA = help-mpi-btl-tcp.txt
24+
include $(top_srcdir)/Makefile.ompi-rules
25+
26+
EXTRA_DIST = $(HELP_INI)
27+
28+
HELP_INI = help-mpi-btl-tcp.txt
29+
HELP_PREFIX = btl_tcp_help
30+
GENERATED_HELP = $(HELP_PREFIX).h $(HELP_PREFIX).c
31+
32+
$(GENERATED_HELP): $(HELP_INI)
33+
$(OMPI_HELP_GEN) \
34+
--in $(abs_srcdir)/$(HELP_INI) \
35+
--out $(HELP_PREFIX) \
36+
--prefix $(HELP_PREFIX)_
2437

2538
sources = \
39+
$(GENERATED_HELP) \
2640
btl_tcp.c \
2741
btl_tcp.h \
2842
btl_tcp_addr.h \
@@ -59,3 +73,5 @@ mca_btl_tcp_la_LDFLAGS = -module -avoid-version
5973
noinst_LTLIBRARIES = $(lib)
6074
libmca_btl_tcp_la_SOURCES = $(lib_sources)
6175
libmca_btl_tcp_la_LDFLAGS = -module -avoid-version
76+
77+
MAINTAINERCLEANFILES = $(GENERATED_HELP)

‎opal/mca/btl/tcp/btl_tcp_component.c

Lines changed: 50 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020
* Copyright (c) 2014-2017 Research Organization for Information Science
2121
* and Technology (RIST). All rights reserved.
2222
* Copyright (c) 2018-2022 Amazon.com, Inc. or its affiliates. All Rights reserved.
23-
* Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
23+
* Copyright (c) 2023-2025 Jeffrey M. Squyres. All rights reserved.
2424
* $COPYRIGHT$
2525
*
2626
* Additional copyrights may follow
@@ -85,6 +85,7 @@
8585
#include "btl_tcp_endpoint.h"
8686
#include "btl_tcp_frag.h"
8787
#include "btl_tcp_proc.h"
88+
#include "btl_tcp_help.h"
8889
#include "opal/constants.h"
8990
#include "opal/mca/btl/base/base.h"
9091
#include "opal/mca/btl/base/btl_base_error.h"
@@ -201,13 +202,17 @@ static void mca_btl_tcp_component_accept_handler(int, short, void *);
201202
static int mca_btl_tcp_component_verify(void)
202203
{
203204
if (mca_btl_tcp_component.tcp_port_min > USHRT_MAX) {
204-
opal_show_help("help-mpi-btl-tcp.txt", "invalid minimum port", true, "v4",
205+
opal_showhelp2("btl_tcp:invalid minimum port",
206+
btl_tcp_help_invalid_minimum_port,
207+
true, "v4",
205208
opal_process_info.nodename, mca_btl_tcp_component.tcp_port_min);
206209
mca_btl_tcp_component.tcp_port_min = 1024;
207210
}
208211
#if OPAL_ENABLE_IPV6
209212
if (mca_btl_tcp_component.tcp6_port_min > USHRT_MAX) {
210-
opal_show_help("help-mpi-btl-tcp.txt", "invalid minimum port", true, "v6",
213+
opal_showhelp2("btl_tcp:invalid minimum port",
214+
btl_tcp_help_invalid_minimum_port
215+
true, "v6",
211216
opal_process_info.nodename, mca_btl_tcp_component.tcp6_port_min);
212217
mca_btl_tcp_component.tcp6_port_min = 1024;
213218
}
@@ -678,7 +683,9 @@ static char **split_and_resolve(char **orig_str, char *name, bool reqd)
678683
tmp = strdup(argv[i]);
679684
str = strchr(argv[i], '/');
680685
if (NULL == str) {
681-
opal_show_help("help-mpi-btl-tcp.txt", "invalid if_inexclude", true, name,
686+
opal_showhelp2("btl_tcp:invalid if_inexclude",
687+
btl_tcp_help_invalid_if_inexclude,
688+
true, name,
682689
opal_process_info.nodename, tmp,
683690
"Invalid specification (missing \"/\")");
684691
free(argv[i]);
@@ -694,7 +701,9 @@ static char **split_and_resolve(char **orig_str, char *name, bool reqd)
694701
free(argv[i]);
695702

696703
if (1 != ret) {
697-
opal_show_help("help-mpi-btl-tcp.txt", "invalid if_inexclude", true, name,
704+
opal_showhelp2("btl_tcp:invalid if_inexclude",
705+
btl_tcp_help_invalid_if_inexclude,
706+
true, name,
698707
opal_process_info.nodename, tmp,
699708
"Invalid specification (inet_pton() failed)");
700709
free(tmp);
@@ -737,7 +746,9 @@ static char **split_and_resolve(char **orig_str, char *name, bool reqd)
737746
/* If we didn't find a match, keep trying */
738747
if (0 == match_count) {
739748
if (reqd || mca_btl_tcp_component.report_all_unfound_interfaces) {
740-
opal_show_help("help-mpi-btl-tcp.txt", "invalid if_inexclude", true, name,
749+
opal_showhelp2("btl_tcp:invalid if_inexclude",
750+
btl_tcp_help_invalid_if_inexclude,
751+
true, name,
741752
opal_process_info.nodename, tmp,
742753
"Did not find interface matching this subnet");
743754
}
@@ -833,7 +844,9 @@ static int mca_btl_tcp_component_create_instances(void)
833844
char *if_name = *argv;
834845
int idx = opal_ifnametokindex(if_name);
835846
if (idx < 0) {
836-
opal_show_help("help-mpi-btl-tcp.txt", "invalid if_inexclude", true, "include",
847+
opal_showhelp2("btl_tcp:invalid if_inexclude",
848+
btl_tcp_help_invalid_if_inexclude,
849+
true, "include",
837850
opal_process_info.nodename, if_name, "Unknown interface name");
838851
ret = OPAL_ERR_NOT_FOUND;
839852
goto cleanup;
@@ -1079,15 +1092,19 @@ static int mca_btl_tcp_component_create_listen(uint16_t af_family)
10791092

10801093
/* set socket up to be non-blocking, otherwise accept could block */
10811094
if ((flags = fcntl(sd, F_GETFL, 0)) < 0) {
1082-
opal_show_help("help-mpi-btl-tcp.txt", "socket flag fail", true, opal_process_info.nodename,
1095+
opal_showhelp2("btl_tcp:socket flag fail",
1096+
btl_tcp_help_socket_flag_fail,
1097+
true, opal_process_info.nodename,
10831098
getpid(), "fcntl(sd, F_GETFL, 0)", strerror(opal_socket_errno),
10841099
opal_socket_errno);
10851100
CLOSE_THE_SOCKET(sd);
10861101
return OPAL_ERROR;
10871102
} else {
10881103
flags |= O_NONBLOCK;
10891104
if (fcntl(sd, F_SETFL, flags) < 0) {
1090-
opal_show_help("help-mpi-btl-tcp.txt", "socket flag fail", true,
1105+
opal_showhelp2("btl_tcp:socket flag fail",
1106+
btl_tcp_help_socket_flag_fail,
1107+
true,
10911108
opal_process_info.nodename, getpid(),
10921109
"fcntl(sd, F_SETFL, flags & O_NONBLOCK)", strerror(opal_socket_errno),
10931110
opal_socket_errno);
@@ -1362,7 +1379,9 @@ static void mca_btl_tcp_component_accept_handler(int incoming_sd, short ignored,
13621379
continue;
13631380
}
13641381
if (opal_socket_errno != EAGAIN && opal_socket_errno != EWOULDBLOCK) {
1365-
opal_show_help("help-mpi-btl-tcp.txt", "accept failed", true,
1382+
opal_showhelp2("btl_tcp:accept failed",
1383+
btl_tcp_help_accept_failed,
1384+
true,
13661385
opal_process_info.nodename, getpid(), opal_socket_errno,
13671386
strerror(opal_socket_errno));
13681387
}
@@ -1409,7 +1428,9 @@ static void mca_btl_tcp_component_recv_handler(int sd, short flags, void *user)
14091428
if (ENOPROTOOPT == errno || EOPNOTSUPP == errno) {
14101429
sockopt = false;
14111430
} else {
1412-
opal_show_help("help-mpi-btl-tcp.txt", "socket flag fail", true,
1431+
opal_showhelp2("btl_tcp:socket flag fail",
1432+
btl_tcp_help_socket_flag_fail,
1433+
true,
14131434
opal_process_info.nodename, getpid(),
14141435
"getsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, ...)",
14151436
strerror(opal_socket_errno), opal_socket_errno);
@@ -1419,7 +1440,9 @@ static void mca_btl_tcp_component_recv_handler(int sd, short flags, void *user)
14191440
tv.tv_sec = 2;
14201441
tv.tv_usec = 0;
14211442
if (0 != setsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv))) {
1422-
opal_show_help("help-mpi-btl-tcp.txt", "socket flag fail", true,
1443+
opal_showhelp2("btl_tcp:socket flag fail",
1444+
btl_tcp_help_socket_flag_fail,
1445+
true,
14231446
opal_process_info.nodename, getpid(),
14241447
"setsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, ...)",
14251448
strerror(opal_socket_errno), opal_socket_errno);
@@ -1472,7 +1495,9 @@ static void mca_btl_tcp_component_recv_handler(int sd, short flags, void *user)
14721495
if (sockopt) {
14731496
/* reset RECVTIMEO option to its original state */
14741497
if (0 != setsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, &save, sizeof(save))) {
1475-
opal_show_help("help-mpi-btl-tcp.txt", "socket flag fail", true,
1498+
opal_showhelp2("btl_tcp:socket flag fail",
1499+
btl_tcp_help_socket_flag_fail,
1500+
true,
14761501
opal_process_info.nodename, getpid(),
14771502
"setsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, ...)",
14781503
strerror(opal_socket_errno), opal_socket_errno);
@@ -1484,14 +1509,18 @@ static void mca_btl_tcp_component_recv_handler(int sd, short flags, void *user)
14841509

14851510
/* now set socket up to be non-blocking */
14861511
if ((flags = fcntl(sd, F_GETFL, 0)) < 0) {
1487-
opal_show_help("help-mpi-btl-tcp.txt", "socket flag fail", true, opal_process_info.nodename,
1512+
opal_showhelp2("btl_tcp:socket flag fail",
1513+
btl_tcp_help_socket_flag_fail,
1514+
true, opal_process_info.nodename,
14881515
getpid(), "fcntl(sd, F_GETFL, 0)", strerror(opal_socket_errno),
14891516
opal_socket_errno);
14901517
CLOSE_THE_SOCKET(sd);
14911518
} else {
14921519
flags |= O_NONBLOCK;
14931520
if (fcntl(sd, F_SETFL, flags) < 0) {
1494-
opal_show_help("help-mpi-btl-tcp.txt", "socket flag fail", true,
1521+
opal_showhelp2("btl_tcp:socket flag fail",
1522+
btl_tcp_help_socket_flag_fail,
1523+
true,
14951524
opal_process_info.nodename, getpid(),
14961525
"fcntl(sd, F_SETFL, flags & O_NONBLOCK)", strerror(opal_socket_errno),
14971526
opal_socket_errno);
@@ -1502,7 +1531,9 @@ static void mca_btl_tcp_component_recv_handler(int sd, short flags, void *user)
15021531
/* lookup the corresponding process */
15031532
btl_proc = mca_btl_tcp_proc_lookup(&guid);
15041533
if (NULL == btl_proc) {
1505-
opal_show_help("help-mpi-btl-tcp.txt", "server accept cannot find guid", true,
1534+
opal_showhelp2("btl_tcp:server accept cannot find guid",
1535+
btl_tcp_help_server_accept_cannot_find_guid,
1536+
true,
15061537
opal_process_info.nodename, getpid());
15071538
CLOSE_THE_SOCKET(sd);
15081539
return;
@@ -1511,7 +1542,9 @@ static void mca_btl_tcp_component_recv_handler(int sd, short flags, void *user)
15111542
/* lookup peer address */
15121543
if (getpeername(sd, (struct sockaddr *) &addr, &addr_len) != 0) {
15131544
if (ENOTCONN != opal_socket_errno) {
1514-
opal_show_help("help-mpi-btl-tcp.txt", "server getpeername failed", true,
1545+
opal_showhelp2("btl_tcp:server getpeername failed",
1546+
btl_tcp_help_server_getpeername_failed,
1547+
true,
15151548
opal_process_info.nodename, getpid(), strerror(opal_socket_errno),
15161549
opal_socket_errno);
15171550
}

0 commit comments

Comments
 (0)