Generate bigcount interfaces for Fortran and C #12226

jtronge · 2024-01-10T21:16:33Z

This adds scripts for generating the C and Fortran mpi_f08 API bindings from template files, while also generating bigcount interfaces for those that require them. On the Fortran side it also adds support for TS 29113 when possible, allowing for better Fortran array handling that matches the standard (some files were imported from PR #10302).

Python >=3.6 is required for running these scripts, which is only necessary when the binding files have not already been generated. Users of the distribution tarball should not need to generate these files and thus should not require Python.

We used https://github.com/cea-hpc/pcvs-benchmarks and the MPI4PY test suite to help ensure all big count interfaces (C and Fortran) are being generated.

PR #12033 is a previous version of this focused specifically on ABI support.

Note that there are additional changes needed to Open MPI for bigcount support. One is that the datatype system needs to be embiggened. This PR includes workarounds for this lack of support that will need to be changed once the datatype system is embiggened. These areas of code are marked with TODO:BIGCOUNT.

jsquyres

I haven't finished reading / understanding / reviewing both generate_bindings.py scripts yet, but since I'm so late for this and the clock is ticking on @jtronge's availability, I'm going to submit what I have so far.

ompi/mpi/fortran/use-mpi-f08/Makefile.am

ompi/include/mpi.h.in

ompi/mpi/c/Makefile.am

ompi/mpi/c/generate_bindings.py

jsquyres · 2024-02-07T16:15:30Z

Should #12033 be closed (if this PR wholly replaces it)?

jtronge · 2024-02-07T17:14:40Z

I think I'll try to remove the extra standard ABI code from the script, as well as the additional items you caught @jsquyres.

I don't think this code completely replaces PR #12033, especially since that one includes the refactor of libmpi into multiple libraries that we discussed.

config/ompi_setup_mpi_fortran.m4

docs/developers/bindings.rst

jtronge · 2024-02-14T22:44:16Z

/azp run

hppritcha · 2024-08-21T19:08:12Z

bot:ompi:retest

for count args. This helps with the big count effort. Related to open-mpi#12226 Related to open-mpi#13152 Signed-off-by: Howard Pritchard <[email protected]>

jsquyres

Replied to a few comments and resolved all the open conversations that I think are now actually resolved. I think my last concern is the question about: are we planning to ship the bigcount APIs without actually supporting big counts?

hppritcha · 2025-03-20T17:04:02Z

Okay. We should put this discussion of what to do with the datatype support in one of our upcoming tuesday meetings. Meanwhile, I think this PR is good to go - modulo commit squashing. Its clear from the initial commit message that this was one of several pieces of work that need to be done before we can claim complete (or reasonably complete) support for big count. Also getting this in to main now will help flush out any problems it may cause well before we branch for the next major release. If there's some fundamental issue with providing bigcount support in the datatype engine we can modify the binding generating infrastructure to not generate the bigcount variants.

Note we've already merged other PRs for bigcount support that had this same problem with the datatype engine - see for example comments in #12621 .

jsquyres

Ok, let's squash and merge.

hppritcha · 2025-03-24T21:35:18Z

@jsquyres for some reason github lost your approval. could you please reapprove?

devreal · 2025-03-26T15:49:10Z

I tried to build this branch on my mac and get this error:

  GENERATE mpi-f08-interfaces-generated.h
Traceback (most recent call last):
  File "/Users/joseph/src/openmpi/git/ompi/build/ompi/mpi/fortran/use-mpi-f08/mod/../../../../../../ompi/mpi/bindings/bindings.py", line 75, in <module>
    main()
    ~~~~^^
  File "/Users/joseph/src/openmpi/git/ompi/build/ompi/mpi/fortran/use-mpi-f08/mod/../../../../../../ompi/mpi/bindings/bindings.py", line 71, in main
    args.handler(args, OutputFile(f))
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joseph/src/openmpi/git/ompi/build/ompi/mpi/fortran/use-mpi-f08/mod/../../../../../../ompi/mpi/bindings/bindings.py", line 43, in <lambda>
    parser_interface.set_defaults(handler=lambda args, out: fortran.generate_interface(args, out))
                                                            ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/joseph/src/openmpi/git/ompi/build/../ompi/mpi/bindings/ompi_bindings/fortran.py", line 277, in generate_interface
    templates = load_function_templates(args.prototype_files)
  File "/Users/joseph/src/openmpi/git/ompi/build/../ompi/mpi/bindings/ompi_bindings/fortran.py", line 248, in load_function_templates
    SourceTemplate.load(fname, type_constructor=FortranType.construct)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joseph/src/openmpi/git/ompi/build/../ompi/mpi/bindings/ompi_bindings/parser.py", line 97, in load
    with open(fname) as fp:
         ~~~~^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '../accumulate_ts.c.in'
make[2]: *** [mpi-f08-interfaces-generated.h] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1

Not sure what is going on, might be a python version issue.

➜  build git:(bigcount) python3 --version
Python 3.13.2

devreal · 2025-03-26T15:55:23Z

The path ../accumulate_ts.c.in seems suspicious. I'm doing an out-of-tree build so the build directory is located somewhere else than the ompi/mpi/fortran/use-mpi-f08/bindings directory where I assume the generator runs.

jsquyres

@devreal I was just doing a VPATH build myself to confirm, but your reply beat me to it. Yeah, I'm guessing there's something not quite right in our Makefile.am substitution to create lists of input files. I.e., instead of making the files be ../BLAH, they'll need to be listed relative to the srcdir.

There's now a small conflict in a Makefile.am that needs to be resolved, too.

hppritcha · 2025-03-26T15:59:38Z

out of path build should work. i checked that a week or so ago and IWFM

hppritcha · 2025-03-26T16:00:56Z

of course that may have been prior to these numerous makefile refactors

hppritcha · 2025-03-26T16:50:39Z

fixing your issue @devreal having problems with the rebase though so i'll push fix for your problem first. thanks for giving this PR a try.

hppritcha · 2025-03-26T18:49:36Z

@devreal try now

This adds scripts for generating the C API bindings from template files, while also generating bigcount interfaces for those that require them. The binding script also include initial support for the mpi_f08 Fortran bindings, but doesn't yet make any changes to fortran/use-mpi-f08 Python >=3.6 is required for running these scripts, which is only necessary when the binding files have not already been generated. Users of the distribution tarball should not need to generate these files and thus should not require Python. Note that there are additional changes needed to Open MPI for bigcount support. One is that the datatype system needs to be embiggened. This PR includes workarounds for this lack of support that will need to be changed once the datatype system is embiggened. These areas of code are marked with TODO:BIGCOUNT. Co-authored-by: mphinney1100 <[email protected]> Co-authored-by: Howard Pritchard <[email protected]> Signed-off-by: Jake Tronge <[email protected]>

devreal

Thanks, I'm able to build it and the RMA API looks correct.

bosilca · 2025-03-26T21:16:16Z

works with and without VPATH with both gcc and nv compilers. 👍

This updates fortran/use-mpi-f08 to generate most of the Fortran bindings from a script and template files. It also adds support for Fortran TS 29113 when possible, allowing for better Fortran array handling that matches the standard. The C files were imported from PR open-mpi#10302 and converted to templates to be fed into the binding script. Co-authored-by: Gilles Gouaillardet <[email protected]> Co-authored-by: Howard Pritchard <[email protected]> Signed-off-by: Jake Tronge <[email protected]>

Signed-off-by: Jake Tronge <[email protected]>

hppritcha · 2025-03-27T16:47:32Z

squashing a bit

github-actions bot added the Target: main label Jan 10, 2024

hppritcha requested a review from jsquyres January 10, 2024 21:18

jtronge force-pushed the bigcount branch from 1a9a108 to 37ada30 Compare January 11, 2024 15:29

jsquyres mentioned this pull request Jan 30, 2024

mpi_f08 configure tests failing for NAG #9795

Open

jsquyres reviewed Feb 7, 2024

View reviewed changes

jtronge mentioned this pull request Feb 7, 2024

Standard ABI generation code and library refactor. #12033

Draft

hppritcha reviewed Feb 8, 2024

View reviewed changes

config/ompi_setup_mpi_fortran.m4 Outdated Show resolved Hide resolved

hppritcha reviewed Feb 12, 2024

View reviewed changes

docs/developers/bindings.rst Outdated Show resolved Hide resolved

hppritcha mentioned this pull request Feb 14, 2024

COLL framework needs work to support MPI Bigcount #12336

Closed

jtronge force-pushed the bigcount branch from 2ae81f2 to cad1e4e Compare April 1, 2024 20:14

jtronge force-pushed the bigcount branch from 66909b4 to e00da70 Compare April 22, 2024 16:44

jsquyres mentioned this pull request Jul 13, 2024

Fortran: first try if type(*), dimension(..) works as-is #11591

Draft

jtronge force-pushed the bigcount branch 6 times, most recently from 8484636 to 90d6570 Compare August 1, 2024 15:07

jtronge force-pushed the bigcount branch 3 times, most recently from 6476640 to 569efa1 Compare August 12, 2024 18:46

jtronge force-pushed the bigcount branch from fa6f257 to 2811c1f Compare August 13, 2024 15:06

wenduwan added the Target: v6.0.x label Aug 13, 2024

github-actions bot removed the Target: v6.0.x label Aug 13, 2024

jtronge force-pushed the bigcount branch 2 times, most recently from 60ea322 to ab496c4 Compare August 15, 2024 21:31

jsquyres reviewed Mar 20, 2025

View reviewed changes

hppritcha force-pushed the bigcount branch from 85b8017 to 4962bda Compare March 20, 2025 15:03

jsquyres previously approved these changes Mar 23, 2025

View reviewed changes

hppritcha dismissed jsquyres’s stale review via 326b65a March 24, 2025 19:50

hppritcha removed the request for review from edgargabriel March 24, 2025 19:51

hppritcha force-pushed the bigcount branch from 326b65a to d33f78b Compare March 24, 2025 21:08

jsquyres approved these changes Mar 24, 2025

View reviewed changes

hppritcha mentioned this pull request Mar 25, 2025

use-mpi-f08: fix prototypes and more #13168

Merged

jsquyres requested changes Mar 26, 2025

View reviewed changes

hppritcha force-pushed the bigcount branch from fdcae7e to 68cd933 Compare March 26, 2025 19:20

devreal approved these changes Mar 26, 2025

View reviewed changes

bosilca approved these changes Mar 26, 2025

View reviewed changes

jsquyres approved these changes Mar 27, 2025

View reviewed changes

jtronge and others added 2 commits March 27, 2025 10:44

Bigcount: Update docs and address other comments

05a5924

Signed-off-by: Jake Tronge <[email protected]>

hppritcha force-pushed the bigcount branch from 68cd933 to 05a5924 Compare March 27, 2025 17:02

bosilca merged commit 92e1378 into open-mpi:main Mar 28, 2025
15 checks passed

github-actions bot mentioned this pull request May 8, 2025

fortran: fix common symbol sizes and alignments #13230

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate bigcount interfaces for Fortran and C #12226

Generate bigcount interfaces for Fortran and C #12226

jtronge commented Jan 10, 2024 •

edited by hppritcha

Loading

jsquyres left a comment

jsquyres commented Feb 7, 2024

jtronge commented Feb 7, 2024

jtronge commented Feb 14, 2024

hppritcha commented Aug 21, 2024

jsquyres left a comment •

edited

Loading

hppritcha commented Mar 20, 2025

jsquyres left a comment

hppritcha commented Mar 24, 2025

devreal commented Mar 26, 2025

devreal commented Mar 26, 2025

jsquyres left a comment

hppritcha commented Mar 26, 2025

hppritcha commented Mar 26, 2025

hppritcha commented Mar 26, 2025

hppritcha commented Mar 26, 2025

devreal left a comment

bosilca commented Mar 26, 2025

hppritcha commented Mar 27, 2025

Generate bigcount interfaces for Fortran and C #12226

Generate bigcount interfaces for Fortran and C #12226

Conversation

jtronge commented Jan 10, 2024 • edited by hppritcha Loading

jsquyres left a comment

Choose a reason for hiding this comment

jsquyres commented Feb 7, 2024

jtronge commented Feb 7, 2024

jtronge commented Feb 14, 2024

hppritcha commented Aug 21, 2024

jsquyres left a comment • edited Loading

Choose a reason for hiding this comment

hppritcha commented Mar 20, 2025

jsquyres left a comment

Choose a reason for hiding this comment

hppritcha commented Mar 24, 2025

devreal commented Mar 26, 2025

devreal commented Mar 26, 2025

jsquyres left a comment

Choose a reason for hiding this comment

hppritcha commented Mar 26, 2025

hppritcha commented Mar 26, 2025

hppritcha commented Mar 26, 2025

hppritcha commented Mar 26, 2025

devreal left a comment

Choose a reason for hiding this comment

bosilca commented Mar 26, 2025

hppritcha commented Mar 27, 2025

jtronge commented Jan 10, 2024 •

edited by hppritcha

Loading

jsquyres left a comment •

edited

Loading