How to handle non-Fortran dependencies

[certik]

I am very confident we can make fpm very robust to work for pure Fortran packages. Just like Cargo works well for pure Rust packages or pip works great for pure Python packages.

The problem is with non-Fortran packages. Pip allows to hook in compiling C (and with some work) Fortran code, but it's very fragile in my experience (thus the motivation for Conda that is a binary package manager). Python doesn't have an option to avoid C++/Fortran dependencies because Python itself is slow. Rust allows to (in principle) rewrite everything in Rust. As an example, take png. The system bindings: https://crates.io/crates/libpng-sys they say are unreliable, and you should use a pure Rust implementation: https://lib.rs/crates/lodepng.

I agree it does make things more robust to stick to pure Fortran and for many things we will do that and people will provide pure Fortran implementation of basic tasks. Python cannot do it well due to performance, but Rust and Fortran can.

However, we still need a robust way of handling non-Fortran dependencies, because if there is a robust and well maintained library in another language, we should just use it instead of reimplementing everything. Take HDF5. Here is the Rust package: https://crates.io/crates/hdf5. If you look at the documentation how to build it: https://github.com/aldanor/hdf5-rust they even mention Conda (to install the HDF5 library itself on all platforms --- which in my experience is much more robust than pip). Anyway, the way it works is by this line: https://github.com/aldanor/hdf5-rust/blob/15ec644977e0bee2b77340272730b34209c3765b/Cargo.toml#L11 which causes Cargo to execute this script: https://github.com/aldanor/hdf5-rust/blob/15ec644977e0bee2b77340272730b34209c3765b/build.rs which emits flags how to link against HDF5 correctly. That way the Cargo itself doesn't need to know almost anything, it just parses the output of this file. I think we should follow the same approach in fpm. The flags in this case are emitted by: https://github.com/aldanor/hdf5-rust/blob/d4c3737772bec477739c75566a3d52d0a44f27ba/hdf5-sys/src/lib.rs#L65. I think this is when you link against hdf5 rust package in your own code. How to link against hdf5 library itself is done here I think: https://github.com/aldanor/hdf5-rust/blob/d4c3737772bec477739c75566a3d52d0a44f27ba/hdf5-sys/build.rs#L566, it's quite complicated unfortunately.

But it's clean from the Cargo side, it offloads the responsibility to the package itself. We can provide helpers that fpm packages can use to work with things like pkg-config, cmake packages, etc.

In Rust it looks like each package is on their own, so for example this HDF5 package has messy code for each platform, e.g., here: https://github.com/aldanor/hdf5-rust/blob/d4c3737772bec477739c75566a3d52d0a44f27ba/hdf5-sys/build.rs#L492, you can see they are checking brew, or Windows registry, etc.

The good news is that Fortran codes do not need many non-Fortran dependencies, and so doing what Rust does might work for us. What I've seen is that Fortran codes mostly need some of: Lapack, MPI, FFT, MKL, HDF5, JSON, NETCDF, HYPRE, ...

Of which MPI and Lapack being the most important. I think fpm will have support for all Fortran compilers and I think it can have built-in support for MPI and Lapack also. One reason to special case MPI and Lapack is so that one can switch MPI implementaitons and Lapack implementations easily, and not have the Fortran packages hardcoded with a particular implementation.

With those out of the way, the rest can be done Cargo style, at least for now. Most other packages have just one implementation, so Fortran packages can just depend on a particular package (say Arpack, or Scalapack).

[milancurcic]

I agree, this sounds quite reasonable as a first stab. Thank you for the research.

My personal sorted list of most used non-Fortran dependencies: MPI, HDF5, NetCDF, zlib, libpng.

If everybody here listed their most used dependencies, we could have an idea of top candidates which to design for and test first.

[everythingfunctional]

I definitely would like to be able to handle some non-Fortran dependencies. As it's currently designed, I think it won't be that difficult.

The end result of building a library in FPM is just a .a file, and all of the relevant .mod files. So, if your package specifies a build script for that, FPM will just call it. There is a small set of things that FPM would like to dictate to that script though. Those being:

1. The compiler to use
2. The compiler flags to use (mostly to ensure the flags are compatible with the given compiler)
3. Where to put the archive and module files
4. Where to find any of the dependencies

In this way, almost anything could be wrapped into an FPM package. Best practice would be to entirely wrap the package into a Fortran API, so consumers don't necessarily even have to know it's not Fortran, but this may not be strictly necessary in every case.

For the build scripts, I really like Rust's way of having the build scripts be written in Rust too. Not sure if Fortran would really be doable for that, but it would make sure building a package doesn't have additional external requirements. We could special case Makefiles and CMakeLists to use the typical environment variables I think.

[pdebuyl]

C has a special relationship to Fortran though. Thanks to the compatibility section, it makes sense for a Fortran developer to include C code in a Fortan project. Could that be taken into account?

My "pet" usage for this is to write a PRNG in C with a Fortran wrapper module. PRNGs often use unsigned integers whose usage is possible in C.

A fpm package could thus, in this scenario, contain Fortran and C source.

[certik]

@everythingfunctional, @milancurcic and I discussed this point on the phone and we think so far that the best way forward is to work with Conda (or Mamba to be specific) together with their developers (@wolfv and others) to provide all non-Fortran dependencies. fpm would link with mamba and from a user perspective things would just work (users would not need to handle Conda environments explicitly).

Regarding C support, I would suggest initially to handle them via Conda, just to keep things simple.

We can think if if want to later extend fpm to handle not only Fortran compilers but also C and C++ compilers.

[everythingfunctional]

@pdebuyl , my recommendation would be to put the C parts of your project in a separate project so that fpm can easily build the "pure Fortran" part of your project with ease.

We're planning to support make and Cmake as separate build scripts, so supporting linking with (almost) any other language and still having it be an fpm package would be possible.

[LKedward]

@everythingfunctional , @certik , what are the disadvantages/difficulties with natively supporting c sources in fpm? Many projects do need to include c code from time to time and this is quite normal for Fortran (even before iso_c_binding came along). I think this would be a good feature.
(I'm not talking about whole package dependencies, just self-contained projects with mixed c/Fortran code).

[milancurcic]

I don't foresee issues with building C alongside Fortran. It may be even simpler as there are no modules to deal with.

[certik]

Regarding C support: a full solution there is this: mamba-org/mamba#223.
What we could do for fpm is not the full solution, but provide functionality for simple C source files, not full packages with a complicated build system.

We have to decide on a layout for C sources:

• Should the .c files be simply files in src/ folder? (I vote yes.)

• Where should the header files be? Either in src/ or in include/.

• What about dependencies and their header files? (I would suggest Conda and that you can use anything from there, so fpm will create an internal environment and you can use any header file from there.)

• What about linking of dependencies?

• Given that we will support C, why not C++ also? Supporting C++ would be very useful to provide Fortran wrappers: we would write a simple C wrapper that calls into a 3rd party C++ library, and call that C wrapper from Fortran, so it requires compiling of a C++ code.

• Also one needs to link libc properly for C, and libc++ properly for C++. I think we have to do this anyway anytime we depend on a C or C++ library.

There will be more issues. Some of them similar to what we have to deal with anyway for Fortran sources. So I think let's keep this option open when we are designing fpm, although I still suggest to concentrate on Fortran sources first.

[milancurcic]

The low hanging fruit here seems to be handling the mixed C and Fortran source in a single project.

Just let .c and .h sit in src/ by default, use gcc by default (like we do now with gfortran), link all object files into a static library and executable (like we do now).

Linking to external (binary) dependencies seems orthogonal to this. Nevertheless important, but we could tackle these incrementally.

[pdebuyl]

I had in mind the "simple" use case of one or a few c files that indeed "just get compiled" as part of the Fortran module. Making the shared C libraries available to other Fortran "consumer code" is probably out of scope. Anyway, thank you for the replies :-)

[certik]

@pdebuyl thanks for bringing it up, I think we can do that. Looking at my own code here: https://github.com/certik/hfsolver/tree/master/src, I have .f90 files, .c and .h files and also .cpp files all in the src directory. So if fpm could eventually compile all that correctly, that would go a long way. The reason I have the C and C++ files are just to interface 3rd party libraries, typically there is no way around that, as one must write some simpler wrapper that is ready to be called from Fortran using the iso_c_binding interface.

So I like this.

[awvwgk]

The Fortran fpm version does support compiling C code and we can link against native libraries already. I'll close this issue as resolved for now.