Compile archive using pre-existing Makefile

[MarkWieczorek]

I have a somewhat complicated archive that is compiled with a pre-existing Makefile, and I would like to make this more accessible to those who might want to use it as a fpm dependency.

Based on the documentation, I thought that I would just need to specify the name of the Makefile in the fpm.toml file, but this obviously didn't work.

I think that it would be very useful for fpm to have have the option of simply executing a pre-existing makefile, and then placing the compiled .mod and .a files wherever it is that they are needed. Ideally, this would be specified in the fpm.toml file something like this:

[library]
makefile = "make all F95=$(FC) DIR=$(BUILD_DIR)"

Alternatively, given that the .mod and .a files are initially found in the src directory, instead of having the makefile manually move them to $(BUILD_DIR), this could be done by fpm itself after the makefile successfully terminates.

[everythingfunctional]

We certainly want to try and make migrating existing projects to fpm as easy as possible. The questions that need to be answered to solve this problem are:

• How does fpm figure out how to call the makefile? You're example shows something that might be workable
• How does fpm figure out where the makefile put the stuff it needs to copy? This one's a bit harder

On the other hand, you could write a script that manually runs the makefile and does the appropriate copy commands. It's a tad less portable, but a pretty straightforward workaround.

[MarkWieczorek]

On the other hand, you could write a script that manually runs the makefile and does the appropriate copy commands.

Is it possible for fpm to run a script now (without compiling files) ? That would probably work for me.

[everythingfunctional]

@MarkWieczorek, yes. Check out the details here. Let us know if anything is unclear or you get stuck.

[MarkWieczorek]

I am starting to make some progress.

First, it turns out that if you have your own Makefile from a pre-existing project

[library]
source-dir="src"
build-script = "make all"

and

[library]
source-dir="src"
build-script = "Makefile"

do not do the same thing. The first example actually does what I want (i.e., just do a make all in shell) but the second seems to try compile my source files using fpm.

Second, if you exclude the src-dir line, the following doesn't work as expected:

[library]
build-script = "make all"

This suprises me, because fpm doesn't need to know where my source files are, given that everything is to be compiled by my pre-existing makefile.

I think that part of the solution will be to refactor the documentation, and describe what is actually happening with the above commands. It would be very useful to have a section in the documentation describing how to port a pre-existing project to fpm.

[certik]

We should have tests for all of the above, then it will at least be clear what is supposed to work.

[MarkWieczorek]

One final thing:

Everything works when using

name = "project"
[library]
source-dir="src"
build-script=make all F95=$FC

However, fpm build ends with the error/warning

make: *** No rule to make target `/path/build/gfortran_debug/project/libproject.a'.  Stop.

This is because (as stated in the docs) "Additionally, script will be called with the name of the archive (*.a file) that should be produced as the command line argument."

Is there a way to disable this behavior?

[MarkWieczorek]

I've encountered one final problem, which unfortunately is the most important for me.

First, I can successfully compile my project locally using a pre-existing makefile, as described above. The makefile moves all the .mod and .a files to BUILD_DIR, which is located in the main directory at

gfortran_debub/project/

However, if I try to use my project as a dependency (downloaded from github) in another project, the .mod and .a files are located at

build/dependencies/project/build/gfortran_debug/project/

and the following directory is empty:

build/gfortran_debug/prioject/

When building the code that makes use of the dependencies, fpm can no longer find where the dependency .a and .mod files are located.

Does anyone have any ideas on how to solve this problem? Obviously the files from build/dependencies/project/build/gfortran_debug/project/ need to be copied to build/gfortran_debug/prioject/, but I don't see how to do this.

[everythingfunctional]

In your case, I'd recommend using a wrapper script to make doing things properly a bit easier. Something like

#!/bin/bash

expected_archive=$1

make all F95=$FC
cp where/your/*.mod $BUILD_DIR
cp where/your/archive.a $expected_archive

because, as you've noticed, the build directory will be different when included as a dependency. If you can, you should try and make use of the FFLAGS environment variable as well, so in the future, projects using yours can try out different compiler flags.

[MarkWieczorek]

That might work, but what would I use as the argument to the script ($1) ?

[everythingfunctional]

fpm calls that script with the appropriate argument. So assuming your script is called build_script.sh, your fpm.toml would look like

...
[library]
source-dir = "src" # presumably
build_script = "build_script.sh"
...

and fpm will call your script (effectively) like

FC=gfortran FFLAGS="-some -flags ..." BUILD_DIR="wherever/fpm/decides" INCLUDE_DIRS="build/thing1 build/package2 ..." build_script.sh some/where/libpackage.a

[certik]

@everythingfunctional why not pass everything as environment variables?

[everythingfunctional]

I'm not sure I have a thoroughly compelling answer, but my thinking is along the lines of the following:

• We should conform to common practices in existing build systems
• Common build commands (or at least the ones I'm used to) are of the form build_script what_I_would_like_built
• A common practice for overriding build parameters is via environment variables

I'd agree it's not the most elegant and consistent design, but if our goal is to make migrating to fpm easier, conforming to existing practices is probably the way to go.

[certik]

I would suggest to follow Cargo's approach, and not invent our own conventions.

…

On Tue, Jul 14, 2020, at 5:05 PM, Brad Richardson wrote: I'm not sure I have a thoroughly compelling answer, but my thinking is along the lines of the following: * We should conform to common practices in existing build systems * Common build commands (or at least the ones I'm used to) are of the form `build_script what_I_would_like_built` * A common practice for overriding build parameters is via environment variables I'd agree it's not the most elegant and consistent design, but if our goal is to make migrating to fpm easier, conforming to existing practices is probably the way to go. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#118 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFAWEBAUPZFHBQJ6YSGPTR3TQD7ANCNFSM4OW6ADFA>.

[MarkWieczorek]

some/where/libpackage.a

And what about the .mod files?

[everythingfunctional]

The mod files need to go in the same place.

[everythingfunctional]

I would suggest to follow Cargo's approach, and not invent our own conventions.
…
On Tue, Jul 14, 2020, at 5:05 PM, Brad Richardson wrote: I'm not sure I have a thoroughly compelling answer, but my thinking is along the lines of the following: * We should conform to common practices in existing build systems * Common build commands (or at least the ones I'm used to) are of the form build_script what_I_would_like_built * A common practice for overriding build parameters is via environment variables I'd agree it's not the most elegant and consistent design, but if our goal is to make migrating to fpm easier, conforming to existing practices is probably the way to go. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#118 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAFAWEBAUPZFHBQJ6YSGPTR3TQD7ANCNFSM4OW6ADFA.

I think Cargo's approach is really good, and we should strive for it. In the mean time, we don't yet support using Fortran as a build script, and their approach doesn't support any other custom (or existing) build scripts, like we would (maybe) like to support.

[certik]

Cargo requires to write Rust code for the script. We should allow other scripts such as Bash or Makefile, as we discussed. But they should be treated exactly the same as the (future) Fortran script (if we decide to allow that, or just require Bash or Makefile).

The API is described here:

https://doc.rust-lang.org/cargo/reference/build-scripts.html

The script is run as is (with no arguments) and everything is passed using environment variables:

https://doc.rust-lang.org/cargo/reference/build-scripts.html#inputs-to-the-build-script

And the outputs are communicated by printing to stdout using the "cargo:..." encoding:

https://doc.rust-lang.org/cargo/reference/build-scripts.html#outputs-of-the-build-script

Can we do the same for fpm?

[everythingfunctional]

I think we should move towards Cargo's API design. Then the communication mechanism can be the same for any build script; inputs as environment variables, outputs as prefixed lines on stdout.

There is a subtle distinction between Bash, Makefile, and Fortran build scripts though that they aren't treated exactly the same. Fortran must be compiled first, potentially with some dependencies if we're following Cargo's design. Bash scripts are executed directly, and Makefiles must be executed with make.

I also don't know that this design will make it any easier to transition existing projects to fpm. There is virtually no chance that an existing build system will "just work" with this design, but that chance was probably pretty small with my design anyway.

[certik]

Perfect, thanks. Yes, I agree it won't make it easier for other projects to port, but by using the same design as Rust, at least they don't have to update their build scripts once they port (currently they will have to update the makefile / bash script after we change the API).

Using Fortran as a script sound weird at first, but make sense from a multiplatform perspective, as it would run natively on Windows and other platforms, while Bash typically does not run natively, but requires a linux subsystem on Windows. I think that's why Cargo chose Rust as the script.

[MarkWieczorek]

I think that there is a very simple solution for projects that compile with pre-existing makefiles: We just need to define two environment variables.

1. BUILD_DIR is already defined, and tells where to put the .mod and .a files within the original project. This would either be in the directory build/gfortran_debug/project if you were simply building the project by itself, or in build/dependencies/gfortran_debug/project if you were installing it as a dependency.
2. INSTALL_DIR, which is where the contents of BUILD_DIR get copied when the project is installed as a dependency. This corresponds to BUILD_DIR/../../gfortran_debug/project.

In practice, the makefile would compile all the .mod and .a files in BUILD_DIR, and only if INSTALL_DIR is defined would they then get copied to INSTALL_DIR.

[certik]

Something like that. Here is the list of environment variables that Cargo defines:

https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-build-scripts

note that most are prefixed with CARGO_, but some are not. I suspect the ones that are not are due to historical reasons. I very strongly suggest we prefix all our environment variables by FPM_, so it would be FPM_BUILD_DIR and FPM_INSTALL_DIR. The reason is that it is very easy to make complex 3rd party build systems fail if you define an environment variable with a common name like BUILD_DIR or INSTALL_DIR, because the customized 3rd party build system can easily do something different if this variable is defined. By prefixing all variables, we ensure that our environment variables do not clash with user defined variables. It's a good habit to do that, not to pollute the environment namespace.

[everythingfunctional]

A quick note, BUILD_DIR is always build/<compiler>_<debug_or_release>/project, whether it's building your project standalone or as a dependency. So when building your project as a dependency, the BUILD_DIR is not within your project's directory. Thus, no need for the INSTALL_DIR variable.

[MarkWieczorek]

Thanks for all the help: I turns out that I was misinterpreting how BUILD_DIR was being set for stand-alone projects and dependencies. (I also made a dumb choice to hardcode the variable build/gfortran_debug/myproject in the makefile for the standalone project). Using

build-script = "make all F95=$FC LIBPATH=$BUILD_DIR MODPATH=$BUILD_DIR"

now works for both cases :) The only thing I need to do to make this work is to be able to link to system-wide libraries.

One final question: how do I change debug to release ?

[jvdp1]

Could the Makefile of stdlib be already used to compile stdlib with fpm?

[zoziha]

We certainly want to try and make migrating existing projects to fpm as easy as possible. The questions that need to be answered to solve this problem are:

• How does fpm figure out how to call the makefile? You're example shows something that might be workable
• How does fpm figure out where the makefile put the stuff it needs to copy? This one's a bit harder

On the other hand, you could write a script that manually runs the makefile and does the appropriate copy commands. It's a tad less portable, but a pretty straightforward workaround.

I have a good idea, it should be able to use different make tools, such as make, cmake.

1. I think the make tool generally only displays simple commands, such as make build and cmake build. Obviously, these are commands and we need to deal with them. They are more like coherent tasks besides static commands.
2. Use the binary files generated by the make tool, such as link libraries and intermediate .obj files. Only the make tool and developers know where they are generated.

Based on the above analysis, we can add items like [make] to fpm.toml:

[package]
name = "fpm-make-test"

[make]
[make.tasks.src]
description = "Generate src_dir objs."
command = "make"
args = ["-f", "makefile", "--directory=src"]
kind = "objs"     # shared/static/objs/binary
objs_dir = ["./build/objs/src1/", 
            "./build/objs/src2/",
            ...
]

[make.tasks.others]
...

We use fpm's absolute control over fpm.toml to control the commands of the make tool and the path of the generated binary file. fpm selects the behavior of fpm by extracting the path of the generated binary file.

We leave this [make] to the developer to consider. fpm just sends make commands and accepts task results.
My inspiration comes from:

1. https://medium.com/@sagiegurari/automating-your-rust-workflows-with-cargo-make-part-1-of-5-introduction-and-basics-b19ced7e7057
2. https://github.com/sagiegurari/cargo-make