Managing compiler flags

[smeskos]

1. Is it possible to add/remove flags and reset back to defaults if needed?
2. Is there an option to display flags (before building)? (eg. fpm show flags)

[certik]

@smeskos what flags do you have in mind? Compiler flags?

[smeskos]

Yes I mean compiler flangs.

[certik]

I don't think it's possible currently, but we need to have it.

Probably fpm.toml should allow to set flags for specific compilers per file.

However, even more important is to extract common flags into general settings that work across compilers.

What flags would you like to set @smeskos ?

[smeskos]

I would like to set -cpp, -Wconversion and -std=f2018. But most importantly to suppress warnings (per case), eg. many times when I am still testing something and is under-development I set -Wno-uninitialized-variables and others as well, the reason is that I don't want to pollute the output with dozens non-relevant warnings.

[LKedward]

+1 for customising compile flags for debug/release.

Also need an option for specifying link flags for linking against external libs like BLAS etc.

[certik]

Ok, so let's take it on a case by case:

1. -cpp should be done in a general manner as discussed in Preprocessor support #78.

2. -std=f2018 should also be done in a general manner (that works with all compilers), probably an fpm.toml setting such as "standard = F2018" or something like that.

3. The -Wconversion should be enabled by default, together with all other common warnings

4. -Wno-uninitialized-variables this is the one where we need to allow to set a specific option per file. It could also be done from a command line.

So we need a general solution, but the most common use cases (such as 1., 2. and 3. above) should be abstracted out and fpm should know how to deal with those.

[jerryd]

I need to include things like this on the compiler command line. Right now this is a blocker for me. Is it possible to set FC to a script to wrap around this:

$(pkgconf --cflags --libs gtk-3-fortran)

As an example:

gfortran -c $(pkgconf --cflags --libs gtk-3-fortran) mystuff.f90

Being able to invoke pkgconf is also important for many other complex libraries.

[urbanjost]

Although it would be preferable for fpm to do this more generically, I ran into similar issues and you can specify something like

build-script = "fpm.mk"

and then put a Makefile in fpm.mk. I had to use that to build a library that contained dozens of C files and where I wanted the output to include a *.a static library instead of *.o files and needed X11 to be loaded in the executables, for example. There are some environment variables defined by fpm(1) as described in the fpm documentation that are helpful in placing the output files in the correct directories, as they go in different places when you use the --release option; for example. M_system and M_draw have fpm.mk files, for example

https://github.com/urbanjost/M_system.git
https://github.com/urbanjost/M_draw.git

Although I apparently did not follow the toml standard in my fpm.toml files (although they work with fpm(1)) which I am hopefully going to correct soon. Maybe not the ideal long-term solution, but works today.

[jerryd]

It seems the simplest way to do this would to have FPM check for an environment variable such as $FFLAGS and just substitute those in or use it to override the default flags. I have not looked at the FPM source code so I 'imagine' this would be straight forward to do

[awvwgk]

Cargo uses profiles to define the compilation arguments: https://doc.rust-lang.org/cargo/reference/profiles.html

We could use a similar scheme to define arguments in fpm and make it extensible for additional compilation profiles (like coverage).

Just as an example how the current release build mode could be expressed:

[profile.release]
fast-math = true
optimization = 3
debug = false
standard = "2008"
lto = false
pic = true
compile-args.gnu = ["-funroll-loops"]  # compiler id required for specifying arguments?

The command line could allow a --profile key to reference a defined profile, with debug or development being the default and release being a special profile selected by the --release argument.

[urbanjost]

Yes. The package ideally has a self-contained definition file so when someone uses it as an external dependency it would build automatically so this seems like a good direction, but harder for Fortran than most other languages to implement.

Using switches on the command line is handy for a developer or someone not planning on distributing their package but is problematic otherwise, especially for anything that is intended to be used automatically as an external dependency by others.

Multi-compiler support becomes difficult unless the keywords are something like a general name like "coarray" or "X11" and there are built-in defaults for how to implement that for each compiler, which quickly gets to be a maintenance problem past a few basic options. Just wondering if the configuration files should be able to be in separate files so they can be shared and reused easily. I would not say that should be mandatory; but an option.

So I am trying to picture when I am using a number of external dependencies and maybe a math library is using coarrays and a graphics package is using X11 libraries and a TUI interface is using ncurses. It seems like external "system" load options should be inherited by the apps by default for loading? It gets complicated because there are potentially many compilers with different options. It would be nice if we had a generic load script and compile script for Fortran that was standardized, but right now if I just had something that let me say for "ifort" add these load options, and for "gfortran", "f90", ... add these it would eliminate my need for a fpm.mk file for several packages, which would be nice.

I could live with the top package having to specify all the options but it would require any external package that depends on "system" libraries to always require manual editing of the main project manifest.

So I think there are still issues but being able to specify options in the manifest file seems the right direction; and being to distinguish between optional and required options might allow "required" options to be inherited by the top package.

Maybe just like you have to add a line in your manifest for a dependency the dependency might be able to specify options as well?

[LKedward]

We could use a similar scheme to define arguments in fpm and make it extensible for additional compilation profiles (like coverage)

Yes, this is very much how I envisaged the problem to be solved. This allows us to support multiple compilers and handle flag compatibility between dependencies but without restricting the general user. I think this is the solution we should pursue for managing compilers and flags. Importantly this will allow us to get something general and flexible working; any issues surrounding particular flags and compilers can then be addressed on a case-by-case basis.

---

In addition to "coverage" the other profile I use often would, perhaps confusingly, be "profile" for performance profiling.

[awvwgk]

An alternative proposal, we already have the build table, so we could put it in here:

[build.config.release]
optimization = 3
debug = false

[build.config.debug-optimized]
optimization = 2
debug = true

[build.config.debug]
optimization = 0
debug = true

[build.config.coverage]
optimization = 0
debug = true

[build.config.profile]
optimization = 2
debug = false

The idea would be to create a table in build containing our compilation profiles. This is the scope I'm envisioning for now:

1. the profiles can have any name, fpm allows to select them at runtime
2. fpm defines release and debug profiles for GCC (this set can be extended in future)
3. any dependency not defining a named profile will inherit the profile from the dependent
4. inheritance of profile options is defined on per option basis (optimization is handled different than fast-math, and so forth)
5. compiler-specific options should be discussed separately (see Compiler support and recognition #223)

I'm also looking for a better synonym of profile; config, option or setting would be possible. Any suggestions or preferences?

[ivan-pi]

Just as an example how the current release build mode could be expressed:

[profile.release]
fast-math = true
optimization = 3
debug = false
standard = "2008"
lto = false
pic = true
compile-args.gnu = ["-funroll-loops"]  # compiler id required for specifying arguments?

I am not sure if the language standard makes sense as part of a profile.

[awvwgk]

I am not sure if the language standard makes sense as part of a profile.

There should be some way to specify it, in most projects I usually set the standard flag for debug builds and leave it out in the release, but specifying it globally in the [build] table would work as well.

[ivan-pi]

To aid the discussion I have tried to summarize some of the available optimization flags. Note that the specific optimizations will slightly differ between compilers. As a single example, to optimize for executable size, gfortran has the flag -Os, however the Intel documentation recommends using -O1.

(perhaps this table can also be moved to a Wiki)

Flag	gfortran	ifort	nagfor	flang	Cray	IBM
-O	equal to -O1	/	equal to -O2	equal to -O2		equal to -O2
-O0	no optimization (default)	no optimization	no optimization	disable optimization	disable optimization	disable optimization
-O1	optimize	optimize for size	minimal quick optimisation	between -O0 and -O2	conservative optimization	no effect, reserved for future use
-O2	optimize even more	maximize speed (default)	normal optimisation (default)	moderate optimization	default optimization	optimize for performance
-O3	optimize yet more	similar to -O2 with additional floating point optimizations	further optimisation	full optimization	aggresive optimisation	additional optimization (similar to gfortran -Ofast)
-O4	/	/	maximal optimisation	reserved for future use	/	aggresive optimization
-O5	/	/	/	/	/	equal to -O4 with additional interprocedure analysis
-Ofast	similar to -O3 with unsafe math	/	/	similar to -O3 with further math optimizations	/	/
-fast	/	maximize speed across entire program	/	/	/	/
-Os	optimize for size	/	/	like -O2 but reduces code size	/	/
-Oz	/	/	/	like -Os, but reduces code size further	/	/
-Og	turns on optimizations which do not interfere with debugging	/	/	/	/	/

Since we are limited by the available compilers for CI, I would suggest to only handle the -O0, -O1, -O2, and -O3 flags for now, which are common for most Fortran compilers.

[awvwgk]

With #322 the package scope of the source files is now available in the fpm model, extending each package object in the fpm model by its compilation profile would allow to implement customized arguments on per package basis.

The main issue to solve is the presentation of the build arguments in the package manifest.

I would suggest to use a table of tables either in build or in the top-level:

[build.profile.*]  # with * = debug, release, ...

The command line argument could be --profile <debug|release|...> to access the compilation profile.

We should abstract as many options as possible in specific entries in a profile (optimization, fast-math, ...), but we won't get around implementing compiler vendor specific arguments. Again I would suggest to represent them with a table of tables:

[build.profile.*]  # with * = debug, release, ...
[build.profile.*.compile-args]  # link-args, global-args ?
gnu = ["...", "..."]
intel = ["...", "..."]  # Intel with GCC like CLI on Unix
intel-cl = ["...", "..."]  # Intel with MSVS like CLI on Windows
flang = ["...", "..."]
# or inverted?
[build.profile.*.gnu]
compile-args = ["...", "..."]
link-args = ["...", "..."]
global-args = ["...", "..."]

The allowed IDs should preferably match existing IDs used in CMake or meson. Internally, all abstracted options could be implemented using the same representation as the compile-args table.

We will require three kinds of arguments, compile arguments for the current project (e.g. -O3), link arguments for the current project (e.g. -flto) and all dependent projects and global compile arguments for all dependent projects (e.g. -fopenmp, -fast-math). We can than fine-tune the exact scopes of the arguments at a later stage.

Implementing the three different kind of arguments allows us to prototype profiles for the different compilers without having to modify fpm. Options and profiles we choose to stabilize in fpm can get specific entries in the profile or build tables later.

[Sideboard]

Yes I mean compiler flangs.

I was confused by the title of this issue. Could it be made more specific?