Implement fpm-install command

[awvwgk]

Noticed I require some working install command, when working on the fpm-dist command. Therefore, I coded up an installer type and implemented the fpm-install command.

• allow installing of executables
• optionally install library and modules if specified in manifest
• add install table to manifest reference
• use in CI to install fpm
• add unit tests for new manifest entry
• add unit tests for new installer type

Related #71

Command-line interface:

NAME
 fpm-install(1) - install fpm projects

SYNOPSIS
 fpm install [--release] [--no-rebuild] [--prefix DIR]
             [--bindir DIR] [--libdir DIR] [--includedir DIR]
             [--verbose]

DESCRIPTION
 Subcommand to install fpm projects.

OPTIONS
 --release         selects the optimized build instead of the debug build
 --no-rebuild      do not rebuild project before installation
 --prefix DIR      path to installation directory (requires write access)
 --bindir DIR      subdirectory to place executables in
 --libdir DIR      subdirectory to place libraries and archives in
 --includedir DIR  subdirectory to place headers and module files in
 --verbose         print more information

Installation configuration:

[install]
# whether or not to install library + modules (default: false)
library = false

[urbanjost]

In the help text I like the new word "archies" but I think "archives" is probably better (It's old meaning for the ackkk-ackkk sound of anti-aircraft gunnery is pretty much obsolete, so I think it might catch on though) :>

[urbanjost]

It would be useful long-term to be able to set access permits, ownership and group but given there is not a platform-independent way to do that with pure Fortran it is problematic, although easy to do on POSIX platforms.

Do you intend it to be able to create the specified directories if needed or must all target directories pre-exisit?

[urbanjost]

Should the default be to not rebuild and a --build switch be available instead of the opposite? I think that would be a more expected behavior. Althought the risk is low I would like to install an executable I have run tests on first, typically.

[urbanjost]

The help_install should be added to the list of help produced when you ask for the entire manual, and the install subcommand should be added to the list of available subcommands in several places so it shows in the output for list and the list of commands in the fpm_help. So the output from fpm --list|fpm list fpm --help|fpm help and fpm help manual should mention the install command, not just help install. I like the function you added so the strings can be unallocated, but when I called sget() directly like lget() to set the types describing the command types I hit a gfortran bug when I built with optimization where I got segfaults. I have not pulled and built this yet, just looked over the code but might want to try that. That is why I set variables and passed those for the most part when I used sget() (in a few places I was actually testing the string several times and preferred to do that anyway).

[awvwgk]

It would be useful long-term to be able to set access permits, ownership and group but given there is not a platform-independent way to do that with pure Fortran it is problematic, although easy to do on POSIX platforms.

I haven't added file permissions yet, because I wasn't sure how to do it on non-POSIX platforms.

Do you intend it to be able to create the specified directories if needed or must all target directories pre-exisit?

Actually, missing directories are automatically created.

Should the default be to not rebuild and a --build switch be available instead of the opposite? I think that would be a more expected behavior. Althought the risk is low I would like to install an executable I have run tests on first, typically.

The CMake install behaviour is exactly this, you have to invoke a separate CMake command to actually build before installing, which I find annoying. In meson install you get automatically a rebuild if required, with an incremental rebuild like in #248 we can always rebuild without much additional cost.

I'm mainly orienting my strategy on the meson commands because it is in fact my main build system and I think it is well thought out.

The help_install should be added to the list of help produced when you ask for the entire manual

Will add it there.

[urbanjost]

I want to see install implemented and this is great; in the mean time I have been using the GNU/Linux install(1) command and the --runner option; but would prefer it integrated as you are doing here. Personally I use platforms with POSIX interfaces (on MSWindows either Cygwin or the Linux subshell) and permissions are relatively straight forward there. Users can verify the permits outside of fpm(1) but other than adding platform-specific commands or having conditional builds it is vexing. I am not sure if stdlib is near releasing system features like this. Should we be building a POSIX look-alike for each routine as we need it for now ( a topic for discussion outside of here, I suppose)? There are a lot of things like CHMOD and CHDIR functionality that would make fpm so much nicer. I actually use a shell wrapper with fpm that finds and moves to a directory with the fpm.toml file before executing because I find that so irritating that I have to move to that directory first; which would be trivially resolved with a CHDIR; but I digress. Looks good but long-term it is likely permissions will be issue; but perhaps for now letting the user handle that is more reasonable than a kludge? Since you are doing a system-dependent copy command though; perhaps adding a chmod after the copy is good for now? I always prefer execute-only for everyone other than the owner for a binary executable myself. I was comparing this to the command install(1) (which has its detractors by the way) and it defaults to backing up existing executables. Personally I do not need that, but install(1) has been around a long time so I thought I would mention it. The cp(1) and chmod(1) are essentially guaranteed on any GNU or Unix platform, I do not think install(1) itself is common enough to use it instead of cp(1) directly(?).

[LKedward]

Thanks @awvwgk, implementation looks great and it appears to work nicely. Left a few minor comments. 👍

It would also be good to add fpm install to the general fpm --help/fpm help/fpm --list outputs.

[everythingfunctional]

At first glance I like it and I've been wanting something like it for a while now. A couple requests/suggestions though.

• I would prefer the default prefix be $HOME/.local. Many package managers are starting to default to this, and I like that it doesn't require sudo or administrative privileges.
• I would like a way to specify that only executable(s) be installed, and a way to specify only a specific executable. I.e. --target my_executable in symmetry with run and test.

[awvwgk]

I would prefer the default prefix be $HOME/.local. Many package managers are starting to default to this, and I like that it doesn't require sudo or administrative privileges.

Sounds good, what is the preferred prefix for Windows? I usually don't use the default prefix and always set my own to cope with the various software from different sources on my machines.

I would like a way to specify that only executable(s) be installed, and a way to specify only a specific executable. I.e. --target my_executable in symmetry with run and test.

Not sure about this, my expectation on the install command is to get complete project exported to another prefix. What is included in the installation should be the decision of the project maintainer rather than the user or packager. It is hard to tell as user or packager which parts of a project can be installed independently. If a project provides executables that can be installed independently, we should rather encourage splitting the project.

[everythingfunctional]

my expectation on the install command is to get complete project exported to another prefix

If you start installing libraries from different projects, you start to reintroduce the problem sandboxing them into the project for building solved in the first place. Right now we are statically linking (at least the Haskell version is). I know it's "inefficient" because you might end up with executables with the same copy of some library, but I think it's far more common that you'll have executables that need different versions of some library. At least that's the experience other languages have had.

With fpm, I no longer need to install libraries or worry about coming up with some common version that satisfies my needs for every project I'm working on. Given that, I don't know why I would ever install a library. I understand that for some people old habits die hard, and there are use cases where you would want to install a library, so I'm fine with having a way to do it, but I want a way not to do it.

[awvwgk]

Maybe we are talking about a different problems here.

If you start installing libraries from different projects, you start to reintroduce the problem sandboxing them into the project for building solved in the first place. Right now we are statically linking (at least the Haskell version is). I know it's "inefficient" because you might end up with executables with the same copy of some library, but I think it's far more common that you'll have executables that need different versions of some library. At least that's the experience other languages have had.

I don't disagree with this point, for executables this works completely fine. As somebody packaging I would like to see a way to opt-out of the static linking, of course 😉. I see the conflict here and don't want to impose this on fpm, because it is out of scope for the project, however I will try to implement a solution with fpm-dist which suits the needs for packaging instead.

With fpm, I no longer need to install libraries or worry about coming up with some common version that satisfies my needs for every project I'm working on. Given that, I don't know why I would ever install a library. I understand that for some people old habits die hard, and there are use cases where you would want to install a library, so I'm fine with having a way to do it, but I want a way not to do it.

But for this point I disagree. There are several scenarios were I want to install a library. Use cases are

1. library-only projects, which do not provide executables by themselves (limited due to ABI issues)
2. exporting non-module Fortran APIs or C APIs
3. dynamically loading symbols from a shared library, e.g. interfacing a C API from Python

Since this cases are the exception, a project has to opt-in on installing the library. Projects that want to install a library usually will do this for a good reason.

[LKedward]

It seems like the default behaviour (of only installing executables) addresses your concern @everythingfunctional — are you happy for this PR move forward?

[urbanjost]

The help text should indicate the default subdirectory names lib,bin,include and indicate what gets installed by default and show a few examples. Since there is not (yet?) any category for executables except test and app currently I often have dozens of demo and performance tests that I do not want installed but are currently "apps". Allowing for a list of names would help; having some way in the toml file to designated demos might be another approach; it is not clear to the user if they say to install executables will they just get executables from app/ or also from test/ and/or other directories specified in the fpm.toml file. I had to look at the code and/or empirically test to detrmine what happens with a more complex project. That needs described to the user.

[awvwgk]

I have to rebase this branch and incorporate the --compiler flag here as well first.

[awvwgk]

Thanks for you patience on this PR. I rebased against the latest changes to allow checking how the fpm-install command works together with the example programs and the new fpm-update feature.

To address the comments above, by default all executables from the projects are installed, examples and tests are not considered for installation. If a project opts-in to installing a library, the archive and the module files are installed as well.

[awvwgk]

The only not addressed issue in the PR are the file permissions on POSIX systems together with the install(1) command. Since this PR is meant to create the foundation to implement any kind of installer, I would like to explore file permissions with a separate PR as this one is already quite large.

Unless there is anything blocking this PR I will merge within the next week.

[milancurcic]

Thank you for this implementation. I just played with it now and it works. I have three suggestions:

1. Make library = true under [install] in the manifest as the default. I don't have a good rationale for this other than that I expect packages to be installable by default. What do others think? Sorry if there has already been a discussion on this--I didn't see it.
2. If a package is not installable, i.e. if library = false is set in the manifest, then fpm install should emit some helpful message, for example:
3. fpm-install as the name in the help message is a tad confusing because there's no such executable as fpm-install. Perhaps a man page is not a suitable format for these CLI help messages. But I'm not convinced either way.

Nothing to install ([install] library is set to false in fpm.toml)

or something similar. Currently fpm install doesn't emit any output which usually implies non-verbose success.

[awvwgk]

Thanks for the comments, I included a check if the project is actually installable.

1. Make library = true under [install] in the manifest as the default. I don't have a good rationale for this other than that I expect packages to be installable by default. What do others think? Sorry if there has already been a discussion on this--I didn't see it.

The preference seems to be an executable-only focus for now, to allow library projects the opt-in install.library is a good compromise in my opinion. We can revisit this once we flesh out the installer command in further PRs.

3. fpm-install as the name in the help message is a tad confusing because there's no such executable as fpm-install. Perhaps a man page is not a suitable format for these CLI help messages. But I'm not convinced either way.

Since our help pages are currently catman pages, I try to follow the conventions for man pages here. It seems to be a convention for man pages of subcommands like git add to be stored as git-add(1). Usually the command man git add is identical to man git-add, possibly to avoid whitespace in filenames. Using install(1) as name is not possible, because there is actually a program install(1) available on most Unix systems. That's my reason for choosing fpm-install(1).

[milancurcic]

Sounds good, +1 from me.

[urbanjost]

Name conficts with man pages were designed for their suffix to be a category, as there are often conflicts. Think of test, for example.
The other packages seem to be taking a short-cut. If the manpages are named topic.1fpm.gz instead of fpm-topic.1.gz then entering

man 1fpm topic 
man topic.1fpm

will go directly to it. And you can list all the related man pages . So if all the man pages have the fpm suffix you can find them.
There are lots of other syntaxes that work too, as man(1) has a long history. Maybe because people did not like using the number
people started adding one-line manpages named fpm-topic.1.gz that where just an ".so" of the topic.1fpm files. Now it looks like several packages just put subcommands in fpm-topic.1.gz files directly as you noted.

In the past the solution was often to add a subcommand to your program like man that would do something like set MANSECT to fpm and call the man(1) command or copy flat versions of the man pages (usually generated by man topic|col -b` if the man command was not available. So man(1) has lots of ways to handle duplicate names (which there are many of) but they are not commonly known and it looks like the program-subcommand syntax is used by at least git and cargo (on my machine those where the ony ones with actual man pages named program-topic).

So there other ways; and perhaps just using topic.1fpm.gz files and adding a "man" subcommand or making fpm help topic smarter and if a whereis 1fpm topic returns a pathname it assumes man(1) is on the system and displays the help with man,
else displays flat text. My impression is there are a good number of people interested in using fpm on non-open systems like MSWindows where man(1) is not available.

Anyway, since I think those questions are unanswered that just writing it up as "install" is less confusing; albeit the git and cargo
examples show this has become a popular solution that requires less intimacy with the man(1) command. But if we are sure that
the documentation will be deployed with man(1) where available (I am good with that) I think we need consistency; so all pages should described themselves with a command name or all should use fpm-topic. Mixing them is confusing.

So, for example, if I enter

git help rebase

It actually calls man(1). Not sure how many users do the man of the git commands directly versus git help ... . Not sure what git does no platforms without man(1). But unless we expect the man pages to be used directly a lot outside of fpm we do not have to use the fpm-topic format. Note when I use the git help I put in the topic "rebase" not "git-rebase".

Anyway, I think we should expect that the main way to get on-line help will be "fpm help topic". Whether that calls man for formatting or a web browser or displays flat text or info files is then hidden from a new user. If that is true, there is no technical reason to follow the git(1) or cargo(1) convention other than how we want someone to see the pages when using man(1) directly.

[urbanjost]

looking around, there seem to be about eight approaches. Note that openssl does both but using links instead of ".so" files:

8 -rw-r--r--. 1 root root 5126 Mar  5  2020 genpkey.1ssl.gz
0 lrwxrwxrwx. 1 root root   15 Mar  5  2020 openssl-genpkey.1ssl.gz -> genpkey.1ssl.gz

So they make a topic.1ssl.gz file and then link openssl-genpkey.1ssl.gz to it so

man openssl-genpkey
man 1ssl genpkey
man genpkey.1ssl
man -a genpkey

all work

[awvwgk]

@urbanjost Thanks for the explanation, my knowledge about man is limited to the content of man man and related documents. While I found the general concept of man page topics helpful and started to use it initially quite extensively in my documentation, I got quickly tired of having to explain man pages and man topics to my co-developers and users and therefore eventually dropped it again. I still see the beauty in man topics, but it seemed like a lost fight to force it upon people.

[urbanjost]

A user developing a library or module (ie. does not even have an app/ directory) that they want installed for usage outside of fpm
in a conventional mode would have to know too much about fpm and TOML and the options allowed in fpm manifest files

   Libraries and module files are only installed for projects requiring the
   installation of those components in the package manifest.

To know they have to edit their fpm.toml file and add
[install]
library=true
so I agree with @milancurcic that either the behavior or help text needs to change. The original fpm only created a src/ directory.

I think a library install should be trigged if you specify the --libdir OR --includedir switch regardless of the setting. As it is if I do

   f pm new A --src
   # change A/src/A.f90 till it does what I want
   fpm install --libdir [anything]

install will do nothing and produce no message, which is very unintuitive. At a minimum the help text could say something like

DESCRIPTION
 Subcommand to install fpm projects. This command can install components
 of the project into external directories. It will install nothing but
 executables no matter what options are supplied unless the "fpm.toml" 
 manifest file includes the equivalent of the entries

    [include]
    library = true

 By default it will install all executables (tests and examples
 are excluded).  If "library=true" it will also install all libraries and
 module files which are part of the project (not files from dependencies).

After trying it with more complex projects I would like it to

• use the --list switch (to be most consistent with the other commands)
  or something like --dryrun to show what it would do without doing it
• allow for a list of executable names to install (which I think others mentioned)

Although theoretically if all your components were in fpm installing libraries and modules would not be needed,
but initially I suspect the opposite is one of the bigger user cases. Use fpm to create a library and export it for conventional use.

External files are going to be around for a while. Users will want to make not just applications but libraries available to non-fpm users; and people will develop dynamic libraries that need to be in user load paths.

[LKedward]

It looks like there is some mixed opinion on the installation of libraries.

I disagree that most users will want to install their library system-wide and I do not believe that we should be encouraging users to do so. Unlike the c programming language, Fortran does not have a standard ABI and hence it does not make sense for users to install their libraries system wide since the compiled library is specific to the compiler vendor and version. Moreover, there is no agreed-upon standard for where to store the necessary module files, which are similarly binary incompatible between compilers and compiler versions.

As @awvwgk pointed out, there are valid exceptions to this – primarily exporting non-module (legacy) Fortran or c APIs – however most users should be using Fortran modules and, in the case of third-party build scripts, should be encouraged to be compatible as an fpm dependency by placing outputs in the correct location.

As such I believe the behaviour currently implemented in this PR (only executables are installed by default) is strongly preferred.

[everythingfunctional]

I'm with @LKedward on this one. There are too many issues associated with installing libraries and so we don't really want to encourage users to do it, so it shouldn't be the default. Not the least of which is that if it doesn't install the dependencies too, then the installed library still isn't usable. I think (maybe hope) that this will very quickly become the vast majority of libraries that have this problem. I think it's fine to have a way to install them, because there are times you really want to do it, but it shouldn't be the default.

Other than that, I'm all good for this to go forward.

[milancurcic]

With all the good arguments, I now agree that not installing the library is a saner default.

[awvwgk]

@urbanjost I added the --list functionality to the install command to show the installable targets in a project. Also the install command will now produce an error if there are no installable targets available. The install.library has been documented in the manifest reference as well.

The installation of libraries is a tricky thing, we currently have no resolution of the dependencies at all, but only one library target in the fpm_model_t available. This means the installed library will contain the root project and parts of the dependencies as well. To have a meaningful way to install and export library targets for external use or other build systems more work has to be done, like writing a pkg-config file or creating a CMake package file.

[urbanjost]

I also do not want installing libraries to be the default unless I specify the --libdir and/or --libinclude pathnames; but the options exist
for libraries and include files so I think the documentation should be very clear they are no-ops and specically describe these kind of limiitations and how to override them, or the options should be removed. I have not tried the changes yet but just having a message should help a lot.

[awvwgk]

I also do not want installing libraries to be the default unless I specify the --libdir and/or --libinclude pathnames; but the options exist
for libraries and include files so I think the documentation should be very clear they are no-ops and specically describe these kind of limiitations and how to override them, or the options should be removed. I have not tried the changes yet but just having a message should help a lot.

Thanks for the clarification, I considered --libdir, --includedir and --bindir no-ops by convention. At least this is the behaviour I observe across the different build tools I use. I can clarify this in the help page as well.

[awvwgk]

Thanks everyone for the review and discussions. I'll go ahead and merge later today.