RFC: Scheme for detecting incomplete package registration

[hvr]

I just had a report from @osa1 where he stumbled into a situation where cabal new-build would run into an inconsistent state such as

$ cabal new-build
Resolving dependencies...
Build profile: -w ghc-8.4.3 -O1
In order, the following will be built (use -v for more details):
 - hpack-0.29.7 (lib) (requires build)
 - stack2cabal-0.1.0.0 (exe:stack2cabal) (first run)
Configuring hpack-0.29.7 (lib)...

Failed to build hpack-0.29.7. The failure occurred during the configure step.
Build log (
/home/omer/.cabal/logs/ghc-8.4.3/hpack-0.29.7-44709858b7d98be4bb71fd9885062ad1294fed050a9bd65cf4bd5b7c63b1ebb9.log
):
Configuring library for hpack-0.29.7..
cabal: The following package dependencies were requested
--dependency='bifunctors=bifunctors-5.5.3-f2f7cceb46cfa237e05d5a645721cd80320bd50a4fff4c72b3fe88c32e1f68f6'
however the given installed package instance does not exist.

cabal: Failed to build hpack-0.29.7 (which is required by exe:stack2cabal from
stack2cabal-0.1.0.0). See the build log above for details.

This looks similar to #5379 but has a different cause:

cabal successfully created the

~/.cabal/store/ghc-8.4.3/bifunctors-5.5.3-f2f7cceb46cfa237e05d5a645721cd80320bd50a4fff4c72b3fe88c32e1f68f6

folder but failed to register it as it was likely terminated externally.

However, it's difficult to efficiently detect such cases as querying the pkg-db has significant latencies involved once the db reaches a certain size.

However, we could keep track of the registration transaction by either creating or removing an additional file in the filesystem.

We could e.g. create a temporary file

~/.cabal/store/ghc-8.4.3/bifunctors-5.5.3-f2f7cceb46cfa237e05d5a645721cd80320bd50a4fff4c72b3fe88c32e1f68f6.wip

which we could additionally lock on (via fcntl or flock), and which would be removed as soon as the pkg-db registration was completed succesfully.

This way, we can see right-away when scanning the ~/.cabal/store/ghc-8.4.3/ which unit-ids are present (NB: exe-only units don't have a pkg-db registry entry!) and which are still in-flight;

when we detect in-flight pkg-db entries we need for the current install-plan, we can abort with an error, and offer directions for the user on how to fix the situation; we may even provide a cabal fsck-store command for this.

[fgaz]

I wonder if there's a way to do it automatically without asking the user to manually check that there are no cabal instances running and execute the store-fixing command