Add Shakefile as possible build script #991

fendor · 2018-12-11T17:03:12Z

Closes #650
I implemented the Shakefile to be run directly from stack.

# one-stop-shop build everything
stack Shakefile.hs build-all

Features:

stack is the only build requirement
shake is used instead of make
should work on all platforms (not fully tested yet)
dist target works on windows out of the box, making it easier to create releases for windows
submodules are synced / updated at the start -> works after clean clone of project

Quirks:

shake.yaml is necessary to start the stack Shakefile.hs ... command before submodules are loaded.
On first execution, a compiler is downloaded. However, no compiler is downloaded more than once.

Tested on:

Working on NixOS with the following commands:

nix-shell -p "haskellPackages.ghcWithPackages (pkgs: with pkgs; [zlib tar shake])" zlib
export PATH=$NIX_USER_PROFILE_DIR/profile/bin:$PATH && ghc -O2 ./Shakefile.hs && ./Shakefile build

However, the ghc versions 8.2.1 and 8.4.2 are not easily accessible and must be removed from the list of hieVersions in Shakefile.hs in order to make it work.

Anrock · 2018-12-12T08:34:35Z

I've noticed there is a downside in requirement to download 8.6.2 to run shake script despite me only wanting to check help and build 8.4.4. Not sure if it's fixable.

lorenzo · 2018-12-12T10:14:34Z

Yeah, having to download the compiler and dependencies to actually install something else is annoying.

Anrock · 2018-12-12T11:16:59Z

I wonder if it's possible to get "whatever ghc is already available" from stack...

lorenzo · 2018-12-12T12:21:47Z

Or at least the ghc I'm intending to use anyway

samuelpilz · 2018-12-12T14:17:58Z

@lorenzo @Anrock It is intended that the same resolver as in the normal stack.yaml is used. Therefore, nothing extra is downloaded.
The initial download of the resolver and the ghc is necessary to eliminate all exterior build dependencies beside stack, which is a major benefit on non-linux platforms.

Anrock · 2018-12-12T15:53:26Z

@power-fungus Can you clarify?

Suppose i have clean stack install without any ghc downloaded yet.

If i run stack Shakefile ... then stack uses resolver from shake.yaml with ghc-8.6.2 to run Shakefile.hs itself. So ghc-8.6.2 is downloaded in order to run Shakefile in addition to whatever ghc i need for building hie.

If i run make hie-8.4.4 it executes stack --stack-yaml stack-8.4.4.yaml ..., so stack downloads ghc-8.4.4 and nothing else.

Where does stack.yaml come in in case with Makefile?

fendor · 2018-12-12T16:03:39Z

@Anrock make hie-8.4.4 still requires the cabal target, which installs cabal with the compiler that is in the local stack.yaml, which currently is ghc-8.6.2.

Anrock · 2018-12-12T16:56:46Z

@fendor i see now! I thought it was just overlooked.

fendor · 2018-12-12T18:42:48Z

I implemented the dist target in a platform independent way, e.g. tar package and zlib package.
I assume that we can use zlib, since git also requires libz library and should be present on the system, anyways.
However, I have to test that first.

Tested on:

Arch Linux
Fedora
Windows
MacOs

fendor · 2018-12-13T16:21:32Z

I added the icu-macos-fix target.
It would be nice if someone can validate that the Shakefile.hs also works on MacOs.

fendor · 2018-12-14T15:39:28Z

The pull request should be ready for reviews now.

Anrock · 2018-12-17T10:28:11Z

Just built hie-8.4.4 on OSX with Shakefile. Looks like it working.

alanz · 2018-12-21T09:26:34Z

Also, it builds all the individual hie versions, and installs them as ~/.local/bin/hie. Then at the end it copies the last one (for GHC 8.2.1) as ~/.local/bin/hie-xxx, so they all end up being for the same GHC.

fendor · 2018-12-21T11:32:43Z

I agree it should have a default, the help message seems like a good idea.

I have to check that. In my opinion, the last ghc should be 8.6.2, but I might be mistaken.

alanz · 2018-12-21T12:57:19Z

The point is that the hie-8.2.1, hie-8.4.3 etc are all copies of the same file, the last hie exe compiled

…

On Fri, 21 Dec 2018, 13:32 fendor ***@***.*** wrote: I agree it should have a default, the help message seems like a good idea. I have to check that. In my opinion, the last ghc should be 8.6.2, but I might be mistaken. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#991 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZAB8bI8a8wkBBSQL6FUEyLDdDqRtQvks5u7MbbgaJpZM4ZN1kb> .

fendor · 2018-12-21T13:17:19Z

If I understand you correctly, this behaviour should be similar to the Makefile.

cp '$(STACKLOCALBINDIR)/hie' '$(STACKLOCALBINDIR)/hie-$*'

We could copy it directly from the compile path, just like when executing the dist target.

EDIT: just realised what you mean

fendor · 2018-12-21T18:58:40Z

@alanz The issue about the hie-versions not properly being copied should be resolved.

alanz · 2018-12-22T17:22:24Z

LGTM

alanz · 2018-12-22T17:34:38Z

One thing though, current master builds for 8.6.3 too.

fendor · 2018-12-22T23:31:50Z

I will update the pull request to also include 8.6.3.
I would also update the Readme to use the Shakefile.

alanz · 2018-12-25T15:10:10Z

shake.yaml

@@ -0,0 +1,7 @@
+# Used to provide a different environment for the shake build script
+resolver: nightly-2018-12-01 # GHC 8.6.2


Is there any way we can do something like "use the system resolver on this machine"?

As far as I can tell, since there is a stack.yaml in the project, the script will always default to that.

However,one can work around this by compiling the shakefile.

An extra stack.yaml is required since the other yamls contain information about extra dependencies in the submodules folder, which is not present on a non-recursive clone or if the submodules change. Since the build script should take care of updating the git submodules, they are not allowed to be referenced within the stack-yaml that is used by Shakefile.hs.

It is possible, however, to include system-ghc: true within the shake.yaml (see option) However, this seems to be incompatible with Docker and nix.

alanz · 2018-12-25T17:35:31Z

Shakefile.hs

@@ -170,7 +176,7 @@ helpMessage = do
  out ""
  out "Targets:"
  out
-    "    build                Builds hie for all supported GHC versions (8.2.1, 8.2.2, 8.4.2, 8.4.3, 8.4.4, 8.6.1 and 8.6.2)"
+    "    build                Builds hie for all supported GHC versions (8.2.1, 8.2.2, 8.4.2, 8.4.3, 8.4.4, 8.6.1, 8.6.2 and 8.6.3)"


We should eventually refactor this (and the next set of targets) to be generated from the list of hieVersions, so we can set that in one place only.

in the next version of the shakefile i would like to refactor this to obtain the list of hie-versions from present stack.yaml files in the directory. But i felt like this should be in another pull request.

alanz · 2018-12-27T17:57:00Z

Is this ready for merge yet?

fendor · 2018-12-27T19:59:58Z

Technically yes, but the README.md needs updates that one should use ./Shakefile.hs. This should be done tomorrow.
Also, it is not clear to me if we should use system-ghc: true in shake.yaml now. I would like to test it tomorrow.
Moreover, should the Makefile and BuilAll.ps1 be deleted in this pull request to avoid confusion?

alanz · 2018-12-27T20:55:10Z

I would say we should add it, and keep the existing Makefile and .ps1 file.

And add a section saying we now have experimental Shake build support, and to please try it and report.

Then we can see what kind of feedback we get.

fendor · 2018-12-27T21:20:12Z

Ok, then I will revert the changes to the README.md and add a section for the Shakefile.hs.

fendor · 2018-12-28T12:55:40Z

README is updated, but I did not manage to use the option system-ghc: true correctly.

I think, it is ready to merge now.

alanz · 2018-12-28T13:01:30Z

Given #1008, I wonder if we should not use an LTS that is not for GHC 8.6.3.

Otherwise it looks good. I think the build is failing because hackage was being restarted.

fendor · 2018-12-28T13:07:41Z

Makes sense, LTS-12.25 looks fine?

fendor · 2018-12-28T13:25:40Z

LTS has been updated

fendor added 10 commits December 10, 2018 22:01

Add Shake Build script

ecb1e26

Rename function names, change line endings and add help message

c394801

Add custom stack.yaml for shake build environment

2a72daf

Add shake build information to gitignore

4f4f9f3

Custom shakel.yaml file for the Shake file

d9387fb

Change cabal old-style build command to v1-*

2142b91

Unset GHC_PACKAGE_PATH, because cabal install complains about it

acfa2ca

Code cleanup, including reformatting

80f2614

Remove unused nix packages

181f386

Replace cp with platform independent copyFile'

019301f

Add experimental support for dist target

16b822a

fendor added 3 commits December 12, 2018 19:58

Fix output name of tar file

276d72f

Add macos fix taken from the Makefile

8fef057

Add explicit import list and add comments

c84e2cd

Add .exe to the executable endings for windows

91bd9a7

2mol mentioned this pull request Dec 19, 2018

improving the build/installation process #1002

Closed

Anrock requested a review from alanz December 20, 2018 13:42

Anrock mentioned this pull request Dec 20, 2018

Bullet-proof installation instructions #409

Closed

fendor added 3 commits December 21, 2018 14:33

Add default target help

796bed9

Try to fix the cp of the executables

d453811

Fix install command and add major hie versions

940ecc2

alanz added this to the 2018-12 milestone Dec 24, 2018

fendor added 2 commits December 25, 2018 16:05

Merge remote-tracking branch 'upstream/master'

a83ae5f

Support ghc 8.6.3

bfc8624

alanz reviewed Dec 25, 2018

View reviewed changes

Build docs of any hie

e9d3639

alanz reviewed Dec 25, 2018

View reviewed changes

Update README to mention Shakefile.hs instead of Makefile

0c70fb9

fendor added 2 commits December 28, 2018 13:47

Update shake.yaml to lts-13.0

9f87ce8

Add section for the Shake build system in README

02feb7c

Update shake.yaml to lts-12.25

0b29bc6

alanz merged commit 03dcb8a into haskell:master Dec 29, 2018

		@@ -0,0 +1,7 @@
		# Used to provide a different environment for the shake build script
		resolver: nightly-2018-12-01 # GHC 8.6.2

Add Shakefile as possible build script #991

Add Shakefile as possible build script #991

Uh oh!

Conversation

fendor commented Dec 11, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Anrock commented Dec 12, 2018

Uh oh!

lorenzo commented Dec 12, 2018

Uh oh!

Anrock commented Dec 12, 2018

Uh oh!

lorenzo commented Dec 12, 2018

Uh oh!

samuelpilz commented Dec 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Anrock commented Dec 12, 2018

Uh oh!

fendor commented Dec 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Anrock commented Dec 12, 2018

Uh oh!

fendor commented Dec 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fendor commented Dec 13, 2018

Uh oh!

fendor commented Dec 14, 2018

Uh oh!

Anrock commented Dec 17, 2018

Uh oh!

alanz commented Dec 21, 2018

Uh oh!

fendor commented Dec 21, 2018

Uh oh!

alanz commented Dec 21, 2018 via email

Uh oh!

fendor commented Dec 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fendor commented Dec 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alanz commented Dec 22, 2018

Uh oh!

alanz commented Dec 22, 2018

Uh oh!

fendor commented Dec 22, 2018

Uh oh!

alanz Dec 25, 2018

Choose a reason for hiding this comment

Uh oh!

fendor Dec 25, 2018

Choose a reason for hiding this comment

Uh oh!

samuelpilz Dec 25, 2018

Choose a reason for hiding this comment

Uh oh!

alanz Dec 25, 2018

Choose a reason for hiding this comment

Uh oh!

fendor Dec 25, 2018

Choose a reason for hiding this comment

Uh oh!

alanz commented Dec 27, 2018

Uh oh!

fendor commented Dec 27, 2018

Uh oh!

alanz commented Dec 27, 2018

Uh oh!

fendor commented Dec 27, 2018

Uh oh!

fendor commented Dec 28, 2018

Uh oh!

alanz commented Dec 28, 2018

Uh oh!

fendor commented Dec 28, 2018

fendor commented Dec 11, 2018 •

edited

Loading

samuelpilz commented Dec 12, 2018 •

edited

Loading

fendor commented Dec 12, 2018 •

edited

Loading

fendor commented Dec 12, 2018 •

edited

Loading

fendor commented Dec 21, 2018 •

edited

Loading

fendor commented Dec 21, 2018 •

edited

Loading