`cabal watch` command

[gbaz]

This ticket is intended to capture ideas for a cabal watch command to give functionality along the lines of ghcid. In particular, one should be able to "watch" a component just as one new-repls a component. I would imagine that there would be a few knobs to turn on what happens when something is altered. One could recompile with no-code (related issue: #1176) or perhaps send some form of signal to a running ghci process that just causes it to execute a :r automatically. In which case cabal watch would really be an option on new-repl. It might also be configurable to run a custom command. The main idea is that since cabal knows which files are in the dependency graph, it is a good place to stick a watch command on them.

[gbaz]

One missing component is just a little tiny api to control ghci from outside -- i.e. send it commands like ":r" and ":q" over some port or the like. This would be hugely useful to ghcid too and other tools in general. Maybe @alanz or @ndmitchell have some thoughts?

[ndmitchell]

There is an API to send :r and :q to ghci from the outside - it's called System.Process. The new API would have to be simpler and more robust, which is hard given that the existing API is super simple and reasonably robust...

If you put it inside ghci/ghc itself there would be a lot of value, but having Cabal fake this on top just seems to be moving the problem around.

Perhaps one thing that would be useful is Cabal declaring what files/directories would need to to restart, and which would require a reload? Just pretty printing Depending on .cabal-metadata-file or similar would be useful.

[gbaz]

My sense is that using stdout/stdin for communication with interactive programs is a bit messy, even though ghcid manages to pull it off? What about a port or a shared fd on the system? Not sure which of these mechanisms works well across platforms either. A different division of labor that just gives a ghci watch and has cabal give it some helpful tips does sound more robust if we're ok with pushing enough fsnotify functionality directly into the ghc codebase.

[ndmitchell]

There are lots of ways the complexity of ghcid leaks out, but stdin/stdout isn't one of them, so you'd be paying a cost (shared fd or whatever) and not really reaping any benefits. I completely agree that stdin/stdout is ugly, especially with buffering, but part of that is that the underlying GHC is a single shared resource - if it were multithreaded (allowed multiple concurrent executions), then you could imagine a richer communication (http being my default favourite, since it's easily debuggable).

[gbaz]

i confess i don't understand how a single shared resources gets in the way of richer communication? suppose we had an http port -- why couldn't communication on that still only be with regards to the single shared ghc resource.

[ndmitchell]

Sorry, to rephrase better, the communication channel we have now (stdin/stdout) sucks if you had a concurrent resource, but we don't. All the other communication channels are almost no better when you have a single-access resource, but quite a lot better when you have a multiaccess resource, so if you made the resource multiaccess there would be more desire to go to richer communication.

[gbaz]

Just a note: some of the data we need exists in new-build --dry and new-repl --dry but we also should have a way to get fully-qualified components, etc

[alanz]

I'm late to this party, sorry.

I know when I looked into using ghci as a subcomponent of hie, one of the things that troubled me is that all the IO happens via stdin/stdout, including when user code is executed and it can do anything with stdio, which means any kind of tooling interaction while a program is running must pause, and also that anything can be sent by the app, since it is completely out of our control.

One of the possible workarounds for this is to use the external interpreter mode, and use specific pipes for the user stdio, which can then be routed to the host IDE in a controlled way.

And in this model putting a proper multiplexed protocol on the stdio, or exposing a different API end point becomes possible.

[ndmitchell]

The user can't do anything with stdio, in particular they can't guess at the magic cookie you use to delineate their stuff from yours. That's what ghcid relies on. However, on the flip side, they can leave a daemon running in ghci which prints to stdout randomly - but if they do they probably aren't going to have an enjoyable time.

That said, I don't think cabal watch solves any of those problems. It solves an entirely different set of problems (that I don't find very problematic...).

[alanz]

True.

And I also agree that I do not see any real use for this feature in hie, nor for me in everyday usage.

[gbaz]

So the question to me is if there is some underlying improvement to ghci that might improve life for hie and ghcid, and as a bonus could also enable a "cabal watch" feature along similar lines.

Similarly if there is a particular feature of cabal that might be useful to ghcid or hie or both in terms of e.g. emitting the --dry output in a better fashion or the like?

[ndmitchell]

The biggest problem that ghcid faces is that ghci is actually quite buggy. It doesn't reload when it should, it over-reloads, and just gets wedged every now and again. What's needed is a dedicated maintainer with lots of times to actually fix the bugs that are already in trac. Otherwise we're putting lipstick on a pig.

For the use of ghcid with Cabal, it's not really a mode I use it in (I tend to use raw ghci with a globally installed package data set) - but I've not really had any complaints from people.

[alanz]

Apart from what @ndmitchell said, my biggest need is to have something in cabal that can make cabal-helper unnecessary. There is an issue for it, and @DanielG has some concerns with what is there. So adressing those and getting it done would help.

And identifying the memory leak would help too.