Progress bars in Jupyter notebooks

[devmotion]

I tried to use TerminalLoggers for displaying progress bars in a Jupyter notebook (I just ran the example from the docs), but unfortunately the progress bar is not updated correctly. It seems TerminalLoggers does not clear existing progress bars, and hence I ended up with multiple lines of incomplete progress bars instead.

By using the superseded ConsoleProgressMonitor I can workaround this issue, but I'm wondering if there exists a workaround or fix for TerminalLoggers?

[tkf]

I think the easiest short-term solution is to make ConsoleProgressMonitor work with ProgressLogging 0.1.1 tkf/ConsoleProgressMonitor.jl#9

In a long run, I think we'd need Jupyter Lab/Notebook -specific progress monitors, just like you get the GUI progress bar in Juno.

[tkf]

ref tkf/ConsoleProgressMonitor.jl#10

[devmotion]

Wow, thanks for the prompt reply and the update!

[c42f]

We should definitely have a solution for Jupyter but I think it may not be in TerminalLoggers itself — TerminalLogger is largely assuming it's talking to a terminal with limited display capabilities (ANSI codes for color and some limited cursor control).

We should probably add a logger sink to IJulia which uses MIME"text/html" for formatting log records rather than text, fancy progress bar widgets etc etc. I'd also love for the log records to come up separately in a way which is somehow easiliy browsable in Jupyter without interfering with the "normal" output of a cell.

[c42f]

This would also fix the problem with IJulia capturing stderr text-based logs and Jupyter coloring them that pink error-looking color regardless of whether they're Info messages, etc.

[tkf]

I think it may not be in TerminalLoggers itself

I agree. I think it should be in somewhere else, like JupyterLogger.jl or something.

We should probably add a logger sink to IJulia

I'm not following Jupyter ecosystem for a while so I'm not 100% sure, don't you need to use different mechanisms for registering dynamic HTML/JavaScript components for each frontend (Jupyter Lab/Jupyter Notebook)?

Looking at nteract/nteract#4573, maybe using whatever ipywidgets is using could be enough for supporting a wide range of frontend (including nteract).

I'd also love for the log records to come up separately in a way which is somehow easiliy browsable in Jupyter without interfering with the "normal" output of a cell.

For "normal" Jupyter usage (i.e., the kernel is local) it might be just enough to open a separate GUI window. Here is a prototype that only handles progress bars: https://github.com/tkf/ElectronProgressBars.jl. Or open a terminal in a separate window and then use TerminalLoggers.jl inside https://github.com/tkf/TmuxDisplays.jl

[c42f]

I think it should be in somewhere else, like JupyterLogger.jl or something.

Fine by me to start off! Does it need to be separate from IJulia in the longer term if IJulia is already supporting both JupyterLab and Jupyter Notebook? I haven't looked into the code behind the jupyter ecosystem so I don't know how responsibilities for frontend and backend are split there.

[tkf]

Actually, I don't personally have a plan to start implementing Jupyter-based loggers (sorry!). So, if whoever wants to do this thinks doing it in IJulia makes sense, maybe that's a good idea.

Does it need to be separate from IJulia in the longer term if IJulia is already supporting both JupyterLab and Jupyter Notebook?

I thought you'd need HTML widgets but it looks like there is now update_display_data in the wiring protocol. It sounds like this is all we need but I'm not sure if IJulia.jl supports this message type.

[c42f]

Actually, I don't personally have a plan to start implementing Jupyter-based loggers (sorry!).

Haha! No worries. As an occasional user of Jupyter I may end up doing this, but not sure when I'd get around to it. (If anyone wants it soon they should jump right in ;-) )

[cossio]

Bump. Any updates on this?

[c42f]

I now use Pluto rather than IJulia so if I was to work on this I'd put the effort into Pluto integration. But I don't even have time to work on that right now. In any case, it's outside the scope of this package to support Jupyter in any kind of properly integrated way so I should close this issue.

If there's any non-intrusive tweaks we can make to have TerminalLoggers and Jupyter interact less strangely then I'm happy to accept patches. But the real solution is a Jupyter-specific logging backend which integrates with the Jupyter GUI in some way.