Cleanup kernel for dashboards run within Jupyter Notebook

[parente]

Deploying completely standalone dashboards can be overweight for situations in which the notebook author wants to quickly share a series of "micro-dashboards" for a couple of trusted users (e.g., teammates) to play with. It would be nice to hand users a URL to a notebook within Jupyter and have it appear as a dashboard. We already do this somewhat with the ?dashboard query arg which hides everything except the Jupyter menubar. A few more actions are required to get a functioning dashboard for every user that hits the notebook.

• Hide all chrome, including the menubar.
• Spawn a completely new kernel to stay independent from other users. (Is this already supported in the notebook?)
• Run all cells. (This has security implications.)

[parente]

Another approach would be to piggyback on the "Deploy as > Local dashboard" which was originally intended for notebook authors to see what their dashboards would look like when deployed. They address all three bullets in the issue description. However, they "leak" kernels. A visit to the dashboard spawns a kernel but nothing ever cleans the kernel up when the user navigates away.

Perhaps cleanup is fixable in the dashboard extension? Periodically check which kernels have connected websocket clients and cleanup those that have none?

[dalogsdon]

Cleanup is an interesting issue. @jhpedemonte and I discussed this a little bit as we were investigating the Thebe alternative in #105. For forever-running processes, such as Spark Streaming, when do we clean up? If the main usage of the dashboard "preview" is to see the layout with less focus on actual content, then navigation away from the page could be ok.

For the deployed case, we discussed manually shutting down standalone dashboard kernels as you can do in the Notebook already. That could be an alternative in this case too.

[parente]

Cleanup has been something we've punted to the container level with tmpnb. When the user has not had interaction with a kernel for a while, the container hosting it goes away.

I think forever-running processes are actually misplaced within the same kernel / container as the dashboard. I know we're doing that in our demos, but I want to believe we only do it for educational purposes. If we wanted to deploy a dashboard that we wanted to scale even to a dozen users, I wouldn't want each one doing its own streaming. Instead, I'd expect the stream to be in its own process with an API and the dashboard notebooks just to sample it.

Today, if I want to build that standalone API, I've got to step outside of the notebook environment. But I think this where kernel gateway and notebooks that becomes micro services can get interesting.

• Notebook Update and rename LICENSE to LICENSE.md #1: all the Spark stream stuff, gets converted to a microservice that either exposes an API and/or dumps to a traditional view DB (like in lambda arch)
• Notebook Setup Travis CI #2: all the UI stuff, gets converted to a dashboard that uses the API / DB from notebook Update and rename LICENSE to LICENSE.md #1

[jhpedemonte]

That makes sense. Our (potential) future work then would be to make those steps as easy as possible for a dev.

[dalogsdon]

Course of action based on discussion above:

1. Expose "Deploy as > Local dashboard" page as some URL
2. Start new kernel
3. Run all code
4. Kill kernel when page or websocket connection is lost

[parente]

Expose "Deploy as > Local dashboard" page as some URL

Not sure we need this one. If user has ability to log into the Jupyter server, they can just hit the URL of the locally deployed dashboard. I'm OK with the notebook author still controlling when that app is deployed.

Kill kernel when page or websocket connection is lost

Immediately when lost is too quick. WS disconnects happen all the time. tmpnb works on a reaping cycle after a period of inactivty.

[dalogsdon]

So then this issue is just to clean up kernels spawned by locally deployed dashboards?

[parente]

Yes. Either:

1. automatically, when there are no active websockets associated with a kernel for some configurable period of time
2. manually, via the Running tab in the Jupyter notebook UI

I believe the reason the kernel doesn't appear in the UI for the second approach is because there is no associated session resource. Thebe only requests the kernel, not a session to go with it (and rightly so because there is no associated notebook with the kernel).

[jhpedemonte]

Regarding the Running tab in notebook UI: how do you feel about adding a 'Dashboards' section there, to go with 'Notebook' and 'Terminals'? We might be able to do it as part of our extension.

[jhpedemonte]

The refresh in the Running tab isn't extensible at all, not even by hacking. We could just add our own 'Dashboards' section and manage it ourselves. Add our own timer and event handler on the refresh button. But we may need to add an /api/dashboards server extension point, to query the running Dashboard kernels.

[parente]

Definitely don't want to add new UI to the notebook frontpage (also called dashboard just to be confusing) given the roadmap of where things are headed with phosphor. That just introduces more things that need to be ported.

I was more wondering if, once we're based on jupyter-js-services, if we could make an /api/session request and pass along the filename of the dashboard HTML file plus a little UUID to allow for multiple instances, instead of a proper notebook to get the associated kernel tracked by the UI automatically.