[WIP] Delay callback validation

[ned2]

This is a proposed approach for handling delayed callback validation. As discussed in #519, immediate callback validation requires the Dash layout to be already set, imposing rigid constraints on where callbacks can be defined within a Dash app. In particular, this making arrangement of pages within multi-page apps that contain both callbacks and layout fragments impossible.

This change was more extensive than expected, so at the moment, this is perhaps more of way to get the conversation going.

• this approach involves adding to a list of registered callbacks in Dash._callback_list (distinct from the Dash.callback_map) when each callback is defined. The list is then used to perform validation when the app starts
• I think it's critical that this callback validation be applied for WSGI apps as well as apps run through run_server, so I did not put this in the enable_dev_tools_folder, where I think there is a risk of users not running it. So instead I've created a _setup() method that should both be run by run_server and also get_wsgi_application, which would become the required way to get the WSGI callable for when using a WSGI server. This is modelled off Django's approach. So this is also a proposed solution to the general problem of keeping the run_server and WSGI app entry points synchronised (which perhaps should go in a different PR though)
• I had to split the current callback validation tests into two, as the current code assumes that buggy callbacks can be caught through exceptions and continue on to use the already defined Dash app, which doesn't work that way now
• A slightly unsatisfying property of this solution is that for an app with invalid callbacks, before validation is applied (and an error thrown), the Dash.callback_map will be in a somewhat dirty state, with eg duplicate callbacks clobbering existing ones.

[ned2]

Here is the error message produced by the below invalid Dash app:

The following outputs have been assigned to multiple callbacks.

<Output `target1.children`> in callback functions: 'callback1', 'callback2'
<Output `target2.children`> in callback functions: 'callback3', 'callback4'

An output can only be associated with a single callback function.
Try combining your inputs and callback functions together into one
function.

The following outputs occur multiple times in a single multi-output
callback:

<Output `target2.children`> in callback function: 'callback4'

Multi-output callbacks cannot target the same output more than
once.

import dash
import dash_core_components as dcc
import dash_html_components as html
from dash.dependencies import Input, State, Output

app = dash.Dash()
app.layout = html.Div([
    html.Div(id='target1'),
    html.Div(id='target2'),
    html.Div(id='target3'),
    dcc.Input(id='my-input', type='text', value=''),
    html.Button(id='submit', n_clicks=0, children='Save')
])

@app.callback(Output('target1', 'children'), [Input('submit', 'n_clicks')],
              [State('my-input', 'value')])
def callback1(n_clicks, state):
    return "callback received value: {}".format(state)

@app.callback(Output('target1', 'children'), [Input('submit', 'n_clicks')],
              [State('my-input', 'value')])
def callback2(n_clicks, state):
    return "callback received value: {}".format(state)

@app.callback(Output('target2', 'children'), [Input('submit', 'n_clicks')],
              [State('my-input', 'value')])
def callback3(n_clicks, state):
    return "callback received value: {}".format(state)

@app.callback([Output('target2', 'children'),
               Output('target2', 'children')],
              [Input('submit', 'n_clicks')],
              [State('my-input', 'value')])
def callback4(n_clicks, state):
    return "callback received value: {}".format(state)

if __name__ == '__main__':
    app.run_server(debug=True)

[ned2]

Keen to get feedback from @T4rk1n, @alexcjohnson, and @chriddyp

[alexcjohnson]

Thanks for pushing this forward @ned2 - I think I'll comment over at #519, then we can come back here if this ends up being the preferred approach.

[alexcjohnson]

Closing - #1103 moved callback validation to the front end.