Update model comparison and SMC notebooks #2855
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
LGTM. |
gBokiau
reviewed
Feb 12, 2018
| @@ -114,20 +73,20 @@ | |||
| " \n", | |||
| " NOTE: latent_sigma_y is used to create a normally distributed,\n", | |||
| " 'latent error' aka 'inherent noise' in the 'physical process' \n", | |||
| " generating thses values, rather than experimental measurement error. \n", | |||
| " generating theses values, rather than experimental measurement error. \n", | |||
There was a problem hiding this comment.
these--s--
…in the ‘physical' data generating process, rather…
would sound more familiar IMO
| "metadata": {}, | ||
| "source": [ | ||
| "Back to the real purpose of this Notebook: demonstrate model selection.\n", | ||
| "Back to the real purpose of this Notebook, demonstrate model selection.\n", |
| @@ -1133,8 +1010,7 @@ | |||
| "\n", | |||
| "+ Quadratic-generated data (rhs):\n", | |||
| " + The WAIC is also quite flat across the models\n", | |||
| " + The lowest WAIC is model **k4**, but **k3** - **k5** are more or less the same. \n", | |||
| " " | |||
| " + The lowest WAIC is model **k4**, but **k3** - **k5** are more or less the same. " | |||
| ] | |||
There was a problem hiding this comment.
Looking at the plots, I believe this should be k3 - k4
| @@ -1224,7 +1093,23 @@ | |||
| "\n", | |||
| "It is important to keep in mind that, with more data points, the real underlying model (one that we used to generate the data) should outperforms other models. \n", | |||
| @@ -1224,7 +1093,23 @@ | |||
| "\n", | |||
| "It is important to keep in mind that, with more data points, the real underlying model (one that we used to generate the data) should outperforms other models. \n", | |||
| "\n", | |||
| "In general, PSIS-LOO is recommended. To quote from [avehtari's comment](https://github.com/pymc-devs/pymc3/issues/938#issuecomment-313425552): \"I also recommend using PSIS-LOO instead of WAIC, because it's more reliable and has better diagnostics as discussed in http://link.springer.com/article/10.1007/s11222-016-9696-4 (preprint https://arxiv.org/abs/1507.04544), but if you insist to have one information criterion then leave WAIC\". Alternatively Watanabe [says](http://watanabe-www.math.dis.titech.ac.jp/users/swatanab/index.html) \"WAIC is a better approximator of the generalization error than the pareto smoothing importance sampling cross validation. The Pareto smoothing cross validation may be the better approximator of the cross validation than WAIC, however, it is not of the generalization error\"." | |||
| "In general, PSIS-LOO is recommended. To quote from [avehtari's comment](https://github.com/pymc-devs/pymc3/issues/938#issuecomment-313425552): \"I also recommend using PSIS-LOO instead of WAIC, because it's more reliable and has better diagnostics as discussed in http://link.springer.com/article/10.1007/s11222-016-9696-4 (preprint https://arxiv.org/abs/1507.04544), but if you insist to have one information criterion then leave WAIC\". Alternatively, Watanabe [says](http://watanabe-www.math.dis.titech.ac.jp/users/swatanab/index.html) \"WAIC is a better approximator of the generalization error than the pareto smoothing importance sampling cross validation. The Pareto smoothing cross validation may be the better approximator of the cross validation than WAIC, however, it is not of the generalization error\"." | |||
There was a problem hiding this comment.
There is some agreement that PSIS-LOO offers the best indication of a model's quality.
| "metadata": {}, | ||
| "source": [ | ||
| "##### Now loop through all the models and calculate the WAIC" | ||
| "In this case we are interested in the WAIC score, we can plot also the standard error of the estimation, which is nice.\n", |
There was a problem hiding this comment.
In this case we are interested in the WAIC score. We also plot error bars for the standard error of the estimated scores. This gives us a more accurate view of how much they might differ.
|
thanks @gBokiau for your help. I included almost all your suggestions. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update notebooks following guidelines in #2834, also fix some typos and update deprecated syntax.