convenience function to help compare models using WAIC or LOO #1775
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@aloctavodia Catching the warnings would be the pythonic way I think. |
twiecki
reviewed
Feb 14, 2017
pymc3/stats.py
Outdated
| @@ -281,10 +281,10 @@ def compare(traces, models, ic='WAIC'): | |||
| Returns | |||
| ------- | |||
| A DataFrame, ordered from lowest to highest IC. The index reflects | |||
| the order in which the models are passed to this function. The columns are: | |||
| the order in wich the models are passed to this function. The columns are: | |||
twiecki
reviewed
Feb 14, 2017
pymc3/stats.py
Outdated
| Smaller IC indicates higher out-of-sample predictive fit ("better" model). | ||
| Default WAIC. | ||
| Smaller IC indicates higher out-of-sample predictive fit ("better" model). | ||
| Default WAIC. |
|
An example would also be good. |
…, I guess there is a more clever/pythonic solution
twiecki
reviewed
Feb 15, 2017
pymc3/stats.py
Outdated
|
|
||
| Parameters | ||
| ---------- | ||
| traces: list of PyMC3 traces |
|
@twiecki thanks for the reviews! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a simple function that computes WAIC or LOO (IC) for a set of models and return a DataFrame ordered from "best" to "worse" model (in terms of the IC). It also computes other quantities like Akaike weights that can be used for example for model averaging and the standard error of the difference in IC between each model and the top-ranked model, that can help to compare and interpret IC for different models. A similar function is used extensively in the book Statistical Rethinking by Richard McElreath.
In a future PR I will add a plot based on this function (I guess it should be a separate function living inside
plots). And I will update the notebook examples to give some details on this function and how it can helps to interpret IC (beyond the "smaller is better" rule ).One current problem with this function is that the warnings returned by waic/loo are not very useful given that the user does not know which model is returning a warning or even if there is more than one warning.
I am not sure which is the best solution here. A couple of alternatives are:
compare, or by modifying waic/loo to return such a value.I think the second option (including the modification of waic/loo) is a better option. What do you think?