Skip to content

Error on NaN's in check_model #888

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 10, 2025
Merged

Error on NaN's in check_model #888

merged 2 commits into from
Apr 10, 2025

Conversation

penelopeysm
Copy link
Member

@penelopeysm penelopeysm commented Apr 10, 2025

Closes #887

Open questions:

  • Should it warn or error?
  • Should it also check for ±Inf?

Copy link
Contributor

github-actions bot commented Apr 10, 2025

Benchmark Report for Commit c215fb2

Computer Information

Julia Version 1.11.4
Commit 8561cc3d68d (2025-03-10 11:36 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × AMD EPYC 7763 64-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)

Benchmark Results

|                 Model | Dimension |  AD Backend |      VarInfo Type | Linked | Eval Time / Ref Time | AD Time / Eval Time |
|-----------------------|-----------|-------------|-------------------|--------|----------------------|---------------------|
| Simple assume observe |         1 | forwarddiff |             typed |  false |                  9.5 |                 1.6 |
|           Smorgasbord |       201 | forwarddiff |             typed |  false |                607.9 |                44.4 |
|           Smorgasbord |       201 | forwarddiff | simple_namedtuple |   true |                423.5 |                49.0 |
|           Smorgasbord |       201 | forwarddiff |           untyped |   true |               1212.9 |                28.3 |
|           Smorgasbord |       201 | forwarddiff |       simple_dict |   true |               3739.2 |                21.2 |
|           Smorgasbord |       201 | reversediff |             typed |   true |               1433.4 |                29.9 |
|           Smorgasbord |       201 |    mooncake |             typed |   true |                934.7 |                 5.3 |
|    Loop univariate 1k |      1000 |    mooncake |             typed |   true |               5508.3 |                 4.1 |
|       Multivariate 1k |      1000 |    mooncake |             typed |   true |               1079.1 |                 8.6 |
|   Loop univariate 10k |     10000 |    mooncake |             typed |   true |              60900.7 |                 3.8 |
|      Multivariate 10k |     10000 |    mooncake |             typed |   true |               9158.8 |                 9.5 |
|               Dynamic |        10 |    mooncake |             typed |   true |                128.8 |                13.3 |
|              Submodel |         1 |    mooncake |             typed |   true |                 25.5 |                 8.0 |
|                   LDA |        12 | reversediff |             typed |   true |                478.2 |                 5.0 |

Copy link

codecov bot commented Apr 10, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.92%. Comparing base (019e41b) to head (c215fb2).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #888      +/-   ##
==========================================
+ Coverage   84.87%   84.92%   +0.04%     
==========================================
  Files          34       34              
  Lines        3815     3814       -1     
==========================================
+ Hits         3238     3239       +1     
+ Misses        577      575       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coveralls
Copy link

coveralls commented Apr 10, 2025

Pull Request Test Coverage Report for Build 14386694475

Details

  • 4 of 8 (50.0%) changed or added relevant lines in 1 file are covered.
  • 1211 unchanged lines in 26 files lost coverage.
  • Overall coverage decreased (-0.5%) to 84.43%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/debug_utils.jl 4 8 50.0%
Files with Coverage Reduction New Missed Lines %
src/DynamicPPL.jl 1 90.91%
src/submodel_macro.jl 2 82.35%
src/extract_priors.jl 5 53.57%
src/test_utils/model_interface.jl 5 22.22%
src/test_utils/varinfo.jl 5 78.26%
src/values_as_in_model.jl 6 43.9%
src/logdensityfunction.jl 7 64.44%
src/model_utils.jl 7 7.14%
src/distribution_wrappers.jl 13 9.76%
src/model.jl 13 80.0%
Totals Coverage Status
Change from base Build 14321484767: -0.5%
Covered Lines: 3221
Relevant Lines: 3815

💛 - Coveralls

@coveralls
Copy link

coveralls commented Apr 10, 2025

Pull Request Test Coverage Report for Build 14387051129

Details

  • 1 of 3 (33.33%) changed or added relevant lines in 1 file are covered.
  • 1211 unchanged lines in 26 files lost coverage.
  • Overall coverage increased (+0.05%) to 85.013%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/debug_utils.jl 1 3 33.33%
Files with Coverage Reduction New Missed Lines %
src/DynamicPPL.jl 1 90.91%
src/submodel_macro.jl 2 82.35%
src/extract_priors.jl 5 53.57%
src/test_utils/model_interface.jl 5 22.22%
src/test_utils/varinfo.jl 5 78.26%
src/values_as_in_model.jl 6 43.9%
src/logdensityfunction.jl 7 64.44%
src/model_utils.jl 7 7.14%
src/distribution_wrappers.jl 13 9.76%
src/model.jl 13 80.0%
Totals Coverage Status
Change from base Build 14321484767: 0.05%
Covered Lines: 3239
Relevant Lines: 3810

💛 - Coveralls

@penelopeysm penelopeysm changed the title Warn on NaN's Error on NaN's Apr 10, 2025
@penelopeysm penelopeysm requested a review from sunxd3 April 10, 2025 17:50
Copy link
Member

@sunxd3 sunxd3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@penelopeysm
Copy link
Member Author

@sunxd3 do you have an opinion on checking for Inf too? (I edited this in at the top but either it was too late or the bot comments distracted from it 😂)

@sunxd3
Copy link
Member

sunxd3 commented Apr 10, 2025

I think checking Inf is a good idea. Also to make it more useful, is there an option where the checking function will do multi-run? (Saying this because Inf can be less deterministic than the other two.)

@yebai
Copy link
Member

yebai commented Apr 10, 2025

Should it warn or error?

This should be an error.

Should it also check for ±Inf?

This depends; certain distributions have ±Inf in support. It is better to check whether ±Inf is in support; if not, an informational message can be given.

@penelopeysm
Copy link
Member Author

Ok! I think let's leave Infs for another time/PR. I am a bit wary of edge cases - NaN is more straightforward because you probably don't want any of them anywhere, so any(isnan, left) suffices; but if Inf is sometimes ok and sometimes not (which makes sense to me) then it gets a bit finicky, especially if the rhs has something like arraydist or product_distribution.

@penelopeysm penelopeysm added this pull request to the merge queue Apr 10, 2025
@penelopeysm penelopeysm changed the title Error on NaN's Error on NaN's in check_model Apr 10, 2025
@penelopeysm
Copy link
Member Author

Inf is now a new issue, #889, so someone can pick it up if they feel like it. But at least having the NaN's checked will fix the immediate request in TuringLang/Turing.jl#2515.

Merged via the queue into main with commit c7bdc3f Apr 10, 2025
17 of 18 checks passed
@penelopeysm penelopeysm deleted the py/check-nans branch April 10, 2025 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

check_model should error on NaN's
4 participants