Skip to content

Add a note that NDCG returns values between 0 and 100 #2637

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rogancarr opened this issue Feb 19, 2019 · 6 comments
Closed

Add a note that NDCG returns values between 0 and 100 #2637

rogancarr opened this issue Feb 19, 2019 · 6 comments
Assignees
Labels
documentation Related to documentation of ML.NET usability Smoothing user interaction or experience

Comments

@rogancarr
Copy link
Contributor

NDCG, the normalized discounted cumulative gain metric that we often use in ranking, ranges in value between 0 and 100 in ML.NET. In the wide world, NDCG is often considered to be between 0 and 1 (e.g. see Wikipedia). We should add a note to the ranking metrics object that the values range between 0 and 100 so that we don't confuse people who are new to the toolkit.

@rogancarr rogancarr added the usability Smoothing user interaction or experience label Feb 19, 2019
@TomFinley TomFinley added the documentation Related to documentation of ML.NET label Feb 19, 2019
@justinormont
Copy link
Contributor

Would it be possible to move the range to 0 to 1.0? This would be inline with most of our other metrics, and the industry standard.

@rogancarr
Copy link
Contributor Author

rogancarr commented Feb 20, 2019

It's definitely possible, but it's a breaking change for a lot of folks out there.

@TomFinley what's your thought?

@TomFinley
Copy link
Contributor

TomFinley commented Feb 20, 2019

It would be more consistent to do as @justinormont suggests -- we phrase no other metric as a percentage, as far as I am aware. At the same time I know that the people that use this metrics are more accustomed to having it expressed as a percentage. (Which I assume is what you mean by a breaking change.) But of course, the custom of some people can be adjusted and changed, whereas if we have an inconsistency now we are probably going to commit to it forever. This strikes me as unattractive.

This leads me to prefer the 0 to 1 range. It seems logically defensible and consistent, though I understand it is more work than the documentation change, and might annoy some people in the short term.

If you decide you agree @rogancarr please change this to be under API and project 13 appropriately, since it will then represent a breaking change.

@rogancarr
Copy link
Contributor Author

@TomFinley @justinormont Great! Let's move it from 0 to 1 and add a note to the API docs.

I'll self-assign and send this in shortly.

Maybe we could write up a doc for internal users with a summary of small changes.

@rogancarr rogancarr self-assigned this Feb 20, 2019
@rogancarr
Copy link
Contributor Author

I found two other measures being scaled to percentages: The relative information gains for Binary Classification LogLossReduction and Multiclass Classification Reduction.

@rogancarr
Copy link
Contributor Author

Just a quick note. I implemented the change, and about 41 tests fail. We'll have to remake baselines for a bunch of tests, so it'll be a rather big PR.

@ghost ghost locked as resolved and limited conversation to collaborators Mar 24, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
documentation Related to documentation of ML.NET usability Smoothing user interaction or experience
Projects
None yet
Development

No branches or pull requests

3 participants