Skip to content

Make tests baseline comparison more lenient  #619

@sfilipi

Description

@sfilipi

There are a few tests that are disabled because the baseline comparisons fail on the 5th decimal for some of the numbers generated, on some OS.

As an example, the RegressorOlsTest() in PredictorTests fails just on the Mac debug version, because only one out of the generated 4896 predictions doesn't match:

baseline:
2625 5 5.09176636 0.091766357421875 0.0084210643544793129

Mac debug run predictions:
2625 5 5.091751 0.0917510986328125 0.0084182641003280878

I think we should make the tests more lenient to failures like this, modifying the comparison with the baseline to:
1- Have a sensitivity threeshold. Compare up to the 4th, or 6th decimal digit.
2- Count the failures, and declare the test as failed if 3% o the predictions/lines differ?

@justinormont @TomFinley @Zruty0 @zeahmed are those acceptable ranges?

Metadata

Metadata

Assignees

Labels

testrelated to tests

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions