Significant time difference between linux, macos and windows test executions on build machines

[Ivanidzo4ka]

Let's look on this build

MacOS Release - tests: 19:08
MacOS Debug - tests: 22:59
Linux Release - test 17:02
Linux Debug - test 21:02
Windows Release - test 5:00
Windows Debug - test 7:53

Windows machine:
2018-09-14T22:47:22.5353796Z Results File: D:\a\1\s\bin/AnyCPU.Debug\Microsoft.ML.Predictor.Tests\VssAdministrator_factoryvm-az366_2018-09-14_22_43_12.trx
2018-09-14T22:47:22.5354312Z
2018-09-14T22:47:22.5355746Z Total tests: 105. Passed: 49. Failed: 0. Skipped: 56.
2018-09-14T22:47:22.5356035Z Test Run Successful.
2018-09-14T22:47:22.5357667Z Test execution time: 4.2240 Minutes

Linux machine:

2018-09-14T22:59:26.1088812Z Results File: /__w/1/s/bin/AnyCPU.Debug/Microsoft.ML.Predictor.Tests/_5438db60ae8a_2018-09-14_22_41_24.trx
2018-09-14T22:59:26.1096444Z
2018-09-14T22:59:26.1109704Z Total tests: 105. Passed: 49. Failed: 0. Skipped: 56.
2018-09-14T22:59:26.1124041Z Test Run Successful.
2018-09-14T22:59:26.1137781Z Test execution time: 18.0608 Minutes

It looks like we run same set of tests but for some reason where is huge difference in execution.
Any one willing to investigate?

[Ivanidzo4ka]

As an example LinearClassifier test:

[Ivanidzo4ka]

@shauheen @Zruty0 @eerhardt
Can we assign someone to take a look at this? Maybe it's a false alarm and we just forgot to turn some knob, but even if so, we would get faster builds. And if it's not configuration issue, it's looks really bad if our Linux and MacOS implementation 30 times slower.

[Ivanidzo4ka]

I think I found a winner:

[TomFinley]

Do we have any idea why the categorical split test is so much worse?

/cc @codemzs

[eerhardt]

@artidoro - any update on investigating why FastTreeBinaryClassificationCategoricalSplitTest takes so long on Linux and macOS?

[Zruty0]

#1131 may be related in some way

[Zruty0]

@artidoro , are you looking at it now?

[artidoro]

Yes I am going to take a look today, and update the thread.

[artidoro]

I looked at 10 different test runs in all OS and for all versions from the PR verification builds. These are the summaries of the results that I obtained:

	linux deb	linux rel	win x64 deb	win x64 rel	win x86 deb	win x86 rel	mac deb	mac rel
mean run tests time (min)	11.3	7.3	16	7.7	10.8	8.5	12.4	9.9
standard dev	0.41	1.73	5.79	0.55	0.80	3.39	1.98	2.26

The above preliminary analysis seems to show that there is no significant slowdown on mac and linux compared to windows. For a more detailed analysis, we should make sure the machines have the same specifications (I just used the results of PR verification builds).

This has probably been solved by #1388, which fixed a bug in the way we were comparing test results with our baselines.