Enable a QuantileRegression Test & Fix Duplicated Baseline Files #1193

eerhardt · 2018-10-09T14:45:24Z

Enable CommandTrainScoreEvaluateQuantileRegression, add the dataset and the necessary baselines.

The only baseline changes were of the form that were caused by dotnet/corefx#31847.

Allow BaseTestBaseline to check Common first.

A lot of baseline tests have duplicated baselines between debug and release. Allow BaseTestBaseline to check the Common baseline directory for a baseline file first.

Also, I cleaned a bunch of dead code from BaseTestBaseline.

Fix #410

Enable CommandTrainScoreEvaluateQuantileRegression, add the dataset and the necessary baselines. The only baseline changes were of the form that were caused by https://github.com/dotnet/corefx/issues/31847.

A lot of baseline tests have duplicated baselines between debug and release. Allow BaseTestBaseline to check the Common baseline directory for a baseline file first. Fix dotnet#410

eerhardt · 2018-10-09T14:50:24Z

/cc @justinormont - in case you want to review the Boston Housing dataset attribution. #Closed

eerhardt · 2018-10-09T14:52:23Z

test/Microsoft.ML.TestFramework/BaseTestBaseline.cs

@@ -838,7 +748,7 @@ protected static StreamReader OpenReader(string path)
        {
            Contracts.CheckNonWhiteSpace(path, nameof(path));
 #if CORECLR
-            return new StreamReader(File.Open(path, FileMode.Open, FileAccess.Read));
+            return new StreamReader(File.Open(path, FileMode.Open, FileAccess.Read, FileShare.Read));


FYI - this is a nicety when debugging tests and trying to use Beyond Compare (or similar) to diff the output and baseline files. As currently implemented, it is impossible to open these files in 2 processes at the same time. #Resolved

sfilipi · 2018-10-09T16:30:38Z

test/data/housing.txt

@@ -0,0 +1,506 @@
+24.00	0.00632	18.00	2.310	0	0.5380	6.5750	65.20	4.0900	1	296.0	15.30	396.90	4.98
+21.60	0.02731	0.00	7.070	0	0.4690	6.4210	78.90	4.9671	2	242.0	17.80	396.90	9.14


Last time i asked about this dataset, I believe we couldn't use it.
I switched my test to the wine quality.
Might have to use the regression dataset that is in the repo already.

@shauheen @artidoro to comment further #Resolved

From all I could find online, this dataset appears to be in the public domain.

This site claims the dataset is in the public domain:
http://b-course.hiit.fi/obc/housingexpl.html
“ B-Course provides a public domain data set called Boston Housing Data so that one can try out B-Course without own data. Below you can find some details of our example data set.”

The dataset is also distributed in an R package here:
https://www.rdocumentation.org/packages/mlbench/versions/2.1-1

It's used in books as well: https://github.com/rasbt/python-machine-learning-book-2nd-edition/blob/master/code/ch10/housing.data.txt

I've added the attribution to the README.

By the way, this is a different dataset from the "King County Housing" (kc_housing) dataset. #Closed

CELA has approved the re-distribution of the Boston Housing dataset. #Closed

sfilipi · 2018-10-09T16:32:18Z

test/Microsoft.ML.TestFramework/BaseTestBaseline.cs

-        protected string ExternalTestBinariesDir { get { return Path.Combine(RootDir, TestExtDir); } }
-
-        protected string TestDirectory { get { return Path.Combine(RootDir, TestDir); } }
-


cc @yaeldekel to check whether this conflicts with her work. #Resolved

Thanks for thinking of that work, @sfilipi.

If there are things that are only used in the internal repo, we should move them to only be in the internal repo. It isn't sustainable to have a bunch of methods in dotnet/machinelearning that are not used.

@yaeldekel - if any of these are necessary for your work, can you add them to a partial class in the internal repo? #Resolved

sfilipi

Anipik

LGTM

justinormont · 2018-10-09T17:55:42Z

test/data/housing.txt

@@ -0,0 +1,506 @@
+24.00	0.00632	18.00	2.310	0	0.5380	6.5750	65.20	4.0900	1	296.0	15.30	396.90	4.98


Would it be reasonable to add a header to the file to help users understand it:
CrimesPerCapita PercentResidental PercentNonRetail CharlesRiver NitricOxides RoomsPerDwelling PercentPre40s EmploymentDistance HighwayDistance TaxRate TeacherRatio BlackIndex PercentLowIncome MedianHomeValue

The headed is also used when producing the feature importance as it will then print the column headers (textual form) instead of just a semi-opaque 'F13'. #Resolved

Done. #Closed

justinormont · 2018-10-09T18:00:53Z

test/data/README.md

@@ -65,6 +65,12 @@ The dataset is under a CC-by 4.0 license.
 }
 ```

+### Boston Housing Data


Can you add a more info link? I also added a link to our copy of the dataset so folks know what we are referencing.

### Boston Housing Data Redistributing the dataset "[housing.txt](housing.txt)" with attribution: > Harrison, D. and Rubinfeld, D.L. 'Hedonic prices and the demand for clean air', J. Environ. Economics & Management, vol.5, 81-102, 1978. More information: http://b-course.hiit.fi/obc/housingexpl.html ``` #Resolved

Done. #Closed

artidoro

Add more info to the README. Add column headers to the housing.txt data file.

justinormont · 2018-10-09T21:44:21Z

Closing/re-opening to restart CI tests

Ivanidzo4ka

eerhardt added 2 commits October 8, 2018 16:07

Enable a QuantileRegression test

ba1b008

Enable CommandTrainScoreEvaluateQuantileRegression, add the dataset and the necessary baselines. The only baseline changes were of the form that were caused by https://github.com/dotnet/corefx/issues/31847.

Allow BaseTestBaseline to check Common first.

1ac7164

A lot of baseline tests have duplicated baselines between debug and release. Allow BaseTestBaseline to check the Common baseline directory for a baseline file first. Fix dotnet#410

eerhardt requested review from codemzs, Ivanidzo4ka, tannergooding, Anipik and sfilipi October 9, 2018 14:45

eerhardt commented Oct 9, 2018

View reviewed changes

sfilipi reviewed Oct 9, 2018

View reviewed changes

sfilipi approved these changes Oct 9, 2018

View reviewed changes

Anipik approved these changes Oct 9, 2018

View reviewed changes

justinormont reviewed Oct 9, 2018

View reviewed changes

artidoro approved these changes Oct 9, 2018

View reviewed changes

PR feedback

259046e

Add more info to the README. Add column headers to the housing.txt data file.

justinormont closed this Oct 9, 2018

justinormont reopened this Oct 9, 2018

Ivanidzo4ka approved these changes Oct 9, 2018

View reviewed changes

eerhardt merged commit 024bd44 into dotnet:master Oct 10, 2018

Ivanidzo4ka mentioned this pull request Oct 12, 2018

Fix ResultProcessor bug, LogisticRegression bug and missing value conversion bug #1236

Merged

eerhardt deleted the QuantileRegressionTest branch November 7, 2018 14:56

ghost locked as resolved and limited conversation to collaborators Mar 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable a QuantileRegression Test & Fix Duplicated Baseline Files #1193

Enable a QuantileRegression Test & Fix Duplicated Baseline Files #1193

eerhardt commented Oct 9, 2018

eerhardt commented Oct 9, 2018 •

edited

Loading

eerhardt Oct 9, 2018 •

edited

Loading

sfilipi Oct 9, 2018 •

edited by eerhardt

Loading

eerhardt Oct 9, 2018 •

edited

Loading

justinormont Oct 9, 2018 •

edited by eerhardt

Loading

sfilipi Oct 9, 2018 •

edited by eerhardt

Loading

eerhardt Oct 9, 2018 •

edited

Loading

sfilipi left a comment

Anipik left a comment

justinormont Oct 9, 2018 •

edited by eerhardt

Loading

eerhardt Oct 9, 2018 •

edited

Loading

justinormont Oct 9, 2018 •

edited by eerhardt

Loading

eerhardt Oct 9, 2018 •

edited

Loading

artidoro left a comment

justinormont commented Oct 9, 2018

Ivanidzo4ka left a comment

		@@ -0,0 +1,506 @@
		24.00 0.00632 18.00 2.310 0 0.5380 6.5750 65.20 4.0900 1 296.0 15.30 396.90 4.98
		21.60 0.02731 0.00 7.070 0 0.4690 6.4210 78.90 4.9671 2 242.0 17.80 396.90 9.14

		protected string ExternalTestBinariesDir { get { return Path.Combine(RootDir, TestExtDir); } }

		protected string TestDirectory { get { return Path.Combine(RootDir, TestDir); } }

Enable a QuantileRegression Test & Fix Duplicated Baseline Files #1193

Enable a QuantileRegression Test & Fix Duplicated Baseline Files #1193

Conversation

eerhardt commented Oct 9, 2018

eerhardt commented Oct 9, 2018 • edited Loading

eerhardt Oct 9, 2018 • edited Loading

Choose a reason for hiding this comment

sfilipi Oct 9, 2018 • edited by eerhardt Loading

Choose a reason for hiding this comment

eerhardt Oct 9, 2018 • edited Loading

Choose a reason for hiding this comment

justinormont Oct 9, 2018 • edited by eerhardt Loading

Choose a reason for hiding this comment

sfilipi Oct 9, 2018 • edited by eerhardt Loading

Choose a reason for hiding this comment

eerhardt Oct 9, 2018 • edited Loading

Choose a reason for hiding this comment

sfilipi left a comment

Choose a reason for hiding this comment

Anipik left a comment

Choose a reason for hiding this comment

justinormont Oct 9, 2018 • edited by eerhardt Loading

Choose a reason for hiding this comment

eerhardt Oct 9, 2018 • edited Loading

Choose a reason for hiding this comment

justinormont Oct 9, 2018 • edited by eerhardt Loading

Choose a reason for hiding this comment

eerhardt Oct 9, 2018 • edited Loading

Choose a reason for hiding this comment

artidoro left a comment

Choose a reason for hiding this comment

justinormont commented Oct 9, 2018

Ivanidzo4ka left a comment

Choose a reason for hiding this comment

eerhardt commented Oct 9, 2018 •

edited

Loading

eerhardt Oct 9, 2018 •

edited

Loading

sfilipi Oct 9, 2018 •

edited by eerhardt

Loading

eerhardt Oct 9, 2018 •

edited

Loading

justinormont Oct 9, 2018 •

edited by eerhardt

Loading

sfilipi Oct 9, 2018 •

edited by eerhardt

Loading

eerhardt Oct 9, 2018 •

edited

Loading

justinormont Oct 9, 2018 •

edited by eerhardt

Loading

eerhardt Oct 9, 2018 •

edited

Loading

justinormont Oct 9, 2018 •

edited by eerhardt

Loading

eerhardt Oct 9, 2018 •

edited

Loading