Port SymSGD #556

codemzs · 2018-07-19T02:30:07Z

This change adds parallel SGD trainer.
fixes #623

TomFinley · 2018-07-19T07:13:18Z

src/Native/SymSgdNative/SparseBLAS.h

+#include "../Stdafx.h"
+#include "mkl.h"
+#ifndef COMPILER_GCC
+#pragma comment(lib, "../../../Libraries/MKL/Win/Microsoft.ML.MklImports.lib")


Libraries/MKL/Win/Microsoft.ML.MklImports.lib [](start = 31, length = 45)

Hi @codemzs thanks for looking at this. We need to either shift this thing to not use MKL, or port the libraries. However, considering that this involves the static linking of the library, and all other usages are using it as a dynamic library, this becomes especially awkward here. #Resolved

I wonder if temporarily (or not so temporarily) we can shift to using regular ops rather than calling MKL, based on some compile time flag (as we do elsewhere). If on, then it will compile with MKL support. If off, it does it manually. Since these are ops like SAXPYI and whatnot, it's not like they would be incredibly hard to support. #Resolved

Hi @TomFinley , There is a PR out that creates the MKL nuget. I have tested the native symSGD code by linking against the MKL binaries assuming they are present in the packages directory. #Resolved

OK. Once that PR is in, assuming it works, you would add the nuget dependency in which project?

In reply to: 205300173 [](ancestors = 205300173)

Microsoft.ML.Tests and Microsoft.ML.Predictor.Test. Anything that uses sym sgd would need the reference to the nuget so that binaries are copied over to the project folder in bin directory.

In reply to: 205467167 [](ancestors = 205467167,205300173)

We have MKL nuget now.

In reply to: 203788370 [](ancestors = 203788370)

TomFinley · 2018-07-26T13:55:50Z

src/Native/SymSgdNative/CMakeLists.txt

+    link_directories(${CMAKE_SOURCE_DIR}/../../packages/MlNetMklDeps/runtimes/win-x64/native)
+else()
+    list(APPEND SOURCES ${VERSION_FILE_PATH})
+	if(APPLE)


[](start = 0, length = 1)

Consistent indentation please. #Resolved

TomFinley · 2018-07-26T13:56:26Z

src/Native/SymSgdNative/SymSgdNative.h

+class SymSGD {
+private:
+	int _numFreqFeat;
+	// Local models that is learned


[](start = 0, length = 1)

Please shift these to use spaces for indentation. #Resolved

TomFinley · 2018-07-26T13:57:07Z

test/Microsoft.ML.Predictor.Tests/Microsoft.ML.Predictor.Tests.csproj

@@ -27,6 +27,7 @@
    <NativeAssemblyReference Include="FastTreeNative" />
    <NativeAssemblyReference Include="CpuMathNative" />
    <NativeAssemblyReference Include="FactorizationMachineNative" />
+	<NativeAssemblyReference Include="SymSgdNative" />


[](start = 0, length = 1)

Yet another place where I see tabs... #Resolved

Ivanidzo4ka · 2018-07-26T17:28:19Z

You probably want new version of SymSGD with respectful changes in ITrainer. Which can be found in PR towards internal repo

codemzs · 2018-07-26T17:54:01Z

@Ivanidzo4ka This PR consumes the latest ITrainer. The code would have not compiled and tests would have not run otherwise, right? Is there anything specific that you are referring to?

In reply to: 408173274 [](ancestors = 408173274)

Ivanidzo4ka · 2018-07-26T17:58:28Z

From what I discover yesterday, you can't just take this code, and update internal repository. SymSgd internally is in separate package, and visibility of LinearTrainerBase don't allow it to be referenced from other projects.
Also from discussion with @TomFinley SymSGD in general shouldn't be based on LinearTrainerBase, but just TrainerBase.

In reply to: 408181017 [](ancestors = 408181017,408173274)

Ivanidzo4ka · 2018-07-26T21:05:04Z

src/Native/build.proj

@@ -90,6 +90,8 @@
                          RelativePath="Microsoft.ML\runtimes\$(PackageRid)\native" />
      <NativePackageAsset Include="$(NativeAssetsBuiltPath)\$(NativeLibPrefix)FactorizationMachineNative$(NativeLibExtension)"
                          RelativePath="Microsoft.ML\runtimes\$(PackageRid)\native" />
+      <NativePackageAsset Include="$(NativeAssetsBuiltPath)\$(NativeLibPrefix)SymSgdNative$(NativeLibExtension)"
+                          RelativePath="Microsoft.ML\runtimes\$(PackageRid)\native" />						  


[](start = 86, length = 8)

something, something, tabs #Resolved

Ivanidzo4ka · 2018-07-26T21:07:34Z

src/Native/SymSgdNative/Macros.h

+// See the LICENSE file in the project root for more information.
+
+#pragma once
+#define MAX(__X__, __Y__) (((__X__) < (__Y__)) ? (__Y__) : (__X__))


MAX never get used, do we need it? #Resolved

Ivanidzo4ka · 2018-07-26T21:08:52Z

src/Native/SymSgdNative/SymSgdNative.h

@@ -0,0 +1,127 @@
+#pragma once


header? #Resolved

codemzs · 2018-07-26T23:27:03Z

I don't follow, can you please comment on SymSgdClassificationTrainer.cs and clarify?

In reply to: 408182382 [](ancestors = 408182382,408181017,408173274)

sfilipi · 2018-07-31T17:27:09Z

src/Microsoft.ML/Microsoft.ML.csproj

@@ -7,6 +7,7 @@
  </PropertyGroup>

  <ItemGroup>
+    <PackageReference Include="MlNetMklDeps" Version="0.0.0.1" />


[](start = 2, length = 63)

note about having to split SymSGD this as a separate project, rather than adding the dependency here, based on @eerhardt feedback on PR #594

Correct. We shouldn't be referencing MlNetMklDeps from the Microsoft.ML nuget package.

sfilipi · 2018-07-31T17:34:25Z

test/Microsoft.ML.Predictor.Tests/TestAutoInference.cs

+                RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
+            {
+                return;
+            }
            using (var env = new TlcEnvironment())


maybe log an issue about re-enabling.

Why are we disabling existing tests when we are adding new functionality? That seems wrong.

sfilipi · 2018-07-31T17:34:46Z

test/Microsoft.ML.Predictor.Tests/TestPipelineSweeper.cs

+                RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
+            {
+                return;
+            }


same, might be worth logging an issue.

sfilipi · 2018-07-31T17:37:18Z

src/Microsoft.ML/CSharpApi.cs

+
+        /// <summary>
+        /// Train a symbolic SGD.
+        /// </summary>


Not sure how much documentation we have about SymSGD. If any, might be nice to include it.

eerhardt · 2018-07-31T17:44:32Z

src/Native/SymSgdNative/CMakeLists.txt

+)
+
+if(WIN32)
+    find_library(MKL_LIBRARY Microsoft.ML.MklImports HINTS ${CMAKE_SOURCE_DIR}/../../packages/mlnetmkldeps/0.0.0.1/runtimes/win-x64/native)


Like we discussed offline, we shouldn't be hard-coding these paths and version numbers into our CMake scripts.

eerhardt · 2018-07-31T17:45:10Z

src/Native/SymSgdNative/CMakeLists.txt

+        find_library(MKL_LIBRARY Microsoft.ML.MklImports.dylib HINTS "${CMAKE_SOURCE_DIR}/../../packages/mlnetmkldeps/0.0.0.1/runtimes/osx-x64/native")
+    else()
+        message("Linking SymSgdNative with MKL on linux.")
+        link_directories(${CMAKE_SOURCE_DIR}/../../packages/mlnetmkldeps/0.0.0.1/runtimes/linux-x64/native)


Why do you need both this and the HINTS below? Isn't one of them enough?

eerhardt · 2018-07-31T17:45:57Z

src/Native/build.cmd

@@ -106,6 +106,15 @@ if exist "%__IntermediatesDir%\INSTALL.vcxproj" goto BuildNativeProj
 goto :Failure

 :BuildNativeProj
+echo Copying MKL library in bin folder. This is a temporary fix.


We should remove this.

eerhardt · 2018-07-31T17:46:18Z

src/Native/build.sh

@@ -93,8 +93,37 @@ __cmake_defines="${__cmake_defines} -DVERSION_FILE_PATH:STRING=${__versionSource

 cd "$__IntermediatesDir"

+#codemzs: temporary fix until mkl nuget binaries are properly renamed so that they can be consumed by CMAKE.


We should also remove this. And the one below.

eerhardt · 2018-07-31T17:48:10Z

test/Microsoft.ML.Tests/Microsoft.ML.Tests.csproj

@@ -21,5 +21,10 @@
    <NativeAssemblyReference Include="CpuMathNative" />
    <NativeAssemblyReference Include="FastTreeNative" />
    <NativeAssemblyReference Include="LdaNative" />
+	<NativeAssemblyReference Include="SymSgdNative" />


(nit) whitespace is off.

eerhardt · 2018-07-31T17:48:51Z

test/Microsoft.ML.Tests/Microsoft.ML.Tests.csproj

+  </ItemGroup>
+
+  <ItemGroup>
+    <PackageReference Include="MlNetMklDeps" Version="0.0.0.1" />


0.0.0.1 - this version number should go in 1 place - in the https://github.com/dotnet/machinelearning/blob/master/build/Dependencies.props file.

* Allow CpuMath to reference C# Hardware Intrinsics APIs. Need to multi-target CpuMath for netstandard and netcoreapp3.0. Also, since we are going to move CpuMath into its own NuGet package, remove the dependency from CpuMath to the ML.Core project. Add a build parameter to enable building against .NET Core 3.0's Runtime Intrinsics APIs. Fix dotnet#534 * Respond to PR feedback.

* failing test case for multiclass * Refactored PipelineSweeperSupportedMetrics Class; added unit test for MultiClassClassification; refactored out unit tests for the PipelineSweeper * take care of review comments; display transforms/learners + metrics in pipeline * taking care of PR comments + refactor PipelineSweeperRunSummary * taking care of review comments

* remove domain from onnx operators for non-ML types. * Make ONNX compatible with Windows RS5 and add more tests. * PR feedback. * PR feedback. * fix build.

* Remove Windows and Linux configurations from netci.groovy * Add end of line to yml files * Add badges and change leg name to Linux * Not merge test results * Add searchFolder to publish test results task

add proper fields to trainer info for FT, FM, and OnlineLearner regarding incremental training and validation datasets

* Fix warning for L2 in SDCA * String interpolation * Word-smithing the warning message

Introduce word embedding transform

Add images support based on System.Drawing

…#611) * moving Ols to a separate project * Revert "moving Ols to a separate project" This reverts commit 9b7eab3. * separating OLS in its own project * adding nupkgproj files to create a package for AdditionalLearners adding AdditionalLearners to the core.tests project fixing the core_ep and core_manifest * CSharpApi should not get generated every time. * Addressing Ivan's comments * referencing package version 0.0.0.4 of MlNetMklDeps that contains new names for the mkl library. * Correcting the error message. * renaming AdditionalLearners to HalLearners < - Hardware Accelerated Learners removign unsafe from the Hal csproj. removing the orphaned member section from the doc.xml * referencing package 0.0.0.5 and updating the name. * regenerating the CsharpApi and the eplist post merge * regenerate the CSharpApi and the eplists post merge. Fix the namespace post merge. * typo * one shall not space * spacing

…to symsgd # Conflicts: # src/Microsoft.ML.HalLearners/Microsoft.ML.HalLearners.csproj # src/Microsoft.ML.HalLearners/doc.xml # src/Microsoft.ML/CSharpApi.cs # test/BaselineOutput/Common/EntryPoints/core_manifest.json

eerhardt · 2018-08-01T23:30:38Z

This can be closed. We merged the other PR

Initial commit plus use ITrainer that returns predictor in SymSGD.

8cdbf0f

TomFinley reviewed Jul 19, 2018

View reviewed changes

TomFinley mentioned this pull request Jul 24, 2018

Adding the MlNetMklDeps package, adding the entry point, tests and docs for OLS. #578

Closed

Add tests.

ca13e8d

codemzs requested a review from eerhardt July 26, 2018 00:27

TomFinley reviewed Jul 26, 2018

View reviewed changes

Ivanidzo4ka reviewed Jul 26, 2018

View reviewed changes

src/Native/SymSgdNative/SymSgdNative.h

@@ -0,0 +1,127 @@

#pragma once

Copy link

Contributor

Ivanidzo4ka Jul 26, 2018 •

edited by codemzs

Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

header? #Resolved

codemzs changed the title ~~WIP Port SymSGD~~ Port SymSGD Jul 31, 2018

sfilipi reviewed Jul 31, 2018

View reviewed changes

eerhardt reviewed Jul 31, 2018

View reviewed changes

eerhardt and others added 4 commits August 1, 2018 04:02

Ensure ONNX export is compatible with Windows RS5 (dotnet#550)

ffeb1c9

* remove domain from onnx operators for non-ML types. * Make ONNX compatible with Windows RS5 and add more tests. * PR feedback. * PR feedback. * fix build.

Move Windows and Linux CI to VSTS (dotnet#566)

91eb4fd

* Remove Windows and Linux configurations from netci.groovy * Add end of line to yml files * Add badges and change leg name to Linux * Not merge test results * Add searchFolder to publish test results task

codemzs and others added 23 commits August 1, 2018 04:02

load mkl native before symSGD is loaded.

8f7e497

load mkl native before symSGD is loaded.

5667701

load mkl native before symSGD is loaded.

e830980

support validation and incremental trainers (dotnet#610)

bc87d62

add proper fields to trainer info for FT, FM, and OnlineLearner regarding incremental training and validation datasets

Add needed param to warning for L2 in SDCA (dotnet#597)

966b0f3

* Fix warning for L2 in SDCA * String interpolation * Word-smithing the warning message

word embedding transform (dotnet#545)

52d96b4

Introduce word embedding transform

Image support (dotnet#528)

2cb7dc6

Add images support based on System.Drawing

load mkl native before symSGD is loaded.

767800c

load mkl native before symSGD is loaded.

89c06c0

fixes.

41cba77

Rpath for linux and os x.

4b27ec8

Rpath for linux and os x.

be8c6c8

Rpath for linux and os x.

6fe7e1a

load mkl native before symSGD is loaded.

2d8c15b

Rpath for linux and os x.

b9fe01d

Rpath for OS X

ebd5bcd

Rpath for linux and os x.

de999a8

Rpath for linux and os x.

573781d

Rpath for linux and os x.

9d8f297

Rpath for linux and os x.

1b4273a

move symsgd under HALLearners.

294c3d7

documentation.

6ec9d70

codemzs force-pushed the symsgd branch from 2425485 to 6ec9d70 Compare August 1, 2018 11:28

codemzs mentioned this pull request Aug 1, 2018

Port SymSGD trainer #624

Merged

eerhardt closed this Aug 1, 2018

codemzs deleted the symsgd branch August 3, 2018 06:03

ghost locked as resolved and limited conversation to collaborators Mar 29, 2022

		@@ -93,8 +93,37 @@ __cmake_defines="${__cmake_defines} -DVERSION_FILE_PATH:STRING=${__versionSource

		cd "$__IntermediatesDir"

		#codemzs: temporary fix until mkl nuget binaries are properly renamed so that they can be consumed by CMAKE.

Port SymSGD #556

Port SymSGD #556

Uh oh!

Conversation

codemzs commented Jul 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomFinley Jul 19, 2018 • edited by codemzs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TomFinley Jul 19, 2018 • edited by codemzs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codemzs Jul 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TomFinley Jul 26, 2018 • edited by codemzs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TomFinley Jul 26, 2018 • edited by codemzs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TomFinley Jul 26, 2018 • edited by codemzs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ivanidzo4ka commented Jul 26, 2018

Uh oh!

codemzs commented Jul 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ivanidzo4ka commented Jul 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ivanidzo4ka Jul 26, 2018 • edited by codemzs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ivanidzo4ka Jul 26, 2018 • edited by codemzs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ivanidzo4ka Jul 26, 2018 • edited by codemzs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codemzs commented Jul 26, 2018

Uh oh!

sfilipi Jul 31, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Jul 31, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

codemzs commented Jul 19, 2018 •

edited

Loading

TomFinley Jul 19, 2018 •

edited by codemzs

Loading

TomFinley Jul 19, 2018 •

edited by codemzs

Loading

codemzs Jul 26, 2018 •

edited

Loading

TomFinley Jul 26, 2018 •

edited by codemzs

Loading

TomFinley Jul 26, 2018 •

edited by codemzs

Loading

TomFinley Jul 26, 2018 •

edited by codemzs

Loading

codemzs commented Jul 26, 2018 •

edited

Loading

Ivanidzo4ka commented Jul 26, 2018 •

edited

Loading

Ivanidzo4ka Jul 26, 2018 •

edited by codemzs

Loading

Ivanidzo4ka Jul 26, 2018 •

edited by codemzs

Loading

Ivanidzo4ka Jul 26, 2018 •

edited by codemzs

Loading

sfilipi Jul 31, 2018 •

edited

Loading

sfilipi Jul 31, 2018 •

edited

Loading