Modified how data is saved to disk #4424

bpstark · 2019-10-31T18:19:48Z

pre-trained meta files are now stored in one location always, this
allows multiple runs to re-use the same meta file without having to
redownload.

Additionally added the ability to cleanup the temporary workspace used
to train the model. This should prevent issues of running out of disk
space when running multiple training session sequentially.

codemzs · 2019-10-31T19:56:56Z

src/Microsoft.ML.Vision/ImageClassificationTrainer.cs

+                options.CleanTmpWorkspace = false;
+            }
+
+            string resourcePath = Path.Combine(Path.GetTempPath(), ResourceDir);


string resourcePath [](start = 12, length = 19)

make it a readonly property of the class. #Resolved

codemzs · 2019-10-31T19:57:23Z

src/Microsoft.ML.Vision/ImageClassificationTrainer.cs

@@ -78,6 +78,7 @@ public sealed class ImageClassificationTrainer :
        internal const string UserName = "Image Classification Trainer";
        internal const string ShortName = "IMGCLSS";
        internal const string Summary = "Trains a DNN model to classify images.";
+        internal const string ResourceDir = "MLNET_VISION";


MLNET_VISION [](start = 45, length = 12)

MLNET #Resolved

codemzs · 2019-10-31T20:00:33Z

src/Microsoft.ML.Vision/ImageClassificationTrainer.cs

+            /// Indicates if the temporary workspace should be deleted on cleanup
+            /// </summary>
+            [Argument(ArgumentType.AtMostOnce, HelpText = "Indicates if the temporary workspace should be deleted on cleanup", SortOrder = 15)]
+            public bool CleanTmpWorkspace = true;


remove. #Resolved

codemzs · 2019-10-31T20:01:56Z

src/Microsoft.ML.Vision/ImageClassificationTrainer.cs

+            else
+            {
+                //Since this is not a tmp workspace, but one provided by the user, do not allow deletion.
+                options.CleanTmpWorkspace = false;


add a readonly private variable and set it to true indicating we need to cleanup workspace in the event user does not provide one. #Resolved

codemzs · 2019-10-31T20:06:16Z

using System.Collections.Generic;

add/modify existing tests to check for below:

meta files exist in temp dir/mlnet
the file that contains number of examples also exists in the workspace directory after the run is completed (this is when user provides a workspace).
we need to think how we can test the scenario when workspace is not passed by the user and we create a temp workspace in temp dir and we later clean it.... we need to verify the clean step but I understand getting hold of this temp directory is challenging. #Resolved

Refers to: src/Microsoft.ML.Vision/ImageClassificationTrainer.cs:6 in 9b1d5b1. [](commit_id = 9b1d5b1, deletion_comment = False)

codemzs · 2019-10-31T21:01:50Z

src/Microsoft.ML.Vision/ImageClassificationTrainer.cs

@@ -1123,11 +1138,19 @@ private int GetNumSamples(string path)

            trainSaver.save(_session, _checkpointPath);
            UpdateTransferLearningModelOnDisk(_classCount);
+            CleanUpTmpWorkspace();
+        }


newline. #Resolved

codemzs · 2019-10-31T21:02:04Z

src/Microsoft.ML.Vision/ImageClassificationTrainer.cs

@@ -1123,11 +1138,19 @@ private int GetNumSamples(string path)

            trainSaver.save(_session, _checkpointPath);
            UpdateTransferLearningModelOnDisk(_classCount);
+            CleanUpTmpWorkspace();
+        }
+        private void CleanUpTmpWorkspace()


CleanUpTmpWorkspace() [](start = 21, length = 21)

TryCleanupTemporaryWorkspace #Resolved

codemzs · 2019-10-31T21:03:36Z

src/Microsoft.ML.Vision/ImageClassificationTrainer.cs

        private int _classCount;
        private Graph Graph => _session.graph;

+        private static readonly string _resourcePath = Path.Combine(Path.GetTempPath(), "MLNET");


why is there a newline between this and the previous field? #Resolved

codemzs · 2019-10-31T21:04:18Z

src/Microsoft.ML.Vision/ImageClassificationTrainer.cs

+            else
+            {
+                _cleanupWorkspace = false;
+            }


No need for this, by default in C# boolean variables are initialized to false. #Resolved

eerhardt · 2019-10-31T22:00:22Z

src/Microsoft.ML.Vision/ImageClassificationTrainer.cs

@@ -630,8 +638,9 @@ private protected override ImageClassificationModelParameters TrainModelCore(Tra
                    _inputTensorName, _bottleneckTensor.name, trainSetBottleneckCachedValuesFilePath,
                    ImageClassificationMetrics.Dataset.Train, _options.MetricsCallback);

+                string sizeFile = Path.Combine(_options.WorkspacePath, "TrainingSetSize.txt");


(minor) you may want to extract this into a helper method so it isn't duplicated where this file is, and what its name is.

eerhardt · 2019-10-31T22:01:21Z

src/Microsoft.ML.Vision/ImageClassificationTrainer.cs

+        {
+            if (_cleanupWorkspace && Directory.Exists(_options.WorkspacePath))
+            {
+                Directory.Delete(_options.WorkspacePath, true);


Do you want a try-catch here? Or if this throws an exception, do you want it to kill the training process?

eerhardt

pre-trained meta files are now stored in one location always, this allows multiple runst to re-use the same meta file without having to redownload. Additionally added the ability to cleanup the temporary workspace used to train the model. This should prevent issues of running out of disk space when running multiple training session sequentially.

justinormont · 2019-11-01T00:28:26Z

src/Microsoft.ML.Vision/ImageClassificationTrainer.cs

+
+        private void TryCleanupTemporaryWorkspace()
+        {
+            if (_cleanupWorkspace && Directory.Exists(_options.WorkspacePath))


Assuming you hold the file pointers open, an elegant method is using the FileOptions.DeleteOnClose when you initially open the temporary files.

This then causes the OS (or filesystem when on macOS/Linux) to cleanup the files when they are closed, or if your program crashes or is killed by the user.

https://docs.microsoft.com/en-us/dotnet/api/system.io.fileoptions?view=netcore-3.0

…o not have extras that need to be downloaded.

bpstark · 2019-11-01T17:48:09Z

Note this change will continue to fail until such time as we update the inception meta file in storage

This will fix #4426

codemzs

* Modified how data is saved to disk pre-trained meta files are now stored in one location always, this allows multiple runst to re-use the same meta file without having to redownload. Additionally added the ability to cleanup the temporary workspace used to train the model. This should prevent issues of running out of disk space when running multiple training session sequentially. * fixed extra line error. * changes based on comments. * comment fixes. * address comments. * fixed error * fix for when to download. * Remove special handling of inception as we have fixed the meta file to not have extras that need to be downloaded.

bpstark requested a review from a team as a code owner October 31, 2019 18:19

codemzs reviewed Oct 31, 2019

View reviewed changes

eerhardt reviewed Oct 31, 2019

View reviewed changes

eerhardt approved these changes Oct 31, 2019

View reviewed changes

Brian Stark added 6 commits October 31, 2019 15:16

fixed extra line error.

2f36bb6

changes based on comments.

b99c917

comment fixes.

6b04ce7

address comments.

a5dfb28

fixed error

c3aae8c

justinormont reviewed Nov 1, 2019

View reviewed changes

Brian Stark added 2 commits November 1, 2019 10:04

fix for when to download.

5b0bcf4

Remove special handling of inception as we have fixed the meta file t…

82c6650

…o not have extras that need to be downloaded.

codemzs approved these changes Nov 1, 2019

View reviewed changes

codemzs merged commit 7a8f063 into dotnet:master Nov 1, 2019

bpstark deleted the cleanup branch November 1, 2019 22:03

justinormont mentioned this pull request Apr 7, 2020

add projects capability in CodeGenerator #5002

Merged

ghost locked as resolved and limited conversation to collaborators Mar 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modified how data is saved to disk #4424

Modified how data is saved to disk #4424

bpstark commented Oct 31, 2019

codemzs Oct 31, 2019 •

edited by bpstark

Loading

codemzs Oct 31, 2019 •

edited by bpstark

Loading

codemzs Oct 31, 2019 •

edited by bpstark

Loading

codemzs Oct 31, 2019 •

edited by bpstark

Loading

codemzs commented Oct 31, 2019 •

edited by bpstark

Loading

codemzs Oct 31, 2019 •

edited by bpstark

Loading

codemzs Oct 31, 2019 •

edited by bpstark

Loading

codemzs Oct 31, 2019 •

edited by bpstark

Loading

codemzs Oct 31, 2019 •

edited by bpstark

Loading

eerhardt Oct 31, 2019

eerhardt Oct 31, 2019

eerhardt left a comment

justinormont Nov 1, 2019 •

edited

Loading

bpstark commented Nov 1, 2019

codemzs left a comment

Modified how data is saved to disk #4424

Modified how data is saved to disk #4424

Conversation

bpstark commented Oct 31, 2019

codemzs Oct 31, 2019 • edited by bpstark Loading

Choose a reason for hiding this comment

codemzs Oct 31, 2019 • edited by bpstark Loading

Choose a reason for hiding this comment

codemzs Oct 31, 2019 • edited by bpstark Loading

Choose a reason for hiding this comment

codemzs Oct 31, 2019 • edited by bpstark Loading

Choose a reason for hiding this comment

codemzs commented Oct 31, 2019 • edited by bpstark Loading

codemzs Oct 31, 2019 • edited by bpstark Loading

Choose a reason for hiding this comment

codemzs Oct 31, 2019 • edited by bpstark Loading

Choose a reason for hiding this comment

codemzs Oct 31, 2019 • edited by bpstark Loading

Choose a reason for hiding this comment

codemzs Oct 31, 2019 • edited by bpstark Loading

Choose a reason for hiding this comment

eerhardt Oct 31, 2019

Choose a reason for hiding this comment

eerhardt Oct 31, 2019

Choose a reason for hiding this comment

eerhardt left a comment

Choose a reason for hiding this comment

justinormont Nov 1, 2019 • edited Loading

Choose a reason for hiding this comment

bpstark commented Nov 1, 2019

codemzs left a comment

Choose a reason for hiding this comment

codemzs Oct 31, 2019 •

edited by bpstark

Loading

codemzs Oct 31, 2019 •

edited by bpstark

Loading

codemzs Oct 31, 2019 •

edited by bpstark

Loading

codemzs Oct 31, 2019 •

edited by bpstark

Loading

codemzs commented Oct 31, 2019 •

edited by bpstark

Loading

codemzs Oct 31, 2019 •

edited by bpstark

Loading

codemzs Oct 31, 2019 •

edited by bpstark

Loading

codemzs Oct 31, 2019 •

edited by bpstark

Loading

codemzs Oct 31, 2019 •

edited by bpstark

Loading

justinormont Nov 1, 2019 •

edited

Loading