lack of feedbak from the ITrainer api becomes an issue

[PeterPann23]

System information

• OS version/distro: Windows 10
• .NET Version (eg., dotnet --info): Core 3.0, preview 1.0 of ML

Issue

Working larger datasets it becomes important to get a progress on what the system is doing.
I am testing the framework against a smaller set of production data (21 days@23:45:00 per day), The smaller training file is ~300 GB and 35630 columns, depending on the multiclass label to predict.

There is not only no documentation in regards to sizing there is also no progress report or early stopping method that one can hook into, or is there?.

One can limit the amount of iterations, however it would be far cooler to be able to hooking to an Iteration have an event that states the current metric as well as a "Cancel" property. there are several events that use such hooks like Form.OnClosing.

Having a cancel would ideally have the model fitted in such a state that one can save it. Ideally I would suggest to have "duration of training" as well as "Iteration performed" in the event. this would nicely hook into progress reporting tools.

I guess it could be a part of IProgressChannel?
Would be nice if this could be considered

[glebuk]

@{PeterPann23

Cancellation

I might have some good news for you. Indeed there is a way to cancel training. We have this feature in the experimental NuGet. note that this will cancel, not stop training.
No warranties implied, use at your own risk! See https://www.nuget.org/packages/Microsoft.ML.Experimental

Logging

Are you subscribing to our logger per chance? There is a way to register a listener for logger to see the messages that are being sent by the trainer. See '''public event EventHandler Log;''' on the context object.

Alternative to Cancellation

Some learners allow continuous training, that means that you can train, stop, save the model, then start training using the trained weights and do this in a loop until desired result is achieved.

Speed up training

Are you as fast as you can be? There are techniques to speed up training significantly. Those would be covered in future documentation, but for now here are few:

1. Ensure to load data as vectors. Meaning if you have 2000k numerical features, load them as a single vector column. all features that are treated the same and are of the same data type should be loaded into a single vector. this gives 10-10000x speedup.
2. for iterative trainers add caching checkpoit before the training. If data is too big to fit into RAM, then save it to Binary format after featurization
3. If you read the same data a lot, convert to Binary IDataView format using readers and writers under context.Data, after vectorizing the data.
4. If futurizing multiple columns, consider using column pairs to apply the same operation on several columns at a time
5. Pick fast trainers like SymSGD, SDCA and AveragedPerceptron.

Hope that helps.

[PeterPann23]

Continue Training would be really cool, I did not find any documentation on this… where do I Need to look? I tried the binary in release 0.7 usually was corrupt so I gave up on it for a while, perhaps I should re-visit it. Gesendet von Mail<https://go.microsoft.com/fwlink/?LinkId=550986> für Windows 10

…

________________________________ Von: Gleb K <notifications@github.com> Gesendet: Friday, April 19, 2019 8:16:50 AM An: dotnet/machinelearning Cc: PeterPann23; Author Betreff: Re: [dotnet/machinelearning] lack of feedbak from the ITrainer api becomes an issue (#3358) @{PeterPann23 Cancellation I might have some good news for you. Indeed there is a way to cancel training. We have this feature in the experimental NuGet. note that this will cancel, not stop training. No warranties implied, use at your own risk! See https://www.nuget.org/packages/Microsoft.ML.Experimental Logging Are you subscribing to our logger per chance? There is a way to register a listener for logger to see the messages that are being sent by the trainer. See '''public event EventHandler Log;''' on the context object. Alternative to Cancellation Some learners allow continuous training, that means that you can train, stop, save the model, then start training using the trained waits and do this in a loop until desired result is achived Speed up training Are you as fast as you can be? There are techniques to speed up training significantly. Those would be covered in future documentation, but for now here are few: 1. Ensure to load data as vectors. Meaning if you have 2000k numerical features, load them as a single vector column. all features that are treted the same and are of the same data type should be loaded into a single vector. this gives 10-10000x speedup. 2. for iterative trainers add caching checkpoit before the training. If data is too big to fit into RAM, then save it to Binary format after featurization 3. If you read the same data a lot, convert to Binary IDataView format using readers and writers under context.Data, after vectorizing the data. 4. If futurizing multiple columns, consider using column pairs to apply the same operation on several columns at a time 5. Pick fast trainers like SymSGD, SDCA and AveragedPerceptron. Hope that helps. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#3358 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AKSYBNRYO6BA7NISKBC6T5LPRFPVFANCNFSM4HGNWYRA>.

[PeterPann23]

The log callback you mentioned is "old", also doesn't do much in ways of providing feedback.
Better would be that it shows the tasks doing and those outstanding for the current pipeline.

And when it is training a callback to the pipeline that shows the progress of each Iteration would be valuable as one can graph the curve of progress and over/ under fitting. It's all a back black box now and it doesn't have to be.

[glebuk]

Please take a look at https://github.com/dotnet/machinelearning/blob/master/test/Microsoft.ML.Functional.Tests/Training.cs for sample continued training. We often have a lot more samples in the tests of the codebase, in case docs are lacking. we are working on getting the docs updated though...
The suggestion of iteration update is great, and we were thinking of doing this in the future. In the meanwhile you can try to do this manually. Note that some learners, like FastTree have early stopping criteria that allow it to automatically stop training when it starts overfitting by comparing score on training and validation sets - https://github.com/dotnet/machinelearning/blob/master/test/Microsoft.ML.Core.Tests/UnitTests/TestEarlyStoppingCriteria.cs

[PeterPann23]

Hi Glebuk

Thanks for these links, I will have a look, at the moment I look at the Ram cycles to loog at GC memeory allocations peaks and lower oscilatioms to see when it release memory.

The log on MLContext can go for hours without providing any feedback. The provided log with crash does not hind why it crashed and the still running log has not posted a update for the last 10 hours as per writing this message.

I am glad that you understand my pain; from a production standpoint this is really bad as I have to fallback to monitoring CPU and RAM to see if I need to notify IT to investigate a failed/ hanging job. When developing it doesn't really matter as we work with smaller files and training takes a few seconds. handing a package over to production becomes rather "diplomatic" as there is no way to integrate it in monitoring tool. At the moment I have a background thread writing a fake status hart-beat.