Skip to content

How to get training progress info? #3752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
seabluescn opened this issue May 21, 2019 · 5 comments
Closed

How to get training progress info? #3752

seabluescn opened this issue May 21, 2019 · 5 comments
Labels
usability Smoothing user interaction or experience

Comments

@seabluescn
Copy link

Hi,Suppose I have 10,000 training data. When I run Fit method, it takes a long time. During this period, how can I get real-time training progress information?
Thank You!

@PeterPann23
Copy link

Hi,

I have opened 2 issue on this topic that are currently under review #3685 and #3358.

You will find some comments in these threads that perhaps can be used for some sort of activity monitoring.

The short answer however is no, even though there are some ways to get feed back, none are really helping determine what's going on.

You have

  1. Use the event MLContext.Log that will give you some text that you can use to parse

2.a You can, if the network you choose use early stopping to help mitigate wasting resources (doesn't tell you your progress though)

2.b Stop a training session periodically, test the current metrics and continue as shown here. Again not really progress reporting

  1. Hoockup an WithOnFitDelegate that get's called by the network, most of them however do not loop and only call it when trained, some of them do. Have a look at the Microsoft.ML.LearningPipelineExtentsions line 138 for more info

  2. Pull the source and alter the source code and publish progress using a delegate and or instrument using WMI

@PeterPann23
Copy link

also have a look at the answers in #3749
could help your issue as well

@justinormont
Copy link
Contributor

As mentioned in #3235, MLContext.Log() doesn't have a verbosity selection, so it's more of a firehose.

If a verbosity argument is added to MLContext.Log(), the log output from there should be human readable to see general progress.

I believe it's still hidden within the firehose of output and once the verbosity is scaled down, you should see messages like:

LightGBM objective=multiclassova
[7] 'Loading data for LightGBM' finished in 00:00:15.6600468.
[8] 'Training with LightGBM' started.
..................................................(00:30.58)	0/200 iterations
..................................................(01:00.9)	1/200 iterations
..................................................(01:31.2)	2/200 iterations
..................................................(02:01.4)	2/200 iterations
..................................................(02:31.9)	3/200 iterations
..................................................(03:02.5)	4/200 iterations
..................................................(03:32.9)	4/200 iterations
..................................................(04:03.6)	5/200 iterations
..................................................(04:34.4)	5/200 iterations
..................................................(05:04.8)	6/200 iterations

And naively extrapolating, there's around 2.7 hours left in the LightGBM training.

@justinormont justinormont added the usability Smoothing user interaction or experience label May 22, 2019
@seabluescn
Copy link
Author

I tried the MLContext.Log event but found that I could not filter out the information I wanted.

@seabluescn
Copy link
Author

I found a way to use CustomMapping: write a CustomMappingFactory subclass, input arbitrary, output as a fixed value such as 1.0f, and input it as a feature, then the Action method of the class will be called, you can write debugging code in this method.
See: https://github.com/seabluescn/Study_ML.NET/tree/master/MulticlassClassification_Mnist

@ghost ghost locked as resolved and limited conversation to collaborators Mar 21, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
usability Smoothing user interaction or experience
Projects
None yet
Development

No branches or pull requests

3 participants