You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi,Suppose I have 10,000 training data. When I run Fit method, it takes a long time. During this period, how can I get real-time training progress information?
Thank You!
I have opened 2 issue on this topic that are currently under review #3685 and #3358.
You will find some comments in these threads that perhaps can be used for some sort of activity monitoring.
The short answer however is no, even though there are some ways to get feed back, none are really helping determine what's going on.
You have
Use the event MLContext.Log that will give you some text that you can use to parse
2.a You can, if the network you choose use early stopping to help mitigate wasting resources (doesn't tell you your progress though)
2.b Stop a training session periodically, test the current metrics and continue as shown here. Again not really progress reporting
Hoockup an WithOnFitDelegate that get's called by the network, most of them however do not loop and only call it when trained, some of them do. Have a look at the Microsoft.ML.LearningPipelineExtentsions line 138 for more info
Pull the source and alter the source code and publish progress using a delegate and or instrument using WMI
Activity
PeterPann23 commentedon May 21, 2019
Hi,
I have opened 2 issue on this topic that are currently under review #3685 and #3358.
You will find some comments in these threads that perhaps can be used for some sort of activity monitoring.
The short answer however is no, even though there are some ways to get feed back, none are really helping determine what's going on.
You have
2.a You can, if the network you choose use early stopping to help mitigate wasting resources (doesn't tell you your progress though)
2.b Stop a training session periodically, test the current metrics and continue as shown here. Again not really progress reporting
Hoockup an WithOnFitDelegate that get's called by the network, most of them however do not loop and only call it when trained, some of them do. Have a look at the Microsoft.ML.LearningPipelineExtentsions line 138 for more info
Pull the source and alter the source code and publish progress using a delegate and or instrument using WMI
PeterPann23 commentedon May 21, 2019
also have a look at the answers in #3749
could help your issue as well
justinormont commentedon May 22, 2019
As mentioned in #3235,
MLContext.Log()
doesn't have a verbosity selection, so it's more of a firehose.If a verbosity argument is added to
MLContext.Log()
, the log output from there should be human readable to see general progress.I believe it's still hidden within the firehose of output and once the verbosity is scaled down, you should see messages like:
And naively extrapolating, there's around 2.7 hours left in the LightGBM training.
seabluescn commentedon May 26, 2019
I tried the MLContext.Log event but found that I could not filter out the information I wanted.
seabluescn commentedon May 26, 2019
I found a way to use CustomMapping: write a CustomMappingFactory subclass, input arbitrary, output as a fixed value such as 1.0f, and input it as a feature, then the Action method of the class will be called, you can write debugging code in this method.
See: https://github.com/seabluescn/Study_ML.NET/tree/master/MulticlassClassification_Mnist