Skip to content

How to get training progress info? #3752

Closed
@seabluescn

Description

@seabluescn

Hi,Suppose I have 10,000 training data. When I run Fit method, it takes a long time. During this period, how can I get real-time training progress information?
Thank You!

Activity

PeterPann23

PeterPann23 commented on May 21, 2019

@PeterPann23

Hi,

I have opened 2 issue on this topic that are currently under review #3685 and #3358.

You will find some comments in these threads that perhaps can be used for some sort of activity monitoring.

The short answer however is no, even though there are some ways to get feed back, none are really helping determine what's going on.

You have

  1. Use the event MLContext.Log that will give you some text that you can use to parse

2.a You can, if the network you choose use early stopping to help mitigate wasting resources (doesn't tell you your progress though)

2.b Stop a training session periodically, test the current metrics and continue as shown here. Again not really progress reporting

  1. Hoockup an WithOnFitDelegate that get's called by the network, most of them however do not loop and only call it when trained, some of them do. Have a look at the Microsoft.ML.LearningPipelineExtentsions line 138 for more info

  2. Pull the source and alter the source code and publish progress using a delegate and or instrument using WMI

PeterPann23

PeterPann23 commented on May 21, 2019

@PeterPann23

also have a look at the answers in #3749
could help your issue as well

justinormont

justinormont commented on May 22, 2019

@justinormont
Contributor

As mentioned in #3235, MLContext.Log() doesn't have a verbosity selection, so it's more of a firehose.

If a verbosity argument is added to MLContext.Log(), the log output from there should be human readable to see general progress.

I believe it's still hidden within the firehose of output and once the verbosity is scaled down, you should see messages like:

LightGBM objective=multiclassova
[7] 'Loading data for LightGBM' finished in 00:00:15.6600468.
[8] 'Training with LightGBM' started.
..................................................(00:30.58)	0/200 iterations
..................................................(01:00.9)	1/200 iterations
..................................................(01:31.2)	2/200 iterations
..................................................(02:01.4)	2/200 iterations
..................................................(02:31.9)	3/200 iterations
..................................................(03:02.5)	4/200 iterations
..................................................(03:32.9)	4/200 iterations
..................................................(04:03.6)	5/200 iterations
..................................................(04:34.4)	5/200 iterations
..................................................(05:04.8)	6/200 iterations

And naively extrapolating, there's around 2.7 hours left in the LightGBM training.

seabluescn

seabluescn commented on May 26, 2019

@seabluescn
Author

I tried the MLContext.Log event but found that I could not filter out the information I wanted.

seabluescn

seabluescn commented on May 26, 2019

@seabluescn
Author

I found a way to use CustomMapping: write a CustomMappingFactory subclass, input arbitrary, output as a fixed value such as 1.0f, and input it as a feature, then the Action method of the class will be called, you can write debugging code in this method.
See: https://github.com/seabluescn/Study_ML.NET/tree/master/MulticlassClassification_Mnist

ghost locked as resolved and limited conversation to collaborators on Mar 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    usabilitySmoothing user interaction or experience

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @justinormont@seabluescn@PeterPann23

        Issue actions

          How to get training progress info? · Issue #3752 · dotnet/machinelearning