Training multiple random forests on common data set

[mjmckp]

I am attempting to train random forests on a fixed 6GB data set of features, with N different labels, using M different random forest parameter settings. Overwhelmingly, the time taken to do this appears to be the disk transpose operation, which occurs N * M times, when ideally it should only be done once (as the feature set is common to all models).

To rectify this, is there any way to either:

• train multiple random forests in the same pipeline, or,
• share the transposed data object between multiple training pipelines?

[codemzs]

@mjmckp This is actually a good question and something we are investigating if transposed object can be shared amongst multiple runs.

[TomFinley]

We might consider exposing TransposeSaver and TransposeLoader, since that's one way we internally try to solve this. But I'm not sure that we've exposed the idea of any saver in the public API yet. That method can be more effective for larger datasets.

(It could of course be called directly -- but that might be subideal. It would be good to have facilities for saving in the public API.)

[mjmckp]

I tried using TransposeSaver/Loader, thanks @TomFinley. However the setup for fast forest training is still very slow. The method ExamplesToFastTreeBins.FindBinsAndReturnDataset seems to spend an inordinate amount of time in both CalculateBins and CreateSingletonFlock when using the DiskImpl implementation of DataConverter - an order of magnitude longer than the actual training done in TrainCore.

[mjmckp]

I profiled training of a FastTreeRegressor with DiskTranspose=true, using 1,000,000 input rows and 200 real-valued feature columns.

The setup takes 3 times as long as the training:

For ConvertData about half of that is spent in each of CalculateBins and CreateSingletonFlock:

Are there any ways to speed this up? The CPU utilization during the ConvertData stage is very low, as all this is done single threaded, would it make sense to do this in parallel across the number of features?

[mjmckp]

I have switched to using the LightGBM wrappers, which are orders of magnitude faster both at the data preparation stage and the training stage.

I would recommend that anyone with large data set to avoid using the managed ML.Net implementations. The data preparation stage does multiple passes over the data and isn't well parallelised. The training stage is ok, but can't match LightGBM on a GPU.