RProp over benchmark problem yields unexpected results

[gnitr]

Please note that this problem may just be a bug in my code but I'd rather report this anyway in case someone is able to confirm whether the problem is in Encog or not.

The PROBEN1 report (page 32) claims that they get 0.027 MSE over 101 epochs on average for that dataset. And the classification error is around 17%.

I can't get anything near that with my code based on Encog. So I've implement the same experiment in MATLAB and Encog to objectively compare the behaviours. I stop training if the MSE goes under 0.05 or the number of iterations is 300. I repeat the training process over 100 randomly initialised networks.

In MATLAB 88 times out of 100 the training stops before the 300 limit as it reaches the MSE target of 0.05. And on average it does so in 95 epochs. And the classification accuracies are in line with those described in PROBEN1.

With my code based on Encog and with the same dataset, the same network architecture and, I believe, the same RProp heuristics, the MSE is 0.14 on average. It goes below 0.11 only once out of the 100 trainings.

In practice this difference means that the trained network should be able to correctly classify 83% of the testing samples but instead it is stuck at 65% even when I let the training run up to 3000 epochs. As you can see below the differences in classification error over the training set is even worse.

Here are the results for 10 trials:

MATLAB:

experiment(gene1, 300, [4,2,3], 10, 0.05);
Trial: 1, Epochs: 97, MSE: 0.0489183, CE(trn): 6.4%, CE(tst): 16.4%
Trial: 2, Epochs: 28, MSE: 0.0482678, CE(trn): 7.6%, CE(tst): 18.0%
Trial: 3, Epochs: 50, MSE: 0.0497159, CE(trn): 8.0%, CE(tst): 22.2%
Trial: 4, Epochs: 56, MSE: 0.0497302, CE(trn): 6.5%, CE(tst): 19.2%
Trial: 5, Epochs: 40, MSE: 0.0493543, CE(trn): 6.4%, CE(tst): 18.9%
Trial: 6, Epochs: 54, MSE: 0.0499842, CE(trn): 6.9%, CE(tst): 16.3%
Trial: 7, Epochs: 94, MSE: 0.0497391, CE(trn): 6.2%, CE(tst): 20.9%
Trial: 8, Epochs: 28, MSE: 0.0496143, CE(trn): 6.9%, CE(tst): 15.6%
Trial: 9, Epochs: 44, MSE: 0.0483782, CE(trn): 7.7%, CE(tst): 17.9%
Trial: 10, Epochs: 64, MSE: 0.0497216, CE(trn): 7.7%, CE(tst): 19.4%
Min MSE: 0.0482678, Max MSE: 0.0499842, Avg MSE: 0.0493424
Min epoch: 28, Max epoch: 97, Avg epoch: 55.5
Successes: 10 / 10
4 2 3

My code using Encog:

Trial: 1, Epochs: 300, MSE: 0.130949, CE(trn): 28.5%, CE(tst): 35.2%
Trial: 2, Epochs: 300, MSE: 0.155891, CE(trn): 41.8%, CE(tst): 41.2%
Trial: 3, Epochs: 300, MSE: 0.167867, CE(trn): 46.3%, CE(tst): 46.0%
Trial: 4, Epochs: 300, MSE: 0.149423, CE(trn): 34.5%, CE(tst): 38.5%
Trial: 5, Epochs: 300, MSE: 0.129148, CE(trn): 28.0%, CE(tst): 34.3%
Trial: 6, Epochs: 300, MSE: 0.167622, CE(trn): 46.0%, CE(tst): 46.5%
Trial: 7, Epochs: 300, MSE: 0.153942, CE(trn): 36.5%, CE(tst): 40.6%
Trial: 8, Epochs: 300, MSE: 0.136173, CE(trn): 29.3%, CE(tst): 32.7%
Trial: 9, Epochs: 300, MSE: 0.170760, CE(trn): 47.5%, CE(tst): 47.2%
Trial: 10, Epochs: 300, MSE: 0.125813, CE(trn): 26.8%, CE(tst): 34.9%
Min MSE: 0.125813, max MSE: 0.170760, Avg MSE: 0.148759
Min epoch: 300, max epoch: 300, Avg epoch: 300.000
Successes: 0 / 10
Architecture: ?:B->SIGMOID->4:B->SIGMOID->2:B->LINEAR->?

[gnitr]

Here's the Java tester to reproduce the issue. Look at the class comments for the information on how to obtain and use the dataset.

/*
 */
package testers;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;

import org.encog.ml.MLMethod;
import org.encog.ml.data.MLDataPair;
import org.encog.ml.data.MLDataSet;
import org.encog.ml.data.basic.BasicMLDataSet;
import org.encog.ml.factory.MLMethodFactory;
import org.encog.ml.factory.MLTrainFactory;
import org.encog.ml.train.MLTrain;
import org.encog.neural.networks.BasicNetwork;
import org.encog.neural.networks.training.propagation.resilient.ResilientPropagation;
import org.encog.util.EngineArray;

/**
 * Tester for gene1.dt
 * Datasets available at: http://www.filewatcher.com/m/proben1.tar.gz.1782734.0.0.html
 * Extract this file : proben1.tar\proben1\gene\gene1.dt
 * Place it in your java working directory.
 * And comment out the first 7 lines with a leading hash. # ...
 */
public class GeneTest {

    public static double XOR_INPUT[][] = { { 0.0, 0.0 }, { 1.0, 0.0 },
            { 0.0, 1.0 }, { 1.0, 1.0 } };

    public static double XOR_IDEAL[][] = { { 0.0 }, { 1.0 }, { 1.0 }, { 0.0 } };

    public static void main(final String args[]) {

        // control parameters
        boolean testXor = false;
        int trials = 10;
        int maxEpoch = 300;
        double maxError = 0.05;

        // create dataset
        MLDataSet trainingSet = null;
        MLDataSet testingSet = null;
        if (testXor) {
            trainingSet = new BasicMLDataSet(XOR_INPUT, XOR_IDEAL);
            testingSet = new BasicMLDataSet(XOR_INPUT, XOR_IDEAL);
        } else {
            trainingSet = loadGeneDataSet("gene1.dt", 1, 1588);
            testingSet = loadGeneDataSet("gene1.dt", 2383, 3175);
        }

        String architecture = "";
        if (testXor) {
            architecture = "?:B->SIGMOID->2:B->LINEAR->?";
        } else {
            architecture = "?:B->SIGMOID->4:B->SIGMOID->2:B->LINEAR->?";
        }

        // stats
        double mseMin = 0;
        double mseMax = 0;
        double mseAvg = 0;
        double epochMin = 0;
        double epochMax = 0;
        double epochAvg = 0;
        double ceAvg = 0;
        int successes = 0;

        // trials
        for (int trial = 1; trial <= trials; trial++) {
            // create net and training
            MLMethodFactory methodFactory = new MLMethodFactory();
            MLMethod method = methodFactory.create(MLMethodFactory.TYPE_FEEDFORWARD, architecture, trainingSet.getInputSize(), trainingSet.getIdealSize());     
            MLTrainFactory trainFactory = new MLTrainFactory(); 
            MLTrain train = trainFactory.create(method, trainingSet, MLTrainFactory.TYPE_RPROP, "");

            int epoch = 0;

            // train
            do {
                train.iteration();
                epoch++;
                //System.out.printf("\tEpoch %d, MSE: %g\n", epoch, train.getError());
            } while((train.getError() > maxError) && (epoch < maxEpoch));

            double mse = train.getError();

            if (train.getError() <= maxError) successes++;

            // record stats
            // MSE
            if (trial == 1) {
                mseMax = mseMin = mse;
            }
            if (mse < mseMin) mseMin = mse;
            if (mse > mseMax) mseMax = mse;
            mseAvg += mse;
            // EPOCH
            if (trial == 1) {
                epochMax = epochMin = epoch;
            }
            if (epoch < epochMin) epochMin = epoch;
            if (epoch > epochMax) epochMax = epoch;
            epochAvg += epoch;
            // ACCURACIES
            double ceTrn = computeClassificationError(train, trainingSet, method);
            double ceTst = computeClassificationError(train, testingSet, method);

            System.out.printf("Trial: %d, Epochs: %d, MSE: %g, CE(trn): %.1f%%, CE(tst): %.1f%%\n", trial, epoch, mse, ceTrn * 100.0, ceTst * 100.0);
        }
        mseAvg /= trials;
        epochAvg /= trials;

        // display summary      
        System.out.printf("Min MSE: %g, max MSE: %g, Avg MSE: %g\n", mseMin, mseMax, mseAvg);
        System.out.printf("Min epoch: %.0f, max epoch: %.0f, Avg epoch: %g\n", epochMin, epochMax, epochAvg);
        System.out.printf("Successes: %d / %d\n", successes, trials);
        System.out.printf("Architecture: %s\n", architecture);

        System.exit(0);
    }

    private static double computeClassificationError(MLTrain train, MLDataSet dataSet, MLMethod method) {
        double ret = 0;
        for (MLDataPair pair: dataSet) {
            if (((BasicNetwork)method).classify(pair.getInput()) == EngineArray.maxIndex(pair.getIdealArray())) ret++;
        }
        return 1.0 - (ret / dataSet.getRecordCount());
    }

    private static MLDataSet loadGeneDataSet(String filename, int fromLine, int toLine) {
        int size = toLine - fromLine + 1;
        int inputCount = 120;
        int outputCount = 3;
        double[][] input = new double[size][inputCount];
        double[][] ideal = new double[size][outputCount];
        int sampleIndex = 0;

        try {
            BufferedReader stream = new BufferedReader(new FileReader(new File(filename)));
            int lineIndex = 0;
            try {
                String line = null; // not declared within while loop
                while ((line = stream.readLine()) != null) {
                    line = line.trim();
                    if (line.startsWith("#") || line.length() == 0) continue;

                    lineIndex++;

                    if (lineIndex > toLine) {
                        break;
                    }
                    if (lineIndex >= fromLine) {

                        String[] values = line.split("\\s+");
                        for (int i = 0; i < (inputCount + outputCount); i++) {
                            if (i < inputCount) {
                                input[sampleIndex][i] = Double.parseDouble(values[i]);
                            } else {
                                ideal[sampleIndex][i - inputCount] = Double.parseDouble(values[i]);
                            }
                        }
                        sampleIndex++;
                    }
                }
            } finally {
                stream.close();
            }
        } catch (IOException ex) {
            ex.printStackTrace();
        }   

        MLDataSet ret = new BasicMLDataSet(input, ideal);

        return ret;
    }
}

[seemasingh]

PROBEN1 is pretty interesting. Discussed with Jeff and I am actually leading an effort to completly implement PROBEN1 inside of Encog. Also using it as a way to teach some of the community the basics of processing data with Encog. This will actually be really helpful for some of the NEAT enhancements I am working on.

http://www.heatonresearch.com/node/2287

[seemasingh]

I added Proben1 to Encog, and ran an RPROP run. I got the following average on gene1.dt. There were a few fixes to various things in RPROP, it could be that something caused this to be fixed.

gene1.dt; Iterations=29.30 (sdev=2.934); Data Size=1,588; Training Error=1.163500% (sdev=0.266863%); Validation Error=7.227267% (sdev=0.214884%)

[gnitr]

Hi Seema,

The results you've posted looks much better! Thanks for taking the time to look into this!

Geoffroy