When and How to use TimeDistributedDense

[joetigger]

When will someone use TimeDistributedDense? Is it only to reduce training data set? Does it have other benefit?

Why is TimeDistributedDense after RNN/LSTM/GRU? Shouldn't it be before? For example, if my LSTM has input_dim=10 and output_dim=1, I want to use TimeDistributedDense to shift in one sample for 10 consecutive time steps; so why would TimeDistributedDense be used after the LSTM?

[fchollet]

TimeDistributedDense applies a same Dense (fully-connected) operation to every timestep of a 3D tensor.

It's not clear what you are trying to do. Please explain in a clearer way...

[joetigger]

Let's say I have a time series {1,2,3,4,...} to feed into an LSTM with 3 inputs and 1 output. Using Dense with prime=2, the training data would be:
X={[(1,2,3),(2,3,4)], [(3,4,5),(4,5,6)],...}; y={[4,5], [6,7], ...}
With TimeDistributedDense, how will the training data look like? My confusion is that the shifting should happen at the input of the LSTM, not the output.

[placebokkk]

@joetigger
As fchollet said ,
TimeDistributedDense applies a same Dense (fully-connected) operation to every timestep of a 3D tensor.

But I think you still don't catch the point. The most common scenario for using TimeDistributedDense is using a recurrent NN for tagging task.e.g. POS labeling or slot filling task.

In this kind of task:
For each sample, the input is a sequence (a1,a2,a3,a4...aN) and the output is a sequence (b1,b2,b3,b4...bN) with the same length. bi could be viewed as the label of ai.
Push a1 into a recurrent nn to get output b1. Than push a2 and the hidden output of a1 to get b2...

If you want to model this by Keras, you just need to used a TimeDistributedDense after a RNN or LSTM layer(with return_sequence=True) to make the cost function is calculated on all time-step output. If you don't use TimeDistributedDense ans set the return_sequence of RNN=False, then the cost is calculated on the last time-step output and you could only get the last bN.

I am also new to Keras, but I am trying to use it to do sequence labeling and I find this could only be done by using TimeDistributedDense. If I make something wrong, please correct me.

[EdwardRaff]

@placebokkk Your explanation I think should be cleaned a little up and added to the docs. It was not initially clear to me how and what the TimeDistribtuedDense layer was for until after some experimentation, the current docs just sort of say "here it is" but with no "why".

[joetigger]

@placebokkk You didn't answer my questions. In my example, can I use TimeDistributedDense? If so, how would the training data look like? If not, why?

I have to agree with @EdwardRaff that the document is not clear at all. It will be very helpful if someone can explain when and how to use TimeDistributedDense. Does it save time or memory or both? Also an example always helps. Can example/lstm_text_generation.py be modified to use TimeDistributedDense?

[EdwardRaff]

@joetigger I think placebokkk did answer your question, but perhaps another way of saying the same thing will help. (And @fchollet please correct me if this wrong)

RNNs are capable of a number of different types of input / output combinations, as seen below

The TimeDistributedDense layer allows you to build models that do the one-to-many and many-to-many architectures. This is because the output function for each of the "many" outputs must be the same function applied to each timestep. The TimeDistributedDense layers allows you to apply that Dense function across every output over time. This is important because it needs to be the same dense function applied at every time step.

If you didn't not use this, you would only have one final output - and so you use a normal dense layer. This means you are doing either a one-to-one or a many-to-one network, since there will only be one dense layer for the output.

[joetigger]

@EdwardRaff I don't think that the "many" outputs must be the same function. For example, I can use an RNN to predict the next X,Y in handwriting synthesis. The two outputs are both for the same time step.

What you meant probably is "when the many outputs are for t, t+1, t+2, ... then TimeDistributedDense must be used to constrain the same function being applied to each time step".

If my understanding is correct, then TimeDistributedDense is not meant for saving time/memory, and "example/lstm_text_generation.py" cannot be modified to use TimeDistributedDense instead of Dense.

[placebokkk]

@EdwardRaff
Thanks for the nice figure and more clear explanation. That is what I want to mean. I have checked the examples of the Keras but all of them are used for classification and no sequence labeling examples are given. I don't know why.

@joetigger Here is a demo for word sequence labeling. It show what the data format you need for the last many-to-many scenario in EdwardRaff's figure
maxlen = 2
batch_size = 1
nb_word = 4
nb_tag = 2

X_train = [[1,2],[1,3]]#two sequences, one is [1,2] and another is [1,3]
Y_train = [[[0,1],[1,0]],[[0,1],[1,0]]] #the output should be 3D and one-hot for softmax output with categorical_crossentropy
X_test = [[1,2],[1,3]]
Y_test = [[[0,1],[1,0]],[[0,1],[1,0]]]

X_train = sequence.pad_sequences(x_train, maxlen=maxlen)
X_test = sequence.pad_sequences(x_test, maxlen=maxlen)

Y_train = np.asarray(y_train, dtype='int32')
Y_test = np.asarray(y_test, dtype='int32')

model = Sequential()
model.add(Embedding(nb_word, 128))
model.add(LSTM(128, return_sequences=True))
model.add(TimeDistributedDense(nb_tag))
model.add(Activation('softmax'))

rms = RMSprop()
model.compile(loss='categorical_crossentropy', optimizer=rms)

model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=4, show_accuracy=True)
res = model.predict_classes(X_test)
print('res',res)

in your example , why not use this format?
X={(1,2,3),(2,3,4),(3,4,5),(4,5,6),...}; y={4,5, 6,7, ...}

[M-Taha]

@joetigger Here is my simple example showing how to use TimeDistributedDense for time series forecasting with RNNs. Note that I have set the RNN to return a sequence.

model = Sequential()  
model.add(LSTM(hidden_neurons, input_dim=in_out_neurons, return_sequences=True))
model.add(Dropout(0.2))
model.add(TimeDistributedDense(in_out_neurons))  
model.add(Activation("linear")) 

[joetigger]

@placebokkk LSTM requires 3D tensor with shape (nb_samples, timesteps, input_dim) and hence I have to use the X,y as described, which is the same as in your example. In your example, I can also use Dense, so what's the difference?

@M-Taha Can you elaborate a bit? I appreciate if you can use my toy example, i.e. to predict the next number in time series {1,2,3,4,5,6,7,8,9, 1,2,3,4,5,6,7,8,9, ...}

Can someone please clarify whether TimeDistributedDense is only to constrain same output function for all outputs at consecutive time steps, or is it a way to save time/memory?

[placebokkk]

@joetigger
If you use a Dense(you need also to set return_sequences=False in previous layer) , then the cost is calculated on the last step output, which is the third one(many to one) in @EdwardRaff 's figure.

[lemuriandezapada]

Unless something changed in the last few builds, you don't NEED to set return_sequences=False. The code will run with normal Dense just fine and give similar results.

[M-Taha]

@joetigger Please take a look at the problem setting in this article: http://danielhnyk.cz/predicting-sequences-vectors-keras-using-rnn-lstm/. The author does not use TimeDistributedDense, but my solution below uses it to make his code more efficient (see the discussion at the end of that article where I answer some questions about my code.)

from keras.models import Sequential  
from keras.layers.core import TimeDistributedDense, Activation, Dropout  
from keras.layers.recurrent import GRU
import numpy as np

def _load_data(data, steps = 40):  
    docX, docY = [], []
    for i in range(0, data.shape[0]/steps-1):
        docX.append(data[i*steps:(i+1)*steps,:])
        docY.append(data[(i*steps+1):((i+1)*steps+1),:])
    return np.array(docX), np.array(docY)

def train_test_split(data, test_size=0.15):  
    #    This just splits data to training and testing parts
    X,Y = _load_data(data)
    ntrn = round(X.shape[0] * (1 - test_size))
    perms = np.random.permutation(X.shape[0])
    X_train, Y_train = X.take(perms[0:ntrn],axis=0), Y.take(perms[0:ntrn],axis=0)
    X_test, Y_test = X.take(perms[ntrn:],axis=0),Y.take(perms[ntrn:],axis=0)
    return (X_train, Y_train), (X_test, Y_test) 

np.random.seed(0)  # For reproducability
data = np.genfromtxt('closingAdjLog.csv', delimiter=',')  # data = a TxN matrix
(X_train, y_train), (X_test, y_test) = train_test_split(np.flipud(data))  # retrieve data
print "Data loaded."

in_out_neurons = 20  
hidden_neurons = 200

model = Sequential()  
model.add(GRU(hidden_neurons, input_dim=in_out_neurons, return_sequences=True))
model.add(Dropout(0.2))
model.add(TimeDistributedDense(in_out_neurons))  
model.add(Activation("linear"))  
model.compile(loss="mean_squared_error", optimizer="rmsprop") 
print "Model compiled."

# and now train the model. 
model.fit(X_train, y_train, batch_size=30, nb_epoch=200, validation_split=0.1)  
predicted = model.predict(X_test)  
print np.sqrt(((predicted - y_test) ** 2).mean(axis=0)).mean()  # Printing RMSE 

[joetigger]

@M-Taha Thanks for the example. According to you, TimeDistributedDense is for saving time/memory. Therefore it should work if I set in_out_neurons to 1. However, according to @EdwardRaff it is for constraining many outputs to use the same function. I'm confused.. Which one is correct?

[viveksck]

@placebokkk : Thanks for the sample code. I tried it, but it seems that it is not able to even memorize the training data.

from keras.models import Sequential  
from keras.layers.core import TimeDistributedDense, Activation, Dropout  
from keras.layers.recurrent import GRU, LSTM
from keras.layers.embeddings import Embedding
from keras.preprocessing import sequence
from keras.optimizers import RMSprop
import numpy as np
maxlen = 2

batch_size = 1
nb_word = 4
nb_tag = 2

X_train = [[1,2],[1,3]]#two sequences, one is [1,2] and another is [1,3]
Y_train = [[[0,1],[1,0]],[[0,1],[1,0]]] #the output should be 3D and one-hot for softmax output with categorical_crossentropy
X_test = [[1,2],[1,3]]
Y_test = [[[0,1],[1,0]],[[0,1],[1,0]]]

X_train = sequence.pad_sequences(X_train, maxlen=maxlen)
X_test = sequence.pad_sequences(X_test, maxlen=maxlen)

Y_train = np.asarray(Y_train, dtype='int32')
Y_test = np.asarray(Y_test, dtype='int32')

model = Sequential()
model.add(Embedding(nb_word, 128))
model.add(LSTM(128, return_sequences=True))
model.add(TimeDistributedDense(nb_tag))
model.add(Activation('softmax'))

rms = RMSprop()
model.compile(loss='categorical_crossentropy', optimizer=rms)

model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=100, show_accuracy=True)
res = model.predict_classes(X_test)
print('res',res)

The predictions on the test set is however is wrong:
2/2 [==============================] - 0s - loss: 0.0023 - acc: 1.0000
Epoch 100/100
2/2 [==============================] - 0s - loss: 0.0023 - acc: 1.0000
2/2 [==============================] - 0s
('res', array([[1, 0],
[1, 0]]))

It should have been [0,1] amd [1,0].

Also should I not expect a 3D tensor as output
like this: [[[0,1],[1,0]],[[0,1],[1,0]]]

Is there anything I am missing here ?