[Discussion] Roadmap and TensorLayer API Changes for Version 2.0

[DEKHTIARJonathan]

Dear friends,

I will try to write and summarize a few thoughts regarding the upcoming changes in TensorLayer, namely for version 2.0 🎉🎉🎉.

Main features that will be released.

I may forget some, if so I will update my post.

• Logging contrib Hyperdash module
• Graph Architecture saving
• Distributed training with Horovod
• Network API and features inspired by Keras and PyTorch.
• Database for data and model life-cycle management

TensorLayer 1.x recurring issues.

TensorLayer has always been a quite fuzzy and messy (and it's really improving 👍). The results have been an incredible number of bugs in the different Layers. Implementing one single feature oftenly assume that you partly rewrite the code for absolutely each Layer (I did it already 2 times, and I'm doing it for the 3rd time with the Network API). This is extremely dangerous with a high risk of introducing an incredible number of bugs (I remind you that we had to release a very large number of release candidate to fix all the bugs 1.8.6rc0, 1.8.6rc1, 1.8.6rc2, 1.8.6rc3, 1.8.6rc4, 1.8.6rc5 1.8.6rc6).

Every so often we find new bugs, just by reading at the code:

• Issue Error caused by using tl.layers.DeformableConv2d function #572 with tl.layers.DeformableConv2d fixed (PR Unittest for Deformable Convolution Added + Issue Fix #572 #573)
• Issue AttributeError: 'ConvLSTMLayer' object has no attribute 'initial_state'? #664 with tl.layers.ConvLSTMLayer fixed (PR Update recurrent.py #676)
• Error in tl.layers.TernaryConv2d fixed - self.inputs not defined (PR Test with Python 3.7 - Dev Branch #658)
• etc ...

Additionally, the current Layer API is slightly counter intuitive:

• why layer.all_params returns the params of a network and not a layer
• same layer.all_layers is quite ironic right ?
• with the Graph API newly introduced, when you save a Layer, you actually save the whole network.
• layer.count_params() send you the number of params in the graph, not inside the layer
• etc.

Proposition: Breaking the breaking the backward compatibility with the Network API

As TensorFlow did when releasing TF 1.0 or Django (with Django 2.0) in a non deep learning context. Very big libraries oftenly decide to let the good old times behind them and clean the code base. I believe that it is not a problem to break backward compibility if it is for the better and done very rarely.

What are the changes, I believe would highly improve TL maintainability and clarity for TL 2.0:

• a huge cleanup in the features of the Layer API, absolutely every features which is a Network feature should move to the newly created Network API.
• keep Layer features at the Layer level.
• add feature Layer-related in a Layer API (e.g. how many params in this Layer ?)

A few words on the Network API

I believe the network API should NOT be mandatory in TL. It should bring additional, non essential features.

The following should be possible

import tensorflow as tf
import tensorlayer as tl

tf.logging.set_verbosity(tf.logging.DEBUG)
tl.logging.set_verbosity(tl.logging.DEBUG)

x = tf.placeholder(tf.float32, shape=[None, 30])

net = tl.layers.InputLayer(x, name='input')
net = tl.layers.DenseLayer(net, n_units=10, name='dense1')

However, the following functionalities should be removed from Layer and move to Network API:

#  Graph API
net.load() 
net.save()

net.print_layers()
net.print_params(False)

print(net.all_drop)
print(net.all_params)
print(net.all_layers)
print(net.all_graphs)

The list above is not exhaustive. In the same time, new functionalities can be added:

layer.count_params()  # returns the number of params in this Layer
layer._local_vars  # returns a list of tf.Variable inside this Layer

The list above is not exhaustive

Presentation of the current Network API

It is not finialized, is subject to changes. I plan on releasing two Network Classes:

Sequential (similar idea than Keras)

For easy and sequential models, the Sequential Network API is here for rapid prototyping.

# Model Definition

model = tl.networks.Sequential(name="My_Sequential_1D_Network")  # Automatically adds the InputLayer, no need to do it

model.add(tl.layers.DenseLayer(n_units=10, act=tf.nn.relu, name="seq_layer_1"))

model.add(tl.layers.DenseLayer(n_units=20, act=None, name="seq_layer_2"))
model.add(tl.layers.PReluLayer(channel_shared=True, name="prelu_layer_2"))

model.add(tl.layers.DenseLayer(n_units=50, act=None, name="seq_layer_3"))
model.add(tl.layers.PRelu6Layer(channel_shared=False, name="prelu6_layer_3"))

model.add(tl.layers.DenseLayer(n_units=40, act=None, name="seq_layer_4"))
model.add(tl.layers.PTRelu6Layer(channel_shared=True, name="ptrelu6_layer_4"))

model.add(tl.layers.DenseLayer(n_units=40, act=tf.nn.relu, name="seq_layer_5"))
model.add(tl.layers.DropoutLayer(keep=0.5, is_fix=True, name="dropout_layer_5"))

# TF Graph Creation

plh = tf.placeholder(tf.float16, (100, 32))

train_model_output = model.compile(plh, reuse=False, is_train=True)  # returns a TF Tensor
test_model_output = model.compile(plh, reuse=True, is_train=False)   # returns a TF Tensor

# PyTorch-Like Layer Access

layer_1 = model["seq_layer_1"]

Custom Model API

CustomModel/CustomNetwork/Model/Network: I haven't really decide on the name yet.
This Class haven't been created yet, it is subject to change.

# Model Definition

class MyCustomNetwork(CustomNetwork):    

    def define_network(self):  # abstract function that needs to be overwritten
    
        net_in = tl.layers.InputLayer(name="input_layer")

        net = tl.layers.DenseLayer(n_units=10, act=tf.nn.relu, name="seq_layer_1")

        net1 = tl.layers.DenseLayer(n_units=20, act=None, name="seq_layer_2")(net)
        net1 = tl.layers.PReluLayer(channel_shared=True, name="prelu_layer_2")(net)

        net2 = tl.layers.DenseLayer(n_units=50, act=None, name="seq_layer_3")(net)
        net2 = tl.layers.PRelu6Layer(channel_shared=False, name="prelu6_layer_3")(net)

        net3 = tl.layers.DenseLayer(n_units=40, act=None, name="seq_layer_4")(net)
        net3 = tl.layers.PTRelu6Layer(channel_shared=True, name="ptrelu6_layer_4")(net)

        net4 = tl.layers.DenseLayer(n_units=40, act=tf.nn.relu, name="seq_layer_5")(net)
        net4 = tl.layers.DropoutLayer(keep=0.5, is_fix=True, name="dropout_layer_5")(net)
        
        net_stack = tl.layers.StackLayer(axis=1, name='stack')([net1, net2, net3, net4])
        
        return net_stack
        
model = MyCustomNetwork(name="My_Custom_Network")

# TF Graph Creation

plh = tf.placeholder(tf.float16, (100, 32))

train_model_output = model.compile(plh, reuse=False, is_train=True)  # returns a TF Tensor
test_model_output = model.compile(plh, reuse=True, is_train=False)   # returns a TF Tensor

# PyTorch-Like Layer Access

layer_1 = model["seq_layer_1"]

[zsdonghao]

TBD

Feel free to add more questions, and add answers below them.

Network API

• How to build ResNet with Network API?
  • Using network["layer_name"], example:
• What if the outputs of different branches have different shape? Will StackLayer stills work?
• Could we code like that?: model.add(DenseLayer(20, None, name="seq_layer_2"))
• The following PyTorch-like layer access can be implemented by def __getitem__(self, key): of core layer. Is there any other advantages of using Network class?

# PyTorch-Like Layer Access

layer_1 = model["seq_layer_1"]

Database for life-cycle management

Here #751

• Any comments about this design?

Graph API design

A list of dictionary with the same order of layer construction that contains all augments

• Any comments about this design?

Distributed training

It is available in master branch, try tl.distributed.Trainer.

• Any comments about this design?

[DEKHTIARJonathan]

I am more concerned about the general idea than the technical details.
For ResNet, I am working on this and for your second point, we can always create an "OutputLayer" that does just this ^^

[lgarithm]

I like the new layer API, in particular for it implements the idea of layer is a (unary, in most case) function of tensors.

With such signature of layer API implemented, we can stack layers in an easier way:

layers = [
    tl.layers.DenseLayer(n_units=20, act=None, name="seq_layer_2"),
    tl.layers.PReluLayer(channel_shared=True, name="prelu_layer_2"),
    tl.layers.DenseLayer(n_units=50, act=None, name="seq_layer_3"),
    tl.layers.PRelu6Layer(channel_shared=False, name="prelu6_layer_3")
]

net_out = transform(layers, net_in)

where

def transform(layers, net):
    y = net
    for l in layers:
        y = l(y)
    return y

We can even introduce a combinator for layers:

# stack :: [Layer] -> Layer
def stack(layers):
    return StackedLayers(layers)

which allows the following:

vgg_unit = stack(...)
net_out = vgg_unit(net_in)

[DEKHTIARJonathan]

Context

After some discussion with @lgarithm, we came up with a conceptual design that would highly clarify the TL architecture and make future development a lot more easy and integrable.

@lgarithm: Please edit my post if you want to add new stuff or correct some stuff I m ight have got wrong.

Conceptual Design

Layer and Network Classes will be converted to factory-like objects. They contain critical information to generate Layers and Network, however, they do not contain anymore any information about the generated Layer.

Layer API

Layer Class becomes a factory encapsulating the required information to generate a TF Ops, TF Tensors and TF Vars. Layer object do not contain anymore any information about the network or TF objects.

Deprecation

Which means that these informations will be removed:

Layer.outputs
Layer.inputs
Layer.all_params
Layer.all_layers
Layer.all_graph
and so on ...

Factory API

In order to insure a stable and consistent API, connecting Layer will now happen using the functional API. This manner to call Layers allows us to use Layer as factories without having to give all the parameters again.

tl.layers.Conv2D(prev_layer, ...)  # Deprecated
tl.layers.Conv2D(...)(prev_layer)  # Using the Layer.__call__() method, factory way.

Layer Output

Now a TL Layer returns a TF Tensor, not a TL Layer anymore.

Network API

Network Classes (CustomNetwork and Sequential) also become factories encapsulating the required information to generate the Model using TL Layers.

Deprecation

Which means that these informations will be removed:

Network.outputs
Network.inputs
Network.all_params
Network.all_layers
Network.all_graph
and so on ...

Factory API

The model generation will be launched using the .compile() method.

Network.compile(input_placeholder, is_train=True, reuse=True)

Network Output

The model output a new class CompiledNetwork, this model will have all the old attributes and methods that makes TL Layers so easy and pleasant to use.

[zsdonghao]

@DEKHTIARJonathan @lgarithm @lgarithm and I just discuss the network API.

For the compatibility, @DEKHTIARJonathan suggest to put existing layers in to tl.depercated.layers, then all new codes should implement in new way.

However, I am worry about the network API, and suggest to support two ways at the same time.

In my opinion, even though some people feel the existing layers don't have good network abstraction, the existing way can build any networks. While, the compile way doesn't have big advantage over the existing way, and it may not able to handle some complex situation.

Let me discuss with more users, I will put more feedback later.

Existing and Keras ways

• Existing way compiles the network in real-time.

net = InputLayer(net)
net = DenseLayer(net, 100)

• Keras way compiles the network in the end.

net = InputLayer()(net)
net = DenseLayer(100)(net)
net.compile()

In my opinion, the main differences between our way and keras way is they compile the network in the end but we simply compile network in real-time, and we mix up layer and network together.
The other parts are the same.

While, to build complex network, in some cases, the next layer will need the information (like shape) of the input or outputs of previous layers. But the input and net.outputs is unknown before we compile it. Take dynamic RNN layer below for example, we need the input placeholder/tensor to compute the sequence_length. Therefore, if we compile the network in the end, we can't give the sequence_length to DynamicRNNLayer .

net = EmbeddingInputlayer(
                     inputs = x,
                     vocabulary_size = 1000,
                     embedding_size = 200,
                     name = 'ebb')
net = DynamicRNNLayer(net,
                     cell_fn = LSTMCell,
                     n_hidden = 200,
                     dropout = (keep_prob if is_train else None),
                     sequence_length = tl.layers.retrieve_seq_length_op2(x),
                     return_last = True,
                     name = 'ann')

Problems need to be solved:

• Next layer need tensor from previous layer like sequence_length of DynamicRNNLayer.
• If a network have two branch of outputs:
  • how to combine them?
    answer: use ConcatLayer
  • apart from using ConcatLayer in the end, any other ways? returns two network via compile?
  • if compile returns two network, how to store model to one file?
• It would be great to support both existing way and new way.
• If we only support the new way.
  • how to prevent losing existing users?
  • how to

[lgarithm]

You don’t really need sequence_length when you constructing the Layer.
in the new API, it would be simply moved to the compile method, like the following:

def transform(net, layers):
    """Apply a sequence of layers to a net to form new net."""
    for layer in layers:
        net = layer(net)
    return net


def rnn_unit(net):
    """A combination of several layers."""
    embed_layer = EmbeddingInputlayer(
        vocabulary_size=1000,
        embedding_size=200,
        name='ebb')

    d_rnn_layer = DynamicRNNLayer(cell_fn=LSTMCell,
                                  n_hidden=200,
                                  dropout=(keep_prob if is_train else None),
                                  # sequence_length is moved to compile
                                  # sequence_length=tl.layers.retrieve_seq_length_op2(x),
                                  return_last=True,
                                  name='ann')

    return transform(net, [embed_layer, d_rnn_layer])

# where DynamicRNNLayer is defined as the folowing

class DynamicRNNLayer(Layer):
    def compile(prev_layer):
        sequence_length = tl.layers.retrieve_seq_length_op2(prev_layer)
        ….

Or, you can make sequence_length a lambda to the constructor of DynamicRNNLayer,
and call it when you have the net variable, which is also in the compile method.

[zsdonghao]

@lgarithm in this example, sequence_length is not get from from previous layer, but from the input tensor. This is just an example, there would be some cases that the next layer need information from previous layers which can't be get before compiling.

for example, the tensor information can be from any previous layers:

n1 = Layer(x)
n2 = Layer(n1)
n3 = Layer(n2, n1.outputs.shape())
n4 = Layer(n3)

I think this is a main drawback of using compiling.

[nebulaV]

Could anyone explain in more detail what advantages of new Network API over the existing one? It would be better to consider thoroughly whether these advantages are really critical, before choosing to break the compatibility. Thanks.