tl.alphas and tl.alphas_like added following the tf.ones/zeros and tf.zeros_like/ones_like

[DEKHTIARJonathan]

In a research project, I encountered the need to create tensors of the same shape than any other tensor with a custom value (not just 0 or 1), could be Boolean, Floats, Integers and so on.

For the record, it was something I originally developed for the TF repository (but they were not interested):

• Issue: Improvement Proposal - tf.alphas_like (merging tf.ones_like and tf.zeros_like into one) tensorflow/tensorflow#16128
• PR: New features: tf.alphas and tf.alphas_like - Related to #16128 tensorflow/tensorflow#16153

The functions prototypes are the following and have been implemented in tensorlayer/array_ops.py:

• tl.alphas (shape, alpha_value, name=None)
• tl.alphas_like (tensor, alpha_value, name=None, optimize=True)

The code is not created from scratch. It is highly inspired by the functions tf.ones, tf.zeros for tl.alphas and by tf.zeros_like, tf.ones_like for tl.alphas_like.

The code use the latest implementation and has been designed to work with eager_mode.
The number of modification is relatively small, thus I am relatively confident on the robustness of the new implementation (largely based on the existing one).

The idea is to reproduce and merge the functions while enabling to set any custom value in the tensor:

• tl.alphas merges:
  • tf.ones
  • tf.zeros
• tl.alphas_like merges:
  • tf.zeros_like
  • tf.ones_like

How is the API Working ?

My new functions take a parameter alpha_value and fill the tensor with this value. This allows me to run such a script:

import tensorflow as tf
import tensorlayer as tl

a = tf.constant([
    [
        [4, 5, 6],
        [1, 2, 3]
    ],
    [
        [4, 5, 6],
        [1, 2, 3]
    ]
])

b1 = tl.alphas_like(a, 0.5431)
b2 = tl.alphas_like(a, 5)
b3 = tl.alphas_like(a, -5)
b4 = tl.alphas_like(a, True)

with tf.Session() as sess:
    _b1, _b2, _b3, _b4 = sess.run([b1, b2, b3, b4])
    
print("b1:", _b1)
print("b2:", _b2)
print("b3:", _b3)
print("b4:", _b4)

############### OUTPUTS ###############

>>> b1: [
  [
    [ 0.5431  0.5431  0.5431]
    [ 0.5431  0.5431  0.5431]
  ]
  [
    [ 0.5431  0.5431  0.5431]
    [ 0.5431  0.5431  0.5431]
  ]
]

>>> b2: [
  [
    [5 5 5]
    [5 5 5]
  ]
  [
    [5 5 5]
    [5 5 5]
  ]
]

>>> b3: [
  [
    [-5 -5 -5]
    [-5 -5 -5]
  ]
  [
    [-5 -5 -5]
    [-5 -5 -5]
  ]
]

>>> b4: [
  [
    [ True  True  True]
    [ True  True  True]
  ]
  [
    [ True  True  True]
    [ True  True  True]
  ]
]

Performance Optimization

Of course, it could be to do the same thing using the following commands:

b1 = tf.ones_like(a, dtype=tf.float32) * 0.5431
b2 = tf.ones_like(a, dtype=tf.int32) * 5
b4 = tf.ones_like(a, dtype=tf.bool)

Each run has been executed 5 times and execution time averaged:

• The method working with TF Code only (shown above) : [47.5s, 47.5s, 48.1s, 47.6s; 47.8s] => Average Time: 47.7 secs

• The method I implemented "alphas_like": [25.0s, 25.0s, 25.0s, 24.9s, 25.0s] => Average Time: 25 secs

My method is almost twice as fast !

Code used to produce this numbers:

import tensorflow as tf
import time

a  = tf.ones(shape=(64,255,255,3))

method1 = tf.ones_like(a) * 2.55
method2 = tl.alphas_like(a, alpha_value=2.55)


for _ in range(5):
    TIC = time.time()
    for __ in range(100):

        with tf.Session() as sess:
            tmp = sess.run(method1)

    TOC = time.time()

    print("Execution with method 1 took: %sms" % str((TOC-TIC)*1000))

print("\n######################################\n")

for _ in range(5):
    TIC = time.time()
    for __ in range(100):

        with tf.Session() as sess:
            tmp = sess.run(method2)

    TOC = time.time()

    print("Execution with method 2 took: %sms" % str((TOC-TIC)*1000))

Which gives me these results:

Execution with method 1 took: 47542.79899597168ms
Execution with method 1 took: 47539.61968421936ms
Execution with method 1 took: 48089.28418159485ms
Execution with method 1 took: 47630.205392837524ms
Execution with method 1 took: 47837.78142929077ms

######################################

Execution with method 2 took: 25034.541845321655ms
Execution with method 2 took: 25057.311058044434ms
Execution with method 2 took: 25051.66268348694ms
Execution with method 2 took: 24936.548471450806ms
Execution with method 2 took: 24955.832958221436ms

[novog]

I think that there may be some minor issues with the comments. First, the documentation for alphas_like does not list str in the list of allowed types of the alpha_value parameter. However, in my testing, it does seem to work for building string tensors with a default value. (A good thing; that is what I wanted it for!) Could this function fail with strings in some cases, or could the documentation be updated to include str in the list of allowed types?

Second, the documentation for alphas describes a nonexistent dtype paramenter instead of the alpha_value paramenter that the function actually uses.

[DEKHTIARJonathan]

The documentation might not be perfectly accurate.
It's me who implemented this the idea is to create a tensor with a shape similar to another one set to any given value.

To be very honest with you, the function will keep supporting string I have designed it to support every kind of dtype (at least I hope so) in a more efficient manner than the APIs existing in TF.

@novog maybe you could submit a PR correcting the documentation. It's always easier to correct something with an external point view.
I would be very happy to review this work if you ever have time 😊