Add Activations

[JimClarke5]

This is a redone PR to make sure the Activations branch did not rely on the Initializers branch.

I have removed all the no-arg CTORs. The original motivation was that I found previously in looking at Keras layers is that objects like “activations”, “loss” and “metric" most likely would need to be constructed
before the TensorFlow Ops is available within the model, hence I make constructors without Ops and used setTF() so the Model could set it later. The model itself hides Ops tf from the user. We have decided to address this requirement later when we start looking at Model.

I fixed HardSigmoid, as I forgot to apply a clipByValue operation on the final result. Also fixed the Unit test.

I have verified all the Unit tests against their TF Python corresponding methods.

For Softmax, we eliminated the IllegalArgmentException on 1D input that was in the Keras version.

In Swish, there is a comment on not handling the grad argument from the corresponding Python class. This is a Python optimization and will not be implemented in Java, at least at this time.

[Craigacp]

Tiny things to do with casing.

[deansher]

What would you think about moving both the declaration and the initialization of negativePart down to immediately before it is used? I realize that would require a change to keep input as the original input instead of reusing it for intermediate calculations, but personally I'd see that as a feature not a bug.

[deansher]

Perhaps rename negativePart -> amountBelowThreshold? For one thing, negativePart is vague to me. For another, it's always positive.

[JimClarke5]

I am not sure if this is used here as a math term, but I found Wolfram MathWorld - Negative Part. Perhaps @Craigacp and @karllessard might want to comment.

[deansher]

Good point, that Wolfram definition does show why negativePart was used in the first place. That said, I'd still advocate that in the presence of threshold, it's a confusing name.

[deansher]

Is zero still the right lower clip if there's a threshold?

[JimClarke5]

The numbers less than the threshold get converted to zero before the clipMax is applied.
So, there is no need to move the lower bound on the clipByValue operation.

if (threshold != 0) {
      // computes input for input > threshold else 0
      Greater greater = tf.math.greater(input, tf.dtypes.cast(tf.constant(threshold), dataType));
      input = tf.math.mul(input, tf.dtypes.cast(greater, dataType));

[deansher]

Perhaps also consolidate this sort of test logic into a helper class? E.g.

assertFloat32Output(tf -> new ELU<Float32>(tf),
            {1f, -0.86466473f, 3f, -0.9816844f, -0.63212055f, 2f, -0.95021296f, 4f},
            {1, -2, 3, -4, -1, 2, -3, 4})
            

[karllessard]

I would try to avoid changing the value of a method parameter, better use a new variable.

[JimClarke5]

OK

[karllessard]

@JimClarke5 , it looks like this PR can almost be merged, can you please go through all remaining comments and resolve them, either by a fix or by an answer? Thanks

[JimClarke5]

I have just pushed the changes requested, the biggest was adding TFloating in the core-api.

[karllessard]

@Craigacp @deansher , there are a few discussions you've started in this PR that is still open, for me it looks like minor changes but can you can please have a quick check and give us your blessing to merge it in its current state or not?

[deansher]

Ship it! This is a huge step forward.

[karllessard]

Thanks @JimClarke5 !