Packaging learning rate schedules with addons?

[tomerk]

We have several experimental learning rate schedules in TensorFlow/Keras currently:

https://www.tensorflow.org/api_docs/python/tf/keras/experimental/CosineDecay
tf.experimental.CosineDecay
tf.experimental.CosineDecayRestarts
tf.experimental.LinearCosineDecay
tf.experimental.NoisyLinearCosineDecay

These have been marked experimental for several releases, but have seen little to no usage (0 for linearcosine/noisylinearcosine, and only a handful of usages for CosineDecay/CosineDecayRestarts)

So, we're thinking of deprecating these in the next TF release, and dropping them entirely in the release after that.

Would Tensorflow-addons be interested in packaging learning rate schedules that have some users but not quite enough to include as part of tensorflow core?
E.g. CosineDecay / cosine decay w/ restarts in this case. As I've found zero users of LinearCosineDecay/NoisyLinearCosineDecay it's not clear that they should be included in addons either.

[WindQAQ]

cc @tensorflow/sig-addons-maintainers

[gabrieldemarmiesse]

That seems like a good fit for addons. We already have some learning rate schedulers. We would just need a codeowner though.

[bhack]

These have been marked experimental for several releases, but have seen little to no usage (0 for linearcosine/noisylinearcosine, and only a handful of usages for CosineDecay/CosineDecayRestarts)

How this usage was identified? Model Garden? GitHub third_party repos?
I think It could make the difference for a codeowner candidate.

[tomerk]

A combination of a github search and a search through our internal codebases.

Github produces enough search results that my estimates could be wrong, but for noisylinearcosinedecay and linearcosinedecay I seemed to just find forks of tensorflow:
https://github.com/search?l=Python&q=%22keras.experimental.noisylinearcosinedecay%22&type=Code
https://github.com/search?l=Python&q=%22keras.experimental.linearcosinedecay%22&type=Code
(other than maybe < 5 usages each)

However, github does show code results for CosineDecay that aren't just forks of tensorflow core:
https://github.com/search?l=Python&q=%22keras.experimental.cosinedecay%22&type=Code
(After 2-3 pages it just returns forks of tensorflow)

[bhack]

I think you need to search also for something like import noisylinearcosinedecay.
In any case I suppose that we are in one of these first case of core2addon (or downstream/degraduating).
In the original ticket at #686 that generated the migration to core policy there was already the downstreamimg topic but was not analyzed at that time so currently the process Is not documented.
We are trying to improve the promotion process at tensorflow/community#239 and It would be nice to have something also for the downstreaming one.
I.e. I think that the inverse process about looking for TF sponsor/maintainer could be something like looking for codeowner with a deadline but also working to common tools on API popularity could be very useful for both the processes etc.
We could have different API popularity tresholds here and in TF like giving more weight to GitHub repo than model Garden when downstreaming etc.. but we could really improve many things if we have some common infra and a bidirectional process that shares similarities as possible.

[seanpmorgan]

TensorFlow Addons is transitioning to a minimal maintenance and release mode. New features will not be added to this repository. For more information, please see our public messaging on this decision:
TensorFlow Addons Wind Down

Please consider sending feature requests / contributions to other repositories in the TF community with a similar charters to TFA:
Keras
Keras-CV
Keras-NLP