Implementing Gradient Accumulation Optimizer #2527

stefan-falk · 2021-07-20T06:58:58Z

Description

This PR is a fork/alternative of #2525 and seeks to implement gradient accumulation.

Type of change

Checklist:

I've properly formatted my code according to the guidelines
- By running Black + Flake8
- By running pre-commit hooks
This PR addresses an already submitted issue for TensorFlow Addons
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
This PR contains modifications to C++ custom-ops

google-cla · 2021-07-20T06:59:26Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

stefan-falk · 2021-07-20T07:00:11Z

tensorflow_addons/optimizers/gradient_accumulator.py

+def GradientAccumulator(
+    optimizer: types.Optimizer,
+    accu_steps: int = 2,
+    trainable_variables=None
+) -> types.Optimizer:
+    if trainable_variables is None:
+        trainable_variables = list()
+
+    if isinstance(optimizer, str):
+        optimizer = tf.keras.optimizers.get(optimizer)
+
+    optimizer.gradient_transformers.append(
+        AccumulationGradientTransformer(
+            optimizer=optimizer,
+            accu_steps=accu_steps,
+            trainable_variables=trainable_variables
+        )
+    )
+
+    return optimizer


@fsx950223 This here is for now just a function. Hence I disabled the test_config() for now.

stefan-falk · 2021-07-20T07:00:42Z

tensorflow_addons/optimizers/tests/gradient_accumulator_test.py

+# def test_config():
+#     sgd_opt = tf.keras.optimizers.SGD(lr=2.0, nesterov=True, momentum=0.3, decay=0.1)
+#     accum_steps = 4
+#     opt = GradientAccumulator(sgd_opt, accum_steps=accum_steps)
+#     config = opt.get_config()
+#
+#     assert config["accum_steps"] == accum_steps
+#
+#     new_opt = GradientAccumulator.from_config(config)
+#     old_sgd_config = opt._optimizer.get_config()
+#     new_sgd_config = new_opt._optimizer.get_config()
+#
+#     for k1, k2 in zip(old_sgd_config, new_sgd_config):
+#         assert old_sgd_config[k1] == new_sgd_config[k2]


See #2527 (comment)

stefan-falk · 2021-07-20T07:01:55Z

@fsx950223 With this implementation I do not get any errors on my side. The (ASR) model I seek to train appears to converge.

fsx950223 · 2021-07-20T07:09:22Z

@fsx950223 With this implementation I do not get any errors on my side. The (ASR) model I seek to train appears to converge.

Do you know the reason?

stefan-falk · 2021-07-20T07:10:13Z

Do you know the reason?

The reason why it works? No idea 😆

stefan-falk · 2021-07-20T07:11:49Z

I can only assume that it's because internally I never have any branching or something like that 🤷‍♂️

I'll admit that I am not really an expert for the internal workings of Tensorflow (yet).

But if you want I can make this PR on your forked repository on https://github.com/fsx950223/addons/tree/ga ?

google-cla · 2021-07-20T07:15:15Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

stefan-falk · 2021-07-20T07:26:25Z

@googlebot I consent.

google-cla · 2021-07-20T07:27:07Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

google-cla · 2021-07-20T07:31:04Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

google-cla · 2021-07-20T07:31:39Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

google-cla · 2021-07-20T07:35:27Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

google-cla · 2021-07-20T07:59:06Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

google-cla · 2021-07-20T08:26:48Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

google-cla · 2021-07-20T08:52:41Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

fsx950223 · 2021-07-20T12:57:20Z

tensorflow_addons/optimizers/tests/gradient_accumulator_test.py

+    for _ in range(accu_steps * 4):
+        opt.apply_gradients(grads_and_vars)
+    np.testing.assert_allclose(
+        var0.read_value(), [[1.0, 2.0, 0.0], [0.2, 1.2, 0.0]], rtol=1e-5


remove rtol=1e-5

I'm not sure why but the result has some rounding errors. I need to increase the tolerance here for the test to succeed. The gradients are computed correctly but the variables are updated with rounding errors.

fsx950223 · 2021-07-20T12:57:58Z

tensorflow_addons/optimizers/tests/gradient_accumulator_test.py

+        tf.keras.optimizers.SGD(lr=1.0), accum_steps, trainable_variables=variables
+    )
+
+    for _ in range(accum_steps + 1):


accum_steps*2+1

You suggest to change this to:

Suggested change

for _ in range(accum_steps + 1):

for _ in range(accum_steps * 2 + 1):

?

google-cla · 2021-07-20T13:02:58Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

fsx950223 · 2021-07-20T13:15:21Z

Close temporarily for deduplicate notes.

fsx950223 and others added 25 commits July 15, 2021 09:09

add gradient accumulator

e49c805

add exceptions

2c0fbae

fix multi gpus bug

11e536d

fix test bugs

1a4c0d4

fix sparse optimizer

eabed95

remove read_value

a6ff7c0

fix sparse test

24ae8a9

fix sparse bug

2760fad

refactor

4ba7a55

add sparse multi gpu test

dc50184

fix rnn bug

8cd65ad

fix step bugs

7d40946

fix _iterations

6949bd3

use gradient transformer

9e423e5

fix bug

7f3b2e9

fix step bug

99dcde5

simpify code

a184581

optimize

d0718f8

fix bug

2af5475

fix bug

42fccea

simpify code

93794ec

add mean reduction

e62cc95

decrease memory usage

64b70b4

add name

4dbc208

Implement GA alternative

b314592

boring-cyborg bot added optimizers test-cases Related to Addons tests labels Jul 20, 2021

google-cla bot added the cla: no label Jul 20, 2021

stefan-falk commented Jul 20, 2021

View reviewed changes

Run black formatter

222e757

Merge branch 'ga' into ga-alt

1c7bb61

Update gradient_accumulator.py

5e75bd7

Fixing code mess up

b31c896

Add embedding test

a03643e

Add currently failing LSTM-test

0a8e686

use default-strategy

40b6e38

Use custom implementation for GA

6db2187

fsx950223 reviewed Jul 20, 2021

View reviewed changes

Some cleaning up

4142f37

fsx950223 closed this Jul 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementing Gradient Accumulation Optimizer #2527

Implementing Gradient Accumulation Optimizer #2527

stefan-falk commented Jul 20, 2021 •

edited

Loading

google-cla bot commented Jul 20, 2021

stefan-falk Jul 20, 2021 •

edited

Loading

stefan-falk Jul 20, 2021 •

edited

Loading

stefan-falk commented Jul 20, 2021

fsx950223 commented Jul 20, 2021

stefan-falk commented Jul 20, 2021 •

edited

Loading

stefan-falk commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

stefan-falk commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

fsx950223 Jul 20, 2021

stefan-falk Jul 20, 2021 •

edited

Loading

fsx950223 Jul 20, 2021

stefan-falk Jul 20, 2021

google-cla bot commented Jul 20, 2021

fsx950223 commented Jul 20, 2021

	for _ in range(accum_steps + 1):
	for _ in range(accum_steps * 2 + 1):

Implementing Gradient Accumulation Optimizer #2527

Implementing Gradient Accumulation Optimizer #2527

Conversation

stefan-falk commented Jul 20, 2021 • edited Loading

Description

Type of change

Checklist:

google-cla bot commented Jul 20, 2021

stefan-falk Jul 20, 2021 • edited Loading

Choose a reason for hiding this comment

stefan-falk Jul 20, 2021 • edited Loading

Choose a reason for hiding this comment

stefan-falk commented Jul 20, 2021

fsx950223 commented Jul 20, 2021

stefan-falk commented Jul 20, 2021 • edited Loading

stefan-falk commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

stefan-falk commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

google-cla bot commented Jul 20, 2021

fsx950223 Jul 20, 2021

Choose a reason for hiding this comment

stefan-falk Jul 20, 2021 • edited Loading

Choose a reason for hiding this comment

fsx950223 Jul 20, 2021

Choose a reason for hiding this comment

stefan-falk Jul 20, 2021

Choose a reason for hiding this comment

google-cla bot commented Jul 20, 2021

fsx950223 commented Jul 20, 2021

stefan-falk commented Jul 20, 2021 •

edited

Loading

stefan-falk Jul 20, 2021 •

edited

Loading

stefan-falk Jul 20, 2021 •

edited

Loading

stefan-falk commented Jul 20, 2021 •

edited

Loading

stefan-falk Jul 20, 2021 •

edited

Loading