'MovingAverage' object has no attribute '_average_weights'

[dipendra009]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• I am reporting the issue to the correct repository. (Model Garden official or research directory)
• I checked to make sure that this issue has not been filed already.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/official/...

2. Describe the bug

A clear and concise description of what the bug is.

3. Steps to reproduce

Steps to reproduce the behavior.

4. Expected behavior

A clear and concise description of what you expected to happen.

5. Additional context

Include any logs that would be helpful to diagnose the problem.

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
• Mobile device name if the issue happens on a mobile device:
• TensorFlow installed from (source or binary):
• TensorFlow version (use command below):
• Python version:
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version:
• GPU model and memory:

[dipendra009]

I am running the official Tensorflow classifier_trainer.py inside vision/image_classification branch of official tf models with moving_average_decay of 0.9999. The model is instantiated correctly, but I get the following bug during training:

INFO:tensorflow:Error reported to Coordinator: in user code:

/vision/image_classification/optimizer_factory.py:131 update_average  *
    self._model_weights),))
/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:719 __getattribute__
    raise e
/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:709 __getattribute__
    return super(OptimizerV2, self).__getattribute__(name)

AttributeError: 'MovingAverage' object has no attribute '_average_weights'

Traceback (most recent call last):
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/training/coordinator.py", line 297, in stop_on_exception
yield
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/distribute/mirrored_run.py", line 323, in run
self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 258, in wrapper
raise e.ag_error_metadata.to_exception(e)
AttributeError: in user code:

/home_dir/dipendra/vision/image_classification/optimizer_factory.py:131 update_average  *
    self._model_weights),))
/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:719 __getattribute__
    raise e
/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:709 __getattribute__
    return super(OptimizerV2, self).__getattribute__(name)

AttributeError: 'MovingAverage' object has no attribute '_average_weights'

I0622 13:41:51.641907 140427864807168 coordinator.py:219] Error reported to Coordinator: in user code:

/home_dir/dipendra/vision/image_classification/optimizer_factory.py:131 update_average  *
    self._model_weights),))
/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:719 __getattribute__
    raise e
/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:709 __getattribute__
    return super(OptimizerV2, self).__getattribute__(name)

AttributeError: 'MovingAverage' object has no attribute '_average_weights'

Traceback (most recent call last):
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/training/coordinator.py", line 297, in stop_on_exception
yield
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/distribute/mirrored_run.py", line 323, in run
self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 258, in wrapper
raise e.ag_error_metadata.to_exception(e)
AttributeError: in user code:

/home_dir/dipendra/vision/image_classification/optimizer_factory.py:131 update_average  *
    self._model_weights),))
/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:719 __getattribute__
    raise e
/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:709 __getattribute__
    return super(OptimizerV2, self).__getattribute__(name)

AttributeError: 'MovingAverage' object has no attribute '_average_weights'

Traceback (most recent call last):
File "classifier_trainer.py", line 463, in
app.run(main)
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "classifier_trainer.py", line 450, in main
stats = run(flags.FLAGS)
File "classifier_trainer.py", line 442, in run
return train_and_eval(params, strategy_override)
File "classifier_trainer.py", line 400, in train_and_eval
**validation_kwargs)
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper
return method(self, *args, **kwargs)
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1098, in fit
tmp_logs = train_function(iterator)
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 780, in call
result = self._call(*args, **kwds)
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 823, in _call
self._initialize(args, kwds, add_initializers_to=initializers)
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 697, in _initialize
*args, **kwds))
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2843, in _get_concrete_function_internal_garbage_collected
graph_function, _, _ = self._maybe_define_function(args, kwargs)
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3201, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3063, in _create_graph_function
capture_by_value=self._capture_by_value),
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 979, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 600, in wrapped_fn
return weak_wrapped_fn().wrapped(*args, **kwds)
File "/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 966, in wrapper
raise e.ag_error_metadata.to_exception(e)
AttributeError: in user code:

/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
    return step_function(self, iterator)
/home_dir/dipendra/vision/image_classification/optimizer_factory.py:131 update_average  *
    self._model_weights),))
/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:719 __getattribute__
    raise e
/home_dir/dipendra/.local/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:709 __getattribute__
    return super(OptimizerV2, self).__getattribute__(name)

AttributeError: 'MovingAverage' object has no attribute '_average_weights'

I am trying to train using two P100 GPUs. Is it because training on multiple GPUs is not supported using MovingAverage?

[allenwang28]

MovingAverage has not been well tested/supported yet - could you try adding this line here?

self._average_weights = []

and see if this resolves?

[allenwang28]

The changes that we had in MovingAverage just got merged into TFA, so you might instead try substituting our MovingAverage with the TFA implementation.

[allenwang28]

This issue was fixed in b035a22

Please try this and if it doesn't work, feel free to re-open this thread. Thanks!