Skip to content

GraphDefs failed to reconcile when using verbose=2 in model.fit() #5745

@raphi1790

Description

@raphi1790

Environment information (required)

  • Ubuntu: 20.04 (via WSL2)
  • Python: 3.10.2
  • TensorFlow: 2.9.1

Diagnostics

Diagnostics output
--- check: autoidentify
INFO: diagnose_tensorboard.py version e43767ef2b648d0d5d57c00f38ccbd38390e38da

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=10, micro=2, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='', release='5.10.16.3-microsoft-standard-WSL2', version='#1 SMP Fri Apr 2 22:23:49 UTC 2021', machine='x86_64')
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: False
INFO: $VIRTUAL_ENV: '/home/<user>/.local/share/virtualenvs/graph-MwCf7F4m'

--- check: installed_packages
INFO: installed: tensorboard==2.9.0
INFO: installed: tensorflow==2.9.1
INFO: installed: tensorflow-estimator==2.9.0
INFO: installed: tensorboard-data-server==0.6.1

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '2.9.0'

--- check: tensorflow_python_version
2022-06-07 15:59:08.759019: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2022-06-07 15:59:08.763534: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-06-07 15:59:08.763586: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
INFO: tensorflow.__version__: '2.9.1'
INFO: tensorflow.__git_version__: 'v2.9.0-18-gd8ce9f9c301'

--- check: tensorboard_data_server_version
INFO: data server binary: '/home/<user>/.local/share/virtualenvs/graph-MwCf7F4m/lib/python3.10/site-packages/tensorboard_data_server/bin/server'
INFO: data server binary version: b'rustboard 0.6.1'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'/home/<user>/.local/share/virtualenvs/graph-MwCf7F4m/bin/tensorboard\n'

--- check: addrinfos
socket.has_ipv6 = True
socket.AF_UNSPEC = <AddressFamily.AF_UNSPEC: 0>
socket.SOCK_STREAM = <SocketKind.SOCK_STREAM: 1>
socket.AI_ADDRCONFIG = <AddressInfo.AI_ADDRCONFIG: 32>
socket.AI_PASSIVE = <AddressInfo.AI_PASSIVE: 1>
Loopback flags: <AddressInfo.AI_ADDRCONFIG: 32>
Loopback infos: [(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::1', 0, 0, 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 0))]
Wildcard flags: <AddressInfo.AI_PASSIVE: 1>
Wildcard infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('0.0.0.0', 0)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::', 0, 0, 0))]

--- check: readable_fqdn
INFO: socket.getfqdn(): 'localdomain'

--- check: stat_tensorboardinfo
INFO: directory: /tmp/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=60228, st_dev=2064, st_nlink=2, st_uid=1000, st_gid=1000, st_size=4096, st_atime=1598434963, st_mtime=1654610299, st_ctime=1654610299)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/home/<user>/.local/share/virtualenvs/graph-MwCf7F4m/lib/python3.10/site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==1.1.0
astunparse==1.6.3
cachetools==5.2.0
certifi==2022.5.18.1
charset-normalizer==2.0.12
flatbuffers==1.12
gast==0.4.0
google-auth==2.6.6
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.46.3
h5py==3.7.0
idna==3.3
keras==2.9.0
Keras-Preprocessing==1.1.2
libclang==14.0.1
Markdown==3.3.7
numpy==1.22.4
oauthlib==3.2.0
opt-einsum==3.3.0
packaging==21.3
pip==22.0.4
protobuf==3.19.4
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==3.0.9
requests==2.27.1
requests-oauthlib==1.3.1
rsa==4.8
setuptools==62.3.2
six==1.16.0
tensorboard==2.9.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.9.1
tensorflow-estimator==2.9.0
tensorflow-io-gcs-filesystem==0.26.0
termcolor==1.1.0
typing_extensions==4.2.0
urllib3==1.26.9
Werkzeug==2.1.2
wheel==0.37.1
wrapt==1.14.1

Issue description

When training a model using Keras'-method model.fit() using the argument verbose=2 the graph, written by the callback, won't be shown. When switching back to verbose=1 everything works as expected.
I haven't found anything in the documentation indicating this behaviour is intended.
Can you shed light on this?

Printscreen:
image

Code (minimal example):

import os
import tensorflow as tf
print("TF:", tf.__version__)

if __name__ == "__main__":

    mnist = tf.keras.datasets.mnist

    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    x_train, x_test = x_train / 255.0, x_test / 255.0

    model = tf.keras.models.Sequential([
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(128, activation='relu'),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10)
    ])

    model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

    tensorboard_callback = tf.keras.callbacks.TensorBoard(
        log_dir="artefacts/logs/", update_freq='epoch', write_graph=True)


    history = model.fit(x_train,
              y_train,
              epochs=3,
              callbacks=[tensorboard_callback],
              verbose=2 # this is the relevant line
              )

Activity

bmd3k

bmd3k commented on Jun 9, 2022

@bmd3k
Contributor

Thanks for the report. I was able to reproduce even by just modifying the "Getting Started" TensorBoard colab to use verbose=2.

https://www.tensorflow.org/tensorboard/get_started

I haven't yet investigated the cause.

self-assigned this
on Jun 10, 2022
tarolling

tarolling commented on Feb 21, 2025

@tarolling

any updates on this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @bmd3k@raphi1790@tarolling

      Issue actions

        GraphDefs failed to reconcile when using verbose=2 in model.fit() · Issue #5745 · tensorflow/tensorboard