Inconsistent bind_all/hostname behavior on Windows 10

[bersbersbers]

I have trouble making the TensorBoard browser interface available on localhost, from external systems, and via an SSH tunnel at the same time. I use LocalForward 6006 localhost:6006 in my client SSH config, using the Windows 10 OpenSSH server and client.

Basically, in the default configuration (no extra parameter), tensorboard is available on localhost:6006 (as expected), but not via hostname:6006 on either client or server (as expected) and not via localhost:6006 on the SSH client system (not as expected).

When I use bind_all, it's accessible via localhost:6006 from client and server (as expected), but not via hostname:6006 on the server or client (not as expected). I had a similar issue in the default configuration of TB 1.14, if I recall correctly.

--host 0.0.0.0 works the same as bind_all.

--host hostname makes it available via hostname:6006 on both, but not via localhost:6006 (all as expected).

So in summary,

• default config should make it available via port forwarding but doesn't, and
• bind_all should make it available via hostname:6006 but doesn't.

Steps to reproduce (required)

This is the code that I use to start tensorboard:

# [X] localhost [X] port forwarding [ ] hostname
remote_argv = ["--bind_all"]
# [X] localhost [X] port forwarding [ ] hostname
# remote_argv = ["--host", "0.0.0.0"]
# [ ] localhost [ ] port forwarding [X] hostname
# remote_argv = ["--host", hostname]
# [X] localhost [ ] port forwarding [ ] hostname
remote_argv = remote_argv if remote else []

argv = ("tensorboard.exe",
        "--logdir", log_dir,
        "--port", str(port),
        *remote_argv,
        )

with open(log_file, 'w') as stdout:
    proc = subprocess.Popen(argv, stdout=stdout, stderr=subprocess.STDOUT)

Diagnostics

Diagnostics output

--- check: autoidentify
INFO: diagnose_tensorboard.py version 4725c70c7ed724e2d1b9ba5618d7c30b957ee8a4

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)
INFO: os.name: nt
INFO: os.uname(): N/A
INFO: sys.getwindowsversion(): sys.getwindowsversion(major=10, minor=0, build=18362, platform=2, service_pack='')

--- check: package_management
INFO: has conda-meta: False
INFO: $VIRTUAL_ENV: None

--- check: installed_packages
INFO: installed: tensorboard==2.0.0
INFO: installed: tensorflow-gpu==1.14.0
INFO: installed: tensorflow-estimator==1.14.0

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '2.0.0'

--- check: tensorflow_python_version
INFO: tensorflow.__version__: '1.14.0'
INFO: tensorflow.__git_version__: 'unknown'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'C:\\Users\\bersbersbers\\AppData\\Local\\Programs\\Python\\Python37\\Scripts\\tensorboard.exe\r\n'

--- check: readable_fqdn
INFO: socket.getfqdn(): 'computer.company.org'

--- check: stat_tensorboardinfo
INFO: directory: C:\Users\bersbersbers\AppData\Local\Temp\.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=5066549581543995, st_dev=2857197680, st_nlink=1, st_uid=0, st_gid=0, st_size=4096, st_atime=1570457602, st_mtime=1570457602, st_ctime=1570439229)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['C:\\Users\\bersbersbers\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
2to3==1.0
absl-py==0.8.0
altgraph==0.16.1
astor==0.8.0
astroid==2.2.5
astropy==3.2.1
attrs==19.1.0
autopep8==1.4.4
backcall==0.1.0
bandit==1.6.2
bleach==3.1.0
certifi==2019.6.16
chardet==3.0.4
colorama==0.4.1
cycler==0.10.0
decorator==4.4.0
defusedxml==0.6.0
dodgy==0.1.9
entrypoints==0.3
flake8==3.7.8
flake8-polyfill==1.0.2
future==0.17.1
gast==0.2.2
gitdb2==2.0.5
GitPython==3.0.2
google-pasta==0.1.7
grpcio==1.23.0
h5py==2.10.0
humanize==0.5.1
idna==2.8
imageio==2.5.0
imageio-ffmpeg==0.3.0
ipykernel==5.1.2
ipython==7.8.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
isbnlib==3.9.8
isbnlib-dnb==0.0.3
isbntools==4.3.20
isort==4.3.21
jedi==0.15.1
Jinja2==2.10.1
joblib==0.13.2
jsonschema==3.0.2
jupyter==1.0.0
jupyter-client==5.3.1
jupyter-console==6.0.0
jupyter-core==4.5.0
Keras==2.2.5
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
lazy-object-proxy==1.4.2
Markdown==3.1.1
MarkupSafe==1.1.1
matlabengineforpython===R2019a
matplotlib==3.1.1
mccabe==0.6.1
mistune==0.8.4
moviepy==1.0.0
mypy==0.720
mypy-extensions==0.4.1
nbconvert==5.6.0
nbformat==4.4.0
nibabel==2.5.0
notebook==6.0.1
numpy==1.17.2
packaging==19.1
pandas==0.25.1
pandocfilters==1.4.2
parso==0.5.1
pbr==5.4.3
pefile==2019.4.18
pep8==1.7.1
pep8-naming==0.8.2
pickleshare==0.7.5
Pillow==6.1.0
pip==19.2.3
pip-review==1.0
pipdeptree==0.13.2
proglog==0.1.9
prometheus-client==0.7.1
prompt-toolkit==2.0.9
prospector==1.1.7
protobuf==3.9.1
psutil==5.6.3
py3c==1.0
pycodestyle==2.5.0
pydicom==1.3.0
pydocstyle==4.0.1
pyflakes==2.1.1
Pygments==2.4.2
pyhibp==4.0.0
PyInstaller==3.5
PyJWT==1.7.1
pylama==7.7.1
pylint==2.3.1
pylint-celery==0.3
pylint-django==2.0.11
pylint-flask==0.6
pylint-plugin-utils==0.5
pynrrd==0.4.0
pyparsing==2.4.2
pypng==0.0.20
pyrsistent==0.15.4
python-dateutil==2.8.0
pytz==2019.2
pywin32==224
pywin32-ctypes==0.2.0
pywinpty==0.5.5
PyYAML==5.1.2
pyzmq==18.1.0
qtconsole==4.5.5
requests==2.22.0
requirements-detector==0.6
rope==0.14.0
scikit-learn==0.21.3
scipy==1.3.1
seaborn==0.9.0
Send2Trash==1.5.0
setoptconf==0.2.0
setuptools==41.2.0
six==1.12.0
smmap2==2.0.5
snowballstemmer==1.9.1
stevedore==1.31.0
tee==0.0.3
tensorboard==2.0.0
tensorflow-estimator==1.14.0
tensorflow-gpu==1.14.0
termcolor==1.1.0
terminado==0.8.2
testpath==0.4.2
tornado==6.0.3
tqdm==4.35.0
traitlets==4.3.2
typed-ast==1.4.0
typing-extensions==3.7.4
urllib3==1.25.3
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.15.6
wheel==0.33.6
widgetsnbextension==3.5.1
wrapt==1.11.2

[stephanwlee]

Hi @bersbersbers, thanks for raising the issue.

When I use bind_all, it's accessible via localhost:6006 from client and server (as expected), but not via hostname:6006 on the server or client (not as expected)

I am not too proficient with ssh tunneling (or Windows) but I can comment on not being able to access <hostname>:6006 on the server where TensorBoard is running. On Linux and macOS, binding to ipv6 does allow connections from <hostname>:<port> which is not true on Windows. As a result, as workaround, you can use ipv6 address like [::1]:6006 or get full ipv6 address from ipconfig on Windows.

@nfelt who is more knowledgeable about ipv6: do you know more about why binding to ipv6 does not allow connections from ipv4 (am I saying it correctly here?)? For instance, I expect below to be accessible from:

• localhost:8000
• <hostname>:8000
• [<ipv6addr>]:8000
• [::1]:8000

# since `python3 -m http.server -b "::"` is buggy
from http.server import HTTPServer
from http.server import SimpleHTTPRequestHandler
import socket

class HTTPServerV6(HTTPServer):
    address_family = socket.AF_INET6

server = HTTPServerV6(('::', 8000), SimpleHTTPRequestHandler)
server.serve_forever()

[bersbersbers]

I have repeated said test with tensorboard==2.1.0 (not sure if anything has changed since using 2.0.0 last year). I have used the following URLs on the SSH server (except noted otherwise):

1 http://localhost:6006/
2 http://127.0.0.1:6006/
3 http://127.0.0.2:6006/
4 http://hostname:6006/
5 http://193.ipv4:6006/
6 http://[::1]:6006/
7 http://[fe80::ipv6]:6006
8 http://localhost:6006 on the SSH client
9 http://127.0.0.1:6006 on the SSH client

http://ssh-client:6006 never worked on the SSH client due to firewall (ERR_NETWORK_ACCESS_DENIED)

With tensorboard --logdir ., only the first two (1 and 2) are accessible.

Same with tensorboard --logdir . --host localhost.

Same with tensorboard --logdir . --host 127.0.0.1.

With tensorboard --logdir . --host 127.0.0.2, only 3 is accessible.

With tensorboard --logdir . --bind_all, only 1, 6 and 7, and 8 and 9 are accessible.

With tensorboard --logdir . --host 0.0.0.0, only 1 through 5 are accessible (all compatible with IPv4, basically).

With tensorboard --logdir . --host hostname, only 4 and 5 are accessible.

Same with --host HOSTNAME.

With tensorboard --logdir . --host [::1], only 1 and 6 (both localhosts) and 8 and 9 (both forwarding) are working.

In summary,

• compared to my first post, --bind_all now behaves differently from --host 0.0.0.0 by including IPv6 traffic (this might have been true before, but I did not test explicitly) and by excluding a few hosts that --host 0.0.0.0 includes,
• I can use tensorboard --logdir . --host [::1] as a workaround to enable port forwarding without --bind_all.

I believe the above makes mostly sense if

• tensorboard does --host localhost by default,
• tensorboard uses getaddrinfo to convert a hostname to an IP both for the --host argument as well as for the actual host (preferably finding an IPv4 address for --host) and then compares hosts by IP,
• my port forwarding (now) works through IPv6 for some reason (which seems to be the case, even when forwarding only IPv4 traffic on the SSH client: it seems to end up on the IPv6 side on the SSH server. I have updated to Win 10 1909 and a newer Win32 OpenSSH server now, so a change is not totally unexplainable.)

What remains unclear to me is

• why bind_all answers to some IPv4 and some IPv6 requests, but not all, in particular, 2-5. This may be partially explained by @stephanwlee's comment, but this does not cover http://127.0.0.1:6006/ for example;
• if tensorboard could be smarter by converting --host hostname to both IPv4 and IPv6 addresses and bind to both: this could better match user expectations, in particular, allowing the use of http://[::1]:6006 with default settings, with does not currently work.