uploader: include metadata in `list` output #2941

wchargin · 2019-11-15T23:18:33Z

Summary:
This commit teaches the uploader to display experiment metadata included
in StreamExperiments responses by supported servers. For servers
without this support, the change is a backward-compatible no-op.

The format is intentionally undocumented and not under any compatibility
guarantees, but is designed to be easily parseable for ad hoc usage. For
instance, this simple one-liner finds experiments with lots of points so
that the user can delete them:

tensorboard dev list |
    awk '$1 == "Id" { id = $2 } $1 == "Scalars" && $2 > 1000 { print id }'

Test Plan:
Running against current prod, which does not yet support the new RPCs,
the behavior is unchanged:

$ bazel run //tensorboard -- dev list
https://tensorboard.dev/experiment/IAVF94GPSWWBTvonQe4kgQ/
https://tensorboard.dev/experiment/LiQNYkOHRSGEWj42xtgtjA/
<snip>
Total: 12 experiment(s)

Running against a local server with support for the new RPCs, we see
lots of additional data (tested on both Linux and Windows):

$ bazel run //tensorboard -- dev --origin http://localhost:8080 --grpc_creds_type ssl_dev list
http://localhost:8080/experiment/WtPawgPIQXi2SZ1fQszOFA/
	Id         WtPawgPIQXi2SZ1fQszOFA
	Created    2019-11-25 10:30:18 (23 seconds ago)
	Updated    2019-11-25 10:30:39 (just now)
	Scalars    18814
	Runs       21
	Tags       7
http://localhost:8080/experiment/jD7Qc7l6S8Wy5gWKYTAHOA/
	Id         jD7Qc7l6S8Wy5gWKYTAHOA
	Created    2019-11-13 18:32:06
	Updated    2019-11-13 18:32:06
	Scalars    0
	Runs       0
	Tags       0
http://localhost:8080/experiment/do8uvvEOSNWOUEANmQIprQ/
	Id         do8uvvEOSNWOUEANmQIprQ
	Created    2019-11-13 18:15:25
	Updated    2019-11-13 18:15:37
	Scalars    3208
	Runs       8
	Tags       4
<snip>
Total: 9 experiment(s)

Also tested that the tensorboard dev export service still works
against both old and new servers.

wchargin-branch: uploader-list-metadata

Summary: This extends the `StreamExperiments` RPC such that the client can specify a set of additional metadata fields that the server should include, like “creation time” or “number of scalar points”. The format is both forward- and backward-compatible. Servers are expected to send responses with both `experiment_ids` and `experiments` until we drop support for clients that do not support `experiments`, at which point they need only send `experiments`. Test Plan: Unit test added to simulate the future behavior of servers. wchargin-branch: streamexperiments-metadata

wchargin-branch: streamexperiments-metadata wchargin-source: a9770c6a0a6f1a56a96721b4437ca33bb15974d0

Summary: This commit teaches the uploader to display experiment metadata included in `StreamExperiments` responses by supported servers. For servers without this support, the change is a backward-compatible no-op. The format is explicitly undocumented and not subject to compatibility guarantees, but is designed to be easily parseable for ad hoc usage. For instance, this simple one-liner finds experiments with lots of points so that the user can delete them: ``` tensorboard dev list | awk '$1 == "Id" { id = $2 } $1 == "Scalars" && $2 > 1000 { print id }' ``` Test Plan: Running against current prod, which does not yet support the new RPCs, the behavior is unchanged: ``` $ bazel run //tensorboard -- dev list https://tensorboard.dev/experiment/IAVF94GPSWWBTvonQe4kgQ/ https://tensorboard.dev/experiment/LiQNYkOHRSGEWj42xtgtjA/ <snip> Total: 12 experiment(s) ``` Running against a local server with support for the new RPCs, we see lots of additional data: ``` $ bazel run //tensorboard -- dev --origin http://localhost:8080 --grpc_creds_type ssl_dev list http://localhost:8080/experiment/qIONOgA7RQKJTDO1llUMzA/ Id qIONOgA7RQKJTDO1llUMzA Created 5 seconds ago Updated just now Scalars 250 Runs 5 Tags 5 http://localhost:8080/experiment/CV0PT3C6RKS3Vdg6D7zt1Q/ Id CV0PT3C6RKS3Vdg6D7zt1Q Created 23:42:10 ago Updated 23:42:10 ago Scalars 250 Runs 5 Tags 5 http://localhost:8080/experiment/do8uvvEOSNWOUEANmQIprQ/ Id do8uvvEOSNWOUEANmQIprQ Created 2019-11-13 18:15:25 Updated 2019-11-13 18:15:37 Scalars 3208 Runs 8 Tags 4 <snip> Total: 9 experiment(s) ``` wchargin-branch: uploader-list-metadata wchargin-source: 8d29c17ea904e092be8cf3e2bf07180aa8777c54

wchargin-branch: streamexperiments-metadata wchargin-source: eb73c9b8d1d48e3a0830ac4c6793a953e75e9b62

wchargin-branch: uploader-list-metadata wchargin-source: 3943f83b2b2f1d7d55949f027abfe718a96dc60c

wchargin-branch: uploader-list-metadata wchargin-source: 148b022bc8ddd2dc220511c856560bf43aa5b69d # Conflicts: # tensorboard/uploader/exporter.py # tensorboard/uploader/exporter_test.py

wchargin-branch: uploader-list-metadata wchargin-source: 148b022bc8ddd2dc220511c856560bf43aa5b69d

nfelt

LGTM. Maybe worth confirming output LGT @GalOshri too if you didn't already.

nfelt · 2019-11-22T05:34:20Z

tensorboard/uploader/uploader_main.py

+
+  ago = datetime.datetime.now().replace(microsecond=0) - dt
+  if ago < datetime.timedelta(seconds=5):
+    return "just now"


How would you feel about providing the exact timestamp in all cases and maybe just add the human-friendly interpretations of recent times in parentheses after the timestamp or something? Maybe just my pet peeve, but as a user I'm often a little annoyed when there's a humanized timestamp with no way to access the exact value (e.g. via the title text or something).

Totally agree; uncharacteristic of me to omit that. Will change.

wchargin

Thanks! @GalOshri, please see output in PR description and let me know
if you have any requests. We can also always change the output format
later, though doing so will require a client-side release rather than
just a server-side deploy.

wchargin · 2019-11-22T06:06:33Z

tensorboard/uploader/uploader_main.py

+
+  ago = datetime.datetime.now().replace(microsecond=0) - dt
+  if ago < datetime.timedelta(seconds=5):
+    return "just now"


Totally agree; uncharacteristic of me to omit that. Will change.

GalOshri · 2019-11-22T06:54:52Z

LGTM with the change mentioned above regarding "Created" and "Updated" formats. Consistency would be useful, and the second example ("23:42:10 ago") is somewhat confusing.

Since we are not including a message on "you are using X% of your quota", let's make sure we have a bug to track that for the future?

GalOshri · 2019-11-22T07:05:02Z

Also, thank you for adding this! This is super helpful.

wchargin-branch: uploader-list-metadata wchargin-source: 8d0f92c7218a8eaa8442fcefed2113ed9161b235

wchargin · 2019-11-25T18:59:51Z

Moved time formatting logic into a utility module so that it can be
tested. See updated PR description for revised output form:

	Created    2019-11-25 10:30:18 (23 seconds ago)
	Updated    2019-11-25 10:30:39 (just now)

Tested on both Linux and Windows against both prod and a dev API server
that supports the new functionality.

wchargin · 2019-11-25T20:47:50Z

Since we are not including a message on "you are using X% of your
quota",

Correct; the backend doesn’t send down that information, and it would be
restrictive to hard-code a value (doesn’t give much flexibility to
change it later without a new uploader version).

let's make sure we have a bug to track that for the future?

Opened #2969.

Summary: This commit teaches the uploader to display experiment metadata included in `StreamExperiments` responses by supported servers. For servers without this support, the change is a backward-compatible no-op. The format is intentionally undocumented and not under any compatibility guarantees, but is designed to be easily parseable for ad hoc usage. For instance, this simple one-liner finds experiments with lots of points so that the user can delete them: ``` tensorboard dev list | awk '$1 == "Id" { id = $2 } $1 == "Scalars" && $2 > 1000 { print id }' ``` Test Plan: Running against current prod, which does not yet support the new RPCs, the behavior is unchanged: ``` $ bazel run //tensorboard -- dev list https://tensorboard.dev/experiment/IAVF94GPSWWBTvonQe4kgQ/ https://tensorboard.dev/experiment/LiQNYkOHRSGEWj42xtgtjA/ <snip> Total: 12 experiment(s) ``` Running against a local server with support for the new RPCs, we see lots of additional data (tested on both Linux and Windows): ``` $ bazel run //tensorboard -- dev --origin http://localhost:8080 --grpc_creds_type ssl_dev list http://localhost:8080/experiment/WtPawgPIQXi2SZ1fQszOFA/ Id WtPawgPIQXi2SZ1fQszOFA Created 2019-11-25 10:30:18 (23 seconds ago) Updated 2019-11-25 10:30:39 (just now) Scalars 18814 Runs 21 Tags 7 http://localhost:8080/experiment/jD7Qc7l6S8Wy5gWKYTAHOA/ Id jD7Qc7l6S8Wy5gWKYTAHOA Created 2019-11-13 18:32:06 Updated 2019-11-13 18:32:06 Scalars 0 Runs 0 Tags 0 http://localhost:8080/experiment/do8uvvEOSNWOUEANmQIprQ/ Id do8uvvEOSNWOUEANmQIprQ Created 2019-11-13 18:15:25 Updated 2019-11-13 18:15:37 Scalars 3208 Runs 8 Tags 4 <snip> Total: 9 experiment(s) ``` Also tested that the `tensorboard dev export` service still works against both old and new servers. wchargin-branch: uploader-list-metadata

wchargin added 4 commits November 7, 2019 23:40

[update diffbase]

3a8a6e1

wchargin-branch: streamexperiments-metadata wchargin-source: a9770c6a0a6f1a56a96721b4437ca33bb15974d0

[update patch]

4c6cda1

wchargin-branch: streamexperiments-metadata wchargin-source: a9770c6a0a6f1a56a96721b4437ca33bb15974d0

wchargin added type:feature core:uploader labels Nov 15, 2019

googlebot added the cla: yes label Nov 15, 2019

wchargin added 3 commits November 18, 2019 18:15

[update patch]

ac011f1

wchargin-branch: streamexperiments-metadata wchargin-source: eb73c9b8d1d48e3a0830ac4c6793a953e75e9b62

[update diffbase]

cbebc2b

wchargin-branch: uploader-list-metadata wchargin-source: 3943f83b2b2f1d7d55949f027abfe718a96dc60c

[update patch]

8196fa2

wchargin-branch: uploader-list-metadata wchargin-source: 3943f83b2b2f1d7d55949f027abfe718a96dc60c

wchargin requested a review from nfelt November 20, 2019 07:52

wchargin changed the base branch from wchargin-streamexperiments-metadata to master November 21, 2019 23:36

wchargin added 2 commits November 21, 2019 15:37

[update diffbase]

b52538f

wchargin-branch: uploader-list-metadata wchargin-source: 148b022bc8ddd2dc220511c856560bf43aa5b69d # Conflicts: # tensorboard/uploader/exporter.py # tensorboard/uploader/exporter_test.py

[update patch]

56da74c

wchargin-branch: uploader-list-metadata wchargin-source: 148b022bc8ddd2dc220511c856560bf43aa5b69d

wchargin requested review from nfelt and removed request for nfelt November 21, 2019 23:37

nfelt approved these changes Nov 22, 2019

View reviewed changes

wchargin commented Nov 22, 2019

View reviewed changes

wchargin added 2 commits November 25, 2019 10:57

[update diffbase]

3675344

wchargin-branch: uploader-list-metadata wchargin-source: 8d0f92c7218a8eaa8442fcefed2113ed9161b235

[update patch]

a2592f3

wchargin-branch: uploader-list-metadata wchargin-source: 8d0f92c7218a8eaa8442fcefed2113ed9161b235

wchargin mentioned this pull request Nov 25, 2019

Show quota usage in tensorboard dev list #2969

Open

wchargin merged commit ae2aaa8 into master Nov 25, 2019

wchargin deleted the wchargin-uploader-list-metadata branch November 25, 2019 20:55

wchargin mentioned this pull request Nov 25, 2019

uploader: add simple list subcommand #2903

Merged

wchargin mentioned this pull request Nov 25, 2019

TensorBoard 2.0.2 #2970

Merged

wchargin mentioned this pull request Apr 2, 2020

uploader: display binary object bytes in tensorboard dev list output #3464

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

uploader: include metadata in `list` output #2941

uploader: include metadata in `list` output #2941

Uh oh!

wchargin commented Nov 15, 2019 •

edited

Loading

Uh oh!

nfelt left a comment

Uh oh!

nfelt Nov 22, 2019

Uh oh!

wchargin Nov 22, 2019

Uh oh!

wchargin left a comment

Uh oh!

wchargin Nov 22, 2019

Uh oh!

GalOshri commented Nov 22, 2019

Uh oh!

GalOshri commented Nov 22, 2019

Uh oh!

wchargin commented Nov 25, 2019

Uh oh!

wchargin commented Nov 25, 2019

Uh oh!

Uh oh!

uploader: include metadata in list output #2941

uploader: include metadata in list output #2941

Uh oh!

Conversation

wchargin commented Nov 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nfelt left a comment

Choose a reason for hiding this comment

Uh oh!

nfelt Nov 22, 2019

Choose a reason for hiding this comment

Uh oh!

wchargin Nov 22, 2019

Choose a reason for hiding this comment

Uh oh!

wchargin left a comment

Choose a reason for hiding this comment

Uh oh!

wchargin Nov 22, 2019

Choose a reason for hiding this comment

Uh oh!

GalOshri commented Nov 22, 2019

Uh oh!

GalOshri commented Nov 22, 2019

Uh oh!

wchargin commented Nov 25, 2019

Uh oh!

wchargin commented Nov 25, 2019

Uh oh!

Uh oh!

uploader: include metadata in `list` output #2941

uploader: include metadata in `list` output #2941

wchargin commented Nov 15, 2019 •

edited

Loading