Add tree #140

jakirkham · 2017-03-03T00:47:05Z

Fixes https://github.com/alimanfoo/zarr/issues/82

Adds a tree method to Group that constructs a pretty-printed hierarchy listing of everything below that group. Similar in nature to the Unix tree command. Borrowed the styling from @alimanfoo's example.

jakirkham · 2017-03-03T01:45:12Z

zarr/hierarchy.py

+
+        def gen_tree(g):
+            r = OrderedDict()
+            d = r.setdefault(self.name.strip("/"), OrderedDict())


There is a question of what to do with the root node, /. Right now I have opted to leave it blank, but could see an argument for making it /. Tweaking how the name is handled here should allow that.

Changed to have /.

alimanfoo · 2017-03-03T10:56:38Z

Thanks for this, big thumbs up!

I'm happy to merge this in then do some work on it. For my own reference, three things worth considering: (1) show root node as "/" as suggested; (2) expose a style argument so output can be customised; (3) consider moving get_tree out as first class method.

alimanfoo · 2017-03-03T10:57:16Z

Oh, and one other thing: (4) add an option to include some indicator showing whether a node is a group or a dataset.

jakirkham · 2017-03-03T14:35:39Z

Glad you like it.

(1) show root node as "/" as suggested;

SGTM.

(2) expose a style argument so output can be customised

Was thinking about giving kwargs that just get forwarded to that function. Maybe could allow user to pass the argument to draw. Whatever works really. 👍

(3) consider moving get_tree out as first class method.

How do you mean? Like a function instead?

(4) add an option to include some indicator showing whether a node is a group or a dataset.

Good call. Does the tree command have some way of showing this (i.e. between directories and files)? If so, maybe we could mimic it. If not, putting say [] around the name is easy.

Edit: Clarified response to 4.

alimanfoo · 2017-03-07T11:36:47Z

On Fri, Mar 3, 2017 at 2:35 PM, jakirkham ***@***.***> wrote: Glad you like it. (1) show root node as "/" as suggested; SGTM. (2) expose a style argument so output can be customised Was thinking about giving kwargs that just get forwarded to that function. Maybe could allow user to pass the argument to draw. Whatever works really. 👍 (3) consider moving get_tree out as first class method. How do you mean? Like a function instead?

I was just thinking that get_tree could be a method on the Group class, rather than being an inner function, which would make the code a bit more straightforward.

(4) add an option to include some indicator showing whether a node is a group or a dataset. Good call. Does the tree command have some way of showing this? If so, maybe we could mimic it. If not, putting say [] around the name is easy.

Yes some simple convention for adding something to the label would be good.

…

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: [email protected] Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

jakirkham · 2017-03-07T16:14:24Z

I was just thinking that get_tree could be a method on the Group class, rather than being an inner function, which would make the code a bit more straightforward.

Good point. Actually we could even skip the stylizing with asciitree altogether and leave that for the examples. Alternatively if we keep it, we could have a function or two for particular styles and let users either take those or make their own.

alimanfoo · 2017-03-07T16:34:06Z

I think it would be nice to keep something that allows printing of a default ascii tree with minimal typing. I imagine this is something I'll do a lot while exploring datasets, so I liked the short and sweet 'tree()'. I almost wondered if calling 'tree()' should actually do the printing as well as generating the ascii text, so I could just write 'g.tree()' rather than 'print(g.tree())' to see the output.

…

On Tue, Mar 7, 2017 at 4:14 PM, jakirkham ***@***.***> wrote: I was just thinking that get_tree could be a method on the Group class, rather than being an inner function, which would make the code a bit more straightforward. Good point. Actually we could even skip the stylizing with asciitree altogether and leave that for the examples. Alternatively if we keep it, we could have a function or two for particular styles and let users either take those or make their own. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/pull/140#issuecomment-284768995>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QrCZ1zoUW5hHpt2GX3dKzyLyy2x4ks5rjYJggaJpZM4MRuec> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: [email protected] Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

jakirkham · 2017-03-07T16:45:17Z

Just to clarify my suggestion, what I think we should do is have g.tree() (or possibly a different name) just return the dictionary representation not even the string. Then we can have some function like show_tree(g) call g.tree() and feed it to asciitree with this configuration to get the string. That way we don't need to muck with exposing asciitree parameters as users can roll their own with a few lines. The net result is this is less brittle for testing and more flexible for displaying without increasing the maintenance burden much.

jakirkham · 2017-03-07T16:46:35Z

zarr/hierarchy.py

+        bar
+         ├── baz
+         └── quux
+             └── baz[...]


Opted to try displaying the datasets like this. Please let me know what you think.

jakirkham · 2017-03-07T16:48:18Z

Have updated to address points 1 and 4. As point 1 is straightforward, there is likely not much to say about it. Point 4 I opted to add [...] to the end of a dataset. We could probably add a flag to control whether this is added or not.

alimanfoo · 2017-04-25T08:41:23Z

Hi @jakirkham, thinking about this some more, how about the following API...

The Group class gets a hierarchy() method. This method returns a nested dictionary data structure representing the hierarchy, of the kind that can be used as input to asciitree.

The hierarchy() method accepts a depth argument which is None by default but can be an integer specifying the maximum depth.

The hierarchy() method also accepts a label argument which is None by default but can be a function that returns a label given a group or dataset, to allow user customisation of how the text labels for each node in the hierarchy are constructed. If None the default label for groups is the name of the group (i.e., last path segment, e.g., 'foo') and the default label for arrays is the name plus dtype and shape (e.g., 'bar int64 (200, 10)').

This is enough to allow users to roll their own display function via asciitree. However, as a user I would very much like a convenience method which can allow me to visualise the hierarchy when working in an interactive terminal or jupyter notebook with minimal typing on the keyboard, and with some reasonable defaults regarding visual representation. The minimal typing is important, because I anticipate I will use this quite a lot. To satisfy this I propose the following.

The Group class also gets a tree() method. This method returns an instance of a new Tree class. The Tree class implements __repr__ which constructs a text representation of the tree via asciitree with some defaults matching the output of the command line tree utility. The Tree class could also implement _repr_html_() and/or _repr_svg_() to generate a representation for the jupyter notebook, although we could leave implementing this for a future release. I particularly like the idea that we could in future generate an HTML representation that had some simple dynamic behaviour where tree nodes can be expanded or collapsed, allowing exploration of the tree without having to view the entire expanded tree.

The Tree class would accept the hierarchy data in the constructor. So the implementation of the Group.tree() method would just be something like:

class Group(object):

    ...

    def hierarchy(self, depth=None, label=None):
        # build dictionary data structure
        ...
        return data

    def tree(self, depth=None, label=None):
        data = self.hierarchy(depth=depth, label=label)
        return Tree(data)


class Tree(object):

    def __init__(self, data):
        self.data = data

    def __repr__(self):
        # pass self.data to asciitree and return

    def _repr_html_(self):
        # for future work, use self.data to construct an HTML representation

This API would mean that all a user has to do is type mygrp.tree() at the terminal or last line of a notebook cell to view the tree.

What do you think?

jakirkham · 2017-04-26T16:54:25Z

Unfortunately I have to run to a meeting soon. So don't have lots of time to think/respond to this ATM, but plan on giving it more thought. Can give some gut reactions though.

Having an object with repr methods sounds like a great idea.

Any thoughts on having this same object also expose a dict-like interface as well? Doing this would allow us to pass the object to asciitree directly. Also this saves us having two similar methods in the Group interface. Can add some methods to it for changing the Group/Dataset representation as well.

alimanfoo · 2017-04-26T22:03:12Z

Cool. Re exposing a dict-like interface, I would say whatever is simplest, happy if you want to try something out and see how it goes.

…

On Wednesday, April 26, 2017, jakirkham ***@***.***> wrote: Unfortunately I have to run to a meeting soon. So don't have lots of time to think/respond to this ATM, but to plan on giving it more thought. Can give some gut reactions though. Having an object with repr methods sounds like a great idea. Any thoughts on having this same object also expose a dict-like interface as well? Doing this would allow us to pass the object to asciitree directly. Also this saves us having two similar methods in the Group interface. Can add some methods to it for changing the Group/Dataset representation as well. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/pull/140#issuecomment-297474103>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QsyeZ_e4C5nfR4lwvI8fqU6my6zBks5rz3bCgaJpZM4MRuec> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: [email protected] Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

jakirkham · 2017-10-24T22:36:04Z

Do we want to try and clean this up for the next release or do we want to table it for after? I'm ok with either.

jakirkham · 2017-10-24T23:38:54Z

On the HTML front, there are not any libraries that I see that meet our constraints (e.g. Python, not GPL'd, etc.), but maybe one could use a strategy like this one to roll their own.

alimanfoo · 2017-10-25T00:32:27Z

I think it would be great to get this into 2.2 if possible.

The HTML example you link to looks great. I don't know enough about jupyter notebook to know how easy it is to have objects with html repr with javascript dependencies. FWIW I'd be happy with an ascii implementation for now if adding html is not trivial.

jakirkham · 2017-10-25T01:02:46Z

Ok, so I tried to update this roughly based on what we talked about before. We now have an object with a __repr__ method. The object takes arguments to configure how it may be viewed. One can also tweak the view after the fact using update_ascii_kwargs, which returns the object. This allows one to use the REPL to tweak the view on the fly as well. Names are rough and am open to suggestions for them. The tests need tweaking, but the general idea seems ok and works fine in IPython.

jakirkham · 2017-10-25T15:07:53Z

I tried the HTML formatting in another branch, but I'm running into issues trying to get what I want from CSS. Not sure if I'm just missing something or if the notebook is overriding some CSS values in such a way that it is messing up my rendering.

jakirkham · 2017-10-25T15:14:14Z

Also the code here has seen a lot of changes in the past 24-hrs. Namely we override what asciitree calls a Traversal to specify how to iterate through a Zarr Group instead of creating a dict from the Zarr Group. Simplifies the code a bit and should make it easier to customize the tree generated.

jakirkham · 2017-10-25T16:32:35Z

Now with an HTML representation and tests! 🎉

Given some code like that below, we can see the regular output and a screenshot of the HTML output. Basically we use an HTML list and apply a specialized CSS to get a tree looking layout.

Code:

import zarr

g1 = zarr.group()
g3 = g1.create_group('bar')
g3.create_group('baz')
g5 = g3.create_group('quux')
g5.create_dataset('baz', shape=100, chunks=10)
g7 = g3.create_group('zoo')

Vanilla __repr__:

/
 +-- bar
     +-- baz
     +-- quux
     |   +-- baz[...]
     +-- zoo

...then _html_repr_:

Edit: Refreshed based on the latest commit ( alimanfoo@017de3f ).

jakirkham · 2017-10-25T17:03:32Z

Added a notebook demonstrating how this works.

jakirkham · 2017-10-25T17:08:34Z

Should add that GitHub blocks CSS when rendering things like Jupyter Notebooks'. So one needs to either look locally or take a look via the nbviewer webservice. Here's the link to new notebook generated with the nbviewer webservice based off of commit ( alimanfoo@017de3f ).

jakirkham · 2017-10-25T21:14:18Z

Made some improvements and simplifications to the CSS. Have updated the comments above to reflect the new state. To my eyes this looks pretty good, but would welcome feedback from others.

alimanfoo

This is looking great, thank you.

alimanfoo · 2017-10-25T21:02:56Z

zarr/hierarchy.py

@@ -523,6 +523,34 @@ def visititems(self, func):
        base_len = len(self.name)
        return self.visitvalues(lambda o: func(o.name[base_len:].lstrip("/"), o))

+    def tree(self):
+        """Provide a ``print`-able display of the hierarchy.


Extra backtick here?

Good catch! Yep, will fix.

alimanfoo · 2017-10-25T21:04:12Z

zarr/hierarchy.py

+        >>> g4 = g3.create_group('baz')
+        >>> g5 = g3.create_group('quux')
+        >>> d1 = g5.create_dataset('baz', shape=100, chunks=10)
+        >>> print(g1.tree())


Suggest just:

>>> g1.tree()

I.e., no need to call print().

Not a strong thought by any means, but print does force str -> repr. Other circumstances (e.g. in a Jupyter Notebook) not using print would have a different effect. Though again not a strong thought happy to change if it is preferred without print.

alimanfoo · 2017-10-25T21:04:27Z

zarr/hierarchy.py

+         |   +-- quux
+         |       +-- baz[...]
+         +-- foo
+        >>> print(g3.tree())


See comment above.

Same point. Again am flexible on this.

alimanfoo · 2017-10-25T21:07:27Z

zarr/hierarchy.py

@@ -523,6 +523,34 @@ def visititems(self, func):
        base_len = len(self.name)
        return self.visitvalues(lambda o: func(o.name[base_len:].lstrip("/"), o))

+    def tree(self):


Consider adding a depth argument here which defaults to None (i.e., render the whole subtree) but can be an integer which would limit the tree depth - useful where you only want to view a few levels down, but whole tree might be unwieldy.

alimanfoo · 2017-10-25T21:14:24Z

zarr/util.py

+
+    result += (
+        """</li>\n""".format(
+            indent, traverser.get_text(group)


No format placeholders in this string, did you mean to call .format() on this?

Good catch! That's likely a remnant from code changes. Will clear that out.

alimanfoo · 2017-10-25T21:16:25Z

zarr/util.py

+        for l in custom_html_sublist(c, indent).splitlines():
+            result += "{0}{0}{1}\n".format(indent, l)
+    if children:
+        result += "{0}{0}</ul>\n{0}".format(indent)


This section maybe more concise as:

children = traverser.get_children(group) if children: result += """\n{0}{0}<ul>\n""".format(indent) for c in children: for l in custom_html_sublist(c, indent).splitlines(): result += "{0}{0}{1}\n".format(indent, l) result += "{0}{0}</ul>\n{0}".format(indent)

alimanfoo · 2017-10-25T21:18:18Z

zarr/util.py

+
+    def get_text(self, node):
+        name = node.name.split("/")[-1] or "/"
+        name += "[...]" if hasattr(node, "dtype") else ""


I like that '[...]' is a concise way to indicate an array. I did wonder if instead people would prefer to see array shape and dtype, which serves to indicate array and also provides some more diagnostics. I'm in two minds. Any thoughts on this?

No strong feelings either way. Went with this initially as it was easy and concise, but am happy to change it if you would prefer something else. We could also let this function be overridden by a user.

alimanfoo · 2017-10-25T21:19:57Z

zarr/util.py

+    result += "</style>\n\n"
+
+    # Insert the HTML list
+    result += """<div class="zarrTree">\n"""


Nit, could you make the CSS class "zarr-tree" for consistency of naming with "zarr-info".

alimanfoo · 2017-10-25T21:21:40Z

zarr/util.py

+    return result
+
+
+class TreeHierarchy(object):


Very small point, but could we call this "TreeViewer" or something like that, just to be clear this is a class for viewing/visualising the tree?

alimanfoo · 2017-10-25T21:24:25Z

zarr/util.py

+
+class TreeHierarchy(object):
+
+    def __init__(self, group):


Related to earlier comment, would it be possible to limit this to a given depth, to help with visualising very large trees?

Sure. Might have to revisit implementing it later, but am open to the idea.

alimanfoo · 2017-10-25T21:36:08Z

Here's what I see via nbviewer as at 017de3f:

Would it be possible to tweak CSS so the list aligns to the left of the output area?

Also, there's quite a bit of vertical space between parent and sublist. Could we make this more compact?
E.g. something like:

alimanfoo · 2017-10-25T21:40:03Z

Just to see what it would look like, here's a version reporting shape and dtype to indicate array nodes:

I think I like this better than using '[...]'?

Provides a quick demo showing how to use `tree` to get text-based and HTML-based representations of a Zarr `Group` object.

Works around some Python 2/3 pains by implementing `__bytes__` and `__unicode__`. This allows a user to get a nicer Unicode text-based representation of `tree` on Python 2 even though `__repr__` will not permit it. Also allows Python 3 to get the nicer Unicode representation for free. Though users can still force the `bytes` representation should they prefer it for some reason.

As all Python `object`s will fallback to `__repr__` if `__str__` is undefined on both Python 2 and Python 3, there is no need for us to implement `__str__` in basically the same way as `__repr__`. So this just drops `__str__` so that `__repr__` can fill in as needed.

alimanfoo · 2017-10-28T00:40:58Z

After a bit of thrashing I have an example working with jstree providing styling and interactive expand/collapse of groups. Here's a screenshot from jupyter notebook:

Here's the notebook with example code. This also works in nbviewer.

What do you think @jakirkham, would you be happy to update your PR to use jstree? It seems worth it to me.

As a side note I think that nbviewer has a problem in that it loads jquery but not via require, and so I had to make sure it was defined because jstree declares a dependency on jquery. Probably worth raising this with the nbviewer folks.

alimanfoo · 2017-10-28T00:50:01Z

xref jupyter/nbviewer#736

alimanfoo · 2017-11-14T12:14:08Z

Hi @jakirkham, I'm getting near done with other work for the 2.2 release, I'd like to include this too. I think we should go for the jstree implementation for _repr_html_(), I figure if we're going to do _repr_html_() we might as well provide the user with something over and above what you can get from __repr__(). If you're busy I'm happy to merge this PR as-is and go from there, but let me know what you'd prefer.

…ree'

jakirkham · 2017-11-14T18:12:18Z

Sorry. The past few weeks have been a bit busy and am now getting over a cold. So might not be super helpful. Have tried to resolve the merge conflicts here at least.

Using jstree sounds nice. Saw it before, but was trying to start simple to begin with. Happy to switch over to jstree. The one with the "spreadsheet" icons is a nice touch. Might be worth thinking about whether we still want depth if we can collapse layers. Though it may still make sense if we expect the HTML to get unwieldy with the number of items added to it. Otherwise I trust your judgement.

alimanfoo · 2017-11-14T23:23:00Z

Thank you, I'll merge now and add in the jstree implementation. I think we can leave the depth option for now. Hope you feel better!

jakirkham · 2017-11-15T00:27:25Z

SGTM. Thanks.

jakirkham mentioned this pull request Mar 3, 2017

Group print hierarchy #82

Closed

jakirkham commented Mar 3, 2017

View reviewed changes

jakirkham commented Mar 7, 2017

View reviewed changes

alimanfoo mentioned this pull request Apr 6, 2017

Release 2.2 #144

Closed

jakirkham changed the title ~~Add tree~~ WIP: Add tree Oct 25, 2017

jakirkham changed the title ~~WIP: Add tree~~ Add tree Oct 25, 2017

alimanfoo reviewed Oct 25, 2017

View reviewed changes

jakirkham added 10 commits October 26, 2017 20:12

Test the HTML representation

0f67052

Include notebook demonstrating tree

f9170cd

Provides a quick demo showing how to use `tree` to get text-based and HTML-based representations of a Zarr `Group` object.

Try unicode and bytes in notebook

590f98d

Fix inline code quote in tree docstring

d9682de

Consolidate ifs in tree's HTML sublist build

7b9a6ae

Drop unneeded formatting at tree's HTML list end

b7a5fd9

Change zarrTree to zarr-tree

8da154a

Change TreeHierarchy to TreeViewer

3a4c65c

Merge remote-tracking branch 'alimanfoo/master' into 'jakirkham/add_t…

d09caa6

…ree'

alimanfoo merged commit 4a5847c into zarr-developers:master Nov 14, 2017

jakirkham deleted the add_tree branch November 15, 2017 00:26

alimanfoo added enhancement New features or improvements release notes done Automatically applied to PRs which have release notes. labels Nov 20, 2017

dependabot bot mentioned this pull request May 17, 2021

Bump pytest-doctestplus from 0.4.0 to 0.9.0 #752

Merged

dependabot bot mentioned this pull request May 25, 2021

Bump pytest-doctestplus from 0.4.0 to 0.9.0 TomAugspurger/zarr-python#2

Closed

dependabot bot mentioned this pull request Jul 2, 2021

Bump pytest-doctestplus from 0.4.0 to 0.10.0 TomAugspurger/zarr-python#10

Closed

dependabot bot mentioned this pull request Jul 21, 2021

Bump pytest-doctestplus from 0.4.0 to 0.10.1 TomAugspurger/zarr-python#13

Closed

dependabot bot mentioned this pull request Sep 21, 2021

Bump pytest-doctestplus from 0.4.0 to 0.11.0 TomAugspurger/zarr-python#18

Closed

dependabot bot mentioned this pull request Nov 17, 2021

Bump pytest-doctestplus from 0.4.0 to 0.11.1 TomAugspurger/zarr-python#28

Closed

dependabot bot mentioned this pull request Dec 10, 2021

Bump pytest-doctestplus from 0.4.0 to 0.11.2 TomAugspurger/zarr-python#31

Closed

dependabot bot mentioned this pull request Feb 28, 2022

Bump pytest-doctestplus from 0.4.0 to 0.12.0 TomAugspurger/zarr-python#38

Closed

dependabot bot mentioned this pull request Sep 27, 2022

Bump pytest-doctestplus from 0.4.0 to 0.12.1 TomAugspurger/zarr-python#59

Open

Add tree #140

Add tree #140

Conversation

jakirkham commented Mar 3, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alimanfoo commented Mar 3, 2017

alimanfoo commented Mar 3, 2017

jakirkham commented Mar 3, 2017 • edited Loading

alimanfoo commented Mar 7, 2017 via email

jakirkham commented Mar 7, 2017

alimanfoo commented Mar 7, 2017 via email

jakirkham commented Mar 7, 2017

Choose a reason for hiding this comment

jakirkham commented Mar 7, 2017 • edited Loading

alimanfoo commented Apr 25, 2017

jakirkham commented Apr 26, 2017 • edited Loading

alimanfoo commented Apr 26, 2017 via email

jakirkham commented Oct 24, 2017

jakirkham commented Oct 24, 2017

alimanfoo commented Oct 25, 2017

jakirkham commented Oct 25, 2017

jakirkham commented Oct 25, 2017

jakirkham commented Oct 25, 2017

jakirkham commented Oct 25, 2017 • edited Loading

jakirkham commented Oct 25, 2017

jakirkham commented Oct 25, 2017 • edited Loading

jakirkham commented Oct 25, 2017

alimanfoo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alimanfoo commented Oct 25, 2017

alimanfoo commented Oct 25, 2017

alimanfoo commented Oct 28, 2017

alimanfoo commented Oct 28, 2017

alimanfoo commented Nov 14, 2017

jakirkham commented Nov 14, 2017

alimanfoo commented Nov 14, 2017

jakirkham commented Nov 15, 2017

jakirkham commented Mar 3, 2017 •

edited

Loading

jakirkham commented Mar 7, 2017 •

edited

Loading

jakirkham commented Apr 26, 2017 •

edited

Loading

jakirkham commented Oct 25, 2017 •

edited

Loading

jakirkham commented Oct 25, 2017 •

edited

Loading