Add "Table of Content" to the Readme tab

[jayoung-lee]

In a recent user study, participants searched within a package's documentation to find a specific feature (e.g., Ctrl+F to search “highlight” within the documentation of “markdown” package). They usually did this when they failed to find useful information after quickly scanning the documentation. They also tried multiple keywords to make sure that they didn't miss any information (e.g., “highlight”, “syntax”, “code”). It showed that search was an easy way to find information from large data, but it could be difficult to find a good keyword.

But when the feature of interest was listed under “Features” section, users didn’t have to search again. It suggests that a standardized Readme format with better navigation UI (e.g., table of contents) might save some time for participants.

We can consider adding table of contents that's similar to the one on Go's package site:

[sigurdm]

We probably want this in a side-bar.

On mobile this could be omitted or dragged in from the side.

[sigurdm]

Github does this as a drop-down menu:

We could consider something similar

[wrbl606]

Before it gets (hopefully) implemented into pub itself, here's a Firefox extension that achieves the same thing.

[Levi-Lesches]

@sigurdm Any update on whether this is planned? On desktops, the right side is reserved for metadata but the left side of the page is empty and can be used for this, similar to Wikipedia and Google Docs.

I imagine this can cut down on a lot of unnecessary text on a lot of packages, and also prevent ToC's from becoming outdated due to "trivial" commits that forget to add a section to the header

[sigurdm]

No updates on this.

[Levi-Lesches]

Are you looking for PRs? 👀

[sigurdm]

@isoos are we? I think the main issue is to decide on a design, more than the code.

[isoos]

I think we need to separate at least two different aspects here:

• a design and implementation of structural parsing of the markdown content (ideally which we could reuse for changelog parsing), ideally alongside the sanitization of the HTML output, and
• a visual design on how it would be an integral part of the site (both desktop and mobile).

@Levi-Lesches: I think the scope of this requires some discussions and preparations before we are even close to the PRs. Having said that, we are open to ideas and collaboration, do you have something specific in mind?

[Levi-Lesches]

Sure:

On desktop, put the ToC on the left sidebar (currently empty, keeping the middle and right side as it is. On mobile: have a collapsible header (collapsed by default) at the top, right underneath the "Readme | Changelog | Example" etc menu but before the actual text of the README

In terms of parsing, a simple tree hierarchy based on heading levels (#'s in the markdown) would work, the same way GitHub does it

[Levi-Lesches]

Obviously the margins are pretty messed up here because I just cut-and-pasted the HTML in DevTools, but something like this:

Desktop	Mobile

[isoos]

@Levi-Lesches: those are valid outlines, but I think they do need to be extended with more nuanced details, just to name a few out of my head:

• how should the ToC scroll (floating vs not)
• how should it highlight the current position
• how to hide nodes that are distant (e.g. leaf nodes in a different top-level header)
• how and where to cut too long text header content, what is considered an empty content
• what to accept as part of the header content (emojis, images, links, code blocks)
• how to handle uneven structures (where one or more level is missing, e.g. document starts with level 3 then continues with level 1)

Some of these are highly subjective, and very likely will be decided by the team internally, though suggestions here certainly could help. There are a few more considerations which probably are outside of the scope of a tracking issue here, e.g. how do we want to handle it on other Dart-team websites.

If somebody is looking to contribute, I'd suggest to focus on the markdown parsing parts first, it has a better chance to get used first (esp. if it helps with the changelog parsing).

[Levi-Lesches]

Some of these are highly subjective, and very likely will be decided by the team internally, though suggestions here certainly could help.

Makes sense. If you're interested in my 2 cents:

• floating ToC on desktop
• a standard "highlight" box (rectangle with low opacity) for focus. Alternatively, underline with slightly different text color
• I'd say leave all nodes expanded, so the reader can quickly jump between sections. The ToC for an introductory README should rarely be so large that collapsing nodes is necessary. If so, imagine how large the README itself must be! That should be a sign for package authors to split it into multiple files and offer a more condensed README that links to those. Doing this process is how I stumbled across this issue in the first place, actually
• not sure what "empty content" means here, but I'm sure a standard ellipses would work with a reasonable width, text size, and a smaller indentation for nested headers
• try to render the header line (everything but the #) as-is. Package authors are specifically designing their package READMEs with Pub in mind, and I haven't seen anything crazier than emojis and maybe links in headers. Links don't have to be clickable
• accurately reflect the actual structure of the README. Again, the package author is in full control and is trying to make their README as, well, readable as possible on the Pub site, and the docs for writing a good package page already talks about keeping your page clean for Pub.dev specifically

Overall, my guiding principles here are:

• enable this with the least friction for the most possible packages
• take the simple and clean route since the ToC will likely be out of the reader's way
• anyone who has a super complex README structure (eg, inconsistent nesting) should be cleaning that up
• give package authors full control, at the cost of them having to make reasonable choices to get a clean appearance

If somebody is looking to contribute, I'd suggest to focus on the markdown parsing parts first, it has a better chance to get used first (esp. if it helps with the changelog parsing).

I could give it a shot. What kind of function are we looking for? How about something like this:

// app/lib/frontend/templates/views/shared/toc.dart

import "package:markdown/markdown" as md;

Iterable<TocNode> getTocFromMarkdown(md.Document markdownDoc);

This file already has code for rendering a toc, specifically in the admin page of any package:

Given that there's already an API here I can try to sneak in a PR soon

[isoos]

anyone who has a super complex README structure (eg, inconsistent nesting) should be cleaning that up

I understand that you want to have something out fast, but our approach so far was more towards having a correct solution even for the complex use cases. This could be a behavior-shaping feature, where a bad (slightly off) first implementation may influence people towards habits that satisfy the tool, but not their intended audience. We could have rushed out a subpar implementation for this already, but decided against it., because it seems to provide more benefit on the long term if we understand the nuances here.

Also, I wouldn't brush of things and push to the users, because there may be important reasons for such structures, e.g. inconsistent nesting can be often found in changelogs, as people may want to change their level for versions and their internal details, and it is not unseen in readmes either, but wouldn't want to change their older log entries. It is better to have an implementation that handles such cases and enables users, and does not put them in lose-lose situations.

Please don't rush PRs. The TocNode is an incomplete API for this use case: it doesn't hold the content, it was created for a different reason, and while it could be simple to reuse it, it may be better to just have a clean start. As we really want to use this for changelog parsing too, it could be a better starting point to collect different kind of requirements for that and how it would look in a parsed form.