Creating a preview of the outline proposed

[kinow]

Closes #222
Closes #228
Closes #157 (invalid/outdated)
Closes #141 (I think)
Closes #64 (in prerequisites)
Closes #6 (I think? Then we can open new PR's or issues for improving the FAQ if needed?)
Closes #2 (I think? Maybe invalid/outdated now with this change, and also with the novice tutorial work?)
Closes #180 (invalid/outdated the missing chapter won't be a problem as we don't have numbers in chapters - although we do have numbers in sections now… but that issue at least is fixed, and it really bothered me too 😄 )
Closes #217 (link added to the first paragraph of the documentation)
Closes #233 (new .github/workflows/documentation-links.yaml file added as per their docs, also includes link to their docs)

A draft PR with a preview of #222. Following @tetron's suggestion so that it's easier for others to review & send feedback, and also allows me to try to write some of the new text (or copy existing text into new locations, add redirects for old pages, etc.).

• Reproduce the proposed outline with Sphinx
• Add content for the first sections:
  • Landing Page
  • Quick Start
  • Prerequisites
  • Basic Concepts
• Build the old pages so there are no/few links broken after the upgrade (orphan pages in Sphinx)
• Add a note to old pages about the new content
  • Episode 1
  • Episode 2
  • Episode 3
  • Episode 4
  • Episode 5
  • Episode 6
  • Episode 7
  • Episode 8
  • Episode 9
  • Episode 10
  • Episode 11
  • Episode 12
  • Episode 13
  • Episode 14
  • Episode 15
  • Episode 16
  • Episode 17
  • Episode 19
  • Episode 20
  • Episode 21
  • Episode 22
  • Episode 23
  • Episode 24
  • Setup

Once the items above are complete (or at least the Basic Concepts page as well) it will be ready for an initial review from @mr-c , @tetron , @swzCuroverse , and others :-) Then what'll be left are the Core Concepts, FAQ, Tutorials, and How-Tos, which I think will be simpler, but may take longer as some require writing new content.

[kinow]

I think I found a way to deploy it to my gh-pages without touching the common-workflow-language/user_guide branch:

# build passing the `html_baseurl` value
rm -rf _build && sphinx-build -b html . _build/html/ -Dhtml_baseurl="/user_guide/" -W
# use `ghp-import` to push to `gh-pages` in my fork called `kinow`, need the option that creates the .nojekyll file apparently
ghp-import --no-jekyll --push --remote=kinow --branch=gh-pages ./_build/html/

It is not rendering 100% correct for me, but I think that might be due to cache on the GH Pages server side. Looking at my gh-pages branch everything looks OK. Will try again later 👋

https://kinow.github.io/user_guide/index.html

[mr-c]

There is also the RTD PR preview: https://common-workflow-languageuser-guide--226.org.readthedocs.build/en/226/

[kinow]

There is also the RTD PR preview: https://common-workflow-languageuser-guide--226.org.readthedocs.build/en/226/

Ah! I saw the GH action for RTD, and that it built the site correctly. But I couldn't find the link to preview. Thanks!!!

[mr-c]

Seems like this would break all existing links to specific pages in the user guide. We can re-organize, but we should add redirects or other text to hidden pages so that old links aren't completely useless.

[kinow]

Seems like this would break all existing links to specific pages in the user guide. We can re-organize, but we should add redirects or other text to hidden pages so that old links aren't completely useless.

My idea is to slowly move the content from the episodes into the new sections of the user guide. Once the content has been moved, or is not necessary any longer, the text could be replaced by some text like:

“The content of this page has been moved to Section A, and Section B. If you would like to access the old content, you can view the source on [GitHub][link-to-old-page-commit-with-content], or to visualize it in the [Internet Archive WayBack Machine][link-to-wayback-machine].”

Ideally, I think it would be nice to also send a 410, permanently gone status with the response, so search engines know the page is not to be considered in the search results. Or maybe a 301 or 30x if it has a single page to redirect. But that's probably hard with GH pages (probably the closest are the meta refreshes or JS navigation changes).

[swzCuroverse]

Hi Bruno -

This is looking really nice. Thank you!

Here are some thoughts. (I will do a different message for each section)

• In Quick Start Section:
  ** maybe consider explaining what an implementation is - and having a note that links to the list on the website of the current implementations
  ** I am not sure why you use the term [JOB_ORDER] for the input file. I have never heard it called that before, but it might be that way in material I haven't come across.
  ** I personally find the name a little confusing so I would probably call it [INPUTS_OBJECT] based off of the CWL help file. I would also mention that it can be YML/YAML or JSON and mention why you might one over the other. I tend to always use YAML but I am not sure about others, so I don't know if the example case should be in YAML or JSON.

[swzCuroverse]

• In Prerequisites sections:
  ** CWL implementation --> again, maybe we should explain what it is. I think people find this confusing -- and don't know there are other implementations out there and/or why you would use them
  ** Is this needed? #!/usr/bin/env cwl-runner
  ** I would explain this in the text instead of the cwl -- also there is typo (extra c)
  ** true is a Linux command that exits with exit code 0 (succcess).
  ** From the output, I find it hard to parse the command you run. I wonder if you want to put the command you will run in the text as well.
  ** Maybe a little explanation of what validation does other than checks that it is valid?
  ** AH - here is the cwl runner part -- I would maybe just make it another cwl file for clarity so that there is as little extra on the intro files as needed.
  ** Oh, I see the implementation list at the bottom. Might be good to explain why you might use another implementation before the list.

[kinow]

Hi @swzCuroverse !

Thanks a lot for the great feedback.

In Quick Start Section:
** maybe consider explaining what an implementation is - and having a note that links to the list on the website of the current implementations

I've replaced that by CWL Runners. There's still a link to CWL Implementations at the very bottom of the page, but hopefully that won't distract readers.

** I am not sure why you use the term [JOB_ORDER] for the input file. I have never heard it called that before, but it might be that way in material I haven't come across.
** I personally find the name a little confusing so I would probably call it [INPUTS_OBJECT] based off of the CWL help file. I would also mention that it can be YML/YAML or JSON and mention why you might one over the other. I tend to always use YAML but I am not sure about others, so I don't know if the example case should be in YAML or JSON.

Glad I'm not the only one! I mentioned it in #222, it took me a while to figure out what was a job order. I had seen it in the cwltool help and in other documents and also being mentioned in issues & discussions. Eventually it clicked and I realized it was the file with the input parameters for the workflow.

I thought it would be good to introduce this term earlier to users, but actually something like Inputs Object would be a lot more intuitive. @tetron , @mr-c maybe we could phase-out the term Job Order, and slowly replace it in places like cli help, user guide, etc?

I've replaced it in this PR by Inputs Object, and also added the note about it being JSON or YAML. Thanks!

In Prerequisites sections:
** CWL implementation --> again, maybe we should explain what it is. I think people find this confusing -- and don't know there are other implementations out there and/or why you would use them

The Quick Start is using CWL Runner, and the user must be first introduced to the term now here in Prerequisites section. I've added the same link to the implementations at the bottom, and also at a note right after where it appears, alongside a short explanation (will self-edit what I wrote later today).

** Is this needed? #!/usr/bin/env cwl-runner
** AH - here is the cwl runner part -- I would maybe just make it another cwl file for clarity so that there is as little extra on the intro files as needed.

Good point. I tried to not assume the reader knew what a shebang was, but maybe introducing it before the section about the cwl-runner command where it's used could confuse readers.

** I would explain this in the text instead of the cwl -- also there is typo (extra c)
** true is a Linux command that exits with exit code 0 (succcess).

Typo fixed! Thanks!

** From the output, I find it hard to parse the command you run. I wonder if you want to put the command you will run in the text as well.

Me too!!! I spent almost 30 minutes fiddling with the Sphinx settings, trying to change the command output (syntax highlighting and formatting&layout), but I decided to keep writing and leave that for later.

I will have another go at improving it later. I also want to make the output less platform & user-dependent. At the moment it includes my user name, home directory, etc. Since Sphinx uses Python, we can probably write something (or use some existing extension) that runs those commands in a container—for instance—to get the same output independent of the user or environment where the documentation it being built on.

Thank you!!! 🚀

[kinow]

Syntax highlighting for code blocks improved a little. Now each code block contains a small caption, with a link (so users can reference it when sharing links, reporting issues, etc.). It's also noticeable the difference between the command and the output in a bash shell session.

Now for tomorrow: start writing the basic concepts section, where I will try to use some images too, like diagrams to illustrate the basic concepts for readers 🤓

[kinow]

Added the first graph. It was created with Graphviz, following what was used in the Cylc doc repository (credit to them, I learned about it there 😬 ).

The first added contains the processing units of the CWL object model. It uses cwl-utils to find the Process class, then calls __subclasses__() and creates the graphviz plot. Sphinx has a built-in extension for rendering these diagrams.

In theory, when we have a new specification, it should either render the updated object model, or require little maintenance to update the graph 🤞

(little hiccup with Graphviz rendering the elbow-joint edges, will try to fix it later)

[kinow]

I've separated the User Guide content, now in src/, and the Python code, now in cwl.doc Python package. The root directory contains the Markdown, license, .zenodo.json, and other files for the GitHub repository.

Also separated the libraries into groups in setup.cfg. I'm fixing the Travis build, but I think the RTD would need to be updated to install the dependencies.

I've updated the libraries too, and noticed the pydata-sphinx-theme was updated to use the dark theme by default. I've reset it to light, and added the switcher (top-right corner) for users to choose between light & dark themes 👍

Added some more text to the Basic Concepts section, as well as TODO's to consult with Peter/Michael/Sarah/and others.

[mr-c]

I'm fixing the Travis build, but I think the RTD would need to be updated to install the dependencies.

https://docs.readthedocs.io/page/config-file/v2.html
suggest adding a .readthedocs.yaml with the following (or similar)

version: 2

mkdocs:
  configuration: null

python:
  install:
    - method: pip
      path: .
      extra_requirements:
        - all

[kinow]

That fixed it @mr-c ! Thanks!!!

[kinow]

To have Sphinx to build a page, without adding it to a toctree, we just need to make it an orphan page 🙂

Example: file.md:

---
orphan: true
---

# Heading 1

....

[kinow]

Pushing a new commit with the old episodes as orphan pages, and will slowly start adding notes in each page telling users the page is out-of-date, and pointing to a new page when we have a new page with the same content.

http://localhost:8080/setup.md (RTD should render the old links correctly too, even though they don't appear on any page, i.e. old links will keep working)

We have two pages from the old User Guide that will be 404, but I can restore if others think it's important:

• https://github.com/common-workflow-language/user_guide/blob/d8d3fc06e19224dcd7753936acd3eed146f4665f/reference.md
• https://github.com/common-workflow-language/user_guide/blob/d8d3fc06e19224dcd7753936acd3eed146f4665f/aio.md

[kinow]

With regards to #141 ; I don't think there should be a distinction between intro topics and advanced topics

? Note that this would make the top menu to have all these items, causing the top menu to have multiple lines, since Sphinx creates the menus at the left sidebar and at the top automatically from the table of contents. Maybe we could move everything under... Reference Material, or another section?
When creating this structure I looked at MDN (JS Guides & References sections are sub-divided in basic/intermediate/advanced levels), VueJS (Essentials where you find most things to write Vue code, and then other sections for Components In-Depth, Best Practices, etc)

Hmm.. one of the motivations was to have all the topics listed on the left, so people don't have to switch between introductory and advanced. Can that be configured?

Alternatively you can put all the intro and advanced sections under a single "topics" entry

Ah, good suggestion. Let me try it locally and I'll either push it here for preview, or attach some screenshots. Thanks @mr-c!

[kinow]

@mr-c last commit changes the outline to have a single "topics" section, which keeps the top menu functional. I simply merged by putting all the links from the old “Essential Topics” first, followed by the entries from “Advanced Topics”.

The RTD preview should render the updated version soon.

[mr-c]

If there are no objections, I would like to merge & deploy this, then work incrementally in smaller PRs to add more content & polish

[kinow]

If there are no objections, I would like to merge & deploy this, then work incrementally in smaller PRs to add more content & polish

/cc @tetron

[tetron]

I approve this but I don't seem to be assigned as a reviewer so I can't formally hit the approve button?

[mr-c]

Congratulations, thank you @kinow and reviewers!

[kinow]

Thank you everybody for the help & patience! Feel free to send feedback as comments, messages, or follow-up issues 👍