Upgrade to jekyll 4.x

[0xdevalias]

Currently we're on v3.8.0 (ref):

• https://jekyllrb.com/news/2018/04/19/jekyll-3-8-0-released/
• https://jekyllrb.com/news/2018/05/01/jekyll-3-8-1-released/
• https://jekyllrb.com/news/2018/05/19/jekyll-3-8-2-released/
• https://jekyllrb.com/news/2018/06/05/jekyll-3-8-3-released/
• https://jekyllrb.com/news/2018/09/19/security-fixes-for-3-6-3-7-3-8/
• https://jekyllrb.com/news/2018/11/04/jekyll-3-8-5-released/
• https://jekyllrb.com/news/2019/07/02/jekyll-3-8-6-released/
• https://jekyllrb.com/news/2019/08/20/jekyll-4-0-0-released/

  • Ruby 2.4.0 or greater is now required.

  • Rouge 3.0 or greater is now required for syntax highlighting.

  • Jekyll builds should be much faster.

  • Kramdown 2.1 is now the default markdown engine.

  • Sass processing should be faster.

  • We dropped support for a lot of stuff, specifically:

    • Pygments

    • RedCarpet

    • RDiscount

    • https://jekyllrb.com/docs/configuration/markdown/

      • Kramdown is the default Markdown renderer for Jekyll.

• https://jekyllrb.com/news/2020/05/08/jekyll-4-0-1-released/
• https://jekyllrb.com/news/2020/05/27/jekyll-4-1-0-released/

Changelog:

• https://jekyllrb.com/docs/history/

Upgrading:

• https://jekyllrb.com/docs/upgrading/
• https://jekyllrb.com/docs/upgrading/3-to-4/

See also:

• Explore alternatives to Jekyll (Hugo, etc) #28

[0xdevalias]

Noting old gem version incompatibilities here for reference:

• You have requested: jekyll ~> 4.1.0 The bundle currently has jekyll locked at 3.8.0. Try running 'bundle update jekyll'
• jekyll-compose was resolved to 0.8.0, which depends on jekyll (~> 3.0)
  • https://github.com/jekyll/jekyll-compose/releases
  • v0.12+ seems to support jekyll 4

---

This was done after updating a couple of things already:

⇒  bundle outdated
Fetching gem metadata from https://rubygems.org/........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...

Outdated gems included in the bundle:
  ..snip..
  * jekyll (newest 4.1.1, installed 3.8.7, requested ~> 3.8.7) in group "default"
  * jekyll-admin (newest 0.10.2, installed 0.8.0) in group "jekyll_plugins"
  * jekyll-feed (newest 0.15.0, installed 0.9.3) in group "jekyll_plugins"
  * jekyll-github-metadata (newest 2.13.0, installed 2.9.4) in group "jekyll_plugins"
  * jekyll-redirect-from (newest 0.16.0, installed 0.13.0) in group "jekyll_plugins"
  * jekyll-sass-converter (newest 2.1.0, installed 1.5.2)
  * jekyll-seo-tag (newest 2.6.1, installed 2.4.0) in group "jekyll_plugins"
  * jekyll-sitemap (newest 1.4.0, installed 1.2.0) in group "jekyll_plugins"
  * jekyll-twitter-plugin (newest 2.1.0, installed 2.0.0) in group "jekyll_plugins"
  * jekyll-webmention_io (newest 3.3.5, installed 2.9.1) in group "jekyll_plugins"
  ..snip..

• https://github.com/jekyll/jekyll-admin/releases
  • https://github.com/jekyll/jekyll-admin/blob/v0.9.0/jekyll-admin.gemspec#L29
    • v0.9.0+ seems to support jekyll 4
  • Update sinatra from 1.4.x to latest (2.0.x) jekyll/jekyll-admin#601
• https://github.com/jekyll/jekyll-feed/releases
• https://github.com/jekyll/github-metadata/releases
• https://github.com/jekyll/jekyll-redirect-from/releases
• https://github.com/jekyll/jekyll-sass-converter/releases
  • bundled as part of jekyll
• https://github.com/jekyll/jekyll-seo-tag/releases
• https://github.com/jekyll/jekyll-sitemap/releases
• https://github.com/rob-murray/jekyll-twitter-plugin/releases
• https://github.com/aarongustafson/jekyll-webmention_io/releases
  • Compatibility with Jekyll 4 aarongustafson/jekyll-webmention_io#130
    • https://github.com/nhoizey/jekyll-webmention_io/tree/nhoizey-jekyll4
  • Update webmention dependency from 0.1.6 to latest (v5.0.0) aarongustafson/jekyll-webmention_io#138

---

• Resolved?

Ruby Sass has reached end-of-life and should no longer be used.

* If you use Sass as a command-line tool, we recommend using Dart Sass, the new
  primary implementation: https://sass-lang.com/install

* If you use Sass as a plug-in for a Ruby web framework, we recommend using the
  sassc gem: https://github.com/sass/sassc-ruby#readme

* For more details, please refer to the Sass blog:
  https://sass-lang.com/blog/posts/7828841

This is a dependency of jekyll-sass-converter which is bundled with jekyll. The latest version appears to correctly update this to use sassc as recommended:

• https://github.com/jekyll/jekyll-sass-converter/blob/master/jekyll-sass-converter.gemspec#L19
  • spec.add_runtime_dependency "sassc", "> 2.0.1", "< 3.0"

[0xdevalias]

⇒  bin/serve
Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
Require just the needed backports instead, or 'backports/latest'.

..snip..

      Generating...
   GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.

       Jekyll Feed: Generating feed for posts
Markdown processor: "redcarpet" is not a valid Markdown processor.
                    Available processors are: kramdown

  Conversion error: Jekyll::Converters::Markdown encountered an error while converting '_posts/2011-09-16-etc-initd-blog-start.md':
                    Invalid Markdown processor given: redcarpet
             ERROR: YOUR SITE COULD NOT BE BUILT:
                    ------------------------------------
                    Invalid Markdown processor given: redcarpet
                    ------------------------------------------------
      Jekyll 4.1.1   Please append `--trace` to the `serve` command
                     for any additional information or backtrace.
                    ------------------------------------------------
GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.

---

• Resolved?

Doing require 'backports' is deprecated and will not load any backport in the next major release. Require just the needed backports instead, or 'backports/latest'.

• backports is required by sinatra-contrib and packable
  • packable is required by nmatrix: packable (~> 1.3, >= 1.3.5), any version of backports
  • sinatra-contrib is required by jekyll-admin: sinatra-contrib (~> 1.4), backports (>= 2.0)

My guess is this deprecated usage will be part of nmatrix since it's so old/seemingly unmaintained.

---

• Resolved?

Markdown processor: "redcarpet" is not a valid Markdown processor. Available processors are: kramdown

Support for redcarpet and other markdown processors was dropped. Need to update our usages to kramdown.

• https://jekyllrb.com/docs/history/#v4-0-0
• Remove support for Redcarpet jekyll/jekyll#6981
• Remove support for Redcarpet jekyll/jekyll#6987

Our Jekyll config will likely require changes too:

• https://jekyllrb.com/docs/configuration/default/
• https://jekyllrb.com/docs/configuration/markdown/

redcarpet:
  extensions:
    - "tables"
    - "autolink"
    - "strikethrough"
    - "space_after_headers"
    #- "with_toc_data"
    - "fenced_code_blocks"

• https://kramdown.gettalong.org/parser/gfm.html
  • https://github.com/kramdown/parser-gfm
  • https://github.com/kramdown/parser-gfm#options
  • this seems to support strikethrough using double tildes: ~~like this~~
• https://kramdown.gettalong.org/converter/html.html#toc
• https://kramdown.gettalong.org/syntax.html#tables
• https://kramdown.gettalong.org/syntax.html#fenced-code-blocks
  • by default this uses a different style than GFM, but the GFM parser supports triple backticks as expected
• https://kramdown.gettalong.org/syntax.html#automatic-links
  • Autolink in kramdown? barryclark/jekyll-now#459 (comment)
  • this seems to require explicit syntax that differs from our previous GFM-styled usage
  • Resolved?
    • eg. http://127.0.0.1:4000/devalias/2018/05/13/usb-reverse-engineering-down-the-rabbit-hole/
    • see Autolinking not working like GFM kramdown/parser-gfm#17

Kramdown seems to have a number of other issues too, eg:

• not autolinking
  • Resolved? yes
  • http://127.0.0.1:4000/devalias/2018/05/13/usb-reverse-engineering-down-the-rabbit-hole/
  • http://127.0.0.1:4000/devalias/2018/04/07/diy-light-therapy-red-infrared-cold-laser-lllt-blue-uv/
  • http://127.0.0.1:4000/devalias/2016/07/07/access-files-from-other-time-machine-backups/
  • http://127.0.0.1:4000/devalias/2014/06/26/play-framework-seperated-ws-library/
• linking syntax being rendered as tables when text includes |
  • Resolved? yes
  • http://127.0.0.1:4000/devalias/2017/07/14/link-dump-clearing-out-my-todo-list/
  • http://127.0.0.1:4000/devalias/2016/09/16/ctf-snippets-xss-gif/
• embedded gist doesn't seem to load
  • Resolved? yes
  • this might be because my github username changed?
  • it also seems to render as 404: Not Found in the generated HTML now.. so we could potentially find it that way?
  • http://127.0.0.1:4000/devalias/2015/04/20/dogedraw-now-with-more-nyaan/
  • http://127.0.0.1:4000/devalias/2014/05/15/java-scala-future-promise-map-headsplode/
  • _posts/2013-07-13-nmap-sh-saving-precious-seconds.md
  • _posts/2013-08-03-rails-lessons-learned-the-hard-way-1-db-migrate.md
  • _posts/2013-08-10-erpscan-automator-because-manual-is-meh.md
  • _posts/2013-08-29-reversing-powershell-securestring-for-fun-and-profit.md
  • _posts/2013-11-21-vfeed-wrapper-helper-scripts-for-speed-and-efficiency.md
  • _posts/2014-04-02-hacking-unicoins-for-fun-and-profit.md
  • _posts/2014-05-15-java-scala-future-promise-map-headsplode.md
  • _posts/2015-04-20-dogedraw-now-with-more-nyaan.md

Not specifically Kramdown related.. but:

• Twitter urls being rendered by lazy twitter plugin even when inside codeblocks
  • Resolved? yes
  • http://127.0.0.1:4000/devalias/2017/07/16/bugcrowd-levelup-2017/

---

• Resolved?

GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.

• https://github.com/jekyll/github-metadata/blob/master/docs/authentication.md
  • JEKYLL_GITHUB_TOKEN
  • ~/.netrc
  • OCTOKIT_ACCESS_TOKEN
  • .env

With my dotfiles layout, I believe we could also add this to ~/.localrc.

I think using a gitignored .env.local file, and also including a .env.example or similar would probably be useful.. if supported.

[0xdevalias]

• Resolved?

/Users/devalias/dev/_sites/devalias.net/_plugins/lazy_tweet_embedding.rb:11: warning: calling URI.open via Kernel#open is deprecated, call URI.open directly or use URI#open

It seems as though this may be the original source for this plugin:

• https://github.com/takuti/jekyll-lazy-tweet-embedding

It hasn't been updated since 2017, and requires manually copying the plugin source into our project. We might be better off just removing it and manually using a twitter filter around links instead?

[0xdevalias]

Kramdown seems to have weird quirks that don't really match to GitHub Flavoured Markdown (GFM) as I would have expected it.

An alternative would be to use commonmark (which also appears to be what GitHub pages uses), but there seem to be 2 main versions that appear to use the same underlying commonmarker dependency, and both plugins seem somewhat outdated/unmaintained.

• https://github.com/jekyll/jekyll-commonmark
  • Updated in 2020, seems active
  • How does this differ from jekyll-commonmark-ghpages? jekyll/jekyll-commonmark#39 (How does this differ from jekyll-commonmark-ghpages?)
• https://github.com/github/jekyll-commonmark-ghpages
  • Updated in 2019, doesn't seem active
  • How does this differ from jekyll-commonmark? github/jekyll-commonmark-ghpages#17 (How does this differ from jekyll-commonmark?)
  • Is this project still actively maintained, or is it abandoned? github/jekyll-commonmark-ghpages#18 (Is this project still actively maintained, or is it abandoned?)
• https://github.com/gjtorikian/commonmarker
  • Updated in 2020, seems active
• https://github.com/github/cmark-gfm
  • Updated in 2020, seems active

The github/pages-gem claims that it provides gem versions in line with what GitHub pages uses, but according to github/pages-gem#699 it seems to use Jekyll's default parser (Kramdown), rather than the (presumably still accurate, but seemingly unmaintained) CommonMarkGhPages gem.

Though as I note in github/jekyll-commonmark-ghpages#18:

Though looking at the pages-gem readme shows that we can also check the GH pages dependency versions online:

• https://pages.github.com/versions/

Which shows a usage of jekyll 3.8.7 and jekyll-commonmark-ghpages 0.1.6, which appears to be the latest released version of this project.

---

commonmarker configuration:

• https://github.com/gjtorikian/commonmarker#options
• https://github.com/gjtorikian/commonmarker#extensions

I explore some relevant sounding choices in github/pages-gem#699 (comment), summarised as:

commonmark:
  options: ["FOOTNOTES", "STRIKETHROUGH_DOUBLE_TILDE", "VALIDATE_UTF8", "GITHUB_PRE_LANG", "HARDBREAKS"]
  extensions: ["autolink", "strikethrough", "table", "tasklist", "tagfilter"]

When I don't have UNSAFE enabled, some of my existing pages fail, instead outputting <!-- raw HTML omitted -->.

The main page that I noticed this on was _posts/2017-07-16-bugcrowd-levelup-2017.md, which appears to use syntax like the following:

{% raw %}
<pre>
Some text here.
</pre>
{% endraw %}

Setting the UNSAFE option 'fixes' this.. but probably isn't an ideal choice.

Searching for <!-- raw HTML omitted --> in the generated pages shows up a number of offending pages:

• _site/author/devalias/rss.xml
• _site/coaching/index.html
  • seems to strip the tags from <a href="mailto:glenn+coaching@devalias.net?subject=Bulletproof%20Coaching">glenn+coaching@devalias.net</a>
• _site/devalias/2012/01/10/plesk-adding-programs-to-a-chrooted-ssh/index.html
  • may have been caused by a typo on a triple backtick code fence?
• _site/devalias/2013/06/17/gists-on-tumblr/index.html
• _site/devalias/2013/07/13/nmap-sh-saving-precious-seconds/index.html
• _site/devalias/2013/08/03/rails-lessons-learned-the-hard-way-1-db-migrate/index.html
• _site/devalias/2013/08/10/erpscan-automator-because-manual-is-meh/index.html
• _site/devalias/2013/08/29/reversing-powershell-securestring-for-fun-and-profit/index.html
• _site/devalias/2013/11/21/vfeed-wrapper-helper-scripts-for-speed-and-efficiency/index.html
• _site/devalias/2014/04/02/hacking-unicoins-for-fun-and-profit/index.html
• _site/devalias/2014/05/15/java-scala-future-promise-map-headsplode/index.html
• _site/devalias/2015/04/20/dogedraw-now-with-more-nyaan/index.html
• _site/devalias/2016/08/24/starting-a-new-web-application-1-an-exploration-of-options/index.html
• _site/devalias/2016/09/07/graphql-why-you-should-care/index.html
• _site/devalias/2017/07/16/bugcrowd-levelup-2017/index.html
  • seems to strip the <a> anchor tags for heading links
  • as well as {% raw %}<pre>foo</pre>{% endraw %} usages (which would be better replaced by triple backticks anyway)
• _site/devalias/2018/05/13/usb-reverse-engineering-down-the-rabbit-hole/index.html
  • seems to strip the <a> anchor tags for heading links
• _site/devalias/2018/09/15/forming-serverless-clouds-aws-cloudformation-sam-cdk-amplify/index.html
• _sites/devalias.net/_site/now/index.html
  • strips the <center> tags and {% include goodreads_reading.html %}
  • strips {% include trakt_watching.html %}
• _site/personal/interests/dance/index.html
• as well as pretty much every one of the pages in _sites/devalias.net/_site/tag/*/rss.xml

Based on this, it seems like we may have to use UNSAFE. Though given this is a static site I control, that's probably not the worst thing in the world.

It also appears that the tagfilter extension breaks some of our usages in pages/templates/includes, for examples the scripts/tags used in the goodreads/watching embeds in now.md

• {% include goodreads_reading.html %}
• {% include trakt_watching.html %}

Again, given I control the source/inputs to this, there probably isn't a lot of real world risk in removing the tagfilter extension as well.

---

According to github/pages-gem#651, the best recommended way to support Jekyll 4 on GitHub pages currently is to use GitHub actions:

• https://jekyllrb.com/docs/continuous-integration/github-actions/
  • https://github.com/marketplace/actions/jekyll-actions

    • A GitHub Action to build and publish Jekyll sites to GitHub Pages

    • https://github.com/MichaelCurrin/jekyll-actions-quickstart
• https://sujaykundu.com/blog/post/deploy-jekyll-using-github-pages-and-github-actions

[0xdevalias]

Fixed up most of the broken gist embeds, but one page still appears to be having issues, seems it may be a bug in jekyll-gist:

• noscript output seems to break sometimes when using Commonmark jekyll/jekyll-gist#82

[0xdevalias]

We have previously used an inlined plugin called postfiles, that appears as though it was copied from:

• https://github.com/indirect/jekyll-postfiles/blob/master/_plugins/postfiles.rb

This is used with syntax such as the following:

 ![who dares enter my domain]({% postfile cat.jpg %})

And appears to be used in the following posts:

• _posts/2011-12-30-who-dares-enter-my-domain.md
• _posts/2013-08-16-rockmelt-blog-rockmelt-browser-retirement-july.md
• _posts/2014-02-13-y-u-no-update-kali.md
• _site/devalias/2011/11/07/bye-bye-russian-spam/index.html
• _site/devalias/2013/11/28/thecus-nas-devalias-says-no/index.html

Both of the .html files appear to contain it as the 'prev/next post' excerpt, so isn't really of concern to us here.

---

There appears to be a maintained (last commit 2020, last release 2019), gem based jekyll plugin that also provides this functionality, though it differs slightly in it's usage:

• https://github.com/nhoizey/jekyll-postfiles

Instead of using a _postfiles folder, it appears to just use per-post folders under the _posts directory.

Instead of having to explicitly use a liquid filter (eg. ![who dares enter my domain]({% postfile cat.jpg %})), we can just use a regular markdown image tag with a relative path (eg. ![who dares enter my domain](cat.jpg))

It's probably a better choice to use going forward.

• Resolved?