Typst 'asis' output

[baptiste]

Description

I'm trying to use a custom knitr engine to enter raw yaml in a chunk and output a raw typst string in a quarto document using the typst output format, but I'm a bit stumped with the output. It seems that the text is escaped, and I don't know how to turn that off.

MRE:

---
title: "Test"
format: 
  typst:
    keep-typ: true
    keep-md: true
---

```{r, echo=FALSE}
library(glue)
library(yaml)
library(knitr)

knitr::knit_engines$set(yamltypst = function(options) {

    options[["results"]] <- "asis"

    # get code together
    code <- paste(options$code, collapse = '\n')

    # parse
    spec <- yaml::yaml.load(code)
    
    # reset language for markdown highlighting
    options$engine <- "yaml"

    out <- glue::glue("#strong(  [{ spec }] )")
    
    knitr::engine_output(
      options,
      options$code,
      out = out
    )
  })

```

## Quarto

Quarto enables you to weave together content and executable code into a finished document. 

```{yamltypst, echo=FALSE, asis=TRUE}
stuff: yamlcontent
```

Output:

[..truncated..]

#block[
\#strong\( \[yamlcontent\] )

]

Output I am hoping to get (no block container, and no escaping of characters):

[..truncated..]

#strong( [yamlcontent] )

So that typst processes it. When I switch to a different format, eg. format: markdown, this quoting doesn't seem to happen. Can it be turned off?

[mcanouil]

If you want to add/emit raw "whatever":

• `something`{=whatever}
• ```{=whatever}
  FYI, the recommended syntax for code cell option in Quarto in YAML not R inline.

[cderv]

knitr has a function to emit raw block - you should use it.

out <- knitr::raw_block(glue::glue("#strong(  [{ spec }] )"), "typst")

Also in ```{yamltypst, echo=FALSE, asis=TRUE} , the asis option is not a known knitr options.

You already used options[["results"]] <- "asis" in your engine, so it should be enough.

Hope it helps

---
title: "Test"
format: 
  typst:
    keep-typ: true
    keep-md: true
---

```{r, echo=FALSE}
library(glue)
library(yaml)
library(knitr)

knitr::knit_engines$set(yamltypst = function(options) {

    options[["results"]] <- "asis"

    # get code together
    code <- paste(options$code, collapse = '\n')

    # parse
    spec <- yaml::yaml.load(code)
    
    # reset language for markdown highlighting
    options$engine <- "yaml"

    out <- knitr::raw_block(glue::glue("#strong(  [{ spec }] )"), "typst")
    
    knitr::engine_output(
      options,
      options$code,
      out = out
    )
  })
```

## Quarto

Quarto enables you to weave together content and executable code into a finished document.

```{yamltypst}
stuff: yamlcontent
```

[baptiste]

Nice! Indeed, the lua filter approach is a good alternative because it won't rely on knitr.

I have something more generic in mind though, because here we're relying on the content being Yaml, which happens to coincide with quarto+pandoc, but it's easy to imagine similar input-wrangling activities with other plain text formats, such as TOML, JSON, or even the likes of Mermaid, TIkz, etc. where, unlike the current workflows, the raw content would be injected into the document's output format rather than evaluated.
Imagine a Typst library that would convert Tikz code on-the-fly into its native Cetz library, for example, or draw diagrams from a Graphviz input representation, etc. – it'd be nice to have the quarto input written in a simple cell (not sure what format is most suitable: ```{=custom} or ::: custom, or ```{custom}etc.) and have a Lua filter grab the content and inject it in a user-specified string. For maximum flexibility this would be a user-defined template a la glue, say:

```{agenda}
#| highlight : yaml
#| wrap : "#let a = yaml.decode( \"{agenda-code}\" )"
...
```

where those arguments could also be set as defaults for the whole document.

[cscheid]

a Lua filter grab the content and inject it in a user-specified string

I don't like this approach because it's very hard to make it work well, especially with the myriad escaping problems that can come up. If you're going to do something like that, you really ought to be sending a very boring string, like a base64-encoded JSON blob. Imagine having to deal with escaping single vs double vs smart quotes, escaping all different possible new line characters, etc.

[baptiste]

True, and i did run into this, but insofar as typst is concerned there seems to be a good workaround, which is to wrap the content as raw, and then retrieve the text with .text :

#let stuff = `{inject here)`
#yaml.decode(stuff.text)

[baptiste]

It's now a typst template: https://typst.app/universe/package/yagenda

I'm still pondering the best approach to use it in quarto; here's the custom knit-engine version (requires using a more recent Typst to compile, I set it with export QUARTO_TYPST=/usr/local/bin/typst at my own risks)

In practice I'm finding that having the YAML in an external file is actually alright, but I still like the idea of being able to embedded it in the qmd file directly, especially when it's very short. Currently the chunk options are a little obtrusive, and obviously the first setup chunk is very distracting.

[cderv]

requires using a more recent Typst to compile

Which version do you need ? Our current 1.5 pre-release now includes Typst 0.11.0 (since 1.5.29).

Currently the chunk options are a little obtrusive, and obviously the first setup chunk is very distracting.

You would need a companion R package that registers this engine on load or through a function call. This would avoid this long setup chunk. (Sourcing a R script for a URL like a gist is also possible but not really a good practice I guess)

In practice I'm finding that having the YAML in an external file is actually alright

One think that could work with knitr built in engine would be to write to file the content of the chunk, so that you can load it in your raw typst block.

---
name: "Pumpkins and peanuts committee"
date: "03/01/2000"
time: "2 pm"
location: "baseball field"
invited: 
  - Sally
  - Shroeder
  - Pig-pen
  - Marcie
format: 
  typst:
    keep-typ: true
    keep-md: true
    template-partials:
      - typst-show.typ
        
engine: knitr
---

```{cat}
#| engine-opts:
#|   file: agenda.yaml  
#| echo: false
"admin":
  Topic: Misc. admin
  Time: 5 mins
  Lead: baptiste 
  Purpose: Inform, Admin
  Preparation: |
    - Anything to change on the agenda? 
    - Read minutes from last meeting
    - Review action points
  Process: |
    - Check if agenda needs to be adjusted
    - Check if everyone's happy with the minutes
    - List top priorities of meeting and items needing decisions
    
"snoopy update":
  Topic: Snoopy's latest 
  Time: 10 mins
  Lead: Charlie Brown
  Purpose: Share, Discuss
  Preparation: |
    - Bring your favorite comic strip
    - Reflect on Snoopy's recent escapades
  Process: |
    - Discuss Snoopy's ongoing "battle" with the Red Baron
    - Share anecdotes of Snoopy's fiercest war moves
    - Plan a group outing to visit Snoopy's doghouse
"woodstock sighting":
  Topic: Woodstock's whereabouts
  Time: 5 mins
  Lead: Linus
  Purpose: Locate, Update
  Preparation: |
    - Bring binoculars for bird-watching
    - Brush up on bird calls
  Process: |
    - Share any recent sightings or chirpings from Woodstock
    - Discuss strategies for bringing Woodstock back to the tunes
    - Plan an upside-down birdhouse-building workshop 
"great pumpkin plans":
  Topic: Prep for Great Pumpkin
  Time: 15 mins
  Lead: Linus
  Purpose: Plan, Excite
  Preparation: |
    - Bring your pumpkin-carving tools
    - Practice your most sincere pumpkin patch speech
  Process: |
    - Discuss tactics for maximizing sincerity in the pumpkin patch
    - Brainstorm new ways to attract the Great Pumpkin's attention
    - Assign roles for the annual pumpkin carving contest
"philosophical discussion":
  Topic: Philosophical Musings
  Time: 20 mins
  Lead: Lucy and Charles
  Purpose: Ponder, Reflect
  Preparation: |
    - Bring your favorite existentialist quotes
    - Contemplate the meaning of life #v(100%)
  Process: |
    - Engage in deep philosophical discussions under the night sky
    - Debate the nature of happiness, existence, and 5c psychiatry
    - Seek wisdom from Linus's trusty security blanket
"beethoven appreciation":
  Topic: Beethoven's Legacy
  Time: 10 mins
  Lead: Schroeder
  Purpose: Appreciate, Discuss
  Preparation: |
    - Bring your favorite Beethoven compositions
    - Practice your air piano skills
  Process: |
    - Listen to Schroeder perform select Beethoven pieces
    - Discuss the timeless appeal of Beethoven's music
    - Plan a Beethoven-themed recital for Christmas
"baseball game":
  Topic: Baseball Game
  Time: 30 mins
  Lead: Charlie Brown
  Purpose: Play, Bond
  Preparation: |
    - Bring your baseball glove and bat
    - Review the rules of baseball
  Process: |
    - Divide into teams for an exciting game of baseball
    - Cheer on Charlie Brown as he attempts to finally kick that football
    - Enjoy snacks and camaraderie under the shade of the old oak tree
    
"summer camp":
  Topic: Summer Camp Adventures
  Time: 15 mins
  Lead: Peppermint Patty
  Purpose: Plan, Excite
  Preparation: |
    - Bring ideas for summer camp activities!
    - Check availability of camping gear!
  Process: |
    - Brainstorm camping trip destinations and outdoor activities
    - Discuss potential guest speakers or counselors
    - Organize a talent show and marshmallow roasting competition
"marcie school update":
  Topic: Marcie's Academic Progress
  Time: 10 mins
  Lead: Peppermint Patty
  Purpose: Discuss, Support
  Preparation: |
    - Bring Marcie's recent report card
    - Reflect on Marcie's study habits and strengths
  Process: |
    - Share updates on Marcie's academic achievements and challenges
    - Discuss strategies to help Marcie excel in school
    - Offer encouragement and support to Marcie in her studies
"linus blanket workshop":
  Topic: Linus and his blanket
  Time: 20 mins
  Lead: Lucy
  Purpose: Understand, *Support*
  Preparation: |
    - Bring your own security blanket (optional)
    - Reflect on the significance of Linus's blanket
  Process: |
    - Discuss the history and symbolism of Linus's security blanket
    - Explore ways to help both Linus and Snoopy 
    - Release a blanket statement
"snoopy plane fights":
  Topic: Snoopy vs the Baron
  Time: 10 mins
  Lead: Woodstock
  Purpose: Support
  Preparation: |
    - Bring your favorite Snoopy flying ace memory
    - Reflect on Snoopy's aerial prowess
  Process: |
    - Share anecdotes from Snoopy's battles
    - Discuss the psychological implications of Snoopy's dogging of bullets
    
```

```{=typst}
#let yamlentries = yaml("agenda.yaml")
#agenda-table(yamlentries)
```

The raw block could even be set in a template partials directly, or a header include directly.

If you package all this in a Quarto extension, you could just get an agenda output using

• A cat knitr engine to write to YAML file
• a agenda.yaml in the project document root

The extension could have a shortcode to place the agenda in the right place in the document body.

Using Lua approach discussed above is also a good one to avoid depending on a computation engine to write YAML from a Markdown formatted content (YAML meta block or special div block, or special raw block with attributes). You could imagine do the same logic (get yaml from the in document and write to a file or inset directly) from a code block

```{.yaml .agenda}
...
```

where you could catch this code block, and process it for your agenda template.

Anyhow, great Typst template!

[baptiste]

Which version do you need ? Our current 1.5 pre-release now includes Typst 0.11.0 (since 1.5.29).

Thanks, I hadn't seen that recent change.

Currently the chunk options are a little obtrusive, and obviously the first setup chunk is very distracting.

You would need a companion R package that registers this engine on load or through a function call. This would avoid this long setup chunk. (Sourcing a R script for a URL like a gist is also possible but not really a good practice I guess)

True, I was mostly aiming for a proof of principle so I didn't wrap it in a package. I wish there was a way to specify a "setup" file in the YAML header, if just to load a set of packages. Something like

knitr:
   startup: 
      text: |
         library(glue)
         source("fun.R")

In practice I'm finding that having the YAML in an external file is actually alright

One think that could work with knitr built in engine would be to write to file the content of the chunk, so that you can load it in your raw typst block.
...
If you package all this in a Quarto extension, you could just get an agenda output using

* A `cat` knitr engine to write to YAML file

* a `agenda.yaml` in the project document root

The extension could have a shortcode to place the agenda in the right place in the document body.

Yep, cat() is the route I initially took (without wrapping it as an extension), and I had the filename specified in the YAML for more flexibility:

---
agenda: "2024-04-05-agenda.yaml"
---

```{r, echo=FALSE, results='hide'}
meta <- rmarkdown::metadata
agenda <- meta$agenda
```

Using Lua approach discussed above is also a good one to avoid depending on a computation engine to write YAML from a Markdown formatted content (YAML meta block or special div block, or special raw block with attributes). You could imagine do the same logic (get yaml from the in document and write to a file or inset directly) from a code block

```{.yaml .agenda}
...

where you could catch this code block, and process it for your agenda template.

Yes, I think this is the best option, so I'm hoping to try it at some point. Unfortunately the Lua side of things is always a bit of a barrier :/
I imagine the logic could be:

function Code(code)
  if not code.classes:includes('agenda') then
    return nil
  end
  -- capture the code block as raw text
  local rawcode = code.content ???
  
  -- append some typst wrapper
  local fullstring = "#let yamlcontent = ` " + rawcode + " `.text "
  -- or write the content in an external file like the cat engine does
  pandoc.write(rawcode, filename)
  
  -- output raw typst block
  pandoc.raw(fullstring, "typst")

  return result
end

Anyhow, great Typst template!

Thanks! Credits should go to PgSuper who wrote the table code; I merely tweaked it and wrapped it up.

[cderv]

I wish there was a way to specify a "setup" file in the YAML header, if just to load a set of packages. Something like

yeah there is no such thing in knitr - you need to include as code cell what you want to be run in the knitting process.

I guess the thinking was that it is always easy enough to add the code as startup chunk

```{r}
#| label: startup
library(glue)
source(fun.R)
```

Potentiall in a share included file to be added in all document.

Startup file like .Rprofile is run by R though - I don't think we run with --vanilla but there will be the topic of R working directory at startup to pick up a project .Rprofile. So not really safe probably.

This topic has come up several time, especially because Quarto starts new process for engine each time and you may want some common configuration for all your runs. Like startup file but tied to quarto execution.

Interesting feedback thanks !

Unfortunately the Lua side of things is always a bit of a barrier :/ I imagine the logic could be:

Yes Lua requires a bit of a learning curve, but we hope our targetted documentation and existing filters help find example to start with.
https://quarto.org/docs/extensions/lua.html

Some comment in the logic to give you hint (that you could have found yourself probably 😅 )

• CodeBlock would be the element to search for - see pandoc's doc for its content https://pandoc.org/lua-filters.html#type-codeblock
• code.text would have the content of the block
• pandoc.RawBlock() would be what to use for creating some Raw Content from AST for pandoc to write asis
• pandoc.write() is not a writing to file function, this is to run a Pandoc writer to convert. See io and file base API for writing to file (https://quarto.org/docs/extensions/lua-api.html#lua-base-api)
• .. is what to use to concatenate (not +)

THe logic is the right one, and you are really close with using the right syntax ! Good little exercise :)

[baptiste]

THe logic is the right one, and you are really close with using the right syntax ! Good little exercise :)

Full disclosure, I did not even try to check any of the syntax when I wrote this at breakfast :P For me the barrier with Lua is more of a debugging one: I have not found a setup where I can just try something and iterate like in R's REPL, but with the quarto+pandoc environments and functions at hand like the quarto process has them set up. Haven't tried for a while though, so I should definitely revisit the docs.

[cderv]

I have not found a setup where I can just try something and iterate like in R's REPL, but with the quarto+pandoc environments and functions at hand like the quarto process has them set up

Yeah I can relate to that. I don't have anything great to offer. I am mainly relying of old good console log debugging. So I use quarto.log.output() a lot to print where I want to check and debug things.

For non specific Quarto lua processing , pandoc lua -i has been also helpful to run Lua interactively in console (available since 3.1.2 https://pandoc.org/releases.html#pandoc-3.1.2-2023-03-26). It works only in bare pandoc context for now.

I never tried setting debugging with Quarto directly : https://pandoc.org/lua-filters.html#debugging-lua-filters

Hope it helps