Defer formatting to adjust format based on the ouput context

[jagerber48]

Discussed in #83

^{Originally posted by Batalex November 20, 2023}
The current sciform implementation eagerly formats values as soon as Formatter.__call__ is called.
This discussion is about making sciform implementation lazy so the actual format would happen when the value is printed.

Here is a quick look at what it could look like

# sciform.formatter.py

class Formatter:
    # ...
    # __init__


    def __call__(self, value: Number, uncertainty: Number = None, /):
-        rendered_options = self.user_options.render()
-        if uncertainty is None:
-            return format_num(Decimal(str(value)), rendered_options)
-        else:
-            return format_val_unc(Decimal(str(value)),
-                                  Decimal(str(uncertainty)),
-                                  rendered_options)
+    return FormattedValue(self, value, uncertainty)

+class FormattedValue(str):
+
+    def __init__(self, fmt: Formatter, value: Number, uncertainty: Optional[Number] = None, /) -> None:
+        self.fmt = fmt
+        self.value = value
+        self.uncertainty = uncertainty
+
+    def __str__(self) -> str:
        rendered_options = self.fmt.user_options.render()
        if self.uncertainty is None:
            return format_num(Decimal(str(self.value)), rendered_options)
        else:
            return format_val_unc(Decimal(str(self.value)),
                                  Decimal(str(self.uncertainty)),
                                  rendered_options)

The example above would be similar to the current implementation, but you could then add _repr_html_ or _repr_latex_ methods so that your displayed value uses \textsuperscript in a LaTeX document or <sup></sup> in a notebook.

[jagerber48]

This change may likely remove the latex Formatter option (technically breaking change) in favor of handling latex conversion downstream in the FormattedValue class.

[jagerber48]

edit: I read the suggestion more carefully. FormattedValue already inherits from str so it will have all the appropriate behaviors.

One issue with this pattern is that right now the output of the Formatter is a string, and people might use it as a string. One example I can think of offhand would be appending units

from sciform import Formatter

sform = Formatter(exp_mode="engineering", exp_format="prefix")
num_str = sform(12345)
units = "Hz"
unit_str = f'{num_str}{units}'
print(unit_str)
# 12.346 kHz

Would this still work as expected? Does passing an object into an f string formatter always cast it to a string? What about

from sciform import Formatter

sform = Formatter(exp_mode="engineering", exp_format="prefix")
num_str = sform(12345)
units = "Hz"
unit_str = num_str + units
print(unit_str)
# 12.346 kHz

[Batalex]

Yes, using f-strings would cast the result as a string. You might be onto something with the + operator, though. Formatter could define __add__ and __radd__ so that it can append units.

[jagerber48]

@Batalex ok, I've thought about this a little bit. The main thing that I like about it is that it provides hooks for __repr_html__ that can be used for jupyter or __repr_latex__ that could be used to send strings to Latex.

However, I don't think those things require deferred formatting. Consider

class Formatter:
    ...

    def __call__(
        self: Formatter,
        value: Number,
        uncertainty: Number | None = None,
        /,
    ) -> str:
        ...
        rendered_options = self._user_options.render()
        if uncertainty is None:
            output = format_num(Decimal(str(value)), rendered_options)
        else:
            output = format_val_unc(
                Decimal(str(value)),
                Decimal(str(uncertainty)),
                rendered_options,
            )
        return FormattedValue(output)

class FormattedValue(str):
    def __repr_html__(self):
        return f'<b>{self}</b>'

    def __repr_latex__(self):
        return fr'\text{{{self}}}'

Of course the __repr_html__ and __repr_latex__ functions would be built out to something meaningful.

So @Batalex, does this realize the main thrust of this recommendation? Or was the deferred formatting actually critical to the suggestion?

[Batalex]

I proposed the lazy formatting so that special characters would be rendered appropriately:

\textsuperscript in a LaTeX document or <sup></sup> in a notebook

Your suggestion above could work; it seems more complex to replace those characters afterward compared to the following approach

class FormattedValue(str):

    def __init__(self, fmt: Formatter, value: Number, uncertainty: Optional[Number] = None, /) -> None:
        self.fmt = fmt
        self.value = value
        self.uncertainty = uncertainty

    def __str__(self) -> str:
        rendered_options = self.fmt.user_options.render()
        if self.uncertainty is None:
            return format_num(Decimal(str(self.value)), rendered_options)
        else:
            return format_val_unc(Decimal(str(self.value)),
                                  Decimal(str(self.uncertainty)),
                                  rendered_options)

    def _repr_html_(self):
        rendered_options = self.fmt.user_options.render()
        rendered_options.html = True
        if self.uncertainty is None:
            return format_num(Decimal(str(self.value)), rendered_options)
        else:
            return format_val_unc(Decimal(str(self.value)),
                                  Decimal(str(self.uncertainty)),
                                  rendered_options)


    def _repr_latex_(self):
        rendered_options = self.fmt.user_options.render()
        rendered_options.latex = True
        if self.uncertainty is None:
            return format_num(Decimal(str(self.value)), rendered_options)
        else:
            return format_val_unc(Decimal(str(self.value)),
                                  Decimal(str(self.uncertainty)),
                                  rendered_options)
        

(without all the copy paste)

[jagerber48]

@Batalex see #134. I've come up with much cleaner way to generate latex compatible strings post-facto. I just had to hit my head against some regexes and string replacements. With this I'm tentatively deciding to drop the latex= option. This gives some decent simplifications throughout the formatting algorithm.

This also made it very easy to implement a FormattedNumber object with _repr_latex_ and _repr_html_ and produce output like the following image in a jupyter notebook.

I'm somehow comfortable having the latex and HTML conversion be outside the scope of the formatter. the Formatter converts python number objects into unicode strings, but then the choice of display in unicode/latex/html depends on the context where the output is going to be used. It is also really nice to REMOVE the latex option rather than ADDING an html option to the Formatter. I'm a bit uneasy about how long the list of options for the Formatter already is.

@Batalex are you aware of other tools (other than IPython/jupyter) that would hook into _repr_latex_ or _repr_html_? If so I'd be curious to test out compatibility with those tools.

[Batalex]

Wow that's perfect!

are you aware of other tools (other than IPython/jupyter) that would hook into repr_latex or repr_html? If so I'd be curious to test out compatibility with those tools.

Yes, I would love to see how this fits into quarto

[jagerber48]

Yes, I would love to see how this fits into quarto

Quarto works somewhat as expected. I have this jupyter notebook cell (the same as appears in the latest docs that looks like

When I render this into a quarto html doc I get

So the display functions aren't interleaved as I would naively expect with the print statements. But I think that's a bit of a standard IPython issue that probably has workaround. So not a sciform issue, but rather an IPython thing to figure out.

I'm curious how people want to use the LaTeX features. The display function is one of the most integrated ways to show LaTeX output with python. maptlotlib plots can also show LaTeX. But other than these two cases, I can't imagine how people would be getting LaTeX out of python and into a LaTeX enabled document without a copy and paste step.

Should probably open another issue about this.