Add accuracy document

[kgryte]

This PR

• adds accuracy as a design topic in order to provide guidance to specification implementors as to expected accuracy of specification-defined numerical functions.

Questions

• Defining accuracy requirements for statistical and linear algebra APIs is a bit fraught, as it is hard to make general rules, in contrast to canonical mathematical functions. At the moment, I have left specification for these API areas as TODOs, and, atm, I am leaning toward non-guidance (i.e., "this standard does not specify accuracy requirements for (insert APIs) due to reasons X, Y, and Z; however, this standard does expect that conforming implementations will make a best-effort attempt to ensure their implementations are theoretically sound and numerically robust."

  • Update: went with non-guidance.

[rgommers]

At the moment, I have left specification for these API areas as TODOs, and, atm, I am leaning toward non-guidance

That does seem reasonable. For the test suite we'll still have to put in some absolute and relative tolerance w.r.t expected values though; those can be derived from libraries' existing test suites. Putting those numbers in the standard could make sense, but since they'd only be based on empirical evidence, it also wouldn't have much significance - mostly just serve as documentation.

[kgryte]

@rgommers Thanks for the review.

Re: non-guidance. I think we have a couple of options.

1. We could specify a reference specification-compliant implementation (e.g., NumPy), and then state that all specification-compliant array libraries must be within a specified relative tolerance.

   • pros: would ensure that all array libraries more or less return the same values across all ranges (and potentially orderings, e.g., during summation), which would help promote numerical reproducibility as users and devs move between libraries.
   • cons: would limit the types of implementations that array libraries can include and could mean that array libraries implement to the reference implementation (akin to "teaching to the test"), meaning that, if the reference implementation includes odd behavior, then other array libraries would also be effectively forced to replicate that same odd behavior. For example, in a more benign case, suppose NumPy is the reference implementation against which all other array libraries should test when implementing sum. NumPy uses pairwise summation which, while more stable, in general, than ordinary recursive (naive) summation, fails to accurately compute the sum when dealing with values of varying magnitude and sign (e.g., (1.0, 1.0e100, 1.0, -1.0e100) => 0.0, rather than 2.0). If we require a universal relative tolerance, even derived from empirical test suites, we risk preventing array libraries from choosing to use, e.g., compensated summation techniques (e.g., KBN) which can accurately handle the aforementioned case.

2. We could require that array libraries accurately compute particular types/subsets of values. For example, for sum, the specification test suite could test certain canonical (non-controversial) sequences, such as sum([1.0, 2.0, 3.0, 4.0]), which should return exactly 10.0 no matter the summation technique. In which case, we could simply specify that all specification-compliant array libraries must pass the specification test suite, where, for these particular algorithms, we test "toy"/simple inputs in order to sanity check array library implementations, but do not attempt to require a particular tolerance (level of accuracy) across all inputs.

[kgryte]

In order to move this PR forward, I've added statements of non-guidance for statistical and linear algebra functions and added notes regarding test suite features for reporting mathematical function accuracy.

[rgommers]

Okay, these explanation of why certain sets of functions cannot have a max accurary spec'ed and an approach of providing comparisons via the test suite is probably the best we can do for now. Merging.