Communicate information about filtered data points

[yurivish]

It would be useful if exploratory plots came with a visual indicator of “discarded data”.

This would improve Plot's capacity for exploratory data analysis by enabling users to become aware of anomalous values that violate their assumptions about the data.

For example, I changed a scale from log to symlog and discovered a bunch of negative values where I wasn’t expecting any.

The data was supposed to be strictly positive and the negative values indicated a processing error, but since the default log scale filtered those data points out I only noticed because I went out of my way to do additional spot checks.

Plot could have made it evident immediately, e.g. with a legend saying something like “100 datapoints not shown”. Even more useful (maybe) would be being able to see a "data pipeline" and how many points are filtered out at each stage.

@Fil observes that some filters use the discarding as a basic mechanism to do their work as intended, so there are subtle questions about what to communicate for this to be a useful signal.

For the exploratory use case I think it makes sense for this to be on by default, since spot-checking every individual assumption manually can get onerous (e.g. checking for null/undefined, zeros where there shouldn’t be any, negative numbers where there shouldn’t be any, values outside of the x/y/color domain, NaN, etc.)

A separate tool such as a summary table could be used to learn about missing/pathological data in a dataset, but it would still be useful for Plot to flag these issues since they can creep in during downstream processing and plot transformations.

[Fil]

The scale.unknown option can be used to this effect — examples.

[Fil]

This would now happen, I guess, in the default filter

plot/src/plot.js

Line 291 in 9d9ba91

index = index.filter(i => filter(value[i]));

. However with each warning we need to indicate a way to fix the situation, and in this case I wouldn't know what to say, in particular because in many charts some data is ignored on purpose.

[eagereyes]

As an additional twist on this, it would be great if we could provide informative error messages for two seemingly common cases:

• somebody gets the capitalization of a key wrong, e.g., city instead of City
• somebody misspells the name of a key, e.g., delivert instead of delivery

Likely candidates for capitalization errors could be found by comparing the key provided to all the keys in the input object in a way that ignores case (i.e., converting both to lowercase before comparing).

Misspellings are more complex than that, possibly using Levenshtein distance and a threshold (or finding the closest match and suggesting that).

The latter is an expensive operation, but it would only have to be run when there's an error (or a presumed error), and it would mostly delay the error message, not interfere with normal Plot operation.