Make use of mapslices consistent throughout Julia

[johnmyleswhite]

I'd like to propose the radical, breaking change of removing all forms of implicit mapslices calls from Julia. I think they make the language much less consistent and create a situation in which one's expectations are routinely violated about the interface for functions. As an example, the list below shows some functions that effectively implement implicit calls to mapslices:

• sum
• prod
• mean
• var
• std

In contrast, the following similar functions do not support the foo(A, dim) interface at all:

• median
• quantile
• indmax / indmin
• findmax / findmin
• Everything defined in the Stats module

I understand that this change would break a great deal of Matlab compatibility and make the language a little more verbose. But I think the gains in consistency would more than make up for that loss by making the language much less confusing. Removing this shorthand would mean that you wouldn't have to memorize which functions belong to a privileged subset that perform implicit mapslices.

As one example of the memorization required to use the foo(A, dim) interface, you need to memorize that empty tuples passed to min are required to trick min into creating slices. I'd much rather know that mapsplices works the same way for all functions in the language.

[stevengj]

You don't have to memorize the subset now; nothing stops you from applying mapslices to sum if you want.

We can promote the mapslices interface as the Julian way to do things, while still retaining the old behavior in a few functions to ease the Matlab transition (as with sum) or for performance (as with fft).

[johnmyleswhite]

Yes, that's certainly true. But we've previously made many decisions to ensure that there aren't two dialects of Julia that provide two mechanisms for doing the same thing.

I suspect I'll be outvoted on this one very quickly, but wanted to raise the issue for discussion.

[BobPortmann]

I think using a DIM keyword consistently would be cleanest. E.G., min(a,dim=2).

[lindahua]

@johnmyleswhite I appreciate the pursuit of consistency over function definitions. This is of course a very important aspect in language design.

However, I think what we should do here is to add the support of computation along specified dimension to other functions over time, instead of removing this capability from sum, max, etc.

Something like sum(x, 1) or sum(x, 2) has been extensively used in various packages. It would be a substantial cost to make changes to all these codes. Also, mapslices(sum, x, dim) are less performant than sum(x, dim), especially when dim > 1, since the memory access pattern of mapslices is not cache-friendly when dim > 1. (However, I think we can add support of dimensions to reduce and mapreduce, which can be implemented in a cache-friendly way though).

That being said, I think we may consider how to improve the consistency of the interface over these functions. (e.g. using dim keyword, etc).

[johnmyleswhite]

I would be very happy with either the consistent use of a dim/slice keyword or any mechanism that ensures that most functions in the language support the foo(A, dim) interface. As @lindahua suggests, we can add this ability incrementally over time, but it's a non-trivial amount of work because every new function needs to implement this functionality. Because functions like min don't quite match the standard interface, we will end up creating a new interface for many functions. And because findmin doesn't return a scalar, we'll have to decide how to allocate the resulting tuples in an array.

As Dahua points out, the major challenge (besides time) is making sure that the resulting interface is still high performance.

[blakejohnson]

I agree with @lindahua that I'd rather add this interface to the stats functions that don't have it rather than remove it from the ones that do. I use these in almost every data analysis script I write, and I find the compact syntax very convenient.

We do need a more friendly syntax for max and min. After N threads of discussion on that point, I think the conclusion was to use a keyword dims argument. Just, no one has gotten around to doing it.

[johnmyleswhite]

Should we have perhaps have a @mapslices macro that constructs the keyword argument version of these functions from the base case?

[dcjones]

I'm very much in favor of mapslices, or rather, very much against reimplementing the same functionality in any function that could conceivably be applied across a matrix.

The performance difference just doesn't strike me as great enough to embrace such an inconsistency. But I also get that Julia has a strong Matlab background, and not having dim arguments in certain functions is jarring to many.

[timholy]

While it would be a fairly radical change, despite my Matlab background in principle I think this may be the right thing to do.

But not now. The performance of mapslices is nowhere close to where it needs to be to become the backbone for such common operations; not just because of the cache-unfriendliness issue @lindahua mentioned (although that indeed is very important), but also because mapslices still has quite a lot of overhead. Fixing this is yet another thing that relies on breaking the SubArray/cartesian iteration logjam (and I don't think we're ready to incorporate Cartesian.jl into base anytime soon).

[stevengj]

Although Julia normally tries to have one way to do things, the cost/benefit ratio of keeping the Matlab-like syntax available (even if we promote mapslices as the "right" way to do it) seems to me to favor keeping the redundant Matlab-like syntax (costs: some redundant syntax; benefits: compatibility and ease of transition for Matlab users, for a few key functions that are used all over the place).

This is independent of whether we can eliminate the performance penalty of mapslices. Note also that for functions like sum we will always be able to do special-case optimizations that will be hard to obtain in a generic mapslices routine.

[JeffBezanson]

Philosophically I totally agree with john, but on the practical side tim's and steve's comments nail it.

[JeffBezanson]

I think we should only keep the dim argument for sum and prod, and maybe any and all, since these happen to have special names (unlike max, min, and other functions in general).

In addition to mapslices, we have reducedim, which is much more efficient when we know the mapped function is doing the equivalent of reduce. reducedim is a silly name since it can actually work on multiple dimensions. It would be nice to name these similarly; maybe mapdims and reducedims, or mapslices and reduceslices (that last one seems a bit long-winded).

[johnmyleswhite]

When you say special names, you mean that sum(1, 2) is a special name for 1 + 2?

I actually like mapslices and reduceslices, but mostly because I'm used to mapslices. I'd be up for mapdims and reducedims as well.

[JeffBezanson]

I mean that sum is a name for x->reduce(+,x). In fact some of the reductions are defined this way:

all(A::AbstractArray{Bool}, region) = reducedim(&,A,region,true)

We can make reduceslices and mapslices the basic thing, but have such definitions for convenience where possible.