Framework for symbolic optimizations

[ViralBShah]

We need a framework to express certain mathematical optimizations in julia itself. These may be expressed as rules that are run after types have been inferred. Examples are:

1. A' * B, A' \ B: Can be computed without computing the transpose by calling BLAS/LAPACK routines
2. A - B + C .* D: Can be computed without temporaries
3. A[m:n, p:q] + B: Avoid computing the ref.
4. A[m:n, p:q] * B: Avoid computing the sref, and call DGEMM directly in cases where this is possible.
5. [ a b c; d f; e]: Compute the entire concatenation in one shot.
6. A[p,q] = B[r,s]: Temporary can be avoided by calling assign(A, sub(B, r, s), p, q)

In all cases, temporaries can be avoided. Either we can do it with optimizations, or have the parser such expressions down to the runtime. If a matching runtime implementation for such cases is not found, the existing base case should certainly be used.

[JeffBezanson]

#5 above is easy since it doesn't generalize (it only deals with 2 dimensions). How should it be called? Maybe

hvcat({a, b, c}, {d, f}, {e})

?

Of course a fallback definition can be provided so implementing it is only an optimization.

Same idea applies to a'*b, a*b', etc. We could have aTb() and abT() methods.

For #2, we already parse runs of + and * to single calls, so +(as::Array...) can be implemented.
Is it worth having special methods for s*x+y etc.?

[ViralBShah]

Yes, #5 is easier - and the suggested syntax seems ok.

I guess aTb() and abT() are good starts for a first release, until a more general framework can be implemented. Ideally, BLAS allows you to efficiently do A[I,J]^T*B[I,J]^T + C[I,J]^T where I and J are ranges. I guess the indexing part can be handled through subarrays, but then the user has to use subarrays explicitly.

For #2, I was only thinking of element-wise operations for a start.

Think of a stencil operation. It will be essentially the element-wise stuff, but with some indexing thrown in. It would be nice to handle these cases with indexing. Perhaps we need to use subarrays more automatically in more places.

[JeffBezanson]

Well I'm saying we can already handle element-wise a+b+c+d+....

What special methods do we want? I guess I could also make special versions of the operators where you know that the first and/or second argument can be overwritten.

[ViralBShah]

I was referring to element-wise a-b._c+4_d?

[JeffBezanson]

Seems to me passing this to something at run time amounts to handling arithmetic operators with an interpreter?

[ViralBShah]

Would be cool if there were a way to do this. I guess that for Dense Arrays, one can just hack it into the compiler as a special case.

[StefanKarpinski]

The A*B' and A'*B cases could be handled by allowing cheap transpose by storing strides for each dimension. That way A' would be a trivial operation that uses the same data as A and just puts a new matrix object around it with different strides. The the multiply call could just pass dgemm the correct stride values and it would work.

Unfortunately, this presumes that you can do T = A' without any copying, which in the current implementation you can't. This brings us back to immutability vs. mutability issues...

[ViralBShah]

I have often thought of this but am not sure the resulting complexity is worthwhile. Also it annoys the user who know what they are doing.

[StefanKarpinski]

It really only annoys the user who knows what they're doing in Matlab and doesn't know what they're doing in Julia. There could be ways to force the actual transpose to be done. With the mutability vs. immutability stuff, this approach wouldn't just apply to just transpose, but also subarray refs — since those could safely be references into the same data as the original array with different strides — and many other cases too.

[ViralBShah]

Here's a suggestion for this - How about extend * to allow:

For A*B:
*(A,B)

For A'*B'+C', where either of the transposes may or may not happen.
*(A, transA, B, transB, C, transC)

Also want one more form for x' *A' *y'. Perhaps * can take a 7th argument that says what type of operation it is - a'b'+c' or x'A'y'.

[StefanKarpinski]

Ahh! These all go away if we deal with mutability and allow cheap transposes.

[ViralBShah]

I don't like the idea of the cheap transpose. Cheap transposes are not always desirable, as they just move the problem elsewhere. Most fast matlab code is written with an assumption on stride and storage, and the cheap transpose just messes up that performance model. Also, every single operation has to be made stride-aware, and also will become a bit slower due to dealing with strides everywhere.

I'd be ok with a cheap transpose that is evaluated lazily, but not one that just changes the stride.

[JeffBezanson]

We can have functions for A'_B'+C', x'_A'_y' etc. but I'd rather they not be part of _ since they don't just multiply. We need other names for them.

[ViralBShah]

muladd() and bilinear()?