[1.0 Requirement] Dimension coalescing.

[csarofeen]

We need to coalesce dimensions that are contiguous and are not (at some point) used in reductions. We should start with something very simple (only coalesce for simple pointwise cases), and then increase complexity later.

[jjsjann123]

log progress:

The original plan is to handle dimension collapsing in integration. So coalescing will be opaque to codegen.

We can progressively implement it in two steps:

1. support profiling executor in its current form.
2. (maybe) support legacy executor. where shape information only arrives at run-time.

Challenges:
Dimension collapsing could make caching tricky (one more knob on contiguity).

I want to note that in the draft/WIP implementation #215, it handles partial collapsing and permuted dimension collapsing well (NHWC memory format is preserved naturally)

Currently investigated alternative is to support dimension collapsing in codegen.
The advantage of this is that it allows finer control of dimension collapsing on individual tensor, since we only collapse it in index computing. This lifts the restriction on a uniform collapsing scheme on all tensors, which could be restricting.

[jjsjann123]

log progress:

Christian is working on supporting dimension collapsing in codegen. An extra vector marking contiguity will be contained in TensorView, indicating that the corresponding dimension is contiguous with the neighboring one and could be collpased if needed.

Meanwhile, codegen still assumes all dimensions come in descending stride order. Permutation to accommodate the assumption should be implemented in integration. (I can tinker #215 to do that). This still does require all inputs to have matching stride order.

[jjsjann123]

I think this one could be considered to be resolved.

We now have dimension coalescing in integration, where it's handled at GraphCache level by best effort #261

And we added contiguity in TensorView to handle dimension collapsing.

One remaining thing is: #230
I'll report back once I got a chance to verify its status.

[csarofeen]

This is all done.