Julep: Generalize indexing with and by `Associative`s

[andyferris]

I've always loved how in Julia (and MATLAB) one can create a new array from an old one, using what is now the APL indexing rules. Basically if you index a collection of values with a collection of indices, you get a new collection of the indexed values. Beautiful, simple. Indexing has also been extended by allowing arrays that don't use 1-based indexing by e.g. the OffsetArrays.jl package.

I'm not sure if this issue exists elsewhere as its own entity (cleaning up distinctions between arrays and associatives was surely mentioned in #20402 and this Julep seems to be a logical extension of #22907), but here I propose specifically that we extend indexing of and by Associative and make related changes so that the semantics are consistent across these two types of container. I prototyped ideas at https://github.com/andyferris/AssociativeArray.jl and basically came up with the ability to (with simple code):

• Index an Associative{K,V} with an Associative{I,K} to get an Associative{I,V}. E.g. Dict(:a=>1, :b=>2, c:=>3)[Dict("a"=>:a, "c"=>:c)] == Dict("a"=>1, "c"=>3).
• Index an Associative{K,V} with an AbstractArray{K,N} to get an AbstractArray{V,N}. E.g. Dict(:a=>1, :b=>2, c:=>3)[[:c, :a]] == [3,1].
• Index an AbstractArray{T,N} with an Associative{K,I} to get an Associative{K,T} (where I might be Int for linear indexing, or a CartesianIndex{N} for Cartesian indexing). E.g. [11,12,13][Dict(:a=>1, :c=>3)] == Dict(:a=>11, :c=>13).

The semantics are consistent across arrays and dictionaries, and provide that for out = a[b]:

• The output container out shares the indices of b (note: these are CartesianRange for arrays)
• The values out[i] correspond to a[b[i]].

This is fully consistent with both the Base arrays and the OffsetArrays.jl package (We can do something similar for setindex!).

To make everything consistent, it helps to make the following associated changes:

• Make Associatives be containers of values, not of index=>value pairs, so that arrays and dictionaries are consistent on this fundamental point. Use the existing pairs function when necessary (and ideally make it preserve indexability).
• Make similar always return a container with the same indices, even for dictionaries. Ideally, unify similar across Associatives and Arrays (for example a dictionary which is similar to a distributed array might also be distributed) via use of the indices.
• Have an new empty function that makes empty Dicts and Vectors to which elements should be added. (Done, Add empty and change similar(::Associative) #24390).
• Consider whether we want to have collection of things you call getindex and setindex! with be called indices, rather than keys (and rename the current indices(::AbstractArray) to something else)
• Have view work for the various combinations where getindex works.

The demonstration package also prototypes making AbstractArray{T, N} <: Associative{CartesianIndex{N}, T} - I don't think this is strictly necessary but it helped (me) to highlight which parts of the existing interface were inconsistent. The package does demonstrate that we can put something simple together without excessive amounts of code (some performance tuning is surely required).

Finally, a word on what motivates this: lately I've been playing with what fundamental data operations (such as mapping, grouping, joining or filtering) would be useful for both generic data structures and tables/dataframes (that iterate rows), and I found whenever I created say a grouping (using a dictionary of groups), I immediately felt the loss of ability to do complex indexing and other operations with the result (as well have to worry whether the output iterates values or key-value pairs, etc).

[andyferris]

I should have also mentioned that "logical indexing" dict[inds], where the values of inds are Bool and the indices of inds match the indices of dict, should probably also work, just like they do for AbstractArray. That would basically be adding Dict(:a=>1, :b=>2, c:=>3)[Dict(:a=>true, :b=>false, :c=>true)] == Dict(:a=>1, :c=>3).

[TotalVerb]

This looks really cool! I haven't had the chance to look through it thoroughly yet, but it's very appealing to me that arrays can be thought of as associative containers with a shape.

[macd]

Would it be a goal of AssociativeArray to support say , if v = Vec([3,2,1]), something like v[[1,1,1,1]] (which should be [3,3,3,3])? It doesn't look like that works at present, or is there special syntax for that?

[andyferris]

@macd Yes, but does v[Vec([1, 1, 1, 1])] do what you expect? I didn't put any effort into making Assoc interoperate with Base types like Vector - it's just for demonstration but we can add more to the package if there is more design space to explore.

[macd]

Very cool, does work as I expect. I didn't get the interoperability issue, but it makes perfect sense.

[timholy]

As someone who has worked on this kind of stuff, I found it interesting that I was initially surprised by some of your choices, until you pointed out the guiding principle. I completely agree that out = a[b] implies out[i] = a[b[i]] is the most reliable foundation for these design decisions. 👍

[andyferris]

Thanks @timholy. I should point out that this is 100% motivated by a few of your comments along the lines of "arrays are really just some special kind of associative container" (in regards to motivating/justifying offset arrays) and I've been trying to make sense of that since :)

[andyferris]

OK, so far I'm seeing positive feedback here, at least on the indexing behavior. I'd like to know people's opinions on my list of suggested breaking changes that IMO would make all this come together consistently. Enumerated:

1. Have Associative iterate values not index-value pairs. When I first saw Jeff mention this some time ago, I was pretty surprised (my initial reaction was scepticism). However, after thinking about this since then, I pretty much consider this a given now - it will help a lot for generic programming to arbitrary indexables (and we aren't going to make Array iterate pairs, now, are we?).
2. similar preserves indices and empty creates an empty container (with no indices). This is breaking for similar(::Associative), but consistent with similar(::AbstractArray). I can already see that this will enormously help in creating code/patterns which work across Associative and AbstractArray. It's about clear semantics.
3. Rename keys to indices and keytype to indextype. While this might be seen as pure bikeshedding, I think it might be more intuitive to work with getindex, setindex! and indices (or else I would be tempted to propose getindex -> getkey, etc).
4. The crazy part of AssociativeArray.jl: make AbstractArray a subtype of Associative. I still don't know how to feel about this one.

Anyway, some design feedback here would help in creating an implementation.

[mbauman]

Very interesting indeed. Still pondering this. One more major breaking point: Allowing non-scalar indexing on Associatives means you'll no longer be able to have arrays or associatives as keys. That is,

julia> d = Dict(:a=>1, :b=>2, :c=>3, [:c,:a]=>4)
Dict{Any,Int64} with 4 entries:
  :a             => 1
  :b             => 2
  :c             => 3
  Symbol[:c, :a] => 4

julia> d[[:c,:a]]
4

[andyferris]

@mbauman Yes this is an interesting case. Note that for cases where the indices are strongly typed (such as Dict{Vector{Symbol}, Int}) we can catch this correctly at dispatch, so not all is lost. What to do exactly when the index type is not concrete or not known is a bit less clear. (I would tend to try and prioritize scalar indexing so this isn't a breaking change at all, but this might introduce a run-time cost in certain situations. However, if the index type is Any, well, things are probably going to be a bit slow anyway).

Also - it would help if I knew of a use case for such index types. I know we've worried about hashing of vectors and unit ranges and so-on, but I haven't come across a use for this yet...

[mschauer]

In any case we need a nicer way to express that an array should be treated as scalar in an expression where arrays usually are splashed, broadcasted or unboxed in other ways, compare the crutch Scalar from StaticArrays to prevent broadcast. If we figure out that on a high enough level, then also this julep benefits.

[mbauman]

Yes, this is also currently a problem for non-scalar indexed assignments for arrays of arrays. That is, A[:] = 1:2 is always interpreted as A[1] = 1; A[2] = 2 and not fill!(A, 1:2). It can really be a pain for generic programming.

I have a very strong distaste for changing semantics based upon eltype changes.