make Base.setindex() public

[aplavin]

setindex() is a widely used well-documented function, it should be marked public, right?

Not sure what's the right way to make a function with multiple methods public, please correct me if I'm wrong here!

[giordano]

I'm missing the point of this PR, as far as I can tell this is already public:

% julia +nightly -q
julia> Base.ispublic(Base, :setindex!)
true

[aplavin]

Note the difference:

julia> Base.ispublic(Base, :setindex!)
true

julia> Base.ispublic(Base, :setindex)
false

[aplavin]

Not sure what's the status here. Should I do some update of this PR or ...?

[nsajko]

If setindex is to be made public, it should be documented, i.e., its doc string should be included into the docs. Thus the "needs docs" label.

[aplavin]

I'm not familiar with documentation generation, could use some pointers on where to update things :)
Of course, assuming that this change is approved.

[nsajko]

If the function is to be made public, we'd need to decide if it's just for tuples, i.e, will adding methods for other collection types be allowed. Perhaps we could instead define something like setindex_tuple(t::Tuple, v, i) = setindex(t, v, i) and then make only setindex_tuple public.

Adding new methods would negatively affect abstract return type inference in the extreme case where the type of the input collection is unknown (inferred as Any), quite possibly requiring the ecosystem to add some type asserts for performance.

[ararslan]

(Removing my review request as @nsajko has got this handled 😉)

[aplavin]

Not sure what you exactly the issue with extending setindex and what exactly you suggest, @nsajko. Currently, setindex is defined (& documented) in Base for tuples and namedtuples. It's also used by lots of packages, with many more methods defined for setindex – eg, StaticArrays do that.

As the semantics of the function is pretty clear, and it's already extensively used in the wild (including new method definitions), what specific issues do you see with marking it public?

---

Perhaps we could instead define something like setindex_tuple(t::Tuple, v, i) = setindex(t, v, i) and then make only setindex_tuple public.

Tbh, I don't see a reason to do that. Defining separate functions for every type, instead of methods of the same functions, goes directly against Julia multiple dispatch design and practice.

[nsajko]

I'm referring to the "world splitting" optimization, see the Base.Experimental.@max_methods doc string. I agree that Base.setindex should probably be available for all collection types. But, it's good to consider the possible downsides upfront, seeing as, in a fresh REPL session, Base.setindex has just three methods - below the max_methods default.

EDIT: I only mentioned this issue for completeness. IMO it's secondary to proper documentation.

[nsajko]

May I ask, Is there a logic to this specific location, between hasproperty and getfield? Mostly just curious, I don't really have a better suggestion.

[nsajko]

Actually, perhaps the doc string would fit better into the Collections and Data Structures page? Under Indexable Collections I guess.

[aplavin]

Properties/fields and indices are kinda close conceptually, and they are exactly the same for tuples and namedtuples (the types setindex is defined for). So it seems to make sense to put it here.
But any location works for me.

[nsajko]

No, it can't go under "Indexable collections" as-is, because of the "fully implemented by" list in that section. The list includes types which don't implement setindex...

[nsajko]

FWIW, given that setindex is indeed already used in many packages for some reason, and its interface seems OK, I now agree that it should probably be made public.

[aplavin]

I'm referring to the "world splitting" optimization, <...>

Still not sure how these considerations are relevant to marking setindex as public.
I don't propose adding any more methods in Base here, although, I would like that as a completely separate independent PR (#43155, #54508, #43338, #33495). Marking it as public should just reflect the practical status quo: this function is widely used & overloaded.

Btw, a lot of high-performance code uses StaticArrays, and with them loaded one has at least 6 setindex methods.

[aplavin]

bump...

[aplavin]

bump :)

[aplavin]

bump... anything else I should do here?

[aplavin]

and another bump

[aplavin]

bump, anyone?..

[aplavin]

should be a tiny change, any issues with it?..

[CameronBieganek]

As the semantics of the function is pretty clear

Actually the semantics are not at all clear. The behavior of Base.setindex is very different for tuples and named tuples. For tuples, an out-of-bounds index throws a BoundsError, whereas for named tuples an out-of-bounds index extends the named tuple.

julia> Base.setindex((1, ), 10, 2)
ERROR: BoundsError: attempt to access Tuple{Int64} at index [2]
Stacktrace:
 [1] setindex(x::Tuple{Int64}, v::Int64, i::Int64)
   @ Base ./tuple.jl:57
 [2] top-level scope
   @ REPL[34]:1

julia> Base.setindex((a=1, ), 10, :b)
(a = 1, b = 10)

Ideally a generic function should have one docstring (per arity). In other words, there should only be one meaning for a generic function. Consider the Base.setindex! docstring, which makes no mention of the input types:

"""
    setindex!(collection, value, key...)

Store the given value at the given key or index within a collection. The syntax `a[i,j,...] =
x` is converted by the compiler to `(setindex!(a, x, i, j, ...); x)`.

...
"""

(I know there is a second docstring for setindex!, but let's ignore that for the purposes of this discussion.) Note how the docstring provides a single generic definition that can work for many different collection types.

I don't think Base.setindex in its current form should be made public. I see two possible paths forward:

1. Change the behavior of Base.setindex(nt::NamedTuple, v, i) so that it errors on an out-of-bounds index. Then the docstring for Base.setindex can be the following:

   """
       setindex(x, v, i)
   
   Create a shallow copy of collection `x` with the value at index `i` set
   to `v`. Throw a `BoundsError` when `i` is out-of-bounds.
   """
   function setindex end
   

2. Make two public functions (names can be bikeshed), something like the following:

   """
       setindex(x, v, i)
   
   Create a shallow copy of collection `x` with the value at index `i` set
   to `v`. Throw a `BoundsError` when `i` is out-of-bounds.
   """
   function setindex end
   
   """
       setkey(x, v, k)
   
   Create a shallow copy of collection `x` with a new key `k` set
   to `v`. If `k` is already in the keys of `x`, `v` replaces
   the old value.
   """
   function setkey end

If we decide to add setindex and setkey to Base, then we might as well export them.

[martinholters]

Comparison to setindex! does not support that argument, actually:

julia> setindex!([1], 10, 2)
ERROR: BoundsError: attempt to access 1-element Vector{Int64} at index [2]
Stacktrace:
 [1] setindex!(A::Vector{Int64}, x::Int64, i1::Int64)
   @ Base ./array.jl:1021
 [2] top-level scope
   @ REPL[6]:1

julia> setindex!(Dict(:a => 1), 10, :b)
Dict{Symbol, Int64} with 2 entries:
  :a => 1
  :b => 10

So the consequence would be to write a docstring for setindex that does not talk about index bounds at all for consistency's sake?

[CameronBieganek]

My comparison to the setindex! docstring was only meant to demonstrate what a generic docstring looks like, but I guess it was not the best example. The iterate docstring is a better example:

    iterate(iter [, state]) -> Union{Nothing, Tuple{Any, Any}}

Advance the iterator to obtain the next element. If no elements remain, `nothing` should
be returned. Otherwise, a 2-tuple of the next element and the new iteration state
should be returned.

Note that the docstring provides a single definition for iterate regardless of the input types.

But you bring up an interesting point. In my previous comment I explicitly ignored the second docstring for setindex!. So, setindex! actually has at least three distinct behaviors, depending on the input type:

Throw a bounds error for an array:

julia> x = [1, 2];

julia> x[3] = 10
ERROR: BoundsError: attempt to access 2-element Vector{Int64} at index [3]

Extend a dictionary:

julia> d = Dict(:a => 1);

julia> d[:b] = 2
2

Set multiple indices at once for an array:

julia> a = [1, 2, 3, 4];

julia> a[[2, 3]] = [10, 20];

julia> a
4-element Vector{Int64}:
  1
 10
 20
  4

So, setindex! is a very non-generic function. I guess that's ok, since setindex! is a bit special due to the square-bracket syntax sugar and it's importance for array indexing. Not every function in Julia has to be a generic function. There are some notable exceptions to the goal of having generic functions, like constructors and convert, where the exact behavior depends on the input type. But I think we should generally prefer to make Base Julia functions generic whenever possible. Base.setindex does not have syntax sugar and it is not ubiquitously used like setindex!, so I think it makes sense to make it generic (and we can add setkey if desired).

So the consequence would be to write a docstring for setindex that does not talk about index bounds at all for consistency's sake?

No, that's not the consequence. Please take a look at the example docstrings that I provided in my previous post.

[aplavin]

Sorry, forgot about this for a while!

triage approves.
@aplavin add a news entry and we can merge.

Added.

[CameronBieganek]

It looks like my concern about the generic definition of setindex still has not been addressed. What happens when an out-of-bounds key is provided? Does it throw an error or does it extend the collection? I think the only plausible solution is to make two functions, setindex and setkey (name can be bikeshed), as I described above.

I think setindex should be defined for tuples but not for named tuples, and setkey should be defined for named tuples but not tuples. In other words, this really needs to be a method error: setkey((1, ), 2, 1_000_000).

[aplavin]

I think the only reasonable way to define setindex semantics is to follow setindex! as closely as possible. Anything else would be very confusing.
AFAICT, this is the case with the current setindex implementation: throws an OOB error for tuples like setindex! for arrays, adds a new element for named tuples like setindex! for dicts.

This function is well-defined and documented, naturally leading to a widespread use in many packages.
I can imagine

I think setindex should be defined for tuples but not for named tuples

would cause quite some breakage if implemented. That's one of the reasons to mark it public – because it effectively is already.

I haven't seen new triage comments here, so assuming they still agree with the PR.

[CameronBieganek]

I haven't seen new triage comments here, so assuming they still agree with the PR.

My concerns were raised after that triage call, so triage may not have considered the issues that I raised.

I think the only reasonable way to define setindex semantics is to follow setindex! as closely as possible. Anything else would be very confusing.

I agree there is some logic to that, but setindex! is very special because it has syntax sugar that is ubiquitously used. Because square bracket indexing is used so ubiquitously, it's not surprising that the various uses do not behave in a consistent, generic way. However, Base.setindex is not special and it is not nearly as ubiquitous. It is overloaded in StaticArrays and Accessors. Where else?

setindex does not even need to be defined in Base. In Accessors.jl you can define your own setindex function with any behavior you like (though I would still prefer a generic definition).

That's one of the reasons to mark it public – because it effectively is already.

We shouldn't be encouraging the usage of Base internals. This PR sets a bad precedent. Once someone starts using a Base internal they can open an issue requesting that the internal is marked public because "it's used widely in the ecosystem."

---

Consider the following two code chunks. The first one uses setindex and the second one uses setkey, as I defined them above. Note that there is no mention of the types of the values that are used in either example. Yet the examples can be understood to be correct due to the generic definitions of setindex and setkey.

setindex example:

b = try
    setindex(a, v, i)
catch e
    e isa BoundsError ? a : rethrow()
end

setkey example:

The following will never throw a BoundsError:

b = setkey(a, v, k)

Of course, it could throw other errors due to type mismatches of the inputs. For ultimate clarity, the following should probably be added to the docstring for setkey (and similarly for setindex):

"""
    setkey(x, v, k)

...

The following must be true, or else an error will be thrown:

- `typeof(k) <: keytype(x)`
- `typeof(v) <: valtype(x)`
"""

---

I think setindex should be defined for tuples but not for named tuples

would cause quite some breakage if implemented.

Ultimately, Base.setindex is not public, so Base has no backwards compatibility obligations for Base.setindex. Though I agree that it would make sense for both setkey and setindex (as I have defined them) to work on named tuples. The important thing is that setkey(("a", ), "b", 1_000_000) should be a method error.

Actually, named tuples are a weird beast... Arguably the second Base.setindex call below should work. (In my formulation, the first call would be a setkey call and the second call would be a setindex call, and they would both work.)

julia> nt = (a = 1, b = 2);

julia> Base.setindex(nt, 100, :c)
(a = 1, b = 2, c = 100)

julia> Base.setindex(nt, 100, 2)
ERROR: MethodError: no method matching setindex(::@NamedTuple{a::Int64, b::Int64}, ::Int64, ::Int64)

[aplavin]

Note that setindex is not some obscure function, it's used and sometimes overloaded in many packages.
I don't really have more arguments for making it public than

• it's a widely used well-documented function with clear and accepted semantics
• if setindex exists, it should be as consistent with setindex! as possible to avoid confusion; and that's exactly the current behavior.

It's clear that you disagree. Neither of us can merge this PR anyways, so now it's up to Julia developers.

[aplavin]

Semiannual bump

[ararslan]

Sorry this has languished for so long, @aplavin. Looks like this will need to be rebased since there's a merge conflict. @oscardssmith, since you commented on behalf of triage when the group discussed this PR last year, can you speak to whether any of the concerns raised in this thread (after triage occurred, so would have been raised independently) were discussed at the time, or if there were any takeaways beyond "merge it"?

[oscardssmith]

To be fully honest, I don't really remember the triage discussion here.

[ararslan]

Not surprising given how much time has passed since then. Worth re-triaging, do you think?

[oscardssmith]

yeah. Can you summarize the issues for triage to consider?

[ararslan]

I'm probably not the one to do that as I haven't taken the time to understand them myself, tbh. 😅

[adienes]

as far as I can tell, @CameronBieganek 's observations of potential inconsistencies in the existing semantics of setindex haven't been too thoroughly addressed, and personally speaking I tend to agree that it doesn't seem like a great idea to declare this function public until those are cleared up. Of course triage's word goes, but it sounds like the previous discussion didn't consider this

[mbauman]

I see a handful of concrete concerns, the biggest one being that the current documentation does not speak to any generic behaviors. I think that could be ok — I'm sure we publicize other functions that are inherently not generic — but the persistent analogies to setindex! and getindex throughout this thread scream generic. There is even a third method defined in base that's not covered by the current definition. I think the appeals to "this function is widely used & overloaded" are even more evidence that this needs to be generic to publicize. Otherwise, how are we to know if a subsequent change is appropriate or even breaking?

Even within the narrow existing documentation for just Tuple and NamedTuple, I have questions:

• Is the second index in setindex(("a",), "b", 2) really out of bounds? It sure seems like it could return ("a", "b"), even within its own narrow documentation as it stands.
• NamedTuple's getindex supports an integer index; should setindex? Maybe? Why? Or Why not? What happens when that is out of bounds? Even just by one?
• Tuple's getindex supports vectors and Colon and CartesianIndex; should setindex? Probably not, but why not?
• The Tuple method says that it returns a tuple "similar to t". By analogy to setindex! for other collection types, one might think that a similar tuple would have the same type and do a convert. It doesn't for tuples — and probably shouldn't — but again, why? Do the StaticArray methods do a convert? I'd bet they do.

[CameronBieganek]

Given the wide variety of indexing behaviors in Base and in the wild, I think it's best to view getindex and setindex! as explicitly not generic. They just happen to be the function names that you have to overload if you want to define square-bracket indexing for your type.

(To clarify, the syntax transformation for square bracket indexing is well-defined. It's just that the desugared functions getindex and setindex! do not have a single, generic behavior. The getindex(::Type{T}, vals...) where T method is an egregious example of the lack of a generic meaning.)

Base.setindex is similarly non-generic, but since there is no syntax sugar associated with Base.setindex, I don't see any value in making it public. Packages are free to define their own setindex variants with whatever meaning they choose. Generally, when we add new public functions to Base (which is not very often), they should have a single, clear, generic meaning in order to be useful to the ecosystem.

[aplavin]

I think it's best to view getindex and setindex! as explicitly not generic.

I struggle to see what you mean here.

getindex(collection, key...)
Retrieve the value(s) stored at the given key or index within a collection.

setindex!(collection, value, key...)
Store the given value at the given key or index within a collection.

What is not generic here?

---

My suggestion going forwards is to treat setindex exactly like setindex!, and aim for its semantics to be as close to getindex/setindex! as possible. Including out-of-bounds handling, and different index types support @mbauman.

But note that making it public (this PR) doesn't mean that methods to support all potentially relevant types have to be added immediately. setindex current behavior is very clearly defined and documented, and many hundreds of uses across the package ecosystem prove that its behavior is also useful.
Adding support for more collection + index types may (or may not) happen in the future. It's not directly tied to making setindex public in any way.

[mbauman]

I can see @CameronBieganek's point. For example, setindex! does not discuss converting behaviors and there simply cannot be a generic implementation (beyond AbstractArray's index shenanigans). But I'd still say that there is a generic concept that it speaks to and that we all understand.

I honestly don't know what the invariants should be for setindex(collection, value, key...). Because it returns a new collection, it's way less constrained than setindex! is. The hand-wavy thing that I understand is that this should give you back a new* collection of the same type** that has the same values*** for all of collection's existing keys except the given key whose value is**** value.***** The hard part is the asterisks.

• new* collection: Could parts of it alias? Or is it assumed to be immutable? For example, should a setindex(::Bidiagonal, 0.0, 1, 1) also make a copy of the off-diagonal, even though it might not need to?
• same type**: Tuple and NamedTuple break this. But we want it to be "pretty close" to the same type. How close is that? Must it similarly be immutable if collection is immutable?
• same values***: Should this promote or widen a singular eltype? If it promotes, we might change the existing values. Is that ok?
• whose value is****: Again, promoting (akin to setindex!) or widening?
• *****: Everything else: this definition wouldn't preclude using setindex as a non-mutating push for arrays and tuples. Should it? What about a setindex("hello", 'ℋ', 1)? Should it be allowed to change the subsequent indices if you plop a differently-sized UTF8 in there? Should AbstractArray have a fallback method that unwraps non-scalar setindex to the scalar "canonical indexing" style akin to setindex!? Should AbstractArray have a fallback that uses similar?

My main hesitancy in making this public without further work on its documentation is that it won't actually resolve any issues. It's not even public now and there are a number of issues/PRs trying to figure out what its behavior should be (#43155, #54508, #43338, #33495). Even more concretely, there's #46453 which tries to define (some of) its implementation requirements in an updated docstring. Making this public when we know there are folks currently extending it—and doing so without an understanding of how they should be extending it—seems to only make it harder to merge something like that in the future.

[adienes]

this should give you back a new collection of the same type that has the same values for all of collection's existing keys except the given key whose value is value

also FWIW I'm not even sure that setindex is the right name in the first place to do this action on Tuple or NamedTuple when the type doesn't match or index out of bounds. that seems like it could be a merge or splice (non-mutating analogue of splice!) type of responsibility, e.g. splice(a::T, I, r) where T = (m = copymutable(a); splice!(m, I, r); T(m))

[aplavin]

Given that Julia already has generic functions that return a copy of a collection with some values changed, @mbauman's questions about setindex semantics already have answers.
In the order of your questions:

julia> replace(Bidiagonal([1,2,3],[4,5], :L), 2 => 30.)
3×3 Bidiagonal{Float64, Vector{Float64}}:
 1.0    ⋅    ⋅ 
 4.0  30.0   ⋅ 
  ⋅    5.0  3.0

julia> replace((1,2), 2 => 3.)
(1, 3.0)

julia> replace([1,2], 2 => 3.)
2-element Vector{Float64}:
 1.0
 3.0

julia> replace([1.,2.], 2 => 3)
2-element Vector{Float64}:
 1.0
 3.0

It's not like setindex introduces some unique concepts in Julia!

---

this definition wouldn't preclude using setindex as a non-mutating push for arrays and tuples. Should it?

also FWIW I'm not even sure that setindex is the right name in the first place to do this action on Tuple or NamedTuple when <...> index out of bounds.

I personally never suggested setindex should work as push and create new elements. Agree that it sounds like a task for a different function.

[LilithHafner]

Triage accepts @mbauman's summary and thinks that we need to wait until that's all figured out to make it public. The standard is "beyond a reasonable doubt" that it's the right API long term, otherwise we can just put it in a package (e.g. SetField.jl or Accessors.jl)

[aplavin]

Interesting, I didn't expect that Base.setindex is indeed considered internal and can change/disappear in any Julia version.

This means opening issues/PRs to remove usage/overload of Base.setindex from (popular) packages is a reasonable thing to do? Otherwise we can get a lot of breakage when Base.setindex does change.

[LilithHafner]

This means opening issues/PRs to remove usage/overload of Base.setindex from (popular) packages is a reasonable thing to do? Otherwise we can get a lot of breakage when Base.setindex does change.

No, that is not a reasonable thing to do. From the FAQ, we do not discourage the attempt to create packages that expose stable public interfaces while relying on non-public implementation details of Julia and buffering the differences across different Julia versions.

[aplavin]

Ok, I see! Basically, the main reason for me opening this PR was to get a "guarantee" that such a widely used function is considered public and can be responsibly used in packages. But given that you (not you personally, Julia devs in general) encourage packages using and overloading this function anyway, my original point becomes kinda moot.
I guess everyone can just continue using setindex going forward...

[LilithHafner]

"We" do not encourage use of internals, no.

[aplavin]

Hmm, honestly, I'm a bit lost now. What should package authors do then?

• aim to get rid of setindex because you don't encourage use of internals
• or don't do that, as

This means opening issues/PRs to remove usage/overload of Base.setindex from (popular) packages is a reasonable thing to do?

No, that is not a reasonable thing to do.

---

Let's look at a specific example, StaticArrays. They both use Base.setindex and overload it for their types.
This function is a part of their public API now:

Should something be changed about it or not? (and many more examples in other packages)

[LilithHafner]

If I were maintaining StaticArrays, I would not take any further action until Julia settles on a final API for setindex. If I were considering adding setindex functionality to an existing package, I would first consider using Accessors.jl, and if that didn't meet my needs I would use MyPackage.setfield or some other package provided API.