Questions about compatibility between nalgebra and ndarray

[jturner314]

I'm working on implementing conversions between nalgebra and ndarray types. This looks fairly straightforward, but I have two questions:

1. Does nalgebra support negative strides? From what I can tell, the answer is "no", but will you please confirm?

2. Does nalgebra allow using strides that result in multiple indices pointing to the same element? ndarray allows this for immutable views (ArrayView) but not for mutable views (ArrayViewMut) due to Rust's aliasing rules. I'm somewhat surprised that nalgebra's mutable slices currently allow this:

   extern crate nalgebra as na;
   
   use na::DMatrixSliceMut;
   
   fn main() {
       let slice = &mut [1, 2, 3];
       let m = DMatrixSliceMut::from_slice_with_strides_mut(slice, 2, 3, 0, 1);
       // Prints:
       //   ┌       ┐
       //   │ 1 2 3 │
       //   │ 1 2 3 │
       //   └       ┘
       println!("{}", m);
   }

   This should be fine fine as long as nalgebra doesn't allow getting mutable references simultaneously for two different indices (e.g. with something like split_at). I'm just surprised at the current behavior, so I'd like to make sure it's intentional and not a bug.

[paddyhoran]

Hey, just wanted to leave a note here on this discussion. I have been contributing to ApacheArrow, I added a tensor implementation but this was mostly following the C++ implementation.

The Rust implementation is relatively new but is growing steadily. I was hoping in time to reach out to projects like nalgebra and ndarray to discuss how we can have different projects interoperate with each other, I'm just not there yet. This is the whole aim behind Arrow, a common memory layout for different projects to build on. I believe that a robust Arrow implementation for Rust could be more important than for the other languages because the Rust community is small so fragmentation of the parties interested in scientific computing is a risk. However, if we were all building off a common base, or more likely provided ways to convert to and from a common base it could help the community grow.

Just wanted to get your thoughts on this let you know about Arrow.

Thanks

[jturner314]

@paddyhoran I came across anowell/are-we-learning-yet#14 today, which is related. I'm glad to see so many data science projects adding support for a common in-memory data format like Apache Arrow, and it looks relatively simple to add conversions of 1-D vectors/arrays to Apache Arrow. Looking at the Apache Arrow docs, though, it seems like the intended application is more for libraries like Pandas (data frames) than libraries like NumPy/ndarray/nalgebra (multidimensional arrays/matrices of a single element type). It appears that to represent more than one dimension, you have to use nested List types, which have variable lengths and require a buffer of offsets in addition to the values themselves. Is that true? How would be be the best way to represent something like a 2-D or 3-D array in Apache Arrow?

[sebcrozet]

@jturner314, @paddyhoran: I've created a separate issue for the discussion about Apache Arrow.

[sebcrozet]

@jturner314

1. Yes, nalgebra does not support negative strides.
2. The fact nalgebra allows mutable strides resulting in multiple indices pointing to the same element is indeed a bug. Actually, the fact strides equal to 0 are allowed unintentional and untested so that might be disallowed in the future, unless this has no performance impact.

[paddyhoran]

@jturner314 I replied re-arrow on the other issue.

[jturner314]

Okay, thanks for the clarifications.

Actually, the fact strides equal to 0 are allowed unintentional and untested so that might be disallowed in the future, unless this has no performance impact.

It's worth noting a couple of things:

• Strides equal to 0 can be useful for read-only slices; they allow broadcasting to larger shapes. ndarray currently has limited support for broadcasting, but I'd like to add support for all the broadcasting that NumPy can do.
• Strides equal to 0 should be fine, even for mutable/owned arrays, for axes with length ≤ 1.

[jturner314]

Another question: From what I can tell, slices support arbitrary positive strides, but owned arrays are always represented internally in contiguous, column-major layout. (For example, you can't create a row-major owned array.) Is that correct?

[sebcrozet]

@jturner314 That's correct!

[jturner314]

Another question: Is there any way to convert an owned matrix with Vec-backed storage into a Vec without cloning the elements? [It would be even nicer to have an .into_vec() method that would convert Matrix into a Vec as efficiently as possible regardless of storage type (cloning if necessary).] It looks like it's possible to get the Vec out of MatrixVec storage using the unsafe resize method, but I don't see any way to get the MatrixVec storage out of the Matrix.

[sebcrozet]

There is no way to do this, no. Though I agree adding an .into_vec() method would be quite useful. I've created an issue for that: #479