Fastest way of converting between OpenCV Mat and Rust ndarray (or other Rust arrays)? Or even better, zero-copy?

[fzyzcjy]

Hi thanks for the lib! I wonder what is the fastest way of converting between OpenCV Mat and Rust ndarray (or other Rust arrays)? Obviously I can iterate each element and assign the value, but it will be terribly slow. I guess there should be something like memcpy? Or even better, can we have zero-copy when converting between Mat and ndarray?

[twistedfall]

Zero-copy would not be safe because the allocation and reallocation of the memory blocks would have to be done by different libraries. But do check out this crate: https://crates.io/crates/cv-convert The author is handling this conversion between popular cv libraries.

[fzyzcjy]

Sounds great. Thanks!

[fzyzcjy]

@twistedfall Hi I think maybe it can be safe sometimes? Pseudocode:

// or other linear algebra lib, say ndarray, not necessarily Vec.
let input_rust = Vec::new(10000000); 
// I mean this -> https://stackoverflow.com/a/23249319/4619958 . then, cv::Mat do not allocate memory by itself. Instead it uses the memory provided by us, which is indeed the memory managed by input_rust.
// Maybe we can make it **mutable borrow**?
let input_cv = cv::Mat::from_the_memory(input_rust); 

// same as above. just pre-allocate the memory that *will* be used by opencv
let output_rust = Vec::new(2000000);
let output_cv = cv::Mat::from_the_memory(output_rust);

cv::resize(input_cv, output_cv, ...);

drop(input_cv);
drop(output_cv);

then use input_rust, output_rust happily!

doc: https://docs.rs/opencv/0.54.0/opencv/imgproc/fn.resize.html

Could you please provide some suggestions? Thanks

[twistedfall]

In version 0.55 you can now call Mat::data_bytes() which will give you access to the underlying buffer as &[u8]. The Mat must be continuous though. This will solve that issue partially. The zero-copy part is still under investigation.

[twistedfall]

I've run some tests and it indeed works without copying the data, you just need to use the Mat::new_*_with_data family of constructors. Be aware that those are all unsafe for obvious reasons, e.g.:

let mat = unsafe {
	Mat::new_rows_cols_with_data(
		1,
		bytes.len() as i32,
		u8::typ(),
		bytes.as_mut_ptr() as *mut c_void,
		core::Mat_AUTO_STEP,
	)?
};

Also if you modify the resulting Mat in some way that would require reallocation then it will copy the data internally into the newly allocated memory area.

[fzyzcjy]

Thanks! Shall we have a wrapper for it? For example, &mut [u8] into Mat, and Mat's lifetime should be shorter than that borrowed u8 slice.

[twistedfall]

For now there is no way to specify the Mat's lifetime, it's planned, but it requires a considerable rework. So in the current circumstances I don't think it make much sense to have such a wrapper.

[fzyzcjy]

Ok... Thank you all the same!

[fzyzcjy]

I agree. Mat is reference counting, so it can make tons of references everywhere. Therefore, even if the Mat we construct from the &[u8] correctly drops within lifetime, there can be lots of references in other places. Then, when the &[u8]'s owner really drops the Vec, all those references become dangling.

[twistedfall]

True that

[soloist-v]

let (mut data,_, _) = pixels.into_raw_parts();  // pixels will be move into_raw_parts，and return a manually drop pointer.
let img = opencv::core::Mat::new_rows_cols_with_data(
    height as i32,
    width as i32,
    opencv::core::CV_8UC4,
    data as *mut c_void,
    opencv::core::Mat_AUTO_STEP).unwrap();
img.addref().unwrap(); // ???