False sharing issue for std::sync::RwLock

[linericyang]

While benchmarking the Hugging Face tokenizer, which uses std::sync::RwLock for managing read/write access to shared data, we've observed a false sharing issue.

In the current implementation of RwLock, the inner lock is positioned adjacent to the data field. In most cases, these elements are placed within a single cache line (typically 128 bytes in x86 systems).
https://github.com/rust-lang/rust/blob/master/library/std/src/sync/rwlock.rs#L80-L84

pub struct RwLock<T: ?Sized> {
   inner: sys::RwLock,
   poison: poison::Flag,
   data: UnsafeCell<T>,
}

When a reader thread attempts to access data that isn't being altered but shares the same cache line as a lock that is being altered (e.g., when a new reader acquires the read lock), the caching protocol necessitates the first reader thread to reload the entire cache block, despite a lack of logical necessity. This significantly impacts performance, especially when the number of readers exceeds the number of writers.

To address this issue, one practice is to move the data to a new cache line by adding some padding. One quick fixing, which use #[repr(align(128))] to align each field to 128 bytes, has shown more than a 10% increase in throughput in our benchmark. However, it's important to note that this change might have a detrimental impact on throughput when the number of readers is less than the number of writers, since a writer has to write data and lock to two different cache lines.

An instinctive solution suggested by @pingzhaozz is split the inner RwLock into two fields, one is used for write access control which resides in the same cacheline with data field, the other lock for reader will be placed in another cacheline to avoid unnecessary reload data cacheline caused by another reader acquired the read lock.

Any comments would be appreciated.

[Jules-Bertholet]

@rustbot label C-bug I-slow T-libs

[mj10021]

perhaps something like this from the cache_padded crate?

#[cfg_attr(
    any(
        target_arch = "x86_64",
        target_arch = "aarch64",
        target_arch = "powerpc64",
    ),
    repr(align(128))
)]
...
#[cfg_attr(
    any(
        target_arch = "arm",
        target_arch = "mips",
        target_arch = "mips64",
        target_arch = "riscv32",
        target_arch = "riscv64",
        target_arch = "sparc",
        target_arch = "hexagon",
    ),
    repr(align(32))
)]

with an exhaustive list of supported architectures so that the cache line never have multiple locks?

[mj10021]

@rustbot claim

[linericyang]

Thanks for your comments.

Yes, using cache_padded will significantly improve the reader performance, since new reader lock acquiring will not invalidate the cacheline that data locate.

However, the writer's performance might be impacted, since acquiring the write lock and write data will access two cachelines instead of one in the current implementation. So it cannot guarantee performance gain if writer is much more than reader.

I will post the micro benchmark data here later for your reference.

[linericyang]

I measured the performance of padded rwlock with the benchmark.

It shows performance gain as expected when only reader needs access to these locks, since the cacheline where the data is located will not be invalidated when another thread acquired read lock.

However, the write performance is impacted by padded rwlock.

For real scenarios with both reader and writer, it improves the performance in high contention for the rwlocks or reader number is significantly more than writer.

I will evaluate the performance of the solution which use separate locks for reader and writer and put them into different cachelines.