Very heavy lock contention under load

[ivanr-rs]

We're experiencing very heavy lock contention with version 5.14. It's in this section of the code https://github.com/java-native-access/jna/blob/5.14.0/src/com/sun/jna/internal/Cleaner.java#L58. In one of the thread dumps we see 647 threads waiting on the lock. We were initially running 5.10, upgrading to 5.14 didn't help.

We've tried with the latest Zulu JDK 22 and JDK 21, same result. Linux amd64. This problem now always triggers in our app immediately after a fresh start, possibly because the JVM is slower. If reduce the amount of threads initially then it doesn't trigger immediately, but it still happens after a while. Sometimes stopping and restarting the threads helps.

The server is not overloaded. We increased the number of cores two-fold and we're still experiencing the problem.

Below is the relevant part of the stacktrace.

Not sure how you can reproduce, in our case we always see this problem from cold start with about 1000 threads.

Any thoughts?

"pool-4-thread-447" #4038 [3373050] prio=5 os_prio=0 cpu=2546.58ms elapsed=612.63s tid=0x00007f5e28011e90 nid=3373050 runnabl
   java.lang.Thread.State: RUNNABLE
        at com.sun.jna.internal.Cleaner.register(Cleaner.java:58)
        - locked <0x00007f6069fdc500> (a com.sun.jna.internal.Cleaner)
        at com.sun.jna.Memory.<init>(Memory.java:122)
        at com.sun.jna.ptr.ByReference.<init>(ByReference.java:57)
        at com.sun.jna.ptr.IntByReference.<init>(IntByReference.java:35)
        at com.sun.jna.ptr.IntByReference.<init>(IntByReference.java:31)
        at java.lang.invoke.LambdaForm$DMH/0x00007f5fbbf19800.newInvokeSpecial(java.base@22.0.1/LambdaForm$DMH)
        at java.lang.invoke.Invokers$Holder.invokeExact_MT(java.base@22.0.1/Invokers$Holder)
        at jdk.internal.reflect.DirectConstructorHandleAccessor.invokeImpl(java.base@22.0.1/DirectConstructorHandleAccessor.j
        at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(java.base@22.0.1/DirectConstructorHandleAccessor.
        at java.lang.reflect.Constructor.newInstanceWithCaller(java.base@22.0.1/Constructor.java:502)
        at java.lang.reflect.Constructor.newInstance(java.base@22.0.1/Constructor.java:486)
        at com.sun.jna.Klass.newInstance(Klass.java:48)
        at com.sun.jna.NativeMappedConverter.defaultValue(NativeMappedConverter.java:65)
        at com.sun.jna.Structure.initializeField(Structure.java:1440)
        at com.sun.jna.Structure.initializeFields(Structure.java:1416)
        at com.sun.jna.Structure.<init>(Structure.java:218)
        at com.sun.jna.Structure.<init>(Structure.java:204)
        at com.sun.jna.Structure.<init>(Structure.java:191)
        at com.sun.jna.Structure.<init>(Structure.java:183)