Possible memory-leak in multi-threading inference using Tensorflow and having org.bytedeco.javacpp.nopointergc=true (CPU)

[skirdey]

Description

Possible memory-leak in multi-threading inference using Tensorflow and having org.bytedeco.javacpp.nopointergc=true

CPU inference.

Expected Behavior

Garbage collection removing objects from Old Generation

Error Message

Java OOM

How to Reproduce?

Run multi-threaded inference for 30minutes with -Dorg.bytedeco.javacpp.nopointergc=true so you don't have JavaCPP Deallocator blocking thread.

/gradlew benchmark -Dorg.bytedeco.javacpp.nopointergc=true -Dai.djl.default_engine=TensorFlow -Dai.djl.repository.zoo.location="https://storage.googleapis.com/tfhub-modules/tensorflow/resnet_50/classification/1.tar.gz?artifact_id=tf_resnet" --args='-n tf_resnet -t 10 -c 1000000 -s 1,224,224,3'

[skirdey]

I have a high level overview of what happens captured via yourkit,
at some point (>30 min of continous predictions), getting a spike in GC collections

and pause time:

Also, OldGen takes most of the space leaving nothing for new allocations

here is memory snapshot, after several forced GCs

[skirdey]

Same benchmark doing multi-threading inference using PyTorch has minimal memory footprint and no GC pauses:
./gradlew benchmark -Dai.djl.default_engine=PyTorch -Dai.djl.repository.zoo.location="https://alpha-djl-demos.s3.amazonaws.com/model/djl-blockrunner/pytorch_resnet18.zip?model_name=traced_resnet18" --args='-c 100000 -d 1440 -t 10 -s 1,3,224,224'

[karllessard]

Please see tensorflow/java#229 for a potential fix for this problem in the TF Java library.

[skirdey]

@karllessard is it possible to release tf java 0.2.1 with the hotfix? it doesn't look like a breaking change so a patch would be nice.

DJl uses release versions of TF Java

[saudet]

@skirdey 0.3.0-SNAPSHOT is pretty much up now, so please give that a try, but like I tried to explain on the pull request, I don't believe this will fix any issues for DJL.

[karllessard]

Good news @skirdey ! Important build issues that were preventing us to release 0.3.0 are now fixed, I will provide more details a bit later but you can also follow this thread to see the progress.

[skirdey]

The fix could be useful when I do not set nopointergc=True, as it was also leaking memory just took 70 hours to catch it.

…

On Friday, 12 March 2021, Karl Lessard ***@***.***> wrote: Good news @skirdey <https://github.com/skirdey> ! Important build issues that were preventing us to release 0.3.0 are now fixed <tensorflow/java#240>, I will provide more details a bit later but you can also follow this thread <tensorflow/java#230> to see the progress. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#690 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHIJQDEYHKNQHHHV6OPGE3TDIGBJANCNFSM4YFNJGOA> .

[saudet]

That doesn't fix any leak. If there's a leak, it's somewhere else.

[stu1130]

DJL team can publish 0.11.0-SNAPSHOT that depends on 0.3.0-SNAPSHOT. I am on it.