Scribe throws "java.lang.IllegalArgumentException: Cannot get String from a null object" - needs simple retry logic

[dermoritz]

i created a little upload app to upload folder structures to flickr. The process took many hours.
But after several minutes or hours (it is completely different) i got an exception:
"
Exception in thread "main" java.lang.IllegalArgumentException: Cannot get String from a null object
at org.scribe.utils.Preconditions.check(Preconditions.java:84)
at org.scribe.utils.Preconditions.checkNotNull(Preconditions.java:31)
at org.scribe.utils.StreamUtils.getStreamContents(StreamUtils.java:22)
at org.scribe.model.Response.parseBodyContents(Response.java:43)
at org.scribe.model.Response.getBody(Response.java:69)
at com.flickr4java.flickr.REST.post(REST.java:292)
at com.flickr4java.flickr.uploader.Uploader.postPhoto(Uploader.java:195)
at com.flickr4java.flickr.uploader.Uploader.upload(Uploader.java:124)
at com.flickr4java.flickr.uploader.Uploader.upload(Uploader.java:104)
"
If i repeat the upload on same file (restarting my app) it works fine.

So i created a retry logic that simply reties all calls (i think it could happen on every flickr api call) i used from your api. This is working fine. But i think a retry logic should be part of the flickr api?! What do you think?

There are many variants to implemnt such a logic. i found mine here:
http://stackoverflow.com/questions/9539845/java-how-to-handle-retries-without-copy-paste-code or here:
http://stackoverflow.com/questions/13239972/how-do-you-implement-a-re-try-catch

[ndtreviv]

This is still a problem:

java.lang.IllegalArgumentException: Cannot get String from a null object at
org.scribe.utils.Preconditions.check(Preconditions.java:82) at
org.scribe.utils.Preconditions.checkNotNull(Preconditions.java:29) at
org.scribe.utils.StreamUtils.getStreamContents(StreamUtils.java:20) at
org.scribe.model.Response.parseBodyContents(Response.java:41) at
org.scribe.model.Response.getBody(Response.java:67) at
com.flickr4java.flickr.REST.get(REST.java:181) at
com.flickr4java.flickr.photos.PhotosInterface.search(PhotosInterface.java:1047)

Although this is version 2.16. What are the chances that this error makes more sense for consumers in 2.19?

[boncey]

@ndtreviv Is this happening every time or intermittently as described in the original report?

[ndtreviv]

@boncey It seems to happen intermittently, but consistently...if that makes any sense at all. In other words: it doesn't happen on every call, but I can almost guarantee that it'll happen within a certain number of calls. I haven't yet spent much time figuring out the conditions around when it happens (eg: "does it represent a failed api call because too of flickr fair-use throttling", etc).

It looks like the problem stems from scribe, rather than flickr4java, though. It'd be nice to find out why this is happening (from scribe), and then have flickr4java throw a more useful exception?

[boncey]

I've had a look at my app that uses F4J and I have been seeing the same error too from time to time.
I think it happens when Flickr responds to an API request with a HTML response (it seems to be quite common).

Anyway, it looks like F4J is using a very old version of scribe so the first step will be to upgrade that.
I'll take a look at that first then see about trying to improve the error handling (if it's still happening).

[boncey]

Hello @ndtreviv
I've now released version 3.0.0 which includes a scribe upgrade (it's a breaking change as their code had changed significantly).

Let's see if this error happens again and then perhaps I can intercept it and throw a special exception to make it easier to handle.

[ndtreviv]

Thanks folks. Sorry for the delay. I've compiled it in (although a breaking change it doesn't appear to have broken our particular usage of it...🤷‍♂️). I'll run it for a few days and let you know.

[ndtreviv]

FYI: After running for a few hours it spat out this exception:

java.lang.NullPointerException: null
        at com.flickr4java.flickr.REST.handleResponse(REST.java:248)
        at com.flickr4java.flickr.REST.get(REST.java:159)
        at com.flickr4java.flickr.photos.PhotosInterface.search(PhotosInterface.java:1045)

Is that expected now? Should I be looking for NPEs there?

[boncey]

Is that expected now? Should I be looking for NPEs there?

No, that shouldn't happen. :-)
It looks like Scribe is coming back with a null body, which we're then trying to trim.

I need a way to replicate this locally.
Any advice on what I can do to make it happen locally?

[ndtreviv]

I've pasted a code snippet into here: #302 that shows what we're doing.

We're basically going back in time over flickr, looking for images. If you do the same, starting at 2008-12-31T00:00:00 and subtracting 5 minutes from the date each time, you'll run across the same problems after a while.

I'll keep an eye on the logs to see if I can narrow down the dates for you. I think, looking at previous logs, you should get a hit pretty soon if you start around: 2008-12-14T05:20:01

[ndtreviv]

I just got another NPE running that code with:

• fetchDate: 2008-12-13T17:20:01
• page: 3

[boncey]

I've managed to replicate it and found a simple fix.
Thanks for your guidance.

I'll do a fix to detect these errors and throw a FlickrRuntimeException with an appropriate message.
You should be able to catch that exception and retry.

New release coming soon.

[ndtreviv]

Amazing! Thanks for all your hard work on it. It's much appreciated.

…

On Thu, Feb 21, 2019 at 9:53 PM Darren Greaves ***@***.***> wrote: I've managed to replicate it and found a simple fix. Thanks for your guidance. I'll do a fix to detect these errors and throw a FlickrRuntimeException with an appropriate message. You should be able to catch that exception and retry. New release coming soon. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#64 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABdbHX4ejrbuxVBqjzMWKw8nMHDwXBuIks5vPxVxgaJpZM4BLNdM> .

[boncey]

Release 3.01 fixes this.