Very slow POST of product with media (base64)

[george-vlahakis]

When adding a new product via REST API (POST/PUT) and the product contains media image with base64 encoding the performance increases by 300% (from .60 sec to over 2 sec depending on base64 image size.

Preconditions

1. Magento CE 2.1.6

Steps to reproduce

1. REST API products POST (add)

Expected result

1. Normal or close to normal speeds

Actual result

1. Above 2 sec response times. Below 1 sec if media is not attached

Any other way of achieving the import without the massive slow-down? Calling the media URL to add the image yields the same slow-down.

[choukalos]

@vlahakis how is your install setup to store the media? How big is the base64 encoded block (on average)? How many images are you sending in the call?

[george-vlahakis]

Hi, Media storage is on disk (versus db). One base media image with each call and on average a few Kbytes, nothing massive. It clearly however shows a considerable slowdown when adding the media to the call.

…

On 18 Apr 2017, at 18:08, Chuck Choukalos ***@***.***> wrote: @vlahakis how is your install setup to store the media? How big is the base64 encoded block (on average)? How many images are you sending in the call? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[choukalos]

I suspect the majority of the slow down is the image resize being fired, but given the size of the image that seems like a lot. What happens to the time if you try it with 3-5 images, does it increase linearly? What about the 2nd API call (with different image)?

[george-vlahakis]

Hello, All calls have an increase regardless. I have had no time to debug backend but I suspect it's somewhere in the pipeline of processing the base64 string. Multiple images in same I have not tried as it does not apply in my scenario. Have tried with and without the image media and saw the increase, hence the report here.

…

On 18 Apr 2017, at 21:35, Chuck Choukalos ***@***.***> wrote: I suspect the majority of the slow down is the image resize being fired, but given the size of the image that seems like a lot. What happens to the time if you try it with 3-5 images, does it increase linearly? What about the 2nd API call (with different image)? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[choukalos]

Hi @vlahakis ,

Thanks for the info. I suspect this is to be expected because the API is triggering an image re-size which is causing the image to be saved along with multiple image resize operations (based on what's needed by your selected theme). I have backlog for a future improvement where we'd make this asynchronous to defer the image resize processing. Would that address your needs here?

Thanks,
Chuck

[george-vlahakis]

Hi, Yes I think anything that could assist in speed improvements over the API usage would be great. If you do put the operation in async mode would it somehow prevent the product from being viewed in the store? I suspect/hope not. Best,

…

On 24 Apr 2017, at 22:11, Chuck Choukalos ***@***.***> wrote: Hi @vlahakis <https://github.com/vlahakis> , Thanks for the info. I suspect this is to be expected because the API is triggering an image re-size which is causing the image to be saved along with multiple image resize operations (based on what's needed by your selected theme). I have backlog for a future improvement where we'd make this asynchronous to defer the image resize processing. Would that address your needs here? Thanks, Chuck — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9280 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMnNUXEbek_80P0FFCNRm1ufhED1iRxlks5rzPPzgaJpZM4M_cHS>.

[george-vlahakis]

Ideally it would also be great (feature request) to be able to temporarily disable

\Magento\Catalog\Model\Product\Image\Cache::processImageData

This takes approximately 48% of the POST call (after profiling).

I believe it would be an option in Cache Management to disable (or set on schedule and so on) the Catalog Images Cache.

It would allow for a considerable boost in performance during these operations and then you can re-enabled it or force an images resize on the console to update it.

(just my thoughts)

[george-vlahakis]

I changed a few things in catalog:images:resize to get some stats on the resizing algorithm and it turns out has a rather poor performance (using ImageMagick in hope of performance boost)...

http://undefined.gr/site/2017/05/18/magento-catalogimagesresize-stats-and-memory/

Current state of affairs on this says that with ~16000 and an average time per product over 4-5 secs I am looking at days for this to finish!

[hostep]

@vlahakis: you can find some hints about improving performance of catalog:images:resize over here: #8145

Also: take a look at your theme's view.xml file, and configure all the image dimensions of image types you don't use in the frontend, to the exact same dimension (we put it at 16x16), this makes the catalog:images:resize command to create many fewer files, and in my experience this speeds up the command a lot (almost half the time in total).

[george-vlahakis]

Thank you @hostep, did read that post but wasn't keen on changing core Magento sources. Will review the image sizes on the theme as well (using luma for now).

[george-vlahakis]

Tried applying the Image.php suggestion from your last comment but it appears that it completely bumped the resizing benchmarks. Rolled back to as it was (2.1.6)

#8145 (comment)

[hostep]

Ok, that's really strange, there is nothing in that code which makes the process slower, it should at least be as "fast" as a stock 2.1.6 installation and normally a lot faster.

[magento-engcom-team]

Hi @georgios-2317
Thanks for reporting this issue.
Can you give more details to reproduce this issue. Maybe examples API request or other details, that helps reproduce this behavior on clean magento installation.

[george-vlahakis]

Hello,

I have since upgraded to 2.2 and no longer use REST API due to the slow down. The ticket here should have enough information for you to replicate behavior. In essence calling REST API to add media to existing product is a simple and direct call.

[magento-engcom-team]

Hi @georgios-2317
You can help to detect the bug faster If you provide json content of your API request, since the response time depends on the size and count of images.

[george-vlahakis]

Dont get me wrong @magento-engcom-team I want to help with the process but finding it a bit difficult to understand your feedback here.

The response times for multiple images is not being reported here, so your comment on "count of images" does not apply.

The original ticket was for one image to be posted on an existing product via REST POST API.

So, a typical call may be to either

POST > catalogProductAttributeMediaGalleryManagementV1
or
POST > catalogProductRepositoryV1

Where the base64 content is an image of lets say 132K. Here's one for your convenience (taken from Google Search - not mine): https://www.organicfacts.net/wp-content/uploads/aromatherapy.jpg

The rest of the information is within this ticket I believe but to recap, POSTing an image to media or product causes a considerable increase in processing time which could translate to either Magento re-processing the image or something wrong in the base64 decoding process (or I/O).

As I mentioned, I no longer use the REST API due to speed delays and have now moved to direct DB calls (knowingly accepting the risk behind this).

Hope I gave some sort of assistance here.

PS: I cannot confirm if the ticket still applies in Magento 2.2

[magento-engcom-team]

@georgios-2317, thank you for your report.
We've created internal ticket(s) MAGETWO-83242 to track progress on the issue.

[guidokudo]

Hi, is there any update on this issue?
We currently have around 5000 products (57000 including variations) each variation has 4-5 images. We insert products (attributes, etc.) with a separate script. Download images with separate script. Although some images might be as big as 3-4mb it takes only 1-2 hours for both scripts finish their jobs. Now then we run our 3 script, to assign image to products. It assigns ~150 products' images in 1 hour. So it is taking 380 hours (16 full non-stop days). And by that time we have to renew more than half of our products. So we are stuck in non-ending loop, where we always have most of our products without images. :)
2.2 GHz Intel Xeon E5 x8 CPU, 30GB Mem. and we use purely SSD persistent disk.
CPU Load get max. 21% very rarely.

I did not post a separate issue, because I think we have the same issue here.

[ankit-sf]

Hi,
the same issue in the case of large images causes timeout error:

Error 503 first byte timeout
first byte timeout
Guru Mediation:
Details: cache-fra19143-FRA 1526474931 18927345314
Varnish cache server

[cmuench]

I also analysed the API call. One idea would be to move the image processing to an asynchronously process. Maybe we can process all images by a worker.
After each call the cache is invalided. This could also be optimized.
Maybe we can refactor this within the BulkAPI project.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 14 days if no further activity occurs. Is this issue still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? Thank you for your contributions!

[sodhancha]

Was this resolved on 2.4+ ?

[AndreaGot]

Was this resolved on 2.4+ ?

I don't think so, I am currently experiencing the same issue. I can just upload images less than ~30kB, but anything over that takes a considerably long time. To be fair, I tried uploading a 2MB image many times but I never saw it finishing successfully.