Batch Push to all users uses a lot of RAM

[benishak]

ENV:

Parse-Server : 2.2.16 on AWS EC2 T2.MEDIUM (CentOS+NGINX)
Parse JavaScript SDK : 1.9.0
Push Adapter : AWS SNS

Problem

Before we were using Parse native to send Push but it fails to send to all installations, so we switched to use AWS SNS Push Adapter
But ... when we send push to all installation (12k), our server get huge CPU load and massive RAM usage more than 1.5GB

How can we get ride of high CPU load and massive RAM usage ? what is the best practice to handle Pushes in general ?

[krzkz94]

Offloading it to a queue like kue.js in batches could help perhaps?

https://github.com/Automattic/kue

That way you could as well add scheduling (something i see missing at the moment)

[gateway]

Yea, I would like to know what the best approach is because we will be facing a similar issue. In the past we used RabbitMQ and had it send though SNS, but in the parse world I'm not sure what to do yet..

[flovilmart]

any queueing system is highly opinionated, that would likely require a new adapter that would take care of slicing/batching as well as leveraging the configured push adapter (or another push mechanism).

As always, PR are welcome :)

[gateway]

Btw we might be looking for a dev to pay to build this.. I personally think its important to tackle and I'm not sure 90% of the parse.com users that are migrating realize that parse server can come to a crawl.. I do understand @flovilmart point of view but this should be tackled somehow..

ideally the flow could be parse -> message/worker que --> push provider

parse -> amazon sqs -> amazon sns
parse -> iron.io -> amazon sns (or other provider).

anyhow needs to be built if anyone thats a node.js guru wants to tackle this let me know.

[benishak]

how would this work with SQS + SNS,
normally it goes this way
Parse -> SNS -> SQS then the user should pull the message from the Queue (maybe some code must be add in the client side)
I think in this case each device using Parse must have it own Queue in SQS and each Queue.

I'm really quite confused here specially if it would come to targetet push notification! But for Batch/Bulk Push it is easy because the message is same for all receiver, I guess, or do I understand this wrong ?

[flovilmart]

@benishak we need to pull all the deviceTokens from the DB and that takes up a lot of memory. Then you need to send those to the push provider, APNS or GCM or other wether it's a single or many, that ends up being pretty much the same amount of calls.