Crash: Cannot read property 'callback' of undefined at normal_reply (Redis)

[System-Glitch]

Describe the bug

We have deployed Cube JS on Google Cloud Run using the official docker image. We use the most basic cube setup (no cube store, no pre-aggregations, etc, as we are using cube as a query builder mostly). We use a redis cache (also deployed on GCP). The Cube instance randomly crashes very often with the following error:

TypeError: Cannot read property 'callback' of undefined at normal_reply (/cube/node_modules/redis/index.js:651:28)
  at RedisClient.return_reply (/cube/node_modules/redis/index.js:753:9)
  at JavascriptRedisParser.returnReply (/cube/node_modules/redis/index.js:138:18)
  at JavascriptRedisParser.execute (/cube/node_modules/redis-parser/lib/parser.js:544:14)
  at Socket.<anonymous> (/cube/node_modules/redis/index.js:219:27)
  at Socket.emit (events.js:314:20) at addChunk (_stream_readable.js:297:12)
  at readableAddChunk (_stream_readable.js:272:9)
  at Socket.Readable.push (_stream_readable.js:213:10)
  at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
/cube/node_modules/redis/index.js:651
if (typeof command_obj.callback === 'function') {
                      ^

The only env variable related to Redis in our configuration: CUBEJS_REDIS_URL: redis://xx.xx.xx.xx:xxxx

We have tried with v0.28, v0.29 and v0.30 and always have the same issue. I think this may related to this issue in the redis package: redis/node-redis#1593

This is affecting our services using Cube pretty badly. If you know a workaround until this is resolved by the redis team, please let me know!

To Reproduce
Steps to reproduce the behavior:

1. Deploy the base cube image on Google Cloud Run with a redis cache
2. Check the logs, wait for a bit and it should crash eventually

Version:
0.28.x, 0.29.x, 0.30.x

[joeledwards]

We are observing the same issue in AWS Lambda on 0.26

Have you tried switching to the ioredis driver to see if the issue persists?

[System-Glitch]

Hi @joeledwards , thank you very much for the suggestion. I deployed with CUBEJS_REDIS_USE_IOREDIS=true and will monitor the logs closely to see if the problem persists.

Edit: after the change, I sometimes get the following message in my logs:

[ioredis] Unhandled error event: Error: connect ETIMEDOUT

However it doesn't crash the application, and ioredis appears to be retrying to connect so the app keeps running. 👍🏻

[joeledwards]

Glad that helped a bit. I think tuning the timeout could help for ioredis, though I haven't attempted that myself, so I don't know whether the option is exposed.