Some error occuered after update 3.0.0

[DingGGu]

I update socket.io-redis 3.0.0 right now,
But few Unknown error occured from error logs.

msgpack-js:msgpack in Decoder.parse
error: Unknown type 0xc7

msgpack-lite.lib:flex-buffer in DecodeBuffer.reserve
error: BUFFER_SHORTAGE

msgpack-js:msgpack in Object.decode
error-686655034 trailing bytes

Maybe It caused by "msgpack-lite": "0.1.26"

[darrachequesne]

Yes, we switched to msgpack-lite here, in order to fix other decoding issues...

Are you able to reproduce consistently?

[DingGGu]

Yes, It happened repeatedly and even now.

Error: BUFFER_SHORTAGE
 File "<PROJECT>/node_modules/msgpack-lite/lib/flex-buffer.js", line 64, col 41, in DecodeBuffer.reserve
   if (end > this.buffer.length) throw new Error(BUFFER_SHORTAGE);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-format.js", line 117, col 23, in uint8
   var start = decoder.reserve(1);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-token.js", line 152, col 15, in null.<anonymous>
   var len = lenFunc(decoder);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-core.js", line 25, col 12, in decode
   return func(decoder);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-format.js", line 52, col 12, in map_to_obj
   v[i] = decode(decoder);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-token.js", line 159, col 12, in null.<anonymous>
   return method(decoder, len);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-core.js", line 25, col 12, in decode
   return func(decoder);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-format.js", line 81, col 16, in array
   value[i] = decode(decoder);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-token.js", line 159, col 12, in null.<anonymous>
   return method(decoder, len);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-core.js", line 25, col 12, in Codec.decode
   return func(decoder);
 File "<PROJECT>/node_modules/msgpack-lite/lib/decode-buffer.js", line 26, col 21, in DecodeBuffer.fetch
   return this.codec.decode(this);
 File "<PROJECT>/node_modules/msgpack-lite/lib/flex-buffer.js", line 166, col 28, in DecodeBuffer.read
   if (!length) return this.fetch();
 File "<PROJECT>/node_modules/msgpack-lite/lib/decode.js", line 10, col 18, in Object.decode
   return decoder.read();
 File "<PROJECT>/node_modules/socket.io-redis/index.js", line 145, col 24, in Redis.onmessage
   var args = msgpack.decode(msg);
 File "events.js", line 106, col 13, in emitTwo
 File "events.js", line 191, col 7, in RedisClient.emit
 File "<PROJECT>/node_modules/socket.io-redis/node_modules/redis/index.js", line 776, col 18, in return_pub_sub
   self.emit('messageBuffer', reply[1], reply[2]);
 File "<PROJECT>/node_modules/socket.io-redis/node_modules/redis/index.js", line 825, col 9, in RedisClient.return_reply
   return_pub_sub(this, reply);
 File "<PROJECT>/node_modules/socket.io-redis/node_modules/redis/index.js", line 188, col 18, in JavascriptRedisParser.Parser.returnReply
   self.return_reply(data);
 File "<PROJECT>/node_modules/redis-parser/lib/parser.js", line 491, col 12, in JavascriptRedisParser.execute
   this.returnReply(response)
 File "<PROJECT>/node_modules/socket.io-redis/node_modules/redis/index.js", line 267, col 27, in Socket.<anonymous>
   self.reply_parser.execute(buffer_from_socket);
 File "events.js", line 96, col 13, in emitOne
 File "events.js", line 188, col 7, in Socket.emit
 File "_stream_readable.js", line 172, col 18, in readableAddChunk
 File "_stream_readable.js", line 130, col 10, in Socket.Readable.push
 File "net.js", line 542, col 20, in TCP.onread

[darrachequesne]

Are you using some external emitter (like socket.io-emitter or socket.io-php-emitter), that could publish packets that msgpack-lite would be unable to decode?

Or with several nodes, some that are using msgpack-lite and others still msgpack-js?

Related: #169

[DingGGu]

Yes of course. I use socket.io-emitter and I just upgraded few hours ago both socket.io-emitter and socket.io-redis

npm ls msgpack-lite
└└──┬┬ socket.io-emitter@2.0.0
  └└──── msgpack-lite@0.1.26

and there is no usage msgpack-js

[darrachequesne]

@DingGGu may I ask what your setup is?

• Redis server version?
• Redis client version (node_redis, ioredis)?

I'd like to make sure that this issue is really related to msgpack-lite, before maybe switching back to msgpack-js..

[mavrick]

I'm getting the same error with my setup:

Redis Server:

  redis_version: 2.8.12
  redis_git_sha1: 0
  redis_git_dirty: 0
  redis_mode: standalone
  os: Linux 3.2.0-48 virtual x86_64
  arch_bits: 64
  multiplexing_api: epoll
  gcc_version: 4.6.3
  process_id: 7677628
  run_id: 
  tcp_port: 0
  uptime_in_seconds: 60156093
  uptime_in_days: 696
  hz: 10
  lru_clock: 0

npm versions:

"redis": "~2.6.3",
"socket.io": "1.7.2",
"socket.io-emitter": "1.1.0",
"socket.io-redis": "3.0.0"

code example:

var redisSub = redis.createClient(redis.port, redis.host, { detect_buffers: true, auth_pass: redis.pass});

redisSub.on("message", function(channel, message)
{
	console.log(channel, message);
});

var redisPub = redis.createClient(redis.port, redis.host, {auth_pass: redis.pass});

redisPub.publish("discord", JSON.stringify({type: "status", data: {uuid: id, status: status}}));

var socketio = require("socket.io");
var io = socketio.listen(server.server);
var ioredis = require("socket.io-redis");

var adapter = ioredis({
	pubClient: redisPub, 
	subClient: redisSub
});

io.adapter( adapter );

data being received:

socket.io#/# : '��y650LF��type\u0002�data��discord::status�\u0000dN4IgLgngDgpiBcIDOYCGYCuSQBoQBN1UFQMMBLfBEARgDYBmAFgE4BWABgHYuAOOxlwY0ATB1zI0mbIgD2AM3kAbcgDs4AXw1A==�nsp�/��roomsĥflags�'

stack trace:

Sun, 15 Jan 2017 10:23:11 GMT uncaughtException: BUFFER_SHORTAGE
Error: BUFFER_SHORTAGE
    at DecodeBuffer.reserve (c:\app\node_modules\msgpack-lite\lib\flex-buffer.js:64:41)
    at bin_buffer (c:\app\node_modules\msgpack-lite\lib\read-format.js:93:23)
    at c:\app\node_modules\msgpack-lite\lib\read-token.js:153:12
    at decode (c:\app\node_modules\msgpack-lite\lib\read-core.js:25:12)
    at map_to_obj (c:\app\node_modules\msgpack-lite\lib\read-format.js:52:12)
    at c:\app\node_modules\msgpack-lite\lib\read-token.js:159:12
    at decode (c:\app\node_modules\msgpack-lite\lib\read-core.js:25:12)
    at array (c:\app\node_modules\msgpack-lite\lib\read-format.js:81:16)
    at c:\app\node_modules\msgpack-lite\lib\read-token.js:159:12
    at Codec.decode (c:\app\node_modules\msgpack-lite\lib\read-core.js:25:12)
    at DecodeBuffer.fetch (c:\app\node_modules\msgpack-lite\lib\decode-buffer.js:26:21)
    at DecodeBuffer.read (c:\app\node_modules\msgpack-lite\lib\flex-buffer.js:166:28)
    at Object.decode (c:\app\node_modules\msgpack-lite\lib\decode.js:10:18)
    at Redis.onmessage (c:\app\node_modules\socket.io-redis\index.js:145:24)
    at emitTwo (events.js:106:13)
    at RedisClient.emit (events.js:191:7)
Sun, 15 Jan 2017 10:23:11 GMT uncaughtException: BUFFER_SHORTAGE
Error: BUFFER_SHORTAGE
    at DecodeBuffer.reserve (c:\app\node_modules\msgpack-lite\lib\flex-buffer.js:64:41)
    at bin_buffer (c:\app\node_modules\msgpack-lite\lib\read-format.js:93:23)
    at c:\app\node_modules\msgpack-lite\lib\read-token.js:153:12
    at decode (c:\app\node_modules\msgpack-lite\lib\read-core.js:25:12)
    at map_to_obj (c:\app\node_modules\msgpack-lite\lib\read-format.js:52:12)
    at c:\app\node_modules\msgpack-lite\lib\read-token.js:159:12
    at decode (c:\app\node_modules\msgpack-lite\lib\read-core.js:25:12)
    at array (c:\app\node_modules\msgpack-lite\lib\read-format.js:81:16)
    at c:\app\node_modules\msgpack-lite\lib\read-token.js:159:12
    at Codec.decode (c:\app\node_modules\msgpack-lite\lib\read-core.js:25:12)
    at DecodeBuffer.fetch (c:\app\node_modules\msgpack-lite\lib\decode-buffer.js:26:21)
    at DecodeBuffer.read (c:\app\node_modules\msgpack-lite\lib\flex-buffer.js:166:28)
    at Object.decode (c:\app\node_modules\msgpack-lite\lib\decode.js:10:18)
    at Redis.onmessage (c:\app\node_modules\socket.io-redis\index.js:145:24)
    at emitTwo (events.js:106:13)
    at RedisClient.emit (events.js:191:7)

[darrachequesne]

redisPub.publish("discord", JSON.stringify({type: "status", data: {uuid: id, status: status}}));

@mavrick JSON.stringify? Shouldn't it be msgpack.encode instead? Else msgpack.decode will obviously fail when trying to parse JSON content.

[DingGGu]

I just downgraded version,
and then resolved.

[DingGGu]

Redis Version: 3.2.4 (from AWS ElastiCache)

Redis connect with socket.io-redis

let Adapter = adapter({host: config.redis.host, port: config.redis.port});
io.adapter(Adapter);

According to source of socket.io-redis, I found node_redis, that was redis client.

[mavrick]

@darrachequesne doesn't msgpack get called inside the publish method? I don't think I should have to call msgpack myself, or do I?

What version did you rollback to @DingGGu ? I may have to do the same at this stage.

[darrachequesne]

redisPub.publish("discord", JSON.stringify({type: "status", data: {uuid: id, status: status}}));

@mavrick I may be missing something here, but if you want to publish some packets, why aren't you using the socket.io-emitter directly, instead of the redis pub?

var io = require('socket.io-emitter')({ host: '127.0.0.1', port: 6379 });
io.emit("discord", JSON.stringify({type: "status", data: {uuid: id, status: status}}));
// msgpack.encode gets indeed called

[mavrick]

@darrachequesne i can't argue with that logic. I'll give it a go this evening and see what results I get.

Cheers

[DingGGu]

@mavrick Here is my versions.

"socket.io-emitter": "^1.0.0",
"socket.io-redis": "^2.0.1"

I guess maybe socket.io-emitter has same issue too.
I downgraded two both of packages.

[mavrick]

After using socket.io-emitter instead of redis.publish() my msgpack-lite errors have gone away. Will do some more testing over the coming days.