More useful error than `toString failed` when Buffer length greater than kMaxLength?

[mhart]

I'm assuming there's some sort of memory constraint trying to convert buffers into strings?

$ uname -a
Darwin ReBuke-Pro.local 15.0.0 Darwin Kernel Version 15.0.0: Wed Aug 26 16:57:32 PDT 2015; root:xnu-3247.1.106~1/RELEASE_X86_64 x86_64
$ node --version
v4.1.1
$ node -p 'new Buffer(268435440).toString().length'
268435440

But:

$ node -p 'new Buffer(268435441).toString().length'
buffer.js:378
    throw new Error('toString failed');
    ^

Error: toString failed
    at Buffer.toString (buffer.js:378:11)
    at [eval]:1:23
    at Object.exports.runInThisContext (vm.js:54:17)
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (module.js:434:26)
    at node.js:566:27
    at doNTCallback0 (node.js:407:9)
    at process._tickCallback (node.js:336:13)

Firstly, I'm not sure that this should even be an error (are we really only limited to 256MB? That's... not very much). This seems to have been addressed in the past, but I feel like a regression might have occurred: #1374 (comment)

But that aside, can we actually detect what the error is and report on it in a more detailed way? The current toString check provides little insight.

[bnoordhuis]

256 MB is the limit for strings in V8 so no, it's not a regression. What would you report in more detail?

[mhart]

Anything would be better than that TBH.

[mhart]

It's not obvious to the user what's actually gone wrong.

How about "Cannot create string larger than 256MB"? Seems a lot clearer to me.

[bnoordhuis]

I think a pull request to that effect would be acceptable.

[mhart]

But is that the only error that can occur? Looking at the code at the moment, the check is pretty basic – is there no way of getting more detail as to why result may be undefined?

[bnoordhuis]

Not at the moment, no. The undefined value trickles down from StringBytes::Encode() in src/string_bytes.h. I don't think there are other code paths that return an empty Local<String> but I didn't check exhaustively.

[mhart]

FWIW, v8 (I assume?) spits out a slightly better error when trying to concat strings:

$ node -p 'var a = "a"; for (var i = 0; i < 27; i++) a = a + a; a.length'
134217728
$ node -p 'var a = "a"; for (var i = 0; i < 28; i++) a = a + a; a.length'
[eval]:1
var a = "a"; for (var i = 0; i < 28; i++) a = a + a; a.length
                                                ^

RangeError: Invalid string length
    at [eval]:1:49
    at Object.exports.runInThisContext (vm.js:54:17)
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (module.js:434:26)
    at node.js:566:27
    at doNTCallback0 (node.js:407:9)
    at process._tickCallback (node.js:336:13)

Is the size limit exposed at all? If it were I'd be happy to add a PR that checks, in the case of an undefined result, whether the Buffer length is greater than it.

[targos]

I don't think it is exposed in JS. The exact limit is (1 << 28) - 16 (268435440) and is defined in

node/deps/v8/include/v8.h

Line 2083 in 64beab0

static const int kMaxLength = (1 << 28) - 16;

[MylesBorins]

I would love to take a stab at this if people are open to it.

/cc @jasnell

[jasnell]

Works for me!
On Oct 6, 2015 12:49 PM, "Myles Borins" notifications@github.com wrote:

I would love to take a stab at this if people are open to it.

/cc @jasnell https://github.com/jasnell

—
Reply to this email directly or view it on GitHub
#3175 (comment).

[trevnorris]

@MylesBorins Seems the discussion addresses more than just the error message. Mind identifying the specifics of what you'll be working on?

[MylesBorins]

I specifically was interested in a better error message when toString is called on a Buffer larger than kStringMaxLength

That seemed like a pretty easy one to knock out (I've basically got it done already).

[mhart]

@MylesBorins I think the discussion was around how to detect the limit though – have you exposed kMaxLength?