req.files empty on nodejs 14

[barisusakli]

I have a feeling this might be a nodejs 14 issue, maybe related to nodejs/node#33050 not sure.

Same code works on node 12.x. On nodejs 14 req.files is just an empty object. Simple app to reproduce below.

const express = require('express')
const path = require('path')
const multi = require('connect-multiparty')
const multiMiddleware = multi();
const app = express()
const port = 3000

app.get('/', (req, res) => res.sendFile(path.resolve(__dirname, './index.html')))

app.post('/upload', multiMiddleware, function (req, res) {
    console.log('got upload', req.files);
    res.send('ok');
});

app.listen(port, () => console.log(`Example app listening at http://localhost:${port}`))

[dougwilson]

Thanks for the report. It does indeed seem to be that issue. And it looks like there is already a PR with a fix on the Node.js repo.

[barisusakli]

14.1.0 got released and above issue was closed but I am still getting the same issue 🤔 https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V14.md#14.1.0

[dougwilson]

Thanks. I just downloaded and testing 14.1.0 and the issue is still something wrong in the fd-slicer dependency, which my understanding of #33050 was it was supposed to fix that (nodejs/node#33050 (comment)). I will take a deeper look.

[barisusakli]

@dougwilson thanks 👍

[ronag]

Does this library anywhere along the dependency chain use stream.pipeline or stream.finished? I can't see that it does, in which case the referenced issue(s) should not be related to this?

[ronag]

Could this be related to fd-slicer and the fact that Node v14 set autoDestroy on streams to true by default (was false before)?

[dougwilson]

No, it does not. It seems there are two main issues: (1) the error from writing too much to fd-slicer no longer comes back in Node.js 14+ (https://github.com/andrewrk/node-fd-slicer/blob/master/index.js#L156) which causes this module to stall and (2) the http request stream is suddenly emitting close event, which is closing the form before file writing has completed.

I can fix (2) in multiparty, but struggling on how to fix (1).

[dougwilson]

Could this be related to fd-slicer and the fact that Node v14 set autoDestroy on streams to true by default (was false before)?

Hmm, I just tried setting autoDestroy: false on all the various streams created in multiparty and the error from fd-slicer is still not getting emitted.

[ronag]

No, it does not. It seems there are two main issues: (1) the error from writing too much to fd-slicer no longer comes back in Node.js 14+ (andrewrk/node-fd-slicer:index.js@master#L156 )

Yea, that's a problem. Calling destroy() without error and then invoking the callback is a bug and won't work anymore. That really should be fixed in fd-slicer... somehow 😞

EDIT: I guess we could maybe somehow add some kind of legacy fallback hack in Node but it won't be pretty.

the http request stream is suddenly emitting close event

Is it incorrectly emitting it?

[dougwilson]

Yea, that's a problem. Calling destroy() without error and then invoking the callback is a bug and won't work anymore. That really should be fixed in fd-slicer... somehow 😞

Hmm, is that an error? The .destroy of that module is not from Node.js, and its own destroy does not accept any argument, so not sure how that could be a bug?

Is it incorrectly emitting it?

I'm not sure yet, but that is the primary reason why the files are always empty here.

[ronag]

Hmm, is that an error? The .destroy of that module is not from Node.js, and its own destroy does not accept any argument, so not sure how that could be a bug?

Hm, good point. Are you sure it doesn't emit error if you set autoDestroy: false on fd-slicer?

If autoDestroy: true then the callback will use destroy(err) to error the stream, which won't emit an error with the overriden destroy impl.

[ronag]

Hmm, is that an error? The .destroy of that module is not from Node.js, and its own destroy does not accept any argument, so not sure how that could be a bug?

So what happens here is:

1. destroy() is called which sets destroyed=true
2. callback(err) is called, which would cause the stream to invoke destroy(err), but because of 1 becomes a noop.

As far as I can see there is no way around this without modifying fd-slicer.

[dougwilson]

Hm, good point. Are you sure it doesn't emit error if you set autoDestroy: false on fd-slicer?

Yea. I even modified fd-slicer source to force the autoDestroy setting to false, and still no error emitted. I added a trace into the .destroy method of fd-slicer and it is never called with an error argument at any point.