[v1 / v2] Takes too long to upload DCM/DICOM file

[andrei0807]

error_file.zip

Unzip the file and try uploading it using formatable.
Then node's memory size will be very increase and it will be take a long time to upload.
I got trouble for only this file.
I think formidable has some error.
Best regards.

[tunnckoCore]

First, that tone isn't okay with me, but anyway. It cannot be urgent while we (open source developers) have TON OF stuff out there out of good will and for free.

And second, what version are you using? How big is this file? And etc. It really greatly depends on hundreds of other stuff.

Try using formidable@canary and report back please.

[andrei0807]

@tunnckoCore
sorry for tone.
There is no other problem with big files.
However, only this file has errors.
I already uploaded file in this issue.
I am using version 1.2.1.
and I tested with examples in the repo and got the same result.

[tunnckoCore]

Try with latest canary version. Can't help much with v1.2.1 version, which is quite buggy and quite old. That formidable@canary is a prerelase of v2, which clear most of the previously opened PRs and issues, plus new features and modernized codebase.

I've seen that the size is around 52mb, but will not open it. Is it binary or it is text? A code snippet of your actual code that uses formidable will be good too.

Or it can be just broken/tampered file which cannot be parsed/handled/processed correctly.

[andrei0807]

It's dcm file.
https://fileinfo.com/extension/dcm
and I tested it too using current repo source code with /example/upload.js
But got same error.

[tunnckoCore]

@andrei0807 it could be just because the type of the file, but it actually uploads it. I tried and it got me around 1 minute or so.

Can you try something like multer, multiparty, or busboy? If it takes around the same time, it's probably because the file type/format.

[andrei0807]

A minute is too long in local for 50mb.
Other dcm files are not problematic, but only one manufacturer's files are problematic.
In this case, what should I set the mime/type to?

[tunnckoCore]

A minute is too long in local for 50mb.

Sure. It's waaaay too much.

Other dcm files are not problematic, but only one manufacturer's files are problematic.

That's the key thing. Maybe they are doing something to them that isn't okay in general, and probably will be a problem for the other nodejs multipart parsers. That's why we should try with them.

I can't help much because I'm not familiar with that format and in the parsers THAT much. The thing is that I don't think it's really because some of our parsers problem. It detects it as application/octet-stream.

And in anyway, with #545 Plugins API, anyone can implement any parser they want.

[andrei0807]

Thanks for your answer.
Can't you find where the error is coming from?
There seems to be an error in the writestream.

[tunnckoCore]

There seems to be an error in the writestream.

When I tried, it doesn't errors, but uploads the file successfully - slow but successfully without errors.

[andrei0807]

Please check node server's memory.
The memory will be increased.
So heap limit occurs when used in a real server.

[andrei0807]

@tunnckoCore
I think the error is in the multipart parser.
There are a lot of 0x0D in the dcm file and it seems like parser is parsing this incorrectly.

[andrei0807]

error_file.zip
fine.zip
I uploaded two file.
These files are same manufacturer's files.
But one is fine and other has problem.
I hope these files will help you find the error.

[GrosSacASac]

Updated wording to conform to coc

[tunnckoCore]

Great, thanks. We will see.

[andrei0807]

Did you fix issue?

[tunnckoCore]

@andrei0807 nothing new. I can't help on that. Pull requests and ideas are welcome.

Is there similar problem with multer or busboy? Please try them both.

[andrei0807]

How about this bug?
I checked with multer. it's fine now.
Can you tell me when fix this bug?
Thanks

[tunnckoCore]

I don't have any ETA on v2, but also not able to fix it because I don't have any ideas.

Maybe check in v3 (this year) which I think will be rewritten, so...

[GrosSacASac]

I cannot reproduce , closing