Unable to archive 1M files

[bouri-mounir]

I want to use archiver to compress more then 1 million files and stream the result to http response.
When using zip-stream : archive not finalized correctly and download is blocked (0 o/s ...)
Seem to work fine when using tar-stream.

Testing code :

archive.pipe(res);
for (var i=0;i<1000000;i++)
            {
                let jsonFileContent = {test:"test"};
                archive.append(JSON.stringify(jsonFileContent, null, 2), {
                    name: path.join('test', "json-test-" + i + ".json")
                })
            }
archive.finalize();

[bouri-mounir]

My workaround to solve the problem :

• Update readble-stream module version used by compress-commons to v3.4.0.
• Add a backpressure mechanism to archiver
  https://nodejs.org/es/docs/guides/backpressuring-in-streams/ :
• Wait for data to drain to avoid memory issue : example

function write(data, cb) {
  if (!stream.write(data)) {
    stream.once('drain', cb);
  } else {
    process.nextTick(cb);
  }
}
// Wait for cb to be called before doing any other write.
write('hello', () => {
  console.log('Write completed, do more writes now.');
});

Instead of :
stream.write();