JavaScript heap out of memory when packaging many functions

[daniel-cottone]

This is a Bug Report

Description

I'm in the process of trying to upgrade serverless-webpack version from 2.2.3, where I do not experience the following issue. Our serverless configuration has package: invididually: true set, and about 40 functions. When I try to upgrade to a later version of serverless-webpack and run sls webpack, the build will run for about a minute and then I get the following error:

lambda:daniel.cottone $ npm run build

> expert-api-lambda@0.1.0 build /Users/daniel.cottone/Projects/expert-api/lambda
> sls webpack --stage dev

Serverless: Bundling with Webpack...
ts-loader: Using typescript@2.5.2 and /Users/daniel.cottone/Projects/expert-api/lambda/tsconfig.json
ts-loader: Using typescript@2.5.2 and /Users/daniel.cottone/Projects/expert-api/lambda/tsconfig.json
ts-loader: Using typescript@2.5.2 and /Users/daniel.cottone/Projects/expert-api/lambda/tsconfig.json
ts-loader: Using typescript@2.5.2 and /Users/daniel.cottone/Projects/expert-api/lambda/tsconfig.json
ts-loader: Using typescript@2.5.2 and /Users/daniel.cottone/Projects/expert-api/lambda/tsconfig.json
ts-loader: Using typescript@2.5.2 and /Users/daniel.cottone/Projects/expert-api/lambda/tsconfig.json
ts-loader: Using typescript@2.5.2 and /Users/daniel.cottone/Projects/expert-api/lambda/tsconfig.json

<--- Last few GCs --->

[42611:0x104001600]    55964 ms: Mark-sweep 1405.7 (1508.8) -> 1405.7 (1508.8) MB, 1721.0 / 0.0 ms  allocation failure GC in old space requested
[42611:0x104001600]    57889 ms: Mark-sweep 1405.7 (1508.8) -> 1405.5 (1487.3) MB, 1923.4 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1923 ms) last resort 
[42611:0x104001600]    59801 ms: Mark-sweep 1405.5 (1487.3) -> 1405.4 (1486.8) MB, 1903.6 / 0.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x37341f01ba79 <JS Object>
    1: set [native collection.js:~247] [pc=0x29d828934f21](this=0x332730f95301 <a Map with map 0x23d2df14319>,p=0x3dd499abec41 <String[11]: MediaSource>,x=0x2589b9b1c819 <a SymbolObject with map 0x399abfecde11>)
    2: /* anonymous */(aka /* anonymous */) [/Users/daniel.cottone/Projects/expert-api/lambda/node_modules/typescript/lib/typescript.js:~23166] [pc=0x29d828ba5830](this=0x37341f002241 <...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/local/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [/usr/local/bin/node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
 6: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, v8::internal::JSMapIterator, 2>::Allocate(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [/usr/local/bin/node]
 7: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, v8::internal::JSMapIterator, 2>::Rehash(v8::internal::Handle<v8::internal::OrderedHashMap>, int) [/usr/local/bin/node]
 8: v8::internal::Runtime_MapGrow(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
 9: 0x29d827e840bd
10: 0x29d828934f21
11: 0x29d828ba5830
12: 0x29d827e86bbb
13: 0x29d828f85beb
Abort trap: 6

If I change my serverless config to not package individually, package: individually: false then this error goes away. I have tested this with version 3.0.0 and the latest, 4.1.0 with the same results. Don't have this issue with 2.2.3.

Additional Data

• Serverless-Webpack Version you're using: 4.1.0
• Webpack version you're using: 3.10.0
• Serverless Framework Version you're using: 1.24.0
• Operating System: macOS 10.12.6
• Stack Trace (if available): see above

[HyperBrain]

Hi @daniel-cottone ,
thanks for reporting. This can be something with your configuration. SLS-webpack since 3.0.0 requires that you use slsw.lib.entries for your entry definitions and have the function handlers declared correctly in your serverless.yml in case you use individual packaging.

Can you post the function definitions from your serverless.yml and the webpack config file?

[daniel-cottone]

Hey @HyperBrain thanks for quick response. Here's the webpack configuration:

var path = require('path');
var slsw = require('serverless-webpack');
var Webpack = require('webpack');

module.exports = {
    entry: slsw.lib.entries,
    resolve: {
        extensions: ['.ts', '.js', '.json']
    },
    target: 'node',
    output: {
        libraryTarget: 'commonjs',
        path: path.join(__dirname, '.webpack'),
        filename: '[name].js'
    },
    externals: [{
        'aws-sdk': 'aws-sdk',
        'mysql2': 'mysql2',
        'sqlite3': 'sqlite3',
        'tedious': 'tedious',
        'pg-native': 'pg-native'
    }],
    module: {
        loaders: [
            { test: /\.ts(x?)$/, loader: 'ts-loader' }
        ]
    }
};

The definitions for all 40 functions is too large to post, but I'll post an example:

users-list:
    handler: src/handler/UserHandler.getUsers
    events:
        - http:
            path: users
            method: get

They pretty much all look the same, I've clipped out VPC, authorizer, and environment config. I'm pretty confident that they're all configured correctly.

[HyperBrain]

The handlers look good. However, version 2.x did not support individual packaging (in fact it only copied the whole artifact per function).
So you should, as next step, add node externals to your webpack configuration to let the externals be automatically determined by webpack, so that individual packaging can make use of it:

// webpack config
const nodeExternals = require('webpack-node-externals');
...
  externals: [ nodeExternals() ]
...

Additionally, webpack > 3.0.0 now uses a module: rules structure instead of module: loaders. You should change that too.

Please also check if you have set custom: webpackIncludeModules: true in your serverless.yml.

Then do a serverless package to test, if it works. You'll find the zip packages that would be uploaded in the .serverless directory.

If aws-sdk should be packaged, you can either put it into your devDependencies or use

# serverless.yml
custom:
  webpackIncludeModules:
    forceExclude:
      - aws-sdk

to keep it outside of your packages.

[daniel-cottone]

I've made your suggested changes to webpack externals and have added the webpackIncludeModules configuration to serverless custom config; I still seem to be experiencing the same problem though.

[daniel-cottone]

It also appears to be related to the fact that there are so many functions in this serverless project; if I comment out all but 5 then sls package works.

[HyperBrain]

Hmmm... that sounds like a memory leak somewhere when using individual packaging.
We also have a project with more than 30 functions which works, but I did not check how the memory consumption is there (i.e. if we're about to hit a limit).

What you can try is, to increase node's heap memory limit (which is at 1.7GB by default) with:
node --max-old-space-size=4096 node_modules/serverless/bin/serverless package to 4GB and check if it then passes with the full amount of functions.

If that works, we have to find out, where exactly the memory leak comes from and if it can be fixed by reusing objects.

[daniel-cottone]

That definitely seems to be the problem. I got much further along, looks like about 50% of the way through. If I bump it up to 12GB then the process finishes after about 8-10 minutes.

[HyperBrain]

Good to know - thanks for testing this 👍 . Can you adjust the title of the issue to reflect that this will happen with many functions? Then it's more clear how to reproduce it and we can find a solution.

Is the workaround using the increased heap ok for you as long as there's no real fix?

[daniel-cottone]

Sure thing. I think the 12GB heap size is probably a bit much; in addition to that it seems to run significantly slower than our build does currently. I'll just opt to not make use of individual packaging for now. If/when this does get fixed I can turn it on then.

[HyperBrain]

The slower runtime is expected, because it takes each webpack compile's output to determine the modules that are really needed for each function and assembles only these for the function package. That takes some time (when using --verbose you should see the exact steps including their timing).
The longer build outweighs the better startup behavior (if the lambdas are cold started) and if some big dependencies are only used by one function.

[daniel-cottone]

@HyperBrain That makes sense, thanks!