Process aborts with 'out of memory' when using 2.0.0

[nick]

• Operating System: OSX 10.14.6
• Node Version: 10.16.0
• NPM Version: 6.9.0
• webpack Version: 4.39.3
• terser-webpack-plugin Version: 2.0.0

Expected Behavior

Process does not abort

Actual Behavior

$ NODE_ENV=production ./node_modules/.bin/webpack --loglevel notice

<--- Last few GCs --->

[84294:0x102843000]    55749 ms: Mark-sweep 1312.4 (1444.3) -> 1305.0 (1446.3) MB, 622.0 / 0.0 ms  (average mu = 0.099, current mu = 0.040) allocation failure scavenge might not succeed
[84294:0x102843000]    56388 ms: Mark-sweep 1315.7 (1446.3) -> 1308.6 (1448.8) MB, 613.1 / 0.0 ms  (average mu = 0.070, current mu = 0.040) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x3aa9ebf5be3d]
Security context: 0x37c8eea1e6e9 <JSObject>
    1: /* anonymous */(aka /* anonymous */) [0x37c895d8d969] [/Users/nick/Projects/origin/node_modules/terser-webpack-plugin/node_modules/webpack-sources/lib/applySourceMap.js:~58] [pc=0x3aa9ed4fe2c4](this=0x37c83a5026f1 <undefined>,chunk=0x37c803a7d979 <String[14]: createElement(>,middleMapping=0x37c8171046d9 <Object map = 0x37c835e45c79>)
    2: SourceNode...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10003cf99 node::Abort() [/usr/local/bin/node]
 2: 0x10003d1a3 node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0x1001b7835 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0x100585682 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 5: 0x100588155 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/usr/local/bin/node]
 6: 0x100583fff v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 7: 0x1005821d4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
 8: 0x10058ea6c v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 9: 0x10058eaef v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
10: 0x10055e434 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/usr/local/bin/node]
11: 0x1007e6714 v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
12: 0x3aa9ebf5be3d 
13: 0x3aa9ed4fe2c4 
14: 0x3aa9ed4b7d28 
Abort trap: 6

Code

https://github.com/OriginProtocol/origin/blob/master/dapps/marketplace/webpack.config.js

How Do We Reproduce?

git clone https://github.com/OriginProtocol/origin.git
cd origin
# Update dapps/marketplace/package.json to use v2.0.0 of terser plugin
yarn
cd dapps/marketplace
yarn build

[alexander-akait]

Thanks for issue, investigate

[alexander-akait]

Just information, change this line to:

"build:js": "NODE_ENV=production webpack --progress --loglevel notice",

and see what happens.

Error appears on source map generation step:

93% after chunk asset optimization SourceMapDevToolPlugin app.9949447c.js generate SourceMap`

Use sourceMap: false fix problem, increase memory for node also fix problem. I keep investigation.

[alexander-akait]

Looks terser leaking

[alexander-akait]

Why problem appears:
Here we change logic 8b88b39

Before for only one file we run new thread for uglification, now we use same thread for this. Removing tasks.length > 1 from https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L48 fix problem. Why we do this - try to reduce memory and cpu usage when you have only one file (creating new thread takes more memory and increase cpu loading).

But another problem occurred - memory leak. terser something doesn’t clear therefore the current thread began to consume more memory.

Simple investigation, i used this code:

const used = process.memoryUsage().heapUsed / 1024 / 1024;
console.log(`The script uses approximately ${Math.round(used * 100) / 100} MB`);

and put this lines after https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/index.js#L272

New version (we don't create new thread for one file):

The script uses approximately 1016.37 MB

Old version (new thread for one file):

The script uses approximately 362.56 MB

Around 700 MB leaking

/cc @fabiosantoscode

[alexander-akait]

/cc @nick also cache is not working with your code, because you always generate something new in code, don't use:

BUILD_TIMESTAMP: +new Date()

Caches is not working in your app, long cache term also

[alexander-akait]

@nick try to use https://github.com/webpack-contrib/terser-webpack-plugin/releases/tag/v2.0.1, anyway we need investigate why terser consume a lot of memory

Let's wait @fabiosantoscode answer

[nick]

Thanks for looking this so quickly, @evilebottnawi - and thanks for the tip on the cache issue 😊

Can confirm 2.0.1 works as before. Feel free to close unless you want to keep it open for the memory issue.

[fabiosantoscode]

Interesting. I'll try to come up with the exact tag where this started to happen.

[fabiosantoscode]

But I kind of have something else I have to fix first, though.

[alexander-akait]

@fabiosantoscode 👍 yep, feel free to ping me i can help you with memory leak searching

[fabiosantoscode]

I tried creating a file that attempts to encrypt a lot of things to try and create a memory leak, but I can't reproduce this at all. I looked into the commit that fixes this. Isn't this just a 2.0.0 specific problem?

[alexander-akait]

@fabiosantoscode no, we just run terser in separate thread instead using currently, but looks terser have memory leak, because no problem in parallel mode, but problem when you use only one thread

[fabiosantoscode]

Can this be reproduced with the default options? The Terser tests aren't leaking memory at all.

[eliseumds]

In my experience, there hasn't been any significant difference in memory usage. Both V1 and V2 are hovering around 500MB of memory (and same compilation time) for one of our smaller apps. Measured with NODE_ENV=production time -l ... on MacOS. The dependencies are basically the same (Node, Webpack, etc).