Next.js development high memory usage

[timneutkens]

Before posting a comment on this issue please read this entire post.

Previous work

The past few weeks we've been investigating / optimizing various memory usage issues. Specifically geared towards production memory usage. In investigating these we were able to find there was one memory leak in Node.js itself when using fetch() in Node.js versions before 18.17.0 (you'll want to use 18.17.1 for security patches though).

Most of the reports related to memory usage turned out to be reports of "it's higher than the previous version" rather than a memory leak. This was expected because in order to run App Router and Pages Router at the same time with different React versions two separate processes were needed. This has been resolved by reducing the amount of processes to two, one for Routing and App Router rendering, and one for Pages Router rendering. So far we haven't received new reports since the latest release.

In some issues there were reports related to Image Optimization in production, however no reproduction was provided so it could not be investigated adequately, if you have a reproduction for that please refer to this issue: #54482

New

With the memory usage in production resolved we've started investigating reports of development memory usage spikes. Unfortunately these reports suffer from the same problem as the production memory usage issues people raised before, they're full of comments saying same issue or posting screenshots of monitoring tools saying "Look, same issue".

Unfortunately, as you can image, these replies are not enough to investigate / narrow down what causes the memory usage, for example in multiple cases that we did get a reproduction and could investigate the reason for the high memory usage was:

• A bug in the application code, causing infinite looping components
• Accidental import of ~11.000 modules through icon libraries -- Yes, many icon libraries ship massive amounts of re-exports and those have to be bundled before it can be tree-shaken. We've been working on an automated way to split these up in optimize_barrel SWC transform and new optimizePackageImports config #54572 that should help a bit to reduce the size (and compilation speed too).
• Webpack customization, e.g. adding external libraries that change webpack settings, for example to write additional sourcemaps

So far I've been able to make one small change to webpack's memory caching to make it garbage collect a bit more aggressively in #54397. I'm not expecting that change to have a big impact on the reported issues though.

We'd like to investigate these reports further, however we're unable to narrow these down if there is no code to run to collect heap snapshots and profiles for, hence this issue. If you are able to please provide runnable code of what you're experiencing.

Comments that don't include runnable code will be automatically hidden in order to keep this issue productive. This includes comments that only have a screenshot and applications that can't run.

I'm going to combine the other reports into this issue as separate comments.

I've made sure that we have 2-3 engineers on our team available to investigate when we get runnable reproductions to investigate.

Thanks in advance!

_NEXT-1569

[tomelliot]

Thanks for the attention on this topic @timneutkens.
I've been experiencing this issue pretty consistently in dev. I'm not sure I'll be able to create a reproduction for it - it seems more to do with the hot-reload operation than with any specific code I'm writing. Is there a way for me to collect some relevant debug info locally until I run into the issue again, and share that instead?

[tomelliot]

I understand your frustration with all the "same" comments, but I'm sorry to say I'm doing my best to be helpful here @timneutkens. I'm not an experienced enough developer to get to the bottom of this myself, and I was just offering the best I could to try and support by collecting data locally.

I'll leave it to you and the pros.

[AhmedChabayta]

i was building an app using the app directory, and i kept getting server out of memory in console, and some sort of input error in the browser, the server crashes and thats that, no errors beyond the ones mentioned, furthermore, the app was slow in loading, i created another app using pages this time, kept everything exactly the same just made it the proper pages format, and all the issues have gone. no more server out of memory, nor slow loading times. Next.js is and most probably will remain my personal favorite react framework, but this shit is ridiculous. maybe we get an explanation at least to why this is happening? i spent f** hours trying to find out what i did wrong, only to realize the problem isnt from my code.

i updated Nextjs to the latest version, updated node, used different versions of node via nvm, removed my code block by block to see if the issue is coming from some broken piece of code, same shit. the only solution was to move on from the app directory.

[timneutkens]

maybe we get an explanation at least to why this is happening?

It's impossible to say what you're running into without code being shared. It could be you import a library in both server components and client components which means it's bundled twice compared to pages and such. Given that you have a reproduction please share it (doesn't have to be public, you can give me / my team access). As said before we'd be happy to take a look.

[AhmedChabayta]

https://github.com/AhmedChabayta/social.me

if you can tell me there's something wrong with my code, id forever appreciate it for helping me move on from this issue.

thank you in any case.

[timneutkens]

@AhmedChabayta It looks like I'll need a Sanity key in order to run the project, can you provide that?

Also do you have a branch with the change to use pages instead? Just so that I can compare them.

[AhmedChabayta]

@timneutkens is it possible i send the keys in private? and no, i made a whole new repo locally to test on pages. i sincerely appreciate your help. thank u

[timneutkens]

@AhmedChabayta feel free to send them on https://twitter.com/timneutkens.

i made a whole new repo locally to test on pages. i sincerely appreciate your help. thank u

Can you share that repository too? Want to make sure I can compare them.

[AhmedChabayta]

@timneutkens i should add this, i tried removing the sidebar all together, in case the icons might be the ones causing the issue, but the server still kept crashing anyway.
i just sent the links to the 2 repos, and the whole bunch of keys i used in both apps.

[timneutkens]

Thanks! Added both to our internal copy of this issue so that your keys don't leak. We'll take a look!

[noetix]

Its interesting that the downvote emojis on the bullying post have been removed.

All the dude did was ask how to collect data in a way that would be helpful. If collecting our own data is out of the question then that should have been the answer.

Poor form for someone representing Vercel.