Size of assets-graph.json is too large

[svarl]

We have a big project (assets_graph.json ~254Mb). Each incremental build takes additional 20-30 seconds for asset graph serialization. I think it's too much. How can I speed up incremental builds for this project? Maybe it's better to save asset graph to a binary file (not json)?

Dart VM version: 2.7.2 

build 1.2.2
build_resolvers 1.2.1
build_modules 2.8.1
build_web_compilers 2.9.0
build_runner 1.7.4
build_config 0.4.2

[jakemac53]

Closing as a dupe of #787 and #41.

We have not had a strong motivation to fix this in the past, although this does seem quite excessive in your case.

However, with a project of this size what are your total build times like?

While there are things that could be done here my guess would be that this has more to do with project structure causing a huge number of edges between nodes. For instance if you have a single very large package we are not very efficient today in how this is handled due to the constraints of the build system and the compiler, especially if you have all your files under lib and not lib/src.

[svarl]

A first build takes 10-12 minutes. An incremental build takes ~1-1.5 minutes. We have migrated the project on webdev from our custom builder (build_runner + some optimizations). Before migration some rebuild takes 30-40s and size of assets-graph.json was 120Mb. Size of asset-graph.json has been increased because some post process was added by build_runner.

Here is the list of added post processors:

build_web_compilers:dart2js_modules 
source_gen:part_cleanup
build_modules:module_cleanup
build_web_compilers:sdk_js_cleanup
build_web_compilers:dart2js_archive_extractor
build_web_compilers:dart_source_cleanup
less_dart:less_source_cleanup
angular:placeholder_cleanup
angular:component_source_cleanup 

[jakemac53]

The only one of those builders that I would expect to create any large number of edges would be dart2js_modules, but even that should not be super large.

If your optimizations listed were around the resolver, that most likely was the reason for the smaller size before. The default (safe) resolver adds a huge number of edges (N^2 worst case).

[svarl]

Thank you for the answers

[natebosch]

@svarl - if you have a chance, could you try adding a dependency override for build_runner_core and use the msgpack branch? I'm curious if switching from json to msgpack would help your project, and by how much.

#2779

[svarl]

@natebosch thx you. I'll try it in a couple of days

[svarl]

I have tested it on the our project. It become better and here is the result:

Before:
[INFO] Caching finalized dependency graph completed, took 23.8s
[INFO] Succeeded after 59.2s with 12 outputs (13 actions)
Size of asset_graph.json is 242M

After (msgpack branch):
[INFO] Caching finalized dependency graph completed, took 15.2s
[INFO] Succeeded after 44.6s with 12 outputs (13 actions)
Size of asset_graph.json is 188M