Add ability to sample the heapsnapshot

[NHDaly]

We've seen two problematic scenarios with the heapsnapshots:

1. We OOM while collecting the snapshot! 🤦
   • A short poem, to represent this case: the memory usage was high, we wondered "why," we asked for a snapshot from the cloud (in the sky), and so we saw our server die. 😢
2. We successfully downloaded the snapshot (which was 7GiB big! 😳), and then the Chrome devtools errors trying to open it.
   • It complains that we've exceeded the max size of a JS string 😑

SO we'd like the ability to sample a heapsnapshot, taking only some edges and/or only some nodes, possibly weighting the sampling by the number of bytes.

[NHDaly]

For problem 1, we also thought that streaming the snapshot could help (rather than building it all up in memory then dumping it), but that won't solve problem 2.

[ArchieCall]

I believe this is your answer:

In the realm of code, where logic intertwines,
There's a snapshot we dread, a perilous sign.
Heap Snapshot, they call it, a tempting pursuit,
But beware, my friend, it can be a bitter fruit.

In the labyrinth of memory, it delves so deep,
To capture each object, in slumber or leap.
But with great power comes a burden, you see,
For it can reveal what you'd rather not be.

When you take a Heap Snapshot, be cautious, take heed,
For it might unveil a performance misdeed.
A tangled web of references, lurking in disguise,
Leaking memory silently, to your program's demise.

It exposes the secrets of your memory's dance,
Unused objects lurking, not given a chance.
To be properly disposed, to free up some space,
Heap Snapshot reveals their hiding place.

So, heed this warning, in your coding quest,
Heap Snapshots can put your skills to the test.
Use them with care, and wisdom embrace,
For they may reveal errors in your code's grace.

In the world of programming, it's understood,
A Heap Snapshot can be both bad and good.
But with vigilance and knowledge, you'll find your way,
To cleaner, faster code, where errors decay.

[ArchieCall]

Just having fun with ChatGPT. On the serious side, I've been using it to look up Julia documentation.

[NHDaly]

Haha i was just teasing - i wasn't meaning to 👎 your post, just the quality of the poem from chatGPT 😉
Sorry it's hard to joke over text 😳

[ArchieCall]

Does RelationalAI have something similar to look up Julia doc?

[vilterp]

One possible approach would be to build the dominator tree (spanning tree) of the full profile, then filter that… I.e. for each child, take only the top N nodes by reachable size, or something.

This would be tricky to implement in a memory constrained fashion, but would ensure that you don't miss the most significant objects…

[jpsamaroo]

Another option: make the snapshot configurable to only capture objects of a certain type, along with some metric to determine which connected objects to capture. This could entail adding a user-programmable C callback that is queried with each traversed object and returns a bool indicating whether the snapshotter should capture and recurse into said object.

[NHDaly]

@Drvi and I think we came up with something that will work.. verifying it first

[NHDaly]

@jpsamaroo i do like that option too! That's a neat idea.

The solution that we put together, which seems to be working for us, is to stream the snapshot out of the process, and then downsample it via post-processing. This gives the greatest possible flexibility to how you sample (you can even downsample the same snapshot lots of different ways to get different views on the data), while preventing OOMing when recording the snapshot.

PR in progress, here: #51518