future improvement on add import #1721
Description
this feature ("add import" - #1656) adds this extra cost to analysis
-
cpu usage - indexing all library files installed once the language server started. it runs on BG so it doesn't interfere with user interactions but the user can see CPU burning while indexing happens.
-
allocations - indexing all library files cause us to parse all those library files causing us to allocate memory we didn't allocate before. (ex, about 200MB spike in memory while processing)
-
VM - we now hold index info on library files as well on top of user code index causing us to hold more in memory. (ex. about 90MB more for our testbed - anaconda + popular libraries)
...
we can improve each part as follow if we believe that is necessary.
for 1, add "persisted index" over libraries so that we only do indexing only once.
for 2, reduce allocations and add pooling.
for 3, compact index data memory footprint.