varinfo runs out of memory

[maartenvd]

this is on julia 1.6.2. If I call

using MPSKit;
varinfo(all=true,imported=true,recursive=true)

my laptop runs out of memory. Is this not supposed to work? I have an issue where julia takes and enormous amount of memory, and I want to inspect where that is being used for.

If I modify varinfo to spit out its intermediary results, I see lines appearing like

MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.BLAS.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.BLAS.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.BLAS.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.BLAS.LinearAlgebra.BLAS.LinearAlgebra.BLAS.LinearAlgebra.#120#122

[Moelf]

It also runs out of memory for me, but I think that's what you would get for all=true,imported=true,recursive=true

[Seelengrab]

Are you sure that you want all=true of module Main? Sounds like you just want varinfo(MPSKit; all=true, recursive=true), since you can't necessarily do anything about imported objects and only really care about things caused/existing in MPSKit.

[maartenvd]

I'm trying to debug a script that allocates 40 gb, and only one of its dependencies is mpskit. That package does not cache anything, but the packages it depends on do tend to do that (though manually inspecting those caches only shows storage of the order 100mb). If an imported package does something weird in memory, then I want to know which it is, so that I can change it or create a pull request.

Then I figured I just want the total size of all objects in memory, roughly categorized per module, which lead me to all=true,imported=true,recursive=true, and I don't understand

• why that call is problematic. I want non-exported objects. I want objects from imported modules, and I want objects in sub-modules within those imported pacakges.

• where statements like MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra come from. MPSKit has no sub-namespace MPSKit.

[Seelengrab]

I'm trying to debug a script that allocates 40 gb

Sounds like you don't really want to investigate all live objects at runtime but whant to find out where those allocations are coming from, in which case the Profile stdlib (maybe combined with ProfileView.jl) is what you probably want to take a look at. If you can't run profiling code on your full script because it takes too long, run it on a smaller test case instead of your full simulation.

varinfo is also not really suited to investigating high memory usage - it's a function meant for interactive use and it's constructing the returned table in full, before returning. It's not directly printing it to the screen.

why that call is problematic.

When doing this full-on call on Main, you're asking for all infos about all things julia knows about and ask it to generate a summary on that. That's a lot of data (how much RAM do you have left over after your script runs?). This will also create A LOT of output, which you probably don't want. For example, in a freshly started julia session (nothing imported yet), you get this:

julia> @time varinfo(;all=true,recursive=true,imported=true)                                                                                       
  0.745999 seconds (1.28 M allocations: 70.874 MiB, 2.33% gc time, 19.06% compilation time)                                                        
  name                                                              size summary                                                                   
  –––––––––––––––––––––––––––––––––––––––––––––––––––––––––– ––––––––––– ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– 
  ##meta#51                                                    1.464 KiB IdDict{Any, Any} with 1 entry                                             
  Base                                                                   Module                                                                    
  Core                                                                   Module                                                                    
  InteractiveUtils                                           257.933 KiB Module                                                                    
  InteractiveUtils.##code_llvm#29                              128 bytes DataType                                                                  
  InteractiveUtils.##code_llvm#30                              128 bytes DataType                                                                  
  InteractiveUtils.##code_native#31                            128 bytes DataType                                                                  
  InteractiveUtils.##code_native#32                            128 bytes DataType                                                                  
  InteractiveUtils.##code_warntype#26                          128 bytes DataType                                                                  
  InteractiveUtils.##code_warntype#28                          128 bytes DataType                                                                  
  InteractiveUtils.##define_editor#1                           128 bytes DataType                                                                  
  InteractiveUtils.##meta#51                                  40.130 KiB IdDict{Any, Any} with 26 entries                                          
  InteractiveUtils.##methodswith#70                            128 bytes DataType                                                                  
  InteractiveUtils.##methodswith#73                            128 bytes DataType                                                                  
  InteractiveUtils.##methodswith#74                            128 bytes DataType                                                                  
  InteractiveUtils.##peakflops#75                              128 bytes DataType                                                                  
  InteractiveUtils.##separate_kwargs#33                        128 bytes DataType                                                                  
  InteractiveUtils.##varinfo#50                                128 bytes DataType                                                                  
  InteractiveUtils.##varinfo#60                                128 bytes DataType                                                                  
  InteractiveUtils.##versioninfo#61                            128 bytes DataType                                                                  
  InteractiveUtils.#10#20                                      128 bytes DataType                                                                  
  InteractiveUtils.#11#21                                      128 bytes DataType                                                                  
  InteractiveUtils.#12#22                                      128 bytes DataType                                                                  
  InteractiveUtils.#13#23                                      128 bytes DataType                                                                  
  InteractiveUtils.#14#24                                      128 bytes DataType                                                                  
  InteractiveUtils.#15#25                                      128 bytes DataType                                                                  
  InteractiveUtils.#2#3                                        120 bytes UnionAll                                                                  
  InteractiveUtils.#34#39                                      128 bytes DataType                                                                  
  InteractiveUtils.#35#40                                      128 bytes DataType                                                                  
  InteractiveUtils.#36#41                                      128 bytes DataType                                                                  
  InteractiveUtils.#37#42                                      128 bytes DataType                                                                  
  InteractiveUtils.#38#43                                      128 bytes DataType                                                                  
  InteractiveUtils.#4#5                                         40 bytes UnionAll                                                                  
  InteractiveUtils.#45#47                                       40 bytes UnionAll                                                                  
  InteractiveUtils.#51#56                                       80 bytes UnionAll                                                                  
  InteractiveUtils.#52#57                                      120 bytes UnionAll                                                                  
  InteractiveUtils.#53#58                                      144 bytes DataType                                                                  
  InteractiveUtils.#54#59                                      128 bytes DataType                                                                  
  InteractiveUtils.#6#16                                       128 bytes DataType                                                                  
  InteractiveUtils.#62#66                                      128 bytes DataType                                                                  
  InteractiveUtils.#63#67                                      128 bytes DataType                                                                  
  InteractiveUtils.#64#68                                      128 bytes DataType                                                                  
  InteractiveUtils.#65#69                                      128 bytes DataType                                                                  
  InteractiveUtils.#7#17                                       128 bytes DataType                                                                  
  InteractiveUtils.#71#72                                      120 bytes UnionAll                                                                  
  InteractiveUtils.#76#77                                       80 bytes UnionAll                                                                  
  InteractiveUtils.#8#18                                       128 bytes DataType                                                                  
  InteractiveUtils.#9#19                                       128 bytes DataType                                                                  
  InteractiveUtils.#@code_llvm                                 128 bytes DataType                                                                  
  InteractiveUtils.#@code_lowered                              128 bytes DataType                                                                  
  InteractiveUtils.#@code_native                               128 bytes DataType                                                                  
  InteractiveUtils.#@code_typed                                128 bytes DataType                                                                  
  InteractiveUtils.#@code_warntype                             128 bytes DataType                                                                  
  InteractiveUtils.#@edit                                      128 bytes DataType                                                                  
  InteractiveUtils.#@functionloc                               128 bytes DataType                                                                  
  InteractiveUtils.#@less                                      128 bytes DataType                                                                  
  InteractiveUtils.#@which                                     128 bytes DataType                                                                  
  InteractiveUtils.#_dump_function                             128 bytes DataType                                                                  
  InteractiveUtils.#_dump_function_linfo_llvm                  128 bytes DataType                                                                  
  InteractiveUtils.#_dump_function_linfo_native                128 bytes DataType                                                                  
  InteractiveUtils.#_methodswith                               128 bytes DataType                                                                  
  InteractiveUtils.#_populate_rows#55                          160 bytes UnionAll                                                                  
  InteractiveUtils.#_subtypes                                  128 bytes DataType                                                                  
  InteractiveUtils.#_subtypes_in                               128 bytes DataType                                                                  
  InteractiveUtils.#clipboard                                  128 bytes DataType                                                                  
  InteractiveUtils.#clipboardcmd                               128 bytes DataType                                                                  
  InteractiveUtils.#code_llvm                                  128 bytes DataType                                                                  
  InteractiveUtils.#code_llvm##kw                              128 bytes DataType                                                                  
  InteractiveUtils.#code_llvm#29                                 0 bytes #code_llvm#29 (generic function with 1 method)                            
  InteractiveUtils.#code_llvm#30                                 0 bytes #code_llvm#30 (generic function with 1 method)                            
  InteractiveUtils.#code_native                                128 bytes DataType                                                                  
  InteractiveUtils.#code_native##kw                            128 bytes DataType                                                                  
  InteractiveUtils.#code_native#31                               0 bytes #code_native#31 (generic function with 1 method)                          
  InteractiveUtils.#code_native#32                               0 bytes #code_native#32 (generic function with 1 method)                          
  InteractiveUtils.#code_warntype                              128 bytes DataType                                                                  
  InteractiveUtils.#code_warntype##kw                          128 bytes DataType                                                                  
  InteractiveUtils.#code_warntype#26                             0 bytes #code_warntype#26 (generic function with 1 method)                        
  InteractiveUtils.#code_warntype#28                             0 bytes #code_warntype#28 (generic function with 1 method)                        
  InteractiveUtils.#define_default_editors                     128 bytes DataType                                                                  
  InteractiveUtils.#define_editor                              128 bytes DataType                                                                  
  InteractiveUtils.#define_editor##kw                          128 bytes DataType                                                                  
  InteractiveUtils.#define_editor#1                              0 bytes #define_editor#1 (generic function with 1 method)                         
  InteractiveUtils.#directsubtype                              128 bytes DataType                                                                  
  InteractiveUtils.#dumpsubtypes                               128 bytes DataType                                                                  
  InteractiveUtils.#dumptype                                   128 bytes DataType                                                                  
  InteractiveUtils.#edit                                       128 bytes DataType                                                                  
  InteractiveUtils.#editor                                     128 bytes DataType                                                                  
  InteractiveUtils.#editor_matches                             128 bytes DataType                                                                  
  InteractiveUtils.#eval                                       128 bytes DataType                                                                  
  InteractiveUtils.#gen_call_with_extracted_types              128 bytes DataType                                                                  
  InteractiveUtils.#gen_call_with_extracted_types_and_kwargs   128 bytes DataType                                                                  
  InteractiveUtils.#include                                    128 bytes DataType                                                                  
  InteractiveUtils.#less                                       128 bytes DataType                                                                  
  InteractiveUtils.#methodswith                                128 bytes DataType                                                                  
  InteractiveUtils.#methodswith##kw                            128 bytes DataType                                                                  
  InteractiveUtils.#methodswith#70                               0 bytes #methodswith#70 (generic function with 1 method)                          
  InteractiveUtils.#methodswith#73                               0 bytes #methodswith#73 (generic function with 1 method)                          
  InteractiveUtils.#methodswith#74                               0 bytes #methodswith#74 (generic function with 1 method)                          
  InteractiveUtils.#peakflops                                  128 bytes DataType                                                                  
  InteractiveUtils.#peakflops##kw                              128 bytes DataType                                                                  
  InteractiveUtils.#peakflops#75                                 0 bytes #peakflops#75 (generic function with 1 method)                            
  InteractiveUtils.#print_highlighted#27                       128 bytes DataType                                                                  
  InteractiveUtils.#print_llvm                                 128 bytes DataType                                                                  
  InteractiveUtils.#print_llvm_operand                         128 bytes DataType                                                                  
  InteractiveUtils.#print_llvm_operands                        128 bytes DataType                                                                  
  InteractiveUtils.#print_llvm_tokens                          128 bytes DataType                                                                  
  InteractiveUtils.#print_native                               128 bytes DataType                                                                  
  InteractiveUtils.#print_native_tokens                        128 bytes DataType                                                                  
  InteractiveUtils.#printstyled_ll                             128 bytes DataType                                                                  
  InteractiveUtils.#recursive_dotcalls!                        128 bytes DataType                                                                  
  InteractiveUtils.#report_bug                                 128 bytes DataType                                                                  
  InteractiveUtils.#separate_kwargs                            128 bytes DataType                                                                  
  InteractiveUtils.#separate_kwargs##kw                        128 bytes DataType                                                                  
  InteractiveUtils.#separate_kwargs#33                           0 bytes #separate_kwargs#33 (generic function with 1 method)                      
  InteractiveUtils.#subtypes                                   128 bytes DataType                                                                  
  InteractiveUtils.#supertypes                                 128 bytes DataType                                                                  
  InteractiveUtils.#sys_arch_category                          128 bytes DataType                                                                  
  InteractiveUtils.#type_close_enough                          128 bytes DataType                                                                  
  InteractiveUtils.#varinfo                                    128 bytes DataType                                                                  
  InteractiveUtils.#varinfo##kw                                128 bytes DataType                                                                  
  InteractiveUtils.#varinfo#50                                   0 bytes #varinfo#50 (generic function with 1 method)                              
  InteractiveUtils.#varinfo#60                                   0 bytes #varinfo#60 (generic function with 1 method)                              
  InteractiveUtils.#versioninfo                                128 bytes DataType                                                                  
  InteractiveUtils.#versioninfo##kw                            128 bytes DataType                                                                  
  InteractiveUtils.#versioninfo#61                               0 bytes #versioninfo#61 (generic function with 1 method)                          
  InteractiveUtils.#warntype_type_printer                      128 bytes DataType                                                                  
  InteractiveUtils.@code_llvm                                    0 bytes @code_llvm (macro with 1 method)                                          
  InteractiveUtils.@code_lowered                                 0 bytes @code_lowered (macro with 1 method)                                       
  InteractiveUtils.@code_native                                  0 bytes @code_native (macro with 1 method)                                        
  InteractiveUtils.@code_typed                                   0 bytes @code_typed (macro with 1 method)                                         
  InteractiveUtils.@code_warntype                                0 bytes @code_warntype (macro with 1 method)                                      
  InteractiveUtils.@edit                                         0 bytes @edit (macro with 1 method)                                               
  InteractiveUtils.@functionloc                                  0 bytes @functionloc (macro with 1 method)                                        
  InteractiveUtils.@less                                         0 bytes @less (macro with 1 method)                                               
  InteractiveUtils.@which                                        0 bytes @which (macro with 2 methods)                                             
  InteractiveUtils.CodegenParams                               192 bytes DataType                                                                  
  InteractiveUtils.EDITOR_CALLBACKS                            1.114 KiB 7-element Vector{Function}                                                
  InteractiveUtils._clipboard_copy                             840 bytes Dict{Symbol, Cmd} with 2 entries                                          
  InteractiveUtils._clipboard_paste                            840 bytes Dict{Symbol, Cmd} with 2 entries                                          
  InteractiveUtils._clipboardcmd                                 0 bytes Nothing                                                                   
  InteractiveUtils._dump_function                                0 bytes _dump_function (generic function with 2 methods)                          
  InteractiveUtils._dump_function_linfo_llvm                     0 bytes _dump_function_linfo_llvm (generic function with 1 method)                
  InteractiveUtils._dump_function_linfo_native                   0 bytes _dump_function_linfo_native (generic function with 1 method)              
  InteractiveUtils._methodswith                                  0 bytes _methodswith (generic function with 1 method)                             
  InteractiveUtils._subtypes                                     0 bytes _subtypes (generic function with 3 methods)                               
  InteractiveUtils._subtypes_in                                  0 bytes _subtypes_in (generic function with 1 method)                             
  InteractiveUtils.apropos                                       0 bytes apropos (generic function with 3 methods)                                 
  InteractiveUtils.arm_cond                                     88 bytes Regex                                                                     
  InteractiveUtils.arm_keywords                                 65 bytes Regex                                                                     
  InteractiveUtils.avx512flags                                  63 bytes Regex                                                                     
  InteractiveUtils.clipboard                                     0 bytes clipboard (generic function with 2 methods)                               
  InteractiveUtils.clipboardcmd                                  0 bytes clipboardcmd (generic function with 1 method)                             
  InteractiveUtils.code_llvm                                     0 bytes code_llvm (generic function with 8 methods)                               
  InteractiveUtils.code_native                                   0 bytes code_native (generic function with 5 methods)                             
  InteractiveUtils.code_warntype                                 0 bytes code_warntype (generic function with 2 methods)                           
  InteractiveUtils.define_default_editors                        0 bytes define_default_editors (generic function with 1 method)                   
  InteractiveUtils.define_editor                                 0 bytes define_editor (generic function with 1 method)                            
  InteractiveUtils.directsubtype                                 0 bytes directsubtype (generic function with 4 methods)                           
  InteractiveUtils.dumpsubtypes                                  0 bytes dumpsubtypes (generic function with 1 method)                             
  InteractiveUtils.dumptype                                      0 bytes dumptype (generic function with 1 method)                                 
  InteractiveUtils.edit                                          0 bytes edit (generic function with 8 methods)                                    
  InteractiveUtils.editor                                        0 bytes editor (generic function with 1 method)                                   
  InteractiveUtils.editor_matches                                0 bytes editor_matches (generic function with 3 methods)                          
  InteractiveUtils.eval                                          0 bytes eval (generic function with 1 method)                                     
  InteractiveUtils.gen_call_with_extracted_types                 0 bytes gen_call_with_extracted_types (generic function with 2 methods)           
  InteractiveUtils.gen_call_with_extracted_types_and_kwargs      0 bytes gen_call_with_extracted_types_and_kwargs (generic function with 1 method) 
  InteractiveUtils.highlighting                                344 bytes Dict{Symbol, Bool} with 3 entries                                         
  InteractiveUtils.include                                       0 bytes include (generic function with 2 methods)                                 
  InteractiveUtils.insert!                                       0 bytes insert! (generic function with 2 methods)                                 
  InteractiveUtils.less                                          0 bytes less (generic function with 5 methods)                                    
  InteractiveUtils.llstyle                                     616 bytes Dict{Symbol, Tuple{Bool, Union{Int64, Symbol}}} with 10 entries           
  InteractiveUtils.llvm_cond                                    75 bytes Regex                                                                     
  InteractiveUtils.llvm_types                                  114 bytes Regex                                                                     
  InteractiveUtils.methodswith                                   0 bytes methodswith (generic function with 4 methods)                             
  InteractiveUtils.num_regex                                   100 bytes Regex                                                                     
  InteractiveUtils.peakflops                                     0 bytes peakflops (generic function with 2 methods)                               
  InteractiveUtils.print_llvm                                    0 bytes print_llvm (generic function with 1 method)                               
  InteractiveUtils.print_llvm_operand                            0 bytes print_llvm_operand (generic function with 1 method)                       
  InteractiveUtils.print_llvm_operands                           0 bytes print_llvm_operands (generic function with 1 method)                      
  InteractiveUtils.print_llvm_tokens                             0 bytes print_llvm_tokens (generic function with 1 method)                        
  InteractiveUtils.print_native                                  0 bytes print_native (generic function with 2 methods)                            
  InteractiveUtils.print_native_tokens                           0 bytes print_native_tokens (generic function with 2 methods)                     
  InteractiveUtils.printstyled_ll                                0 bytes printstyled_ll (generic function with 2 methods)                          
  InteractiveUtils.recursive_dotcalls!                           0 bytes recursive_dotcalls! (generic function with 2 methods)                     
  InteractiveUtils.report_bug                                    0 bytes report_bug (generic function with 1 method)                               
  InteractiveUtils.separate_kwargs                               0 bytes separate_kwargs (generic function with 1 method)                          
  InteractiveUtils.subtypes                                      0 bytes subtypes (generic function with 2 methods)                                
  InteractiveUtils.supertypes                                    0 bytes supertypes (generic function with 1 method)                               
  InteractiveUtils.sys_arch_category                             0 bytes sys_arch_category (generic function with 1 method)                        
  InteractiveUtils.type_close_enough                             0 bytes type_close_enough (generic function with 1 method)                        
  InteractiveUtils.typesof                                       0 bytes typesof (generic function with 1 method)                                  
  InteractiveUtils.varinfo                                       0 bytes varinfo (generic function with 4 methods)                                 
  InteractiveUtils.versioninfo                                   0 bytes versioninfo (generic function with 2 methods)                             
  InteractiveUtils.warntype_type_printer                         0 bytes warntype_type_printer (generic function with 1 method)                    
  InteractiveUtils.x86_ptr                                      75 bytes Regex                                                                     
  Main                                                                   Module                                                                    
  eval                                                           0 bytes eval (generic function with 1 method)                                     
  include                                                        0 bytes include (generic function with 2 methods)                                 

which is probably already more than you bargained for. Now imagine that but with a bunch of included modules, their dependencies, their dependencies' dependencies...

---

These kinds of questions about where allocations are coming from are usually better suited to first ask on https://discourse.julialang.org and only after a problem in julia itself has been identified should an issue be opened.

[maartenvd]

The primary question is where MPSKit.MPSKit.LinearAlgebra.MPSKit.MPSKit.LinearAlgebra.* comes from. I don't understand how that is even a correct module. LinearAlgebra does not have a .MPSKit field. If that is not a bug, then I will just close this issue.

[Moelf]

that probably is not meant to be read like a normal Julia code. Can't help without knowing what produced that

[maartenvd]

It's the output of varinfo. varinfo internally populates a list, and then directly returns it. These are the elements of the list.

[Seelengrab]

No, it's definitely meant to be read like julia code. That's supposed to be an import chain. varinfo is allocating left and right, recursively, making it sloooow. I'll see if I can speed it up.

Part of the problem is that Markdown.Table will only accept Vector{Vector{Any}} - so I can't just throw in something else :/

[maartenvd]

I can look at it tomorrow as well. I really suspect there's something flawed going on, where it recursively iterates over modules that it already went over. I know it'll be a lot of data, and perhaps not the most optimal tool, but I don't understand why it would fail outright. I mean, I have 120gb on that system, and I cannot do "using mpskit; varinfo(...)".

Going at it the profileview kind of way is also difficult, as it certainly allocates a lot in tight loops. I don't mind it using a lot of memory there, but after all is said and done, and the simulation is finnished, the remaining data should be rather small, and it isnt.

[vtjnash]

It sounds like it is having trouble recursing over the module graph, and not correctly filtering out modules that are not submodules

[Seelengrab]

Agreed - the code looks like it only tries to check if the child isn't the same as the parent, but ignores whether the child ever occured before?

I have a small lazy unique iterator lying around from a different project (which wouldn't work at all without that), so maybe using something like that could help? Combined with something that properly walks that tree, it may be helpful. Carries around a history if its state though in form of a list of hashes. In my other application I have it walk over a tree to produce unique nodes and I was surprised something like this wasn't already in Base.

lazy unique iterator, feel free to use, MIT licensed

struct UniqueIterator{T,F}                                                           
    itr::T                                                                           
    by::F                                                                            
    UniqueIterator(itr::T, f::F) where {T,F} = new{T,F}(itr, f)                      
end                                                                                  
                                                                                     
function Base.iterate(itr::UniqueIterator)                                           
    t = iterate(itr.itr)                                                             
    t === nothing && return nothing                                                  
    el, state = t                                                                    
    (el, (state, [hash(itr.by(el))]))                                                
end                                                                                  
                                                                                     
function Base.iterate(itr::UniqueIterator, (state, cache))                           
    local el, state, h                                                               
    while true                                                                       
        t = iterate(itr.itr, state)                                                  
        t === nothing && return nothing                                              
        el, state = t                                                                
        h = hash(itr.by(el))                                                         
        h ∉ cache && break                                                           
        yield() # so we don't hog ressources in case we never return
    end                                                                              
    push!(cache, h)                                                                  
    return el, (state, cache)                                                        
end                                                                                  
Base.IteratorEltype(::Type{UniqueIterator{T,F}}) where {T,F} = Base.IteratorEltype(T)
Base.eltype(::Type{UniqueIterator{T,F}}) where {T,F} = eltype(T)                     
Base.IteratorSize(::Type{UniqueIterator{T,F}}) where {T,F} = Base.SizeUnknown() # we don't know how many unique elements there are
      
iunique(itr; by=identity) = UniqueIterator(itr, by)                                  
iunique(itr...; by=identity) = UniqueIterator(Iterators.flatten(itr), by)

[maartenvd]

a possible fix is this:

if recursive
            for row in newrows
                if row[3] == "Module" && 
					!in(split(row[1], ".")[end], ["Base", "Main", "Core"]) &&
					!in(string(getfield(m2, Symbol(split(row[1], ".")[end]))),first.(allrows)) 
					
					row[1] = string(getfield(m2, Symbol(split(row[1], ".")[end])));
					push!(allrows,row);
                    _populate_rows(getfield(m2, Symbol(split(row[1], ".")[end])), allrows, false)
				else
					row[1] = "$(m2).$(row[1])"
					push!(allrows,row)
                end
            end
        end

where we just alltogether do away with prep in _populate_rows

this is a very quick'n dirty hacky-ish approach though
derp.txt