Consolidate internal caches

[denravonska]

There are a lot of caches in the code designed to map between CPID metadata, blocks, projects etc. Some data such as block hashes and CPID strings are duplicated among these caches, causing unnecessary memory waste. There should be a more clever way of storing this data so that we can still cache without wasting too much RAM.

• Move relevant CPID data into StructCPID
• Implement a StructCPID pool
• Use pointers to CPID structs instead of duplicating them
• Traverse relevant caches when blocks are detached

[denravonska]

I was looking into replacing some variables in StructCPID with a pointer to CBlockIndex since that data is just duplicated. However, in AddNMRetired we add data to the structure without having a block index connected to it. AddNMRetired is used by RetiredTN which is only used when tallying 14-day network averages when the highest block is at max 364500. If we can live without calculating averages that old we can remove a lot of obsolete code and open up StructCPID for better optimizations.

I guess we could also pass the block index to AddNMRetired and still optimize, but the more obsolete code we can remove the better.

[gridcoin]

Yes, good idea, I think you can retire the RetiredTN, since we have the nGrandfather > than that threshold now anyway, meaning the client should sync past 364500 with a non working RetiredTN function. If it does, sure just remove it if possible.
Thanks.

[RoboticMind]

Has any of this been fixed with the recent scraper?

[tomasbrod]

Yes, but there is still work needed.

[jamescowens]

I think this is largely done. #1583 and #1558