Migrate CPU Backend to allocate memory at runtime.

[gcatron]

Description: Migrate CPU Backend to allocate and initialize memory at runtime. Added a new runtimeInfo object that contains information needed for runtime memory allocation.
Testing: ninja test passes, AOT bundles still work, testing with run.sh --iterations=20 -cpu -time showed minimal performance difference.
Documentation: N/A
Progress on #1904

[nadavrot]

Thank you for doing this work @gcatron . I am excited to see the fast development on this PR. I made a few comments in the code, mainly regarding comments and style.

I asked a few questions, such as "is it possible to forward declare a class". if the answer is "no" then feel free to ignore the question and move on.

Please make sure to run the ./run.sh tests with the --iterations=10 and -cpu flags and check that there are no noticeable performance regressions. I expect none but we need to be sure.

Also, please make sure to squash the PR down to one or two commits, or whatever is reasonable.

@opti-mix is the expert in this part of the project, so if he approved the design then feel free to land (after fixing the style issues).

[rdzhabarov]

I'm not very familiar with this code path :)

Question:
Do you allocate memory for weights and activations every time on inference request?

I think longer term we could keep activations in thread local memory, facilitate thread safety and avoidance of allocation/deallocation overhead.

[rdzhabarov]

How do you test this change :)?

[qcolombet]

Looks reasonable as a first step.
I believe with want to be more aggressive on the refactoring going forward.

[rdzhabarov]

Have you verified that AOT still works :)?

[rdzhabarov]

We also have information about JIT under /docs in the repo, could you check if anything needs to be updated there?

Looking good.

[opti-mix]

Great job! Looking good!

[nadavrot]

Congrats! Nice work.