Move default Vela/Regor configurations to Sram_Only

[3l1]

Summary: Move default Vela/Regor configurations to Sram_Only

Differential Revision: D73212758

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10279

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• CI workflows being skipped on PR

❌ 1 New Failure

As of commit 758e29c with merge base 3997ae9 ():

NEW FAILURE - The following job has failed:

• pull / test-llava-runner-linux / linux-job (gh)
  RuntimeError: Command docker exec -t eb76794d24814d35e800e008cbaafc982230d45254fcd99975871df052afd3e1 /exec failed with exit code 139

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D73212758

[3l1]

@pytorchbot label "topic: not user facing"

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D73212758

[digantdesai]

what is the rationale for changing the default? I don't see any issues, asking because I am not sure why this was the default before.

cc @freddan80, @zingo

[gggekov]

Hi @digantdesai @3l1 ,
For U55, Shared_Sram is the most widely used memory mode. The reason is that on most embedded SoCs, you have limited amount of SRAM, hence most people prefer to place the NN in the Flash and use the SRAM only for the scratch_buffer(the SRAM will be used also for SW stack outside of the inference, e.g. pre/post processing, running an RTOS, etc). If you have a small model and SoC with enough SRAM for the NN, scratch buffer and your overall SW stack, then yes, it makes sense to use Sram_Only and you will get the performance benefit from the lower latency/higher bandwidth for the NN, but I would say this is the exception rather than the rule.

Also, the Corstone-300 has 2MB of SRAM and 256MB of DDR(can behave like Flash after the Timing Adapter parameters). Right now, for greater flexibility, we place everything in the DDR in the linker script. Then, via the REGIONCFG registers we tell the NPU to read the weights via AXI1(AXI1 providing Flash latency/BW thanks to the Timing Adapter settings) and read the scratch buffer via the AXI0(AXI0 providing SRAM latency & BW thanks to the TAs). On silicon, you can't do that- to get good performance from Sram_Only, you have to place NN & scratch buffer in the SRAM. I have a patch for internal review where i will move the allocation for the scratch buffer into an array in the SRAM.

The same considerations go for the U85 as well. In addition to Shared_Sram, the U85 will often be used on SoCs with Cortex-A and DRAM which is why we provide the Dedicated_Sram mode.

@3l1 maybe we have to look how we can enable you to easily change the memory mode, perhaps from the cmd line ? Is there a way we can do it that doesn't involve changing the default memory mode we test against?

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D73212758

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D73212758

[gggekov]

Hi @digantdesai @3l1 ,
For U55, Shared_Sram is the most widely used memory mode. The reason is that on most embedded SoCs, you have limited amount of SRAM, hence most people prefer to place the NN in the Flash and use the SRAM only for the scratch_buffer(the SRAM will be used also for SW stack outside of the inference, e.g. pre/post processing, running an RTOS, etc). If you have a small model and SoC with enough SRAM for the NN, scratch buffer and your overall SW stack, then yes, it makes sense to use Sram_Only and you will get the performance benefit from the lower latency/higher bandwidth for the NN, but I would say this is the exception rather than the rule.

Also, the Corstone-300 has 2MB of SRAM and 256MB of DDR(can behave like Flash after the Timing Adapter parameters). Right now, for greater flexibility, we place everything in the DDR in the linker script. Then, via the REGIONCFG registers we tell the NPU to read the weights via AXI1(AXI1 providing Flash latency/BW thanks to the Timing Adapter settings) and read the scratch buffer via the AXI0(AXI0 providing SRAM latency & BW thanks to the TAs). On silicon, you can't do that- to get good performance from Sram_Only, you have to place NN & scratch buffer in the SRAM. I have a patch for internal review where i will move the allocation for the scratch buffer into an array in the SRAM.

The same considerations go for the U85 as well. In addition to Shared_Sram, the U85 will often be used on SoCs with Cortex-A and DRAM which is why we provide the Dedicated_Sram mode.

@3l1 perhaps we have to look how we can enable you to easily change the memory mode, perhaps from the cmd line ? Is there a way we can do it that doesn't involve changing the default memory mode?

[gggekov]

Hi @digantdesai @3l1 ,
For U55, Shared_Sram is the most widely used memory mode. The reason is that on most embedded SoCs, you have limited amount of SRAM, hence most people prefer to place the NN in the Flash and use the SRAM only for the scratch_buffer(the SRAM will be used also for SW stack outside of the inference, e.g. pre/post processing, running an RTOS, etc). If you have a small model and SoC with enough SRAM for the NN, scratch buffer and your overall SW stack, then yes, it makes sense to use Sram_Only and you will get the performance benefit from the lower latency/higher bandwidth for the NN, but I would say this is the exception rather than the rule.

Also, the Corstone-300 has 2MB of SRAM and 256MB of DDR(can behave like Flash after the Timing Adapter parameters). Right now, for greater flexibility, we place everything in the DDR in the linker script. Then, via the REGIONCFG registers we tell the NPU to read the weights via AXI1(AXI1 providing Flash latency/BW thanks to the Timing Adapter settings) and read the scratch buffer via the AXI0(AXI0 providing SRAM latency & BW thanks to the TAs). On silicon, you can't do that- to get good performance from Sram_Only, you have to place NN & scratch buffer in the SRAM. I have a patch for internal review where i will move the allocation for the scratch buffer into an array in the SRAM.

The same considerations go for the U85 as well. In addition to Shared_Sram, the U85 will often be used on SoCs with Cortex-A and DRAM which is why we provide the Dedicated_Sram mode.

@3l1 maybe we have to look how we can enable you to easily change the memory mode, perhaps from the cmd line ? Is there a way we can do it that doesn't involve changing the default memory mode we test against?

[3l1]

You can ignore this pull request :) (Im not sure how to cancel it in this UI)

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D73212758

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D73212758

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D73212758

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D73212758