Qualcomm AI Engine Direct - Streaming Mimi Enablement

[winskuo-quic]

Summary

Stateless(Static) Streaming Mimi Decoder Enablement.

Scripts to run

1. Run examples/models/moshi/mimi/install_requirements.sh, manually change pip install moshi==0.2.3 for this since we are currently building static mimi decoder on top of 0.2.3 version

2. Run the example script

• CPU Encoder with QNN Decoder
  python examples/qualcomm/oss_scripts/moshi/mimi.py -b build-android -s $DEVICE -m SM8650 --use_cpu_encoder

• QNN Encoder/Decoder
  python examples/qualcomm/oss_scripts/moshi/mimi.py -b build-android -s $DEVICE -m SM8650

Stats for SM8650

• SQNR Scores Improvement after Streaming Mode Enablement
  • CPU Encoder with QNN Decoder: 4.7 -> 7.8
  • QNN Encoder/Decoder: 0.46 -> 0.88
• Encoder: 3.0ms/chunk
• Decoder: 5.9ms/chunk

Follow ups

• Add README
• UT and Community CI enablement
• Inference Speed Optimization
• Accuracy Improvement

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10570

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures

As of commit e44e919 with merge base df75088 ():

NEW FAILURES - The following jobs have failed:

• Check Labels / Check labels (gh)
  RuntimeError: Error checking labels: PR does not have required labels
• Lint / link-check / lint-urls / linux-job (gh)
  RuntimeError: Command docker exec -t 22f9bf33b66145cac2de68661af105f4f14431224f79ecb77d6d21de1dfbc16e /exec failed with exit code 127
• Lint / link-check / lint-xrefs / linux-job (gh)
  RuntimeError: Command docker exec -t c4f4843964412f281728ef5a903ea5c0b144266b813ed357ec41df96685fc05f /exec failed with exit code 127
• Lint / lintrunner / linux-job (gh)
  >>> Lint for backends/cadence/hifi/operators/op_mm.cpp:
• pull / unittest-arm-backend-with-no-fvp (test_pytest_models) / linux-job (gh)
  RuntimeError: Command docker exec -t 858f76b67fc5ac5d0e4c39d7c8b044892b714e9a4586a412aa81c33a389a3c6f /exec failed with exit code 127
• pull / unittest-arm-backend-with-no-fvp (test_pytest_ops) / linux-job (gh)
  RuntimeError: Command docker exec -t eb669ec9859f22476ed5685c87747fb5f4b3f5ac502dbba724010ace171fb86f /exec failed with exit code 127

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[winskuo-quic]

Hi @cccclai, @iseeyuan,
This PR is to enable Stateless (Static) Mimi.
Execution commands and some of the stats are shared in the Summary section.
Also, it would be appreciated if you could confirm the license for this PR is written correctly.
As discussed previously, it requires source code modification in order to enable the model.
The following 4 files listed below involves source code modification.

• examples/qualcomm/oss_scripts/moshi/model/static_conv.py
• examples/qualcomm/oss_scripts/moshi/model/static_convtr.py
• examples/qualcomm/oss_scripts/moshi/model/static_mimi.py
• examples/qualcomm/oss_scripts/moshi/model/static_seanet_decoder.py

Please let me know if you believe something is not done correctly.
Thanks