Test suite overhaul

[mahesh-hegde]

The current test suite does not scale. It takes ~1 minute to run tests on a powerful machine, and more on a less powerful machine, which includes the CI machines which might have 2-4 vCPUs at best.

Performance

- [ ] Large amount of time is spent in invoking maven commands, although most of the time the dependencies are in the local folder. #742 should fix this.
- [ ] javac is invoked in a few test cases, this can be cached transparently too.

Update: These seem to take some time during binding generation, but do not occupy large part of our testsuite.

• More tests should be able to be run from same summary. Probably we should implement Decouple summarizer and code generator #678 .

Overall, not running any Java based tool in the fast path should be the goal.

Testing summary

• Summary golden file tests: This can be moved from JUnit/Java to dart, since we are using git diff --no-index anyway to compare. This should also enable us to add more summary tests.

• Summary characteristic tests: Pattern match known details on summary, since most details will be common between asm and doclet summaries.

Examples

Unfortunately some examples are rather android-specific and cannot be frequently run in usual dev workflow.

But pdfbox standalone test can be probably run.

• Create a program which creates a minimal PDF using PDFBox, and run the other (currently existing example) on the same PDF file.

CI

I believe CI shouldn't take 10+ minutes and there should not be 17 CI jobs. (Except those which eg: upload the results).

The worst offender seems to be the android build jobs.

• Explore adding more caches to long-running CI jobs.
  • ?? No idea if this handles SDK updates well. ??

- [ ] Some long running Android builds install multiple versions of the NDK. Remove as many NDK constraints as possible. (Shouldn't everything build with same NDK?).

• check_java_format and test_summarizer should not be different jobs.

- [ ] test_jni can run in 20s, it's better to merge this and analyze_jni.

- [ ] Is there a technical reason why collect coverage is run separately once all tests pass? I think coverage can be collected during first time itself?
(Error reporting)

---

[HosseinYousefi]

Once doclet is removed (#1079), a lot of the tests will be simplified. Closing this.