-
Notifications
You must be signed in to change notification settings - Fork 169
Add on-target tests to CI #355
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This is a great idea and thanks for putting the draft together!
My vote would be to plan to go ahead with this, but first merge cortex-m-rt into this repository; any thoughts @rust-embedded/cortex-m ? |
Out of interest, what cortex-m features can be tested on real hardware, that can't be tested on virtual hardware (e.g. qemu)? |
There are RTL accurate simulators that can test everything except analog behavior, but they are not practical here because of licensing restrictions for ARM IP, and because they are insanely slow. QEMU is a wonderful tool, most simulators are written by hardware designers for their use-case, QEMU is written by software developers for their use-case. As a result of this QEMU skips a few steps.
It is also worth noting that QEMU does not solve the "works on my machine problem"; QMEU is not deterministic, same as on-target testing. Zephyr (RTOS) uses QEMU extensively, and they have to stay on-top of this problem: zephyrproject-rtos/zephyr#12553 I have used a lot of different of pre-silicon simulation at work; typically when silicon is available we focus our testing efforts on-target because it is the best representation available. That said, QEMU can still be a good choice because it avoids bus factor / QOS. Neither option is a bad choice, I can make another draft of this PR using QEMU if desired. |
I appreciate the comprehensive response - thanks! I think I would like to see QEMU support as well, simply because it scales better and doesn't involve physical infrastructure that is voluntarily hosted. But I totally buy your arguments that on-device testing adds something beyond what QEMU can do and is hence highly desirable. Maybe we can talk to someone at Arm about a Fast Model licence... |
QEMU support is definitely doable. To clarify, is your desire to have QEMU be the CI target with the option to run on-target locally or the other way around?
I would like whatever testing method we choose to be easily accessible for outside contributors. The ARM fast model would be a better fit for this usecase than QEMU, but I am concerned that it would introduce too much of a barrier for contributions. |
This was discussed in the weekly meeting: https://github.com/rust-embedded/wg/blob/master/minutes/2021-10-26.md There is some hesitation to add on-target testing without an SLA, which is completely understandable. Based on that feedback I updated the draft PR:
The tests can still be run on physical hardware. There is still a discussion to be resolved as to what approach to take with regards to gating/non-gating and automatic/manual. @thejpster noted that the Cortex-M55 FPV is free for use (no licensing issues). I plan to explore that option this weekend as well. |
391: Merge cortex-m-rt into this repository r=therealprof a=adamgreig This PR merges the cortex-m-rt repository (with history) into this repo, inside the `cortex-m-rt` folder which is added to the workspace. The main advantage is easier combined testing of cortex-m with cortex-m-rt (including on-hardware tests e.g. #355), and in the future easier changes across the two projects. The MSRV of cortex-m-rt is bumped 1.39 -> 1.40 to align it with cortex-m itself. I've updated the CI to run the same tests and checks as before, and updated references to the old URL. If/after this is merged, I propose adding a note to the old repo's README and then archiving it. An alternative to this technique would be adding all the files in one new commit (not preserving history), if anyone thinks that would be neater. NB: This PR also adds an inline to ITM to fix a clippy hard error. For future reference, the git work was: ``` cd cortex-m-rt git filter-repo --to-subdirectory-filter cortex-m-rt cd ../cortex-m git remote add rt ../cortex-m-rt git fetch rt git merge --allow-unrelated-histories rt/master ``` Co-authored-by: bors[bot] <bors[bot]@users.noreply.github.com> Co-authored-by: Raz Aloni <[email protected]> Co-authored-by: Vadim Kaushan <[email protected]> Co-authored-by: Jonathan 'theJPster' Pallant <[email protected]> Co-authored-by: Adam Greig <[email protected]> Co-authored-by: Jonas Schievink <[email protected]> Co-authored-by: Jorge Aparicio <[email protected]> Co-authored-by: Emil Fresk <[email protected]> Co-authored-by: Daniel Egger <[email protected]> Co-authored-by: Niklas Claesson <[email protected]> Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
391: Merge cortex-m-rt into this repository r=thejpster a=adamgreig This PR merges the cortex-m-rt repository (with history) into this repo, inside the `cortex-m-rt` folder which is added to the workspace. The main advantage is easier combined testing of cortex-m with cortex-m-rt (including on-hardware tests e.g. #355), and in the future easier changes across the two projects. The MSRV of cortex-m-rt is bumped 1.39 -> 1.40 to align it with cortex-m itself. I've updated the CI to run the same tests and checks as before, and updated references to the old URL. If/after this is merged, I propose adding a note to the old repo's README and then archiving it. An alternative to this technique would be adding all the files in one new commit (not preserving history), if anyone thinks that would be neater. NB: This PR also adds an inline to ITM to fix a clippy hard error. For future reference, the git work was: ``` cd cortex-m-rt git filter-repo --to-subdirectory-filter cortex-m-rt cd ../cortex-m git remote add rt ../cortex-m-rt git fetch rt git merge --allow-unrelated-histories rt/master ``` Co-authored-by: bors[bot] <bors[bot]@users.noreply.github.com> Co-authored-by: Jonathan 'theJPster' Pallant <[email protected]> Co-authored-by: Adam Greig <[email protected]> Co-authored-by: Jonas Schievink <[email protected]> Co-authored-by: Jorge Aparicio <[email protected]> Co-authored-by: Emil Fresk <[email protected]> Co-authored-by: Daniel Egger <[email protected]> Co-authored-by: Niklas Claesson <[email protected]> Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com> Co-authored-by: Vadim Kaushan <[email protected]>
ba8945f
to
c90228d
Compare
I updated this pull-request for #391 which solves the bootstrapping problem; CI is passing now! 🎉 I also removed the demonstration do-nothing tests and added a couple basic tests. |
This is looking great, thanks! Perhaps it would be worth adding the new CI flow to bors' required statuses? I'd still be interested in having it run automatically on hardware as a best-effort thing: perhaps qemu on GitHub's own CI and required for bors, and also (tries to) runs a job on your hardware, but without being required for bors so it wouldn't block merging if it wasn't working? Edit: To be clear, no need for on-hardware testing as part of this PR or anything, we could go ahead with just qemu if you like. Just making it clear I still think there's a lot of value in automated on-hardware testing too, in addition to it being easy to run on-hardware tests manually from the repo. Is there anything else before this is "ready for review"? |
Missed that, fixed, thanks!
I can do that, everything is setup because I run on-target testing for the @thejpster expressed some SLA concerns before - what do you think about this approach instead?
Just cleaning up the introduction message so that the now-irrelevant details do not get committed by bors 😄 |
@thejpster any concerns with the above - adding on-target testing with real hardware to CI (in addition to QEMU), but without the real hardware CI runs being a gating check to mitigate SLA issues? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great. Love the really thorough documentation.
Apologies it took so long to get to this, too. |
No problem!
I cleaned up the PR body to reflect the current plan, re-added the HIL tests, and marked this as ready for review 👍 Before this gets merged the self-hosted runner needs to get added; I will need the github actions runner registration token for this repository, or access to the repository to do this myself. |
605: Add @newAM to the Cortex-M team r=adamgreig a=newAM As discussed in the rust-embedded meeting today; see [this comment](rust-embedded/cortex-m#355 (comment)) for additional context. Co-authored-by: Alex Martens <[email protected]>
The final steps are done and this should be good to go now! I added the self-hosted runner and rebased onto Repo admins can see the runner in the settings here: https://github.com/rust-embedded/cortex-m/settings/actions/runners Here's the new job with the runner: https://github.com/rust-embedded/cortex-m/actions/runs/2048592598 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks for your perseverance! I'll leave this to give other @rust-embedded/cortex-m peolpe a chance to look over and if there's no objections we can merge in a couple of days?
bors r+ |
429: Exclude testsuite from cron, fixes #427 r=therealprof a=adamgreig Turns out #427 was caused by #355 after all, because the CI workflow was updated to exclude testsuite from its default `cargo test`, but the cron workflow wasn't. For now this seems like the simplest fix; eventually it might be nice to have cron run the testsuite too (at least in qemu), but really cron's there to catch surprise compilation failures when new Rust versions or new dependency releases come out, so I don't think it's urgent. Co-authored-by: Adam Greig <[email protected]>
A week ago in the rust-embedded matrix chat @adamgreig mentioned that on-target CI would be helpful for the
cortex-m
andcortex-m-rt
crates. This is a pull-request to make that happen.History: Bootstrapping and
cortex-m-rt
The first problem I came across was the amount of boilerplate required to build an on-target binary without
cortex-m-rt
.There are two paths forward here:
cortex-m
, which will largely be copy-pasted fromcortex-m-rt
.cortex-m-rt
to this workspace.In (#391)
cortex-m-rt
was relocated to this workspace to fix the bootstrapping problem.Test Execution
Tests run with QEMU and physical hardware.
QEMU uses the LM3S6965 Cortex-M3 target because it is widely used.
The physical hardware is a NUCLEO-F070RB connected to a self-hosted runner. In the future more hardware can be added to cover more CPUs.
Due to reliability concerns with self-hosted runners only QEMU will be a required status check in CI, the physical hardware checks will be informational only.
CI Software
The CI software for running on physical hardware is simply a self-hosted github actions runner. A working demonstration of this can be found here (the repository does not appear as a fork to work around github actions limitations with forks).
The runner uses
probe-run
to execute the tests on embedded hardware.probe-run
was chosen for several reasons:probe-rs
.defmt-test
).Test Harness
This PR introduces a test harness,
minitest
.minitest
is almost identical todefmt-test
, the only difference is that it replacesdefmt
withrtt-target
becausedefmt
introduces a dependency cycle oncortex-m
.This is harness is very minimal, adding only 327 lines of rust:
The test harness does introduce some abstraction, and may not be suitable for all tests. Lower-level tests are still possible without the harness using
asm::udf
to fail the test, andasm::bkpt
to exit without failure.Reliability and Uptime
I have been doing automatic on-target testing for the
stm32wlxx-hal
usingprobe-run
. Over hundreds of automatic runs spanning several months I have had no failures as a result of external factors (USB connectivity, programming errors, ect.). I do not anticipate on-target CI being perfect, but at the same time I do not anticipate frequent problems.