Description
We have a few tier 3 targets that are 16-bit, AVR and MSP430. These have numerous issues but a number of them have been resolved recently. It's almost like you can really compile for them! Unfortunately these targets have almost no regression testing, as they are bare-metal, cross-compile-only targets. If we are serious about supporting them, it would be prudent to run tests. Fortunately, AVR has a sophisticated simulation testing ecosystem which several users report as reliable-enough for "the program builds and functions, and maybe even as-intended". However, as these are tier 3 targets, it would be somewhat of a policy change to add testing for much more than "you can technically build code for them ever".
On the other hand, the simulation testing in this case would probably be for bare minimums like "arithmetic and bitwise operators work". Embedded users are often willing to work around the inadequacy of std
and the language, but it's hard to expect them to build around us if they don't have access to even basic computation primitives.
It may be surprising these targets need regression testing for such minimums, but nonetheless it's hard to expect them to be elevated past even tier 3 if we don't allow regression testing for those basics. On the other hand, if they break, very few compiler programmers will know how to fix them, and the policy of not running testing for tier 3 targets is there for a reason. So, this may need an MCP or something like that? I don't know. I'm just raising it as an issue here to, for now, track issues that either currently or would enjoy this. Or maybe this will just wait until we elevate support for these targets past tier 3.
- [AVR] result of
i32 as f32
is incorrect #77131 - [AVR] integer division incorrectly yields result value 1 #82242
- [AVR] Shift left
<<
doesn't work correctly on >= 32-bit integers on a 16-bit platform #82380 - An apparent regression introduced by virtually no regression testing for AVR:
Activity
the8472 commentedon Feb 3, 2023
A while ago there was some disussion about adding hello world or libs-only tests for tier2 targets where qemu is available. It wouldn't provide full test coverage but at least serve as a smoke test. #101703 was a first step in that direction.
Maybe for tier-3 they'd have to be allow-fail tests.
Rahix commentedon Feb 3, 2023
Just a thought about this: Even just knowing that regressions have happened would be nice. I am in favor of not blocking anything on these targets but I think giving people a heads up that something breaks AVR and friends would be a great start.
Mark-Simulacrum commentedon Feb 3, 2023
I am a little confused by this statement. Part of the tier policy is explicitly that raising to a higher tier is what "allows" putting additional burden on CI and maintainers, justified by greater commitment from the target maintainers and stricter requirements on the target itself.
In general, the policy we have today explicitly indicates that target maintainers (which don't exist for every target) are encouraged to run build and tests regularly. Basically, anything our CI doesn't cover should be covered externally, and fixes submitted as needed.
I personally am willing to interpret the tier 2 guarantees around testing in a way that includes tests which can be run without architecture-specific tooling. For example, the asm! macro has a number of tests for tier 2 and tier 3 targets that I believe are even no_core, explicitly to allow easily running those on our tier 1 platforms.
workingjubilee commentedon Feb 3, 2023
I guess part of the reason I opened this issue is because I am
2 << 2 == 8
or such, even w/o testingIn other words I am questioning things a bit in the vein of "do the existing policies make sense for all targets?" As you noted, we have what might be considered exceptions to the rule. But they're exceptions that might make sense, and if they do, it is likely because they are testing fairly fundamental compiler functionality in these cases.
cr1901 commentedon Feb 9, 2023
FWIW,
mspdebug
has an MSP430 simulator. I have batted around using that for testing my own code. But I would have no idea what integration intocargo test
orcompiletest
would look like.Patryk27 commentedon Feb 24, 2023
Self-advertisement: I've written a tool that does that! -- https://github.com/Patryk27/avr-tester/.
It's a wrapper over simavr (which, well, simulates AVRs) and it can be easily used to test whether various stuff works as intended, in fully black-box tests - in fact, one of the first tests I've written there is an expression evaluator which allowed me to make sure that at least the math stuff works as intended when I was sending patches to compiler-builtins and whatnot.
The AVR program is:
https://github.com/Patryk27/avr-tester/blob/3ea7049c813a4c12a0278f26112bb199fe07f69b/avr-tester-tests/xx-eval/src/main.rs
... and the Rust test:
https://github.com/Patryk27/avr-tester/blob/3ea7049c813a4c12a0278f26112bb199fe07f69b/avr-tester/tests/tests/xx/eval.rs
... and what basically happens there is that the Rust test generates random expression trees (like
(123 * 7) - 32
), sends them to a simulated AVR, the simulated firmware then computes the result and sends back to the "host", which then compares its own evaluation result with the one retrieved from AVR.Maybe it would come handy here?
cr1901 commentedon Feb 26, 2023
Your avr-tester project is cool, but I was looking to try to use the same test function bodies on a
std
andno_std
platform, and have "some bridge program that knows what to do whencargo test
is called for a target withoutlibstd
." I.e. pass--target
tocargo test
".For msp430, "knows what to do" would be "wrap the test function into a
main
ELF binary", and send that freshly-compiled ELF binary off tomspdebug
's simulator`."avr-tester
seems to be more "simulate peripherals in the context ofcargo test
running on a target withstd
(i.e. don't pass--target
tocargo test
").cr1901 commentedon Mar 16, 2023
Apropos of a crate I've been writing, I suspect I found an arithmetic bug on msp430. When called, the following function
shift_test
panic
s in bothrelease
anddebug
mode fortarget=msp430-none-elf
, but passes for my hostx86_64-pc-windows-gnu
(and presumably others with a multiple-bit shift insn):I've been writing a tool to make msp430 debugging easier, and have been using
mspdebug
s simulator effectively to create a test case:Perhaps a driver program/library could be useful for integrating into the
#[test]
framework. Is there any precedent for runningcargo test
for targets using a simulator?Erm, well... they mostly work on MSP430 :P. I lost many hours to this bug. It caused a deserialization routine to fail when the result didn't fit into a
u16
thanks to a bit in the high 6 bytes being set incorrectly by the "shift-by-0"...