More extensive testing

[japaric]

Once #43 lands, I think we should be ready to test generating all the peripherals that a SVD contains. svd2rust -i $svd would give us the full list of peripheral then we can use svd2rust -i $svd $peripheral for each peripheral in that list.

We could even repeat this approach for several SVD files in the cmsis-svd repo. We should be careful to not overdo ourselves and end up with unreasonable test times though.

[japaric]

Here's a script I used to smoke test svd2rust against posborne/cmsis-svd.

set -ex

main() {
    local url='https://github.com/posborne/cmsis-svd/archive/python-0.4.tar.gz'
    local data_dir=$(mktemp -d)

    curl -L $url | tar -C $data_dir --strip-components 1 -xz

    local errors=()
    set +ex
    find $data_dir -type f -name '*.svd' -print0 | \
        while IFS= read -r -d $'\0' line; do
            svd2rust -i $line >/dev/null 2>&1 || \
                echo $(basename $line)
        done

    set -ex

    rm -rf $data_dir
}

main

These svd2rust failed to generate code from these files:

M36B.svd - BAD SVD - svd-parser-0.5.1/src/lib.rs:216
M369.svd - PARSER? - svd-parser-0.5.1/src/lib.rs:216
M368.svd - PARSER? - svd-parser-0.5.1/src/lib.rs:216
M367.svd - PARSER? - svd-parser-0.5.1/src/lib.rs:216
M365.svd - PARSER? - svd-parser-0.5.1/src/lib.rs:216
SIM3L1x8_SVD.svd - PARSER? - svd-parser-0.5.1/src/lib.rs:58
LPC408x_7x_v0.7.svd - BAD SVD - error: Register P0_12 has no reset value
LPC178x_7x_v0.8.svd - BAD SVD - error: Register P0_12 has no reset value
LPC178x_7x.svd - BAD SVD - error: Register P0_12 has no reset value
LPC176x5x_v0.2.svd - BUG? - error: EnumeratedValue RESERVED has no <value> field
LPC11Axxv0.6.svd - BAD SVD - error: Register PIO0_4 has no reset value
ATSAMA5D35.svd - BAD SVD - error: Register CR has no reset value
ATSAMA5D34.svd - BAD SVD - error: Register CR has no reset value
ATSAMA5D33.svd - BAD SVD - error: Register CR has no reset value
ATSAMA5D31.svd - BAD SVD - error: Register CR has no reset value
ATSAM4SD32C.svd - BAD SVD - error: Register CR has no reset value
ATSAM4SD32B.svd - BAD SVD - error: Register CR has no reset value
ATSAM4S8C.svd - BAD SVD - error: Register CR has no reset value
ATSAM4S8B.svd - BAD SVD - error: Register CR has no reset value
ATSAM4S16C.svd - BAD SVD - error: Register CR has no reset value
ATSAM4S16B.svd - BAD SVD - error: Register CR has no reset value
ATSAM3X8E.svd - BAD SVD - error: Register CR has no reset value
ATSAM3X8C.svd - BAD SVD - error: Register CR has no reset value
ATSAM3X4E.svd - BAD SVD - error: Register CR has no reset value
ATSAM3X4C.svd - BAD SVD - error: Register CR has no reset value
ATSAM3U4E.svd - BAD SVD - error: Register CR has no reset value
ATSAM3U4C.svd - BAD SVD - error: Register CR has no reset value
ATSAM3U2E.svd - BAD SVD - error: Register CR has no reset value
ATSAM3U2C.svd - BAD SVD - error: Register CR has no reset value
ATSAM3U1E.svd - BAD SVD - error: Register CR has no reset value
ATSAM3U1C.svd - BAD SVD - error: Register CR has no reset value
ATSAM3SD8C.svd - BAD SVD - error: Register CR has no reset value
ATSAM3SD8B.svd - BAD SVD - error: Register CR has no reset value
ATSAM3S8C.svd - BAD SVD - error: Register CR has no reset value
ATSAM3S8B.svd - BAD SVD - error: Register CR has no reset value
ATSAM3S4C.svd - BAD SVD - error: Register CR has no reset value
ATSAM3S4B.svd - BAD SVD - error: Register CR has no reset value
ATSAM3S4A.svd - BAD SVD - error: Register CR has no reset value
ATSAM3S2C.svd - BAD SVD - error: Register CR has no reset value
ATSAM3S2B.svd - BAD SVD - error: Register CR has no reset value
ATSAM3S2A.svd - BAD SVD - error: Register CR has no reset value
ATSAM3S1C.svd - BAD SVD - error: Register CR has no reset value
ATSAM3S1B.svd - BAD SVD - error: Register CR has no reset value
ATSAM3S1A.svd - BAD SVD - error: Register CR has no reset value
ATSAM3N4C.svd - BAD SVD - error: Register CR has no reset value
ATSAM3N4B.svd - BAD SVD - error: Register CR has no reset value
ATSAM3N4A.svd - BAD SVD - error: Register CR has no reset value
ATSAM3N2C.svd - BAD SVD - error: Register CR has no reset value
ATSAM3N2B.svd - BAD SVD - error: Register CR has no reset value
ATSAM3N2A.svd - BAD SVD - error: Register CR has no reset value
ATSAM3N1C.svd - BAD SVD - error: Register CR has no reset value
ATSAM3N1B.svd - BAD SVD - error: Register CR has no reset value
ATSAM3N1A.svd - BAD SVD - error: Register CR has no reset value
ATSAM3N0C.svd - BAD SVD - error: Register CR has no reset value
ATSAM3N0B.svd - BAD SVD - error: Register CR has no reset value
ATSAM3N0A.svd - BAD SVD - error: Register CR has no reset value
ATSAM3N00B.svd - BAD SVD - error: Register CR has no reset value
ATSAM3N00A.svd - BAD SVD - error: Register CR has no reset value
ATSAM3A8C.svd - BAD SVD - error: Register CR has no reset value
ATSAM3A4C.svd - BAD SVD - error: Register CR has no reset value
AT91SAM9X35.svd - BAD SVD - error: Register CR has no reset value
AT91SAM9X25.svd - BAD SVD - error: Register CR has no reset value
AT91SAM9N12.svd - BAD SVD - error: Register CR has no reset value
AT91SAM9M11.svd - BAD SVD - error: Register LUTENTRY[%s] has no reset value
AT91SAM9M10.svd - BAD SVD - error: Register LUTENTRY[%s] has no reset value
AT91SAM9G35.svd - BAD SVD - error: Register CR has no reset value
AT91SAM9G25.svd - BAD SVD - error: Register CR has no reset value
AT91SAM9G20.svd - BAD SVD - error: Register CCR0 has no reset value
AT91SAM9G15.svd - BAD SVD - error: Register CR has no reset value
AT91SAM9G10.svd - BAD SVD - error: Register LUTENTRY[%s] has no reset value
AT91SAM9CN12.svd - BAD SVD - error: Register CR has no reset value
AT91SAM9CN11.svd - BAD SVD - error: Register CR has no reset value

I have yet to investigate the cause of the errors. The error rate is 71 out of 490 or about 14.49%.

[japaric]

I have now classified all the 71 errors. Most of them are caused by the SVD not being valid. As per the specification all registers must have a reset value but the SVD listed above are missing reset values in some registers.

[japaric]

Updated script to detect SVD files for which svd2rust does generate code without crashing but the generated code doesn't compile:

set -ex

main() {
    local url='https://github.com/posborne/cmsis-svd/archive/python-0.4.tar.gz'
    local cargo_dir=$(mktemp -d)
    local data_dir=$(mktemp -d)

    curl -L $url | tar -C $data_dir --strip-components 1 -xz

    cargo init --lib --name device $cargo_dir
    ( cd $cargo_dir && cargo add cortex-m vcell )

    local errors=()
    set +ex
    find $data_dir -type f -name '*.svd' -print0 | \
        while IFS= read -r -d $'\0' line; do
            svd2rust -i $line >$cargo_dir/src/lib.rs 2>/dev/null || \
                ( echo "I: $(basename $line)" && continue )

            cargo check --manifest-path $cargo_dir/Cargo.toml >/dev/null 2>&1 || \
                echo "II: $(basename $line)"
        done

    set -ex

    rm -rf $cargo_dir
    rm -rf $data_dir
}

main

The compile errors are referred to as class II errors, and the generation errors are referred to as class I errors.

Here's the list of class II errors I have seen so far. (The script is still running on my machine; it's taking a while to test all the SVD files). The errors have already been classified:

• LPC11Cxx_v9.svd | BAD SVD | two enumeratedValues have the same name
• LPC11E6x_v0.8.svd | BAD SVD | two enumeratedValues have the same name
• LPC11Exx_v5.svd | BAD SVD | two enumeratedValues have the same name
• LPC11Uxx_v7.svd | BAD SVD | two enumeratedValues have the same name
• LPC11xx_v6.svd | BAD SVD | two enumeratedValues have the same name
• LPC11xx_v6a.svd | BAD SVD | two enumeratedValues have the same name
• LPC13Uxx_v1.svd | BAD SVD | two enumeratedValues have the same name
• LPC15xx_v0.7.svd | BAD SVD | two enumeratedValues have the same name
• LPC800_v0.3.svd | BAD SVD | two enumeratedValues have the same name
• MKV56F20.svd | BAD SVD | Bad enumeratedValue values
• MKV56F22.svd | BAD SVD | Bad enumeratedValue values
• MKV56F24.svd | BAD SVD | Bad enumeratedValue values
• MKV58F20.svd | BAD SVD | Bad enumeratedValue values
• MKV58F22.svd | BAD SVD | Bad enumeratedValue values
• MKV58F24.svd | BAD SVD | Bad enumeratedValue values
• nrf52.svd | BAD SVD | two enumeratedValues have the same value

(up to this point these errors are not due to svd2rust bugs)

• STM32L063x.svd | support <access>read-writeOnce</access> #91
• STM32L062x.svd | support <access>read-writeOnce</access> #91
• STM32L053x.svd | support <access>read-writeOnce</access> #91
• STM32L052x.svd | support <access>read-writeOnce</access> #91
• STM32L051x.svd | support <access>read-writeOnce</access> #91
• LPC5410x_v0.4.svd | Noncontiguous register clusters result in name collisions #90
• LPC43xx_svd_v5.svd | SYNTAX ERROR
• LPC18xx_svd_v18.svd | SYNTAX ERROR
• LPC13xx_svd_v1.svd | SYNTAX ERROR
• LPC11D14_svd_v4.svd | SYNTAX ERROR
• LPC1102_4_v4.svd | SYNTAX ERROR
• SKEAZN84.svd | "bits" is a valid field name #92
• SKEAZN642.svd | "bits" is a valid field name #92
• SKEAZ1284.svd | "bits" is a valid field name #92
• MKW01Z4.svd | "bits" is a valid field name #92
• MK81F25615.svd | Name mangling is lossy around numbers #76
• MK82F25615.svd | Name mangling is lossy around numbers #76
• MKE02Z2.svd | "bits" is a valid field name #92
• MKE02Z4.svd | "bits" is a valid field name #92
• MKE04Z1284.svd | "bits" is a valid field name #92
• MKE04Z4.svd | "bits" is a valid field name #92
• MKE06Z4.svd | "bits" is a valid field name #92
• MKE14D7.svd | "bits" is a valid field name #92
• MKE15D7.svd | "bits" is a valid field name #92
• MKL02Z4.svd | "bits" is a valid field name #92
• MKL03Z4.svd | "bits" is a valid field name #92
• MKL04Z4.svd | "bits" is a valid field name #92
• MKL05Z4.svd | "bits" is a valid field name #92
• MKL13Z644.svd | "bits" is a valid field name #92
• MKL14Z4.svd | "bits" is a valid field name #92
• MKL15Z4.svd | "bits" is a valid field name #92
• MKL16Z4.svd | "bits" is a valid field name #92
• MKL17Z4.svd | "bits" is a valid field name #92
• MKL17Z644.svd | "bits" is a valid field name #92
• MKL24Z4.svd | "bits" is a valid field name #92
• MKL25Z4.svd | "bits" is a valid field name #92
• MKL26Z4.svd | "bits" is a valid field name #92
• MKL27Z4.svd | "bits" is a valid field name #92
• MKL27Z644.svd | "bits" is a valid field name #92
• MKL33Z4.svd | "bits" is a valid field name #92
• MKL33Z644.svd | "bits" is a valid field name #92
• MKL34Z4.svd | "bits" is a valid field name #92
• MKL36Z4.svd | "bits" is a valid field name #92
• MKL43Z4.svd | "bits" is a valid field name #92
• MKL46Z4.svd | "bits" is a valid field name #92
• MKL81Z7.svd | Name mangling is lossy around numbers #76
• MKL82Z7.svd | Name mangling is lossy around numbers #76
• MKM14ZA5.svd | "bits" is a valid field name #92
• MKM33ZA5.svd | "bits" is a valid field name #92
• MKM34Z7.svd | "bits" is a valid field name #92
• MKM34ZA5.svd | "bits" is a valid field name #92

[japaric]

Testing all the SVD files in cmsis-svd takes a few hours on my laptop so it seems very unlikely that will be able to run it as part of the CI testing of this repo. I think we should sample some files, specially those listed above that are causing trouble today, from the database and just use those in our test suite. At the very least we should pick on SVD file per microcontroller vendor.

[pftbest]

Also there are some new svd files in cmsis-svd repo (on master branch) that are not present in 0.4 archive.

[japaric]

@pftbest thanks for the heads up. I'll re-run the test script on the diff between HEAD and v0.4.0.

[japaric]

I'll re-run the test script on the diff between HEAD and v0.4.0.

I have now tested and classified the extra files. All the TM4C files fail (svd2rust works but compilation errors) due to #97. The rest work fine.

• TM4C1230C3PM.svd support <alternateGroup> in registers? #97
• TM4C1230D5PM.svd support <alternateGroup> in registers? #97
• TM4C1230E6PM.svd support <alternateGroup> in registers? #97
• TM4C1230H6PM.svd support <alternateGroup> in registers? #97
• TM4C1231C3PM.svd support <alternateGroup> in registers? #97
• TM4C1231D5PM.svd support <alternateGroup> in registers? #97
• TM4C1231D5PZ.svd support <alternateGroup> in registers? #97
• TM4C1231E6PM.svd support <alternateGroup> in registers? #97
• TM4C1231E6PZ.svd support <alternateGroup> in registers? #97
• TM4C1231H6PGE.svd support <alternateGroup> in registers? #97
• TM4C1231H6PM.svd support <alternateGroup> in registers? #97
• TM4C1231H6PZ.svd support <alternateGroup> in registers? #97
• TM4C1232C3PM.svd support <alternateGroup> in registers? #97
• TM4C1232D5PM.svd support <alternateGroup> in registers? #97
• TM4C1232E6PM.svd support <alternateGroup> in registers? #97
• TM4C1232H6PM.svd support <alternateGroup> in registers? #97
• TM4C1233C3PM.svd support <alternateGroup> in registers? #97
• TM4C1233D5PM.svd support <alternateGroup> in registers? #97
• TM4C1233D5PZ.svd support <alternateGroup> in registers? #97
• TM4C1233E6PM.svd support <alternateGroup> in registers? #97
• TM4C1233E6PZ.svd support <alternateGroup> in registers? #97
• TM4C1233H6PGE.svd support <alternateGroup> in registers? #97
• TM4C1233H6PM.svd support <alternateGroup> in registers? #97
• TM4C1233H6PZ.svd support <alternateGroup> in registers? #97
• TM4C1236D5PM.svd support <alternateGroup> in registers? #97
• TM4C1236E6PM.svd support <alternateGroup> in registers? #97
• TM4C1236H6PM.svd support <alternateGroup> in registers? #97
• TM4C1237D5PM.svd support <alternateGroup> in registers? #97
• TM4C1237D5PZ.svd support <alternateGroup> in registers? #97
• TM4C1237E6PM.svd support <alternateGroup> in registers? #97
• TM4C1237E6PZ.svd support <alternateGroup> in registers? #97
• TM4C1237H6PGE.svd support <alternateGroup> in registers? #97
• TM4C1237H6PM.svd support <alternateGroup> in registers? #97
• TM4C1237H6PZ.svd support <alternateGroup> in registers? #97
• TM4C123AE6PM.svd support <alternateGroup> in registers? #97
• TM4C123AH6PM.svd support <alternateGroup> in registers? #97
• TM4C123BE6PM.svd support <alternateGroup> in registers? #97
• TM4C123BE6PZ.svd support <alternateGroup> in registers? #97
• TM4C123BH6PGE.svd support <alternateGroup> in registers? #97
• TM4C123BH6PM.svd support <alternateGroup> in registers? #97
• TM4C123BH6PZ.svd support <alternateGroup> in registers? #97
• TM4C123BH6ZRB.svd support <alternateGroup> in registers? #97
• TM4C123FE6PM.svd support <alternateGroup> in registers? #97
• TM4C123FH6PM.svd support <alternateGroup> in registers? #97
• TM4C123GE6PM.svd support <alternateGroup> in registers? #97
• TM4C123GE6PZ.svd support <alternateGroup> in registers? #97
• TM4C123GH6PGE.svd support <alternateGroup> in registers? #97
• TM4C123GH6PM.svd support <alternateGroup> in registers? #97
• TM4C123GH6PZ.svd support <alternateGroup> in registers? #97

[japaric]

Done in #101

[japaric]

The ci/script.sh file is now keeping track of the SVD files that are not working due to missing feature is svd2rust. Each of those files has been classified and issues have been filed under the 'cmsis-svd' milestone.