Handle NVStore tests in a low memory environment #7746

davidsaada · 2018-08-09T17:01:53Z

Description

Previous fix to NVStore tests measured how much memory is available for threads (in the multi thread tests) and lowered number of threads accordingly. This was problematic in two aspects:

Other memory allocations weren't checked. In low memory boards, we can fail on them as well (probably the problem we see in #7555).
Lowering number of threads hurts the test credibility.

Therefore, this PR simply checks each and every memory allocation in these tests (using std::nothrow), and skips the remainder of the test (without failing it) if memory can't be allocated.

Pull request type

[x] Fix
[ ] Refactor
[ ] Target update
[ ] Feature
[ ] Breaking change

yossi2le

Looks fine for me. Approved

bentcooke · 2018-08-14T19:36:53Z

@davidsaada After applying this patch, the MOTE_L152RC still hard faults for the nvstore tests:

mbedgt: mbed-host-test-runner: started
[1534274654.46][HTST][INF] host test executor ver. 1.3.1
[1534274654.46][HTST][INF] copy image onto target...
[1534274654.46][COPY][INF] Waiting up to 60 sec for '41000221E2B81D47DDFCCD7A' mount point (current is '/mnt/pci-0000_00_1a_0-usb-0_1_6_1_1-scsi-0_0_0_0')...
[1534274664.73][HTST][INF] starting host test process...
[1534274664.73][CONN][INF] starting connection process...
[1534274664.73][CONN][INF] notify event queue about extra 60 sec timeout for serial port pooling
[1534274664.73][CONN][INF] initializing serial port listener...
[1534274664.73][PLGN][INF] Waiting up to 60 sec for '41000221E2B81D47DDFCCD7A' serial port (current is '/dev/ttyACM0')...
[1534274664.73][HTST][INF] setting timeout to: 60 sec
[1534274664.75][SERI][INF] serial(port=/dev/ttyACM0, baudrate=9600, read_timeout=0.01, write_timeout=5)
[1534274664.75][SERI][INF] reset device using 'default' plugin...
[1534274664.85][SERI][INF] waiting 1.00 sec after reset
[1534274665.85][SERI][INF] wait for it...
[1534274665.86][SERI][TXD] mbedmbedmbedmbedmbedmbedmbedmbedmbedmbed
[1534274665.86][CONN][INF] sending up to 5 __sync packets (specified with --sync=5)
[1534274665.86][CONN][INF] sending preamble '78f58122-fc63-4761-b95f-5c905dcd2719'
[1534274665.89][SERI][TXD] {{__sync;78f58122-fc63-4761-b95f-5c905dcd2719}}
[1534274665.99][CONN][RXD] mbedmbedmbedmbedmbedmbedmbedmbed
[1534274666.04][CONN][INF] found SYNC in stream: {{__sync;78f58122-fc63-4761-b95f-5c905dcd2719}} it is #0 sent, queued...
[1534274666.04][HTST][INF] sync KV found, uuid=78f58122-fc63-4761-b95f-5c905dcd2719, timestamp=1534274666.039608
[1534274666.06][CONN][INF] found KV pair in stream: {{__version;1.3.0}}, queued...
[1534274666.06][HTST][INF] DUT greentea-client version: 1.3.0
[1534274666.08][CONN][INF] found KV pair in stream: {{__timeout;120}}, queued...
[1534274666.08][HTST][INF] setting timeout to: 120 sec
[1534274666.11][CONN][INF] found KV pair in stream: {{__host_test_name;default_auto}}, queued...
[1534274666.11][HTST][INF] host test class: '<class 'mbed_host_tests.host_tests.default_auto.DefaultAuto'>'
[1534274666.11][HTST][INF] host test setup() call...
[1534274666.11][HTST][INF] CALLBACKs updated
[1534274666.11][HTST][INF] host test detected: default_auto
[1534274666.14][CONN][INF] found KV pair in stream: {{__testcase_count;3}}, queued...
[1534274666.17][CONN][RXD] >>> Running 3 test cases...
[1534274666.22][CONN][INF] found KV pair in stream: {{__testcase_name;NVStore: Basic functionality}}, queued...
[1534274666.26][CONN][INF] found KV pair in stream: {{__testcase_name;NVStore: Race test}}, queued...
[1534274666.32][CONN][RXD]
[1534274666.32][CONN][INF] found KV pair in stream: {{__testcase_name;NVStore: Multiple thread test}}, queued...
[1534274666.38][CONN][RXD] >>> Running case #1: 'NVStore: Basic functionality'...
[1534274666.43][CONN][INF] found KV pair in stream: {{__testcase_start;NVStore: Basic functionality}}, queued...
[1534274666.45][CONN][RXD] NVStore areas:
[1534274666.75][CONN][RXD] Area 0: address 0x0803e000, size 4096 (0x1000)
[1534274666.80][CONN][RXD] Area 1: address 0x0803f000, size 4096 (0x1000)
[1534274667.34][CONN][RXD] Max keys 20 (out of 255 possible ones)
[1534274667.75][CONN][RXD]
[1534274667.78][CONN][RXD] ++ MbedOS Fault Handler ++
[1534274667.78][CONN][RXD]
[1534274667.81][CONN][RXD] FaultType: HardFault
[1534274667.82][CONN][RXD]
[1534274667.82][CONN][RXD] Context:
[1534274667.84][CONN][RXD] R0   : 20002008
[1534274667.86][CONN][RXD] R1   : 3803E0F8
[1534274667.87][CONN][RXD] R2   : FFFFFFE8
[1534274667.89][CONN][RXD] R3   : 00000000
[1534274667.91][CONN][RXD] R4   : 3803E0F8
[1534274667.92][CONN][RXD] R5   : 20002008
[1534274667.94][CONN][RXD] R6   : 00000008
[1534274667.95][CONN][RXD] R7   : 20002978
[1534274667.97][CONN][RXD] R8   : 00000000
[1534274667.99][CONN][RXD] R9   : 20002008
[1534274668.00][CONN][RXD] R10  : 000000BF
[1534274668.02][CONN][RXD] R11  : 00000017
[1534274668.04][CONN][RXD] R12  : 80000000
[1534274668.05][CONN][RXD] SP   : 20001FD0
[1534274668.07][CONN][RXD] LR   : 0800B43B
[1534274668.08][CONN][RXD] PC   : 080015C4
[1534274668.10][CONN][RXD] xPSR : 81004400
[1534274668.12][CONN][RXD] PSP  : 20001FB0
[1534274668.13][CONN][RXD] MSP  : 20007FD8
[1534274668.15][CONN][RXD] CPUID: 412FC230
[1534274668.17][CONN][RXD] HFSR : 40000000
[1534274668.18][CONN][RXD] MMFSR: 00000000
[1534274668.21][CONN][RXD] BFSR : 00000082
[1534274668.22][CONN][RXD] UFSR : 00000000
[1534274668.24][CONN][RXD] DFSR : 00000008
[1534274668.26][CONN][RXD] AFSR : 00000000
[1534274668.27][CONN][RXD] BFAR : 3803E0F8
[1534274668.29][CONN][RXD] Mode : Thread
[1534274668.31][CONN][RXD] Priv : Privileged
[1534274668.32][CONN][RXD] Stack: PSP
[1534274668.32][CONN][RXD]
[1534274668.35][CONN][RXD] -- MbedOS Fault Handler --
[1534274668.35][CONN][RXD]
[1534274668.35][CONN][RXD]
[1534274668.35][CONN][RXD]
[1534274668.37][CONN][RXD] ++ MbedOS Error Info ++
[1534274668.42][CONN][RXD] Error Status: 0x80FF013D Code: 317 Module: 255
[1534274668.47][CONN][RXD] Error Message: Unrecoverable fault excaption.
[1534274668.49][CONN][RXD] Location: 0x800BB1B
[1534274668.51][CONN][RXD] Error Value: 0x80015C4
[1534274668.62][CONN][RXD] Current Thread: Id: 0x200022A8 Entry: 0x800D741 StackSize: 0x1000 StackMem: 0x200012A8 SP: 0x20007FA0
[1534274668.65][CONN][RXD] -- MbedOS Error Info --
[1534274786.77][HTST][INF] test suite run finished after 120.69 sec...
[1534274786.78][CONN][INF] received special event '__host_test_finished' value='True', finishing
[1534274786.81][HTST][INF] CONN exited with code: 0
[1534274786.81][HTST][INF] No events in queue
[1534274786.81][HTST][INF] stopped consuming events
[1534274786.81][HTST][INF] host test result(): None
[1534274786.81][HTST][WRN] missing __exit event from DUT
[1534274786.81][HTST][WRN] missing __exit_event_queue event from host test
[1534274786.81][HTST][ERR] missing __exit_event_queue event from host test and no result from host test, timeout...
[1534274786.81][HTST][INF] calling blocking teardown()
[1534274786.81][HTST][INF] teardown() finished
[1534274786.81][HTST][INF] {{result;timeout}}

davidsaada · 2018-08-14T20:24:11Z

@bentcooke If you could connect the board to a raas node, I may have a look at this error from remote.

bentcooke · 2018-08-14T20:29:24Z

@davidsaada it is on a raas node now... address provided by email.

davidsaada · 2018-08-17T20:27:41Z

Rebased following merge of #7670 (as both modify the NVstore tests).

davidsaada · 2018-08-21T15:49:11Z

Rebased again.
And issue reported by @bentcooke is not related to this PR. Plz see discussion in #7534.

davidsaada · 2018-08-27T13:54:37Z

@0xc0170 @kjbracey-arm Can you please review? As the problem in MOTE_L152RC board has long been resolved.

davidsaada · 2018-09-11T10:06:59Z

@0xc0170 @kjbracey-arm Would appreciate a review here, as this PR resolves a few open issues.

jeromecoutant · 2018-09-11T10:55:47Z

Hi
Is it possible to solve merge conflicts ?
Thx

davidsaada · 2018-09-11T11:42:54Z

Is it possible to solve merge conflicts ?

Ah yes. Sorry about that. Whole NVStore directory has moved. Rebased now.

jeromecoutant

Tested OK with DISCO_L072

0xc0170 · 2018-09-18T08:04:35Z

/morph build

mbed-ci · 2018-09-18T10:28:30Z

Build : SUCCESS

Build number : 3093
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/7746/

Triggering tests

/morph test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-09-18T15:49:12Z

Exporter Build : SUCCESS

Build number : 2702
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/7746/

mbed-ci · 2018-09-18T16:50:26Z

Test : FAILURE

Build number : 2882
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/7746/2882

davidsaada · 2018-09-20T09:03:43Z

Fixed test issues in problematic board.

cmonr · 2018-09-22T21:45:12Z

/morph build

mbed-ci · 2018-09-23T00:06:13Z

Build : SUCCESS

Build number : 3129
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/7746/

Triggering tests

/morph test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-09-23T04:49:31Z

Test : SUCCESS

Build number : 2931
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/7746/2931

mbed-ci · 2018-09-23T05:18:36Z

Exporter Build : SUCCESS

Build number : 2737
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/7746/

This was referenced Aug 9, 2018

enable MOTE_L152 for OS5 #7534

Merged

NVStore tests hard fault for MOTE_L152RC #7555

Closed

cmonr requested review from 0xc0170, kjbracey and a team August 10, 2018 00:04

cmonr added the needs: review label Aug 10, 2018

yossi2le approved these changes Aug 12, 2018

View reviewed changes

davidsaada force-pushed the david_nvstore_test_low_mem branch from 65f03c9 to 8f17daa Compare August 17, 2018 20:26

davidsaada force-pushed the david_nvstore_test_low_mem branch 2 times, most recently from 7412310 to dbff75e Compare August 21, 2018 15:46

davidsaada mentioned this pull request Sep 11, 2018

NVSTORE tests with small memory targets #8077

Closed

kjbracey approved these changes Sep 11, 2018

View reviewed changes

davidsaada force-pushed the david_nvstore_test_low_mem branch from dbff75e to 744300f Compare September 11, 2018 11:41

davidsaada force-pushed the david_nvstore_test_low_mem branch 2 times, most recently from 0a05f3c to 2ebf9ff Compare September 12, 2018 10:42

jeromecoutant approved these changes Sep 12, 2018

View reviewed changes

cmonr added needs: CI and removed needs: review labels Sep 12, 2018

Handle NVStore tests in a low memory environment

999042d

davidsaada force-pushed the david_nvstore_test_low_mem branch from 2ebf9ff to 999042d Compare September 20, 2018 08:54

cmonr added ready for merge and removed needs: CI labels Sep 23, 2018

0xc0170 approved these changes Sep 24, 2018

View reviewed changes

0xc0170 added the release-version: 5.10.1 label Sep 24, 2018

0xc0170 merged commit e7cd6ae into ARMmbed:master Sep 24, 2018

0xc0170 removed the ready for merge label Sep 24, 2018

davidsaada deleted the david_nvstore_test_low_mem branch January 14, 2019 10:03

Handle NVStore tests in a low memory environment #7746

Handle NVStore tests in a low memory environment #7746

Uh oh!

Conversation

davidsaada commented Aug 9, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Pull request type

Uh oh!

yossi2le left a comment

Choose a reason for hiding this comment

Uh oh!

bentcooke commented Aug 14, 2018

Uh oh!

davidsaada commented Aug 14, 2018

Uh oh!

bentcooke commented Aug 14, 2018

Uh oh!

davidsaada commented Aug 17, 2018

Uh oh!

davidsaada commented Aug 21, 2018

Uh oh!

davidsaada commented Aug 27, 2018

Uh oh!

davidsaada commented Sep 11, 2018

Uh oh!

jeromecoutant commented Sep 11, 2018

Uh oh!

davidsaada commented Sep 11, 2018

Uh oh!

jeromecoutant left a comment

Choose a reason for hiding this comment

Uh oh!

0xc0170 commented Sep 18, 2018

Uh oh!

mbed-ci commented Sep 18, 2018

Build : SUCCESS

Triggering tests

Uh oh!

mbed-ci commented Sep 18, 2018

Exporter Build : SUCCESS

Uh oh!

mbed-ci commented Sep 18, 2018

Test : FAILURE

Uh oh!

davidsaada commented Sep 20, 2018

Uh oh!

cmonr commented Sep 22, 2018

Uh oh!

mbed-ci commented Sep 23, 2018

Build : SUCCESS

Triggering tests

Uh oh!

mbed-ci commented Sep 23, 2018

Test : SUCCESS

Uh oh!

mbed-ci commented Sep 23, 2018

Exporter Build : SUCCESS

Uh oh!

Uh oh!

davidsaada commented Aug 9, 2018 •

edited

Loading