Implement FlashSimBlockDevice - flash simulated block device over RAM #6559

davidsaada · 2018-04-06T13:20:25Z

Description

Implement a simulated flash block device over RAM. This will allow us to test features that use flash block devices (like StorageLite) on targets that don't have a real flash block device (practically all targets).

Pull request type

[ ] Fix
[ ] Refactor
[ ] New target
[X ] Feature
[ ] Breaking change

geky · 2018-04-06T14:35:37Z

Ah! This looks great.

Though do you think it would be possible to make this a block device adapter? That way you could use it on top of any block device without a proper erase:

// Flash on heap
HeapBlockDevice heap(512, 512*1024);
FlashSimBlockDevice sim(&heap);

// Flash on I2C EEPROM
I2CEEBlockDevice i2c(D11, D12);
FlashSimBlockDevice sim(&i2c);

Thoughts? Sorry if this is more effort to implement.

davidsaada · 2018-04-06T14:41:20Z

Though do you think it would be possible to make this a block device adapter? That way you could use it on top of any block device without a proper erase.
...
Thoughts? Sorry if this is more effort to implement.

No big deal. This is indeed a cleaner solution.

davidsaada · 2018-04-06T15:09:18Z

Updated as a BD adaptor.

geky · 2018-04-06T15:34:05Z

features/filesystem/bd/FlashSimBlockDevice.cpp

+#include <algorithm>
+#include <stdlib.h>
+
+static const uint8_t blank = 0xFF;


Do you think we should make this configurable in the constructor? Since this variable is the point of the class.

Good question. Would argue that this value is device specific, and that a user of either this class or a real flash block device would need to read this value via the get_erase_value API, hence wouldn't be interested in the predefined value.

In that case I suggest we should add an API to block device class to read this value from block device

There is such an API - get_erase_value, and this class implements it (just returns blank).

Sorry missed that. If it is device specific and not application/user specific we can have a macro and target device based on the RAM in hardware can define this macro in targets.json macro section. But I am not sure if any RAM device will have erase state as 0.

You are right. If the underlying BD's erase value is 0xFF, this will not work well (as we will allow programming of unerased blocks). Will add a parameter to the constructor (with a default value of 0xFF).

geky · 2018-04-06T15:38:01Z

features/filesystem/bd/FlashSimBlockDevice.cpp

+        return ret;
+    }
+    _blank_buf_size = align_up(min_blank_buf_size, _bd->get_read_size());
+    _blank_buf_size = align_up(_blank_buf_size, _bd->get_program_size());


non-blocker: Program size must be a multiple of the read size for the BlockDevice API, so you probably don't need both of these align-ups.

Maybe not blocking, but also not very nice. Will fix.

geky

Looks good to me 👍 Thanks for putting this together

deepikabhavnani · 2018-04-06T16:10:18Z

features/filesystem/bd/FlashSimBlockDevice.cpp

+    bd_size_t curr_size = size;
+
+    while (curr_size) {
+        bd_size_t read_size = std::min(_blank_buf_size, curr_size);


Query: What should be the value of _blank_buf_size in case read size of block device is 1 byte and program size is 512 bytes. ?

From code it looks like blank_buf size and buffer parameters are to find out if the buffer is blank to be programmed then program it else erase (which is re-program with 0x0/0xFF) first.
If my understanding is correct then you don;t have to consider read size at all, it should be atleast program size big always.

Right. Also raised by @geky. Will fix.

deepikabhavnani · 2018-04-06T16:16:23Z

features/filesystem/bd/FlashSimBlockDevice.cpp

+#include <stdlib.h>
+
+static const uint8_t blank = 0xFF;
+static const bd_size_t min_blank_buf_size = 32;


I think even min_blank_buf_size should be configurable

Can you explain? Was trying to define a value that should be small enough, so it wouldn't effect heap usage. Do you think it should be smaller? A larger value can make the blank programming/checking a bit more efficient, but this seems negligible to me.

It depends on the use case of this class. If you have it as variable, we keep it configurable for application to have more flexibility.

Ideally for me it should be equivalent to program size and not separate parameter. (As in case of real flash only if entire programmable block is empty than we can say it is free and not by checking few bytes of block)

Ah, I think you missed the purpose of this buffer. If you look at the code (both in erase and program APIs) you'll see that the entire prog_size is checked. This buffer is just used as an intermediate one for this purpose, read/programmed in a few iterations until prog_size is reached.

Yeah I am kind of confused with the usage of this buffer, instead of having it equivalent to program size it is less to save on heap space, and since it is RAM the iterations will be fast. But in the test example program size (8) is less then blank buffer size (32), shouldn't we cap it to program size.
If you would have used defines, and static allocation I would not have made this argument, but we are using dynamic allocation here.

Capping it to program size will make it extremely inefficient if program size is 1 (saw flash devices with such parameters), meaning that both program and erase functions will read/write chunks of 1. Hence the minimum of 32, which should work even if program size is 1.

deepikabhavnani

Looks good to me 👍

davidsaada · 2018-04-06T18:12:59Z

Pushed a fix with the following:

Removed redundant alignment to read_size
Added erase value as a parameter to the constructor

0xc0170 · 2018-04-09T15:35:26Z

Looks good to me 👍

Can you please use approve/request changes rather than comment (helps to see the actual status in reviewers) ?

geky

Looks great 👍
This will be a nice block device wrapper to have

0xc0170 · 2018-04-10T12:35:13Z

waiting for @SenRamakri approval, meanwhile

/morph build

mbed-ci · 2018-04-10T13:11:48Z

Build : SUCCESS

Build number : 1707
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/6559/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

0xc0170 · 2018-04-10T13:17:43Z

@AnotherButler Please review documentation

davidsaada · 2018-05-06T04:04:34Z

Sorry, figured out the problem (pointer increment in an optimized condition). Should be OK now.
/morph build

mbed-ci · 2018-05-06T04:43:03Z

Build : SUCCESS

Build number : 1923
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/6559/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

mbed-ci · 2018-05-06T06:08:25Z

Test : SUCCESS

Build number : 1743
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/6559/1743

mbed-ci · 2018-05-06T06:49:40Z

Exporter Build : SUCCESS

Build number : 1568
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/6559/

0xc0170 · 2018-05-07T10:55:28Z

Waiting for approvals

@davidsaada Docs will be added here as a reference (handbook PR) ?

davidsaada · 2018-05-08T13:17:22Z

Pushed astyle related fixes

davidsaada · 2018-05-09T09:29:55Z

Doc PR here.

cmonr · 2018-05-15T15:30:47Z

/morph build

cmonr · 2018-05-15T15:31:51Z

Requesting re-review from reviewers. Should be quick.

@0xc0170 @geky @AnotherButler @deepikabhavnani @SenRamakri

mbed-ci · 2018-05-15T16:39:50Z

Build : SUCCESS

Build number : 2028
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/6559/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

geky

Looks great to me 👍

mbed-ci · 2018-05-16T03:58:24Z

Test : FAILURE

Build number : 1834
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/6559/1834

mbed-ci · 2018-05-16T07:00:58Z

Exporter Build : SUCCESS

Build number : 1677
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/exporter/6559/

davidsaada · 2018-05-16T09:34:55Z

@cmonr @0xc0170 Looks like the failure is CI related (no space left on disk on couple of the failed devices). Would appreciate a morph test restart.

0xc0170 · 2018-05-16T11:06:33Z

That is correct, we noticed the failures today morning. We will restart all jobs as soon as the devices are back running (just restart does not help in this case).

deepikabhavnani · 2018-05-16T15:15:26Z

features/filesystem/bd/FlashSimBlockDevice.cpp

+    _blank_buf_size = align_up(min_blank_buf_size, _bd->get_program_size());
+    if (!_blank_buf) {
+        _blank_buf = new uint8_t[_blank_buf_size];
+        MBED_ASSERT(_blank_buf);


Non blocking and can be fixed in future - New will assert if memory allocation fails, additional check is not needed. Do you intent to use malloc instead here?

Thanks, will remove the redundant check. And no - will keep using new[].

cmonr · 2018-05-16T17:59:06Z

/morph test

cmonr · 2018-05-17T01:09:13Z

/morph test

mbed-ci · 2018-05-17T08:13:31Z

Test : SUCCESS

Build number : 1853
Test logs :http://mbed-os-logs.s3-website-us-west-1.amazonaws.com/?prefix=logs/6559/1853

0xc0170 added the needs: review label Apr 6, 2018

0xc0170 requested review from geky and deepikabhavnani April 6, 2018 14:13

davidsaada force-pushed the david_flash_sim_bd branch from 0aa6320 to 0dfae76 Compare April 6, 2018 15:08

davidsaada force-pushed the david_flash_sim_bd branch from 0dfae76 to 16391e8 Compare April 6, 2018 15:33

geky reviewed Apr 6, 2018

View reviewed changes

geky previously approved these changes Apr 6, 2018

View reviewed changes

deepikabhavnani reviewed Apr 6, 2018

View reviewed changes

deepikabhavnani previously approved these changes Apr 6, 2018

View reviewed changes

davidsaada dismissed stale reviews from deepikabhavnani and geky via 4a169b6 April 6, 2018 18:12

davidsaada force-pushed the david_flash_sim_bd branch from 16391e8 to 4a169b6 Compare April 6, 2018 18:12

davidsaada force-pushed the david_flash_sim_bd branch 2 times, most recently from b8609c1 to cb992de Compare April 8, 2018 10:58

0xc0170 requested a review from SenRamakri April 9, 2018 15:34

deepikabhavnani previously approved these changes Apr 9, 2018

View reviewed changes

geky previously approved these changes Apr 9, 2018

View reviewed changes

0xc0170 added needs: review and removed needs: review labels Apr 10, 2018

0xc0170 requested a review from AnotherButler April 10, 2018 13:17

0xc0170 added needs: review and removed needs: work labels May 7, 2018

Implement FlashSimBlockDevice - flash simulated block device adaptor

c3e3999

davidsaada force-pushed the david_flash_sim_bd branch from 38cea35 to c3e3999 Compare May 8, 2018 13:15

cmonr added the release-version: 5.9.0-rc1 label May 14, 2018

deepikabhavnani approved these changes May 15, 2018

View reviewed changes

davidsaada mentioned this pull request May 15, 2018

Add the StorageLite feature #6915

Closed

0xc0170 approved these changes May 15, 2018

View reviewed changes

geky approved these changes May 15, 2018

View reviewed changes

deepikabhavnani reviewed May 16, 2018

View reviewed changes

cmonr merged commit 4b72158 into ARMmbed:master May 18, 2018

cmonr removed the needs: review label May 18, 2018

davidsaada deleted the david_flash_sim_bd branch July 9, 2018 13:00

Implement FlashSimBlockDevice - flash simulated block device over RAM #6559

Implement FlashSimBlockDevice - flash simulated block device over RAM #6559

Uh oh!

Conversation

davidsaada commented Apr 6, 2018

Description

Pull request type

Uh oh!

geky commented Apr 6, 2018

Uh oh!

davidsaada commented Apr 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidsaada commented Apr 6, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

geky left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deepikabhavnani left a comment

Choose a reason for hiding this comment

Uh oh!

davidsaada commented Apr 6, 2018

Uh oh!

0xc0170 commented Apr 9, 2018

Uh oh!

geky left a comment

Choose a reason for hiding this comment

Uh oh!

0xc0170 commented Apr 10, 2018

Uh oh!

mbed-ci commented Apr 10, 2018

Triggering tests

Uh oh!

0xc0170 commented Apr 10, 2018

Uh oh!

davidsaada commented May 6, 2018

Uh oh!

mbed-ci commented May 6, 2018

Triggering tests

Uh oh!

mbed-ci commented May 6, 2018

Uh oh!

mbed-ci commented May 6, 2018

Uh oh!

0xc0170 commented May 7, 2018

Uh oh!

davidsaada commented Apr 6, 2018 •

edited

Loading

0xc0170 commented May 16, 2018 •

edited

Loading