Skip to content

Add the StorageLite feature #6915

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 18 additions & 10 deletions features/filesystem/bd/ExhaustibleBlockDevice.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,7 @@ int ExhaustibleBlockDevice::program(const void *buffer, bd_addr_t addr, bd_size_
MBED_ASSERT(is_valid_program(addr, size));

if (_erase_array[addr / get_erase_size()] == 0) {
// TODO possibly something more destructive here
return 0;
return BD_ERROR_ERASE_UNIT_WORN_OUT;
}

return _bd->program(buffer, addr, size);
Expand All @@ -79,17 +78,26 @@ int ExhaustibleBlockDevice::erase(bd_addr_t addr, bd_size_t size)
{
MBED_ASSERT(is_valid_erase(addr, size));

// use an erase cycle
if (_erase_array[addr / get_erase_size()] > 0) {
_erase_array[addr / get_erase_size()] -= 1;
}
bd_size_t eu_size = get_erase_size();
while (size) {
// use an erase cycle
if (_erase_array[addr / eu_size] > 0) {
_erase_array[addr / eu_size] -= 1;
}

if (_erase_array[addr / get_erase_size()] == 0) {
// TODO possibly something more destructive here
return 0;
if (_erase_array[addr / eu_size] == 0) {
return BD_ERROR_ERASE_UNIT_WORN_OUT;
}

int err = _bd->erase(addr, eu_size);
if (err) {
return err;
}
addr += eu_size;
size -= eu_size;
}

return _bd->erase(addr, size);
return 0;
}

bd_size_t ExhaustibleBlockDevice::get_read_size() const
Expand Down
4 changes: 4 additions & 0 deletions features/filesystem/bd/ExhaustibleBlockDevice.h
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,10 @@

#include "BlockDevice.h"

enum {
BD_ERROR_ERASE_UNIT_WORN_OUT = -3301,
};


/** Heap backed block device which simulates failures
*
Expand Down
92 changes: 92 additions & 0 deletions features/filesystem/storagelite/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
## StorageLite

StorageLite is a storage solution, providing a key value store like API of set/get data or a reduced POSIX API of open-write-close and open-read-close. It provides fast write and read functions, power failure resilience, security and built-in backup and factory reset support.

Uses: keeping credentials and cryptographic keys, application settings, application states and small application data to be sent to servers.

Hardware: designed for external NOR flash, it can work also on internal flash and with additional block devices on SD cards.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably mention the user will need the FlashSimBlockDevice to use SD cards. Did we measure the performance on SD cards?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FlashSimBlockDevice is simulation of flash over RAM, SD is already flash memory.

it can work also on internal flash and with additional block devices on SD cards.

Please clarify this. Which additional block device and why

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point @geky. Will add it to documentation. Got a few performance numbers, can send them later today. Maybe we should run your benchmark on this as well.
@deepikabahavani The Flash simulator adaptor applies to any block device not having flash attributes (i.e. erasing). SD is such a device thus requiring the flash simulator adaptor.


StorageLite is a lightweight file system storage module for files in external memory. Files are stored and accessed by file-name. StorageLite has the following attributes:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If StorageLite is not a FileSystem should we still call its entries files?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't think of a better name. Entries can be OK, but it can also be misleading.

Copy link
Contributor

@geky geky May 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe "values" or "data" or "blobs" or "elements". Up to you.


- High performance through a small RAM table.
- Security:
- Integrity through file authentication.
- Confidentiality through file encryption.
- Rollback protection.
- Wear level: StorageLite stores files in a sequential order. This concept uses the entire external memory space. It is optimized for NOR flash that supports a limited number of erase per sector.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can also suggest using a SlicingBlockDevice or MBRBlockDevice to not use the entire external memory space since one feature StorageLite is bring is small storage usage.

- Backup: built-in backup and factory reset support.
- Low RAM memory footprint: indexing each file uses 8 bytes of RAM.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe better wording is "indexing using 8 bytes of RAM per file." We want to hint that users can optimize RAM consumption by reducing the number of files.


Please see the [StorageLite design document](./StorageLiteDesign.md) for a detailed design description of StorageLite.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this going to come in with this pr?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will update the link once the handbook PR merges.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah! Didn't see the handbook PR, will look into that 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@offirko, @davidsaada, I realize this is coming late, so not blocking this pr, but could we also have a SPEC.md to cover just the layout of data on disk? Similar to the SPEC.md in LittleFS:
https://github.com/ARMmbed/mbed-os/blob/master/features/filesystem/littlefs/littlefs/SPEC.md

This would help with forward compatibility and tooling.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, layout of data on disk is described in the design part of the handbook documentation.
ARMmbed/mbed-os-5-docs@85f1c15#diff-c3ac614f9d51ef7de09c0f38a05b4e75


#### Flash memory

StorageLite is optimized for NOR flash memory. It is based on dividing the allocated memory storage to areas: active and standby. Data is written to the active area until it becomes full. When the active area becomes full, an internal garbage collection mechanism moves only the updated relevant files to the standby area and switches between the active and standby areas. The new standby area is then erased.

This concept is ideal for low wear leveling of the memory, but it comes with a price of using half of the external memory size.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should say half here. Maybe a quarter? Otherwise users may measure out the minimum amount of storage needed and slice a block device to fit 2*(files+metadata). Unfortunately, in this case StorageLite will be forced to perform a garbage collect every write.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe "about half" would be better here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe "a portion" or "a subset"?

Oh, or "but it comes with a price of not being able to use more than half of the external memory size".

I just don't want to encourage users to use exactly half, thinking that's the optimal amount of storage.

Copy link
Contributor

@geky geky May 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some quick maths:

Writing n byte files to StorageLite with d dynamic files and s static files gives us this equation:

n

If we let r be the ratio of static space to the size of storage, we can build these guys:

s
d

This gives us a nice formula for the cost of writing n byte files with a ratio r of how full the storage is:

n

Lets assume 100 bytes files on a 1MB block device, we can plot this:
image

So at 25% usage, we're seeing on average 200 B written per file, and at 37.5% usage we're seeing on average 400 B written per file.

I think this is fine, compared to every other resource flash is massive. But we should probably encourage users to stick to around 37.5% flash usage. (Feel free to steel these maths for documentation).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The analysis is good, but I don't believe we should get into this detailed level of analysis or recommendation. The main point to emphasize to the user was that StorageLite reduces the storage potential by half to begin with. "but it comes with a price of not being able to use more than half of the external memory size" sounds great to me (-:

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May be useful to have in the design documentation, but up to you 👍


#### APIs

You can either use StorageLite with set- and get-based APIs or it can be defined as StorageLiteFS and work with a limited POSIX open-write or read-close based API. (Only a **single** write or read operation is allowed.)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

- Only a **single** write or read operation is allowed.
+ Only a **single** write or read operation is allowed per file.


##### StorageLite

- `init`: binds StorageLite to a specific block device and constructs and initializes StorageLite module elements.
- `deinit`: deinitialize the StorageLite module.
- `set`: saves data into persistent storage by file name.
- `get`: retrieves file data from storage after authenticating it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're introducing a new storage API, we should provide an abstract interface for it (like the FileSystem class). Maybe KVStore? Feel free to come up with your own name.

The new API should be provided in its own abstract class (no members) and implemented by StorageLite. We'll need an mbed-os-example-kvstore (like mbed-os-example-filesystem), and in the future KVStore tests (we don't have portable filesystem tests so this may all have to come later).

Additionally we should provide KVStore implementations for all FileSystems that can support the new API (maybe this should be in the FileSystem abstract class itself?).

- `remove`: removes an existing file from storage.
- `get_file_size`: returns the file size after authenticating the data.
- `file_exist`: verifies that file exists and authenticates it.
- `get_file_flags`: returns files supported features (such as rollback protection, encryption and factory reset).
- `get_first/next`: provides the ability to iterate through existing files, starting from the first one.
- `factory_reset`: returns storage to contain only files that are backed up and returns those files to their backup version.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we mention reset here?

Out of curiousity does factory_reset fall back to reset if there's no factory? Should these functions be combined?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reset equals format and it should definitely be mentioned. And factory reset indeed falls back to reset if no factory backup file has been created. These shouldn't be combined as they are two totally different features fulfilling different requirements.


##### StorageLiteFS

- Mount or unmount: attaches or detaches a block device to or from the file system.
- Open or close: establishes the connection between a file and a file descriptor.
- Write: Writing data to the file associated with the opened file descriptor (replacing existing data - if exists). You can only call write once.
- Read: reads data from the file associated with the open file descriptor. Read can be called only once.
- Remove: deletes a file name from the file system.
- Reformat: clears a file system, resulting in an empty and mounted file system.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can probably just link to the FileSystem reference (or copy and paste it here):
https://os.mbed.com/docs/v5.8/mbed-os-api-doxy/classmbed_1_1_file_system.html#a387bceba0e4696753a16bab3a501f8a4


#### Usage

##### Using StorageLite

First, define a block device, and define the maximum number of files you want StorageLite to support. Next, create an instance of StorageLite, and initialize it with the above block device and max files setup. Then, you can start saving new files and getting their data.

``` c++
#define MAX_NUM_OF_FILES 50
SPIFBlockDevice spif(PTE2, PTE4, PTE1, PTE5);
StorageLite sl;
sl.init(&spif, MAX_NUM_OF_FILES);
```

##### Using StorageLiteFS

First, define the block device you want StorageLiteFS to use, and create an instance of StorageLite. Next, create a StorageLiteFS instance, giving it the name under which it is cataloged in the filesystem tree, the StorageLite instance it uses and its feature flags. Finally, mount the selected block device onto the StorageLiteFS instance. Now you can start opening new files for write and read.

``` c++
SPIFBlockDevice spif(PTE2, PTE4, PTE1, PTE5);
StorageLite sl;
StorageLiteFS slfs("/sl", &sl, StorageLite::encrypt_flag);
slfs.mount(&spif);
```

#### Testing StorageLite

Run any of the StorageLite tests with the `mbed` command:

```mbed test -n features-storagelite-tests-storagelite-whitebox```
```mbed test -n features-storagelite-tests-storagelite-fs_tests```
```mbed test -n features-storagelite-tests-storagelite-general_tests```

### StorageLite API

TODO: Link to StorageLite class reference once code merges.

### SotrageLite example

TODO: Link to transcluded example once it exists.
Loading