Skip to content

v2: Metadata logging, custom attributes, inline files, and a major version bump #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 142 commits into from
Apr 11, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
142 commits
Select commit Hold shift + click to select a range
49698e4
Separated type/struct fields in dir entries
geky Mar 3, 2018
4c35c86
Added different sources for commits, now with disk->disk moves
geky Mar 4, 2018
73d29f0
Adopted a tiny LISP-like DSL for some extra flexibility
geky Mar 11, 2018
e3daee2
Changed dir append to mirror commit DSL
geky Mar 11, 2018
692f0c5
Naive implementation of resizable entries
geky Mar 11, 2018
ca3d6a5
Made implicity tag updates explicit
geky Mar 11, 2018
f30ab67
Traded enum-based DSL for full callback-based DSL
geky Mar 11, 2018
e4a0cd9
Take advantage of empty space early in dir search
geky Mar 12, 2018
362b0bb
Minor improvement to from-memory commits
geky Mar 13, 2018
03b262b
Separated out version of dir remove/append for non-entries
geky Mar 16, 2018
9273ac7
Added size field to entry structure
geky Mar 16, 2018
fb23044
Fixed big-endian support for entry structures
geky Mar 16, 2018
836e238
Shoehorned in hacky implementation of inline files
geky Mar 17, 2018
d8cadec
Better implementation of inline files, now with overflowing
geky Mar 18, 2018
701e4fa
Fixed a handful of bugs as result of testing
geky Mar 19, 2018
d0e0453
Changed how we write out superblock to use append
geky Mar 23, 2018
ad74825
Added internal lfs_dir_get to consolidate logic for reading dir entries
geky Mar 23, 2018
9555458
Added internal lfs_dir_set, an umbrella to dir append/update/remove o…
geky Mar 27, 2018
6362afa
Added disk-backed limits on the name/attrs/inline sizes
geky Apr 1, 2018
6774276
Expanded inline files up to a limit of 1023 bytes
geky Apr 3, 2018
65ea6b3
Bumped versions, cleaned up some TODOs and missing comments
geky Apr 3, 2018
6ffc8d3
Added simple custom attributes
geky Apr 6, 2018
6c754c8
Added support for atomically committing custom attributes
geky Apr 6, 2018
636c0ed
Modified commit regions to work better with custom attributes
geky Apr 8, 2018
93244a3
Added file-level and fs-level custom attribute APIs
geky Apr 8, 2018
746b909
Added lfs_fs_size for finding a count of used blocks
geky Apr 9, 2018
2a8277b
Added test coverage for filesystems with no inline files
geky Apr 10, 2018
ea4ded4
Fixed big-endian support again
geky Apr 10, 2018
61f454b
Added tests for resizable entries and custom attributes
geky Apr 11, 2018
8070abe
Added rudimentary framework for journaling metadata pairs
geky May 19, 2018
87f3e01
Progressed integration of journaling metadata pairs
geky May 21, 2018
fe553e8
More progress integrating journaling
geky May 22, 2018
0695862
Completed transition of files with journalling metadata
geky May 23, 2018
a3c67d9
Reorganized the internal operations to make more sense
geky May 26, 2018
0405ceb
Cleaned up enough things to pass basic file testing
geky May 26, 2018
0bdaeb7
More testing progress, combined dir/commit traversal
geky May 27, 2018
11a3c8d
Continued progress with reintroducing testing on the new metadata log…
geky May 28, 2018
483d41c
Passing all of the basic functionality tests
geky May 28, 2018
85a9638
Fixed issues discovered around testing moves
geky May 28, 2018
9278b17
Trimmed old names and functions from the code base
geky May 29, 2018
eaa9220
Renamed lfs_entry_t -> lfs_mattr_t
geky May 29, 2018
f458da4
Added the internal meta-directory structure
geky May 29, 2018
116c1e7
Adopted EISDIR as internal error for root path as argument
geky May 29, 2018
e39f7e9
Introduced xored-globals logic to fix fundamental problem with moves
geky May 29, 2018
3ffcedb
Restructured tags to better support xored-globals
geky May 30, 2018
cebf7aa
Switched back to simple deorphan-step on directory remove
geky Jul 2, 2018
b46fcac
Fixed issues with finding wrong ids after bad commits
geky Jul 4, 2018
2ff32d2
Fixed bug where globals were poisoning move commits
geky Jul 8, 2018
d7b0652
Removed old move logic, now passing move tests
geky Jul 9, 2018
c1103ef
Changed type info to be retrieved from name tag instead of struct tag
geky Jul 9, 2018
b7bd34f
Restructured types to use a more flexible bit encoding
geky Jul 9, 2018
fd121dc
Dropped "has id" bit encoding in favor of invalid id
geky Jul 9, 2018
fe31f79
Consolidated find/parent scanning functions
geky Jul 9, 2018
7ad9700
Integrated findscan into fetch as a built in side effect
geky Jul 10, 2018
67d9f88
Combined get functions into one
geky Jul 11, 2018
e3b8678
Modified results from find-like functions to use tags
geky Jul 11, 2018
7c88bc9
Restructured get/traverse functions
geky Jul 12, 2018
2b35c36
Renamed tag functions and macros
geky Jul 13, 2018
5fc53bd
Changed internal functions to return tags over pointers
geky Jul 13, 2018
d3f3711
Cleaned up attributes and related logic
geky Jul 13, 2018
5d24e65
Cleaned up commit logic and function organization
geky Jul 13, 2018
d9a24d0
Fixed move handling when caught in a relocate
geky Jul 17, 2018
392b2ac
Refactored the updates of in-flight files/dirs
geky Jul 28, 2018
3914cdf
Pulled in fixes for additional path corner cases
geky Jul 28, 2018
15d1560
Added support for custom attributes leveraging the new metadata logging
geky Jul 29, 2018
3e246da
Fixed the orphan test to handle logging metadata-pairs
geky Jul 30, 2018
2257060
Fixed test bugs around handling corruption
geky Jul 30, 2018
df1b607
Removed the implicit lfs_t parameter to lfs_traverse
geky Jul 30, 2018
105907b
Cleaned up config usage in file logic
geky Jul 30, 2018
1a58ba7
Fixed ENOSPC issues with zero-granularity blocks
geky Jul 31, 2018
64df0a5
Added orphan bit to xored-globals
geky Jul 31, 2018
112fefc
Added back big-endian support again on the new metadata structures
geky Aug 1, 2018
01d837e
Removed redundant lfs_scan in lfs_init
geky Aug 1, 2018
bd1e0c4
Cleaned up several TODOs
geky Aug 1, 2018
35f68d2
Squished in-flight files/dirs into single list
geky Aug 1, 2018
97f35c3
Simplified the internal xored-globals implementation
geky Aug 4, 2018
3cfa086
Introduced cache_size as alternative to hardware read/write sizes
geky Aug 4, 2018
1941bbd
Cleaned up config options
geky Aug 4, 2018
f369f80
Added tests for global state stealing
geky Aug 4, 2018
a88230a
Updated custom attribute documentation and tweaked nonexistant attrib…
geky Aug 5, 2018
213530c
Changed LFS_ERR_CORRUPT to match EILSEQ instead of EBADE
geky Aug 5, 2018
dbcbe4e
Changed name of upper-limits from blah_size to blah_max
geky Aug 5, 2018
3186e89
Changed littlefs-fuse target for testing purposes
geky Aug 5, 2018
6d0a6fc
Merge remote-tracking branch 'origin/master' into v2-alpha
geky Aug 5, 2018
c3e36bd
Standardized naming for internal functions
geky Aug 5, 2018
7c70068
Added root entry and expanding superblocks
geky Aug 6, 2018
d8f930e
Modified CTZ struct type to make space for erased files in the future
geky Aug 7, 2018
3b3981e
Fixed testing issues introduced by expanding superblocks
geky Aug 7, 2018
10f45ac
Changed lfs_crc to match more common API
geky Aug 7, 2018
20b669a
Fixed issue with big-endian CTZ lists intertwined in commit logic
geky Aug 8, 2018
e4a0d58
Added building blocks for dynamic wear-leveling
geky Aug 8, 2018
126ef8b
Added allocation randomization for dynamic wear-leveling
geky Aug 9, 2018
38011f4
Fixed minor memory leak
geky Aug 11, 2018
21217d7
Dropped lfs_fs_getattr for the more implicit lfs_getattr("/")
geky Aug 12, 2018
d5e8005
Collapsed recursive deorphans into a single pass
geky Aug 13, 2018
a2532a3
Fixed inline files when inline_max == cache_size
geky Aug 13, 2018
4db96d4
Changed unwritable superblock to ENOSPC for consistency
geky Aug 13, 2018
478dcdd
Revisited caching rules to optimize bus transactions
geky Aug 20, 2018
a43f9b3
Modified lfs_dir_compact to avoid redundant erases during split
geky Aug 21, 2018
6db5202
Modified valid bit to provide an early check on all tags
geky Aug 24, 2018
6046d85
Added support for entry insertion
geky Sep 9, 2018
c67a41a
Added support for deleting attributes
geky Sep 9, 2018
617dd87
Added deletion to custom attributes
geky Sep 9, 2018
5eeeb9d
Revisited some generic concepts, callbacks, and some reorganization
geky Sep 11, 2018
7bacf9b
Removed xored-globals from the mdir struct
geky Sep 12, 2018
cf87ba5
Combined superblock scan and fetch of xored-globals during mount
geky Sep 12, 2018
29b8810
Revisited xored-globals and related logic
geky Sep 15, 2018
cafe6ab
Fixed issue with splitting metadata-pairs in full filesystem
geky Sep 15, 2018
d7e4aba
Edited tag structure to balance size vs id count
geky Sep 15, 2018
f010d2a
Fixed issue with reads ignoring the pcache
geky Oct 2, 2018
ad96fca
Changed attr_max to be specific to custom attributes
geky Oct 2, 2018
7af8b81
Changed lookahead configuration unit to bytes instead of bits
geky Oct 3, 2018
aeca766
Switched to strongly ordered directories
geky Oct 4, 2018
97a7191
Fixed issue with creating files named "littlefs"
geky Oct 4, 2018
795dd8c
Fixed mkdir when inserting into a non-end block
geky Oct 5, 2018
c8a39c4
Merge remote-tracking branch 'origin/master' into v2-rebase-part2
geky Oct 21, 2018
4a1b8ae
Fixed issues found by more aggressive rename tests
geky Oct 21, 2018
5b26c68
Tweaked tag endianness to catch power-loss after <1 word is written
geky Oct 22, 2018
dc507a7
Changed required alignment of lookahead_size to 64 bits
geky Oct 22, 2018
a548ce6
Switched to traversal-based compact logic
geky Dec 27, 2018
b989b4a
Cleaned up tag encoding, now with clear chunk field
geky Dec 29, 2018
66d7515
Modified global state format to work with new tag format
geky Jan 4, 2019
51b2c7e
Changed custom attribute descriptors to used arrays
geky Jan 8, 2019
e1f9d2b
Added support for RAM-independent reading of inline files
geky Jan 13, 2019
916b308
Fixed excessive waste from overly large inline files
geky Jan 14, 2019
5fb8fa9
Fixed issue with global state updates being lost during relocates
geky Jan 22, 2019
8cca1b6
Fixed several small issues found during wider testing
geky Jan 22, 2019
173c212
Added scripts/prefix.py for automatically prefixing version numbers
geky Jan 25, 2019
95c1a63
Fixed corner case in block_cycles eviction logic
geky Jan 30, 2019
10dfc36
Fixed issue with long names causing unbounded recursion
geky Jan 31, 2019
512930c
Updated SPEC.md to reflect v2 changes
geky Feb 1, 2019
a064479
Fixed several small issues
geky Feb 12, 2019
7d8f8ce
Enabled -Wextra
geky Feb 12, 2019
4ad09d6
Added migration from littlefs v1
geky Feb 23, 2019
bdff4bc
Updated DESIGN.md to reflect v2 changes
geky Mar 3, 2019
9568f8e
Added v1->v2 migration into CI
geky Apr 2, 2019
a32be1d
Merge remote-tracking branch 'origin/master' into v2-alpha
geky Apr 8, 2019
0b76635
Added better handling of large program sizes (> 1024)
geky Apr 9, 2019
c2c2ce6
Fixed issue with handling block device errors in lfs_file_sync
geky Apr 9, 2019
1ff6432
Added clarification on buffer alignment.
geky Apr 9, 2019
651e14e
Cleaned up a couple of warnings
geky Apr 9, 2019
48bd2bf
Artificially limited number of file ids per metadata block
geky Apr 9, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 59 additions & 6 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,11 @@ script:
- make test QUIET=1

# run tests with a few different configurations
- make test QUIET=1 CFLAGS+="-DLFS_READ_SIZE=1 -DLFS_PROG_SIZE=1"
- make test QUIET=1 CFLAGS+="-DLFS_READ_SIZE=512 -DLFS_PROG_SIZE=512"
- make test QUIET=1 CFLAGS+="-DLFS_BLOCK_COUNT=1023 -DLFS_LOOKAHEAD=2048"
- make test QUIET=1 CFLAGS+="-DLFS_READ_SIZE=1 -DLFS_CACHE_SIZE=4"
- make test QUIET=1 CFLAGS+="-DLFS_READ_SIZE=512 -DLFS_CACHE_SIZE=512 -DLFS_BLOCK_CYCLES=16"
- make test QUIET=1 CFLAGS+="-DLFS_BLOCK_COUNT=1023 -DLFS_LOOKAHEAD_SIZE=256"

- make clean test QUIET=1 CFLAGS+="-DLFS_INLINE_MAX=0"
- make clean test QUIET=1 CFLAGS+="-DLFS_NO_INTRINSICS"

# compile and find the code size with the smallest configuration
Expand Down Expand Up @@ -103,7 +104,7 @@ jobs:
if: branch !~ -prefix$
install:
- sudo apt-get install libfuse-dev
- git clone --depth 1 https://github.com/geky/littlefs-fuse
- git clone --depth 1 https://github.com/geky/littlefs-fuse -b v2-alpha
- fusermount -V
- gcc --version
before_script:
Expand All @@ -113,7 +114,7 @@ jobs:

- mkdir mount
- sudo chmod a+rw /dev/loop0
- dd if=/dev/zero bs=512 count=2048 of=disk
- dd if=/dev/zero bs=512 count=4096 of=disk
- losetup /dev/loop0 disk
script:
# self-host test
Expand All @@ -126,7 +127,59 @@ jobs:
- mkdir mount/littlefs
- cp -r $(git ls-tree --name-only HEAD) mount/littlefs
- cd mount/littlefs
- ls
- stat .
- ls -flh
- make -B test_dirs test_files QUIET=1

# self-host with littlefs-fuse for fuzz test
- stage: test
env:
- STAGE=test
- NAME=littlefs-migration
install:
- sudo apt-get install libfuse-dev
- git clone --depth 1 https://github.com/geky/littlefs-fuse -b v2-alpha v2
- git clone --depth 1 https://github.com/geky/littlefs-fuse v1
- fusermount -V
- gcc --version
before_script:
# setup disk for littlefs-fuse
- rm -rf v2/littlefs/*
- cp -r $(git ls-tree --name-only HEAD) v2/littlefs

- mkdir mount
- sudo chmod a+rw /dev/loop0
- dd if=/dev/zero bs=512 count=4096 of=disk
- losetup /dev/loop0 disk
script:
# compile v1 and v2
- make -C v1
- make -C v2

# run self-host test with v1
- v1/lfs --format /dev/loop0
- v1/lfs /dev/loop0 mount

- ls mount
- mkdir mount/littlefs
- cp -r $(git ls-tree --name-only HEAD) mount/littlefs
- cd mount/littlefs
- stat .
- ls -flh
- make -B test_dirs test_files QUIET=1

# attempt to migrate
- cd ../..
- fusermount -u mount

- v2/lfs --migrate /dev/loop0
- v2/lfs /dev/loop0 mount

# run self-host test with v2 right where we left off
- ls mount
- cd mount/littlefs
- stat .
- ls -flh
- make -B test_dirs test_files QUIET=1

# Automatically create releases
Expand Down
Loading