-
Notifications
You must be signed in to change notification settings - Fork 3k
Filesystem bugs, including volume corruption #5780
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
@deepikabhavnani @geky Please review |
Note, # |
# |
@bmcdonnell-ionx - Thanks for reporting the issues. ChanFS reports FR_OK for end of directory, hence the check should be just
I do not have target device with that big HEAP memory, and testing with SD card will take long. Will confirm once I have verified. |
Yes it will, as per your example files are created in same directory, and while open call all files are scanned in the directory before creation. See https://github.com/ARMmbed/mbed-os/blob/master/features/filesystem/fat/ChaN/ff.cpp#L3046 You can split files in multiple directories to get better performance. We would appreciate a PR to improve performance here, else we might pick it up in future based on priorities. |
OK, so @deepikabhavnani is working on #
I don't anticipate doing so myself, but for reference, how could one improve the performance here? |
@bmcdonnell-ionx - @geky might help you with that, else you can add query in ChanFs community (http://elm-chan.org/fsw/ff/bd/) and see if you can get some help. |
I've published test cases for the following. mbed team (@deepikabhavnani, @geky): Is this enough information for you to troubleshoot the issue?
bug03a-mbed-os-5.7.2-filesystem-corruption (650c265) Notes:
|
Test runs of my unpublished program did not fail or corrupt the filesystem on 2 GB or 4 GB SD Cards, which can't fit the number of files it took to get to the failure demonstrated above. |
I edited my comment above, including the table. I added the version ( I'm experimenting with pre-filling the test directory. I'll update later. |
OK. Finally I published:
mbed team (@deepikabhavnani, @geky) - is this test enough good enough for you to use to troubleshoot? This version doesn't require the external memory. And it runs much more quickly - around 15 minutes on my board. But you must pre-populate the files. To pre-populate the SD Card, with your PC, create a folder A
I recommend you create the folder on your hard drive, so you can just copy it repeatedly for multiple test runs. If using Windows, I recommend using Windows Explorer to copy/paste the folder to the SD Card. (I had an issue where the mbed device isn't seeing all the files under certain circumstances, such as when things were moved from one location to another within the SD Card, or when files were copied to the SD Card using Cygwin. If I can nail it down, I'll report here or in another issue. But Ctrl+C/Ctrl+V on the I did three test runs, on three different units, and on all three it failed to create file (Sorry it took so long. I had tried changing several things at once, and I couldn't reproduce the errors, so I made one change at a time. With test runs taking 1+ days, it took a while...) |
@bmcdonnell-ionx Were you able to create 65534 files successfully on HEAP partition/SD card? I was trying with 8GB SD card formatted using
|
Oops - somehow the call to |
@bmcdonnell-ionx - I already did those changes and tried once. Below is the observation, I will be trying again with format using FatFilesystem::format() Steps:
Output: Next Step: Try format with FatFilesystem::format() and repeat steps |
@deepikabhavnani, you bring up a good point - I did not track which methods I used to format the SD Cards as it related to the tests. I think I mostly used Windows 10 to format them (with "default allocation size"), but occasionally used the
Which changes? Looks like your gist was based on my Rev
I don't have a Linux machine to test with, so I can't duplicate that test. I'm using Cygwin.
Why would that matter? I don't recall it giving me any trouble (Windows 10). |
Yes, this demo code does that with empty files on the HeapBlockDevice. |
@bmcdonnell-ionx I was able to reproduce issue 3 with FatFilesystem::format(), more on that once we find out actual root cause of corruption. Thanks. |
@bmcdonnell-ionx - Can you please verify the fix in PR#5829 for issue 3? |
@deepikabhavnani - Does it matter if the card was formatted by the mbed device or the PC? |
No it should not matter. Block division is done differently by different format utilities, with this case with fat filesystem format we easily got the case where write to 0x800000 block was done. With other methods, it might take more time or some extra file writes. |
I'm running some other testing on my devices now. I'll probably try test Thanks. |
@bmcdonnell-ionx - Thanks for helping us find this issue |
@deepikabhavnani - You're welcome. Thanks for being responsive and fixing things. :) |
@deepikabhavnani - Until now, I've been testing with mbed-os-5.7.2. Should I The merge brings in a lot of other stuff, as shown below.
|
Fix did not work on re-running test, let me see more why I am still getting addr 0x0 for block 0x800000. May be some compiler optimization. |
@bmcdonnell-ionx - Fix in place, you can verify as per your convenience. Thanks |
Where? |
I updated the PR with fix #5829 |
@deepikabhavnani, what do you think of my suggested course of action above (wait & see on # Also, what's the status of issue # For ease of reference, here's our conversation so far on that. I said:
You said:
You said:
I said:
|
I am pretty sure file creation in SD card failed beyond 32765 files on Linux system, hence was unable to reproduce that issue. Also, I don’t have any system with huge heap to try this. Will let you know if I am working back on that issue. |
For issue #3, I would say leave it if you are not able to reproduce it. |
Well, you showed the code here for your proposed fix. I can test it.
|
I am pretty sure file creation in SD card failed beyond 32765 files on Linux system, hence was unable to reproduce that issue.
Was the card formatted by an mbed device, without your PR?
|
Yes |
Were you creating the files directly on the SD card? Could the buggy mbed formatting be the reason why you couldn't create all the files?
|
Bug was not in format, still I will give it a try sometime next week. |
"Buggy mbed formatting" was poor phrasing on my part. I meant that Anyway, if you want to decouple questions, you could try creating the empty files on your HDD, and copy them to the SD card later. |
@deepikabhavnani, re # I see two lines in |
@bmcdonnell-ionx - I tried creation after new format and same result.
Yes, but that change might not help, I missed the return statement for failure case, so checking fname[0] only in case of success makes sense. |
@deepikabhavnani, if you want me to test, can you just put the commit in a branch somewhere, and point me to it? |
@deepikabhavnani, can you give a status update? Do you think you know what needs to be fixed for I'm not clear on the required changes from what you've said here. I'd love to wrap this up and get the changes into Mbed OS 5.7.4. |
@bmcdonnell-ionx As mentioned earlier, the proposed change might not work as I missed the return statement in failure case during analysis. |
Re |
ARM Internal Ref: MBOTRIAGE-296 |
@bmcdonnell-ionx IMHO issues 2 and 3 (slow performance on big directories and corruption of FATFS) are well known issues with FAT file system. |
Apparently [2] is a known inherent issue. [3] was fixed. Agreed, [1] appears to be an issue w/ ChaN FatFs.
Can you document [1] in Mbed OS as a known issue, until ChaN fixes it? I reported it to them (link above), but there's been no response, other than one other user confirming the issue. Do you know if that's the right place, or any other way to encourage them to take action on it? |
@bmcdonnell-ionx Do you still has any concerns regarding closing this bug? |
@yossi2le, thanks for your input. I hadn't realized that ChaN had taken any action on my bug report, since there was no mention in my thread on their forum.
On your [option 1], it doesn't look like the patch has been included in any FatFs release yet. I will try your [option 2] when I get a chance. Then hopefully I can close this with a PR. 😃 |
PR is merged, hence closing this. Please reopen if anything is missed out. |
(Oops, I forgot to close it. Thanks!) |
Uh oh!
There was an error while loading. Please reload this page.
Description
Bug
Target
LPC4088
Toolchain:
GCC_ARM - yes
ARM|IAR - untested
Toolchain version:
mbed-cli version:
1.2.2
mbed-os sha:
2b4ff78
Expected behavior
readdir()
returnsNULL
to terminate a directory listing.FATFileSystem
writes files to SD Card at a relatively consistent speed.FATFileSystem
never corrupts the file system on the SD Card.Actual behavior
readdir()
never returnsNULL
.FATFileSystem
writes.FATFileSystem
(orSDBlockDevice
?) corrupts the file system on the SD Card. (i.e. It can no longer be read byFATFileSystem
, nor by a Windows PC.)Steps to reproduce
I forked ARMmbed/mbed-os-example-filesystem to create demo code for items
1
and2
. There are tags (linked below) for each of these. I don't have code to share for3
yet, but I give details below.All testing was done with an Embedded Artists' LPC4088 QuickStart Board connected to an LPC4088 Experiment Base Board (EBB). The EBB is used for its SD Card slot. Jumpers are configured to use the SPI interface to the SD Card (per Figure 7 in the EBB user guide (PDF)).
I made a small patch to mbed OS so that GCC_ARM will
malloc()
from the LPC4088's external SDRAM like the mbed online compiler does. Demo programs1
&2
use megabytes of memory for aHeapBlockDevice
. You should be able to replace this with anSDBlockDevice
to avoid the large memory use and test on another target device, but of course this is slower to run.Of course, add your choice of
Serial
device to outputprintf()
text.bug02b-mbed-os.5.7.1-write-slowdown-small-files (d891ad3)
fprintf()
to a metadata file in the directory, and a log file in the parent directory.fwrite()
. Filesystem is corrupted after writing to the 8160th file. I observed this particular failure mode on at least two units.fprintf()
. Filesystem is corrupted after filling up the volume. Observed on one unit so far.The text was updated successfully, but these errors were encountered: