Which Filesystem is Best for CompactFlash? 100
HungWeiLo asks: "We're currently using a Debian-based distribution for an embedded device where we're placing our primary kernel and filesystem on a 1GB CompactFlash card. The kernel will be placed in a read-only partition, while the other partition will be used for logging actions in the system and hosting a flatfile database. The concern here is the need to journalize the data (ext2 corrupts pretty badly since we power it on and off), and the need to minimize thrashing of the CompactFlash (we're using industrial-strength, million-write-cycle+ versions, but that can quickly get us into trouble if the filesystem constantly writes to the flash). Does anyone have any experience using filesystems in this situation? Which one should I look into for this type of application? Ext2? Ext3? Reiser? JFFS2? Help!"
Odd, ext2 works well for me (Score:5, Informative)
I run about a dozen machines running pebble (and soon, voyage) which are both debian based CF distros, and we don't have much problem with them at all. They get powered on and off a lot, I do quite a few live updates of specific files, etc, no problems.
Is it possible you're not actually suffering FS corruption but instead having problems with CF that just isn't suited for the task? We started this project using kingston, which is good flash for cameras, but we ran into lots of dead sectors. We've been using Lexar since, with no issues at all (of the 13 machines, i think we've lost 1 sector in 2 years).
let me guess - gaming terminal? (Score:5, Informative)
jffs was designed (Score:2, Informative)
If you did even a small amount of research you would have seen that JFFS2 was designed for flash memory devices, and has journaling. Its a little harder to make file systems for (at least the last time I was doing so) but given your criteria its what you are looking for.
I'd recommend doing experiments (Score:4, Informative)
Don't discount unsophisticated filesystems such as FAT and its variations.
For the read-only filesystem, FAT, MINIX, or even a read-only OS like cramfs might be better under certain circumstances.
Whatever filesystem you use, consider immediate-write-commit for file-system operations or better yet all operations rather than worrying about journaling. Write-on-commit everything is a little slow but it's hard to beat for data integrity after sudden power loss.
As for thrashing due to memory limits - don't use swap space. Ante up for more memory and write your code so it fails gracefully if it is out of RAM.
Consider having your power-down the desktop equivalent of "sleep" or "hibernate" rather than "off." That way, you either never save RAM or only save it at power-off. Use battery-backed-up RAM or NVRAM to make the "sleep" mode if you have to. These also mostly-solve the journaling problem, as you won't have a lot of unexpected fresh-starts.
Re:Unwriteprotecting a CF card? (Score:4, Informative)
You can download the program from here [totalbiscuit.com] - even though its a totally unrelated problem I think this might help ya.
Can't guarantee it'll work or if its not spyware but it's worked for me before
Re:Can any explain how wear-leveling really works? (Score:3, Informative)
Media with built-in wear-levelling (CF Flash, for example) use whatever methods/structures/etc that they want to keep track of flash blocks, how often they've been written/erased/etc (the 'media' is actually storage media (like NAND flash) plus some sort of microcontroller to implement the wear levelling). This underlying physical structure is hidden from the application/filesystem code. Instead, a 'virtual disk drive' of "N" 512-byte logical sectors is presented to the application/FS code (for example, to the FAT filesystem).
When FAT wants to update (example) 'sector 2' of the 'drive', it asks the CF drive to do this. The CF drive may end up moving a bunch of data around, erasing/rewriting physical blocks, discarding bad blocks, etc, and may actually write the updated data to 'block 500' on the flash array. The underlying CF structures keep track of where 'logical sector 2' exists in the physical media - even after things are rearranged.
So, the app/FS code doesn't need to worry about wear levelling - it's taken care of.
On the other hand, if the app/FS code is written with the underlying media type in mind (example: NAND flash), and does its own wear levelling/bad block management, it's possible that this would be more efficient/do a better job than the 'virtual layer' created by media like CF Flash.
Re:A little large for JFFS2, but that is being fix (Score:4, Informative)
Re:Too bad you can't use ZFS (Score:2, Informative)
Furthermore, when blocks inevitably do go bad, ZFS can detect and correct the faulty blocks. At present, this only applies to metadata, though it will be possible to replicate data as well in the future. ZFS uses replicated metadata for pool and filesystem structures, making it nearly impossible to damage an entire filesystem or pool. Even if data blocks are damaged, in addition to maintaining access to all other good data, you will know exactly what has been lost.
While I'm not 100% certain, I don't think most flash uses ECC. Once blocks start going bad, other filesystems will simply fall apart.
CF cards do NOT do atomic writes (Score:5, Informative)
Thus, any classical fixed-location file system (inode or FAT style) is NOT suitable for embedded appliance use on compactFlash cards.
This severely pissed me off, because the essence of wear-leveling is out-of-place writes, and I just assumed that any CF manufacturer with an ounce of brains would implement a two-phase commit, ao each sector write would be atomic: after a power cycle, either you'd see the new contents, or the old contents, but never anything else. The window is narrow, so I hadn't noticed it during development; we had shipped products and got field failures.
It MAY be possible to adapt a block-based journaling FS like Ext3 to this brain-damage, since it can unconditionally replay the journal on power-up and overwrite the problematic sectors. You just need to ensure that single-block corruption can't mess up the journal. or the superblock. And you need to journal the data as well as the metadata.
FS specifically for Flash & embedded systems (Score:5, Informative)
-- Emery Berger [umass.edu], Dept. of Computer Science, UMass Amherst
JFFS2 may be best, but it isn't a good match (Score:5, Informative)
CompactFlash looks like IDE, not raw flash. CompactFlash has built-in wear leveling. It's designed to support the typical FAT filesystem that normal people use. All the fancy wear-leveling things in JFFS2 will be wasted.
Of course, the other filesystems are lame too. The others are optimized for rotating media, so they waste space and CPU time trying to avoid disk seeks.
CompactFlash and similar devices really could use special filesystems optimized for the odd combination of near-free seeks and device-supplied wear leveling.
Re:Odd, ext2 works well for me (Score:5, Informative)
Re:Entire CF Drive should be Read Only! (Score:3, Informative)
These are MRAM chips. Fast as flash but as reliable as ram. Write, page and thrash all you want.
Re:A solution in search of a problem. (Score:2, Informative)
Re:Block Emulation in Compact Flash (Score:2, Informative)
Wikipedia on xD cards [wikipedia.org]
Instructions to fit an xD card to a Mattel Juicebox [elinux.org]
A comment on LWN from Wookey, who knows a lot about flash [lwn.net]
xD card pinout [pinouts.ru]
My best advice is to ask people on the linux-mtd mailing list [infradead.org] for specific details.
Re:Odd, ext2 works well for me (Score:1, Informative)