Which Filesystem is Best for CompactFlash?

HungWeiLo asks: "We're currently using a Debian-based distribution for an embedded device where we're placing our primary kernel and filesystem on a 1GB CompactFlash card. The kernel will be placed in a read-only partition, while the other partition will be used for logging actions in the system and hosting a flatfile database. The concern here is the need to journalize the data (ext2 corrupts pretty badly since we power it on and off), and the need to minimize thrashing of the CompactFlash (we're using industrial-strength, million-write-cycle+ versions, but that can quickly get us into trouble if the filesystem constantly writes to the flash). Does anyone have any experience using filesystems in this situation? Which one should I look into for this type of application? Ext2? Ext3? Reiser? JFFS2? Help!"

A little large for JFFS2, but that is being fixed

[hpa]

Last I talked to David Woodhouse (author of JFFS2), he told me JFFS2 didn't really handle filesystems in the gigabyte range very well, but that that was being remedied in order to support OLPC.

So if you can wait a bit, then JFFS2 is probably the right answer.

Re:Odd, ext2 works well for me

[notarus]

Actually, rereading your original comment, I'm not sure my experiences actually are anything like yours.

While we update a couple files a few times a month, we're not writing a database to the cf over and over.

Still, we like the CF based solution-- if we have a hard ware failure, it's quick to swap out, and if the CF fails, we have archives of the file systems made automatically weekly so it's just a matter of untarring to a disk, re lilo'ing, and install.

Unwriteprotecting a CF card?

[NeMon'ess]

I have a CF card that I use with one of the handful of mp3 players that use them. A few weeks ago I visited my family and took the player and my camera which uses SD. I left the camera cable at home. My uncle and I wanted to try and view my photos on his TV. He has a Canon SLR that takes CF. So I attempted to use the multi-card reader on his printer to copy the pictures from the SD card to the CF. After formatting my CF in his camera, Win2k said the card was write-protected and could not copy my pictures to it. It also wouldn't let me format it. I used my mp3 player to format the card, but Windows still says it's write-protected. Anyone know how I can restore my card? It's only 256MB, but I'm peeved because it's my card, non Canon's. It could have been a 2GB and worth much more. I know this is offtopic. Mods, please maybe just leave this comment be at +1?

Re:Doesn't CF already do this for you?

[oz_paulb]

I believe all CF flash cards do wear-levelling, so you can just drop a FAT (or whatever) filesystem on top of it without worrying about over-using certain sectors (like the FAT/root dir on a FAT filesystem).

It sounds like the OP requires journalling - so I'd suggest just choosing any journalling FS that you're comfortable with (and don't worry about whether/not it re-writes sectors a lot, as the CF card will take care of that for you).

Re:A solution in search of a problem.

[Anonymous Coward]

and your other suggestions are?

do you ever need to delete a file?

[Anonymous Coward]

if you don't need to delete anything--if you're just appending to the log indefinitely, w/ no chance of running out of space--how about a fs that never truly deletes anything?

there was a slashdot story about such a filesystem a while ago:
    http://linux.slashdot.org/article.pl?sid=05/10/04/ 1410241 [slashdot.org]

and a quick googling turned up a paper on the idea:
    http://www.hpl.hp.com/personal/Alistair_Veitch/pap ers/elephant-hotos/elephant.pdf [hp.com]

-- kieran hervold

Damnsmalllinux agrees with you.

[Anonymous Coward]

www.damnsmalllinux.org
If you're using dsl on a compact flash, you're advised to do a frugal install. DSL reads from the cf on boot but then runs completely in memory. The flash is almost never written and only read once per boot. The result is that it should live forever.

As you note, a journaling file system will trash your cf reasonably fast. As for Reiser, he's been charged with murder and the future of the file system is somewhat in doubt. http://en.wikipedia.org/wiki/Hans_Reiser [wikipedia.org]

Ramdisk

[slimjim8094]

Do all of your work in a ramdisk. Flash has write-limitations, so you want to be limiting the writes to it at all costs.

Obviously, any important changes should be written, but any FS should work for that, since you probably only will need to write on database change, and/or OS shutdown.

Re:Beware journaling on flash.

[notarus]

Wear leveling will have the effect of keeping the writes evenly distributed across the flash. What it won't do is save you from reaching your max-write counts. While many flashes are good for 1000+ writes per block, writing a journal entry for each change is going to dramatically increase the speed you reach that number.

Further, I'd point out that you shouldn't be writing to the CF much except during the setup phase. The reason you want a journal is to recover from a failure before the disk is fully written to and synced; if you never write to the disk, what is a journal saving you from?

I personally have great experience with EXT2 on CF, so it's what I recommend.

Re:Unwriteprotecting a CF card?

[Anonymous Coward]

Fine, yeah Anonymous Coward, blah, blah, blah... If slashdot.org would let me log in...

Anyway, had a similar problem with a SD card for a Canon camera. I inserted it into a Sony laptop, and the camera wouldn't recognize the card from that point on. As far as I was able to determine, Canon uses a non-standard format for their cards. I think they set a reserved bit (or register) on the card. It was a pain-in-the-ass, but there are free utils to help.

Here are some links that I found useful.

      http://www.engatiki.org/2005/04/28/4 [engatiki.org]

      http://www.engatiki.org/flashrecover [engatiki.org]

      http://acmedigitalphotography.com/blog/?p=105 [acmedigita...graphy.com]

      http://wwww.mckeay.net/secure/2004/10/using_dd_to_ clone_a_hd.html [mckeay.net]

Good Luck.

ext3 in full journaling mode

[bucksontour]

Im also running a cutdown knoppix OS on an SBC using 1GB CF. The stock distro shipped with 2.4 kernel with no ext3 support and we experienced fs corruption on shutdowns during periods of high disk write activity using ext2. I hand rolled a 2.6.13 kernel with compiled in ext3 support and this has increased realibility when using full jouranal mode (not ordered or writeback). A word of warning, most underlying flash is going DMA enabled; we've been buying Kingston pro elite which was working perfectly in pio mode. Kingston bless them have now changed to samsung DMA enabled flash, and still call the product Pro Elite, its even got the same P/N, but its completly different underlying technology. Our CF slot has pin 34 (DMA pin) NC (not connected). You can set ide=nodma in grub / lilo but we're still seeing CF corruption on DMA enabled CF. Ive changed to Lexar 40X standard - non DMA part and 100% relibility restored. YMMV. But different CF brands seem to behave differently wrt curruption in DMA enabled mode. Ive called Kingston tech support and sandisk both claim all their technology is going this way.

Re:A solution in search of a problem.

[dgatwood]

You'd still have to mount it in synchronous mode so the kernel doesn't do any write caching. External journals only go so far. Journals guarantee filesystem consistency, but not data consistency. So you could still have a journal that says that a transaction was written to disk but that was actually in-flight in the buffer cache. At least flash parts don't do write reordering, so it's not as bad as it could be, but journaling to SRAM is still only half the problem.