Hey again David,

I'm not against block level access: I just think that most cases where you mix low level access to dead space with simultaneous filesystem (or page) level access are edge cases - whether it's RAM or disk you're talking about.

Looking at the 5 suggestions: I asked for realistic, real-world use cases.

1 - Lazy conversion of filesystems on a single medium using 'dead' space on the disk without creating a temporary filename corresponding to those blocks to cover their ass? This is a 'realistic', 'real-world' use case? Show me a program that does this.

2 - Defragging: similarly, if you're moving data around in dead space without safely duplicating it or having a filename pointing to the blocks in use at any given time, you're not being careful. Also, which defraggers have random 3-minute gaps in operation that would even allow GC to kick in?

3 - Raw filesystem image included as a file inside another filesystem: I don't get it - if the blocks for the image are marked as associated with a filename, where's the problem?

4 - UDF format over NTFS old filesystem. Good point, though I would argue that any UDF formatting tool which leaves in a disk in a state where it could be interpreted ambiguously in terms of it's filesystem, is a broken formatting tool.

5 - Interesting theoretical idea, but not 'real world'. this is not actually the case and MS are well aware of SSD tech, as seen in the latest iteration of Windows. A problem for the future though, for anyone adding new tricks to the GC.

Graeme.

"This is of course a spurious comparison. SMB, FTP, and NFS are presented to the operating system as file trees."

This GC only works with NTFS filesystems. If you are operating an SSD device using an NTFS file tree to store data, then you (as a programmer) are not using the drive as a block device as you suggest; you're accessing it (or you should be accessing it) via the abstract file tree.

"Again, RAM is presented to the OS as logical addresses, and it does faithfully restore the data that was stored in those logical addresses."

While there is a difference in the semantics that occur when marking a reference to a memory page as free (and subsequently losing access to the data) and marking a reference to file data as deleted, they are essentially the same operation in terms of saying 'I don't need to store this data now'.

The only difference is that people seem to take block-level access to the disk for granted; we now know with memory that block-level access isn't really a good idea at all, in the interests of protecting memory from accidental or deliberate attempts to misuse it.

The problem, from my perspective, is that your arguments hinge on the idea that marking data as 'deleted, and the filesystem can now overwrite it at some random future point, perhaps instantly or never' (the HDD model) is in some way better than 'deleted, purge at first opportunity' (the SSD model). From my perspective, I'd prefer the latter; at least then I know what's happened to data after it's been marked for deletion.

I'm still keen to see those realistic real-world use cases. If another poster has posted them, can you provide a link?

Graeme.

"followed by a drive being connected to a non-TRIM OS with a write-blocker"

"Now write 10GB of data to the drive overwriting 10GB of the data written earlier."

Write-blocker. Also, here, we did not do any writes to the drive, only reads. I wasn't asking you to suggest a way in which an OS can (generally) overwrite deleted file data, that's easy. I was asking you to explain how that could happen in the experimental setup we described, if you're suggesting it wasn't the drive-based GC that did it.

Graeme.

Hi David

"A SSD that performs automatic garbage collection by interpreting the filesystem in firmware is not, in my opinion, a storage device."

Well - you can hold any opinion you like, but the physical fact of reality is that this device can and is being used to store things quite safely by millions of people. To use an analogy: would you say that a library that reorganises their shelves and throws out items marked for removal from time to time - but only when there's no one using the library at 3AM in the morning - was not really a reliable repository for information? Because in reality, that's how most libraries work. You might hold the opinion that such a library was not really be an 'information storage facility' because in fact, you were arranging to have some books thrown out in such a way that a secret message was conveyed to the trashman. But 99.9999% of the population would consider the loss of such a peculiar backchannel a reasonable tradeoff, for say, a 3-fold to 10-fold improvement in how quickly they can find things on the shelf - which is of course the primary purpose of a library.

The analogy is fair, because the disk only runs GC when the disk isn't in use for several minutes; at the OS level we can use write-through caching to ensure there's no logical data left unwritten at this point.

"Suppose I am a filesystem developer. Suppose I want to modify NTFS in such a way that deleted segments of an NTFS disk layout become (in my modified filesystem) a repository for meaningful data. This is not as absurd a concept as it appears. In my line of work (cryptography), storing actual meaningful data in deleted segments might be something that you want to do, for example in steganography.

This is so far into being an edge case it's not funny. The phrase 'for example in steganography' is not reasonable here - can you give me 5-10 other examples to demonstrate this isn't a one-off edge case? Steganography seems like the only example I can think of, and as a stego researcher myself (see my site) I can tell you that you won't get much capacity or reliability from rearranging deleted files - the OS will need to use that space for something. I think huge numbers of deleted files or metadata entries that are not being overwritten would stand out a bit to a forensic investigator too, given that the second thing they'll look for is deleted files.

"In this sense, it is, by definition, impossible for a valid storage device to implement automatic garbage collection at the filesystem level."

I'm guessing you don't consider networked computers (e.g. SMB shares, FTP sites, NFS mounts) to be storage devices either then, since the remote host will merrily overwrite deleted files with other people's data however it likes there too? Why do you think so many people are willing to use remote hosts to store data when they don't have control over how deleted files are garbage-collected/re-used/arranged below the logical layer?

"Sure, those deleted sectors are safe to erase in an NTFS volume, but how do you know that my operating system is using this NTFS volume as an NTFS volume? What if I'm doing steganography or something where those deleted sectors matter?"

What other somethings do you have in mind?

Thanks for your feedback though, it's interesting to see people's gut reactions to this tech.

I will be very interested if you can enumerate some more realistic examples of why performance-boosting GC is a bad thing, other than an edge case (NTFS file deletion stegosystem) which, I would expect, does not presently exist as an implementation.

Graeme.

p.s. Just thought of another example. RAM. Where you store data in memory logically, and how it is arranged physically - including zeroing of dead pages - are completely out of your control and even out of your view. Does this mean you consider RAM not to be a storage device, since you can't reliably construct a stego side-channel using dead pages of memory?

Hey,

"What you're describing is impossible. It might be possible for some of the more common filesystems"

You seem to have changed your mind rather quickly there?

given the difficulty of supporting NTFS in Linux, I highly doubt that embedded firmware on a drive can parse the NTFS format)

Well, if you think I (and the tech support staff on various SSD manufacturers forums) are wrong, you're welcome to buy an SSD and check for yourself. It's not quite as easy as typing 'it's impossible' a bunch of times, but it's a lot more likely to be correct.

It is utterly impossible in the case of new filesystems. Think about it -- if a piece of hardware predates the creation of ext4, or ext5, or whatever, then how can the hardware understand the filesystem?"

The same way they added GC to older models of SSD drives where it didn't already exist of course, and the same way they update features on any hardware. You flash the firmware with revised code.

"How would it do this without maintaining firmware for all possible types of file systems?"

It wouldn't. But, how many different filesystems are in realworld use on 95%+ of systems? HFS+, ext2/3, NTFS, FAT32?

I can imagine that it does this for simple FAT16 or FAT32, but I doubt they deal with NTFS or EXT3 or HFS.

It does work on NTFS, and if you set up an experimental rig similar to the one we describe at the back of the paper (page 16), hopefully you can satisfy your doubts :-)

Graeme.

"This story comes just 2 weeks after another story on slashdot saying that your data doesn't actually get deleted:"

The two papers are similar in that they both highlight unexpected and poorly documented behaviours of real world SSDs that diverge from traditional hard disk behaviour, and raise issues for digital forensics and data recovery. The irony is that the USENIX authors have shown SSDs may not purge data even when you try your best to make them do so (i.e. a problem for information security); whereas here, we show they may purge data permanently when you don't want or expect them to (i.e. a problem for digital forensics & law).

Graeme

"The laws regarding hard drives will become much closer to the laws regarding biologicals which can alter themselves between collection and trial, and can be altered by the investigative process."

I agree, that's one way it could go; other options might include 'ignorance is bliss, except for defendants' and 'attempt to legislate ineffectively against a moving target'.

Broadly speaking, I think a can of worms will be opened in the next few years in regard to the legal idea of volition and intention, as computers increasing come under the control of viruses, self-repairing computer systems and so on (this is arguably an example of a self-repairing system).

Graeme.

"To clarify - is this deletion occurring because the drive analyzed the filesystem metadata and decided it would be a good idea to zero some sectors, or is it an asynchronous response to a fast TRIM command applied to an extent covering the whole drive?"

It's not TRIM. We used a non-TRIM OS and a write blocker for this reason. The manufacturer has mentioned a garbage collector that works with NTFS, and the only way it could work is by zero-ing areas of the SSD using information gleaned from the metadata. Hope this clarifies the situation for you! :-)

Graeme.

" but standards of evidence will emerge over time."

Agreed! But, surely this will only happen if courts realise that new standards *need* to emerge because there is a problem with the old ones not keeping up with technology?

Graeme.

"Is there a list of SSDs that do this?"

P64 Corsair and OCZ vertex 2 are known to have GC ; it's probably worth checking firmware notes for any drive you're thinking of buying.

Graeme.

Who would buy it, knowing that the feature could be used against them? Also, there's the legacy problem.

"Your honour, the defendant knowingly purchased an old SSD drive rather than one with ATA Freeze!"

Graeme.

You're welcome, it's great to hear people's thoughts about our work and to have the opportunity to get the message out about this result to a wider audience.

Graeme.