Changes in HDD Sector Usage After 30 Years

freitasm writes "A story on Geekzone tells us that IDEMA (Disk Drive, Equipment, and Materials Association) is planning to implement a new standard for HDD sector usage, replacing the old 512-byte sector with a new 4096-byte sector. The association says it will be more efficient. According to the article Windows Vista will ship with this support already."

Comment removed

[account_deleted]

Comment removed based on user account deletion

No, that's not 'sector'

[wesley96]

You're thinking of 'cluster'. This is tied to the file system that is actually used on the disk. Even with the current 512-byte sector, a normal NTFS partition of, say, 200GB, uses 4KB cluster and a single file takes up a minimum of 4KB already.

Re:That's nice

[Beryllium Sphere(tm)]

NTFS will write something that small into the MFT.

Re:That's nice

[AuMatar]

OSes allocate disk space by the sector. If the size of a file is less than 1 sector, the rest is wasted. This is called internal fragmentation. Every file wastes sector_size-file_size%sector_size bytes. On average, thats sector_size/2 bytes per file.

Re:That's nice

[ars]

Um, it already does take up 4K or more. Unless you have a hard disk smaller then 256MB.

See: http://www.microsoft.com/technet/prodtechnol/winxp pro/reskit/c13621675.mspx [microsoft.com] and scroll down to Table 13-4

If you notice, in most of the useful cases the custer size is 4K. Making the hard disk match this seems like a good idea to me.

And EXT2 also uses a 4K block size.

Also remember it's for large disks, no FS that I know of supports a cluster (or block) size smaller then 4K for large disks.

Re:That's nice

[Foolhardy]

Actually, if you're using NTFS, the data will be stored directly in the file entry in the MFT, taking zero dedicated clusters or sectors. The maximum size for this to happen is like 800 bytes.

Here [ntfs.com]'s a short description of how NTFS allcates space. On volumes larger than 2GB, the cluster size (the granularity the FS uses to allocate space) was 4k already unless you specified something else when formatting the drive. Also, Windows NT has supported disk sector sizes larger than 512 bytes for a long time; it's just that anything else has been rare.

Re:Ah, error correction.

[Ark42]

Hard drives do the same thing - for each 512 bytes of real data, they actually store near 600 bytes onto the disk with information such as ECC and sector remapping for bad sectors. There is also tiny "lead-in" and "lead-out" areas outside each sector which usually contain a simple pattern of bits to let the drive seek to the sector properly.
Unlike CD-ROMs, I don't believe you can actually read the sector meta-data without some sort of drive-manufacturer-specific tricks.

It's all about Format Efficiency

[alanmeyer]

HDD manufacturers are looking to increase the amount of data stored on each platter. With larger sector sizes, the HDD vendor can use more efficient codes. This means better format efficieny and more bytes to the end user. The primary argument being that many OSes already use 4K clusters.

During the transition from 512-byte to 1K, and ultimately 4K sectors, HDDs will be able to emulate 512-byte modes to the host (i.e. making a 1K or 4K native drive 'look' like a standard 512-byte drive). If the OS is using 4K clusters, this will come with no performance decrease. For any application performing random single-block writes, the HDD will suffer 1 rev per write (for a read-modify-write operation), but that's really only a condition that would be found during a test.

Re:Hrm, that kind of makes sense...

[Anonymous Coward]

You're talking bullshit. In SCSI/SATA you can read/write big chunks of data (even 1MB) in just one command. Just read the standards.

Seems good to me.

[mathew7]

Almost all filesystems I know of use at least 4Kb clusters. NTFS does come with 512 byte on smaller partitions.
LBA accesses on sector boundaries, so for larger HDD's, you need more bits (currently 28-bit LBA, which some older bioses support, means a maximum of 128GB- 2^28*512=2^28*2^9=2^37) Since 512-bytes were used for 30 years, I think it is easy to assume it will not last for 10 more years (getting to LBA32 limit). So why not shave off 3 bits and also make it an even number of bits (12 against 9).
Also there is something called "multible block access" where you make only one request for up to 16 (on most HDD's) sectors. For 512-byte sectors you have 8K, but for 4K sectors that means 64K. Great for large files (IO overdead and stuff).
On the application side this sould not affect anyone using 64-bit sizes (since only the OS would know of sector sizes), as for 32-bit sizes it already is a problem (4G limit).
So this sould not be a problem because on a large partition you will not have too much wasted space (i have around 40MB wasted space on my OS drive for 5520MB of files, and I would even accept 200MB)

Re:Cluster size?

[scdeimos]

I thought cluster sizes were already 4KB for efficiency, and LBA for larger drive sizes.

Cluster sizes are variable on most file systems. On our NTFS web servers we tend to have 1k clusters because it's more efficient to do it that way with lots of small files, but the default NTFS cluster size is 4k. LBA is just a different addressing scheme at the media level to make a volume appear to be a flat array of sectors (as opposed to the old CHS or Cylinder Head Sector scheme).

Re: Apple in the forground again

[n.wegner]

You could have added MS with FAT32 and NTFS. The problem is we're not talking about filesystem cluster sizes, which are software-configurable, but the disks' actual sector size, which is hardware that HFS+ has no effect on.

Re:What's the case for Linux?

[Anonymous Coward]

All major Linux file systems (except XFS) already support arbitrary sector sizes up to 4096 bytes, e.g. for s/390 Mainframes that traditionally use 4096 byte sectors on Linux.
The poeple who would need to write support for this are Jeff Garzik (libata) and James Bottomley (scsi). It's not that this would require a terribly complicated patch though.

Re: Apple in the forground again

[Anonymous Coward]

On the filesystem layer everbody moved the block size up more than 15 years ago. Even Microsoft's POS FAT32 has a 4096 byte cluster. This time Microsoft was only 10 years behind Unix. We're talking hardware layer here. If you can address 2^48 sectors (or whatever is the limit these days) and your sector size is 8x you can have much bigger disks. Think in term of SAN arrays and you'll see that such a need might be here right now on big iron already.

Re:Because of R-M-W

[Mortlath]

Want to write a single byte? Then read 4MB, modify 1 byte, and write 4MB back to the disk.

That's why there is a system (Level 1, 2, and main memory) cache. Write-backs to the physical disk only occur when needed. That doesn't mean that 4MB would be a good sector size; it just means that write-backs are not the issue to consider here.

Re:No, that's not 'sector'

[Anonymous Coward]

Even with the current 512-byte sector, a normal NTFS partition of, say, 200GB, uses 4KB cluster and a single file takes up a minimum of 4KB already

No. If the file can fit into the MFT record (resident file) then it takes 0 bytes outside of the file's metadata, which is usually 1 KB altogether. The maximum size of such files are usually 700-800 bytes. Though not many other filesystems have this capability.

Re:4MB

[Alioth]

4Kbyte is the size of a page of memory on all modern architectures. Given all modern operating systems use demand page loading of executables, and implement paging (swap space), a sector size that matches the size of a memory page will probably result in better performance.

Re:30 years and now it's bumped up only 8x?

[Alioth]

All modern operating systems do demand page loading of executables and use paging space on disk (the swapper). Memory pages are all 4Kbyte on all the CPU architectures we are using at the moment in a personal computer. Therefore, 4Kbyte is probably the ideal size (since now loading a page into memory takes only one read command instead of 8). Making it bigger than 8Kbyte would complicate VMM design (since if you only need to load one page, you now wind up loading two and having to throw one away, or at best, you'd wait twice as long while 8kbyte loads instead of 4kbyte).

Re:Hrm, that kind of makes sense...

[Anonymous Coward]

I'm pretty sure he was talking about operations performed by the drive's internal controller, not those sent through the interface cable.

Re:Ah, error correction.

[baadger]

NTFS has a cluster/allocation size from 512 bytes to 64K. This determines the minimum possible ondisk filesize, but I don't think it has too much to do with the sector size.

Re:You're all complaining about tiny files...

[mgblst]

It is not just files that are less than 4k. It is almost all small files. Think about a 5k file, that now uses 8k. - almost 40% waste. A 9k file uses 12k - about 25% waste. So the more small files you get, the more waste. The larger files you get, the less waste.

Which is good, you don't really want lots of small files anyway.

If you are using windows, you can see how much is space is wasted at the moment, just right click on a directory, and it will tell how much data is in the files, and how much disk space is actually used. It never really gets much.

Re:4MB

[DavidRawling]

Sorry, 4Kb is 4 Kilobits, not Kilobytes. Bytes is abbreviated with a capital B to distinguish from bits - so we are looking at 4KB sectors (32Kb).

On the other hand, don't you just love the fact that capacity is in powers of 2 when dealing at the sector level, but powers of 10 at the device level - and this from the same organisations!

Re:Ah, error correction.

[alexhs]

Unlike CD-ROMs, I don't believe you can actually read the sector meta-data

What are you calling meta-data ?
CDs also have "merging bits", and what is read as a byte is in fact coded on-disk as 14 bits, and you can't read C2 errors either, that are beyond the 2352 bytes that really are all used as data on an audio CD, an audio sector being 1/75 of a second, 44100/75*2(channels)*2(bytes per sample) = 2352 bytes and it has correction codes in addition too. You can however read subchannels (96 bytes / sector)

When dealing with such low-level technologies, reading bits on disk doesn't mean anything as there really are no bits on the disc, just pits and lands (CD) or magnetic particles (HD) causing little electric variations on a sensor, then no variation is interpreted as 0 and a variation is interpreted as a 1, and you need variations even when writing only 0's as a reference clock.

without some sort of drive-manufacturer-specific tricks.

Now of course, as you cannot change HD platters within different drive with different heads like you can do with a CD, each manufacturer can (and will !) encode differently. It has been reported that hard disks with the same reference wouldn't "interoperate" exchanging the controller part because of differing firmware versions, while the format is standardized for CDs or DVDs.

they actually store near 600 bytes

(that would be 4800 bits) In that light, they're not storing bytes, just magnetizing particles. Bytes are quite high-level. There are probably more than a ten thousands magnetic variations for a 512 byte sector. What you call bytes is already what you can read :) But there is more "meta-data" than that.

Here's an interesting read [laesieworks.com] quickly found on Google just for you :)

Re:Ah, error correction.

[Glooty-Us-Maximus]

It doesn't, other than the FS block size should be a multiple of the disk sector size to avoid wasting extra read/writes to access/store a FS block, as well as to avoid wasting space storing an FS block.

Size != storage

[tomstdenis]

You're all missing one key point. Your 512 byte sector is NOT 512 bytes on disk. The drive stores extra track/ecc/etc information. So a 4096-byte sector means less waste, more sectors, more useable space.

Tom

Re:That's nice

[91degrees]

Uhmm... NO!

This is a quick and dirty hack to check that the generated data is correct. I'm not going to spend weeks designing a data file format, and an API plus conversion tools to export the files to an excel compatible format.just because I've got an inefficient file system.

A new hard drive would be a better investment. Or alternatively just ignore the problem since NTFS seems to hande these adequately.

And sometimes its simply impossible to write a solution that will work like this. Some applications require a large number of discrete files.

Re:It can't be.

[kthejoker]

Slight pedeantry:

Actually, it's 7200rpm, not rps. You get 120rps, so a platter rotation is actually 1/120 second.

Re:Hrm, that kind of makes sense...

[darkmeridian]

Grandparent is discussing "native command queueing", where the hard disk will parse the OS read/write calls and stack them in a way that optimizes hardware access. Pretend there are three consecutive blocks of data on the hard drive: 1, 2, and 3. The OS calls for 1, 3, and then 2. Instead of going three spins around, NCQ will read the data in one spin in 1, 2, 3 order but then toss it out to the OS in 1, 3, 2 order. Now, I'm not sure how much higher sector sizes will affect NCQ capability, because I thought was limited by the amount of hardware cache.

Re:4MB

[hackstraw]

4Kbyte is the size of a page of memory on all modern architectures.

Huh? Which modern architectures?

The only systems I run that still have 4k page sizes are x86 systems.

x86-32 = 4k
x86-64 = 4k
G4,G5 = 4k
alpha (64bit) = 8k
sparc (64bit) = 8k
ia64 = 16k

and at least on the ia64 platform the page size is configurable at compile time.

History of the 512-byte Sector Size

[John_Sauter]

In 1963, when IBM was still firmly committed to variable length records on disks, DEC was shipping a block-replacable personal storage device called the DECtape [wikipedia.org]. This consisted of a wide magnetic tape wrapped around a wheel small enough to fit in your pocket. Unlike the much larger IBM-compatible tape drives, DECtape drives could write a block in the middle of the tape without disturbing other blocks, so it was in effect a slow disk. To make block replacement possible all blocks had to be the same size, and on the PDP-6 DEC set the size to 128 36-bit words, or 4608 bits. This number (or 4096, a rounder number for 8-bit computers) carried over into later disks which also used fixed sector sizes. As time passed, there were occasional discussions about the proper sector size, but at least once the argument to keep it small won based on the desire to avoid wasting space within a sector, since the last sector of a file would on average be only half full.

Re:You got modded up funny but?

[ArsenneLupin]

Won't this also affect lilo and the like?

... and it will affect FAT. Not only does FAT use 512 byte sector sizes, but it also makes sure almost the entirety of the filesystem in aligned on an odd boundary of sectors.

(Boot sector is one, so we start off odd right after boot sector. There are usually 2 FAT copies (even), so after FAT offset stays odd. For root directory size, there is usually no compelling reason to make it an even size, however usually Windows makes it an even size anyways, guaranteeing that start of cluster space stays odd).

So, to make a long story short, even if cluster size is a multiple of 4K, this wont help, because it is oddly aligned (meaning that each write of a 4K cluster would always straddle 2 sectors!)

Presumably, Windows will make appropriately parametrized FAT systems once these disks become available, but there will be implications when restoring old FAT images on the new drives.

BIOSes will also need to deal with these disks, or how will you be able to boot if you replace your old PC's hard disk with a 4K sector disk, while still keeping the old motherboard?

And even if the BIOS can deal with it, forget about dd'ing your old system over to the new disk, because of the FAT issue mentioned above.

Re:4MB

[John Courtland]

Yeah the PSE bit (bit 4) in CR4, here's some info: http://www.ddj.com/documents/s=961/ddj9605n/ [ddj.com]

Re:What's the case for Linux?

[Alan Cox]

Linux has supported media with 4K and 2K blocksize for some years (about 7 I think offhand). 2K media comes up with optical disks a lot.

Re:Because of R-M-W

[jadavis]

Many applications require that write cache is flushed.

In his example, let's say it was a text editor. You change one letter in a document, and save it, it must sycnhronously write the sector to disk, to the actual physical media. Otherwise, if the system crashes, you lose it, and most people don't like that in a text editor.

Write cache at the disk level can be very bad. Databases may have no way of knowing that write cache is enabled, and tell you that your transaction is comitted when it's really not. Of course, battery-backed RAID controllers are safe, but the consumer level disks with write cache enabled can mean trouble.