Massive Storage Advances

pra9ma writes: "Scientists from Keele University, in England, have suceeded creating a system that enables up to 10.8 terabytes of data to be stored in an area the size of a credit card, with no conventionally moving parts. This along with 3 other forms of memory which could revolutionize storage. The company said the system could be produced commercially within two years, and each unit should cost no more than $50 initially, with the price likely to drop later. " I'm unconvinced about their compression algorithm, but if it works, this is gonna be amazing.

Re:This may never happen . . .

Personally, I would love to have one! Just think of all the Pron and l33t warez I could store :)

Oh, get real. Both you and I know that by the time this technology (if it's real) makes it to market a standard OS install (take your pick, it won't matter) will be 5TB, using up half of it right off the bat. I, for one, will not be looking forward to buying Linux Kernel Internals -- 33rd printing, volumes 1-53.

And, in ten years, I'll STILL be on a fucking 56k-when-hell-freezes-over-more-like-26.4 dialup while Suzy N'Syncempeethrees and Sammy Likestoforwardjokes III have blistering Ultra-DSL at 30Gbps. Grrr.

Sorry for the rant.

Re:The compression algorithm...

I'm no compression expert but I regularly get 10:1 compression on text files using guess what? WinZip.

As the other poster has pointed out, your text files are almost certainly *not* straight English ASCII text (they're HTML, Word files, C code, Unicode-encoded, or something else again). I'm sorry I wasn't more clear in my post to explain that I was referring to straight ASCII text, not anything else.

As to my colleague, he'd read virtually all the published literature in the area, and he's a pretty smart cookie (he's now on a PhD scholarship at Princeton working with people like Tarjan). I think the thing I learned most from his efforts were that text compression is in a period of diminishing returns for improved algorithms - they're not likely to get much better.

Re:The compression algorithm...

I'm not at all surprised that they can get 8:1 compression of plain text.

I am. One of my postgrad colleagues (back when I was a postgrad) did research into text compression. The best that he could get on the KJV Bible was a little over 1.8 bits per character (about 4.4:1 compression), and IIRC the best *anyone* has ever done with a general-purpose compression scheme is a bit over 1.7, and it turns out that the bible is generally a little more compressible than most other text ;) Generally, you'll struggle to get better than 4:1 on most text, and that's using using compressors that are substantially slower than gzip or even bzip2.

While it is correct that studies with humans have indicated that English text has about one bit of entropy per byte, suggesting a natural limit of about 8:1 compression, humans have the use of a whole lot of semantic information (they understand the meaning of the text and can therefore predict words based on that) that no compression algorithm I'm aware of has used.

I'm taking this with a large grain of salt, thanks.

Re:The compression algorithm...

Wow, a chemist using a term wrong... amazing. To be specific, entropy doesn't come from chemistry, it comes from physics. Granted, these two were identical fields at the time, or at least, closely related, but the term came from studying thermodynamics, not chemistry.

OK. A little background on information theory for you - you know, from Shannon, back in the early 1900s, I believe, though correct me if I'm wrong. There is an object in information theory called the partition function of an experiment - it is essentially the chance of getting any result from that experiment. There is then an object called 'information', which is proportional to the log of the partition function. The lower a chance of getting a result, the higher the information content gleaned from that experiment. For instance, if you had a box full of quarters, and you randomly pulled out a coin, the partition function would be (quarters, dimes, nickles, pennies) {1,0,0,0}. The information content of that experiment is klogW: (0,inf,inf,inf)- you don't learn anything if you pull out a quarter. You knew there were only quarters to begin with. If you get any of the other ones, damn, you're surprised.

What does all of this have to do with entropy? Well, in thermodynamics, which is, you know, where the term COMES from, entropy is klogZ, where Z is the partition function of the system: essentially the same thing. k is Boltzmann's constant - it comes from the Celsius temperature scale.

So, here's the news flash: Entropy is information. Period. Therefore, he was using the term CORRECTLY, not INCORRECTLY. Entropy is USEFUL information, not USELESS information. Guess what? This is the same in chemistry, too. The universe doesn't care whether or not you can use energy for work, and entropy has nothing to do with 'randomness'. A 'random' distribution of matter in a universe will collapse to a 'nonrandom' sphere, thanks to gravity: if entropy was randomness, then the universe would have just violated the second law - it went from 'random' energy to 'nonrandom' energy. (mass=energy, so don't even try it)

Entropy is information. Period. Hence the second law of thermodynamics- entropy increases because the information content of the universe is increasing. If you doubt me on this, here's a simple bit to convince you: you have a system which goes from state 1 to state 2, both of which have the same entropy. Therefore, there is a reversible process which connects the two states, which means that one can go from state 1 to state 2 and leave no tracks inside the system that the change had happened - i.e., the information content of the system is static.

I'm really getting sick of having to explain this constantly - I wish they would never teach entropy as 'useless energy' or 'unusable energy' - like the universe cares whether or not something can be used for work.

The link between information and entropy is entirely well known and extraordinarily important. For instance, if an object falls into a black hole, is there no record of its existance anymore? Is all the information that was inside that object lost? No - a black hole's area is related to its entropy, which increases with mass. Therefore, the 'information' (as far as the Universe cares) in that object is now somehow stored in the black hole's event horizon. Curiously enough, an object which falls into a black hole is, from the outside world, constantly getting infinitesimally closer to the event horizon. This is a weak argument, yes, and changing a few words could make it stronger, but this is offtopic, so I don't care.

In closing - you're wrong. Entropy is useful information. 1 bit of entropy out of 8 means an 8:1 compression ratio. Here, you've 'extracted' 7 bits of 'work' out of the system. The remaining 1 bit of entropy cannot be removed from the system, as entropy can never decrease. (or in this case, cannot decrease without destroying the system)

Re:Nonsense

I think that "no conventionally moving parts" means that they are using a Wankel rotary engine to move the parts rather than the conventional 4-stroke design. I must admit that it is a pretty clever hack to figure out how to use mechanical storage to get that kind of density and even more curious that they chose the rather oddball Wankel design over battery power which is usually used for small devices.
_____________

The compression algorithm...

I'm not at all surprised that they can get 8:1 compression of plain text. It is a rule of thumb for encryption that plain text, at least in English and with ASCII, has only about one bit of entropy per byte. While it is impressive that they've managed to get rid of almost all of the slack, it doesn't strike me as that hard to believe.

Re:hmmm

If you don't want to read about vaporware, then you should probably read buy.com instead of slashdot.org.
--

More Info

Here is some more info I got from Google's cache for http://www.cmruk.com/cmrKHD.html

KEELE HIGH DENSITY LTD

UPDATE - November 2000 During 1999 Keele High Density Ltd. (KHD) announced that it had developed a very high density memory system capable of holding 2.3TB of memory in the space of a credit card. Further work since then has resulted in some significant upward changes to both the capacities previously stated and to the applications the KHD technology addresses. Some of this work is continuing, and there are further patent applications to be filed. The information available publicly is necessarily restricted until those patents have been filed. The very high data densities are achieved through a combination of many different factors - some relating to the physical properties of the recording media, and some to the way of processing and handling data. The physical memory system is a hybrid combination of magneto-optics and silicon. The KHD memory system is applicable to both rotating and fixed media, and is not dependent on the laser-based media-addressing system used. Following the work undertaken since last year, the following data capacities are achievable: a) For rotating media, at DVD size, a single-sided capacity of 245 GB using a red laser. b) For fixed media, a single-sided capacity of 45 GB/cm, giving a total capacity of 3.6 TB on the surface area of a credit card, double-sided and using a red laser. Using a violet laser (now being introduced), the capacity at credit card size will be 10.8 TB. In last year's announcement from KHD the primary focus was on the fixed media application, which with a novel form of laser addressing, could be described as 'near solid state' - involving no moving parts in the conventional sense. However, this aspect of the technology will require some further R&D work to bring it to a mass-production scale - although it is believed that this will not present insurmountable difficulties. These constraints do not apply to existing rotating media applications (for example, DVD), using conventional laser systems, and there are no reasons why the KHD technology cannot be implemented within a short timescale - measured literally in months. A major development arising out of KHD's work over recent months, is that the technology achieving these very high data density figures has application not just for memory systems, but will also produce significant enhancements for the transmission and processing of data generally. This means that KHD's technology can achieve an effective increase in bandwidth capacity, because the very high data density properties, which are in addition to those from conventional compression methods, allow so much more data to be transmitted over a given bandwidth. The same advantages are also felt in terms of processing speeds. Work on this aspect of KHD's technology is continuing, but the current calculations show that an effective eight-times increase in bandwidth capacity and processor speed can be achieved. KHD's development represents a fundamental advance in computing technology, with the benefits being felt across many industry areas. Following completion of the patenting position, KHD will be looking to license the technology to companies for mass-production, and for the ongoing R&D work needed to make the 'solid-state' memory commercially viable. The technology has been developed by Professor Ted Williams at Keele University, Staffordshire, England, over a period of thirteen years. PROFILE: Ted Williams is Professor Emeritus of Optoelectronics at Keele University, Staffs, England, and visiting Professor of Electronic Engineering at South Bank University, London. Professor Williams was Director of Research with Sir Godfrey Hounsfield, Nobel Prizewinner, working on the invention and creation of the first NMR Scanner at Hammersmith Hospital, London. He has also held directorships with major international companies. His main focus over the last thirteen years has been the research and development of 3-dimensional magneto-optical recording systems. KHD's licensing and funding arrangements are managed by Mike Downey, Managing Director of Cavendish Management Resources. CMR is a venture capital and executive management company, based in London. CMR has supported the development of this technology. Further information from: Mike Downey Managing Director CMR, 31 Harley Street, London W1N 1DA Tel: [phone] Fax: [phone] Email: [e-mail] [mailto] Web: www.cmruk.com [cmruk.com]

Re:The compression algorithm...

Yeah, uh, no. The absolute best text compression algorithm today (RK) achieves 1.42 bits per character. For some REAL STATS (!), see The Archive Comparison Test [geocities.com]. Considering the amount of work that has gone into text compression in the past few years (going from 2.0 to 1.42!) and knowing the theory myself, I find it ludicrous that someone unknown in the compression scene would come up with such an algorithm, which by the sounds of the simple description would likely already be covered by many patents. It's definitely overstated, and likely far inferior to the current state-of-the-art in compression.

a bit light on the details

This article is pure crap. Professor soggybottoms invents ten fabulous new technologies that will instantaneously revolutionize the entire computer industry, all while fixing himself a ham sandwich...

film at eleven...

New storage ratings...

10.8TB = 1064 DVD's (presuming 10.4GB per DVD) = 17,400 CD's (presuming 650MB per CD) = 7,864,320 floppies (presuming 1.44MB per floppy) = 371,085,174,374 of those new MOT 256bit MRAM chips.

Anyone want to come up with some other ratings ?

Mark Duell

This is a year and a half old...

Let's see...

$ nc www.keele.ac.uk 80
HEAD /research/cmrkeele.htm HTTP/1.0

HTTP/1.1 200 OK
Date: Tue, 13 Feb 2001 11:39:15 GMT
Server: Apache/1.3.12
Last-Modified: Fri, 20 Aug 1999 12:16:30 GMT
ETag: "239a2-f60-37bd471e"
Accept-Ranges: bytes
Content-Length: 3936
Connection: close
Content-Type: text/html

Last modified 20 Aug 1999? Not what I'd call "breaking news"... If you don't believe the server date, try this corroborating evidence: http://www.cs.colorado.edu/pipermail/postpc/1999-S eptember/000002.html [colorado.edu]

Why news.ft.com decided to post the story now, I couldn't say...

== Sparrow

Nonsense

This is a highly unconvincing attempt at hyping what is in all likelihood a non-existant product.

The first invention is a method of compressing text stored in binary form, which expresses information as a series of noughts and ones, by comparing each word with its predecessor and recording only the differences between words

Well that's pretty unremarkable. They've written a compression algorithm.

Oh, by the way, they have also invented

"a memory system that enables up to 10.8 terabytes of data to be stored in an area the size of a credit card, with no conventionally moving parts"

If that were true, why are they bothering to even *think* about their text compression algorithm? Fifty dollars a go? Who wants compression? If these people are telling the truth, we are talking about a thousand-fold increase in gigabytes per dollar over the space of two years.

The phrase "no conventionally moving parts" also brings to mind images of really whacky, non-linear moving parts flailing about. What the hell do they mean?

Absolutely no technical detail is given in the article, and as far as I'm concerned, this is yet another false alarm on the long road to entirely solid-state computer systems.

Always the size of a credit card

Why is it every piece of new tech is the size a a credit card? Can't be the size of a dollar bill? or what about a piece of sliced bread, considering all this new tech is the greatest thing since.

I just want to know what every tech inventor's opbession is with everything being the size of a credit card. It's not like we are going to fit these in our wallets. "Sure Mr. Tanaka, I have my 20 terabyte database here in my wallet, care to swap?"

I dunno, I just wish technology came in different sizes I guess.

Wow this is GREAT!

Man, I am so glad that I read slashdot. Without slashdot I would have to sift through tons and tons of bullshit every day just to find the new and amazing technological advances of the age. But no, I read slashdot, so I can come here and find the best of the best, such as this dandy invention.

Wow 10.8 TB on a credit card, wahooo! What will they think of next? How do I send them guys my money? I couldn't find any address or nothing, but those english 'blokes' sure look like they is gunna go far with this invention - specially that text compression thingy - pretty damned original if I do say so myself. And then that storage mechanism 'no conventional moving parts' - I can't imagine how they got those conventional parts to stop movin, sound like quite a trick.

Anyway, don't you slashdot guys let the criticism get you down. I am with you. Don't listen to them naddering nabobs of negativism. They always persecute the dreamers!

I am looking forward to your next 'Light speed limit possibily violated' post with anticipation.

-josh

Re:Always the size of a credit card

what about a piece of sliced bread

The size of bread slices varies widely from region to region, this prevents multinational corporations from referring to their products as the size of a piece of sliced bread. Although ANSI created a sliced bread standard in 1986 and updated their standard in 1992 to account for the coarseness of pumpernickel, this is an American standard which prevents any companies wishing to sell their product outside of the United States from using it and unfortunately the ISO has been dragging their heels on forming a sliced bread standard, so until the day when we get the ISO sliced bread standard you can expect many more credit card sized comparisons.

Re:The compression algorithm...

All right,

lynx http://slashdot.org/article.pl?sid=01/02/13/024025 4&mode=nested&threshold=-1 > slash.txt

(no -source option because this is Slashdot, and as we all know too well, the content is much more redundant than repeating html tags, much, much more redundant)

shelf:~$ ls -l slash.*
-rw-r--r-- 1 stu users 20394 Feb 12 21:09 slash.bz2
-rw-r--r-- 1 stu users 23750 Feb 12 21:09 slash.gz
-rw-r--r-- 1 stu users 93867 Feb 12 21:09 slash.txt
shelf:~$

This gives a ratio of 0.22. Surprisingly, if you feed the same page to bzip2, but at +2, the ratio increases to 0.27, implying that there is more entropy and thus, more information, in higher scoring posts, which of course, we know to be false :)

Perhaps with this firm mathematical footing, /. can proceed to a new chapter in moderation - moderation by bzip2. Articles which receive high compression ratios are marked down automatically. Of course, this would make it possible to earn a lot of karma, simply by posting random garbage. oh wait..

This is unbelievable!

Here's a link from the university. [keele.ac.uk]It sounds like it's real to me.