Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×

Submission + - TRIM and Linux: Tread Cautiously, And Keep Backups Handy

Submitted by Anonymous Coward
An anonymous reader writes: Algolia is a buzzword-compliant ("Hosted Search API that delivers instant and relevant results") start-up that uses a lot of open-source software (including various strains of Linux) and a lot of solid-state disk, and as such sometimes runs into problems with each of these. Their blog this week features a fascinating look at troubles that they faced with ext4 filesystems mysteriously flipping to read-only mode: not such a good thing for machines processing a search index, not just dishing it out. The NGINX daemon serving all the HTTP(S) communication of our API was up and ready to serve the search queries but the indexing process crashed. Since the indexing process is guarded by supervise, crashing in a loop would have been understandable but a complete crash was not. As it turned out the filesystem was in a read-only mode. All right, let’s assume it was a cosmic ray :) The filesystem got fixed, files were restored from another healthy server and everything looked fine again. The next day another server ended with filesystem in read-only, two hours after another one and then next hour another one. Something was going on. After restoring the filesystem and the files, it was time for serious analysis since this was not a one time thing. https://blog.algolia.com/when-... The rest of the story explains how they isolated the problem and worked around it; it turns out that the culprit was TRIM, or rather TRIM's interaction with certain SSDs: The system was issuing a TRIM to erase empty blocks, the command got misinterpreted by the drive and the controller erased blocks it was not supposed to. Therefore our files ended-up with 512 bytes of zeroes, files smaller than 512 bytes were completely zeroed. When we were lucky enough, the misbehaving TRIM hit the super-block of the filesystem and caused a corruption. Since SSDs are becoming the norm outside the data center as well as within, some of the problems that their analysis exposed for one company probably would be good to test for elsewhere. One upshot: As a result, we informed our server provider about the affected SSDs and they informed the manufacturer. Our new deployments were switched to different SSD drives and we don’t recommend anyone to use any SSD that is anyhow mentioned in a bad way by the Linux kernel.

Comment 2.5ms from Rackspace Cloud (Score 1) 558

by donnyspi (#43546459) Attached to: Average latency to Slashdot.org?
2.5ms from my Rackspace Cloud server: ping slashdot.org PING slashdot.org (216.34.181.45) 56(84) bytes of data. 64 bytes from slashdot.org (216.34.181.45): icmp_seq=1 ttl=246 time=2.56 ms 64 bytes from slashdot.org (216.34.181.45): icmp_seq=2 ttl=246 time=3.05 ms 64 bytes from slashdot.org (216.34.181.45): icmp_seq=3 ttl=246 time=2.90 ms 64 bytes from slashdot.org (216.34.181.45): icmp_seq=4 ttl=246 time=2.52 ms 64 bytes from slashdot.org (216.34.181.45): icmp_seq=5 ttl=246 time=3.79 ms 64 bytes from slashdot.org (216.34.181.45): icmp_seq=6 ttl=246 time=2.63 ms 64 bytes from slashdot.org (216.34.181.45): icmp_seq=7 ttl=246 time=2.75 ms ^C --- slashdot.org ping statistics --- 7 packets transmitted, 7 received, 0% packet loss, time 6008ms rtt min/avg/max/mdev = 2.524/2.890/3.791/0.408 ms

Comment Re:Am I the only one? (Score 2) 42

by donnyspi (#36426356) Attached to: Inside Amazon's Data Centers
Yeah, the thing is it already has taken off. They can hardly add additional capacity fast enough. AWS is a very profitable business for Amazon. Our company switched from a traditional data center setup to running fully on AWS and it did reduce our costs and increase our agility and scalability. That said, like anything it's not a fit for everybody.

Comment price (Score 1) 314

by donnyspi (#34332612) Attached to: Verizon Speeds Up FiOS To 150Mbps
I was an early adopter of FiOS in 2006. Had the 5/2 plan for $29.99. Since then the price has slowly increased. Last year they doubled by download speed and started charging me $49.99 for 10/2. If I had it to do over again, I'd have stuck with DSL. I don't need anything faster than 5/2. Now I'm stuck with a minimum price of $50/month. Lame.

Slashdot Top Deals

Science is to computer science as hydrodynamics is to plumbing.

Close