How Much Bandwidth is Required to Aggregate Blogs? 209
Kevin Burton writes "Technorati recently published that they're seeing 900k new posts per day. PubSub says they're seeing 1.8M. With all these posts per day how much raw bandwidth is required? Due to innefficiencies in RSS aggregation protocols a little math is required to understand this problem." And more importantly, with millions of posts, what percentage of them have any real value, and how do busy people find that .001%?
How much? If everyone GZipped, a lot less! (Score:5, Insightful)
So my plea to the internet community today.. make sure your web server is configured to send gzipped content. TFA says he doesn't know how many RSS feeds can support gzip. The answer is easy really, any feed being served by Apache (plus a LOT of other webservers. AOLserver even added gzip support recently). Here's how to setup Apache [whatsmyip.org] and here's where to check [whatsmyip.org] if your site is using GZip or and get an idea of the bandwidth savings you should see get. If you're site isn't gzipping, show your admin (if it's someone else) the 'how-to' above and ask them to implement it -- it's an absolute no-brainer win-win for everyone that takes no time at all to setup really. It's really absurd IMO that it's not enabled in Apache by default.
Slashdot? (Score:4, Insightful)
On slashdot.... Oh wait....
busy people read 9000 blogs per day?? (Score:1, Insightful)
Some Answers (Score:4, Insightful)
Less than it currently takes, what with pull, HTTP, and XML used instead of more efficient technologies.
``what percentage of them have any real value, and how do busy people find that
Using a scoring system, like Slashdot's?
It's not like all of this is rocket science. It's just that people go along with the hyped technology that's "good enough for any conceivable purpose", ignoring the superior technology that had been invented before and wasn't hyped as much. Nothing new here.
Definition of quality and value == arbitrary (Score:3, Insightful)
In actuality, my guess is that there are few blogs you might decide to visit, and of those you do, several may have content you find worthwhile. Remember, worthwhile is all in the perception of the reader - there is no real definition for quality or value. Perhaps through trial and error - in essence digital tinkering - you find and derive your own value.
cheers, --dave
Re:How much? If everyone GZipped, a lot less! (Score:4, Insightful)
Re:All at once (Score:5, Insightful)
Re:How much? If everyone GZipped, a lot less! (Score:3, Insightful)
Finding the Worthwhile Content in Blogs (Score:5, Insightful)
If a friend is going through cancer treatment, her blog is worthwhile. If you find a youth group leader like yourself and can learn from his posts, his blog is worthwhile. A mother fighting for her health so that she can take care of her two sons and husband can share insights that are worthwhile. Someone fighting depression might have a worthwhile blog. A grandmother might have a view of the world that makes her blog worthwhile, just to get a different view. Perhaps a blog by someone who totally disagrees with you will be worthwhile, just to stretch your mind.
I've just described why I read the blogs on my blog roll. You can choose differently.
Top political blogs? You can find them easily among Technorati's top 100 list. Tags at Technorati will let you pick out specialties like science or "Master Blasters" or diabetes or the Tour de France. Google will turn up blogs if you search right, which is the trick for using Google.
"Worthwhile" is a much more difficult variable to calculate than "bandwidth." Perhaps it's the sheer variety of blogs that makes them interesting, because they are so individual and someone, somewhere will speak to your mind or your heart.
Worthwhile is what's worthwhile to you, and maybe to very few others. Not everyone will agree, and that's not a bad thing.
Re:we've tried gzip on our server... (Score:1, Insightful)
This reads like a generic troll. "We actually had been using $PRODUCT_NAME for quite a long time on a server at home..."
Re:How much? If everyone GZipped, a lot less! (Score:3, Insightful)
One thing is for certain though, for many users bandwidth is NOT cheap.
Re:Wheat from chaff (Score:3, Insightful)
Re:How much? If everyone GZipped, a lot less! (Score:3, Insightful)
Re:How much? If everyone GZipped, a lot less! (Score:4, Insightful)
That results in a 10 times shorter transfer time,
Which results in 10 times fewer simultaneous connections,
Which results in 10 times fewer apache processes,
Which results in massively reduced memory and processor requirements.
That unused processor and memory is what would be used to perform the gzip operations. Lets say for arguments sake compressing the output doubles the processor usage (a ridiculously high number) cutting the number of apache processes by an order of magnitude only has to reduce CPU requirements by 50% to come out on top.
If the gzip operation only inflicts a 10% overhead cutting the apache processes by ten only needs to free more than 9% to come out on top.
Look at your server, would cutting the number of apache processes from 400 to 40 save more than 10% of the CPU usage, would it save more than 50%?
[All numbers in this post were selected for ease of calculation not for their real world precision,]
Re:How much? If everyone GZipped, a lot less! (Score:5, Insightful)
Re:How much? If everyone GZipped, a lot less! (Score:4, Insightful)
Re:Don't forget the robots (Score:3, Insightful)
s/blog/website (Score:3, Insightful)
Time to ditch the World Wide Web, right?.