Catch up on stories from the past week (and beyond) at the Slashdot story archive

 



Forgot your password?
typodupeerror
Facebook

Submission + - Facebook's Corona: When Hadoop MapReduce Wasn't Enough (slashdot.org)

Nerval's Lobster writes: "Facebook’s engineers face a considerable challenge when it comes to managing the tidal wave of data flowing through the company’s infrastructure. Its data warehouse, which handles over half a petabyte of information each day, has expanded some 2500x in the past four years—and that growth isn’t going to end anytime soon.

Until early 2011, those engineers relied on a MapReduce implementation from Apache Hadoop as the foundation of Facebook’s data infrastructure. Still, despite Hadoop MapReduce’s ability to handle large datasets, Facebook’s scheduling framework (in which a large number of task trackers that handle duties assigned by a job tracker) began to reach its limits. So Facebook’s engineers went to the whiteboard and designed a new scheduling framework named “Corona.”"

This discussion was created for logged-in users only, but now has been archived. No new comments can be posted.

Facebook's Corona: When Hadoop MapReduce Wasn't Enough

Comments Filter:

In the realm of scientific observation, luck is granted only to those who are prepared. - Louis Pasteur

Working...