Become a fan of Slashdot on Facebook

 



Forgot your password?
typodupeerror
Check out the new SourceForge HTML5 internet speed test! No Flash necessary and runs on all devices. Also, Slashdot's Facebook page has a chat bot now. Message it for stories and more. ×
Facebook

Submission + - Facebook's Corona: When Hadoop MapReduce Wasn't Enough (slashdot.org)

Nerval's Lobster writes: "Facebook’s engineers face a considerable challenge when it comes to managing the tidal wave of data flowing through the company’s infrastructure. Its data warehouse, which handles over half a petabyte of information each day, has expanded some 2500x in the past four years—and that growth isn’t going to end anytime soon.

Until early 2011, those engineers relied on a MapReduce implementation from Apache Hadoop as the foundation of Facebook’s data infrastructure. Still, despite Hadoop MapReduce’s ability to handle large datasets, Facebook’s scheduling framework (in which a large number of task trackers that handle duties assigned by a job tracker) began to reach its limits. So Facebook’s engineers went to the whiteboard and designed a new scheduling framework named “Corona.”"

This discussion was created for logged-in users only, but now has been archived. No new comments can be posted.

Facebook's Corona: When Hadoop MapReduce Wasn't Enough

Comments Filter:

"Engineering meets art in the parking lot and things explode." -- Garry Peterson, about Survival Research Labs

Working...