Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror
Check out the new SourceForge HTML5 internet speed test! No Flash necessary and runs on all devices. Also, Slashdot's Facebook page has a chat bot now. Message it for stories and more. ×
Facebook

Submission + - Facebook open sources its Hadoop job scheduling enhancement, Corona

stormspirit writes: Today, Facebook open sources its scalable Map-Reduce job scheduling framework, Corona, as an enhancement for Hadoop. Facebook has been using Hadoop as MapReduce framework for years. According to the engineering notes, by early 2011, it started to hit the scalability wall of Hadoop. In particular, the job scheduling module became the bottleneck for large workload as there's only a single job tracker in Hadoop to both manage cluster resources and task trackers. Corona mainly divide the two responsibilities between a central resource manager and per task job tracker, allowing the module to scale. Other improvement includes: a pushed-based task assignment, dynamic resource management and better fairness guarantee. Corona is now deployed in all production systems with its design goals achieved.
This discussion was created for logged-in users only, but now has been archived. No new comments can be posted.

Facebook open sources its Hadoop job scheduling enhancement, Corona

Comments Filter:

Top Ten Things Overheard At The ANSI C Draft Committee Meetings: (9) Dammit, little-endian systems *are* more consistent!

Working...