Become a fan of Slashdot on Facebook

 



Forgot your password?
typodupeerror
Facebook

+ - Facebook open sources its Hadoop job scheduling enhancement, Corona

Submitted by stormspirit
stormspirit writes: Today, Facebook open sources its scalable Map-Reduce job scheduling framework, Corona, as an enhancement for Hadoop. Facebook has been using Hadoop as MapReduce framework for years. According to the engineering notes, by early 2011, it started to hit the scalability wall of Hadoop. In particular, the job scheduling module became the bottleneck for large workload as there's only a single job tracker in Hadoop to both manage cluster resources and task trackers. Corona mainly divide the two responsibilities between a central resource manager and per task job tracker, allowing the module to scale. Other improvement includes: a pushed-based task assignment, dynamic resource management and better fairness guarantee. Corona is now deployed in all production systems with its design goals achieved.
This discussion was created for logged-in users only, but now has been archived. No new comments can be posted.

Facebook open sources its Hadoop job scheduling enhancement, Corona

Comments Filter:

The UNIX philosophy basically involves giving you enough rope to hang yourself. And then a couple of feet more, just to be sure.

Working...