Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!


Forgot your password?

Facebook's Corona: When Hadoop MapReduce Wasn't Enough 42

Nerval's Lobster writes "Facebook's engineers face a considerable challenge when it comes to managing the tidal wave of data flowing through the company's infrastructure. Its data warehouse, which handles over half a petabyte of information each day, has expanded some 2500x in the past four years — and that growth isn't going to end anytime soon. Until early 2011, those engineers relied on a MapReduce implementation from Apache Hadoop as the foundation of Facebook's data infrastructure. Still, despite Hadoop MapReduce's ability to handle large datasets, Facebook's scheduling framework (in which a large number of task trackers that handle duties assigned by a job tracker) began to reach its limits. So Facebook's engineers went to the whiteboard and designed a new scheduling framework named Corona." Facebook is continuing development on Corona, but they've also open-sourced the version they currently use.

Comment Yes (Score 4, Informative) 293

If you are working as a consultant, then the biggest advantage of incorporating will be in tax savings.

In Canada (Ontario specifically) there is a break-even point around $42k/yr income, where the personal income tax and corporate income tax (and accountant fees, etc.) you pay will be approximately equal. Above $42k/yr income, the corporate tax will become less and less compared to personal tax. This is due to the fact that the corporate tax rate is fixed at 16.5% (until $500k or $1M annual income... I can't recall) while personal tax rates have brackets that increase as you make more money.

To take an example from my past, the last year before I incorporated I made roughly $86,000 and paid about $22,000 in personal income taxes. The accountant that helped me incorporate did some calculating, and if I were incorporated, the corp would have had to pay only about $13,000-14,000 in taxes.

There are some costs associated with running a corporation. There are the initial costs of setting it up, usually between $2000-4000 for lawyers and accountants. Then annually, you will probably have an accountant prepare your corporate taxes, which will cost around $750-2000 depending on who does it and how organized your paperwork is. These are extra hassles that some people find unpalatable, and it is a bit of extra administrative work on your part, but altogether, it saves you thousands and is very much worth it. (Unless you have some kind of ADHD and psychologically cannot deal with paperwork.)

Another tax saving tool available in Canada is that you can make $50k/yr in dividend income, tax free. Therefore, if you and your significant other are both part owners in your newly formed corp, then you can essentially have a combined household (personal) income of $100k/yr, tax free because your corp will pay out dividends to its owners, rather than salary (which is all taxable). You will probably not make exactly $100k/yr tax free (but it will still be around $95k or $98k) because in order to take advantage of various tax credits you have to show some personal income. How this is works is that, whenever you need money from your corp, you just withdraw it. At the end of your fiscal year, you and your accountant will figure out how to label those withdrawals, be it dividends, salary, whatever, to maximize the tax savings. That is how I have been doing it in Canada, anyway, and your accountant will be more familiar with how this stuff works in your area.

The best thing you can do (aside from asking the experts on Slashdot, of course) is to go see an accountant who deals with corporate stuff. Explain to him or her what you are thinking about doing and outline your current situation. Using your 2011 net income as an example, they can then draw up a spreadsheet for you, showing what would be your taxes and other numbers if you had been incorporated in 2011. This will let you know with little uncertainty what is your best course of action.

There are other benefits that come with having a corporation, your corp can purchase the equipment (e.g. laptops, mobile devices for testing, etc.) that you will use to do the service that the corporation sells. This can be recorded as an expense of the corp, which reduces the corporate taxes. In contrast if you bought equipment personally, it would not affect your tax situation at all. This is nice if you like toys, and would like some extra reasons to rationalize their purchase.

In summary, if you plan to make more than $42k (*) this year from your moon-lighting activities, just get it done already.

* $42k, or whatever is the break-even number for the tax system you live in.

Comment Re:Government Contract in Search of a Problem? (Score 1) 329

Wrong, "toqué" is an adjective. It is never used, anywhere, to refer to the hat, which is a noun. You'll never find a Frenchman who says "toqué". If someone does write "toqué" or "tuke" when they mean "toque" then they belong to the same group that misuses quotes and hyphens (a.k.a. illiterates of questionable moral standing).

Toque is the correct spelling.

Comment Re:Kazkek (Score 0, Flamebait) 337

Listen, you Labview fan boy, I am not smoking crack. (Although I will admit it: Labview makes me wish I was a toothless meth addict, hitting rock bottom, giving blowjobs to married, middle-aged closeted gay men in a movie theater for drug money.)

I spent more than 10 minutes learning that crap. More than a few weeks in fact. I got to know some Labview tech support "engineers" by name. The problem, I later realized, was that I knew exactly what I wanted the hardware to do, except I had to jump through all kinds of Labview hoops to get there. In C or C++, I could have had the majority of the hard parts done in a few days, and then polish up the details. Not so in Labview.

Ever try to force yourself to do something the hard way, when you know there is a much easier way to do it? Ever try to pee sitting down when you're a few pints in, and you really have to go? That's what working with Labview is like -- it will block your peehole until you are calmly sitting on the toilet like a good boy. If you are a girl it will make you pee standing up through one of those cardboard funnels.

Let's look at what you'd have to spend to get a hardware and software package that lets you look at signals at 10MHz.

Here is the "Buy Labview" (a.k.a. get raped in the ass with a cactus branch) web page:


I could go with Labview Base or Labview Full, but since I want to deploy stuff to customers' machines without forcing them to buy Labview, I'm going to go with Labview Professional for US$4299, although I should probably go for NI Developer Suite for US$4699 because it's the "best value."

Next, I want to sample at 10MHz or greater. In 5 minutes of searching, these are the only products that I could find, that will let me do that. Note that these both have a maximum rate of 10MHz; there don't seem to be NI products that can go higher.

  • PXI-6115 for US$4199, or if we go with the "NI recommended" version, US$5249
  • PCI-6115 which is just the PCI bus version of the PXI-6115, for US$3799, or US$4849 for the "NI recommended" version

If you go with the PCI version, you will need a PC with some pretty good horsepower to handle the 10MHz data stream. This may add an extra $1000 or $2000 to your total price.

If you go with the PXI version, then bend over and grab your ankles again, for into your bleeding rectum NI shall happily insert a well-lubricated PXI chassis of your choice, for an additional few thousands of $$$.

Add in all taxes and other costs (shipping, and oh yeah, $600 cables, anybody??) and it's pretty damn near $10k if not well above. Just for the privilege of writing multi-colored spaghetti code in order to make your measurements.

In conclusion, it is better to just get a real 60MHz digital scope for a few grand. Labview treats intelligent humans like dogs.

Comment Re:Kazkek (Score 5, Informative) 337

By all means, buy National Instruments hardware. It is fantastic. I have deployed it on a number of production systems that run for days and days and days without a glitch.

As for Labview, stay the fuck away from that steaming pile of dogshit. It is a great way to waste lots of time and give up your sanity (and possibly your anal virginity) unless you feel like fucking around at your lab bench and drag-n-dropping some blinky lights and text boxes to impress your PHB. Oh yeah, and it's also great if you enjoy having fresh-out-of-college, inexperienced National Instruments tech support fuckwads (i.e. never having done any actual work with data acquisition or signal processing in their lives) repeatedly tell you, "OMG, change the way you think! You're so wrapped up in the text-based language tunnel vision! LOL!" whenever you get frustrated because Labview actually slows you down and doesn't help you get your results.

Oh yeah, I almost forgot, it will also cost you an arm and a leg. A hardware and software (i.e. Labview) package will let you look at signals in the 10MHz range and above will probably run you at least $10-15k.

In my day to day work, I prefer to remain in the "text-based language tunnel vision." That is, I prefer to use a well-designed C or C++ API to write programs that actually work in a predictable and reliable manner. (The NI-DAQmx API is actually very powerful and easy to use.) That way, I don't have to scroll around in a blinding maze of brightly colored connector lines and boxes and stuff, just because some National Instruments fucktard decided that "text-based languages" are just too, like, texty and complicated and not very much fun, yay! Yuck.

Comment Re:Physicists? (Score 1, Insightful) 466

You're not qualified to answer the question. Information technology is not computer science. They are called different things for a good reason.

There is little or no similarity between computer science and IT because they are completely different fields. In IT, you use programs and systems that real computer scientists created, and write scripts and Visual Basic things to glue them together, in order to get something that does what you want. IT is about integrating systems. Computer science is about math, scientific analysis, designing optimal algorithms, and so on. Therefore your lack of need for math in your IT work has no connection to the question being asked.

Comment Re:attack is a primitive response (Score 1) 836

Thank you for your correction. Using "origin" would have indeed made more sense.

Not to be a nit picker, but I did clearly state that I kept reading in spite of the cringe-inducing opening of the article. I'm not sure why you're calling me unwise because of that. But, your trite network communication metaphors are almost as cringe-inducing as Mr. Spiegel's writing.

You're right, the article is in fact "a stream of information not forced onto [my] consciousness." But it appeared on the Slashdot front page, and when there is something on the front page of Slashdot that annoys us, we speak up.

I fail to understand your generalization that "attack is a primitive response to a stream of information not forced onto your consciousness." To clarify, are you saying that when information is not forced on me, and I am primitive, my response is to attack? I beg your pardon. That makes no sense at all. I hope that English is not your first language; if it is, you are probably a sloppy programmer.

Comment Re:proofreading for the college graduate? (Score 5, Interesting) 836

I agree with acidfast.

Furthermore, Mr. Spiegel, you are keen to use cliche phrases without even putting in the effort to understand their meaning, or know their correct spelling. This helps you come across as a pompous idiot.

For example: "Queue awkward silence."

The correct spelling is "cue awkward silence." It comes from stage and movie production, where the producer will "cue" actors, lights, or special effects. How does one "queue" awkward silence?

I almost stopped reading there. But I kept going, hoping to find some redeeming value.

It was hard to finish your article, as your tone makes it clear that you are a cocky, holier-than-thou ladder climber. You provoke a regular guy eating his lunch into a pissing match, and then you claim to have said things like, "Everyone is making valid points," in actual conversation. Who does that?

God help any of us who may have to work with you, or even worse, for you. I don't care if you have Asperger's or not. You are a douchebag, period.

Comment Re:Sure, let's have more unschooling... (Score 1) 1345

I'm pretty sure most kids wander around and are curious about things anyway, whether or not they go to school. Nothing dangerous about it. What supporters of "unschooling" are suggesting is that that is all that kids should have to do.

Classes go from 9am to 3pm in most schools. That leaves plenty of time after school and during weekends for wandering around, acquiring knowledge, and "being dangerous" as you call it. I guess my question is, why go to the extreme of "unschooling" and let kids do whatever the hell they want, all the time? Or am I missing something?

Slashdot Top Deals

Adding features does not necessarily increase functionality -- it just makes the manuals thicker.