Follow Slashdot stories on Twitter

 



Forgot your password?
typodupeerror

Comment Re: It almost writes itself. (Score 1) 55

I'm sure for many people it works that way. For me, I got more out of actively listening to the professor in real time than from words scratched on a page. The only class I ever used heavy notes in was linear algebra, because there wasn't really another source for the formulas.

Comment Re: was this (Score 1) 42

I've never seen a preview stage here. Weird. Anyway, I understand enough to know that there was a weird combination of circumstances (including insufficient unit testing) that caused this to hit everything at once. Normally, you'd see these crashes during a 1% canary and stop it before it becomes user visible, either during a canary of the binary or of the configuration change that diverted traffic. This somehow missed both.

Comment Re: was this (Score 1) 42

"Cut in." Not my fault slashdot has no edit function.....anyway, yes, that's exactly the problem. The CODE was rolled out slowly; but that didn't matter because that path was not in use at the time. It began getting traffic hitting that bug all at once. That should not have happened. Usually you gradually divert traffic onto a new code path.

Comment Re: was this (Score 2) 42

Google uses a blameless postmortem process to identify areas for improvement. It isn't about making an example, it's about making sure it can't happen again. This is pretty good for an external report. The problem here is that the code path with the bug was cut it abruptly rather than gradually; that isn't supposed to happen. A secondary problem was hitting a backend too hard when all the jobs tried to come up at once; probably insufficiently random backoff.

Slashdot Top Deals

"If I do not want others to quote me, I do not speak." -- Phil Wayne

Working...