Snowden mentioned "hashing" the emails that were not obvious duplicates. That means generating an SHA1 key (or similar) for the entire contents of the email - to/from/cc/bcc/subject/etc. Do that for both the new emails and for the "old" emails. Now anytime you have a matching SHA1 key on both sides, you have a duplicate email. Discard those. Now run the remainder through full text indexing (only about a business day of processing time) and run keyword searches for your specific topics of interest. Flag any results for further review/analysis. Some of that further work may be applying more scripting to remove false positives. The results could be that there are very few results that would impact the previous decision regarding Clinton. And with the apparent manpower that was thrown at this, I'm sure any emails that made it through that filtering were vetted thoroughly. I think the initial declaration by the FBI was the bullshit part, not the time it took to process the "new" emails.

See my note below. Email ultimately is just a text file formatted a specific way. Now if you insist on using MS Exchange this fact is obscured by all the Microsoft-isms they like to do. Not all mail servers treat email as a singular binary object that requires email to be "extracted into a readable format". And to take that a step further - the script deciding if the message is pertinent or not doesn't need to be able to read it the same way a human does - so the full SMTP headers are fine to leave alone. Once you have a collection of text files, then you can apply modern tech (full text indexing / search) to allow keyword searches very quickly. Anything that is a "hit" there needs further human review (perhaps). But that suddenly takes you from 650k emails to maybe a few thousand (perhaps). After all aren't they searching for some very specific points? So they must have a handy set of keywords to be looking for... My own rough calculations suggest the full text indexing could be done is as little as 10 hours for 650k email messages. (I routinely take about 45 minutes to index approx 50k records of product data that covers more data than a typical email - headers and all. 650k / 50k = 13 "batches". 13 * 45 minutes comes in at between 9 and 10 hours to get full text indexing in place. On a single desktop PC, without considering clustering the search servers for faster processing. Throw in a few hours of the actual keyword searches, and then a quick review of any possible hits, and making a judgement call in 8 days becomes VERY feasible.)

Using Solr and Magento, it only takes me 45 minutes to run full text indexing against 50K enriched product records (color, weight, vendor, description, short description, title, nicknames, etc. - easily way more data than in a typical email for each product) And my box is not especially fast and does nothing in terms of clustering to improve performance. Now do that 13 times to arrive at approx 650k items, and it only takes approx 9 or 10 hours. Now you could run keyword searches against the entire lot to see if there is anything of interest. And that is BEFORE removing duplicates like Snowden suggested, or applying some Natural Language Processing algorithms, or any other relevant AI code... No conspiracy here, I think. Rather I think you see just who is truly out of touch with what modern tech can do.

Beanstalkapp is down for us here (prairie region of Canada). Beanstalkapp is a git repository similar to GitHub and Bitbucket, kinda important when you are a development shop. Twitter is down too. Current time is 1:30pm MDT.

The guy (assuming) who posted the private video to a public forum. THAT is who deserves the blame, and the bullying.

The physical act of rape is not involved here. However Rape Culture is about blaming the victims for the crap they had no control over. The moment one of her so-called friends took a private email/message/video and made it public without her consent, there was a kind of figurative rape. And you and others are blaming her for sending out the video in the first place. While that may have been an error in judgement on her part, she did not make it public. Someone chose to do that for her in a very damaging way. Continuing to insist "she deserved it" is bullshit. She deserved to be PRIVATELY admonished for the lapse in judgement. Not bullied for her actions in PUBLIC. Canada had a similar case a few years ago where a teen was involved. Rehteh Parsons (look it up) was subjected to the same sort of thing after being gang raped, with the same outcome. And the Internet blamed her too. Blaming the victim is what makes this a Rape Culture issue - whether you like it or not.

+1 for calling out the rape culture (no mod points right now)

Imagine you have 30 year old desk of a nice design. A specific plastic piece of the desk (for argument's sake) breaks. That piece cannot be purchased because the desk went out of production 25 years ago and replacement parts are not available. So you go ahead and print the piece you need - either downloading the 3D model or creating a copy of it yourself. You have a fixed desk. A few days later you get a visit from the boys in blue for copyright infringement. Replace the desk with a car, toy, or some other widget. According to your argument - you should spend possibly thousands on getting a new thing, rather than spending $1 and a little time to maintain the thing. According to your argument the rest of the world should shame you into wasting your money. Me, I'd rather not shame anyone for doing anything reasonable like keeping their stuff maintained.

If A = [Some Muscian]
and B = [ Some coporation ].

"We ask you to enact sensible reform that balances the interests of creators with the interests of the companies who exploit music for their financial enrichment" can can be reworded as as
"We ask you to enact sensible reform that balances the interests of 'A' with the interests of 'B' who exploit music for their financial enrichment.
Or do I have that backwards?
"We ask you to enact sensible reform that balances the interests of 'B' with the interests of 'A who exploit music for their financial enrichment.

I'm a little confused who the "bad guy" is intended to be here.

I remember a parable story about how a dev team lamented about the 20 standards there were for XXXXX. So they decided to merge all the standards into one comprehensive standard. They worked long and hard and finally completed the mammoth task and released it to the public. Now there were 21 standards.

This story about "pop computing" seems similar for some reason.

If the FBI does not reveal the hack so they can hack other phones, well that means the bad guys can also continue using that hack. After all we know that there are now at least 3 organizations who can access a locked iPhone 5c without the owner's password.

John Oliver famously coined the "dick-pic" angle of looking at the surveillance programs Snowden helped reveal. The resulting understanding in the masses when you boiled down the question to "can the government see my dic pics" showed a massive reversal of general opinion (IMO).

Something similar is needed here. Perhaps the question should be reworded to "Should the FBI be able to force Apple to rewrite their systems so that an Apple phone will unzip your pants to see if you have a penis or not?" Because at this point there is no evidence (that I've heard) that there is anything pertinent on the phone. Only the possibility that there *might* be. Much the same as there is a 50/50 chance that any particular person may have male genitalia under their pants. Hmm.. Schrodinger's Dick Pic???

So my simple web server, serving up some basic info - like maybe my most recent cat photos.. Are you saying that I *must* use SSL to do this? And to make SSL work I have to pay to get a certificate (cuz I don't really trust the freebie options yet). All so that visitors to my site will *know* that they are looking at cat pictures securely? That doesn't really make too much sense, and seems to suggest a broad assumption about the main purpose of web sites. Not everything requires an encrypted channel. Won't someone think of the kitties? All this hype about safeguarding the Internet for the kids, and not enough to remember that kitties need love too.

Your rights to publish revenue generating advertisements stops the moment that ad attempts to appear on my screen. The screen belongs to me. The hardware that runs it belongs to me. The bandwidth to retrieve your ad belongs to me (by virtue of the fact I am paying for it). What I choose to have appear on my screen is MY choice - not yours. If I choose to replace all advertisements of one-eyed midget Unicorns, that is up to me. If I choose to utilize a tool to prevent the ads from using my bandwidth in the first place, that is up to me. If I choose to blacklist all the IPs that your ads appear on, that is also up to me. You have every right to publish your ad. You do NOT have the right to force me to look at it.

I haven't seen it in the comments yet, but by seizing the site and NOT shutting it down, the government chose to run a child porn server. Does that not then put them under the same legal scrutiny as those they were investigating? Of course I did not read the article and may be missing a bunch of detail, but if the gov was actively serving child porn, then THAT is a crime in my eyes - regardless if it was a honeypot or not.