The same exact reasoning to justify TSA

They're incomparable. TSA is mandated by governments, you have no choice in the matter. Using a particular brand of smartphone is not. You are free to use a smartphone that doesn't use Google services and indeed are free to buy a Nexus 5 and then say "no" to the billion and one "trade data for feature?" prompts that appear when switched on the first time. No government goon is going to step in and insist that you send all your data to Google.

In fact, if you would prefer a smartphone that has a different data/features tradeoff then - conveniently! - Google provides a rather good open source operating system for free that you can use to build one. If others feel the same way you do you can even sell them without paying Google a dime.

Depends how you define "very popular" I guess. The most popular way to bypass state-level censorship in the Arab world and elsewhere is a product called HotSpot Shield. When Turkey blocked Twitter some time ago, HSS experienced 1000% growth and reached 1.1 million installs in the iOS App Store alone within only four days, with 800,000 regular users.

In contrast Tor went from 30,000 to 40,000 "direct connects" from Turkey.

HSS doesn't get much press in the geek world as it's just a plain old VPN run by a company in California that inserts ads into people's web pages to pay for the bandwidth costs. But usage wise it utterly dominates Tor.

hi mr thinly-sliced, thank you this is awesome advice, really really appreciated.

RTFS? It says that in the summary. The goal here is to alert people who don't know their internet connection is being used for piracy and who aren't OK with freeloading, parents being the given example.

Nothing I have read about Snowden indicates that he is actually some sort of uber-hacker

Except the stuff about how a 29 year old completely pwnd the NSA, probably the most technically sophisticated part of the US Government there is?

Sheesh. Your standards are high. What would it take, exactly?

Additionally, just because you have read nothing about his programming skills doesn't mean he has none. He once mentioned finding XSS holes in some CIA app so apparently he is good enough to do that.

There are already plenty of CA's in countries that are not under US jurisdiction. However, so far the CA's that issued bad certs were all outside the USA, and appear to have only done so because they got hacked and not because they were e.g. forced to by court order.

Unless you have a magical solution to hacking I don't think your new root CA would solve much.

Additionally, citation needed for "routine man in the middle". SSL MITM has been studied by academics at scale. They did not find evidence of much. Governments don't need to MITM SSL for as long as users browse non-SSLd sites like Slashdot and browser exploits exist.

try googletranslating http://lb.ua/news/2014/07/20/2... [lb.ua] - ukrainian army detains 23 terrorists. somehow all 23 turn out to be citizens of the russian federation.

That page is merely reporting a press release from the Ukrainian government in Kiev. Are you suggesting we should treat everything they say as factually true?

let's bisect the other thing you said - "at most Russia is supplying weapons to them".
"at most". as if they were given bows and arrows. they get armoured vehicles. they get... tanks. they get bloody sam systems that can reach targets up to 25km.

Yes. That's what I said. Perhaps this is a language issue.

Whatever is happening in Ukraine it is not a full-blown invasion by Russia in the "classical" style that Iraq or Afghanistan were. That would be far more obvious. It seems to be much more similar to what's been happening in Syria where the west has been supplying weapons, training and expertise to anti-Assad groups there. If you were to say the west has "at most been supplying weapons and training to the Syrian rebels" you would be correct, given that (fortunately) Syria was not invaded by a foreign army.

Not exactly. There is a distinct difference between a soldier and a combatant. A soldier is trained and is a member of a standing military. The separatists can at best be described as "irregulars", or insurgents or rebels if you want to go with slightly more charged terminology.

Yes, really? With that definition it'd be impossible for a new military to ever be created, because anyone who joins and fights with one is not joining a standing army therefore cannot be soldiers. That is obviously nonsense, it must be possible for someone to be a soldier in a newly formed army, which is what it looks like is happening here.

Additionally, you claim that the fighters in Donetsk cannot be soldiers because soldiers are trained, and then immediately claim they're receiving training from Russia. So which is it?

And given the fact that the missiles were launched from inside territory controlled by the rebelsis a very important detail. Why would the Ukrainians have anti-air equipment deployed in an area they do not control, against an enemy with no air power?

You're quite right - it probably was the separatists. This does not change the accuracy of the Wikipedia edit that's being discussed, because unless/until the separatists win, they are still Ukrainians.

Although I'd note that given the amount of bullshit emanating from all sides in this conflict it's hard to really know anything about what's going on. The area of Ukraine that's in revolt is next to the Russian border, which is exactly where you'd expect the Ukrainian military to have had lots of soldiers and equipment stationed. Missiles might have been trucked over the Russian border, or they might simply have been there already. The separatists might be being trained by Russians (this would be unsurprising and not exactly unprecedented - see how the USA supported rebels in Syria), or alternatively they might be operating the equipment without really knowing what they're doing - indeed, having no clue what you're targeting would be rather indicative of not being properly trained, no? Or perhaps they're being trained by people who are ethnically Russian but lived in Ukraine at the time of the rebellion, or one of many other more complex cases that won't neatly fit into the "Putin fired the missiles himself" story the west is busy pushing.

All we can say for sure is that whatever you read about this incident is going to be full-blown propaganda, and should be treated as such.

I don't think Russian state media should be editing Wikipedia entries especially not on matters of current affairs.

But still, interpreted literally the new statement is far more factually correct and unbiased than what it replaced. Whoever shot down the plane, they were "soldiers" or fighters of some variety and almost certainly can be described as Ukrainian, given that everyone seems to agree that the fighters are actually eastern Ukrainians and at most Russia is supplying weapons to them.

The original text, on the other hand, more or less exactly sums up western/west Ukrainian line despite the obvious abuse of the word terrorist to mean "rebel fighter" and the [citation needed] assertion about who did it and the source of the weapons.

There is no need to get rid of Tor: in theory, Tor could have a "hidden service policy" mechanism not much different to the exit policy mechanism. HS Policies would allow a node operator to state that they aren't willing to act as an introduction point for a list of hidden services (or point to lists maintained elsewhere to stop fast-flux type behaviour).

Tor already accepts that not all relay operators will want to support all kinds of behaviour and that some kinds of traffic can be abusive, that's why they implement exit policies which allow exits to ban port and IP ranges. Taking this philosophy to hidden services seems like the next natural step. After all, Tor volunteers are ultimately acting as human shields for other people's anonymous behaviour. Requiring them to shield everything just restricts the number of people who would be willing to donate bandwidth to general privacy but are not interested in enabling botnets.

Not sure what you're getting at, but the Azul collector is well known for pulling off apparently magical GC performance. They do it with a lot of very clever computer science that involves, amongst other things, modifications to the kernel. I believe they also used to use custom chips with extended instruction sets designed to interop well with their custom JVM. Not sure if they still do that. The result is that they can do things like GC a 20 gigabyte heap in a handful of milliseconds. GC doesn't have to suck.

because we have an extremely limited amount of time as an additional requirement, and we can always rewrite critical portions or later the entire application in c once we have delivered a working system that means that the client can get some money in and can therefore stay in business.

also i worked with david and we benchmarked python-lmdb after adding in support for looped sequential "append" mode and got a staggering performance metric of 900,000 100-byte key/value pairs, and a sequential read performance of 2.5 MILLION records. the equivalent c benchmark is only around double those numbers. we don't *need* the dramatic performance increase that c would bring if right now, at this exact phase of the project, we are targetting something that is 1/10th to 1/5th the performance of c.

so if we want to provide the client with a product *at all*, we go with python.

but one thing that i haven't pointed out is that i am an experienced linux python and c programmer, having been the lead developer of samba tng back from 1997 to 2000. i simpy transferred all of the tricks that i know involving while-loops around non-blocking sockets and so on over to python. ... and none of them helped. if you get 0.5% of the required performance in python, it's so far off the mark that you know something is drastically wrong. converting the exact same program to c is not going to help.

we don't have anything like that [strict timing guarantees] - not for the data itself. the data comes in on a 15 second delay (from the external source that we do not have control over) so a few extra seconds delay is not going to hurt.

so although we need the real-time response to handle the incoming data, we _don't_ need the real-time capability beyond that point.

.... you know, i think this is extremely sensible advice (which i have heard from other sources) so it is good to have that confirmed... my concerns are as follows:

questions:

* how do you then ensure that the process receiving the incoming UDP messages is high enough priority to make sure that the packets are definitely, definitely received?

* what support from the linux kernel is there to ensure that this happens?

* is there a system call which makes sure that data received on a UDP socket *guarantees* that the process receiving it is woken up as an absolute priority over and above all else?

* the message queue destination has to have locking otherwise it will be corrupted. what happens if the message queue that you wish to send the UDP packet to is locked by a *lower* priority process?

* what support in the linux kernel is there to get the lower priority process to have its priority temporarily increased until it lets go of the message queue on which the higher-priority task is critically dependent?

this is exactly the kind of thing that is entirely missing from the linux kernel. temporary automatic re-prioritisation was something that was added to solaris by sun microsystems quite some time ago.

to the best of my knowledge the linux kernel has absolutely no support for these kinds of very important re-prioritisation requirements.

i am running into exactly this problem on my current contract. here is the scenario:

* UDP traffic (an external requirement that cannot be influenced) comes in
* the UDP traffic contains multiple data packets (call them "jobs") each of which requires minimal decoding and processing
* each "job" must be farmed out to *multiple* scripts (for example, 15 is not unreasonable)
* the responses from each job running on each script must be collated then post-processed.

so there is a huge fan-out where jobs (approximately 60 bytes) are coming in at a rate of 1,000 to 2,000 per second; those are being multiplied up by a factor of 15 (to 15,000 to 30,000 per second, each taking very little time in and of themselves), and the responses - all 15 to 30 thousand - must be in-order before being post-processed.

so, the first implementation is in a single process, and we just about achieve the target of 1,000 jobs but only about 10 scripts per job.

anything _above_ that rate and the UDP buffers overflow and there is no way to know if the data has been dropped. the data is *not* repeated, and there is no back-communication channel.

the second implementation uses a parallel dispatcher. i went through half a dozen different implementations.

the first ones used threads, semaphores through python's multiprocessing.Pipe implementation. the performance was beyond dreadful, it was deeply alarming. after a few seconds performance would drop to zero. strace investigations showed that at heavy load the OS call futex was maxed out near 100%.

next came replacement of multiprocessing.Pipe with unix socket pairs and threads with processes, so as to regain proper control over signals, sending of data and so on. early variants of that would run absolutely fine up to some arbitrarry limit then performance would plummet to around 1% or less, sometimes remaining there and sometimes recovering.

next came replacement of select with epoll, and the addition of edge-triggered events. after considerable bug-fixing a reliable implementation was created. testing began, and the CPU load slowly cranked up towards the maximum possible across all 4 cores.

the performance metrics came out *WORSE* than the single-process variant. investigations began and showed a number of things:

1) even though it is 60 bytes per job the pre-processing required to make the decision about which process to send the job were so great that the dispatcher process was becoming severely overloaded

2) each process was spending approximately 5 to 10% of its time doing actual work and NINETY PERCENT of its time waiting in epoll for incoming work.

this is unlike any other "normal" client-server architecture i've ever seen before. it is much more like the mainframe "job processing" that the article describes, and the linux OS simply cannot cope.

i would have used POSIX shared memory Queues but the implementation sucks: it is not possible to identify the shared memory blocks after they have been created so that they may be deleted. i checked the linux kernel source: there is no "directory listing" function supplied and i have no idea how you would even mount the IPC subsystem in order to list what's been created, anyway.

i gave serious consideration to using the python LMDB bindings because they provide an easy API on top of memory-mapped shared memory with copy-on-write semantics. early attempts at that gave dreadful performance: i have not investigated fully why that is: it _should_ work extremely well because of the copy-on-write semantics.

we also gave serious consideration to just taking a file, memory-mapping it and then appending job data to it, then using the mmap'd file for spin-locking to indicate when the job is being processed.

all of these crazy implementations i basically have absolutely no confidence in the linux kernel nor the GNU/Linux POSIX-compliant implementation of the OS on top - i have no confidence that it can handle the load.

so i would be very interested to hear from anyone who has had to design similar architectures, and how they dealt with it.