Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror
×

Comment Re:Correction (Score 1) 97

How so? What did it accomplish or change?

There's more than a touch of irony in military project that reached the ultimate high ground only to show us that the world domination game was not worth playing.

But I guess you had to be there to really grasp the significance of Apollo's role in the cold war. Personally I think the 1968 "earthrise" photo from apollo 8 was the most significant contribution, it's often credited with igniting the environmental movement (along with the book "Silent Spring").

The notion of the "pale blue dot" (google it) came out of that photo and exploded in our cultural consciousness several years before Carl Sagan gave it an eloquent voice. The Earthrise photo made it very clear in a lot of people's minds that there is nowhere else to go in the foreseeable future. It was clear that mankind had run out of territory to conquer on Earth and it asked the question at the height of the Vietnam war - why are we still squabbling over the spoils?

Earthrise and the PBDot are now popular cultural icons, they say something to us in the same way a red cross says something to a soldier on a battlefield.

Comment Re:I was six years old watching that (Score 1) 211

The National Geographic that came out with the wonderful moon maps and photos was a treasure of my childhood.

I still have a copy of that issue. :)

The "mankind" thing was just poetry for a domestic audience, read Kennedy's speech and it's crystal clear that the Apollo project was a military response to the "threat" posed by sputnik.

Comment Re:complex application example (Score 4, Informative) 161

> the first ones used threads, semaphores through python's multiprocessing.Pipe implementation.

I stopped reading when I came across this.

Honestly - why are people trying to do things that need guarantees with python?

because we have an extremely limited amount of time as an additional requirement, and we can always rewrite critical portions or later the entire application in c once we have delivered a working system that means that the client can get some money in and can therefore stay in business.

also i worked with david and we benchmarked python-lmdb after adding in support for looped sequential "append" mode and got a staggering performance metric of 900,000 100-byte key/value pairs, and a sequential read performance of 2.5 MILLION records. the equivalent c benchmark is only around double those numbers. we don't *need* the dramatic performance increase that c would bring if right now, at this exact phase of the project, we are targetting something that is 1/10th to 1/5th the performance of c.

so if we want to provide the client with a product *at all*, we go with python.

but one thing that i haven't pointed out is that i am an experienced linux python and c programmer, having been the lead developer of samba tng back from 1997 to 2000. i simpy transferred all of the tricks that i know involving while-loops around non-blocking sockets and so on over to python. ... and none of them helped. if you get 0.5% of the required performance in python, it's so far off the mark that you know something is drastically wrong. converting the exact same program to c is not going to help.

The fact you have strict timing guarantees means you should be using a realtime kernel and realtime threads with a dedicated network card and dedicated processes on IRQs for that card.

we don't have anything like that [strict timing guarantees] - not for the data itself. the data comes in on a 15 second delay (from the external source that we do not have control over) so a few extra seconds delay is not going to hurt.

so although we need the real-time response to handle the incoming data, we _don't_ need the real-time capability beyond that point.

Take the incoming messages from UDP and post them on a message bus should be step one so that you don't lose them.

.... you know, i think this is extremely sensible advice (which i have heard from other sources) so it is good to have that confirmed... my concerns are as follows:

questions:

* how do you then ensure that the process receiving the incoming UDP messages is high enough priority to make sure that the packets are definitely, definitely received?

* what support from the linux kernel is there to ensure that this happens?

* is there a system call which makes sure that data received on a UDP socket *guarantees* that the process receiving it is woken up as an absolute priority over and above all else?

* the message queue destination has to have locking otherwise it will be corrupted. what happens if the message queue that you wish to send the UDP packet to is locked by a *lower* priority process?

* what support in the linux kernel is there to get the lower priority process to have its priority temporarily increased until it lets go of the message queue on which the higher-priority task is critically dependent?

this is exactly the kind of thing that is entirely missing from the linux kernel. temporary automatic re-prioritisation was something that was added to solaris by sun microsystems quite some time ago.

to the best of my knowledge the linux kernel has absolutely no support for these kinds of very important re-prioritisation requirements.

Comment complex application example (Score 4, Insightful) 161

i am running into exactly this problem on my current contract. here is the scenario:

* UDP traffic (an external requirement that cannot be influenced) comes in
* the UDP traffic contains multiple data packets (call them "jobs") each of which requires minimal decoding and processing
* each "job" must be farmed out to *multiple* scripts (for example, 15 is not unreasonable)
* the responses from each job running on each script must be collated then post-processed.

so there is a huge fan-out where jobs (approximately 60 bytes) are coming in at a rate of 1,000 to 2,000 per second; those are being multiplied up by a factor of 15 (to 15,000 to 30,000 per second, each taking very little time in and of themselves), and the responses - all 15 to 30 thousand - must be in-order before being post-processed.

so, the first implementation is in a single process, and we just about achieve the target of 1,000 jobs but only about 10 scripts per job.

anything _above_ that rate and the UDP buffers overflow and there is no way to know if the data has been dropped. the data is *not* repeated, and there is no back-communication channel.

the second implementation uses a parallel dispatcher. i went through half a dozen different implementations.

the first ones used threads, semaphores through python's multiprocessing.Pipe implementation. the performance was beyond dreadful, it was deeply alarming. after a few seconds performance would drop to zero. strace investigations showed that at heavy load the OS call futex was maxed out near 100%.

next came replacement of multiprocessing.Pipe with unix socket pairs and threads with processes, so as to regain proper control over signals, sending of data and so on. early variants of that would run absolutely fine up to some arbitrarry limit then performance would plummet to around 1% or less, sometimes remaining there and sometimes recovering.

next came replacement of select with epoll, and the addition of edge-triggered events. after considerable bug-fixing a reliable implementation was created. testing began, and the CPU load slowly cranked up towards the maximum possible across all 4 cores.

the performance metrics came out *WORSE* than the single-process variant. investigations began and showed a number of things:

1) even though it is 60 bytes per job the pre-processing required to make the decision about which process to send the job were so great that the dispatcher process was becoming severely overloaded

2) each process was spending approximately 5 to 10% of its time doing actual work and NINETY PERCENT of its time waiting in epoll for incoming work.

this is unlike any other "normal" client-server architecture i've ever seen before. it is much more like the mainframe "job processing" that the article describes, and the linux OS simply cannot cope.

i would have used POSIX shared memory Queues but the implementation sucks: it is not possible to identify the shared memory blocks after they have been created so that they may be deleted. i checked the linux kernel source: there is no "directory listing" function supplied and i have no idea how you would even mount the IPC subsystem in order to list what's been created, anyway.

i gave serious consideration to using the python LMDB bindings because they provide an easy API on top of memory-mapped shared memory with copy-on-write semantics. early attempts at that gave dreadful performance: i have not investigated fully why that is: it _should_ work extremely well because of the copy-on-write semantics.

we also gave serious consideration to just taking a file, memory-mapping it and then appending job data to it, then using the mmap'd file for spin-locking to indicate when the job is being processed.

all of these crazy implementations i basically have absolutely no confidence in the linux kernel nor the GNU/Linux POSIX-compliant implementation of the OS on top - i have no confidence that it can handle the load.

so i would be very interested to hear from anyone who has had to design similar architectures, and how they dealt with it.

Comment Re:Other Systems (Score 1) 127

My bones say - close enough to 40 in round numbers.

Never played myself, I was 17-18 when I first heard of it. It appeared to me to have nothing to do with motorbikes or girls so it failed to hold my attention. We did however as younger teens play a (nameless) game that used a die, a ruler, an eraser, some pencils and a roll of wallpaper or similar.

The idea was to set up a battle, drawing in pencil the units of your army on your end of the paper. To move a unit you erased and redrew it, the dice determined how far a unit could move (in inches). To shoot you put the pencil point on your unit, put one finger on top of the pencil to hold it upright, and flick it with your fingers on the other hand. The pencil would leave a line on the paper which represented your bullet. Other than that there weren't any fixed rules, you could make up your own rules for each game...how small can a unit be, how many hits could a unit take, different coloured pencils for different bullets, a bouncing bomb is a series of pencil flicks, etc, etc. Had just as much fun playing that as a kid as I do playing WoT as an grandfather.

Have no idea if the game has a name or where it originated, it seems to be quite old, my dad showed us how to play but it was not his invention since I found other kids at school who had learnt it from their dad, not sure but I think dad played it as a kid in the late 30's, early 40's. While on the subject of kids games, here's something a bit geeky that will blow a grandchild's mind.

Comment Re:Apple has 'done nothing'??? (Score 1) 139

World of Tanks and other titles from wargaming are IMO "free to play" in the original spirit of the idea, you can get the same in-game advantages with points as you can with a credit card. The credit card just means you progress in the meta-game much faster. But the meta-game is never ending, so who really cares how fast they progress?

Disclaimer: I have been playing video games on and off since ~1970-71, WoT is the only game I currently subscribe to, after a year of playing for free I was convinced they were a company worth supporting. Suitable for kids, no blood and guts. Speaking of teenage kids, don't ever let them use your credit card - end of story.

Comment Re:depends. VBA is very different from systems arc (Score 3, Interesting) 241

TFA is really about the human mind. We understand patterns as different forms of language, music is the most basic and universal, it lights up all areas of the brain, then you have spoken language built on top of musical patterns, then along comes symbolism in the form of writing and icons, math is our most recent and most precise form of natural language.

The take home message is, expose your kids to maths without boring them to death.

Comment Re:it is the wrong way... (Score 4, Interesting) 291

How do you recommend governments act to reduce carbon emissions?

The same way Ronald Regan and the Iron Lady acted to reduce sulphur emmissions that cause acid rain, international cap and trade treaty. Cap and trade is a market solution proposed and implemented by the founders of the neo-conservative movement, that has actually worked as advertised. The problem today is that influential "conservatives" are sitting on coal mines that could easily become stranded assests ten years from now. Funny how the politics turns itself upside down if you watch for long enough.

Comment Re:Or maybe it works the other way arround (Score 1) 57

The thing is there are very good feedback therapies, I saw an application recently that was originally designed to help dancers perfect their moves. A neuroscientists working near the Sydney opera house who was interested in dance found it also helped stroke patients, undermining 35yrs of her own work in the field. But like any real scientists she had found a "better answer", so her opinion spun on a dime.

The fact that scammers make similar claims for a $5 app doesn't distract from the real benefits "biofeedback" can have, it distracts from the app store selling it and the ignoramus buying it.

Comment Re:Such harassment (Score 1) 362

Agreed, the only thing they all have in common is bad taste. Ironic how a sex offended was the head cheerleader for the sex-offenders list in the US congress (until he was caught). I always found it suspicious when a male habitually goes into impromptu rants against sexual offenders, he who shouts loudest and all that....

Comment legal ramifications of identity verification (Score 1) 238

i think one of two things happened, here. first is that it might have finally sunk in to google that even just *claiming* to have properly verified user identities leaves them open to lawsuits should they fail to have properly carried out the verification checks that other users *believe* they have carried out. every other service people *know* that you don't trust the username: for a service to claim that they have truly verified the identity of the individual behind the username is reprehensibly irresponsible.

second is that they simply weren't getting enough people, so have quotes opened up the doors quotes.

Slashdot Top Deals

The one day you'd sell your soul for something, souls are a glut.

Working...