Sloppy's Journal: A concern about Python

I'm getting a little concerned about one of Python's weaknesses. It's not a big deal yet, but it's coming up in the future. It's just a question of how soon and how bad.

It's the "global interpreter lock" and the fact that CPython programs don't scale well with multiprocessor machines. In a way, I used to have the same concern about OpenBSD, but it eventually got addressed. And the problem is this: with the meanstreamization (WTF kind of word is that?!) of multi-core chips, soon everyone is going to have multiprocessing hardware. Not just geeks. Not just people who want lots of processing power. Everyone.

And when that happens, any OSes, environments, language runtimes, etc that don't scale well and take advantage of the ubiquitous existence of such hardware, are going to look lame compared to their competitors. I know no one would use Python for CPU-intense stuff (similar to the OpenBSD situation) but even so, it matters that a threaded Python program doesn't scale up. It's merely a question of how much it matters, and we'll all disagree about that.

The especially sad thing is that the Python language and standard library actually make writing threaded programs really easy. It has some great modules (e.g. Queue) that make all the locking issues transparent and just so incredibly trivial. But it's not enough, if the most popular interpreter happens to implement the scaling poorly.

And don't tell me to break stuff up into processes. I know that. It's a good answer sometimes. But it's not always a good answer. Sometimes multithreading is the most natural way to do things.

Any thoughts? Does switching from CPython to Jython make things better?

/ducks

[ellem]

Isn't Python that language you learn so you'll be better at Ruby?

Re:/ducks

[gmhowell]

No, someone python is for people (like me) who can't get past pseudo code.

No idea about python

[jawtheshark]

I used to have the same concern about OpenBSD, but it eventually got addressed.

And that changed how? Have you ever tried running OpenBSD on a real SMP machine (not talking about Hyperthreading, and I do not know about Dual Core)? Yes, the support is there now, but when I run it on my Athlon MP 2400+, OpenBSD locks dead after a time. I can play hangman on function calls, yeah... fun... Okay, I didn't try OpenBSD 3.9 yet on this machine, but 3.8 certainly doesn't handle SMP well. It might be better on

enter Microsoft

[naoursla]

Microsoft has a version of Python called IronPython that runs in managed code on top of its Common Language Runtime (CLR).

Microsoft has also released a community technology preview of their Robotics Studio (MRS). MRS uses the "next generation" programming model for multi-processor and distributed systems called the Concurrency and Coordination Runtime (CCR) and a hosting environment for the CCR called Decentralized System Services (DSS). Since they both use the CLR, it should be fairly straightforward to us

Python problems

[Tet]

It's the "global interpreter lock" and the fact that CPython programs don't scale well with multiprocessor machines.

Well, yes, but for me, the biggest problem is the suckiness of python's libraries. We're writing a bunch of screen scrapers and data feed decoders. So rather than implement an XML parser from scratch, it makes sense to use one that someone else has already provided, right? We don't need a heavyweight parser, so minidom [python.org] from the standard python library sounds ideal. Lightweight, you say? I be