alien_blueprint's Journal: Python decorators - an overview

I said I'd stay away from code today, but I found myself with nothing to do tonight after kicking people out and so started mucking around with the new Python "decorator" syntax. Trouble is, the documentation on this new feature is very terse and there are certainly no examples, and so it took me a little bit of fiddle work and experimenting to figure out exactly how to write my own decorators.

So, as my Xmas present to my /. friends who put up with my occasional rantings here, I will try to present a quick overview of what it's all about plus a couple of examples that I've come up with that I think are pretty interesting and, even better, that no-one else seems to have implemented - namely a "synchronized" and a "private" method decorator, both of which I've wanted Python syntax for since, well, since forever.

Here's the problem: Python has a simple, clean syntax. But often there's a lot of things you want to declare about a method that isn't necessarily functional - its scope (class or instance), its accessibility (private, public, etc) and so on for the superset of all such declarations in your favourite other language.

The solution: The common idiom in Python is to "wrap" your method in *another* method that provides the new additional capability. For example, to declare a method "class scoped", you might say something like:

class Klass:
    def method(class):
        # Does something
        pass

    method = classmethod(method)

So, the "classmethod" call creates a new method that wraps the original method and implements whatever is required to make it a class method.

Trouble is, as you may have realized, this isn't even slightly clear to the reader - it's backwards for a start (the fact that it's a class method should be declared up front), and this "wrapping" phase could in fact occur *anywhere* in the class after the initial method declaration, making for incredibly unreadable code. So the answer the Python development people came up with was to preserve the whole idea of flexibly extending methods by wrapping them with other methods - which is very Python-ic and very extensible - but to have a clear and convenient syntax for it. So they decided on the "@" notation, where the code above becomes:

class Klass:
    @classmethod
    def method(class):
        pass

This has exactly the same semantics as the earlier code. Even better, they can be chained:

@some_decorator
@other_decorator
def method(self, arg1, arg2):
    pass

This is exactly equivalent to:

def method(self, arg1, arg2):
    pass

method = some_decorator(other_decorator(method))

So, what's this give you? Out of the box, Python 2.4 comes with two standard decorators, "classmethod" and "staticmethod". Which makes sense as one of the most common questions is "how do I create a static/class method in Python?" C++ and Java coders may not see why there needs to be separate "classmethod" and "staticmethod" decorators, but for the curious it's explained in the library reference.

What did disappoint me a little was the lack of a standard "synchronized" decorator. We've needed this in Python for a while - using the "try/finally" idiom, while okay, was just noisy and verbose and possibly error-prone. So I thought I'd have a go at this one myself as a learning exercise.

First of all, obviously the synchronized decorator is going to need a lock to work with, so any class that uses this decorator will have to derive from a base class that provides the lock. In Python, this isn't so bad as multiple inheritence is actually supported.

So, here's the base class:

class Synchronizable:
        def __init__(self):
                self.lock = RLock()

This just gives us a lock to work with. Now here's the decorator function itself:

def synchronized(method, *args):

        def sync_wrapper(object, *args):
                object.lock.acquire()
                try:
                        # Call method
                        return method(object, *args)
                finally:
                        object.lock.release()

        return sync_wrapper

This function has to return a new function that will be called in place of the decorated method. So as you can see we create a new function that simply wraps a call to the *actual* method in an acquire/release pair. Note the use of "try/finally" to guarantee that the lock is released even if the method exits with an exception.

Here's how you might use it, from my test code:

class SafeInteger (Synchronizable):
    def __init__(self, i = 0):
        Synchronizable.__init__(self)
        self.i = i

    @synchronized
    def get_value(self):
        return self.i

    @synchronized
    def increment(self, inc = 1):
        old_value = self.i
        sleep(0.01) # Deliberately create a race condition.
        new_value = old_value + inc
        self.i = new_value

By the way, "increment" is unnecessarily complex because I wanted to force a race condition. It's hardly a rigorous test, more of a demonstration, but without the @synchronized decorator you get problems when more than one thread tries to call "increment" on the same object - which is exactly what you would expect.

I might bug the Python dev team and see if they want to add this - but I can see now why they're holding off. Basically, the problem is this: what if people don't want to use an RLock? It is possible for decorators to be function calls themselves that accept things like arguments with default values, so maybe I'll make it customizable that way first. I'll have to think about it.

The other kind of declaration I've always (kind of) wanted Python to have was "private". In practice, this never comes up. You declare that these methods here are your public interface with a comment, and these over here aren't to be used, and I've never had any problems with people being evil. However, it is one of the first things people want for whatever reason, so after seeing it suggested on usenet when looking for a "@synchronized" implementation, but with no actual implementation yet, I thought I'd try it. Basically, you have to work up through the call stack and see that your caller is another method in the same class, and if it's not, throw an exception. Here's my rough and highly experimental first try:

def private(func, *args):
        def private_wrapper(object, *args):
                self_class = object.__class__

                frame = inspect.currentframe()
                while frame != None:
                        try:
                                local_self = frame.f_locals['self']

                                caller_class = local_self.__class__

                                if caller_class != self_class:
                                        # Trouble
                                        raise "Private method called!"

                                # We're okay, an internal method called this method
                                break

                        except KeyError:
                                pass

                        frame = frame.f_back

                # Look's like everything's fine, so just call the method
                return func(object, *args)

        return private_wrapper

Okay it *works*, but obviously we need to throw some specific class instance, not just a string. Also, it would be nice to have a test up front to see if this checking has been switched off - you would want to switch it off transparently in production code.

It's easy to use, you just say something like:

class SafeInteger (Synchronizable):
    def __init__(self, i = 0):
        Synchronizable.__init__(self)
        self.i = i

    @synchronized
    @private
    def get_value(self):
        return self.i

    @synchronized
    def increment(self, inc = 1):
        old_value = self.get_value()
        sleep(0.01) # Deliberately create a race condition.
        new_value = old_value + inc
        self.i = new_value

And now "get_value" can only be called from methods of the "SafeInteger" class - notice I've slightly altered "increment" to call "get_value" to test that. It seems to work okay, and the traceback error message you get when the exception is thrown is actually pretty informative. I guess the next step is to add "protected", which checks that the caller is either the same class or a derived class.

So, I'm going to see what people think, and maybe get these decorators or something like them added for a future Python release. However, I'm probably just going to find it's happening already, and in a much neater way than I've done. I hope, at least, for those of you that do use Python my examples can get you started with writing your own decorators - as I said at the start the documentation on this new feature is incredibly sparse and brief so it took a little mucking about just to get anything working at all. I think there's a lot of potential here for adding all kinds of interesting decorators - tracing, remote accessibility, optional argument type declarations ... and those are just the *sane* ones that I can think of!