A Primer on Python Metaclass Programming

by David Mertz
04/17/2003

This series of articles by David Mertz assumes readers have a familiarity with basic object-oriented programming concepts: inheritance, encapsulation, and polymorphism. We pick up where the basics leave off; how some "exotic" techniques can make applied programming tasks easier, more maintainable, and just plain more elegant. This series will primarily present examples in Python, but the concepts apply to other programming languages too.

This first installment examines metaclasses. Just as ordinary instances (objects) are built out of classes, classes themselves are built out of metaclasses. Most of the time, you do not need to worry about the implicit metaclass (named type in Python) that is used in class construction, but occasionally, deliberate control of a class' metaclass can produce powerful effects. For example, metaclasses allow "aspect oriented programming," meaning you can enhance classes with features like tracing capabilities, object persistence, exception logging, and more.

An Object-Oriented Programming Review

Related Reading Python in a Nutshell By Alex Martelli

In general principles, OOP works the same way across many programming languages, modulo minor syntax differences. In an object-oriented programming language, you can define classes that bundle together related data and behavior. These classes can inherit some or all of their qualities from their parents, but can also define attributes (data) or methods (behavior) of their own. Classes generally act as templates for the creation of instances (or simply objects). Different instances of the same class typically have different data but share the same "shape"--e.g., the Employee objects bob and jane both have a .salary and a .room_number, but not the same room and salary as each other.

A Metaprogramming Rejoinder

In Python (and other languages), classes are themselves objects that can be passed around and introspected. Just as regular classes act as templates for producing instances, metaclasses act as templates for producing classes.

Python has always had metaclasses. The metaclass machinery became exposed much better with Python 2.2. Specifically, with version 2.2, Python stopped being a language with just one special (mostly hidden) metaclass that created every class object. Now, programmers can subclass the built-in metaclass type and even dynamically generate classes with varying metaclasses.

You do not need to use custom metaclasses to manipulate the production of classes, however. A slightly less brain-melting concept is a class factory. An ordinary function can return a class that is dynamically created within the function body. In traditional Python syntax, you can write the following.

Example 1. An old-fashioned Python 1.5.2 class factory

Python 1.5.2 (#0, Jun 27 1999, 11:23:01) [...]
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> def class_with_method(func):
...     class klass: pass
...     setattr(klass, func.__name__, func)
...     return klass
...
>>> def say_foo(self): print 'foo'
...
>>> Foo = class_with_method(say_foo)
>>> foo = Foo()
>>> foo.say_foo()
foo

The factory function class_with_method() dynamically creates and returns a class that contains the method/function passed into the factory. The class itself is manipulated within the function body before being returned. The new module provides a more concise spelling, but without the same options for custom code within the body of the class factory:

Example 2. A class factory in the new module

>>> from new import classobj
>>> Foo2 = classobj('Foo2',(Foo,),{'bar':lambda self:'bar'})
>>> Foo2().bar()
'bar'
>>> Foo2().say_foo()
foo

In all of these cases, the behaviors of the class (Foo, Foo2) are not directly written as code, but are instead created by calling functions at runtime, with dynamic arguments. Not only are the instances dynamically created, so are the classes themselves.

From Class Factories to Metaclasses

Methods (i.e., of classes), like plain functions, can return objects. In that sense, it is obvious that class factories can be classes just as easily as they can be functions. In particular, Python 2.2+ provides a special class called type that is just such a class factory. The new class type is backwards-compatible with the older function of the same name, by the way. The class type works as a class factory in the same way that the function new.classobj does:

Example 3. type as a class factory metaclass

>>> X = type('X',(),{'foo':lambda self:'foo'})
>>> X, X().foo()
(<class '__main__.X'>, 'foo')

Since type is now a (meta)class, you are free to subclass it:

Example 4. A type descendent as class factory

>>> class ChattyType(type):
...     def __new__(cls, name, bases, dct):
...         print "Allocating memory for class", name
...         return type.__new__(cls, name, bases, dct)
...     def __init__(cls, name, bases, dct):
...         print "Init'ing (configuring) class", name
...         super(ChattyType, cls).__init__(name, bases, dct)
...
>>> X = ChattyType('X',(),{'foo':lambda self:'foo'})
Allocating memory for class X
Init'ing (configuring) class X
>>> X, X().foo()
(<class '__main__.X'>, 'foo')

The magic methods .__new__() and .__init__() are special, but in conceptually the same way that they are for any other class. The .__init__() method lets you configure the created object and the .__new__() method lets you customize its allocation. The latter is not widely overridden, but does exist for every Python 2.2 new-style class.

There is one feature of type descendents to be careful about; it catches everyone who first plays with metaclasses. The first argument to methods is conventionally called cls rather than self, because the methods operate on the produced class, not the metaclass. Actually, there is nothing special about this. All methods attach to their instances, and the instance of a metaclass is a class. A better name makes this more obvious:

Example 5. Attaching class methods to produced classes

>>> class Printable(type):
...     def whoami(cls): print "I am a", cls.__name__
...
>>> Foo = Printable('Foo',(),{})
>>> Foo.whoami()
I am a Foo
>>> Printable.whoami()
Traceback (most recent call last):
TypeError:  unbound method whoami() [...]

All of this surprisingly non-remarkable machinery comes with some syntax sugar that both makes working with metaclasses easier and confuses new users. There are several elements to the extra syntax. The resolution order of these new variations is tricky though. Classes can inherit metaclasses from their ancestors--notice that this is not the same thing as having metaclasses as ancestors (another common confusion). For old-style classes, defining a global __metaclass__ variable can force a custom metaclass to be used. Most of the time, the safest approach is to set a __metaclass__ class attribute for a class that wants to be created via a custom metaclass. You must set the variable in the class definition itself, since the metaclass is not used if the attribute is set later (after the class object has already been created). For example:

Example 6. Setting a metaclass with a class attribute

>>> class Bar:
...     __metaclass__ = Printable
...     def foomethod(self): print 'foo'
...
>>> Bar.whoami()
I am a Bar
>>> Bar().foomethod()
foo

Pages: 1, 2

Next Page