---

Background

During our recent discussion of Functional Programming, we found that the difficulty of combining existing functions into new ones dramatically decreased when we realized we could do one thing: write a function that returns a new function as its result. In essence, what we were doing was writing functions that wrote functions for us, so we wouldn't have to write them ourselves. It wasn't some kind of science fiction — there was no sentient artificial intelligence doing our bidding — but merely the recognition that some of the ways we combine functions together follow patterns, so if we implement a pattern once, we'll never have to combine them that way by hand again.

• Instead of writing a function that composes two particular functions and returns their result, we can write a function that composes any two functions.
• Instead of writing a function that returns the Boolean negation of another function's result, we can write a function that returns the Boolean negation of any function's result.
• Instead of writing a function that partially calls a function, we can write a function that partially calls any function.

In each case, the result is a new function that we otherwise would have had to write by hand. And, of course, if we need to apply that same pattern many times, we've saved ourselves from writing many functions. The more of these commonly occurring patterns we've implemented, the fewer functions we'll need to write, test, and maintain over time.

So, we found a lot of value in the idea that we might like to combine functions in an automated way, and that higher-order functions presented a nice way to do that, so that we could call them, ask them to build new functions for us, and pass them as arguments to still other higher-order functions.

But what do we do when we want a function in a module to be a combination of existing functions? What about the methods of a class? What if we have ten functions in a module that all need to share a common characteristic? For these kinds of purposes, our best bet is another Python feature called a decorator, which, like a lot of Python features, is a simple idea with a rich and complex set of uses.

---

What are decorators?

A decorator, in its simplest form, is a function that accepts a function as an argument and returns a function as its result.

>>> def unchanged(func):
...     return func
...

Being a function, this decorator can be called like one.

>>> def square(n):
...     return n * n
...
>>> another_square = unchanged(square)
>>> another_square(3)
    9

But the name "decorator" implies that they're somehow used to decorate a function, which gives us an idea that there's more to this feature than we've seen so far. How do we decorate functions? What happens when we do?

We decorate a function by adding a decorator expression above it, in which we specify the name of the decorator preceded by the @ symbol.

>>> @unchanged
... def square(n):
...     return n * n
...
>>> square(3)
    9

When a decorator expression appears above a function, it's as though we wrote something a little more long-winded.

>>> def square(n):
...     return n * n
...
>>> square = unchanged(square)
>>> square(3)
    9

In other words, when we decorate a function, we've asked Python to transform it just after defining it. In this case, the transformation was unimpressive, because our decorator returned the function as-is; that's why calling our decorated square function did the same thing an undecorated version of square would have done.

Writing a decorator that transforms a function

Once we understand the mechanics of decorators, and since we've already become familiar with writing functions that transform other functions, we're fully equipped to make use of decorators. Let's try writing one that returns a newly built function.

>>> def with_greeting(func):
...     def greet_and_execute(n):
...         print('Hello!')
...         return func(n)
...     return greet_and_execute
...
>>> @with_greeting
... def square(n):
...     return n * n
...
>>> square(3)
    Hello!          # square is really a version of greet_and_execute
    9               # that calls square, so 'Hello!' is printed here.
>>> @with_greeting
... def negate(n):
...     return -n
...
>>> negate(3)
    Hello!          # negate has the same greeting behavior as square,
    -3              # but we only had to write it once.
>>> @with_greeting
... def multiply(n, m):
...     return n * m
...
>>> multiply(6, 17)
    Traceback (most recent call last):
      ...
    TypeError: with_greeting.<locals>.greet_and_execute() takes 1 positional argument but 2 were given

We had early success with our with_greeting decorator, but it fell over when we tried to call our decorated multiply function, resulting in a somewhat perplexing error message. What happened when we called multiply, and why was it different from our decorated square and negate functions?

• Because it was decorated with with_greeting, multiply is actually an instance of the greet_and_execute function, with func bound to the original (undecorated) implementation of multiply.
• That means multiply is really a function that has one parameter, n, which will then attempt to call the original multiply function and pass it n as an argument.
• When we called multiply, we passed it two arguments, but that's one more than it's able to accept, so we got an error message to that effect.

If multiply is now a function that has one parameter, then what happens if we call it with one argument instead?

>>> multiply(17)
    Hello!
    Traceback (most recent call last):
      ...
    TypeError: multiply() missing 1 required positional argument: 'm'

Interestingly, we got further this time. The 'Hello!' message was printed to the Python shell, which means we made it into greet_and_execute this time — because it can be called with the one argument we passed it — but it then failed when trying to pass one argument to the original multiply function, which requires two.

So, what does this tell us about the functions returned by decorators? It means that we need to think more carefully about the signatures of the functions involved. What are the signatures of the functions we'd like to be able to decorate? How would we like the signatures of the transformed functions to be different, if at all?

In this case, there's no reason we shouldn't be able to decorate any function with our with_greeting decorator, since all it does is add the ability for a function to print Hello! before it does its regular business. Whether the function requires no arguments, multiple arguments, a variable number of arguments, allows keyword arguments, or anything else, we should be able to decorate it, which means we need to address two related problems.

• The greet_and_execute function needs a signature that accepts any combination of positional and/or keyword arguments.
• When greet_and_execute calls func, it needs to pass along whatever arguments it was passed, regardless of what kind of arguments they are.

Tuple- and dictionary-packing parameters can allow greet_and_execute to accept a more flexible range of possible arguments, while unpacking in the call to func can allow that same flexibility when passing those arguments on.

>>> def with_greeting(func):
...     def greet_and_execute(*args, **kwargs):
...         print('Hello!')
...         return func(*args, **kwargs)
...     return greet_and_execute
...
>>> @with_greeting
... def distance_from_origin(x, y, z = 0):
...     return math.hypot(x, y, z)
...
>>> distance_from_origin(11, 18)
    Hello!
    21.095023109728988        # That's more like it!
>>> distance_from_origin(x = 1, y = 17, z = 6)
    Hello!
    18.05547008526779         # Keyword arguments work, too.

Multiple decorators on the same function

When we decorate a function, the result is a transformed function, but a function nonetheless. This raises the question of whether we can decorate that transformed function to transform it further. In other words, can we write two decorator expressions above a function? If so, what does it mean?

>>> from datetime import datetime
>>> def with_time_display(func):
...     def show_time_and_execute(*args, **kwargs):
...         print(f'Current Time: {datetime.now():%I:%M:%S %p}')
...         return func(*args, **kwargs)
...     return show_time_and_execute
...
>>> @with_greeting
... @with_time_display
... def square(n):
...     return n * n
...                 # We can decorate a function with more than one decorator.
>>> square(3)
    Hello!
    Current Time: 11:07:18 PM
    9

When we decorate a function with multiple decorators, they're applied in the reverse of the order in which we specified them, which means that our definition of square is roughly equivalent to having written this instead.

>>> def square(n):
...     return n * n
...
>>> square = with_time_display(square)
>>> square = with_greeting(square)

To the extent that we transform functions in a way that leaves their signature requirements unchanged, the order in which we list decorators will sometimes be irrelevant, but when those transformations change a function's signature, or otherwise change the reasonable assumptions we can make about how it's called or what it does, then the order of decorators becomes vitally important. And, of course, the order in which we apply decorators with side effects — such as the ones we've written here, which print output to the Python shell — will also be important if the relative order of those side effects is important.

---

Decorators that accept arguments

We've seen that when we decorate a function f with a decorator d, d is called with f passed as an argument to it, with d's result ultimately replacing f in its scope. The purpose of decorating a function is to allow us apply an automatic transformation to a function, rather than having to do it manually, and where this technique is most useful is where the same transformation can be applied to many functions instead of just one.

But what happens when we want a transformation to apply to many functions, but for the transformation itself to be subtly different from one situation to another? Ordinarily, we can influence a function's behavior by passing arguments to it, but decorators seem set in stone; the original function is passed to them, but there doesn't seem to be any other way to pass arguments to it.

To put this question into context, let's imagine that we want to write a decorator that specifies that a function should be retried when it fails (i.e., when an exception is raised), but only a limited number of times. The general pattern one might follow to retry on failure could look like this if, for example, we wanted to retry on the first four failures, but give up after the fifth one.

def read_int():
    for _ in range(4):
        try:
            return int(input('Enter an integer: '))
        except Exception:
            pass

    return int(input('Enter an integer: '))

This is a rote pattern, though, so we could instead implement a decorator retry_on_failure to perform that transformation for us, so that any function that requires retrying could have it grafted in automatically. This would allow us to write this decorated function, eliminating all of the noise of looping and catching exceptions, while leaving behind a decorator that nicely describes that missing code for us.

@retry_on_failure
def read_int():
    return int(input('Enter an integer: '))

But how many times will read_int be retried if it fails? Given only what we've seen, we have nowhere to specify this. It can't be part of read_int's signature or body, because the retrying occurs outside of read_int. Up to this point, our only option is to hard-code a value into retry_on_failure, the way we did in our original read_int function, so that the decorator really means something specific like "Retry five times on failure," but if our decorator is meant to be a general-use tool, it leads to the unfortunate problem of having to write retry_five_times, retry_three_times, and many other decorators: one for each number of times we might like to retry a function.

That would be a disappointing outcome indeed, because what we'd really like is for the decorator itself to be parameterized, so that we can specify separately, each time we apply it, how many times it should retry a function's body. In lieu of building many different retry_on_failure decorators ourselves, we want them to be built for us on demand automatically. What we want is this.

@retry_on_failure(3)
def read_int():
    return int(input('Enter an integer: '))

This turns out to be legal Python syntax with a definitive meaning, though it can be a little tricky to understand and implement, so we'll need to pay careful attention to the details. What we wrote above is essentially equivalent to having written this instead.

def read_int():
    return int(input('Enter an integer: '))

read_int = retry_on_failure(3)(read_int)

Notice, in particular, that there are two calls on the last line.

• First, retry_on_failure is called, with 3 passed as an argument to it. Notably, read_int hasn't been passed yet.
• The result of retry_on_failure is expected to itself be a decorator — a function that transforms another function — so we then pass it read_int, asking it to transform read_int into a function that makes three attempts to read an integer before giving up.

To implement retry_on_failure, then, we have a couple of hoops to jump through.

• The retry_on_failure function is not a decorator, but instead a function that builds a decorator that knows specifically how many times a particular decorated function should be retried.
• The decorator built by retry_on_failure can, in turn, take a function and transform it into one that's retried the specified number of times.

So, we'll need retry_on_failure to return a decorator, which is a function. That function will, in turn, need to return a function. All in all, we need three functions, nested one inside the other, which leads to a pattern like this one.

retries.py (click here for a commented version)

def retry_on_failure(times):
    def decorate(func):
        def run(*args, **kwargs):
            for _ in range(times - 1):
                try:
                    return func(*args, **kwargs)
                except Exception:
                    pass

            return func(*args, **kwargs)

        return run

    return decorate

Once we have retry_on_failure written, we can use it to decorate functions.

>>> @retry_on_failure(3)
... def read_int():
...     return int(input('Enter an integer: '))
...
>>> read_int()
    Enter an integer: ​Boo​
    Enter an integer: ​Alex​
    Enter an integer: ​13​
    13
>>> read_int()
    Enter an integer: ​31​
    31
>>> read_int()
    Enter an integer: ​Boo​
    Enter an integer: ​is​
    Enter an integer: ​happy​
    Traceback (most recent call last):
      ...
    ValueError: invalid literal for int() with base 10: 'happy'

It's important to stop at this point and realize what we've just done, amidst the details of arguments being passed, functions building other functions, and the various levels of nesting. From a big-picture perspective, we just accomplished something bigger than it might seem, so let's be sure we take notice of that accomplishment.

• We started with the realization that there was an aspect of what many functions do that we'd like to write and test once, rather than writing it repeatedly.
• We then realized that there was a slight difference in how it should be done from one function to another — particularly, the number of retries desired — but would otherwise be the same every time.
• With a parameterized decorator, we were able to implement the retrying pattern once, in a way that it could be applied to any function we'd like, with each function being able to separate specify how many times it should be retried.

One might reasonably argue about whether this particular problem is one that would best be solved this way, but the important point here is that this kind of thing is possible. When we can transform functions, we can slice them up into pieces and reassemble them in a variety of ways, which gives us a level of design flexibility we'd never have otherwise.

Higher-order functions indeed.

---

Characterizing how decorators can (and can't) transform functions

Before we get too excited about our newfound abilities, though, we ought to be sure we understand not only what we can accomplish with a decorator, but also what we can't. Decorators can transform functions into new ones, but those transformations aren't entirely arbitrary; they automate certain patterns beautifully, while being entirely unable to automate others. So, we should understand where those limits are.

When we decorate a function, we're essentially limited to making one of four kinds of modifications to it.

• Adding code that runs before the original function's body. We did this in our with_greeting decorator, for example, where we printed 'Hello!' before calling the original function. It's worth noting that this code might adjust the arguments in some way before calling the original one, so that the arguments passed to the transformed function are different from the ones passed to the original function, which means that the transformed function might have a different signature from the original one.
• Adding code that runs after the original function's body. This might be code that has a side effect or code that adjusts the original function's result in some way.
• Combining the first two kinds of modifications, so that we have new code surrounding the original function's body, such as what we did in the retry_on_failure decorator. This would allow us to adjust arguments and return values, but also to surround the original function's body with additional exception handling, or with additional context management (e.g., by calling the original function within a with statement that sets something up before the function is called, while ensuring that it's cleaned up afterward).
• Replacing the original function's body with a new one altogether, which is to say that the transformed function never calls the original function at all. It's not often that you'd make this decision in practice — why write a function and then decorate it in a way that renders it meaningless? — but it's at least one of the possibilities, one that might be of particular value if that wholesale replacement was done conditionally in some way.

Notably missing from this list is the ability to inject or alter code somewhere within the original function's body. We can't use a decorator to make a function do something different itself, but we can alter the way that it's called, the way its result is returned, the ways in which it's considered to have failed (by catching exceptions and/or raising others), or the context in which it will be executed. There's a good reason for this limitation, though; you can't alter what's in a function's body without knowing precisely what's in it, but when we write decorators, our aim is to transform many functions with it, not just one. We can inject code before, after, or surrounding a function's body without depending too heavily on the details of what's in it.

There's one other thing we ought to be sure we understand: Decorators run when functions are defined, not when they're called. In other words, the transformation done by a decorator is done once, and that will generally be when the decorated function is first created (e.g., when the module containing it is first loaded). That means we can't easily decorate a function differently at different points in a program's execution; once it's done, it's done. If we want a function to behave differently depending on the situation at run-time, passing arguments to the function remains the best way to do that. Decorators have the purpose of allowing us to build functions more flexibly, not to call them more flexibly.

Still, despite these limitations, decorators are great to have in our arsenal, because a surprising number of transformations we might like to automate fall into either the "before the function", "after the function", or "surrounding the function" categories.

---

Implementing decorators using classes

We've seen recently that functions aren't the only objects in Python that are callable. Any object whose class has a __call__ method can be called, with the parameters of the __call__ method determining how arguments can be passed to it. If that's true, then why couldn't decorators return objects of classes other than function, as long as those objects are callable?

As a starting point, let's try writing a decorator that returns something other than a function, so we can understand what happens when we use it.

>>> def bad_idea(func):
...     return 'Boo!'
...
>>> @bad_idea
... def square(n):
...     return n * n
...
>>> square(3)
    Traceback (most recent call last):
      ...
    TypeError: 'str' object is not callable
                       # Why can't we call square?
>>> square
    Boo!
>>> type(square)
    <class 'str'>      # Because it's a string!

We've seen that when a decorator expression appears above a function, it's translated into a call to the decorator, with the function passed as an argument to it, as though we had written this instead:

>>> def square(n):
...     return n * n
...
>>> square = bad_idea(square)

So, when we decorated square with bad_idea, bad_idea(square) was called, which returned 'Boo!', which was then stored back into square. Consequently, square is 'Boo!' — a string! — and not a function. (Why we named this decorator bad_idea is because it's probably a bad idea in practice to replace functions with strings like this, but it's helpful to see how literally the rules around decorators are applied, because there might be good ideas that rely on the same mechanisms as this bad one.)

Suppose instead, though, that we wrote a class InterestingIdea with two methods.

• An __init__ method taking a function as a parameter (in addition to self), which would make InterestingIdea(func) legal for any function. Since InterestingIdea is a class, InterestingIdea(func) is an attempt to construct an InterestingIdea object, so the result would be an InterestingIdea object.
• A __call__ method, so that InterestingIdea objects are callable, meaning that we can call them as though they're functions.

If we did that, what would we expect to happen if we used InterestingIdea as a decorator?

@InterestingIdea
def square(n):
    return n * n

By our previous understanding, we'd expect that to be translated into this instead.

def square(n):
    return n * n

square = InterestingIdea(square)

Wouldn't this mean that square is an InterestingIdea object, rather than a function? But since InterestingIdea has a __call__ method, wouldn't we still be able to call it? Let's try it.

>>> class InterestingIdea:
...     def __init__(self, func):
...         pass
...     def __call__(self, *args, **kwargs):
...         return 'Boo!'
...
>>> @InterestingIdea
... def square(n):
...     return n * n
...
>>> type(square)
    <class '__main__.InterestingIdea'>
>>> square(3)
    Boo!

Everything happened just as we expected: square became an InterestingIdea object, and calling it turned into a call to the __call__ method in the InterestingIdea class.

But why would we want something like this? Isn't this just a longer-winded way of writing something that we could have written a lot more simply by just writing an interesting_idea function instead? For a decorator that transforms a function into one that always returns 'Boo!', that's almost certainly true. But classes provide us abilities that functions don't provide as easily:

• Classes have an __init__ method that lets us initialize whatever attributes we'd like.
• Classes can have additional methods, meaning that we could interact with the resulting object in ways other than just calling it, so that our transformed function can have abilities that our original one didn't.

Many times, we won't need either of those abilities, but when we do, we can write a decorator as a class.

call_counting.py (click here for a commented version)

class WithCallsCounted:
    def __init__(self, func):
        self._func = func
        self._count = 0

    def __call__(self, *args, **kwargs):
        self._count += 1
        return self._func(*args, **kwargs)

    def count(self):
        return self._count

Despite the fact that we wrote it as a class, WithCallsCounted is also a decorator, because it conforms to the same requirements that decorators must conform to: I can call WithCallsCounted (the class) like a function and pass it the original function, and the resulting object acts like a transformed version of the original function.

>>> @WithCallsCounted
... def square(n):
...     return n * n
...
>>> square(3)
    9          # The decorated square function acts like the original one when it's called.
>>> square.count()
    1          # But square also knows how many times it's been called.
>>> square(5)
    25         # Does it really?
>>> square.count()
    2          # Yes, it does!
>>> type(square)
    <class 'call_counting.WithCallsCounted'>
               # And this is why.

The sky's the limit, when it comes to the amount of complexity one could write in a decorator implemented as a class. And while we generally want to build simple solutions to simple problems, more complex problems require more complex solutions, so it's best for us to have a way to organize that complexity when the need arises.

---

Decorators within classes

In the previous section, we saw that we can write classes that act as decorators, with their objects taking the place of the functions they decorate. While we're on the subject of classes, though, there's another question worth exploring. Can we use decorators within classes and, if so, what does it mean?

Decorating classes in their entirety

The first thing we might like to try is decorating classes in their entirety, i.e., by writing a decorator expression above a class statement instead of a def statement. When placed above the definition of a function, a decorator expression is evaluated by passing the function as an argument to it, with the result being stored in place of the original function. So, we might expect decorating classes to work the same way, by passing the original class to the decorator function, and then replacing the class with the decorator function's result. Let's see if our expectation holds.

>>> def unchanged(cls):
...     print(f'decorating class: {cls.__name__}')
...     return cls
...              # This is a decorator that tells us it's been called and what its parameter
                 # is, but returns its parameter unchanged.
>>> @unchanged
... class Thing:
...     pass
...
    decorating class: Thing
                 # We see that our decorator function was called, and that its parameter was
                 # the Thing class.
>>> Thing
    <class '__main__.Thing'>
                 # Because the decorator function returned its parameter unchanged, Thing
                 # is still the same class.
>>> def booize(cls):
...     return 'Boo!'
...              # This is another decorator that ignores its parameter and returns a string.
>>> @booize
... class Another:
...     pass
...
>>> Another
    'Boo!'       # Our Another class has been entirely replaced with the string returned
                 # from the booize decorator function.

Those two experiments have given us a pretty good idea that our suspicions are correct. Decorating a class is just like decorating a function, except, of course, it's a class being passed to the decorator function, so the kinds of modifications you might want to make to it are different. Still, the principle is a familiar one, which is always a good place to start.

If the primary job of a class decorator is to modify a class, then there's one more thing we'd better be sure we know how to do before we go any farther. How do you modify a class after it's been built?

>>> class Person:
...     def __init__(self, name, age):
...         self._name = name
...         self._age = age
...              # We'll start by defining a class with an __init__ method and nothing else.
>>> def name(self):
...     return self._name
...              # What if we now create a name function?  How do we add it to the class?
>>> Person.name = name
                 # Classes have attributes.  Why not assign the name function into the
                 # class' name attribute?
>>> p = Person('Boo', 13)
>>> p.name()
    'Boo'        # And now we can call name as a method of our Person class.
>>> def age(self):
...     return self._age
...
>>> Person.age = age
>>> p.age()
    13           # We can even add a method to a class and call it on an object of that class
                 # that existed previously.  This is because the methods aren't stored in the
                 # object's dictionary; they're stored in the dictionary belonging to
                 # the class.

So, now we're fully equipped to write a class decorator. It's just like a function decorator, except its goal is to add, update, or remove attributes in the decorated class. We could use this technique to automate the creation of formulaic code that we might otherwise have to write in a class.

For example, suppose we wanted to build a class like our usual Person example.

class Person:
    def __init__(self, name, age):
        self._name = name
        self._age = age

    def name(self):
        return self._name

    def age(self):
        return self._age

A class like this one is made up entirely of a rote pattern. If we wanted a third value, such as height, to be stored in each Person, the change would be formulaic.

• Add a height parameter to the __init__ method.
• In the body of the __init__ method, store the height parameter's value in a _height attribute within self.
• Add a height() method that returns self._height.

What would I rather be able to say? How about something like this instead?

@fields(('name', 'age'))
class Person:
    pass

To make this possible, we'll need to write a decorator function called fields, which takes an iterable of field names as a parameter, then returns a decorator that makes modifications to a class as though we had written the rote pattern ourselves.

fields.py (click here for a commented version)

def fields(field_names):
    field_names = list(field_names)

    def make_field_getter(field_name):
        def get_field(self):
            return getattr(self, f'_{field_name}')

        return get_field

    def decorate(cls):
        for field_name in field_names:
            setattr(cls, field_name, make_field_getter(field_name))

        def __init__(self, *args):
            for field_name, arg in zip(field_names, args):
                setattr(self, f'_{field_name}', arg)

        cls.__init__ = __init__
        return cls

    return decorate

Note that the built-in functions getattr and setattr can be used to get or set the values of attributes whose names are determined dynamically, which is certainly needed here: We can't know the names of the fields until the decorator is used after we've written it.

So, what happens when we try to use our decorator on an empty class?

>>> @fields(('name', 'age', 'height'))
... class Person:
...     pass
...
>>> Person
    <class '__main__.Person'>
                  # A Person class was defined, as we expected.
>>> Person.__init__
    <function fields.<locals>.decorate.<locals>.__init__ at 0x00001F912FBD900>
                  # We didn't write an __init__ method, but Person has one, which
                  # appears to have been built within our decorate function.
>>> Person.name
    <function fields.<make_field_getter>.get_field at 0x000001F912FBD7E0>
>>> Person.age
    <function fields.<make_field_getter>.get_field at 0x000001F912FBD870>
                  # It also has name and age methods, each of which are functions
                  # returned from our make_field_getter function.
>>> p = Person('Boo', 13, 18)
>>> p.name()
    'Boo'
>>> p.age()
    13
>>> p.height()
    18

There are plenty of things about our fields decorator that could be better, if we were to try to implement something like this for wide-scale use. Our decorated classes will make no attempt to handle too few or too many arguments to their __init__ methods, for example. As we'll see soon, there are tools built into the Python standard library that automate exactly this problem much more completely than we have here.

Nonetheless, the idea here is an important one. If computers are best at boring and repetitive work, they might do a better job of boring and repetitive code-writing than we will, too. If we can boil a function or a class down to a rote pattern, we can automate that pattern instead. Once we've done that, subsequent modifications to the class become much simpler; adding a field to our Person class is a matter of adding another field name at the top. As long as we thoroughly implement our pattern, and as long as our needs don't change to the point where the pattern no longer applies, this can be a great way to eliminate boring, repetitive, and error-prone acts of writing, testing, and maintaining this kind of code.

Decorating methods in classes

When we define a function in Python, the usual way to do so is by using a def statement, which allows us to give the function a name (and associate the function with that name within its scope), specify its parameters, and implement its body. This works largely identically, whether we do it in the Python shell, at the top level of a Python module, or locally within a function. However, when we write a def statement in a class, its meaning is slightly different.

>>> class Thing:
...     def value(self):
...         return 13
...
>>> t = Thing()
>>> Thing.value(t)
    13      # We can call methods the long-handed way, by specifying the class name,
            # the method name, and passing the self object as an argument.
>>> t.value()
    13      # We can also use the shorthand (as we usually do), by specifying the
            # object before the dot, then leaving it out of the argument list.

Aside from the ability to call them with a shorthand syntax, methods in a class aren't particularly different from other functions, so we would expect that it would be possible to decorate the methods of a class, just the same way we could decorate other functions. So, what happens if we try to apply our WithCallsCounted decorator to Thing's value method?

>>> class Thing:
...     @WithCallsCounted
...     def value(self):
...         return 13
...
>>> t = Thing()
>>> t.value()
    Traceback (most recent call last):
      ...
      File "C:/Examples/call_counting.py", line 37, in __call__
        return self._func(*args, **kwargs)
    TypeError: Thing.value() missing 1 required positional argument: 'self'

Unfortunately, our attempt to call our decorated value method failed spectacularly, with a rather perplexing error message emanating from within our decorator's __call__ method, at the point where it was trying to call the original method, which says it needs a self argument, even though we've provided one (by virtue of having called value on t).

The specifics of what went awry here revolve around the way that the methods of a class are different from other functions. There's a special mechanism that makes it possible for us to either write Thing.value(t) or t.value() to call our value method on t. The functions that are built when we write def statements have that mechanism built into them, but our decorated method — which isn't a function, but is actually an object of our WithCallsCounted class — doesn't have it. So, if we want to make this work, WithCallsCounted will need to provide that missing mechanism.

Using attribute descriptors to decorate methods

We've looked previously at the rules around attribute access in Python, which spell out what happens when we ask an object for one of its attributes. The following example summarizes those rules succinctly.

>>> class Something:
...     c = 0           # Classes can have attributes.
...
>>> s = Something()
>>> s.b = 13            # Objects can also have attributes.
>>> s.b
    13                  # We can access the b attribute we just assigned a value into.
>>> s.__dict__['b']
    13                  # The value of the b attribute is stored in the dictionary within s.
>>> s.c
    0                   # We can access the c attribute, since it was initialized in the class.
>>> 'c' in s.__dict__
    False               # But we won't find c in the object's dictionary, since it
                        # belongs to the class rather than each object separately.
>>> Something.__dict__['c']
    0                   # On the other hand, we will find c in the class' dictionary.

But, like so many of Python's inner workings, these rules, too, can be customized when the need arises. Values obtained from the attributes of an object always come back unchanged, but values obtained from the attributes of a class are treated slightly differently. Usually, this is a distinction without a difference; once in a while, though, this difference becomes important.

Methods, ultimately, are functions. We write them with def statements, we call them and pass arguments to them, the result of calling them is whatever value they return, and all of these things are true whether they're defined within classes or not. When we call methods, they need an extra parameter whose name is normally self, but what gives the methods of a class their special ability to be called in a way that doesn't require us to pass self, but to instead have another part of the expression become a self instead? It's because functions have an additional feature built into them: They customize what happens when their values are obtained from within a class. In other words, when called through the object (e.g., t.value() instead of Thing.value(t)), functions turn themselves into methods automatically. Since all functions do this, we don't have to do it ourselves; it's automatic and pervasive. But, rather than functions being magical, this is instead driven by a mechanism that we can hook into, as well.

An attribute descriptor — or, as they're most often called, a descriptor — is an object that can customize its own value when obtained from an attribute within a class. The presence of a particular dunder method is what makes an object into a descriptor.

• __get__(self, obj, objtype), which is given the object from which the attribute is being obtained (if any), as well as the object's class. Whatever __get__ returns becomes the value we get back when accessing the attribute.

In other words, when we obtain the value of a class attribute that is itself a descriptor, it transforms itself into the value we see, which might be slightly or wildly different from what's stored in the dictionary belonging to the class.

>>> class AlwaysBoo:
...     def __get__(self, obj, objtype):
...         return 'Boo!'
...
>>> class HasBoo:
...     boo = AlwaysBoo()          # It looks like boo is an AlwaysBoo object.
...
>>> x = HasBoo()
>>> x.boo
    'Boo!'                         # But when we ask for its value, we get a string.
>>> type(x.boo)
    <class 'str'>                  # And when we ask for its type, we see it's a string, too.
>>> type(HasBoo.__dict__['boo'])
    <class '__main__.AlwaysBoo'>   # But in the class' dictionary, it's an AlwaysBoo object, after all.

Now, that seems like a bizarre trick, but it's exactly how functions become bound methods, which are a bit like the partial functions we've seen before. A bound method is one in which the self parameter's value has already been determined, so it can be called by passing all of the remaining arguments to it.

>>> class Whatever:
...     def do_something(self):
...         return 13
...
>>> w = Whatever()
>>> w
    <__main__.Whatever object at 0x0000028862F46F80>
>>> Whatever.do_something
    <function Whatever.do_something at 0x000002886305D7E0>
              # When we ask the class for its do_something attribute, we get back
              # a function.  To call it, we'll need to pass all of the necessary
              # arguments, including self.
>>> w.do_something
    <bound method Whatever.do_something of <__main__.Whatever object at 0x0000028862F46F80>>
              # Notice that the address of the Whatever object listed here is
              # the same as the address of w.  That's because w is the object
              # to which the method has been bound.  w will be its self.

That last expression works the way it does because do_something is a function, and functions are descriptors whose __get__ method turns them into bound methods when they're accessed via objects.

>>> Whatever.do_something.__get__
    <method-wrapper '__get__' of function object at 0x000002886305D7E0>

And this, ultimately, is exactly what's missing from our WithCallsCounted decorator. With only a __call__ method, it can be called like a function, but has no way to transform itself into a bound method. But if we added a __get__ method to it, it could react in the same way that functions do when called like a method, by transforming itself into something that calls the bound-method version of the original function, instead of the original function without a self already bound. And since the underlying function already knows how to transform itself into a bound method — when its own __get__ method is called — all we need to do is ask it to do that transformation for us.

call_counting2.py (click here for a commented version)

class WithCallsCounted:
    def __init__(self, func):
        self._func = func
        self._count = 0

    def _call_original(self, func, *args, **kwargs):
        self._count += 1
        return func(*args, **kwargs)

    def __call__(self, *args, **kwargs):
        return self._call_original(self._func, *args, **kwargs)

    def __get__(self, obj, objtype):
        def execute(*args, **kwargs):
            original_func = self._func.__get__(obj, objtype)
            return self._call_original(original_func, *args, **kwargs)

        execute.count = self.count
        return execute

    def count(self):
        return self._count

With all of this in place, we expect that we can decorate either functions or methods with our WithCallsCounted decorator, and we can call methods either with or without self bound. Let's see if that's true.

>>> @WithCallsCounted
... def square(n):
...     return n * n
...
>>> class Thing:
...     @WithCallsCounted
...     def value(self):
...         return 13
...
>>> square(3)
    9            # We can call a decorated function.
>>> square.count()
    1            # Decorated functions have counts.
>>> t1 = Thing()
>>> t2 = Thing()
>>> t1.value()
    13           # We can call the value method with self bound already.
>>> t1.value.count()
    1
>>> Thing.value(t1)
    13           # We can call the value method with self unbound.
>>> t2.value.count()
    2            # Methods have counts, which are independent of which object they're called on.
>>> Thing.value.count()
    2            # Method counts can be obtained either through classes or their objects.

So, as we see here, while decorators offer us a lot of power, they require attention to a lot of detail in order to implement them correctly, especially when we want them to be applicable to the widest variety of targets. What made our WithCallsCounted decorator a bit complicated was our desire to be able to decorate both functions and methods, regardless of how they're called, and regardless of what arguments are passed to them. That turned out to be a tall order — requiring two kinds of argument packing and unpacking, as well as both __call__ and __get__ methods — but once it's done, it's done, and we can decorate as many functions or methods as we want at no additional cost.

Going forward in this course, as our skills continue to build and compound, we'll see techniques we could use to generalize this a bit further, so that if we needed ten different decorators with these characteristics, we'd only have to implement the difficult part once. All things in time.

---

Decorators in the Python standard library

One reason we want to learn about decorators is so that we can write them, but once we know what they're capable of, we can also use decorators written by others, while being able to more quickly understand what they can (and can't) do. Before we leave this topic, let's take a quick look at a few of the decorators you can find in Python's standard library.

Caching

When it comes to performance, one of the classic tradeoffs you'll see is the tradeoff between time and memory. It's hardly a universal truth, but it's quite often the case that you can save yourself time by using more memory, or use less memory by being willing to spend more time. The finesse is in how you use more memory — so that you're actually avoiding having to do more work — or how you use time — so that you spend it doing things that don't cost more memory than you're trying to save. Applied profitably, though, these kinds of tradeoffs are a great choice when you have a surplus of one resource (time or memory) and a shortage of the other.

The most common technique we employ to spend memory in a way that saves time is called caching, which simply means remembering data so we don't have to obtain it again. In practice, this can mean a lot of subtly different things, from remembering the result of an expensive calculation to avoid having to re-calculate it, to saving a copy of a downloaded file on a local storage device to avoid having to download it again. Caching is employed at every level of computing, from the internals of hardware like processors, memory, and storage devices, to low-level software like operating systems, to large-scale distributed systems that run on cloud infrastructure. Consequently, you'll find that caching emerges as a recurring theme in a computing curriculum.

To put this idea into some context, let's imagine how we might use it to good effect in a Python program. Suppose we have a function that potentially takes a long time to run, but that we'll need to use often. One way we could mitigate its cost is to augment it with an additional ability.

• Build an initially empty dictionary that maps the values of the function's arguments to the result of the function when given those arguments.
• After calling it, but before using its result, store the result in the dictionary, keyed by the values of the arguments.
• Before calling it, check the dictionary to see if the result for the given arguments is already known. If so, skip calling it altogether, returning the result from the dictionary instead.

When applied to functions in this way, caching is sometimes called memoization, a concept we discussed previously in the context of Recursion.

You may have noticed that this technique sounds quite similar to the kinds of things that decorators can automate, because it doesn't modify what a function does when it's called; it only modifies whether a function is called, based on things that happen before and after a call to a function, all of which are within the realm of the changes that decorators can apply. Indeed, you can write a decorator that provides automatic function memoization. But because this is such a common technique, Python's standard library provides a decorator that automates it for us.

>>> import functools
>>> @functools.cache
... def square(n):
...     print(f'calculating square of {n}')
...     return n * n
...
>>> square(3)
    calculating square of 3
    9
>>> square(3)
    9

Where the technique offers the greatest benefit is where it's applied to a function that's expensive to run, such as the fibonacci function we wrote when we were discussing Recursion.

>>> def fibonacci(n):
...     return n if n < 2 else fibonacci(n - 1) + fibonacci(n - 2)
...
>>> fibonacci(35)
    9227465       # You'll probably notice that this takes a while to run.  If not,
                  # try slightly higher arguments until you start noticing the run time.
                  # Don't go too far, though, or you'll be waiting a very long time indeed.
>>> @functools.cache
... def fast_fibonacci(n):
...     return n if n < 2 else fast_fibonacci(n - 1) + fast_fibonacci(n - 2)
...
>>> fast_fibonacci(300)
    222232244629420445529739893461909967206666939096499764990979600
                  # This should be (more or less) instantaneous.

Ultimately, all functools.cache does is automate the same memoization technique we employed by hand to speed up our fibonacci function previously. Being a general-purpose solution, it does the job somewhat differently, by using a dictionary to store the results instead of the list that we used. (We could use a list because we knew when fibonacci(n) is called, the possible arguments are integers from 0 through n, inclusive. A general-purpose caching tool can't know that about a function ahead of time.) This means that the time spent storing the results and looking them up will be a little slower than our solution, though it also means that we don't have to implement or test anything ourselves, which is obviously not without benefit.

We should be aware that this memoization trick is not universally beneficial, or even universally possible. First of all, it'll only work under two conditions.

• The arguments passed to a function must all be hashable, because it must be possible to use them as the keys in a dictionary.
• The function must be pure (i.e., its outputs are determined only by its inputs). If it causes side effects that we need, or if it relies on previous side effects, using a cache might give us an answer that's incorrect or prevent a side effect that we need.

Secondly, the technique will only be beneficial under two additional conditions.

• The amount of time spent looking the value up in a dictionary is meaningfully less than the amount of time spent performing the calculation.
• The amount of memory spent storing the values in the dictionary is not prohibitive. If we'll be using the same function with enough different sets of arguments over the course of a long-running program, the dictionary will start consuming enough memory to become problematic. The amount of memory on a modern computer is quite large, but certainly not infinite.

There is a way to mitigate the second problem — caches growing too large to be beneficial — by setting a boundary on how large the cache is allowed to be. Once the cache has reached that boundary, adding a value to it requires removing another first. There are different strategies for deciding which value to remove, but a common strategy is to remove the one that is least-recently used (i.e., Of all the values in the cache, what was the one that was looked up the farthest in the past?), under the theory that it's least likely to be needed again. That's what functools.lru_cache does — where lru stands for "least-recently used".

>>> @lru_cache
... def square(n):
...     return n * n
...

You can find out plenty of details about lru_cache in its Python standard library documentation — such as how you specify its maximum size — but we won't belabor them here.

Automating the implementation of total ordering

We saw previously that we can add six dunder methods (__eq__, __ne__, __lt__, __le__, __gt__, and __ge__) to a class, to give its objects the ability to be compared to each other both for equality and for ordering. There's a fair amount of repetitive work involved, though. __lt__ and __le__ aren't that different from each other, for example, but we have to implement them separately.

Given a class with __eq__ and __lt__ methods, though, we'd expect to be able to turn it into a class that implements all six of these methods by following a rote pattern.

• __ne__ could return the negation of what __eq__ returns.
• __ge__ could return the negation of what __lt__ returns.
• __le__ could use __eq__ and __lt__ and combine their results.
• __gt__ could return the negation of what our automatically-generated __le__ returns.

The functools.total_ordering decorator in Python's standard library automates exactly this kind of pattern. When used to decorate a class containing __eq__ and __lt__ methods, it implements the remaining methods according to the pattern similar to the one described above, so that we don't have to.

>>> @functools.total_ordering
... class Value:
...     def __init__(self, value):
...         self.value = value
...     def __eq__(self, other):
...         return self.value == other.value
...     def __lt__(self, other):
...         return self.value < other.value
...                # All we've implemented are __eq__ and __lt__.
>>> Value(3) > Value(2)
    True           # The others are implemented automatically.
>>> Value(3) > Value(4)
    False
>>> Value(3) <= Value(9)
    True
>>> Value(3) <= Value(3)
    True
>>> Value(3) <= Value(2)
    False

This isn't the last we'll see of decorators in this course, but the thing to appreciate at this stage is what they provide, in general: a leveling-up in our ability to solve kinds of problems instead of solving problems, so that we can implement and test complexity once and use it repeatedly, rather than having to implement it over and over again. We should always be in search of tools that let us do that, because even though not all complexities occur repeatedly and not everything can be automated away, when we can automate the less interesting problems, we can focus our attention instead on the important ones.