Note: This is the first post in what I hope will be a series leading to my talk at PyCon US, 2013.

Up until the late 1890s, atoms were thought to be the smallest possible division of matter. And then someone discovered electrons. It was easy to relate this new model of atom to the British dessert, plum pudding - negatively charged “plums” surrounded by positively charged “pudding”(though, in Thomsom’s model, electrons were not stationary, but that’s another story).

Now things seemed to be going fine, till someone else gave a new theory. And we all corrected ourselves: atom doesn’t resemble the plum pudding, instead it resembles a planetary model, with a cloud of electrons surrounding a nucleus of positive charge. Alright.

But no, looks like this change in metaphors wasn’t enough, someone set out to improve this model, prove and disprove things further…

Oh I could go on. But, my purpose here is not to revisit whatever little quantum theory I know, instead it’s to emphasize on the basic human tendency to compare new things to the concepts and objects we already know, before accepting it. We link the unfamiliar to something familiar, in order to extend understanding. Sometimes, this helps. But at other times (more often than not), this proves limiting - we tend to disregard anything that doesn’t fit our metaphors, that we find difficult to build a link with the known; refusing to use, apply, learn, or analyze the unfamiliar - good or bad.

This problem of building unfit analogies also shows up when we start learning how to write programs. As this article nicely addresses the issue: “It’s misleading; it obscures the relevant while confusing people with the irrelevant.”

Once we’ve learnt how to write a program, the metaphors we have used limit us from learning new things. We find it hard to accommodate new features that don’t fit into our metaphors, and end up avoiding them, or worse, forcefully bending them till they fit. This seems to be one of the many reasons people find it difficult to use Twisted - because it doesn’t fit into their “metaphor”.

As people with more experience in answering Twisted users’ questions put it (and I’m repeating the words in here, because I couldn’t have explained it any better myself): The most common problem people have with Twisted doesn’t really have anything to do with Twisted.
It is that they find themselves with part of a program, and coincidentally it uses Twisted. And they want to add some new behavior to some part of the program.

Say, for instance, the program is an IRC bot and the behavior one wants to add is sending a message to an IRC server. To send a message using Twisted’s IRC client API, you need a connected IRCClient instance. And very often, we come across people who face trouble figuring out how to get and keep a reference to such an object.

The problem here, is that most people understand the program as a big box. The main function is a box, the ClientFactory is a box, etc. They are used to putting boxes inside of boxes. For example, they put a factory box into the main function box by doing f = ClientFactory(...). They know how boxes work. If you put a small box inside a large box, then later on you can reach into the large box and get the small box out again. That is, after you define f like that, you can get the factory again by using the f variable. But then, they see reactor.connectTCP(..., f) and it doesn’t fit into their mental model. It’s not obviously creating an IRCClient, and it’s certainly not putting it into any of the boxes they have at hand. They’ve usually already customized IRCClient with a subclass. They have a loggedIn method or a connectionMade method on it overriding some behavior - and by overriding those methods, they’ve already done what they think they don’t understand how to do. They’ve gotten a reference to the connected instance (since it is passed as the “self” argument, as with any method in a Python program).

But they didn’t get it in a way that’s easily compatible with the box metaphor, so they didn’t realize it.

The easy answer to their question is to tell them to use connectionMade to put the instance into a box on the factory (i.e., self.factory.connections.append(self)). That’s mostly just a trick, though. It gives them a new tool for pushing boxes around. They may be able to make good use of the tool, but a better answer is to help them lose the box metaphor and understand the program at a more direct level.

A better solution, probably, is to understand that a program is not a box, after all. At a slightly deeper level, it’s references and objects, which implement namespaces (modules, classes, instances), scopes (locals, globals). These are still metaphors. The program is electrical potentials on semiconductors. But stripping away all the abstractions isn’t very useful.

The box metaphor is a problem because it’s too limiting. It excludes possibilities. If you understand programming using a metaphor, and the metaphor cannot explain certain things that programs can do, then you cannot write or understand programs that do those things.

Not until you come to understand programming without using that metaphor. References, objects, namespaces, scopes, etc. - those are better metaphors to understand Python programs with - because they’re the same metaphors Python itself is developed with - they can explain 99.99% of things a Python program can do.

I know that bluntly telling you your box-metaphor is wrong is almost as easy for me as it is hard for you to discard it. To be fair, I also made an attempt to try and find out where this box-model comes from.
We figured that it could, perhaps, be blamed on the language we use to talk about programs. Even the programmers who are not limited to thinking of programs as boxes, say “how do I send an IRC message inside this function?” And if you hear just enough people talk about “inside” enough, eventually maybe you can’t help but understand things in terms of containers.

There’s a subtle distinction in that use of “inside” that’s probably rarely understood though. The way we organize programs, as bytes in files.
But wait, the “in” is a metaphor as well. Files don’t literally contain bytes. It’s abstraction.
And some of those bytes represent functions and classes, and the functions are “in” the classes - but not really. More abstraction.
It’s pretty useful to talk about the contents of files though. And it’s not that misleading to talk about functions in classes, or even the code in functions. If you have a function that is a GUI button press event handler and you want to send an IRC message when the button is pressed, then you are probably going to put the code for sending that message “in” the event handler function.

That’s a different kind of “in” than the box-programmers think about though. The lexical structure of your program - bytes in files, functions in classes, lines of code in functions - can certainly be related to how objects, references, namespaces, scopes, etc. are arranged. Ultimately it defines those relationships, after all. But the two structures are, by no means, the same. So, and this is all quite speculative, perhaps the box-programmer’s understanding of the one (which usually develops first, since early programming experience is mostly about learning to get syntax right) bleeds over into the other (which follows later, and is perhaps sometimes understood in terms of the former).

So next up, the question is why doesn’t a program that uses Twisted fit into this box model well? Why is it that box programmers can fit everything else in this little metaphor - everything but Twisted?

Probably because, Twisted is the first library they’ve ever tried to use that actually does a significant amount of work for them. They’re used to writing the whole thing, or very close to the whole thing, so they could write it in a way that is compatible with their box model. As I already mentioned, they write programs that their model can completely describe, and ignore program structures that it cannot (because they don’t understand them).

There’s also this another possibility that people feel Twisted takes control away from them and forces them to structure their program differently. It turns their program inside out, which necessarily hides some of the relationships from them. They’re used to being on the outside of the biggest box and being able to look in. Suddenly the code they’re writing is now inside several projections of a single hyperbox and they can’t see the hyperbox that’s outside of them.

I had once said that Twisted is a car, you have to know how to drive it to reach your destination. Sure, it’s awesome because you can make it work like a spaceship as well, but it works as a darn nice car too. You just need to know where the brakes and accelerators are.

At the time, the metaphor seemed really profound to me. But then, I discovered that Twisted can act as a submarine as well…

(Thanks to exarkun, who always has the answers.)