A couple of assumptions I make about you – that you are familiar with basic socket programming, and know a little bit about I/O multiplexing (select etc.)
In this first post, we’d see what a reactor is, and how it works. In the second part, we’ll talk about how you should and should not use it. The third part will talk about what’s wrong with the way Twisted’s reactor works now, and how it should be instead.
I wish I was only doing an imitation of a dumb user rather than really being one.
My first interaction with the reactor was when I was writing tests for an endpoint. I’d only heard about The Reactor – no one ever talked about how it worked, but everyone knew that it worked. And that it worked pretty well. To understand what a reactor was really doing, and what all the fuss was about, I looked for it hoping to unveil some magic. Alas, not much was found there; it again led to me looking through other some places, which, again, had little besides docstrings in a skeleton.
Since I did not find much in the twisted.internet.reactor module, I looked at the next best alternative, the MemoryReactor, which is a test double that one could use to test code that uses a reactor (e.g. TCP connection attempts). If you’ve been paying attention so far, you’d know that my goal was to test an endpoint. And a fake reactor was sufficiently serving my purpose. Also, MemoryReactor had enough code to tell me what it does when an endpoint is used to establish a connection. Case closed. Until one day, when I had to write tests for another endpoint, which required me to write my own fake reactor. One thing led to another and I ended up understanding the Twisted reactor architecture. And here’s an attempt to help you understand it too.
It’s frightening to remember that twisted is an overgrown MUD
Let us begin with a look at Twisted itself. So, what does Twisted do? It helps you write servers and clients (among other things). If you’ve worked with Network Protocols before, you’d know that, e.g. to work with TCP sockets, we create a socket, bind it, make it listen and accept or connect, and uh, basically write a lot of code with a lot of function calls having a lot of arguments. So, with Twisted, we have a framework that’ll help us work with those sockets without having to worry about the low-level platform specific details. This means that it is a middle layer between the very low-level infrastructure (i.e. OS-level I/O), and the very high-level infrastructure (e.g. web servers), that makes it easier for us to write our servers and clients. This means that we can now concentrate on developing our application, rather than reinventing the wheel – it also reduces the potential mistakes we could make by a large amount.
Note that we could instead use, for example, a low-level framework and language (C or C++), which will give us access to all the capabilities and APIs provided by the operating system (and can be used to make a very fast program as well!). But pervasively using platform-specific functionality results in a platform-specific program, so portability suffers. Also, we’ll have to be very careful about buffer overflows and system crashing bugs that C and C++ bring with them. This may lead to either a longer development process with lots of rigorous testing, or a more fragile system. (There’s also the “lowest common denominator” approach – i.e. provide functionality which only provides access to the lowest common denominator between all supported platforms. More on this here.)
evol: But what kinda design is the goal here
itamar: not ugly?
moshez: a good one.
dash: “non sucky”
Now that we’re done playing with basic socket programming, let us go to I/O multiplexing – select() system calls, and the likes.
An object that implements an event-loop in Twisted is called a reactor. Event loops implement a pluggable interface to OS-level functionality such as networking, timers, and certain other commonly available utilities such as SSL encryption. There are many mechanisms in the area of high-performance multiplexing to increase performance. select is one of them. A few others are /dev/poll, epoll, poll, kqueue, completion ports, and POSIX AIO. Twisted provides reactors that run on top of these mechanisms - the reactor implementation may be chosen at run-time, depending on which ones are available and what functionality is required. In Twisted, the reactor is basically a Singleton. There is only one reactor object and it is created implicitly when you import it. As mentioned earlier, there is very little code in the reactor module in twisted.internet. The actual implementation resides in other files, such as twisted.internet.selectreactor, twisted.internet.epollreactor, twisted.internet.pollreactor etc. Though that does mean that Twisted supports reactor-specific functionality, but this support does not mean that all applications written with Twisted are not portable. You can choose to use platform-specific functionality (e.g. use the file descriptor support on Unix to write a curses-based console interface), or to use only those interfaces that are cross-platform and supported in all reactors (e.g. use TCP support to write a telnet-based console interface). The choice is made by you, not Twisted. If you import twisted.internet.reactor without first installing a specific reactor implementation, then Twisted will install the default reactor for you. The particular one you get will depend on the operating system and Twisted version you are using. For that reason, it is general practice not to import the reactor at the top level of modules to avoid accidentally installing the default reactor. Instead, import the reactor in the same scope in which you use it.
To reiterate, first we saw that Twisted is a middle-layer between the low-level system calls and the high-lever applications, and it is better to not interact directly with low-level infrastructure when there are higher-level tools to do that for you. Next, we learnt that a reactor is nothing but an object that implements an event-loop in Twisted, and runs on top of mechanisms for high-performance multiplexing, like select, poll, epoll, etc. And finally, we saw how the reactor is a Singleton (we’ll talk a bit more about this in the upcoming posts), and how various different reactor implementations exist (and co-exist).
In the next part, we’ll see how you can use the reactor in your application, a bit more about the event-loop and the asynchronous architecture, and a lot more about what you shouldn’t do with the reactor when you use it.
]]>Now there are two ways in which I can tell you how it was. I can use some really big words and give you a dry report of events. But that’s boring, and there’s enough boring stuff out there.
“Speak English!” said the Eaglet. “I don’t know the meaning of half those long words, and I don’t believe you do either!”
So here’s a recollection of whatever I can remember, without using many big words.
It started on March 12, at least the interesting part did.
I was welcomed into SJC with a PyLadies banner and a smiling Lynn Root. I also met Hynek, and we drove to the hotel and I guess that’s about it for the first day.
The next day, i.e. March 13, I went to the Convention Centre, met Yarko, who printed my badge. I met Steve Holden - no wait, didn’t really meet, I jumped into his way and talked to him - I wonder how weird it looked. I did some volunteering - moved boxes, helped out at the registration desk - was fun. I then trotted around the venue with Stuart, this time pretending to do some volunteering. Met Doug - he’s nice, and gave me more work. I saw Guido at the Language Summit, and he was getting coffee alone, but looked somewhat tired, so I decided to leave him alone. I wasn’t equally kind to Glyph though, and chatted a lot, everything from the history of Twisted to, well… DuckDuckGo.
I also wasn’t that kind to Asheesh, and badgered him loads. He called me weird because Twisted didn’t scare me. Jessica, who’s super-humanly awesome, by the way, if you didn’t already know, gathered lots of folks from Boston and OpenHatch, and we all hung out and ate food. Good food, actually, in case you were wondering. I also got to know the secret to how paulproteus is omnipresent on IRC (he’s everywhere you look!).
Later that night, I met nedbat and a few CPython core devs who told me stories about old Twisted days and some insider-info on Glyph, his name and identity and stuff. Not going to write about that though - if you want to know, get your own tickets to PyCon. We also discussed some cool MIT Media labs stuff, and I met Steve again, and he seemed to remember my name (I guess that means I must have weirded him out significantly earlier).
Moving on to the 14th of March. The day started with me meeting Itamar, who had a tutorial later that day. Then had lunch with some Twisted people, and then there was that tutorial. I met Jean-Paul for the first time, and learnt that I had caused him so much anguish during the summer that he decided to get a new job this summer. I apologised profusely, but I think the damage is done. Oh, and I threw my laptop on the floor, because you know, no one wants to not look like a fool in a room full of people.
Then I went back to my room, after revealing a nervous-wrecked self to Glyph and Ying. I woke up next day and went to attend the Keynote. Now, I was quite sleepy and tired and lost and fretting a bit over my talk, so I wasn’t expecting to find a 30 minute lecture too interesting. To my immense surprise, I actually enjoyed it. I met Michelle and a few other PyLadies, and we all got a Raspberry Pi each, and then I vanished until my talk.
I saw that there were actual people present in my talk, which was quite surprising. Oh wait, there were some really helpful people in the Green Room, a special mention must go to David, who was great at calming nerves - he asked me to take two bottles of water to the talk - and I did. Anyway, it seemed to have gone by okay - I did throw the mic on the floor in the middle of my talk (I guess I throw things around a lot) and then there were some issues with the computer, but anyway, I kept talking, it’s hard to shut me up. People were nice, they laughed at my jokes and Glyph let me call him a liar. After the talk, I saw that the Twisted core team was holding some signs up, which spelt “Go Ashfall/Ashwini”, and that’s nice enough to make you cry. I didn’t cry though, because everything was being recorded.
Later that day, I ate some more good food with the Twisted people, and heard more stories.
The next day, the 16th of March, I got to the Keynote. And then there was Glyph’s talk. He mentioned my name and referred to some stuff I’d said in my talk (and did not, unlike me, call me a liar - I guess he’s nice), and everyone is saying that this makes me a celebrity. I don’t think it works that way, but if you believe so and want an autograph or something, feel free to drop me a mail.
Then there was the PyLadies lunch, where I met more awesome people, and had some more good food. I attended Lynn’s talk next, and then went to dinner with Guido. Yes, the Guido who wrote Python. My friends insist that I must describe this part well, so here it is: we drove in his car to a steakhouse, and I sat to his left. Ate food. Drank lemonade. Heard some interesting stories. Yes, PyCon exposes you to loads of stories, it is cool that way. Then Guido dropped me back to the Convention Center where everyone got together and talked about goats, which was very informative.
Oh, also, Disney gave away Wreck-it Ralph action figures to all the speakers (and hence, me), and Paul Hildebrandt is one of the nicest and kindest people I have talked to. Just saying. Also, Armin Rigo is awesome - he’s perhaps the only incredibly intelligent person involved with programming I’ve met who seemed genuinely happy about life (and not grumpy at all).
The next day, which was also my last day at PyCon, I attended Guido’s Keynote, and then everyone agreed that I wasn’t lying when I said that I had become cynical about software. We had the annual Twisted dinner and I had an interesting conversation with Zooko, who accepted (and even adopted) my mispronunciation of IRC nicks without hesitation or revolt. I sat with Bradley and learnt a lot, with interesting insights from Duncan - about Usenet, about embarrassing email conversations, and about proprietary software pains, I should probably write more about that sometime.
And then there was the Twisted sprint, which I couldn’t attend for long, but everyone signed the Twisted book for me (also, Jessica mentioned me in the credits of that book - again, people tell me it makes me a celebrity of sorts).
Well, that’s that. I met most of the Twisted team, and they were all incredibly nice. But before I finish, I would like to thank the Python Software Foundation, PyLadies, and everyone involved with PyCon, for making it possible for me to attend, and I hope I see you all again next year!
So then, how was PyCon 2013? Welcoming? Engaging? Awesome? I guess you should decide for yourself.
]]>Up until the late 1890s, atoms were thought to be the smallest possible division of matter. And then someone discovered electrons. It was easy to relate this new model of atom to the British dessert, plum pudding - negatively charged “plums” surrounded by positively charged “pudding”(though, in Thomsom’s model, electrons were not stationary, but that’s another story).
Now things seemed to be going fine, till someone else gave a new theory. And we all corrected ourselves: atom doesn’t resemble the plum pudding, instead it resembles a planetary model, with a cloud of electrons surrounding a nucleus of positive charge. Alright.
But no, looks like this change in metaphors wasn’t enough, someone set out to improve this model, prove and disprove things further…
Oh I could go on. But, my purpose here is not to revisit whatever little quantum theory I know, instead it’s to emphasize on the basic human tendency to compare new things to the concepts and objects we already know, before accepting it. We link the unfamiliar to something familiar, in order to extend understanding. Sometimes, this helps. But at other times (more often than not), this proves limiting - we tend to disregard anything that doesn’t fit our metaphors, that we find difficult to build a link with the known; refusing to use, apply, learn, or analyze the unfamiliar - good or bad.
This problem of building unfit analogies also shows up when we start learning how to write programs. As this article nicely addresses the issue: “It’s misleading; it obscures the relevant while confusing people with the irrelevant.”
Once we’ve learnt how to write a program, the metaphors we have used limit us from learning new things. We find it hard to accommodate new features that don’t fit into our metaphors, and end up avoiding them, or worse, forcefully bending them till they fit. This seems to be one of the many reasons people find it difficult to use Twisted - because it doesn’t fit into their “metaphor”.
As people with more experience in answering Twisted users’ questions put it (and I’m repeating the words in here, because I couldn’t have explained it any better myself): The most common problem people have with Twisted doesn’t really have anything to do with Twisted.
It is that they find themselves with part of a program, and coincidentally it uses Twisted. And they want to add some new behavior to some part of the program.
Say, for instance, the program is an IRC bot and the behavior one wants to add is sending a message to an IRC server. To send a message using Twisted’s IRC client API, you need a connected IRCClient instance. And very often, we come across people who face trouble figuring out how to get and keep a reference to such an object.
The problem here, is that most people understand the program as a big box. The main function is a box, the ClientFactory is a box, etc. They are used to putting boxes inside of boxes. For example, they put a factory box into the main function box by doing f = ClientFactory(...).
They know how boxes work. If you put a small box inside a large box, then later on you can reach into the large box and get the small box out again.
That is, after you define f like that, you can get the factory again by using the f variable. But then, they see reactor.connectTCP(..., f) and it doesn’t fit into their mental model. It’s not obviously creating an IRCClient, and it’s certainly not putting it into any of the boxes they have at hand. They’ve usually already customized IRCClient with a subclass. They have a loggedIn method or a connectionMade method on it overriding some behavior - and by overriding those methods, they’ve already done what they think they don’t understand how to do. They’ve gotten a reference to the connected instance (since it is passed as the “self” argument, as with any method in a Python program).
But they didn’t get it in a way that’s easily compatible with the box metaphor, so they didn’t realize it.
The easy answer to their question is to tell them to use connectionMade to put the instance into a box on the factory (i.e., self.factory.connections.append(self)). That’s mostly just a trick, though. It gives them a new tool for pushing boxes around. They may be able to make good use of the tool, but a better answer is to help them lose the box metaphor and understand the program at a more direct level.
A better solution, probably, is to understand that a program is not a box, after all. At a slightly deeper level, it’s references and objects, which implement namespaces (modules, classes, instances), scopes (locals, globals). These are still metaphors. The program is electrical potentials on semiconductors. But stripping away all the abstractions isn’t very useful.
The box metaphor is a problem because it’s too limiting. It excludes possibilities. If you understand programming using a metaphor, and the metaphor cannot explain certain things that programs can do, then you cannot write or understand programs that do those things.
Not until you come to understand programming without using that metaphor. References, objects, namespaces, scopes, etc. - those are better metaphors to understand Python programs with - because they’re the same metaphors Python itself is developed with - they can explain 99.99% of things a Python program can do.
I know that bluntly telling you your box-metaphor is wrong is almost as easy for me as it is hard for you to discard it. To be fair, I also made an attempt to try and find out where this box-model comes from.
We figured that it could, perhaps, be blamed on the language we use to talk about programs. Even the programmers who are not limited to thinking of programs as boxes, say “how do I send an IRC message inside this function?” And if you hear just enough people talk about “inside” enough, eventually maybe you can’t help but understand things in terms of containers.
There’s a subtle distinction in that use of “inside” that’s probably rarely understood though. The way we organize programs, as bytes in files.
But wait, the “in” is a metaphor as well. Files don’t literally contain bytes. It’s abstraction.
And some of those bytes represent functions and classes, and the functions are “in” the classes - but not really. More abstraction.
It’s pretty useful to talk about the contents of files though. And it’s not that misleading to talk about functions in classes, or even the code in functions. If you have a function that is a GUI button press event handler and you want to send an IRC message when the button is pressed, then you are probably going to put the code for sending that message “in” the event handler function.
That’s a different kind of “in” than the box-programmers think about though. The lexical structure of your program - bytes in files, functions in classes, lines of code in functions - can certainly be related to how objects, references, namespaces, scopes, etc. are arranged. Ultimately it defines those relationships, after all. But the two structures are, by no means, the same. So, and this is all quite speculative, perhaps the box-programmer’s understanding of the one (which usually develops first, since early programming experience is mostly about learning to get syntax right) bleeds over into the other (which follows later, and is perhaps sometimes understood in terms of the former).
So next up, the question is why doesn’t a program that uses Twisted fit into this box model well? Why is it that box programmers can fit everything else in this little metaphor - everything but Twisted?
Probably because, Twisted is the first library they’ve ever tried to use that actually does a significant amount of work for them. They’re used to writing the whole thing, or very close to the whole thing, so they could write it in a way that is compatible with their box model. As I already mentioned, they write programs that their model can completely describe, and ignore program structures that it cannot (because they don’t understand them).
There’s also this another possibility that people feel Twisted takes control away from them and forces them to structure their program differently. It turns their program inside out, which necessarily hides some of the relationships from them. They’re used to being on the outside of the biggest box and being able to look in. Suddenly the code they’re writing is now inside several projections of a single hyperbox and they can’t see the hyperbox that’s outside of them.
I had once said that Twisted is a car, you have to know how to drive it to reach your destination. Sure, it’s awesome because you can make it work like a spaceship as well, but it works as a darn nice car too. You just need to know where the brakes and accelerators are.
At the time, the metaphor seemed really profound to me. But then, I discovered that Twisted can act as a submarine as well…
(Thanks to exarkun, who always has the answers.)
]]>Before we get to PyPy, we need to understand what the implementation of a language means.
A program that interprets the programs you write is called an implementation (of that programming language). For example, gcc and icc are different implementations of C (and clang), while CPython and PyPy are implementations of Python.
Thus, PyPy is a Python program that is compiled to C ( or other things) which is then compiled to machine code which then takes your Python programs and compiles them directly to machine code (or not!) without the “C” step in between.
So, when you say python myScript.py, you are using CPython.
You could use PyPy instead, and run pypy myScript.py. In most cases, your script would work just the same. (And if it doesn’t, it’s probably a bug!)
Ultimately the goal of PyPy is to do the exact same work as CPython, but a lot faster.
PyPy comes with a JIT compiler, so that when PyPy is translated into an executable such as pypy-c, the executable contains a full virtual machine that can optionally include a Just-In-Time compiler.
What that basically means is, when you compile a C program, the machine code gets put into a file so you can run it; whereas, when you use PyPy’s JIT to “compile” your program, the machine code goes into memory and is run right away, because it only compiles little bits of your program at a time. So it never saves a copy (saving a copy would not be useful due to a million technical details, I am told).
PyPy has a whole ton of different modes of operation. JITing is one of them. Usually it’s the one with the most performance impact, but your code is not necessarily JITed immediately. PyPy has an interpreter as well, and it decides which to use depending on a number of factors.
Now, if you dig into how PyPy came into existence, you will discover that PyPy is, in fact, a followup to the Psyco project. Wikipedia says, “PyPy’s aim is to have a just-in-time specializing compiler with scope, which was not available for Psyco.”
So, I inevitably wondered what happened to Psyco? Why PyPy?
Why couldn’t Psyco be improved in itself?
The answer being: The improvement is called PyPy!
But let’s dig into this a bit more, there is an important software design concept here that one must understand.
In this particular case, it so happens that starting a new project called for less effort than improving the slightly flawed one. Actually, “slightly flawed” is not the best way to put it. Psyco was basically a hook for improving the performance of integer math. In order to do more than that, it needed vastly more information. Speeding up object operations proved to be impossible.
Now let’s step back and look at the bigger picture. At what point does one decide give up improving an existing project and decide to make something new?
Turns out, never. It’s (almost) always the wrong decision.
Hmm so, why PyPy and not Psyco?
In the case of Psyco/PyPy it is kind of a different thing. It’s hard to explain exactly why one doesn’t become the other, but it’s like if you wrote a command-line program to make an image greyscale, and then later, you wanted to make a competitor to Photoshop. Both things manipulate images, but the photoshop clone is so much bigger, does so much more stuff, that it doesn’t necessarily help to build the whole thing around the infrastructure you built for a little command-line tool that just decreases saturation. So it’s not like you are throwing out one perfectly good greyscaler so you can build a new one, you’re just starting a new project with a different purpose. (Which may, eventually, contain code to make an image greyscale, but that’s almost beside the point).
So, we are saying Psyco didn’t have the same goals as PyPy, PyPy is more ambitious and has broader goals.
Now, once again, what exactly is PyPy?
It’s just another Python implementation, that’s pretty bleeding edge when it comes to compiler design.
The thing about GSoC (and other such programs) is, well… it’s a conspiracy! It hardly is just a summer of code. It ends up making you a part of the bigger team; it attempts to make you a part of the community.
I can say I finally understood the affable tone that most hackers take when speaking of the software they build. It’s hard to describe what you get out of it - the satisfaction of making a difference? The thrill of learning something new everyday? The challenge of trying your hand at something you never did? The wait for someone to come prove you wrong? Or maybe something as trivial as the joy of discovering the living replicas of some fictional characters you respect.
Twisted is a great to contribute to. There are many places where the awesomeness of Twisted as a software can be found, but I’d like to talk a bit about what I see Twisted as - a collection of many well-written lines of code. No, really, it would be unfair for me to highlight the advantages that Twisted offers as a networking engine, because I haven’t used it that way (yet). What I do see Twisted as, is a nice codebase, with a lot of smart people behind it who are continuously working to make it even nicer. And it feels good to be associated with something this cool. Everything, right from the review process to the code development policy, ensures that both, you and Twisted grow together as you contribute. As you try and fix the flaws in it, Twisted gives back by exposing your flaws and giving you a chance to improve.
And honestly, I had no clue about Twisted and its splendidness or reputation in the Python community when I started contributing. I probably would’ve been slightly more overwhelmed had I been aware of how people are awed by it’s size and features. Instead, I dug in because there were simple guidelines and instructions that showed me how I could begin contributing code, and some friendly people willing to help me get started.
I’m still quite surprised at times that I have commit access to something this big - I am still learning the basics of Computer Science, and it’s pretty exciting to be able of think of Deferreds and callbacks and reactors and endpoints in the midst of studying those sorting algorithms and tree-traversals that the routine college classes require of me. And to think I got to release a whole new version of Twisted out to the world… well it feels kind of nice!
A big and huge thanks to the Twisted community for being awesomely supportive. Seriously.
Oh, and a +1 to exarkun for president, for being an awesome mentor.
]]>“Twisted python…. it’s featurrific!”
Another endpoint!
Twisted now supports IPv6 TCP client endpoints.
And guess what, it accepts hostnames too!
Other than enabling you to create a TCP client that connects to IPv6 hosts, this endpoint is special because it does name-resolution: When you pass a hostname to the endpoint, it uses socket.getaddrinfo to resolve it into IPv6 addresses and uses the first address on the list to connect.
This was the endpoint where I came to appreciate the importance and necessity of using fake components for testing, instead of testing our code with the real stuff. Thanks to a discussion with glyph, tomprince, and radix.
It started with Glyph telling me that I shouldn’t be talking to the real system resolver; it is insufficiently reliable for a unit test. So… the real thing is not reliable enough? How is a unit test different from any other code which will be using it?
And then came the answers:
The difference with a unit test is that test coverage must cover all the conditions, even those which do not currently exist. The real code only cares about the current state of the universe, but since we expect real code to run in many different universes at many different times, we must have tests that cover all of it. For example, your computer may exist in one of two states - ethernet cable plugged in, or not plugged in. Name resolution will behave differently in these cases. So we have to have a unit test that behaves as if your ethernet cable were plugged in, at least for the purpose of the code under test, and one that behaves as if it isn’t. And of course, this extends all the way up through DNS configuration and routing tables and whether the server you’re trying to connect to actually exists or not, which produce similar (but distinct) error conditions.
So yes, the real code can’t reliably fail on demand. The whole point of the real code is telling you the real, current state of the ethernet cable, after all.
Also, a lot of this just has to do with the speed of the tests. If you have 1000 tests which do DNS resolutions, it’s a lot faster to call a function that just does return “1.1.1.1” than to call a function that has to do a whole ton of real work on your network hardware in order to just get that same answer.
Thus, fake code will create a virtual universe within the real one to test how the code will behave there. Basically the test is fooling the feature’s code with certain environments, to test how it would behave in them.
This also enables someone who doesn’t have access to one of those environments (for example, someone on a network that just firewalls out IPv6 traffic) can still maintain the code and submit fixes.
Except ‘virtual universe’ is probably thinking way too big. Unit testing is all about individual components talking to each other. If you have a widget that connects to a gizmo, and you need to test that the widget works, you just have a testing gizmo that behaves differently from a real one. It’s not really a “fake” gizmo just because it only does one thing instead of all of the things that some other gizmo might normally do, and you certainly don’t need to simulate the entire universe that produced your testing gizmo.
But then, there’s also a tiny glitch: This way, we get to test if name resolution works with some (fake) function that I make, and pass, but how can I reliably check if it will work with socket.getaddrinfo too?
To which, I got this:
For our purposes right now, the way you know that it works the same as getaddrinfo is that you read the docs, you called getaddrinfo in a python shell and you compared the output carefully. And since it’s unlikely that the behavior of getaddrinfo will change (in any way that we care about), that kind of manual verification is fine. We don’t need to test its behavior, we need to test our behavior. There is something to be said for testing that we are compatible with it, but in a case such as this, where it’s such a relatively stable API, it’s not that critical.
But the better technique is to have a library of verified fakes where a specially-configured machine is used to make sure that the real one and the fake one have the same behavior in one test, so other tests can confidently use the fake one in N tests.
I thought I got comfortable with the idea of Deferred callbacks and using them effectively, but, this sums it up best I think:
radix: well, running it works well :>
glyph: yeah, but don’t ask what it does because it’ll KILL YOU WITH ITS TEETH!
(I almost got killed, by the way, a few weeks later, while making another endpoint, by them Deferreds.)
Oh, and I also got to know about using lambda forms.
(Note: In case you are interested, Twisted’s got a smarter name-based TCP connection endpoint under development.)
]]>I’ve learnt loads. Each endpoint brought with it new speedbumps. TCP IPv6 (server and client) endpoints led me to get to know IPv6 and address resolution. Standard I/O endpoint made me learn how to deal with Deferreds and Deferred callbacks. With every new endpoint, I faced new errors and exceptions, and learned how to tackle them. Right from ValueError’s to DirtyReactorWarning’s, I think I’ve faced a nice lot of issues.
Serial port endpoint taught me how ImportError’s can mess up your life, while child processes showed me how it feels to be back to the square one, just when you thought you were done (Damn you, Law of Demeter!).
Well, I still stutter with test driven development, and keep unintentionally evading writing tests first and the main feature later, thus piling up more work for myself and jumping back and forth between improving the tests and feature. Though that also led me to some very helpful (and cool!) tools like python-coverage, I’m trying to get better at TDD.
Some internet outages aside, it’s been fun all through. So loads of thanks to everyone who’s been lending a helping hand every now and then, and the incredible Twisted community for bearing with the excessive noise I make in the channel!
Well, don’t be surprised to see more endpoints, and (lots) more noise in #twisted in the near future!
Cheers!
]]>If we knew what we were doing, we would not call it programming.
Well, before I start rambling, here’s some more news: Twisted now has an endpoint interface that listens on stdin/stdout. We merged it today, and Twisted just became slightly more awesome.
And now, with a warning that what follows is my endless rant about how I wrote this, you may read further.
So, the story starts after I finished the TCP IPv6 server endpoint, which was more or less a small modification of the pre-existing TCP IPv4 server endpoint. Yes, what I’m saying is that it involved very little effort from my end. Some small changes here and there, and we had a new endpoint.
Then came this Standard I/O endpoint, which I had to write (somewhat) from scratch.
First, I tried to understand how code involving stdio is written in Twisted, for which these examples were about all I had. And then I tried to figure out what stdio actually meant, and what it does, and why we need an endpoint for it. This in turn made me question life and our existence in this world as a whole. No, I’m kidding, basically I had no idea what I was doing. In spite of many discussions and my extensive pestering at #twisted, I was still not convinced I knew what I needed to do.
Now, I had a problem. And to solve it, I went back to basics. Right down to OSI layers, what each layer did, how TCP’s 3-way handshake works, how UDP, unlike TCP, is non-reliable, etc. Every related or unrelated thing. And then, once I had a little confidence about understanding at least how networking is supposed to work, I went back to reading the how-to on endpoints. It was there, that I realized, that I was thinking too much.
Yes, I was worried that I won’t be able to write the said code because I didn’t know exactly what went on inside the system with it. But then, after my crazy learning spree, it dawned on me that none of it was actually a pre-requisite to writing an endpoint. You see, an endpoint is just an implementation of an interface. This ‘interface’ bit is important because it makes my work easier by telling me exactly what methods I need to implement. And then, twisted.internet.stdio is just a module. So now, instead of following the theory, I followed the code. A module that imports a class, that implements some more classes, etc. And since I (thankfully) can read code, it made enough sense to enable me to know what do next and use the existing code to write some new stuff.
So that was one part - writing the endpoint. It took a few days, 5 actually, of staring at scripts and modules to finally come up with 6 lines of real code that was the endpoint (of which one line, as it happens, was unnecessary and had to be removed). But nevertheless, it was done. And then, came the second part - writing the unit tests. Now before I proceed, I must mention that this is not the ideal order of writing code. One must follow, what is called Test Driven Development to write good code. As you probably know by now, I haven’t been doing that in this case. Anyway, continuing.
So this was new. Very new. I knew what each part of my code was supposed to do, and because I wrote it, I also believed that it would do what it was supposed to do. Writing tests, was like questioning my own piece of code, and considering a possibility that it might not work as expected, and checking if it actually does. And that just didn’t make any sense. I am not that experienced a programmer yet, to get to live with the privilege of a big ego, but, hey, I liked what I wrote, and it made sense to expect it to work, I mean everyone does, don’t they? (That’s what keeps you sane, hoping that codes will work, bugs will go away, etc., right?) But sadly, a refusal to consider the alternate possibility, is what makes writing tests harder. “Will this part of the code work?” ”Obviously it will.” “Then why should I test it?”
I eventually got over this by accepting the fact that the code you write is not constant. Even if you believe that what you are writing now is perfect (and trust me, it isn’t. If you ask the right people, they’ll probably hand you a huge list of drawbacks, with nothing but a mere glance at your programs), you have to understand that it will be changed, maybe not today, not tomorrow, but a few months/years later, someone would want to modify it, add to it, or even remove it. And to stop them from breaking the feature you are implementing, you need to write tests. Once you see it from this perspective, it becomes slightly less frightening to write tests. I still took a few days to come up with some (very) simple lines of code, but eventually (and with a lot of help), I did.
Though even now I take a while to write good tests, and more often than not depend on loads of guidance, I’m (hopefully) improving. Maybe in some time, I will actually be able to follow the real Test Driven Development process and write code in the right order.
]]>That was years ago, and it didn’t even apply to me back then (fortunately, I was fairly decent with my spellings, etc.) but I was reminded of this a while back when I was stuck with some piece of Twisted code, was badly messing it up, and (hence) not committing it to my respective Subversion branch. Mostly because I knew what I was writing wasn’t going to work, so why commit it?
This also reminded me of this talk I’d seen last year about the Myth of a Genius Programmer. Probably not the whole of the 55 minutes, but some of its initial bits are pretty relevant to my case. As Brian Fitzpatrick and Ben Collins-Sussman say there, we’ve mythologized the ideal programmer: “Hey, here’s a genius, and the genius goes off in a cave, writes this brilliant thing, reveals it to the world and he’s famous forever”. And as Fitz and Ben bust this myth, we get to know that this is not the way it really works. Also, in cases where you are probably not working on something that brilliant and clever (like my reluctance to commit the said code) we think (quoting Fitz and Ben once again) “Okay maybe I won’t be seen as a genius, but at least I don’t want people to see my trail of mistakes, and I’m gonna cover my tracks.” People want to be seen as clever. And clever people don’t make mistakes, right?
But as Jean-Paul Calderone told me, when I was shying away from committing my share of some ugly (and beastly) code, “Everyone writes ugly code sometimes. Often you have to, before you can write something nice.” The truth is, not committing code, good or bad, inhibits progress. Not just personal, but also the progress of the project.
While I’m working, I might, say, write comments that make sense only to myself, use some weird extra whitespaces here and there, write loads and loads of rough code that comes mostly from the time when I started out to do something and realised it’s a completely retarded direction and I just don’t want to go there now. And I don’t want others to see this, do I? Mostly when I commit, I want to make sure that at least something works, and remove the weirdness from the code so it makes sense to people.
Though this is alright when you are working on something you’ve worked on before and just need some time to figure the right way out, but when you are trying something new in a project, or, for lack of better words, when you have no idea what you’re doing, it’s better to commit every bit of that random-mindless-direction you started to go in every once in a while, before coming back on track.
This ensures that you (and others who are following your commits) learn from it. People might help you work it out and understand how you could improve that approach or see some side of it which didn’t occur to you.
Not just that, having a history of your code will be good for you even when you take a look at it. Surely, there’ll be cases where you’d go “Hey, that was some piece of complete nonsensical gibberish I wrote, what was wrong with me?”, but then there might also be cases of “Well, that approach led me nowhere, or it makes the system fail, let’s not repeat it.” or better still, you might want to later fix something that’s breaking now.
So, it turns out, committing sad code is okay. It’s just like documenting what happened, to give you an idea of what you can do in future to deal with it, should that happen again.
The trick here, which I learnt after going through the above-mentioned Google I/O talk again, is to iterate failure. Fail quickly, pick yourself up and try something different. The faster you fail, the faster you can iterate and do something better. Apparently, practice makes your iteration failure cycle faster, and (hopefully) failures tend to get smaller over time, and successes get larger.
]]>Everything tastes better with a little internet.
So here’s some news: New endpoint interfaces were added to twisted.internet.endpoints and Twisted now supports IPv6 TCP server endpoints.
It all started (at least my role did, the long term plan is here) with a proposal to provide an API which can support either IPv4 or IPv6 (endpoints). Since my Google Summer of Code project this year is to add more endpoint support to Twisted, my project mentor Jean-Paul Calderone suggested I begin with this one.
We decided to make this enhancement more specific and narrow it down by splitting into TCP server and client endpoints. And we just finished the implementation of the server part!
After initially adding TCP6ServerEndpoint, we found that it was mostly identical to the already existing TCP4ServerEndpoint, with only a minute change in the default value for the interface parameter. To avoid code duplication, we introduced a private interface TCPServerEndpoint, and the public interfaces TCP4ServerEndpoint and TCP6ServerEndpoint now implement it as needed.
Well, that’s all for now!
]]>But what happens if you can’t clear the buffer (e.g. you are a web server sending a 100MB file to someone with a slow 2G smartphone)? If you keep writing, you will accumulate more and more data in the memory.
The producer API lets Twisted notify an object of your choice (i.e. the producer) when it should stop writing as the buffers are full, and resume writing once the buffer has been cleared. That way you can make sure you don’t write stuff to the transport if it’s just going to sit there in the memory.
The consumer (typically a transport) keeps track of the buffer condition and sends signals to stop/resume writing to producer (depending on whether the buffer is full/cleared) which in turn controls the write operation in the transport and does as the consumer signalled.
In the simplest case, the protocol might register itself as a producer with the protocol’s transport and then it will just need to implement a few more methods (pauseProducing(), resumeProducing(), stopProducing()) in addition to the usual methods (connectionMade(), connectionLost(), dataReceived, etc). Hence, for a simple application, there’s no need to create a separate object.
But in some cases (e.g. a web server), you might want different kinds of producers (e.g. one that produces a file, another that forwards requests to another web server, etc). Although one object can be made to deal with more than one source, it is better to have each object responsible for one thing (else the code will be full of conditional statements to handle every case).
(Thanks to Itamar for the info, more on this can be found here.)
]]>