This document is a instroduction to the asynchronous programming model, and to Twisted's Deferred abstraction, which symbolises a 'promised' result and which can pass an eventual result to handler functions.
This document is for readers new to Twisted who are familiar with the Python programming language and, at least conceptually, with core networking conepts such as servers, clients and sockets. This document will give you a high level overview of concurrent programming(interleaving several tasks) and of Twisted's concurrency model:
non-blocking code or asynchronous code.
After discussing the concurrency model of which deferreds are a part, it will introduce the methods of handing results when a function returns a Deferred object.
Many computing tasks take some time to complete, and there are two reasons why a task might take some time:
1. it is computationally intensive(for example factorising large numbers) and requires a certain amount of CPU time to calculate the answer;
2. it is not computationally intensive but has to wait for data to be available to produce a result.
Waiting for answers
A fundamental feature of network programming is that of waiting for data. Imagine you have a function which sends an email summarising some information. This function needs to connect to a remote server, wait for the remote server to reply, check that the remote server can process the email, wait for the reply, send the mail ,wait for the confirmation, and then disconnect.
Any one of these steps may take a long period of time. Your program might use the simplest of all possible models, in which it actually sits and waits for data to be sent and received, but in this case it has some very obvious and basic limitations: it can't send many emails at once; and in fact it can't do anything else while it is sending an email.
Hence, all but the simplest network programs avoid this model. You can use one of several different models to allow your program to keep doing whatever tasks it has on hand while it is waiting for something to happen before a particular task can continue.
Not waiting on data
There are many ways to write network programs. The main ones are:
1. handle each connection in a separate operating system process, in which case the operating system will take care of letting other processes run while one is waiting.
2. handle each connection in a separate thread in which the threading framework takes care of letting other threads run while one is waiting for.
3. use non-blocking system calls to handle all connections in one thread.
Non-blocking calls
The normal model when using the Twisted framework is the third model: non-blocking calls.
When dealing with many connections in one thread the scheduling is the responsibility of the application.not the operating system. and is usually implemented by calling a registered function when each connection is ready to for reading or writing - commonly known as asynchronous, event-driven or callback-based programming.
In this model, the earlier email sending function would work something like this:
1. it calls a connection function to connect the remote server;
2. the connection function returns immediately, with the implication that the notify the email sending library will be called when the connect has been made.
3. once the connection is made, the connect mechanism notifies the email sending function that the connection is ready.
What advantage does the above sequence have over our original blocking sequence?The advantage is that while the email sending function can't do the next part of its job until the connection is open, the rest of the program can do other tasks, like begin the opening sequence for other email connections. Hence, the entire program is not waiting for the connection.
Callbacks
The typical asynchronous model for alerting an application that some data is ready for it is known as a callback. The application calls a function to request some data, and in this call, it also passes a callback function that should be called when the data is ready with the data as an argument. The callback function should therefore perform whatever tasks it was that the application needed that data for.
In synchonous programming, a function requests data, waits for the data, and then processes it. In asynchronous programming, a function requests the data, and lets the library call the callback function when the data is ready.