Thoughts About Elixir

I’ve been programming Erlang for several years now and have always liked it. Erlang syntax has it’s quirks, but I’ve always considered OTP so cool that I don’t mind all the braces, colons, semicolons and stuff. Then I heard about Elixir that runs on Erlang VM, has a Rubyish syntax and can take advantage of all the cool stuff that the Erlang OTP offers. It took a while for Elixir to reach version 1.0 and become really a viable option for developing server software, but I really think it’s now there and I though that I should write about why I think so, and why I’m probably going to do more Elixir programming than Erlang programming in the near future.

Syntax

Elixir syntax is pretty close to Ruby. You don’t have to add braces to function calls if you don’t feel like it and all that stuff. Personally, this is what I don’t like about Ruby. I’m more of a Python guy anyways, forgive me. The Zen of Python states that…

There should be one– and preferably only one –obvious way to do it.

I kind of agree with that. Although, shamed to admit, but Elixir’s Rubyish syntax is growing on me and I’m liking it more and more.

The one thing where Erlang’s syntax is at it’s best is writing recursive code using pattern matching. It combines the different parts of the recursive function nicely using semicolon. Elixir doesn’t make this distinction basically at all. Pattern matched functions all look the same and are somewhat more difficult to spot from the code. For example in Erlang a simple recursive function looks like this…

loop([], Acc) ->;
  Acc;
loop([Head|Tail], Acc) ->;
  loop(Tail, Acc + Head).

And in Elixir…

def loop([], acc) do
  acc
end

def loop([head|tail], acc) do
  loop(tail, acc + head)
end

Note that both examples only have one function, which is maybe not that visible from Elixir code. Not a big deal, but makes code somewhat less readable when, for example, you have gen_fsm module with a lot of callbacks. The Elixir syntax does not group the def clauses in any way, but the compiler politely warns you if you happen to have your function definitions spread all over.

Handling Nested Data Structures

One area where Elixir particularly shines over Erlang is handling nested data structures. It is quite common to have some entity’s state represented by a nested map data structure and you usually all your application does is modify that state. Due to the fact that Erlang (and Elixir) data structures are immutable, modifying these nested data structures gets rather cumbersome. Elixir has some neat utilities to help you deal with these.

Keyword Lists vs Proplists

Proplists are common way of passing on configuration data in Erlang applications. Proplist is a list of two element tuples, where the first element is an key atom and second element is the value which can be any Erlang term. Data in proplists is accessed using the proplists module. So you’ll end up doing a lot of this…

Opts = [{key, "value"}].
Value = proplists:get_value(key, Opts).

Or if you happen to have nested proplist…

Opts = [{key1, [{key2, "value"}]}].
Value = proplists:get_value(
                    key2,
                    proplists:get_value(key1, Opts)).

Elixir has a concept of keyword lists. Which are essentially the same as proplists in Erlang. Well, proplists can contain also atoms which are just a shorthand for {Atom, true}. But basically they are the same. Elixir syntax just has a lot more convenient way of accessing the data. In Elixir you can…

opts = [key: "value"]
value = opts[:key]

Or with nested…

opts = [key1: [key2: "value"]]
value = opts[:key1][:key2]

Updating Nested Maps

Due to the fact that the data structures are immutable you will end up doing something like this when you need to modify a map. Let’s say we have a player state where we have player’s “wallet” containing all the soft/hard currency the player has and we want to reduce the amount of diamonds by ten. We’d do it like this…

state = %{username: "Player1", wallet: %{diamonds: 100, oil: 100, gold: 100}}
wallet = Dict.put(state.wallet, :diamonds, state.wallet.diamonds - 10)
state = Dict.put(state, :wallet, wallet)

Looks unnecessarily complex doesn’t it? Luckily Elixir is all about making things easier for you so it has Kernel.put_in and Kernel.get_in just for this. So here’s how we would use those…

state = %{username: "Player1", wallet: %{diamonds: 100, oil: 100, gold: 100}}
state = put_in(state.wallet.diamonds, state.wallet.diamonds - 10)

Pipe Operator

Simple, yet one of the most useful syntax feature in my opinion is the Elixir pipe operator. Similar to that found from F# language, pipe operator passes on the output of the previous function as the first parameter of the next function. Let’s compare to Erlang once again. Coding in Erlang you easily end up doing plenty of this…

List = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9].
%% take out all odd numbers
List1 = lists:filter(fun(X) -> X rem 2 == 0 end, List).
%% power of two
List2 = lists:map(fun(X) -> X * X end, List1).
%% sum of all
Result = lists:foldl(fun(X, Acc) -> X + Acc end, 0, List2).

And the same using Elixir pipe operator…

list = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
result = list
|> Enum.filter(fn(x) -> rem(x, 2) == 0 end)
|> Enum.map(fn(x) -> x * x end)
|> Enum.reduce(0, fn(x, acc) -> x + acc end)

I just gathered here pretty randomly the syntax related stuff that came to my mind. I really like how it is very clear that Elixir designers have really focused on making a functional language that is easy and pleasant to write and produces easily readable code.

Tooling

It’s common knowledge that Erlang’s build/release tools are quite horrible. There are 3rd party tools like rebar, Erlang.mk and relx, which are pretty neat but are not included in Erlang by default and don’t help the person writing his first Erlang software at all.

Luckily Elixir has really nice tools included. The standard distribution contains a tool called mix that is one of the best I’ve seen in any language. It lets you bootstrap your project, package it, run tests, install dependencies etc. Really nice. One thing it does not do is create a release like Erlang reltool or relx. There is a plugin for mix that handles this called exrm.

Documentation

One more thing where Elixir beats Erlang is documentation. Everything is well documented and documentation is easily accessible either through the website or even more easily through the Elixir shell. For example, try this in iex.

iex(1)> h Enum.map

Interfacing Erlang Code

The thing that finally got me convinced that Elixir is to be taken seriously is the way it interacts with Erlang code. Well, since Elixir compiles to Erlang it is quite straightforward. You just call Erlang modules and for the most part it just works. I can call Erlang like this…

iex(1)> :erlang.now
{1422, 88172, 596308}

Pretty much the only thing that can lead to problems is that since Elixir strings are Erlang binaries it is good to keep in mind that if some Erlang module takes string as parameter you need to give it Elixir’s character list which is the single quoted string in Elixir syntax.

All in all

I’ve been extremely happy working with Elixir. I quite recently started as a Lead Server Programmer in a mobile gaming company called Ministry of Games, based in Helsinki, Finland. We’re building our entire application server using Elixir. What is cool is that, while we’re using Elixir, we have the whole Erlang ecosystem at our disposal as well. E.g., we’re using ranch and locker which both are awesome pieces or Erlang software that do some of the most difficult parts of writing distributed servers extremely well, and we can write our application code on top of those using Elixir. A match made in heaven. :)

Ok, now… go give Elixir a try!

Thoughts About Elixir

String encodings in Erlang

Erlang is famous for the way it deals with strings. Being that strings are “just a list of integers”. Sounds easy, doesn’t it? I’ve been having some issues with Erlang and UTF-8 strings lately and I thought I would write down some of my findings here.

I’m playing with the Erlang shell here so first I need to figure out which encoding my shell is using. This is how I can figure that out.

1> lists:keyfind(encoding, 1, io:getopts()).
{encoding, unicode}

According to the Erlang manual, the shell should be able to read and write UTF-8 strings if your environment has been configured properly. Now we can see that our Erlang shell supports Unicode, so we’re happy. Let’s start playing around with some strings.

Let’s start with string “äiti” which is a Finnish word for mom (I find myself missing my mom often when dealing with strings in Erlang). By default, Erlang strings use Latin-1 encoding. The list of integers representing this string looks like this.

1> String = [228, 105, 116, 105].
"äiti"

As you can see, Erlang is “clever” enough to show the string represented by this list of integers. These integers represent code points for the characters and since the default encoding in Erlang is Latin-1 the code point for character “ä” is 228. For Latin-1, these code points are integers from 0 to 255. So you can represent 256 different characters with Latin-1. If you’re using ASCII encoding your code points are between 0 and 127 (ASCII uses seven bits). Since ASCII is a subset of Latin-1 the characters, “i” and “t” have same code points in ASCII and Latin-1.

Ok, pretty easy so far. But what happens when 256 characters just is not enough. Say hello to Unicode. With unicode, one can represent basically any number of characters. The code points are not limited to just one byte anymore. Let’s stick with the word “äiti” still for a while. We know that ASCII is a subset of Latin-1 and actually Latin-1 is also a subset of Unicode. Makes sense, ha? So Latin-1 string “äiti” looks in terms of code points exactly the same as Unicode string “äiti“.

1> Latin1String = [228, 105, 116, 105].
"äiti"
2> UnicodeString = [228, 105, 116, 105].
"äiti"

Now this is just convenient, since the character “ä” has the same code point in Latin-1 and Unicode and it can be represented using only one byte. Ok, this looks pretty easy. Nothing can go wrong here since the Latin-1 and Unicode strings look exactly the same. Well, not quite. Since Unicode can represent way more characters than Latin-1 we need to agree on how the Unicode strings are represented on the byte level. You cannot represent the Unicode character “snow man” (☃) with one byte since it’s Unicode code point is 9731. Here we need UTF-8 encoding. It is very commonly used and it is the encoding you need to use nowadays. For example, popular data serialization format JSON assumes that the JSON string is encoded in UTF-8. Ok, let’s look at how the string “äiti” looks like in UTF-8. The integers here no longer represent the code point in Unicode, but the byte of the UTF-8 string. As you can see we are using Erlang binaries here to represent the string.

1> Utf8String = <<195, 164, 105, 116, 105>>.
<<"äiti">>

Now Erlang shows the string a bit messed up, since it tries to convert the binary to string using one byte per character and as you can see UTF-8 uses two bytes to represent the character “ä” and the string looks to be messed up. How we can deal with this in Erlang? We need to use the unicode module.

2> unicode:characters_to_list(Utf8String, utf8).
"äiti"

A very common way to mess up things here is to use erlang:binary_to_list/1. The same thing happens as with the shell print out. binary_to_list/1 converts binary to list byte by byte and now your string has five characters instead of four. This can easily lead to “exploding” strings if you write this string to database and read it from there and decode it again with binary_to_list/1.

Like I already mentioned UTF-8 is the “de facto” encoding nowadays. So here are few pointers about how to deal with UTF-8 strings in Erlang.

Creating strings

If you create strings the usual way (String = "äiti".) remember that it is Latin-1 encoded.

If you are reading strings from somewhere as a byte stream, you should use unicode:characters_to_list/2 and give the function the proper encoding e.g.,

1> unicode:characters_to_list(<<226,152,131>>, utf8).
[9731]

Here we have the UTF-8 encoded binary representation of the “snow man” (☃) character. The output is list with Unicode code point integers. This Unicode string you can use with e.g., functions from the string module.

Convert Unicode strings to binaries

Some functions like crypto:sha_mac/2 requires the input to be an iolist(), which a list that does not care about the encodings but must be just a list of bytes. This cannot take Unicode string as a parameter since those might have integers in them that need more than one byte. So if you’re handling strings as Unicode, you will need to convert them to UTF-8 binaries to functions like crypto:sha_mac/2. Like this:

1> UnicodeString = [9731].
[9731]
2> Utf8Bin = unicode:characters_to_binary(UnicodeString, unicode, utf8).
<<226,152,131>>
3> crypto:sha_mac(<<"key">>, Utf8Bin).

As a conclusion. Everything should go well if you stick with Unicode strings and remember to encode/decode them to/from UTF-8 every now and then.

String encodings in Erlang

Playing Around With Openstack’s Object Storage

Couple of weeks ago I found out about the Openstack project and I found it immediately to be very interesting. What I’ve been playing around with the most is the object storage part of Openstack called Swift. I’ll show here how you can use Swift with a couple of different libraries. The nice thing about Swift is that it is basically the Rackspace Cloudfiles storage, so the same libraries that work with Cloudfiles, should work with Swift as well. Well, they require some small modifications. But, I’ll show you here two libraries that I know are working already. Of course, you will need a Swift instance running somewhere and instructions on how to setup one you can read the “Swift All In One” document that shows how you can run Swift on a single server.

The first library I’ll show here is the python-cloudfiles. I recommend using the latest one from Github, since the one that you can get for example from Ubuntu repositories does not support Swift and the one you can get from Python Package Index had a bug that made it not work with Swift.

Here I’ll show you how you can connect to your local Swift instance using the authurl parameter and how you can create containers and objects using python-cloudfiles.

from cloudfiles.connection import Connection
 
conn = Connection("test:test", "test", authurl="http://127.0.0.1:11000/v1.0")
 
container = conn.create_container("test")
 
obj = container.create_object("test.txt")
obj.content_type = "text/plain"
obj.write("test")

Pretty straightforward… right?

Next, I’ll show you another library that works with Swift called cferl. It’s a Erlang library for Cloudfiles and I made some simple patches to it to make it work with Swift.

Here’s how you can do the same things as in previous example using cferl.

ibrowse:start().
{ok, Connection} = cferl:connect("test:test", "test", "http://127.0.0.1:11000/v1.0").
 
{ok, Container} = Connection:create_container(<<"test">>).
 
{ok, Object} = Container:create_object(<<"test.txt">>).
ok = Object:write_data(<<"test">>, <<"text/plain">>).

Ok, that’s it. Now you can start playing with Swift and storing petabytes of data in it.

Playing Around With Openstack’s Object Storage

Exercise in Python Decorators

Recently, I found myself in need of a web server that I can use to simulate a behavior of a certain website. I wanted to just copy the output of that website and deliver it using this web server. The problem was that serving static content is naturally way faster than serving dynamic web application, so for my simulation I needed to make the web server wait for a certain period of time before returning the static file. Being a Python fan I decided to use Tornado as the web server. Now, all I needed to do is slow it down.

Ok, I could just simply do this…

time.sleep(0.5)

… to make my server wait half a second before returning, but since Tornado is using asynchronous networking and hence runs in a single thread, this would block all other requests made to my server. Not good…

I need to do this asynchronously. Here is an example on how this can be done with Tornado without blocking.

import time
 
import tornado.web
import tornado.ioloop
 
class RequestHandler(tornado.web.RequestHandler):
 
    def _finish_request(self):
        self.finish()
 
    @tornado.web.asynchronous
    def get(self):
        ioloop = tornado.ioloop.IOLoop.instance()
        ioloop.add_timeout(time.time() + 0.5, self._finish_request)
        self.add_header("Content-Type", "text/plain")
        self.write("Hello, world")

So, this is an example of an asynchronous Tornado request handler that will wait for half a second before returning and will not block other requests while doing that. Here we are using decorator tornado.web.asynchronous on line 11 to tell Tornado that this request should not be returned immediately and we need to call tornado.web.RequestHandler.finish() on our own. The timeout is implemented by calling tornado.ioloop.IOLoop.add_timeout() method which is given a callback method that will finish the request.

Now, the problem with this is that, if I need other request handlers to do the same thing, I would need to copy paste this peace of code all over the place. And I don’t like that. We can do this bit more elegantly by using Python decorators. By writing a decorator we can avoid duplicating the same code to every request handler. This is how the same example will look using a decorator.

import tornado.web
import tornado.decorators
 
class RequestHandler(tornado.web.RequestHandler):
 
    @tornado.decorators.wait_for(milliseconds=500)                                               
    def get(self):                                                            
        self.set_header("Content-Type", "text/plain")                         
        self.write("Hello, world") 

Looks nice and clean. Doesn’t it? Well, the complex part has been moved now to the decorator tornado.decorators.wait_for. Let’s look now how we can implement that.

import time
 
from functools import partial
 
from tornado.web import asynchronous
from tornado.ioloop import IOLoop
 
def wait_for(milliseconds=0):
    def _finish_request(request, start_time):
        timeout = (time.time() - start_time) * 1000.0
	request.write("\n\nServer waited for %.3f ms" % timeout)
        request.finish()
    def _decorator(func):
	func = asynchronous(func)
        def _wrapper(*args, **kwargs):
            ioloop = IOLoop.instance()
            callback = partial(_finish_request, args[0], time.time())
            ioloop.add_timeout(time.time() + milliseconds / 1000.0, callback)
            func(*args, **kwargs)
	return _wrapper
    return _decorator

Here I have implemented the decorator wait_for that takes the number of milliseconds to wait as a parameter. Let’s start from line 13 where the actual decorator is implemented. Decorator’s parameter is always the function that is being decorated. You can think of this…

@my_decorator
def my_function():
    // do something

…being same as this…

def my_function():
    // do something
my_function = my_decorator(my_function)

Now, on line 14 decorate the function with Tornado’s tornado.web.asynchronous decorator. Just like we did in the first example. So that our request does not return before we call tornado.web.RequestHandler.finish(). Then on line 15 we write a wrapper method that adds the timeout and callback to tornado.ioloop.IOLoop and after that calls the original function that we are decorating. The trickiest part here probably is that we need to give our callback function some parameters and Tornado’s add_timeout method only takes the callback function as the parameter. For that we use Python’s functools.partial to generate the callback and give some parameters to it on line 17.

To conclude the blog post here is a complete example of a script that you can test this with. You need to create the tornado/decorators.py using the code above for this to work.

import time
 
import tornado.httpserver
import tornado.ioloop
import tornado.web
 
from tornado.decorators import wait_for
 
 
class MainHandler(tornado.web.RequestHandler):
 
    @wait_for(milliseconds=500)
    def get(self):
        self.set_header("Content-Type", "text/plain")
        self.write("Hello, world")
 
application = tornado.web.Application([(r"/", MainHandler)])
 
if __name__ == "__main__":
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(8080)
    tornado.ioloop.IOLoop.instance().start()

If everything goes right your server should return something like this…

Hello, world
 
Server waited for 500.371 ms
Exercise in Python Decorators

Generating random strings in Erlang

I could not find any decent examples from the web on how to generate a random string with a certain set of characters and length in Erlang. The basic idea for such a method is to take a string of allowed characters and loop N times where the N is the length of the resulting string. Then at each loop we take some random character from the string that contains the required set of characters. Sounds relatively simple, right? Next we have to write this in Erlang. This is what I came up with…

get_random_string(Length, AllowedChars) ->
    lists:foldl(fun(_, Acc) ->
                        [lists:nth(random:uniform(length(AllowedChars)),
                                   AllowedChars)]
                            ++ Acc
                end, [], lists:seq(1, Length)).

Ok, Erlang is not the most readable language in the world and a simple thing such as generating a random string can look pretty tedious. No worries. I’ll go through the method line by line.

I’m using the lists:foldl method here. What it does is that it goes through a list (from left to right) and calls a function that has as it’s parameter a value from that list and the result form the previous iteration. The result of the method is the result of the last call to the function. The list I give as a parameter to lists:foldl is a sequence of numbers from one to the length of the resulting random string. For that I use the lists:seq method. This is how we define how many times we loop.

I’ll explain the fun() that is the first parameter of lists:foldl. Here is what it looks like separate from the whole code.

fun(_, Acc) ->
     [lists:nth(random:uniform(length(AllowedChars)), AllowedChars)]
          ++ Acc
end

The first parameter of the function is the value from the given list ([1, 2, 3, 4,..., N]) and we don’t use it (hence the underscore). The second parameter Acc is called the accumulator that is the result from the previous iteration. To achieve our goal of producing random strings we use lists:nth and random:uniform method calls to pick a random character from the AllowedChars string. Note that the lists:nth returns the integer value of that character so that is why the method call is wrapped in square brackets making the result a string (in Erlang strings are lists of integers). What we do then is that we add the Acc (the result of the previous iteration) to the result and this way build our random string.

There is also a third parameter for the lists:foldl method that you probably have guessed already. Naturally, you also have to give the value of the accumulator for the first iteration, which in this case is empty list [] or empty string since strings in Erlang are actually lists.

Here is an example of the result that the method produces.

test:get_random_string(32, "qwertyQWERTY1234567890").     
"8qttW01wQET1qRTt1r4tr2T392QY94Re"
Generating random strings in Erlang

Automatic code reloading in Erlang

I’ve recently got back to coding Erlang and noticed a neat module that I didn’t know existed that is probably worth writing a blog entry about. I’ve started developing a PubSubHubbub hub in Erlang called Hubbabubba and I’m using the great Mochiweb HTTP library as the HTTP server implementation. I discovered the reloader.erl module that comes with Mochiweb. It automatically reloads the code when you have the application running and you modify the code (remember to compile as well). This is something that I’ve found very useful when developing with Django or AppEngine and I’m really satisfied that there is a similar solution for Erlang as well.

Automatic code reloading in Erlang

Using zc.buildout in a Twisted project

I’ve been learning how to use zc.buildout in my Python projects. It seems to be used mostly by the Zope and the Plone communities, but I feel that it might be a useful tool to be used in any Python project. I do lots of work with Twisted and I wanted to try zc.buildout with a Twisted project. It wasn’t trivial and I wasn’t able to find any good examples, so I thought I might as well document my experiences here.

Let’s start this exercise by setting a new virtual Python environment using virtualenv. It gives you a nice constrained Python environment that does not mess up your system. Easiest way to get virtualenv on your system is by using setuptools. When you have setuptools installed you can get virtualenv by typing:

$ easy_install virtualenv

Now that you have virtualenv installed it’s time to create the virtual environment. In this example I will create the environment in a directory called twistedenv. You can place it anywhere you want in your system.

$ mkdir twistedenv
$ virtualenv --no-site-packages twistedenv/
New python executable in twistedenv/bin/python
Please make sure you remove any previous custom paths from your /Users/teemu/.pydistutils.cfg file.
Installing setuptools............done.

Now that the virtual environment is setup you can activate it by typing:

$ cd twistedenv
$ source bin/activate

This will activate the new Python interpreter that was installed in twistedenv/bin/python.

Ok, now we can get to the actual topic, that is how to use zc.buildout with a Twisted project. What you will need is a bootstrap script that will install zc.buildout inside your environment and a buildout configuration file. Let’s start with the configuration file, since we need that before running the bootstrap script. In zc.buildout the configuration file is called buildout.cfg. It will tell the buildout system, what packages are needed and setup your development system in a blink of an eye. I’ll start with an example that will install Twisted and some dependencies and a sample Twisted application that is under development. Here’s the buildout.cfg that will do this.

[buildout]
parts =
    depends
    twisted
    twisteds
develop = ./src

[versions]
Twisted = 8.2.0
pyOpenSSL = 0.8
pyserial = 2.4
pycrypto = 2.0.1

[depends]
recipe = minitage.recipe:egg
eggs =
    pyOpenSSL
    pyserial
    pycrypto

[twisted]
recipe = minitage.recipe:egg
eggs = Twisted

[twisteds]
recipe = minitage.recipe:scripts
interpreter = twistedpy
extra-paths = ${buildout:directory}/src
eggs =
  ${twisted:eggs}
  ${depends:eggs}

I’ll briefly explain what it does. Section buildout is the first thing to put in the config file. It lists what parts of the configuration file are used and the location of the code that is being developed. The section called depends installs the dependencies for Twisted using recipe minitage.recipe. Section twisted naturally installs Twisted, but it does not install the scripts that are installed to the bin directory. That is what the section twisteds is for. By the way, you are free to choose the names of the sections or parts except for the buildout section. In the twisteds section we also define the variable interpreter to create us a new Python interpreter that will be used when running these scripts. For this interpreter we define which eggs are included in the interpreter path and using extra-paths variable we can include our development code there also. It is useful that we can keep our own code separate from 3rd party code while developing it. Section versions can be used to specify which versions of 3rd party libraries we want to use. If you don’t specify a version, a newest version available is used.

Now that we have the buildout.cfg file ready, we can bootsrap the buildout. Download the bootstrap script from this URL and put it in the twistedenv directory. Before running the bootstrap script you can make sure that you are using the Python interpreter from the virtual environment to run it instead of the syste Python. Do that by typing:

$ which python
/Users/teemu/Programming/twistedenv/bin/python

As you can see, for me it shows that I’m using the Python interpreter from inside the twistedenv where we created the virtual environment. That is good. Naturally, for you the complete path is something else. Then we can run the bootstrap script by typing:

$ python bootstrap.py
Creating directory '/Users/teemu/Programming/twistedenv/parts'.
Creating directory '/Users/teemu/Programming/twistedenv/eggs'.
Creating directory '/Users/teemu/Programming/twistedenv/develop-eggs'.
Generated script '/Users/teemu/Programming/twistedenv/bin/buildout'.

As you can see from the output the script created couple of directories and generated a script called buildout to the bin directory. Now our buildout environment is almost ready. The only thing is missing is that our ./src directory does not contain any code. You will see an error if you now try to run bin/buildout.

Let’s create a quick sample Twisted application to demonstrate a more complete buildout environment. This is what we need. Create the required directories for our code:

mkdir -p src/sample
mkdir -p src/twisted/plugins
touch src/sample/__init__.py

Then we need the following files:

src/sample/tap.py:

from twisted.application import internet
from twisted.web import server
from twisted.web.resource import Resource
from twisted.python import usage

class SimpleResource(Resource):
    isLeaf = True
    def render_GET(self, request):
        return "

It Works!

" class Options(usage.Options): pass def makeService(config): site = server.Site(SimpleResource()) return internet.TCPServer(8080, site)

src/twisted/plugins/sample.py:

from twisted.application.service import ServiceMaker

Sample = ServiceMaker(
    "Sample",
    "sample.tap",
    "Twisted sample application",
    "sample"
)

And last, but not least, src/setup.py:

from setuptools import setup

setup(name='sample',
      version='0.0.1',
      description='Sample Twisted application',
      author='Teemu Harju',
      author_email='teemu.harju@gmail.com',
      url='http://blog.teemu.im',
      packages=['sample'],
      package_data={'twisted.plugins': ['twisted/plugins/sample.py']},
      zip_safe=False
)

These files will create a Twisted application called sample that runs a webserver on port 8080 that responds “It works!” when the root URL is accessed.

Now, run:

$ bin/buildout
...

I’ve not put the output to the example since there is so much of it. But after the buildout is ready and if everything worked fine you can test that your environment is ready by running the bin/twistd script like this:

$ bin/twistd --help

The output should contain our sample application and you can run it by typing:

$ bin/twistd -n sample

Now you can start developing your Twisted application and test it in your safe and restricted environment using virtualenv and buildout. By the way, if you want to get out of your virtual Python environment you can do that simply by typing:

$ deactivate

It would be probably good to include some kind of a test runner in the buildout script to make sure that your buildout is not broken. How that can be done is probably a blog post topic of it’s own. That’s all for now. Have fun with buildout and Twisted. Feel free to ask in the comments if this didn’t work for you for some reason.

Using zc.buildout in a Twisted project