String encodings in Erlang

Erlang is famous for the way it deals with strings. Being that strings are “just a list of integers”. Sounds easy, doesn’t it? I’ve been having some issues with Erlang and UTF-8 strings lately and I thought I would write down some of my findings here.

I’m playing with the Erlang shell here so first I need to figure out which encoding my shell is using. This is how I can figure that out.

According to the Erlang manual, the shell should be able to read and write UTF-8 strings if your environment has been configured properly. Now we can see that our Erlang shell supports Unicode, so we’re happy. Let’s start playing around with some strings.

Let’s start with string “äiti” which is a Finnish word for mom (I find myself missing my mom often when dealing with strings in Erlang). By default, Erlang strings use Latin-1 encoding. The list of integers representing this string looks like this.

As you can see, Erlang is “clever” enough to show the string represented by this list of integers. These integers represent code points for the characters and since the default encoding in Erlang is Latin-1 the code point for character “ä” is 228. For Latin-1, these code points are integers from 0 to 255. So you can represent 256 different characters with Latin-1. If you’re using ASCII encoding your code points are between 0 and 127 (ASCII uses seven bits). Since ASCII is a subset of Latin-1 the characters, “i” and “t” have same code points in ASCII and Latin-1.

Ok, pretty easy so far. But what happens when 256 characters just is not enough. Say hello to Unicode. With unicode, one can represent basically any number of characters. The code points are not limited to just one byte anymore. Let’s stick with the word “äiti” still for a while. We know that ASCII is a subset of Latin-1 and actually Latin-1 is also a subset of Unicode. Makes sense, ha? So Latin-1 string “äiti” looks in terms of code points exactly the same as Unicode string “äiti“.

Now this is just convenient, since the character “ä” has the same code point in Latin-1 and Unicode and it can be represented using only one byte. Ok, this looks pretty easy. Nothing can go wrong here since the Latin-1 and Unicode strings look exactly the same. Well, not quite. Since Unicode can represent way more characters than Latin-1 we need to agree on how the Unicode strings are represented on the byte level. You cannot represent the Unicode character “snow man” (☃) with one byte since it’s Unicode code point is 9731. Here we need UTF-8 encoding. It is very commonly used and it is the encoding you need to use nowadays. For example, popular data serialization format JSON assumes that the JSON string is encoded in UTF-8. Ok, let’s look at how the string “äiti” looks like in UTF-8. The integers here no longer represent the code point in Unicode, but the byte of the UTF-8 string. As you can see we are using Erlang binaries here to represent the string.

Playing Around With Openstack’s Object Storage

Couple of weeks ago I found out about the Openstack project and I found it immediately to be very interesting. What I’ve been playing around with the most is the object storage part of Openstack called Swift. I’ll show here how you can use Swift with a couple of different libraries. The nice thing about Swift is that it is basically the Rackspace Cloudfiles storage, so the same libraries that work with Cloudfiles, should work with Swift as well. Well, they require some small modifications. But, I’ll show you here two libraries that I know are working already. Of course, you will need a Swift instance running somewhere and instructions on how to setup one you can read the “Swift All In One” document that shows how you can run Swift on a single server.

The first library I’ll show here is the python-cloudfiles. I recommend using the latest one from Github, since the one that you can get for example from Ubuntu repositories does not support Swift and the one you can get from Python Package Index had a bug that made it not work with Swift.

Here I’ll show you how you can connect to your local Swift instance using the authurl parameter and how you can create containers and objects using python-cloudfiles.

Pretty straightforward… right?

Next, I’ll show you another library that works with Swift called cferl. It’s a Erlang library for Cloudfiles and I made some simple patches to it to make it work with Swift.

Here’s how you can do the same things as in previous example using cferl.

Ok, that’s it. Now you can start playing with Swift and storing petabytes of data in it.

Exercise in Python Decorators

Recently, I found myself in need of a web server that I can use to simulate a behavior of a certain website. I wanted to just copy the output of that website and deliver it using this web server. The problem was that serving static content is naturally way faster than serving dynamic web application, so for my simulation I needed to make the web server wait for a certain period of time before returning the static file. Being a Python fan I decided to use Tornado as the web server. Now, all I needed to do is slow it down.

Ok, I could just simply do this…

… to make my server wait half a second before returning, but since Tornado is using asynchronous networking and hence runs in a single thread, this would block all other requests made to my server. Not good…

I need to do this asynchronously. Here is an example on how this can be done with Tornado without blocking.

So, this is an example of an asynchronous Tornado request handler that will wait for half a second before returning and will not block other requests while doing that. Here we are using decorator tornado.web.asynchronous on line 11 to tell Tornado that this request should not be returned immediately and we need to call tornado.web.RequestHandler.finish() on our own. The timeout is implemented by calling tornado.ioloop.IOLoop.add_timeout() method which is given a callback method that will finish the request.

Now, the problem with this is that, if I need other request handlers to do the same thing, I would need to copy paste this peace of code all over the place. And I don’t like that. We can do this bit more elegantly by using Python decorators. By writing a decorator we can avoid duplicating the same code to every request handler. This is how the same example will look using a decorator.

Looks nice and clean. Doesn’t it? Well, the complex part has been moved now to the decorator tornado.decorators.wait_for. Let’s look now how we can implement that.


import time

from functools import partial

from tornado.web import asynchronous
from tornado.ioloop import IOLoop

def wait_for(milliseconds=0):
    def _finish_request(request, start_time):
        timeout = (time.time() - start_time) * 1000.0
	request.write("\n\nServer waited for %.3f ms" % timeout)
    def _decorator(func):
	func = asynchronous(func)
        def _wrapper(*args, **kwargs):
            ioloop = IOLoop.instance()
            callback = partial(_finish_request, args[0], time.time())
            ioloop.add_timeout(time.time() + milliseconds / 1000.0, callback)
            func(*args, **kwargs)
	return _wrapper
    return _decorator

Here I have implemented the decorator wait_for that takes the number of milliseconds to wait as a parameter. Let’s start from line 13 where the actual decorator is implemented. Decorator’s parameter is always the function that is being decorated. You can think of this…

…being same as this…

Now, on line 14 decorate the function with Tornado’s tornado.web.asynchronous decorator. Just like we did in the first example. So that our request does not return before we call tornado.web.RequestHandler.finish(). Then on line 15 we write a wrapper method that adds the timeout and callback to tornado.ioloop.IOLoop and after that calls the original function that we are decorating. The trickiest part here probably is that we need to give our callback function some parameters and Tornado’s add_timeout method only takes the callback function as the parameter. For that we use Python’s functools.partial to generate the callback and give some parameters to it on line 17.

To conclude the blog post here is a complete example of a script that you can test this with. You need to create the tornado/ using the code above for this to work.

import time

import tornado.httpserver
import tornado.ioloop
import tornado.web

from tornado.decorators import wait_for

class MainHandler(tornado.web.RequestHandler):

    def get(self):
        self.set_header("Content-Type", "text/plain")
        self.write("Hello, world")

application = tornado.web.Application([(r"/", MainHandler)])

if __name__ == "__main__":
    http_server = tornado.httpserver.HTTPServer(application)

If everything goes right your server should return something like this…

Generating random strings in Erlang

I could not find any decent examples from the web on how to generate a random string with a certain set of characters and length in Erlang. The basic idea for such a method is to take a string of allowed characters and loop N times where the N is the length of the resulting string. Then at each loop we take some random character from the string that contains the required set of characters. Sounds relatively simple, right? Next we have to write this in Erlang. This is what I came up with…

Ok, Erlang is not the most readable language in the world and a simple thing such as generating a random string can look pretty tedious. No worries. I’ll go through the method line by line.

I’m using the lists:foldl method here. What it does is that it goes through a list (from left to right) and calls a function that has as it’s parameter a value from that list and the result form the previous iteration. The result of the method is the result of the last call to the function. The list I give as a parameter to lists:foldl is a sequence of numbers from one to the length of the resulting random string. For that I use the lists:seq method. This is how we define how many times we loop.

I’ll explain the fun() that is the first parameter of lists:foldl. Here is what it looks like separate from the whole code.

The first parameter of the function is the value from the given list ([1, 2, 3, 4,..., N]) and we don’t use it (hence the underscore). The second parameter Acc is called the accumulator that is the result from the previous iteration. To achieve our goal of producing random strings we use lists:nth and random:uniform method calls to pick a random character from the AllowedChars string. Note that the lists:nth returns the integer value of that character so that is why the method call is wrapped in square brackets making the result a string (in Erlang strings are lists of integers). What we do then is that we add the Acc (the result of the previous iteration) to the result and this way build our random string.

There is also a third parameter for the lists:foldl method that you probably have guessed already. Naturally, you also have to give the value of the accumulator for the first iteration, which in this case is empty list [] or empty string since strings in Erlang are actually lists.

Here is an example of the result that the method produces.

Automatic code reloading in Erlang

I’ve recently got back to coding Erlang and noticed a neat module that I didn’t know existed that is probably worth writing a blog entry about. I’ve started developing a PubSubHubbub hub in Erlang called Hubbabubba and I’m using the great Mochiweb HTTP library as the HTTP server implementation. I discovered the reloader.erl module that comes with Mochiweb. It automatically reloads the code when you have the application running and you modify the code (remember to compile as well). This is something that I’ve found very useful when developing with Django or AppEngine and I’m really satisfied that there is a similar solution for Erlang as well.

Using zc.buildout in a Twisted project

I’ve been learning how to use zc.buildout in my Python projects. It seems to be used mostly by the Zope and the Plone communities, but I feel that it might be a useful tool to be used in any Python project. I do lots of work with Twisted and I wanted to try zc.buildout with a Twisted project. It wasn’t trivial and I wasn’t able to find any good examples, so I thought I might as well document my experiences here.

Let’s start this exercise by setting a new virtual Python environment using virtualenv. It gives you a nice constrained Python environment that does not mess up your system. Easiest way to get virtualenv on your system is by using setuptools. When you have setuptools installed you can get virtualenv by typing:

$ easy_install virtualenv

Now that you have virtualenv installed it’s time to create the virtual environment. In this example I will create the environment in a directory called twistedenv. You can place it anywhere you want in your system.

$ mkdir twistedenv
$ virtualenv --no-site-packages twistedenv/
New python executable in twistedenv/bin/python
Please make sure you remove any previous custom paths from your /Users/teemu/.pydistutils.cfg file.
Installing setuptools............done.

Now that the virtual environment is setup you can activate it by typing:

$ cd twistedenv
$ source bin/activate

This will activate the new Python interpreter that was installed in twistedenv/bin/python.

Ok, now we can get to the actual topic, that is how to use zc.buildout with a Twisted project. What you will need is a bootstrap script that will install zc.buildout inside your environment and a buildout configuration file. Let’s start with the configuration file, since we need that before running the bootstrap script. In zc.buildout the configuration file is called buildout.cfg. It will tell the buildout system, what packages are needed and setup your development system in a blink of an eye. I’ll start with an example that will install Twisted and some dependencies and a sample Twisted application that is under development. Here’s the buildout.cfg that will do this.

parts =
develop = ./src
Twisted = 8.2.0
pyOpenSSL = 0.8
pyserial = 2.4
pycrypto = 2.0.1
recipe = minitage.recipe:egg
eggs =
recipe = minitage.recipe:egg
eggs = Twisted
recipe = minitage.recipe:scripts
interpreter = twistedpy
extra-paths = ${buildout:directory}/src
eggs =

I’ll briefly explain what it does. Section buildout is the first thing to put in the config file. It lists what parts of the configuration file are used and the location of the code that is being developed. The section called depends installs the dependencies for Twisted using recipe minitage.recipe. Section twisted naturally installs Twisted, but it does not install the scripts that are installed to the bin directory. That is what the section twisteds is for. By the way, you are free to choose the names of the sections or parts except for the buildout section. In the twisteds section we also define the variable interpreter to create us a new Python interpreter that will be used when running these scripts. For this interpreter we define which eggs are included in the interpreter path and using extra-paths variable we can include our development code there also. It is useful that we can keep our own code separate from 3rd party code while developing it. Section versions can be used to specify which versions of 3rd party libraries we want to use. If you don’t specify a version, a newest version available is used.

Now that we have the buildout.cfg file ready, we can bootsrap the buildout. Download the bootstrap script from this URL and put it in the twistedenv directory. Before running the bootstrap script you can make sure that you are using the Python interpreter from the virtual environment to run it instead of the syste Python. Do that by typing:

$ which python

As you can see, for me it shows that I’m using the Python interpreter from inside the twistedenv where we created the virtual environment. That is good. Naturally, for you the complete path is something else. Then we can run the bootstrap script by typing:

$ python
Creating directory '/Users/teemu/Programming/twistedenv/parts'.
Creating directory '/Users/teemu/Programming/twistedenv/eggs'.
Creating directory '/Users/teemu/Programming/twistedenv/develop-eggs'.
Generated script '/Users/teemu/Programming/twistedenv/bin/buildout'.

As you can see from the output the script created couple of directories and generated a script called buildout to the bin directory. Now our buildout environment is almost ready. The only thing is missing is that our ./src directory does not contain any code. You will see an error if you now try to run bin/buildout.

Let’s create a quick sample Twisted application to demonstrate a more complete buildout environment. This is what we need. Create the required directories for our code:

mkdir -p src/sample
mkdir -p src/twisted/plugins
touch src/sample/

Then we need the following files:


from twisted.application import internet
from twisted.web import server
from twisted.web.resource import Resource
from twisted.python import usage
class SimpleResource(Resource):
    isLeaf = True
    def render_GET(self, request):
        return "<h1>It Works!</h1>"
class Options(usage.Options):
def makeService(config):
    site = server.Site(SimpleResource())
    return internet.TCPServer(8080, site)


from twisted.application.service import ServiceMaker
Sample = ServiceMaker(
    "Twisted sample application",

And last, but not least, src/

from setuptools import setup
      description='Sample Twisted application',
      author='Teemu Harju',
      package_data={'twisted.plugins': ['twisted/plugins/']},

These files will create a Twisted application called sample that runs a webserver on port 8080 that responds “It works!” when the root URL is accessed.

Now, run:

$ bin/buildout

I’ve not put the output to the example since there is so much of it. But after the buildout is ready and if everything worked fine you can test that your environment is ready by running the bin/twistd script like this:

$ bin/twistd --help

The output should contain our sample application and you can run it by typing:

$ bin/twistd -n sample

Now you can start developing your Twisted application and test it in your safe and restricted environment using virtualenv and buildout. By the way, if you want to get out of your virtual Python environment you can do that simply by typing:

$ deactivate

It would be probably good to include some kind of a test runner in the buildout script to make sure that your buildout is not broken. How that can be done is probably a blog post topic of it’s own. That’s all for now. Have fun with buildout and Twisted. Feel free to ask in the comments if this didn’t work for you for some reason.

Python For Series 60 v.1.9 Released For Testing

Python for Nokia’s Series 60 platform has been around for four years now and to be honest, not much has happened on that front during the past year. However, suddenly on 24th of December Nokia releases a version 1.9.0 of the Python for Series 60 that is a major rewrite of the whole thing and comes now with Python 2.5 version of the core language. As usual, the odd-number version branch means that the release is for testing purposes only and the even-numbered 2.0 version branch should be released once it becomes more stable. Blog at has done pretty nice wrap up of the new release and here are my thoughts.

First of all I should mention that as I write this blog entry I’ve not yet had time to test the new Python for Series 60 release. So I apologize beforehand for all false comments I’m about to make.

It seems that they are giving a lot more focus on the ease of development in this new release. Which is a really good thing I must say. What pleases me the most is that they are planning on improving the Python runtime deployment so that the end-user or the guy that is installing Python applications should not have to worry too much about having or not having the Python runtime installed on his phone. Definitely a good thing. The new release includes a packaging tool that is not part of the S60 SDK and is basically ensymble with added GUI. Ensymble is an excellent tool and it is very nice to see it included in the official PyS60 release.

Like mentioned already, the new release includes the version 2.5 of the Python interpreter and most of it’s standard libraries. Ok, the word “most” does not sound good here. I’ll focus first on the good things. First, the new release has Expat XML parser in it. Definitely a good thing since XML is something that pretty much every application out there uses and so far people have had to make ugly regexp XML parsers or use 3rd party package of expat to be able to parse XML in their PyS60 applications. However, I would say that including JSON parser as the official Python 2.6 release did lately, would probably be a good idea too.

Also, inclusion of asyncore and more compliant socket module sounds nice. I can’t wait to try Twisted on PyS60 and see how it works. One could do some crazy things with that on a mobile phone.

Then to the things that are not included from the standard Python 2.5 libraries. Now, I don’t have too much information on this since, like said before, I haven’t tried the new release yet. However, to my disappointment I noticed that sqlite3 is not included. SQLite should probably be a platform component in S60 because it is kind of becoming a de facto standard in mobile platforms since both iPhone and Android platforms use it. I don’t know what is the equivalent in S60 or does such exist but having some kind of storage other than just plain text file for PyS60 applications would be extremely nice thing to have.

Ok, I have to try out the new PyS60 release soon. I’m hoping that I will be pleasantly surprised. I’ll definitely write more about it later.

Setting your SRV records straight for XMPP

I’ve noticed that DSN SRV records are something that are not very common knowledge among even some of the really techy people. Well, the reason is probably that most people never need to know what they actually are. I’m gonna explain them here briefly and show how I set the DNS SRV records for my XMPP server hosted at

Ok, so what these DNS SRV records are. If you’re familiar with the DNS MX record you know that it is used to indicate where is the email server that is hosting a certain domain. For example, I have domain and I’m running my webserver at and my email server is at Naturally, I would want my users to have email addresses like Here I would set the DNS MX record to point to so when email servers communicate with each other they can ask from DNS server that “where is the email server for domain”.

DNS SRV records were specified for a bit more general purpose than MX records that can be used with email only. You might have XMPP server at and would like your users have also XMPP address of the format This is how the SRV records for the XMPP look like. This is an example of SRV records I’ve set for my XMPP server at 82698 IN SRV 10 0 5222 86400 IN SRV 10 0 5269

What there actually mean? Well, the line is the line that XMPP clients use to ask from DNS that “where I can find the XMPP server for domain”. The DNS server responds that “the XMPP server can be found from host port 5222″. The other line that has is used by the XMPP servers when they talk to each other using the XMPP server-to-server protocol, similarly as the email servers use the MX records to find out information about each other.

The SRV records can also be used to do “poor mans load balancing” by using the priority and weight attributes, but I won’t go in to that now. Instead, I’ll show you how you can configure DSN SRV records on server. This is because I use Slicehost, but the same principles apply for other hosting providers as well.

First, go to your SliceManager and login. There choose the “DNS” tab and you should see something like this:

Slicehost's SliceManager DNS Zones Tab

Slicehost's SliceManager DNS Zones Tab

Now click “Records” and then “new record”. Fill the form as shown below:

Slicehost SliceManager new DNS SRV record for XMPP clients

Slicehost SliceManager new DNS SRV record for XMPP clients

Note that the value in “Name” field starts with underscore although it is not visible in the screenshot.

Do the same for the server-to-server protocol, except for the name set and for the data use 0 5269

You can use tool called dig for testing your configuration. Type the following line to see if your configuration is correct:

dig SRV

The result should be:

;; ANSWER SECTION: 86400 IN SRV 10 0 5222

By the way… when configuring DNS it is good to set the time-to-live value to something relatively small. In case you make mistakes, you don’t need to wait until the DNS record will be updated. Other good practice is to use the parameter for dig command so that the entry does not propagate to other DNS servers and you can change it pretty much when ever you want.

Google Calendar on Nokia 770 HOWTO

Dan Tye posted a comment on my blog describing how it is possible to make your Google Calendar viewable from your Nokia 770 or from Nokia N800 using the good old Opera browser. It seemed really useful so I decided to post it here so more people will be able to find it.

Anyone who has editing control of their own web site can see their Google calendar on the Nokia 770. here is HOW:

1. On your PC, Click the down arrow next to your calendar listing on the left side of your google calendar page and choose CALENDAR SETTINGS from the drop-down.
2. Under “Private Address” in this page, choose the HTML icon.
3. A POP-UP will appear. Click the link to “Configuration Tool”
4. This page will format HTML you can paste into a private address web page you create to display your Google calendar. (choose option 2 under CONTROLS so the Google Icon will not take the top 1/3 of the page.
5. After copying the resulting code to a file or clipboard, simply create your own calendar page on your own web site. Paste the html into the body of the code view of your web page and publish/save it. You may want to set your web site robots.txt file to ignore this calendar page, thus keeping it a bit more private (KGP, not PGP!)

Shortcomings – yes, but better than no access to your Google Calendar!


Python for Series 60 Goes Final

Finally it happened. It was announced today in the Forum Nokia Python section that a 1.4.0 final version of Python for Series 60 has been released. This is something that I’ve been waiting this for a long time since I’ve been kind of pissed of with the Symbian signing policy. And guess what… this final version of PyS60 has been officially signed by Nokia with all developer certificate capabilities so no more signing is required for Python application in order to access e.g., the GSM location or GPS APIs. How cool is this? Finally, it is humanly possible to develop applications for S60 mobiles. You can expect to see more PyS60 related stuff in this blog once the summer holidays are over. Until then, have a good summer…. I know I will. :D is Stephen Fry proof thanks to caching by WP Super Cache