blog.teemu.im

Couple of weeks ago I found out about the Openstack project and I found it immediately to be very interesting. What I’ve been playing around with the most is the object storage part of Openstack called Swift. I’ll show here how you can use Swift with a couple of different libraries. The nice thing about Swift is that it is basically the Rackspace Cloudfiles storage, so the same libraries that work with Cloudfiles, should work with Swift as well. Well, they require some small modifications. But, I’ll show you here two libraries that I know are working already. Of course, you will need a Swift instance running somewhere and instructions on how to setup one you can read the “Swift All In One” document that shows how you can run Swift on a single server.

The first library I’ll show here is the python-cloudfiles. I recommend using the latest one from Github, since the one that you can get for example from Ubuntu repositories does not support Swift and the one you can get from Python Package Index had a bug that made it not work with Swift.

Here I’ll show you how you can connect to your local Swift instance using the authurl parameter and how you can create containers and objects using python-cloudfiles.

from cloudfiles.connection import Connection
 
conn = Connection("test:test", "test", authurl="http://127.0.0.1:11000/v1.0")
 
container = conn.create_container("test")
 
obj = container.create_object("test.txt")
obj.content_type = "text/plain"
obj.write("test")

Pretty straightforward… right?

Next, I’ll show you another library that works with Swift called cferl. It’s a Erlang library for Cloudfiles and I made some simple patches to it to make it work with Swift.

Here’s how you can do the same things as in previous example using cferl.

ibrowse:start().
{ok, Connection} = cferl:connect("test:test", "test", "http://127.0.0.1:11000/v1.0").
 
{ok, Container} = Connection:create_container(<<"test">>).
 
{ok, Object} = Container:create_object(<<"test.txt">>).
ok = Object:write_data(<<"test">>, <<"text/plain">>).

Ok, that’s it. Now you can start playing with Swift and storing petabytes of data in it.

Recently, I found myself in need of a web server that I can use to simulate a behavior of a certain website. I wanted to just copy the output of that website and deliver it using this web server. The problem was that serving static content is naturally way faster than serving dynamic web application, so for my simulation I needed to make the web server wait for a certain period of time before returning the static file. Being a Python fan I decided to use Tornado as the web server. Now, all I needed to do is slow it down.

Ok, I could just simply do this…

time.sleep(0.5)

… to make my server wait half a second before returning, but since Tornado is using asynchronous networking and hence runs in a single thread, this would block all other requests made to my server. Not good…

I need to do this asynchronously. Here is an example on how this can be done with Tornado without blocking.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

import time
 
import tornado.web
import tornado.ioloop
 
class RequestHandler(tornado.web.RequestHandler):
 
    def _finish_request(self):
        self.finish()
 
    @tornado.web.asynchronous
    def get(self):
        ioloop = tornado.ioloop.IOLoop.instance()
        ioloop.add_timeout(time.time() + 0.5, self._finish_request)
        self.add_header("Content-Type", "text/plain")
        self.write("Hello, world")

So, this is an example of an asynchronous Tornado request handler that will wait for half a second before returning and will not block other requests while doing that. Here we are using decorator tornado.web.asynchronous on line 11 to tell Tornado that this request should not be returned immediately and we need to call tornado.web.RequestHandler.finish() on our own. The timeout is implemented by calling tornado.ioloop.IOLoop.add_timeout() method which is given a callback method that will finish the request.

Now, the problem with this is that, if I need other request handlers to do the same thing, I would need to copy paste this peace of code all over the place. And I don’t like that. We can do this bit more elegantly by using Python decorators. By writing a decorator we can avoid duplicating the same code to every request handler. This is how the same example will look using a decorator.

1
2
3
4
5
6
7
8
9

import tornado.web
import tornado.decorators
 
class RequestHandler(tornado.web.RequestHandler):
 
    @tornado.decorators.wait_for(milliseconds=500)                                               
    def get(self):                                                            
        self.set_header("Content-Type", "text/plain")                         
        self.write("Hello, world")

Looks nice and clean. Doesn’t it? Well, the complex part has been moved now to the decorator tornado.decorators.wait_for. Let’s look now how we can implement that.

tornado/decorators.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

import time
 
from functools import partial
 
from tornado.web import asynchronous
from tornado.ioloop import IOLoop
 
def wait_for(milliseconds=0):
    def _finish_request(request, start_time):
        timeout = (time.time() - start_time) * 1000.0
	request.write("\n\nServer waited for %.3f ms" % timeout)
        request.finish()
    def _decorator(func):
	func = asynchronous(func)
        def _wrapper(*args, **kwargs):
            ioloop = IOLoop.instance()
            callback = partial(_finish_request, args[0], time.time())
            ioloop.add_timeout(time.time() + milliseconds / 1000.0, callback)
            func(*args, **kwargs)
	return _wrapper
    return _decorator

Here I have implemented the decorator wait_for that takes the number of milliseconds to wait as a parameter. Let’s start from line 13 where the actual decorator is implemented. Decorator’s parameter is always the function that is being decorated. You can think of this…

@my_decorator
def my_function():
    // do something

…being same as this…

def my_function():
    // do something
my_function = my_decorator(my_function)

Now, on line 14 decorate the function with Tornado’s tornado.web.asynchronous decorator. Just like we did in the first example. So that our request does not return before we call tornado.web.RequestHandler.finish(). Then on line 15 we write a wrapper method that adds the timeout and callback to tornado.ioloop.IOLoop and after that calls the original function that we are decorating. The trickiest part here probably is that we need to give our callback function some parameters and Tornado’s add_timeout method only takes the callback function as the parameter. For that we use Python’s functools.partial to generate the callback and give some parameters to it on line 17.

To conclude the blog post here is a complete example of a script that you can test this with. You need to create the tornado/decorators.py using the code above for this to work.

http_server.py

import time
 
import tornado.httpserver
import tornado.ioloop
import tornado.web
 
from tornado.decorators import wait_for
 
 
class MainHandler(tornado.web.RequestHandler):
 
    @wait_for(milliseconds=500)
    def get(self):
        self.set_header("Content-Type", "text/plain")
        self.write("Hello, world")
 
application = tornado.web.Application([(r"/", MainHandler)])
 
if __name__ == "__main__":
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(8080)
    tornado.ioloop.IOLoop.instance().start()

If everything goes right your server should return something like this…

Hello, world
 
Server waited for 500.371 ms

I’ve been learning how to use zc.buildout in my Python projects. It seems to be used mostly by the Zope and the Plone communities, but I feel that it might be a useful tool to be used in any Python project. I do lots of work with Twisted and I wanted to try zc.buildout with a Twisted project. It wasn’t trivial and I wasn’t able to find any good examples, so I thought I might as well document my experiences here.

Let’s start this exercise by setting a new virtual Python environment using virtualenv. It gives you a nice constrained Python environment that does not mess up your system. Easiest way to get virtualenv on your system is by using setuptools. When you have setuptools installed you can get virtualenv by typing:

$ easy_install virtualenv

Now that you have virtualenv installed it’s time to create the virtual environment. In this example I will create the environment in a directory called twistedenv. You can place it anywhere you want in your system.

$ mkdir twistedenv
$ virtualenv --no-site-packages twistedenv/
New python executable in twistedenv/bin/python
Please make sure you remove any previous custom paths from your /Users/teemu/.pydistutils.cfg file.
Installing setuptools............done.

Now that the virtual environment is setup you can activate it by typing:

$ cd twistedenv
$ source bin/activate

This will activate the new Python interpreter that was installed in twistedenv/bin/python.

Ok, now we can get to the actual topic, that is how to use zc.buildout with a Twisted project. What you will need is a bootstrap script that will install zc.buildout inside your environment and a buildout configuration file. Let’s start with the configuration file, since we need that before running the bootstrap script. In zc.buildout the configuration file is called buildout.cfg. It will tell the buildout system, what packages are needed and setup your development system in a blink of an eye. I’ll start with an example that will install Twisted and some dependencies and a sample Twisted application that is under development. Here’s the buildout.cfg that will do this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

[buildout]
parts =
    depends
    twisted
    twisteds
develop = ./src
 
[versions]
Twisted = 8.2.0
pyOpenSSL = 0.8
pyserial = 2.4
pycrypto = 2.0.1
 
[depends]
recipe = minitage.recipe:egg
eggs =
    pyOpenSSL
    pyserial
    pycrypto
 
[twisted]
recipe = minitage.recipe:egg
eggs = Twisted
 
[twisteds]
recipe = minitage.recipe:scripts
interpreter = twistedpy
extra-paths = ${buildout:directory}/src
eggs =
  ${twisted:eggs}
  ${depends:eggs}

I’ll briefly explain what it does. Section buildout is the first thing to put in the config file. It lists what parts of the configuration file are used and the location of the code that is being developed. The section called depends installs the dependencies for Twisted using recipe minitage.recipe. Section twisted naturally installs Twisted, but it does not install the scripts that are installed to the bin directory. That is what the section twisteds is for. By the way, you are free to choose the names of the sections or parts except for the buildout section. In the twisteds section we also define the variable interpreter to create us a new Python interpreter that will be used when running these scripts. For this interpreter we define which eggs are included in the interpreter path and using extra-paths variable we can include our development code there also. It is useful that we can keep our own code separate from 3rd party code while developing it. Section versions can be used to specify which versions of 3rd party libraries we want to use. If you don’t specify a version, a newest version available is used.

Now that we have the buildout.cfg file ready, we can bootsrap the buildout. Download the bootstrap script from this URL and put it in the twistedenv directory. Before running the bootstrap script you can make sure that you are using the Python interpreter from the virtual environment to run it instead of the syste Python. Do that by typing:

$ which python
/Users/teemu/Programming/twistedenv/bin/python

As you can see, for me it shows that I’m using the Python interpreter from inside the twistedenv where we created the virtual environment. That is good. Naturally, for you the complete path is something else. Then we can run the bootstrap script by typing:

$ python bootstrap.py
Creating directory '/Users/teemu/Programming/twistedenv/parts'.
Creating directory '/Users/teemu/Programming/twistedenv/eggs'.
Creating directory '/Users/teemu/Programming/twistedenv/develop-eggs'.
Generated script '/Users/teemu/Programming/twistedenv/bin/buildout'.

As you can see from the output the script created couple of directories and generated a script called buildout to the bin directory. Now our buildout environment is almost ready. The only thing is missing is that our ./src directory does not contain any code. You will see an error if you now try to run bin/buildout.

Let’s create a quick sample Twisted application to demonstrate a more complete buildout environment. This is what we need. Create the required directories for our code:

mkdir -p src/sample
mkdir -p src/twisted/plugins
touch src/sample/__init__.py

Then we need the following files:

src/sample/tap.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

from twisted.application import internet
from twisted.web import server
from twisted.web.resource import Resource
from twisted.python import usage
 
class SimpleResource(Resource):
    isLeaf = True
    def render_GET(self, request):
        return "<h1>It Works!</h1>"
 
 
class Options(usage.Options):
    pass
 
 
def makeService(config):
    site = server.Site(SimpleResource())
    return internet.TCPServer(8080, site)

src/twisted/plugins/sample.py:

1
2
3
4
5
6
7
8

from twisted.application.service import ServiceMaker
 
Sample = ServiceMaker(
    "Sample",
    "sample.tap",
    "Twisted sample application",
    "sample"
)

And last, but not least, src/setup.py:

1
2
3
4
5
6
7
8
9
10
11
12

from setuptools import setup
 
setup(name='sample',
      version='0.0.1',
      description='Sample Twisted application',
      author='Teemu Harju',
      author_email='teemu.harju@gmail.com',
      url='http://blog.teemu.im',
      packages=['sample'],
      package_data={'twisted.plugins': ['twisted/plugins/sample.py']},
      zip_safe=False
)

These files will create a Twisted application called sample that runs a webserver on port 8080 that responds “It works!” when the root URL is accessed.

Now, run:

$ bin/buildout
...

I’ve not put the output to the example since there is so much of it. But after the buildout is ready and if everything worked fine you can test that your environment is ready by running the bin/twistd script like this:

$ bin/twistd --help

The output should contain our sample application and you can run it by typing:

$ bin/twistd -n sample

Now you can start developing your Twisted application and test it in your safe and restricted environment using virtualenv and buildout. By the way, if you want to get out of your virtual Python environment you can do that simply by typing:

$ deactivate

It would be probably good to include some kind of a test runner in the buildout script to make sure that your buildout is not broken. How that can be done is probably a blog post topic of it’s own. That’s all for now. Have fun with buildout and Twisted. Feel free to ask in the comments if this didn’t work for you for some reason.

Python for Nokia’s Series 60 platform has been around for four years now and to be honest, not much has happened on that front during the past year. However, suddenly on 24th of December Nokia releases a version 1.9.0 of the Python for Series 60 that is a major rewrite of the whole thing and comes now with Python 2.5 version of the core language. As usual, the odd-number version branch means that the release is for testing purposes only and the even-numbered 2.0 version branch should be released once it becomes more stable. Blog at Croozeus.com has done pretty nice wrap up of the new release and here are my thoughts.

First of all I should mention that as I write this blog entry I’ve not yet had time to test the new Python for Series 60 release. So I apologize beforehand for all false comments I’m about to make.

It seems that they are giving a lot more focus on the ease of development in this new release. Which is a really good thing I must say. What pleases me the most is that they are planning on improving the Python runtime deployment so that the end-user or the guy that is installing Python applications should not have to worry too much about having or not having the Python runtime installed on his phone. Definitely a good thing. The new release includes a packaging tool that is not part of the S60 SDK and is basically ensymble with added GUI. Ensymble is an excellent tool and it is very nice to see it included in the official PyS60 release.

Like mentioned already, the new release includes the version 2.5 of the Python interpreter and most of it’s standard libraries. Ok, the word “most” does not sound good here. I’ll focus first on the good things. First, the new release has Expat XML parser in it. Definitely a good thing since XML is something that pretty much every application out there uses and so far people have had to make ugly regexp XML parsers or use 3rd party package of expat to be able to parse XML in their PyS60 applications. However, I would say that including JSON parser as the official Python 2.6 release did lately, would probably be a good idea too.

Also, inclusion of asyncore and more compliant socket module sounds nice. I can’t wait to try Twisted on PyS60 and see how it works. One could do some crazy things with that on a mobile phone.

Then to the things that are not included from the standard Python 2.5 libraries. Now, I don’t have too much information on this since, like said before, I haven’t tried the new release yet. However, to my disappointment I noticed that sqlite3 is not included. SQLite should probably be a platform component in S60 because it is kind of becoming a de facto standard in mobile platforms since both iPhone and Android platforms use it. I don’t know what is the equivalent in S60 or does such exist but having some kind of storage other than just plain text file for PyS60 applications would be extremely nice thing to have.

Ok, I have to try out the new PyS60 release soon. I’m hoping that I will be pleasantly surprised. I’ll definitely write more about it later.