Saturday, January 7, 2012

Coding: Python proxy support, thread-safety, and SocksiPy

Depending on the task, HTTP requests in Polly can be made by four different modules: pycurl, python-httplib2, python-oauth2, and urllib. The first has built-in proxy support and to the others I add proxy support with a little bit of monkey-patching: I replace the standard socket.socket class by a proxy-aware subclass from the SocksiPy module.

This is massively thread-unsafe, of course, because there are different threads creating instances of socket.socket all the time. This issue is present in the current version of Polly because I admittedly implemented proxy support in a little bit of a rush. I decided to get rid of it now, during the rewritings and refactorings I'm doing for the GTK3 port.

Now all HTTP requests are wrapped by methods of a proxy singleton, responsible for ensuring thread-safety. This is done via a reader/writer lock instead of a simple lock to allow multiple requests simultaneously. I also use this wrapping to replace the plethora of different exceptions given by the different modules by simpler Polly exceptions.

At least for now, I'm happy with this approach. One of the things I like about it is that the syntax makes sense. Now instead of things like

    http_client = httplib2.Http()
    http_client.request(url)

I have things like

    proxy.http_request(url)

which makes syntactically clear that the request is being handled, and possibly rerouted, by a proxy manager.

Thoughts, everyone? Simpler alternatives?

No comments:

Post a Comment