Wishful Coding

Didn't you ever wish your computer understood you?

Mature HTTP client for asyncio

I have recently started writing a server for some obscure protocol using Python’s new asyncio module.

The module works great for writing the server, but any external IO the server has to do is tricky. There are simply not so many libraries, and asyncio doesn’t do patching the way gevent does.

The quick and dirty solution is to use run_in_executor to run blocking code in a thread.

The only other game in town for HTTP is aiohttp, which is relatively young and occasionally buggy.

Then I found that Tornado has support for running on the asyncio event loop. Tornado includes a much more mature HTTP client that can optionally use libcurl.

The Tornado HTTP client returns a Future that is similar but not compatible with Futures from asyncio. So in order to use Tornado in a asyncio coroutine, a little wrapper is needed.

from tornado.platform.asyncio import AsyncIOMainLoop
from tornado.httpclient import AsyncHTTPClient
import asyncio

# Tell Tornado to use the asyncio eventloop
AsyncIOMainLoop().install()
# get the loop
loop = asyncio.get_event_loop()
# the Tornado HTTP client
http_client = AsyncHTTPClient()

# wrap the Tornado callback in a asyncio.Future
def aio_fetch(client, url, **kwargs):
    fut = asyncio.Future()
    client.fetch(url, callback=fut.set_result, **kwargs)
    return fut

# enjoy
@asyncio.coroutine
def main():
    print("fetching my site")
    mysite = yield from aio_fetch(http_client, "http://pepijndevos.nl/")
    print("my site said", mysite.reason)
    print("hello httpbin")
    httpbin = yield from aio_fetch(http_client, "http://httpbin.org/get?code=%d" % mysite.code)
    print(httpbin.body.decode())

print(loop.run_until_complete(main()))
Pepijn de Vos

Learning F# as a lesson in feminism

Some of my Hacker School friends tweet a lot about feminism, and sometimes I fail to see why things like this are such a big deal.

But since I started learning F#, it suddenly makes sense. Here’s why: F# is default Windows.

You can use F# on Mac or Linux, but at every step along the way you see screenshots, commands, programs and people that implicitly asume you’re using Visual Studio. You can ask questions in ##fsharp on Freenode, but you’ll need to explain you’re using Mono every single time. You can compile cross-platform programs, but with dll and exe extensions.

With this experience, I started looking at some other things.

Why do I use Linux anyway? Is it superior to Windows? In some ways, but also worse in other ways. The real reason is that outside of the .NET community, everything is thoroughly default Linux. Half of the tools and libraries I use will just not work. On some bigger projects, Windows support might be added as an afterthought, possibly through Cygwin.

Why do I work in English? I’m a Dutch guy. Same answer. Programming is default English, and only the biggest of the biggest projects have multilingual documentation and community. It would be incredibly frustrating to program without all the English resources.

Realising I fit the default programmer pretty accurately, and experiencing the annoyances of not being the default, it starts to be really easy to see why it might be frustrating to be an afterthought as a woman.

If you talk about a random guy on the internet, your probably talk about him like this. Read that again, then install this extension

I just assumed you where using Chrome implicitly. How rude. If you’re not, you can install it with apt-get install chromium.

Free beer meetups seem like a great idea at first, but why not make it free drinks instead? Let me sip my lemonade while you all get drunk.

I hope this post will get you thinking about how to alienate less people of any kind. It’s not about Linux or about woman, it’s about being aware of your assumptions. You can’t always cater to everyone, but you can at least acknowledge the people you’re not catering to.

Pepijn de Vos

Redis Pipelining

I’d like to announce Pypredis, a Python client for Redis that tries to answer the question

How fast can I pump data into Redis?

There are many answers to that question, depending on what your goal and constraints are. The answer that Pypredis is exploring is pipelining and sharding. Let me explain.

The best use case is a few slow and independent commands. For example, a couple of big SINTER commands.

The naive way to do it using redis-py is to just execute the commands one after the other and wait for a reply.

r = redis.StrictRedis()
r.sinter('set1', 'set2')
r.sinter('set3’, 'set4’)
r.sinter('set5’, 'set6’)
r.sinter('set7’, 'set8’)

In addition to the CPU time, you add a lot of latency by waiting for the response every time, so a better solution would be to use a pipeline.

r = redis.StrictRedis()
pl = r.pipeline()
pl.sinter('set1', 'set2')
pl.sinter('set3’, 'set4’)
pl.sinter('set5’, 'set6’)
pl.sinter('set7’, 'set8’)
pl.execute()

That is pretty good, but we can do better in two ways.

First of all, redis-py does not start sending commands until you call execute, wasting valuable time while building up the pipeline. Especially if other work is done in-between Redis commands.

Secondly, Redis is — for better or worse — single-threaded. So while the above pipeline might use 100% CPU on one core, the remaining cores might not be doing very much.

To utilise a multicore machine, sharding might be employed. However, sequentially executing pipelines on multiple Redis servers using redis-py actually performs worse.

pl1.execute() #blocks
pl2.execute() #blocks

The approach that Pypredis takes is to return a Future and send the command in another thread using an event loop.

Thus, pipelining commands in parallel to multiple Redis servers is a matter of not waiting for the result.

eventloop.send_command(conn1, “SINTER”, "set1", "set2")
eventloop.send_command(conn2, “SINTER”, "set3”, "set4”)
eventloop.send_command(conn1, “SINTER”, "set5”, "set6”)
eventloop.send_command(conn2, “SINTER”, "set7”, "set8”)

A very simple benchmark shows that indeed Pypredis is a lot faster on a few big and slow commands, but the extra overhead makes it slower for many small and fast commands.

pypredis ping
1.083333
redis-py ping
0.933333
pypredis sunion
0.42
redis-py sunion
11.736665
Pepijn de Vos