Wishful Coding

Didn't you ever wish your
computer understood you?

Archaic API helpers

I’ve been working with a number of HTTP API’s using Python recently, and have accumulated some helper functions that I hope you don’t need.

PHP

It might be possible to write a good REST API in PHP, but it’s certainly not easy… or common.

Array parameters

Lets say your brilliant API design requires an object to be passed as a parameter. How do you go about this? Hint: It’s not JSON.

Either you call serialise on the object and send it, or you use this other PHPism that lets you say ?param[field]=val.

In the first case, use the phpserialise package, in the second case, use this handy pre-processing function:

def php_flatten(data):
    res = {}

    def inner_flatten(path, value):
        if isinstance(value, dict):
            for k, v in value.items():
                newpath = "%s[%s]" % (path, k)
                inner_flatten(newpath, v)
        elif isinstance(value, list):
            for k, v in enumerate(value):
                newpath = "%s[%s]" % (path, k)
                inner_flatten(newpath, v)
        else:
            res[path] = value

    for k, v in data.items():
        inner_flatten(k, v)

    return res

XML

I heard Java people like XML, so your API should probably offer it as an alternative to JSON, serialised PHP and INI files.

I was surprised there is no such thing as xml.dumps in Python, so I wrote it. There is no 1-on-1 mapping between dicts and XML, but this a a way to go from Python data to XML.

import lxml.etree as ET

def maybesub(el, tag):
    """
    Create a subelement if a tag was given.
    Otherwise return the current node.
    Used for the root node.
    """
    if tag:
        return ET.SubElement(el, tag)
    else:
        return el

def walk(el, tag, data):
    """
    Recursively add child nodes named tag to el based on data.
    """
    if isinstance(data, dict):
        sub = maybesub(el, tag)
        for key, value in sorted(data.items()):
            walk(sub, key, value)
    elif isinstance(data, list):
        for value in data:
            walk(el, tag, value)
    else:
        sub = maybesub(el, tag)
        sub.text = str(data)

def dumps(data, root="response"):
    root = ET.Element(root)
    walk(root, None, data)
    return ET.tostring(root, pretty_print=True)

INI

That was no joke. Take a look at configparser if you ever encounter this.

The internet is s-l-o-w

So take this handy Redis-based memoization function with you, and twiddle with the TTL.

import redis
from functools import wraps
import pickle
import logging

pool = redis.ConnectionPool()

def memoize(ttl=3600):
    def decorator(f):
        @wraps(f)
        def wrapper(*args, **kwargs):
            try:
                r = redis.StrictRedis(connection_pool=pool)
                # Compute the function signature
                sig = (f.__module__, f.__name__, args, kwargs)
                pargs = pickle.dumps(sig) 
                # Try to get the result
                pres = r.get(pargs)
                if pres:
                    return pickle.loads(pres)
                else:
                    # Or compute and store
                    res = f(*args, **kwargs)
                    r.setex(pargs, ttl, pickle.dumps(res))
                    return res
            except redis.RedisError:
                # Show must go on!
                logging.exception("redis oopsed")
                return f(*args, **kwargs)

        return wrapper
    return decorator
Published on

BlackBerry 10 IRC Client

screenshot of TinCan

While working on TRC I made a collection of popular IRC clients per platform, only to realise BlackBerry 10 had no real IRC clients. The best one available seemed to be an Android client running in an emulator.

So I asked a friend, collector of phones and owner of Heris IT to help me build a native IRC client for BlackBerry 10 using their Qt based Cascades framework.

A few months later, the first usable version is available for sale on the BlackBerry World.

This is a beta release that costs half of the final product, but also has half the features. We are doing this early release to collect some feedback, so please let us know what is important to you.

There are a few known issues:

  • Sending messages is broken on 10.2
  • The “Settings” button is a dummy.
  • The “Change nick” button is a dummy.
  • When the phone is standby the server kicks you of after a few minutes.

Some planned features:

  • Richer message display
  • Smarter notifications
  • Tighter ZNC integration
  • Nick autocomplete
  • Remember channels
  • SASL support

And finally a recommendation:

Anatomy of a Channel

I have been trying to implement a CyclicBarrier in core.async, but it took me a while to understand how a channel works.

To support IOC and thread and be fast, the implementation is a lot more verbose and complicated than the bunch of promises I initially expected.

It seems all verbosity and callback hell that can be avoided by using channels is bundled in their implementation. Take this loop from cleanup that removes inactive handlers, can you spot the 2 actual lines of code?

(when-not (.isEmpty takes)           ;bla
  (let [iter (.iterator takes)]      ;bla
    (loop [taker (.next iter)]       ;bla
      (when-not (impl/active? taker) ;inactive?
        (.remove iter))              ;KILL!
      (when (.hasNext iter)          ;bla
        (recur (.next iter))))))     ;bla

But let’s go back to the beginning. The basic unit of a channel is a handler, a glorified callback. It has a commit method with returns the callback, an id for avoiding deadlock and a method to check if it is active.

The latter is used with alts!!, so that after one possible channel has been acted upon, the other handlers turn inactive, but lets stick to a simple <!!.

(defprotocol Handler
  (active? [h] "returns true if has callback. Must work w/o lock")
  (lock-id [h] "a unique id for lock acquisition order, 0 if no lock")
  (commit [h] "commit to fulfilling its end of the transfer, returns cb. Must be called within lock"))

;; no locking, always active.
(defn- fn-handler
  [f]
  (reify
   Lock
   (lock [_])
   (unlock [_])
   
   impl/Handler
   (active? [_] true)
   (lock-id [_] 0)
   (commit [_] f)))

Now let’s look at <!! itself, which is pretty simple. It creates a promise and a callback that delivers the promise. It then calls take! with it. If take! returns nil, deref the promise, otherwise deref the box.

(defn <!!
  "takes a val from port. Will return nil if closed. Will block
  if nothing is available."
  [port]
  (let [p (promise)
        ret (impl/take! port (fn-handler (fn [v] (deliver p v))))]
    (if ret
      @ret
      (deref p))))

Now let’s look at the main course, ManyToManyChannel.

If you look through all the verbosity, take! and put! are pretty similar. There are 3 main code paths in each.

  1. There is someone at the other end. Match up the handlers, call their callbacks and return the value or nil in an IDeref.
  2. There is room in the buffer. Take/put the value in the buffer and return the value or nil in an IDeref.
  3. There is no room. Put the handler in a list and return nil.
(take!
 [this handler]
 (.lock mutex)
 ; remove all inactive handlers
 (cleanup this)
 (let [^Lock handler handler
       ; get the actual callback if it is active
       commit-handler (fn []
                        (.lock handler)
                        (let [take-cb (and (impl/active? handler) (impl/commit handler))]
                          (.unlock handler)
                          take-cb))]
   ; If there are items in the buffer,
   ; take one, put it in a box and return it.
   ; We now have a free spot in the buffer
   ; so we try to find an active put handler.
   ; if we find one, put its value in the buffer
   ; and commit its callback.
   (if (and buf (pos? (count buf)))
     (do
       (if-let [take-cb (commit-handler)]
         (let [val (impl/remove! buf)
               iter (.iterator puts)
               cb (when (.hasNext iter)
                    (loop [[^Lock putter val] (.next iter)]
                      (.lock putter)
                      (let [cb (and (impl/active? putter) (impl/commit putter))]
                        (.unlock putter)
                        (.remove iter)
                        (if cb
                          (do (impl/add! buf val)
                              cb)
                          (when (.hasNext iter)
                            (recur (.next iter)))))))]
           (.unlock mutex)
           (when cb
             (dispatch/run cb))
           (box val))
         (do (.unlock mutex)
             nil)))
     ; There is nu buffer to take from,
     ; so we search for an active putter
     ; for each putter check if both ends are active.
     ; If so, remove it from the list and let the callbacks and value,
     (let [iter (.iterator puts)
           [take-cb put-cb val]
           (when (.hasNext iter)
             (loop [[^Lock putter val] (.next iter)]
               ; funny bit where deadlock is avoided
               (if (< (impl/lock-id handler) (impl/lock-id putter))
                 (do (.lock handler) (.lock putter))
                 (do (.lock putter) (.lock handler)))
               (let [ret (when (and (impl/active? handler) (impl/active? putter))
                           [(impl/commit handler) (impl/commit putter) val])]
                 (.unlock handler)
                 (.unlock putter)
                 (if ret
                   (do
                     (.remove iter)
                     ret)
                   (when-not (impl/active? putter)
                     (.remove iter)
                     (when (.hasNext iter)
                       (recur (.next iter))))))))]
       ; if we found 2 callbacks in the previous step
       ; immediately return the value in a box and commit the putter.
       ; if the channel is closed, return nil in a box.
       ; if the channel is open and there is no matching callback
       ; add the handler to the list of takers
       ; and return nil, without a box.
       (if (and put-cb take-cb)
         (do
           (.unlock mutex)
           (dispatch/run put-cb)
           (box val))
         (if @closed
           (do
             (.unlock mutex)
             (if-let [take-cb (commit-handler)]
               (box nil)
               nil))
           (do
             (.add takes handler)
             (.unlock mutex)
             nil)))))))

I wonder if the code could be made more readable with a few macros for locking and iterators.

Understanding put! is left as a exercise for the reader. Understanding the IOC part is left as an exercise for the Clojure gods.

I will write more about my ideas for barriers and transactions later.