Wishful Coding

Just before I went to EuroClojure I read about Gershwin, a concatenative programming language based on Clojure.

With a lot of time to think and code on the train between Amsterdam and Berlin and back, I thought: Why make this a language at all?

Using the threading macro you can make a basic stack program as follows.

(def add [stack]
  (conj (drop 2 stack)
        (apply + (take 2))))

(-> () ; the stack
  (conj 1)
  (conj 2)
  add)
;; => (3)

After that, things got interesting. What if I used a queue instead of a stack? What if I used any number of queues and stacks?

I switched to using thrush, wrote a function to generate words from Clojure functions, and did some magic to apply words to multiple stacks.

(defn nth-stack [n f]
  #(apply update-in % [n] f %&))

(thrush [() ()]  ; data stack and return stack
  (lit 2) dup c+ ; add numbers
  (lit 4) c=     ; verify result
  >r)            ; put bool on return stack
;; => [() (true)]

What if I used clojure.core.reducers/fold instead of reduce in thrush? Could I use dataflow programming and continuations to parallelize programs?

I failed at parallelization, but I surely achieved dataflow programming.

The idea is that you have a contiunation-passing-style function for popping of the stack. That function either calls the continuation with the head and tail of the stack, or returns a negative stack which will call the continuation when an item is conj‘d.

(deftype NegStack [popfn stack]
  clojure.lang.ISeq
  ...
  (cons [_ o] (popfn o stack))
  ...)

(defn mpeek [stack cont]
  (if (seq stack)
    (cont (first stack) (next stack))
    (NegStack. cont stack)))

(defn add [stack]
  (mpeek stack (fn [car cdr]
    (mpeek cdr (fn [cdar cddr]
      (conj cddr (+ car cdar)))))))

(thrush () (lit 2) add (lit 2))
;; => (4)

Yes, you can have prefix, infix, postfix and otherfix all you want. But not on a fork-join pool.

I had it all figured out, including concatenating negative stacks. It’s just not conceptually sound. Imagine this program

(thrush () (lit 2) (lit 2) mul (lit 5) (lit 6) add add)
;; => (15)

Using my fork-join-based pthrush thish might be split up as

(mconcat
  (thrush () (lit 2) (lit 2))
  (thrush () mul (lit 5) (lit 6) add add))

The first block produces (2 2), the execution of the second block is more tricky.

1. mul produces a negative stack
2. 5 and 6 are conj‘d onto this stack, resulting in (30)
3. The add’s consume 30 and produce a negative stack
4. This negative stack is concatenated to (2 2), resulting in (34)

The problem is that mul got applied to (5 6) instead of (2 2) because of how the program was split.

It might still be possible to paralellize by analyzing the stack effect of words, but dataflow is definitely not the way to go.

All my experiments so far can be found on Github at pepijndevos/staack

I’ve been working with a number of HTTP API’s using Python recently, and have accumulated some helper functions that I hope you don’t need.

PHP

It might be possible to write a good REST API in PHP, but it’s certainly not easy… or common.

Array parameters

Lets say your brilliant API design requires an object to be passed as a parameter. How do you go about this? Hint: It’s not JSON.

Either you call serialise on the object and send it, or you use this other PHPism that lets you say ?param[field]=val.

In the first case, use the phpserialise package, in the second case, use this handy pre-processing function:

def php_flatten(data):
    res = {}

    def inner_flatten(path, value):
        if isinstance(value, dict):
            for k, v in value.items():
                newpath = "%s[%s]" % (path, k)
                inner_flatten(newpath, v)
        elif isinstance(value, list):
            for k, v in enumerate(value):
                newpath = "%s[%s]" % (path, k)
                inner_flatten(newpath, v)
        else:
            res[path] = value

    for k, v in data.items():
        inner_flatten(k, v)

    return res

XML

I heard Java people like XML, so your API should probably offer it as an alternative to JSON, serialised PHP and INI files.

I was surprised there is no such thing as xml.dumps in Python, so I wrote it. There is no 1-on-1 mapping between dicts and XML, but this a a way to go from Python data to XML.

import lxml.etree as ET

def maybesub(el, tag):
    """
    Create a subelement if a tag was given.
    Otherwise return the current node.
    Used for the root node.
    """
    if tag:
        return ET.SubElement(el, tag)
    else:
        return el

def walk(el, tag, data):
    """
    Recursively add child nodes named tag to el based on data.
    """
    if isinstance(data, dict):
        sub = maybesub(el, tag)
        for key, value in sorted(data.items()):
            walk(sub, key, value)
    elif isinstance(data, list):
        for value in data:
            walk(el, tag, value)
    else:
        sub = maybesub(el, tag)
        sub.text = str(data)

def dumps(data, root="response"):
    root = ET.Element(root)
    walk(root, None, data)
    return ET.tostring(root, pretty_print=True)

INI

That was no joke. Take a look at configparser if you ever encounter this.

The internet is s-l-o-w

So take this handy Redis-based memoization function with you, and twiddle with the TTL.

import redis
from functools import wraps
import pickle
import logging

pool = redis.ConnectionPool()

def memoize(ttl=3600):
    def decorator(f):
        @wraps(f)
        def wrapper(*args, **kwargs):
            try:
                r = redis.StrictRedis(connection_pool=pool)
                # Compute the function signature
                sig = (f.__module__, f.__name__, args, kwargs)
                pargs = pickle.dumps(sig) 
                # Try to get the result
                pres = r.get(pargs)
                if pres:
                    return pickle.loads(pres)
                else:
                    # Or compute and store
                    res = f(*args, **kwargs)
                    r.setex(pargs, ttl, pickle.dumps(res))
                    return res
            except redis.RedisError:
                # Show must go on!
                logging.exception("redis oopsed")
                return f(*args, **kwargs)

        return wrapper
    return decorator

While working on TRC I made a collection of popular IRC clients per platform, only to realise BlackBerry 10 had no real IRC clients. The best one available seemed to be an Android client running in an emulator.

So I asked a friend, collector of phones and owner of Heris IT to help me build a native IRC client for BlackBerry 10 using their Qt based Cascades framework.

A few months later, the first usable version is available for sale on the BlackBerry World.

This is a beta release that costs half of the final product, but also has half the features. We are doing this early release to collect some feedback, so please let us know what is important to you.

There are a few known issues:

• Sending messages is broken on 10.2
• The “Settings” button is a dummy.
• The “Change nick” button is a dummy.
• When the phone is standby the server kicks you of after a few minutes.

Some planned features:

• Richer message display
• Smarter notifications
• Tighter ZNC integration
• Nick autocomplete
• Remember channels
• SASL support

And finally a recommendation:

• Get a bouncer, it makes everything better