Wishful Coding

Fast PyCAM on Mac using Homebrew

2012-04-23T00:00:00-07:00

Ever since I started working on my CNC mill, I have been looking for decent software to generate G-code for it.

I wrote a G-code interpreter for my machine, after I found out LinuxCNC doesn’t support my type of LPT stepper driver, but finding good software to turn a 3D model into G-code turned out to be hard.

I went down this list, but most of them where hard to install or impossible to configure. In 2D I had some luck with the Inkscape plugin, but I was lost for 3D.

With the mill in a working state, and nothing to mill, I decided to push forward with PyCAM. I skipped PyCAM previously, because of its dependencies, which weren’t all available in Homebrew.

Some experimentation revealed a few important things:

It runs okay on my Ubuntu netbook.
Toolpath calculations take a looong time.
Its CLI runs without OpenGL and GTK+
It supports multiple and distributed processors.

After I cancelled the Ubuntu calculations, I ran it headless on my Mac. While somewhat faster, it still took a long time to generate G-code.

The real speedup came later, when I found PyCAM supports Psyco. But rather than messing with Psyco, I followed the advice on the Psyco homepage, and found the headless PyCAM runs great, and very fast, on PyPy.

Psyco is unmaintained and dead. Please look at PyPy for the state-of-the-art in JIT compilers for Python.

After successfully running PyCAM headless, I finally put my teeth in the dependencies. Installation steps using Homebrew follow:

Download PyCAM.
Apply my patch to make sure it finds pygtk.
brew install pygtk
sudo easy_install PyOpenGL
Wait for my pygtkglext Formula to be merged, or get it from my branch.
brew install pygtkglext or brew install https://raw.github.com/pepijndevos/homebrew/master/Library/Formula/pygtkglext.rb

Now you should be able to just run ./pycam to see the GUI pop up. However, I use the following 2 commands to run a server on PyPy, for extra speed.

pypy pycam --start-server-only --server-auth-key=fietspomp --number-of-processes=4

python pycam --enable-server --remote-server=localhost --server-auth-key=fietspomp --number-of-processes=0

I’ll blog about the mill itself later. This morning I tried to mill a small sample project, but the mill is very inaccurate and jammed after a few layers.

Crazy Seq

2012-04-17T00:00:00-07:00

crazy-seq, the solution to all your head-holding induced memory problems.

(defn crazy-seq [head f]
  (reify
    clojure.lang.ISeq
    (first [_] head)
    (more [_] (crazy-seq (f head) f))
    (next [this] (rest this))
    (seq [this] this)
    (equiv [_ _] false)))

(def s (crazy-seq 0 inc)) ; hold head
(dorun (take 100000000 s)) ; low fat
(nth s 10) ; recomputes
;=> 10

A normal lazy seq is made up of a car and a function that returns the cdr, which is then cached.

This means that if you store the first item and walk to the millionth, it will keep all these million items in memory.

What crazy-seq does, is that it simply does not cache the value. So you can walk to the millionth item, and it will only store the first and the current, the others are GC’d.

This means that walking twice will cost more CPU cycles, and it means that you get in trouble if the fn is not side-effect free.

Have fun.

Sciento CS-113

2012-04-09T00:00:00-07:00

After my success with the Cyber arm, I had to try my luck at the Sciento one.

This time I was out of luck with Dutch or English documentation, but my brother found a Spanish PDF, from which I was able to get the needed information using Google Translate.

Unlike the Cyber, the Sciento is controlled by sending actual ASCII commands, rather than stepping the motors directly.

It is a lot smarter than the other arm, as it can remember its position. A list of commands:

”Z” Sets the reference point
”N” Go to the reference position
”C”/”O” Close/open the claw
“M100,0,-250,0,0,0” Move the motors for the specified steps.
“H7” Save the current position in the specified register(1-100)
“P5,500,200,-5,0,0,0” Save the specified position in the specified register(first number, 1-100)
“G1” Move to the location in the specified register
“S5” Set the motor speed(1-5)
“D1” Wait for x seconds. Useless?
“L1” Check the limits of the motors. Recommended for testing, off by default.

Note that numbers are represented as ASCII text and that commands are upper-case. All commands are terminated by a carriage return.

A downside to this arm is that in comparison to the one-byte commands for the Cyber, it takes ages to send an ASCII command across, making this arm unsuitable for smooth control with a gamepad.

I programmed a coarse and ugly controller, with buttons for storing positions and moving to them. In the video I’m just stepping through the motions I stored earlier.

The controller is a real mess, based on the Cyber code, and not very interesting.

The code for the arm itself is nice though. Note the use of the actual strobe line, as opposed to the strobe bit in the Cyber code.

import parallel
from time import sleep

p = parallel.Parallel()

def write(s):
    bs = map(ord, s)
    bs.append(13)
    for b in bs:
        p.setData(b)
        p.setDataStrobe(1)
        p.setDataStrobe(0)
        sleep(0.001)

def command(cmd, *params):
    write(cmd+",".join(map(str, params)))

def move(m1=0, m2=0, m3=0, m4=0, m5=0, m6=0):
    "Move a number of steps"
    command("M", m1, m2, m3, m4, m5, m6)

def here(n):
    "Save current position in slot n 1-100"
    command("H", n)

def position(n, m1, m2, m3, m4, m5, m6):
    "Save the specified position in slot n 1-100"
    command("P", n, m1, m2, m3, m4, m5, m6)

def zero():
    "Set the reference point"
    command("Z")

def goto(n):
    "Go to position in slot n 1-100"
    command("G", n)

def home():
    "Return to the reference"
    command("N")

def open():
    "open the claw"
    command("O")

def close():
    "Close the claw"
    command("C")

def speed(n):
    "Set the speed 1-5"
    command("S", n)

def delay(n):
    "Wait for n seconds"
    command("D", n)

def limit(b):
    "enable/disable limits"
    commands("L", b)

Cyber 310

2012-04-07T00:00:00-07:00

We got 2 robotic arms from a school lab that did them away. After we got them, they had been accumulating dust for ages in our shed, with one failed attempt at making them work in between.

This was my second attempt. As you can imagine, there aren’t nearly as many people writing about robotic arms as there are writing about Clojure or Node.js, so documentation is sparse.

The most useful (English!) resource I found was this collection, which has original manual pages he got from a museum(!). I contacted him for a couple more pages, which where very helpful.

I can only say it was very hard to find out how easy it was to control this arm.

Basically, you have one pin for every motor, and 2 control pins. To send a command you just set the motor pins, and then turn the corresponding toggle pin on and off.

To reverse motor 2:

01000000
01000001
01000000

To turn motor 3 one step:

00100000
00100010
00100000

Once I figured that out, I probably spent most of my time installing stuff on Windows, then installing Ubuntu and installing there.

To make PyParallel work, I had to rmmod lp and to run as root.

The code, might you have a dusty Cyber 310 in your shed:

import parallel
from time import sleep

base = 1
shoulder = 2
elbow = 4
leftwrist = 8
rightwrist = 16
grip = 32
prf = 64
strobe = 128

p = parallel.Parallel()

def direction(m):
    p.setData(m)
    p.setData(m+strobe)
    p.setData(m)

def step(m):
    p.setData(m)
    p.setData(m+prf)

def move(steps, *motors):
    dir_pattern = abs(sum([m for m in motors if m < 0]))
    step_pattern = sum(map(abs, motors))
    
    direction(dir_pattern)
    for i in range(steps):
        step(step_pattern)
        sleep(0.01)

import os, pygame, cyber
from time import sleep

os.environ["SDL_VIDEODRIVER"] = "dummy"
pygame.display.init()
pygame.init()
j = pygame.joystick.Joystick(1)
j.init()

#axes = {cyber.base:0, cyber.shoulder:1, cyber.elbow:2, cyber.leftwrist:3, cyber.rightwrist:4, cyber.grip:5}
axes = {cyber.base:0, -cyber.shoulder:1, -cyber.elbow:4}

def dead_int(real, deadzone=0.5):
    if real > deadzone:
        return 1
    elif real < -deadzone:
        return -1
    else:
        return 0

print 'Initialized Joystick : %s' % j.get_name()
try:
    while True:
        pygame.event.pump()
        #sleep(0.01)
        movements = [k*dead_int(j.get_axis(v)) for k, v in axes.iteritems()]

	x, y = j.get_hat(0)
	if x == 0 and y != 0:
		wr = (cyber.leftwrist+cyber.rightwrist) * y
		movements.append(wr)
	elif x != 0 and y ==0:
		movements.append(cyber.leftwrist * x)
		movements.append(cyber.rightwrist * -x)

	if j.get_button(0):
		movements.append(cyber.grip)
	elif j.get_button(1):
		movements.append(-cyber.grip)	

        cyber.move(1, *movements)
        
except KeyboardInterrupt:
    j.quit()

Lat/Lon to Meter

2012-03-26T00:00:00-07:00

Easy, 1 Latitude is the circumference of the earth divided by 360: 111 km

1 Longitude depends on your Latitude, so… copy-paste

math.radians(6378137.0 * math.cos(math.radians(lat)))

Now, I know this all breaks down horribly when you consider large distances or require high precision(it also upsets mathematicians), but I don’t care(much).

All I¹ am interested in is, if I stand at 42° Latitude and I walk North 10 meter, about what is my new Latitude?

And probably half the googlers and questioners on Stack Overflow.
↩

Thoughts about generating stripboard layouts

2012-03-13T00:00:00-07:00

I am at the moment working on a servo controller for the LEGO NXT based on the PICAXE 20X2¹.

When you design a circuit, you usually use schematics like these:

However, when you actually want to solder the circuit, you either get a PCB, or you use stripboard, which looks like this:

It is basically a board full of holes, with horizontal copper lanes connecting rows of holes.

All you have to do is insert the pieces so that things that should connect are on the same lane. To make things even easier, you can break lanes, or join lanes with a wire link.

Wait a moment… If we lay out the components like on the schematic, nothing will connect correctly!

Right, this brings me to the meat of this post. How can we automate this, to get an optimal and flawless translation from what is basically a graph, to a set of lanes?

I think the best tool for this job is constraint logic programming, of which cKanren is a neat implementation on top of MiniKanren.

Our program is somewhat related to the N-Queens problem, for which a solution is implemented in section 4.2 in the cKanren paper.

In he N-Queens problem, N queens must be arranged on an NxN board, so that no queen attacks another queen. This is implemented using all-difffd², which forces the queens to all be on different rows, columns and diagonals.

Likewise, we can require our components to be all-difffd when it comes to lanes, but our situation is a little more complicated.

Components have multiple points which have constraints of themselves³
Lanes need to be broken under uC’s and might be broken for compactness.
Lanes might need to be joined with a wire link.

Let me state right away that I have not solved all of these complications, but I’ll let you in on my thought process.

For starters, lets define lanes as a finite domain of numbers. This allows us to use all-difffd, but gets us in trouble when we need to express broken lines.

(fresh (lane)
  (infd lane (range 0 50)))

In this simplified version, I also just defined components as a list of points they are connected to. Components that are connected, use the same fresh variable. This also gets us into trouble when we insert wire links.

> (run 1 (q)
    (fresh (l1 l2 l3 l4)
      (infd l1 l2 l3 l4 (range 0 10))
      (== q (list (list 'r1 l1 l2) (list 'r2 l3 l4) (list 'fet l1 l3 l2)))
      (distinctfd (list l1 l2 l3 l4))))
(((r1 0 1) (r2 2 3) (fet 0 2 1)))

The result is rather boring, as it just echoes back what is connected to what. It does get a little more interesting when we add constraints.

(define lanes (range 0 50))

(define lane
  (lambda (l)
    (infd l lanes)))

(define resistor
  (lambda (l1 l2 r)
    (fresh (l2+)
      (lane l1)
      (lane l2)
      (lane l2+)
      (plusfd l2 2 l2+)
      (<fd l2+ l1)
      (== r (list 'r l1 l2)))))

Now we can get some valid resistor positions, and incorporate their constraints into the full application.

> (run 5 (q) (fresh (l1 l2) (resistor l1 l2 q)))
((r 3 0) (r 4 0) (r 4 1) (r 5 0) (r 5 1))
> (run 1 (q)
    (fresh (l1 l2 l3 l4 r1 r2)
      (infd l1 l2 l3 l4 lanes)
      (resistor l1 l2 r1)
      (resistor l3 l4 r2)
      (== q (list r1 r2 (list 'fet l1 l3 l2)))
      (distinctfd (list l1 l2 l3 l4))))
(((r 3 0) (r 4 1) (fet 3 4 0)))

How to proceed from here? I don’t know. I think we need to find a different representation for lanes, to support breaking and joining them.

I started writing a goal called connectedo, to consider 2 lanes with a wire link between them as connected, but this is unfinished.

COMPANY <3 CAPS
↩
Called distinctfd in the code on Github.
↩
Such as “a resistor must span at least 4 holes” and “a uC has 2 continuous columns of pins with 2 columns between them”
↩

Hidden Pages in Wordpress

2012-01-02T00:00:00-08:00

More often that not, I need to hide pages from the menu and search pages, when setting up a Wordpress blog.

Usually for download and thank-you pages, or other stuff that is only accessible via a direct link.

By default, Wordpress includes a visibility settings, which has 3 values.

Public – anyone can see it.
Protected – Anyone with the password can see it.
Private – Only you can see it

None of them is useful for my goals, however, some digging revealed that a small change would allow you to link directly to password protected pages.

Copy wp-pass.php to unlock.php, and make the changes below.

--- wp-pass.php	2011-09-19 05:17:26.000000000 +0200
+++ unlock.php	2012-01-02 17:38:47.000000000 +0100
@@ -10,8 +10,8 @@
 require( dirname(__FILE__) . '/wp-load.php');
 
 // 10 days
-setcookie('wp-postpass_' . COOKIEHASH, stripslashes( $_POST['post_password'] ), time() + 864000, COOKIEPATH);
+setcookie('wp-postpass_' . COOKIEHASH, stripslashes( $_GET['post_password'] ), time() + 864000, COOKIEPATH);
 
-wp_safe_redirect(wp_get_referer());
+wp_safe_redirect($_GET['return']);
 exit;
 ?>

Now you can simply link to http://example.com/unlock.php?post_password=foobar&return=/thank-you/

Lego NXT With a Gamepad

2011-11-26T00:00:00-08:00

I built this robot a while back, but I can’t stand the software that comes with the NXT, my own compiler isn’t ready yet, and any other remote controls I found didn’t work particularly well, if at all.

If you want it done right, you have to do it yourself – Bernd Paysan

So I threw together PyGame and nxt-python, and wrote my own little application to control the NXT. Notice that it looks very similar to this PyMouse code

I have no building plans for the robot, it’s the standard wheel base with my own pincer. I do have the code:

import pygame
from nxt import locator, motor
from time import sleep

# edit this to reflect your joystick axis and buttons
axis = {'x':0, 'y':1, 'x*':3, 'y*':5}

b = locator.find_one_brick()

left = motor.Motor(b, motor.PORT_B)
right = motor.Motor(b, motor.PORT_A)
action = motor.Motor(b, motor.PORT_C)

closed = False

def limit(nr):
    if nr > 50 or nr < -50:
        return min(127, max(-128, nr))
    else:
        return 0

def move(fwd=0, turn=0):
    lp = int((fwd - turn) * -100)
    rp = int((fwd + turn) * -100)
    left.run(limit(lp))
    right.run(limit(rp))

def pincer(button):
    global closed
    try:
        if button and not closed:
            closed = True
            action.turn(-40, 70, emulate=False)
        elif not button and closed:
            closed = False
            action.turn(30, 70, emulate=False, brake=False)
    except motor.BlockedException:
        print action.get_tacho()

pygame.init()
j = pygame.joystick.Joystick(0) # first joystick
j.init()
print 'Initialized Joystick : %s' % j.get_name()
try:
    while True:
        pygame.event.pump()
        sleep(0.1)
        
        # get_axis returns a value between -1 and 1
        move(j.get_axis(axis['y']), j.get_axis(axis['x']))
        pincer(j.get_button(0))
        
except KeyboardInterrupt:
    j.quit()

Twisted on Dotcloud

2011-11-22T00:00:00-08:00

A while back, I wrote a Twisted email server that would sit between you and Twitter and let you reed and send tweets via email.

With the advent of free cloud hosting, I thought it’d be fun to put it online. I’ll share with you how it’s done, for those of you who are looking for a place to put your Twisted app.

The major problem with these easy hosts is that they assume a simple WSGI app or a background worker, but then the background worker is not accessible from the outside.

First I tried Heroku, then Dotcloud, and finally a couple of others, and just when I had given up, @solomonstre came in:

＠pepijndevos dotcloud has beta support for arbitrary tcp/udp ports. Want to try? :)

After some fruitless tries, he shared this repo, which contains scaffolding for a web-accessible Python worker.

The core parts of this thing are dotcloud.yml, where you define a setup script, the ports you want, and the command to run your app.

worker:
  type: custom
  buildscript: builder
  ports:
    smpt: tcp
    pop3: tcp
  process: ~/run
  approot: twemail
  environment:
    PYTHON_VERSION: 2.7

builder contains a whole lot of pip/virtualenv code, while run contains something like twistd -ny yourapp.tac.

Study Group

2011-11-10T00:00:00-08:00

I’m starting a study group in Loenen (Gelderland), and you are invited. The primary focus of the group will be on programming and computer science.

After having been to several learning institutions, I found out it’s not something that works for me. I like to study at my own pace, whatever it is that I want to study.

However, I found that collaborating and peer learning make things a lot more fun. Therefore I want to organize a study group, to learn and create together.

Do you want to join? To work out the details, and to facilitate further discussion, I have created a wiki and a mailing list.

I sketched out a few things that’d work for me, but if you feel like “I want to join, but…”, speak up, nothing is set in stone.

Vacuum Balloon

2011-11-09T00:00:00-08:00

You know these helium balloons? They float because helium is lighter, less dense then air. You know what is even lighter than air? No air at all!

I came across the idea on this website, where he notes

I haven’t done any math, but I assume this isn’t feasible at the present time.

I have done some math, or rather, asked Wolfram Alpha.

So, first of all, how heavy is a cubic meter of air anyaway?

Ok, so our balloon can weigh just over a kilo and still float. But what is the surface area of a balloon of that size?

Almost five square meter. So with one kilo of matter, we need to make a balloon of nearly 5 square meter surface area.

Aww, not looking good. If we use aluminum¹, we can make a shell of a tenth of a millimeter thick. This shell will have to resist 1 bar, which equals 10 newton per square centimeter.

I just picked this up from Wikipedia, but presumably this means the stress on the shell will be 3162 newton per square millimeter.

Now, remember that this is a vacuum, so while a pressurized balloon retains it shape because of the pressure, the biggest problem with our vacuum balloon is the buckling force.

The smallest dent, a gust of wind, or even gravity itself, will cause the balloon to implode. I asked my dad², and he asked his engineer, how to calculate this force. Third order magic.

We should be using carbon or some composite probably, but it doesn’t really get much beyond 1mm anyway.
↩
He’s an architect.
↩

Google Reader on Kindle

2011-11-04T00:00:00-07:00

I recently bought a Kindle, and I love it! It takes some effort to convert scanned PDF books, but other than that, you just email stuff to Amazon, and it gets delivered.

For long articles, I found that Instapaper has a ‘Read Later’ bookmarklet, that can be configured to send a digest to my Kindle.

The only hard part was Google Reader, so I thought I’d share that with you. Reader has a ‘Send To’ feature that can send articles to Kindle, but you have to go trough all the articles manually.

We’ll have to involve an extra step, called Instascriber. Instascriber lets you automatically send up to three feeds to Instapaper. Not enough for all my feeds, but luckily Reader has an RSS feed hidden deep down.

The broad approach is to collect your feeds in a public place in Google reader, so you can put the RSS feed in Instascriber, which submits them to Instapaper, which in turn emails them to Amazon, which then finally delivers the issue on my Kindle.

Finding a public RSS feed in Reader is a bit tricky. Information I found about public folders and tags seemed outdated, with the new interface. What you need to do is, go to ‘Browse for stuff’, and create a bundle there.

When you’re done, you need to figure out how to get the feed. I did this by clicking ‘Add a link’, to get to the public page. Depending on your browser, you might already see the RSS icon there. For FireFox, it can be found by pressing CMD+I and going to the ‘Feeds’ tab.

For reference, this is what my feed looks like http://www.google.com/reader/public/atom/user%2F09723460687514136094%2Fbundle%2Fkindle

From here on, it’s standard form-filling and do-as-you-are-told business.

First register at Instapaper, and set up Kindle delivery according to their instructions

Finally, register at Instascriber, enter the Reader feed, and enjoy reading!

Clojure Wiki Gitorial

2011-10-23T00:00:00-07:00

At the last Amsterdam Clojurians meetup, I gave a presentation with Hubert Iwaniuk, in which I wrote a wiki in 15 minutes, and Hubert explained what I was doing.

Unfortunately no video is available, but I made a Gitorial out of it to show you what I did. Typos included.

commit 9d2f91089dadbe795a16da9d84cb0da33cea9a37

Welcome to this gitorial. I just created an empty Leiningen project with lein new wiki and initiated git with git init

commit e2156be6cbfc421e384ebb016699497501395131

I’m using Ring for serving my application, and Moustache for routing.

commit 1898d45e54d1ae6045742f8e5db3b8fbc1fb7141

I defined a router that will at this point match an empty address and return “hello world”. The -main function can be run with lein run -m wiki.core.

Browsing to http://localhost:8080/ now should display “hello world”.

commit 5e68a78f30586ad5313072201df25e5b1eea76ac

A lot is going on here, form top to bottom:

I imported some Ring middleware. These modify the request and response on the fly. Note that wrap-reload and wrap-stacktrace are for debuging only.

I added the middleware to the Moustache app.

I added routes. The first route matches and WikiLink and binds it to title. The second one redirects all other links to the MainPage.

The #’ syntax is for getting the var instead of the value, to make reloading work.

At this point, visiting the same url should redirect to /MainPage and display “hello tester”

Changing this text does not require a server restart, so we can keep it running from now on.

commit ae78c9d640420d0154b22c66df581fa684b8aa48

I lied, to add a new dependency, you need to restart the server.

Hiccup is a DSL for generating HTML.

I defined a HTML template and a function for showing it that takes a request and a title.

Note that I used the underscore to denote we’re not using the request.

Delegate is a HOF that returns a function with the title alreadu supplied. Moustache supplies the request.

Try visiting /FooBar now.

commit cd24f552f7eefb470ac7b4357d01486e0fc9c888

I added Clutch as a dependency. Clutch is a library for CouchDB.

Install CouchDB and use Futon to create a wiki database and insert a “MainPage” document with a “content” key.

I defined my own Ring middleware that takes a handler and returns another function that calls the old handler in the context of our database.

The show function now gets a document from the database and passes its content to page.

commit f0c831708f397137b9aadf397376de2202f5cd60

You can now edit and create pages.

Page now takes a revision, which is used in the web form to update.

The update function does use the request object, and uses destructuring to extracts form data as parsed by wrap-params.

Depending if a revision was supplied, a new document is created or an existing one updated. Then, the page is shown.

Note how Moustache now delegates POST requests to update and GET requests to show.

commit 11588b8f1e591ea6415515ffa21e70ed630eef91

I added a Java library for Markdown parsing.

The markup function also replaces WikiLinks with an HTML link.

commit 235c551d57786f79c1867452de37b079a59606bd

Getting ready for deployment.

I removed the debugging wrappers and the var syntax(#’wiki).

Jetty now gets the port number from the environment/Foreman. This means it now runs at port 5000.

Install Foreman with gem install foreman

I added a Procfile according to http://devcenter.heroku.com/articles/clojure

The Origin of Language

2011-10-22T00:00:00-07:00

Dutch is an interesting language. It seems we’re at a crossroad of German, French and English influences.

During the Roman empire, the river Rijn was the border of the Germanic and Roman empire, of which German and Italian are direct descendants¹.

Dutch is very close to German, but it was and is influenced by the ‘world languages’ of the time. (The world was a lot smaller at that time) And so it happened that during 1800-1900 or so, French was spoken by the elite, and we where part of the French empire at times.

If you look at a Dutch dictionary today, you can divide most words in 3 categories, Words that are a lot like German, Archaic and chic words from France and modern words from English.

Dutch is a lot like Common Lisp or Scala². If we’d make a cheesy map of Europe with languages and paradigms overlaid, you’d see that natural languages are complected and very multi-paradigm.

Programming languages are much more designed, and mostly unambiguous, like Esperanto or Lojban. Computers don’t like ambiguity. Some languages, like Scheme, are still designed for growth though.

Near the end Guy Steele argues that programming languages need to be more like natural ones, and the other way around.

As we have seen earlier, natural languages grow, and grow a lot. But I know from experience that adding words to your language that are not native, feels forced at times, does not go well with grammar, and leads to extra complexity.

On the other hand, natural languages should be simpler. But I argue that this has nothing to do with syllables, but with choosing simple words with but one meaning, concerning one thing.

But do our programming languages have this property? Rich Hickey argues this is often not the case, and explains the word ‘simple’ in more detail.³

I can only wonder what future languages will look like. Since our applications will be limited by our understanding, programming will be an art of omission and simplicity.

Last video, I promise. ‘Uncle Bob’ Martin shows us what progress we have made in software development. Not much, compared to Moore’s law. We’re still programming Lisp, and doing assignment, branching and iteration.

What we did with structured programming, with object orientated programming and with functional programming, is taking stuff away. We converted conventions to rules.

Maybe future languages will have rules about simplicity?

I’m not a historian, I never followed a single history lesson. Take with a spoon of salt.
↩
More salt please; At least we’re back at programming.
↩
I can’t seem to embed an InfoQ video.
↩

My Bookshelf 5/5: Making Ideas Happen

2011-10-10T00:00:00-07:00

Great book, read it. And if you don’t, scan the index, it reads like a list of do’s and don’ts.

The book describes the Action Method, which resolves around persisting, following up, managing action steps(todo’s that start with a verb) and relentless execution. It’s not easy, but it makes sense.

I actually read this book during summer holiday. I’ve been, uh… practicing since then. On thing that stuck with me in particular is that connectivity is inverse productivity.

I found it much easier to focus on the action step at hand without all the distractions from email, IM, Twitter, etc. But when you are offline, you need to prepare yourself with all the docs and libs you need, put the whole internet on a floppy if you can¹.

Behance developed a web application specifically for following the Action Method, but as you can imagine, having your task manager ‘in the cloud’ while working offline, is not ideal.

I need something that syncs between my laptop and my desktop, but works great offline. CouchDB seemed to fit the bill perfectly. After I started a homebrew solution and one based on the Backbone MVC framework, I found this.

Those who don’t understand UNIX are condemned to reinvent it, poorly. – Henry Spencer

Why not use mighty tools such as text files and rsync(or Dropbox)? Googling for todo.txt shows I’m not the first to have that idea. There is a more or less agreed upon format, and even a GUI.

Good luck with that, I’m going back to my task list, which features cleaning the kitchen.

Half of my action steps at the time consisted of “Download X”
↩

Understand SQL, learn NoSQL

2011-10-04T00:00:00-07:00

I learned SQL when I started PHP. I found a website named Tizag, where they had SQL tutorials. I installed PhpMyAdmin, created tables, ran queries like SELECT * FROM pages WHERE foo IS bar LEFT JOIN ON comments or whatever. It was magic.

No one ever explained to me how it stored the information, or how it was so fast (or slow). They did say that indexes made stuff faster, sometimes.

On the other hand, when you read the CouchDB guide, they do not primarily teach you their query language, but also a lot about how stuff works. A lot of this also applies to SQL databases.

Storage

CouchDB uses a B-tree to store documents. This provides O(log n) lookup, update, etc. rather than O(n) scanning of all documents. It seems most SQL databases use a B-tree as well, but not always.

Indexes

When you add a WHERE clause to your query, the database has to look at all documents for a match.

If you add an index to the field, you get a sorted representation of that field. This way you can get single items or ranges(time > 123456) in logarithmic time, using binary search.

CouchDB gives you a ‘view’ of the _id of a document, but other views will have to be created to create the equivalent of a WHERE clause. (What _id is in CouchDB, is your primary key in SQL)

Locking & Transactions

NoSQL databases are infamous for their lack of locking and transactions. Why? For the sake of scalability.

Let’s ignore for a moment that 90% of all apps can run on a single server. The idea is that creating a transaction synchronously on a whole cluster is nearly impossible, let alone fast. So you you just don’t to it at all, in NoSQL land.

The flip side is that not locking at all allows reads to be faster. More on that in the next section.

Interesting to note is that both CouchDB and PostgreSQL use MVCC, allowing for reads without locking. So this is not unique to NoSQL databases.

History & Recovery

CouchDB stores its data in append-only B-trees, meaning that data is never changed.

Because old data is still there and immutable, readers can access it without waiting for a write to complete. It is even possible to read old revisions of the data.

What is maybe even more interesting is that, if the server crashes in the middle of an update, the old data is still there.

InnoDB also applies a similar technique, unlike Mysam, which needs to scan and repair the whole database.

Joins

Basically the same thing as transactions, you don’t want to scavenge your whole cluster looking for all comments referencing a blogpost.

The high-performance way to do joins is to not do joins, instead unlearn everything your learned about normalization, and denormalize.

The other way to do it teaches us more about SQL joins though. Basically you create an index or view on the ‘foreign key’, and run a separate query to get the correct documents. Here is an elaborate example.

Conlusion

Neither SQL or NoSQL databases are magic, and they are even pretty similar in most ways. Don’t follow the hype, choose wisely.

Minecraft 1.8 (P)review

2011-09-12T00:00:00-07:00

Like most Minecraft addicts¹, I downloaded the prerelease this weekend and played it all day. There are some really cool things in there, but also a few crapy ones, and of course a few bugs and glitches.

I rally love the new biomes. I spawned in a desert biome, and if it wasn’t for the NPC village, I would have died for sure. Rivers are also great for exploring by boat.

I really like all the stuff that is specific to certain biomes. Without compasses and maps, it’s quite an adventure to get wood, animals and later vines and mushrooms from other biomes.

There are quite a few odd new items, like Ender pearls, rotten meat, and huge mushrooms, but they win on atmosphere, so I built my home in a shroom. Beat that, treehouse!

The new map features all look really neat, but I found the abandoned mineshafts impossible to navigate. They cross right through strongholds, other caves and themselves.

The only thing I rally don’t like about the Adventure Update is the food and fighting mechanics. It’s like playing Quake 3 on peaceful².

I’m not going to say anything about experience points, because I don’t know what they will be used for, but let me say that I don’t think Minecraft is a classic RPG.

The idea of food healing you indirectly is really nice, but having it replenish my life slowly makes me play as reckless as if I’m playing on peaceful. Maybe it could heal you while sleeping?

When you run, you are able to jump 4 blocks. Though, mid-air, you go faster than running, so the fastest way to move is to jump running. I pray that running diagonally isn’t even faster.

There is a new fighting mechanic that lets you deal critical hits by jumping down on a mob while hitting. This means the best way to fight is to jump up and down.

The last thing that adds to the Quake experience is hammering the mouse button. Previously, it was possible to place blocks and hit mobs by just holding the button, now you need to hammer the mouse like a real FPS. dislike

So, overall, I really enjoy 1.8, but can I say it’s having a bit of an identity crisis?

Well, I played all Sunday, but except for that I don’t play much.
↩
I Play on hard.
↩

CSS tips and tricks

2011-08-17T00:00:00-07:00

While working on Sombrero, I encountered a lot of things I did not know how to do properly. I hope they help someone.

width: cling;

Block elements normally fill up all the available width, except when you apply position or float to them. I needed some element to take up the minimal width, without the side effects of floating or positioning. display: table; is the solution I was looking for.

Full-width form fields

Another problem in my magic alignment layout where full-width form fields. Input elements do not take space like normal block elements, no matter the display property.

width: 100%; works, but only as long as you have absolutely no borders, margin, or padding. To get around that, CSS 3 has a property to use the IE 5 box model, which includes padding and borders in the width: box-sizing: border-box;

Collapsing margins

I can’t imagine why I never noticed before, but it turns out CSS collapses margins on “regular” block elements. It’s complicated.

display: run-in;

I never used it, but it seems like an interesting property. It makes things like headings appear inline with the content of the block below it.

Styling range input fields

Some modern browsers support input fields of type="range", for imprecise number values. However, style on these elements is normally completely ignored.

Bring in -webkit-appearance: none;, which turns the slider track back into a normal box. Then apply the knob styling to the ::-webkit-slider-thumb pseudo element.

position all four corners

Another part in my alignment puzzle. Note that not all browsers support this. If you can’t do with 2 sides plus size, it is possible to specify all 4 and leave the sizing implicit.

Playing Minecraft Offline

2011-08-11T00:00:00-07:00

Me and my friends like to play Minecraft sometimes, but it often happens to be in the middle of nowhere, without internet. The most interesting part is most often the setup, not the playing.

We usually start by sharing the latest Minecraft binaries and setting up an ad-hoc WiFi network. Then someone boots a server, and we’re set… NOT!

Minecraft usually performs some DRM and verification with the Minecraft website. For the server, this can be disabled by setting online-mode to false in server.properties, but with the client, you’re out of luck.

Initially, the solution consisted of sending someone with a smartphone out to find an open WiFi network. Later, before the new launcher, it used to be possible to fake the login server¹. But recently, I found a way to play in offline mode on the client as well.

Spoiler alert: Minecraft is The Matrix.

That’s it. It had to be said. Now, let’s hack Minecraft Matrix style.

Normally, when you click “Play Offline” after a failed login, you are named “Player”, which means that you are going to kick each other because you all have the same name.

Now, fire up your terminals², and change directory to the bin folder of your Minecraft directory³.

Now, I can’t read and write Matrix like Mouse⁴, but luckily, the GNU toolchain can. What we’re going to do is change a few occurrences of “Player” with any string of equal length⁵.

The first step is to extract the jar. It turns out it works just fine as a directory, and it’s much easier to work with that way.

mv minecraft.jar minecraft-orig.jar
mkdir minecraft.jar
cd minecraft.jar
jar -xf ../minecraft-orig.jar

Now we need to figure out which files need to be modified. Beware of Déjà vu!

grep -r Player .
# Binary file ./ei.class matches
# ./lang/stats_US.lang:stat.playerKills=Player Kills
# Binary file ./net/minecraft/client/Minecraft.class matches
# Binary file ./net/minecraft/client/MinecraftApplet.class matches
# Binary file ./ow.class matches

Some experimentation shows that MinecraftApplet.class is the one that matters. Now you need to use sed to replace “Player” with another name of equal length.

Mac users will need to install GNU sed, as BSD sed scrambles the binary file beyond repair. I used brew install gnu-sed, but Macports and Fink might also work.

gsed -ibak s/Player/_Notch/g net/minecraft/client/MinecraftApplet.class

Done!

Not anymore, it uses SSL now.
↩
Mac users can use the “Homebrew” style at fullscreen for extra Matrix factor.
↩
Mac: ~/Library/Application\ Support/minecraft/bin, Linux: ~/.minecraft/bin, Windows: ~/.AppData/Roaming/.minecraft/bin
↩
Mouse, please tell me how you wrote the lady in the red dress.
↩
Equal length you hear me, you’ll crash Minecraft otherwise.
↩

The NoSQL Burden

2011-07-29T00:00:00-07:00

Some people claim that NoSQL is premature optimization and places an extra burden on the developer. The main points seem to be that NoSQL has no schemas and drops consistency in favour of the other parts of CAP(availability, partitioning).

I largely agree, however, I think we should differentiate between two kinds of NoSQL databases.

ScaleDB: Hard-core MapReduce, thousands of nodes, sacrifices everything for speed and scalability. If you use this below Google-size, you have no idea what you’re doing.
EasyDB: Relax, life is to short to update your schema and master SQL, give me an easy API to persist and query my data and I’ll be happy. I really think some NoSQL databases should realize this and drop scalability as a prepackaged buzzword feature and focus on the real needs of the majority of their user base.

Relax, does that word ring a bell? It’s the tag line of CouchDB. With its nice REST API and with replication as the only way to scale horizontally, I think CouchDB classifies as an EasyDB.

However, CouchDB uses MVCC, this avoids locking and provides a form of consistency, but it does place the burden of handling update conflicts on the client. Or… does it?

I would like to draw a parallel with how Clojure handles controlled shared mutable state. The simplest form present in Clojure is the Atom.

An atom is a MVCC construct that provides a low-level compare-and-set! function that executes an atomic update if the expected old value matches the actual value. Much like CouchDB compares the _rev of a document before updating.

Interestingly, though, atoms also provide the very convenient swap!, which takes a pure function that takes the old value and returns the new one. swap! calls compare-and-set! in a loop, recomputing the new value on every iteration until the update succeeds.

So what about CouchDB? Can we have easy fire-and-forget updates there as well? Yes we can! I previously hacked together an atom implementation on top of CouchDB, but it turns out CouchDB already offers a little known feature called Document Update Handlers, which does exactly this.

Sadly, the Clojure view server included with Clutch does not yet include support for document update handlers, but this can be easily remedied!

Lion Hacks

2011-07-22T00:00:00-07:00

I installed Mac OS X 10.7 Lion on my Mac yestereday. It did not went very smooth. I’m not going to do a review, but I am going to share my story and some tricks used along the way.

Backup

Always backup, always, and make sure it works.

I run Time Machine backups on an external 500GB HDD, but TM has the nasty habbit to gobble up all available disk space. So when I wanted to make a disk image of my complete HD, I had to remove the TM backup.

Then I did something stupid, and proceeded with the installation without checking the image worked.

Trick: Remove Single Revision

Go into Time Machine to the revision and folder you want to remove. Now right-click and select “Delete Backup”.

Note that the UI is not suitable for removing all but a few revisions, which is why I opted to remove the whole thing.

Trick: Limit Time Machine space

Only after I was finally running Lion did I discovered a way to limit the disk space used by Time Machine.

There are a few complicated hints to achieve this for remote volumes, but nothing for local ones.

This trick involves another Mac, because Lion refuses to connect to a “remote” disk that is actually the local disk shared over AFP.

First, we need to trick Time Machine into using a sparsebundle instead of plain files.

Connect the disk to another Mac
Share it in System Preferences
Initiate a backup, so that a sparsebundle is created
Cancel the backup and disconnect the drive

Now Time Machine will continue using the disk image when the drive is connected directly. The only thing that remains is to limit the bundle in size, using this command:

hdiutil resize -size 150G -shrinkonly /path/to/image.sparsebundle

Where 150G and /path/to/image.sparsebundle need to be replaced with the desired size and the path to the bundle, residing on your backup drive.

Installation

You’re normally supposed to install Lion straight from the App Store, but some smart guy found out you can burn your own DVD or USB drive.

I ended up going the DVD route, because my main drive appeared unbootable by the installer and my USB stick was a few MB to small.

Possible trick: Clean install without DVD

I could not try this because of my broken HD, please let me know if it works.

Lion installs a recovery partition, which plays an important role in the installation. You can’t simply replace a running system, you know.

Run the installation straight from the App Store up to the point where you need to reboot. Now the trick is to hold ⌘R to boot into the recovery mode rather than into the installer.

Now, use Disk Utility to wipe your disk, and run the full installation from the recovery partition.

Migration/Recovery

As I said, I did a clean install, wiped my drive, but forgot to check the image works. tension… It did not work of course.

The right way, of course is to always make sure your Time Machine backup or your disk image works, and to verify and repair your hard drive before you make an image or upgrade.

If you do all of that, Mac has a neat Migration Assistent which can import your old data.

Trick: Mount correct image of damanged partition

Disk Utility is able to perform first aid on damaged disk images, but apparently not mine. Another weird thing is that the checksum matched, but no mountable volumes where found.

So what I have is a working image of a broken disk.

Luckly HFS+ has a feature called journaling, to recover from a bad state, but Disk Utility told me that the journal could not be replayed because the media was read-only.

After using Disk Utility to convert the disk image to read/write, it would mount, and let me extract my files, but upon applying first aid, matters got worse again, and Disk Utility told me to take my files and ruuuun!

Settling in

This part went pretty smooth. I’m swapping some software, waiting for some Lion updates, and looking at all the things that go swoosh.

Homebrew is in, Macports is out. Homebrew makes sysadmins cringe, because it’s written by a “Ruby hippie” that has no sens for dependency management. It does not include its own copy of everything included in OS X, which saves a lot of time and space IMO.

I read iChat is now extensible to support “legacy” protcols, like MSN. Because I’m barely using MSN and becaus everyone is hiring Adium devs, I’m using iChat now. I’ll wait until someone staples libpurple to iChat.

I also heard Growl 1.3 will be a sexy Lion app available in the App Store. Not sure if I’ll wait or install 1.2 anyway.

One thing that has to be done in Lion is un-dumbing Finder.

Customize the toolbar to include that path widget
Show full paths defaults write com.apple.finder _FXShowPosixPathInTitle -bool YES
Show hidden files defaults write com.apple.finder AppleShowAllFiles -bool YES

Then restart finder, killall Finder

Good luck!

My Bookshelf 4/5: Mining the Social Web

2011-07-16T00:00:00-07:00

This book seems almost written for me. It starts of with my favourite platform: Twitter. Next it covers micro formats, which I was just getting interested in. And finally, working with similarity and clustering released a slew of new ideas.

Wht is interesting to note is that this book references a lot of material. This means that it contains more good stuff than fits between the covers, but also that some of the stuff you want to know cannot be found between the covers.

All code used in the book (and more) can be found on github, which saves a lot of typing when you want to play around. The downside is that some longer examples and utilities are only on github, which is sub-ideal when you are reading away from the internet or the computer.

The code used in the book is written in Python, arguably because of its readable syntax and library support, especially the NLTK.

The book also uses Redis and CouchDB extensively, which is not so easily justified at this small scale in my opinion. Later in the book, Pickle is used most of the time.

This book covers a lot of ground in this broad subject, and gives you the tools to explore subjects in-depth yourself. Definitely recommended reading.

Can't take value of a macro

2011-07-07T00:00:00-07:00

Do. not. ever. make me see that exception. Because in my opinion that means you failed at designing your shiny DSL.

A macro is not a thing, but a transform from one thing to another. If your DSL does not have things – domain specific values, as Christophe Grand calls them – at its core, you’re doing it wrong.

First and foremost, a macro should be simple, where simple means ‘not compound’, as explained by Stuart Halloway.

I would like to define a simple macro as one that is written as a compound of simple functions returning code that evaluates to a simple or composite value.

(+ 1 1) returns a simple value
(a-record. 1 2 3) evaluates to a composite of simple things
(do (def foo 2) (def bar 4) (do-thing-to foo)) is not simple

Another good non-reason to use macros is to defer execution.

(defmacro assoc-once [m k v]
  `(if-not (get ~m ~k)
     (assoc ~m ~k ~v)
     ~m))

(assoc-once {:a 1} :a (println "foo"))
; {:a 1}
(assoc-once {:a 1} :b (println "foo"))
; foo
; {:b nil, :a 1}

Works nicely, uh? Now assume we have an atom, try (swap! a assoc-once ...) It’ll honor our post title: Can’t take value of a macro

You know what else defers execution? A function.

(defn assoc-once [m k v]
  (if-not (get m k)
    (assoc m k (v)) ; note parens
    m))

(assoc-once {:a 1} :a #(println "foo"))
; {:a 1}
(assoc-once {:a 1} :b #(println "foo"))
; foo
; {:b nil, :a 1}

This is of course not as good-looking as the macro, but the swap! case works fine here. If you care enough, you could define a macro on top of the function, giving you a simple macro.

(defmacro assoc-once [m k v]
  `(assoc-once* ~m ~k (fn [] ~v)))

Now, this is a very long and opinionated post for me already, so I’m going to stop here. I highly recommend that you watch the 2 video’s I linked to.

I had in mind to take you through another example of using protocols and function composition to simplify and avoid macros, but I’ll just give you this, this and that as a homework assignment.

Generative Music

2011-07-01T00:00:00-07:00

It’s been a long time since I played with Overtone. What I’ve always wanted to do is generate music automatically. Today I ran into jFugue, which makes this really easy, listen to this:

Markov generated music by pepijndevos

I just took some music from wikifonia, constructed a Markov-chain and did a random note-walk.

This could work just as well with Overtone, but I used jFugue because it comes with a parser for MusicXML files.

To run the code below, you need to have jFugue on the classpath. Then just pipe a couple of notes from generate into play.

(ns ponger.core
  (:require clojure.string))

(defn inflate-mxl [file]
  (let [z (java.util.zip.ZipFile. file)]
    (slurp
      (.getInputStream z
        (.nextElement (.entries z))))))

(defn parse-musicxml [file]
  (let [parser   (org.jfugue.parsers.MusicXmlParser.)
        renderer (org.jfugue.MusicStringRenderer.)]
    (.addParserListener parser renderer)
    (.parse parser (inflate-mxl file))
    (.getPattern renderer)))

(defn notes [tokens]
  (filter #(re-matches #"[A-GR].*" %) tokens))

(defn voices [tokens]
  (let [[f [_ & r]] (split-with (complement #(.startsWith % "V")) tokens)]
    (when (seq f)
      (cons f (voices r)))))

(defn make-chain [tokens]
  (into {}
    (for [[k slice] (group-by first (partition 2 1 tokens))]
      [k (map last slice)])))

(defn walk-chain [chain start]
  (let [nxt (rand-nth
              (get chain start
                   (apply concat (vals chain))))]
    (cons
      nxt
      (lazy-seq
        (walk-chain chain nxt)))))

(defn generate [path]
  (-> path
    parse-musicxml
    (.getTokens)
    notes
    make-chain
    (walk-chain nil)))

(defn play [token-seq]
  (let [player (org.jfugue.StreamingPlayer.)]
    (doseq [token token-seq]
      (.stream player token))))

(defn save [token-seq path]
  (.saveMidi (org.jfugue.Player.)
         (clojure.string/join " " token-seq)
         (java.io.File. path)))

NIO in Clojure

2011-06-18T00:00:00-07:00

So, I’m playing with the idea of writing a little webserver in Clojure, much like those asynchronous Python ones like Twisted and Tornado, but actually I’m just writing this to play with my new blog.

There you have the code, not as a gist this time. The code actually lives in a separate file on Github, and is included here by a few lines of Liquid markup.

This code was modeled after this Java code. If you have anything interesting to say about this code, NIO, webservers or Clojure, I’d love to hear from you.

(ns nio
  (:import
     [java.net
      InetSocketAddress]
     [java.nio
      ByteBuffer
      CharBuffer]
     [java.nio.channels
      ServerSocketChannel
      Selector
      SelectionKey]
     [java.nio.charset
      Charset]))

(def *buffer* (ByteBuffer/allocate 16384))

(defn selector [server-socket-channel]
  (let [selector (Selector/open)]
    (.register server-socket-channel selector SelectionKey/OP_ACCEPT)
    selector))

(defn setup [port]
  (let [server-socket-channel (ServerSocketChannel/open)
        _ (.configureBlocking server-socket-channel false)
        server-socket (.socket server-socket-channel)
        inet-socket-address (InetSocketAddress. port)]
    (.bind server-socket inet-socket-address)
    [(selector server-socket-channel)
     server-socket]))

(defn state= [state channel]
  (= (bit-and (.readyOps channel) state) state))

(defn buffer->string
  ([byte-buffer]
   (buffer->string byte-buffer (Charset/defaultCharset)))
  ([byte-buffer charset]
   (.toString (.decode charset byte-buffer))))

(defn string->buffer
  ([string]
   (string->buffer string (Charset/defaultCharset)))
  ([string charset]
   (.encode charset string)))

(defn accept-connection [server-socket selector]
  (let [channel (-> server-socket (.accept) (.getChannel))]
    (println "Connection from" channel)
    (doto channel
      (.configureBlocking false)
      (.register selector SelectionKey/OP_READ))))

(defn read-socket [selected-key]
  (let [socket-channel (.channel selected-key)]
    (.clear *buffer*)
    (.read socket-channel *buffer*)
    (.flip *buffer*)
    (if (= (.limit *buffer*) 0)
      (do
        (println "Lost connection from" socket-channel)
        (.cancel selected-key)
        (.close (.socket socket-channel)))
      (.write socket-channel *buffer*))))

(defn react [selector server-socket]
  (while true
    (when (> (.select selector) 0)
      (let [selected-keys (.selectedKeys selector)]
        (doseq [k selected-keys]
          (condp state= k
            SelectionKey/OP_ACCEPT
            (accept-connection server-socket selector)
            SelectionKey/OP_READ
            (read-socket k)))
        (.clear selected-keys)))))

(defn run []
  (apply react (setup 2323)))

Blogging like a Ruby Hacker

2011-06-16T00:00:00-07:00

Kind of a weird thing to say for someone who does not program Ruby, don’t you think?

I have been wanting to move away from Posterous for quite a while now. I’ve searched for solutions in languages I know and even spent some time writing something in Clojure.

All in all, I wasn’t getting anywhere, so I just said to myself that I wanted to have my blog in a static site generator today.

I chose Jekyll because it’s the first and most popular one I knew. It’s also used by Github and is named after “The Strange Case of Dr Jekyll and Mr Hyde”. Only, it’s in Ruby.

It turns out Ruby syntax is quite easy to guess right for a Python dev. Before I could get started, I had to hack a Posterous migrator to support my old links, tags and images.

So, I hope you like my new old theme and that you don’t find to much broken stuff.

I don’t have comments for the moment, so you’ll have to email me when you have problems. I do plan to add search from Tapir soon.

My Bookshelf 3/5: Seven Languages in Seven Weeks

2011-05-22T00:00:00-07:00

Phew, that took a while. I actually did some languages in under a day, but overall, 7 weeks is about what it took me.

You can argue endlessly about which languages to include in the book, but I think he made a balanced choice that leads to a progressive increase in new concepts.

The book starts easy with Ruby, a mainstream OO language that is the native tongue of the author. Then, we move to prototype inheritance(Io), Logic programming(Prolog), and take the plunge into functional programing with Scala, followed by Erlang, Clojure, and finally the most functional of them all; Haskell.

If you want to learn on specific language well, this book is not for you. But if you are the kind of person who wants to taste every cake in existence, it's great!

The author recommends you to do all the exercises "otherwise it'll just be some syntax". I disagree here. For languages where I did not do the exercises, I remember only concepts, not any syntax at all. This is why I'm reading it, because of concepts, not to actually work with Io(though it's a gem).

So, it's a nice book, but if you want to learn some really wicked stuff, I suggest you also check out these languages suggested by Michael Fogus. (who is also the co-author of the next book on my shelf, Joy of Clojure)

Lazy sorting

2011-05-02T00:00:00-07:00

I was inspired by Edmund's post to try some other lazy sorts. These are far from competitive to Java's timsort, but they are an interesting concept.

It is not possible to make timsort lazy because it uses insertion sort which does not work that way.

Insertion sort is very similar to selection sort. As in selection sort, after k passes through the array, the first k elements are in sorted order. For selection sort these are the k smallest elements, while in insertion sort they are whatever the first k elements were in the unsorted array.

A chunked lazy timsort might be attempted though. Another option might be to use selection sort instead, or use another compounded sort like introsort.

Update: I added an insertion sort and a chunked lazy timsort, without all the frills that make it actually run fast.

all    #'clojure.core/sort "Elapsed time: 67.987 msecs"
sorted #'clojure.core/sort "Elapsed time: 22.214 msecs"
first  #'clojure.core/sort "Elapsed time: 56.249 msecs"

all    #'test/qsort "Elapsed time: 717.93 msecs"
sorted #'test/qsort "Elapsed time: 12388.316 msecs"
first  #'test/qsort "Elapsed time: 30.976 msecs"

all    #'test/msort "Elapsed time: 1317.334 msecs"
sorted #'test/msort "Elapsed time: 945.077 msecs"
first  #'test/msort "Elapsed time: 708.803 msecs"

all    #'test/hsort "Elapsed time: 83.909 msecs"
sorted #'test/hsort "Elapsed time: 75.235 msecs"
first  #'test/hsort "Elapsed time: 28.91 msecs"

all    #'test/isort "Elapsed time: 140.147 msecs"
sorted #'test/isort "Elapsed time: 23.097 msecs"
first  #'test/isort "Elapsed time: 110.453 msecs"

all    #'test/tsort "Elapsed time: 310.026 msecs"
sorted #'test/tsort "Elapsed time: 187.124 msecs"
first  #'test/tsort "Elapsed time: 72.164 msecs"

(ns lazy-sort)

(defn qsort [[head & tail]]
  (when head
    (lazy-cat (qsort (filter #(< % head) tail))
              [head]
              (qsort (remove #(< % head) tail)))))

(defn merge*
  [[f1 & r1 :as l1] [f2 & r2 :as l2]]
  (cond
    (nil? f1) l2
    (nil? f2) l1
    (<= f1 f2) (lazy-seq (cons f1 (merge* r1 l2)))
    (> f1 f2) (lazy-seq (cons f2 (merge* l1 r2)))))

(defn msort [s]
  (if (next s)
    (let [[left right] (split-at (/ (count s) 2) s)]
      (merge* (msort left) (msort right)))
    s))

(defn hsort [s]
  (let [heap (java.util.concurrent.PriorityBlockingQueue. s)]
    (repeatedly (count s) #(.take heap))))

(defn insert [^java.util.LinkedList s x]
  (let [i (.listIterator s (.size s))]
    (while (and (.hasPrevious i) (> (.previous i) x)))
    (.add i x)
    s))

(defn isort [s] ; not lazy
  (seq (reduce insert (java.util.LinkedList.) s)))

(defn treeduce [f s]
  (loop [s (partition-all 2 s)]
    (let [s (map (fn [[x y]] (f x y)) s)]
      (if (< 1 (count s))
        (recur (partition-all 2 s))
        (first s)))))

(defn tsort [s] ; naive
  (treeduce merge* (map isort (partition-all 128 s))))

(defn speed []
  (doseq [s [#'sort #'qsort #'msort #'hsort #'isort #'tsort]
          :let [sort (var-get s)]]
    (println)
    (print "all   " s "")
    (time (dotimes [_ 100] (dorun (sort (shuffle (range 1000))))))
    (print "sorted" s "")
    (time (dotimes [_ 100] (dorun (sort (range 1000)))))
    (print "first " s "")
    (time (dotimes [_ 100] (first (sort (shuffle (range 1000))))))))

Leiningen project-in-a-gist

2011-04-30T00:00:00-07:00

Since I wrote a pattern matcher in under 50 lines of code, I do not feel like setting up an entire project for it, but I do want to use it in other projects and share it with you.

Those great folks at Github added git access to Gists a while back, and it turns out you can make Leiningen run in a flat structure with a few extra lines of project.clj.

This allows you to make cute mini projects with git access that can be installed and published and are embeddable and sharable.

Here is the project.clj from my pattern matcher, I also pushed the jar to clojars.

(defproject seqex "1.0.0-SNAPSHOT"
  :description "a tiny pattern matcher"
  :dependencies [[org.clojure/clojure "1.2.1"]]
  :source-path ""
  :aot [seqex]
  :omit-source true)

First we tell Leiningen that our :source-path is the root dir. The other 2 lines are a bit of a hack to make sure only seqex.clj gets into the jar, Otherwise you'll get conflicts and broken jars.

Clojure micro pattern matcher

2011-04-28T00:00:00-07:00

Today I had some ideas for my game related to cellular automata that involved pattern matching. I googled, and found Matchure.

It might be a great pattern matcher, but the syntax and the amount of macro trickery makes me feel uncomfortable.

In an attempt to write one that relies only on functions, I made this.

Update: This is now a cute project-in-a-gist that can be used as a checkout dependency in Leiningen and Cake.

(ns seqex
  (:import [clojure.lang
            Fn
            IPersistentVector
            IPersistentMap]))

(defn cps [funct cont]
  (fn [[f & r]]
    (when-let [f (funct f)]
      (when-let [r (cont r)]
        (if (true? r)
          (if (true? f) [] [f])
          (if (true? f) r (cons f r)))))))
;    (and (funct f) (cont r))))

(def _ (constantly true))

(def end nil?)

(defmulti matcher type)

(defn match [pattern items]
  ((reduce #(cps (matcher %2) %1)
           (rseq pattern))
     items))

(defmethod matcher :default [l]
  (partial = l))

(defmethod matcher Fn [f] f)

(defmethod matcher IPersistentVector [pattern]
  (partial match pattern))

(defmethod matcher IPersistentMap [pattern]
  (fn [m]
    (every? identity
      (for [[k v] pattern]
        (when-let [mv (get m k)]
          ((matcher v) mv))))))

(defmethod matcher java.util.regex.Pattern [pattern]
  (partial re-find pattern)) ; returns match

(defn store [pattern]
  (fn [s]
    (when ((matcher pattern) s)
      s)))

(defmacro cond-let
  [& clauses]
    (when clauses
      (list `if-let (first clauses)
            (if (next clauses)
                (second clauses)
                (throw (IllegalArgumentException.
                         "cond-let requires an even number of forms")))
            (cons `cond-let (next (next clauses))))))

(match
 [1
  [3 _ end]
  {:a even?, :b #(= 2 (count %))}
  end]
 [1 [3 5] {:a 4, :b [1 2], :d :a}])
; [] (is truthy)

(match [1 2 3 _] [1 2 3 4 5 6 7 8])
; []

(let [[x y] (match
              [1 #"\w.+?\w" 3 (store _) end]
              [1 "foo bar"  3        4])]
  (dotimes [_ y] (println x)))
; foo
; foo
; foo
; foo
; nil

(defproject seqex "1.0.0-SNAPSHOT"
  :description "a tiny pattern matcher"
  :dependencies [[org.clojure/clojure "1.2.1"]]
  :source-path ""
  :aot [seqex]
  :omit-source true)

A new seque

2011-04-21T00:00:00-07:00

I have used clojure.core/seque quite a bit lately, and I love it.

Creates a queued seq on another (presumably lazy) seq s. The queued

seq will produce a concrete seq in the background, and can get up to

n items ahead of the consumer. n-or-q can be an integer n buffer

size, or an instance of java.util.concurrent BlockingQueue. Note

that reading from a seque can block if the reader gets ahead of the

producer.

First I used it in Begame for decoupling the frame computation from the rendering. But for most games, this needs to work without the frames getting ahead of rendering.

This morning I wanted to make a lazy heapsort in one line, rather than with my own heap implementation.

That is when problems started to occur. While seque optionally takes a BlockingQueue as argument, it is not made to support exotic queues like SynchronousQueue(for Begame) and PriorityBlockingQueue(for heapsort), which do a bit more than just threading items through a queue.

Long story short, I submitted a bug report and wrote something that works for me.

Update: I'm not even sure It's possible anymore to write seque so that it works in all cases.

(ns test
  (:import [java.util.concurrent
            BlockingQueue
            LinkedBlockingQueue
            SynchronousQueue
            PriorityBlockingQueue
            CyclicBarrier])
  (:use clojure.test)
  (:refer-clojure :exclude [seque]))

;; some BlockingQueues are...
;; not ordered
;; not finite
;; unable to contain nil
;; without content
;; mutable
;; dropping or recieving extra items

;; seque...
;; fills the que on another thread
;; propagates errors
;; replaces nil with a sentinel
;; detects the end of the input seq
;; can be consumed on any thread

(defn seque
  "Creates a queued seq on another (presumably lazy) seq s. The queued
  seq will produce a concrete seq in the background, and can get up to
  n items ahead of the consumer. n-or-q can be an integer n buffer
  size, or an instance of java.util.concurrent BlockingQueue. Note
  that reading from a seque can block if the reader gets ahead of the
  producer."
  {:added "1.0"}
  ([s] (seque 100 s))
  ([n-or-q s]
   (let [^BlockingQueue q (if (instance? BlockingQueue n-or-q)
                             n-or-q
                             (LinkedBlockingQueue. (int n-or-q)))
         NIL (Object.) ;nil sentinel since BQ doesn't support nils
         s (map (fnil identity NIL) s)
         channel (LinkedBlockingQueue.)
         fill (fn fill [s]
                (try
                  (if (seq s)
                    (do
                      (.put channel #(.take q))
                      (.put q (first s))
                      (recur (next s)))
                    (.put channel #(throw (InterruptedException.))))
                  (catch Exception e
                    (.put channel #(throw e)))))
         fut (future (fill s))
         drain (fn drain []
                 (lazy-seq
                   (try
                     (cons ((.take channel)) (drain))
                     (catch InterruptedException e nil))))]
     (map #(if (identical? % NIL) nil %) (drain)))))

(defn trickle
  [slow]
  (map #(do (Thread/sleep 100) %) slow))

(deftest identity?
  (is (= (seque (range 10)) (range 10))))

(deftest priority
  (is (= 4 (count (seque (PriorityBlockingQueue.)
             [:a :b :c :a])))))

(deftest synchronous
  (is (= (seque (SynchronousQueue.)
                (range 10))
         (range 10))))

(deftest nils
  (is (= (seque [nil true false []])
         [nil true false []])))

(deftest errors
  (is (thrown? Throwable
               (dorun (seque (map #(throw (Exception. (str %))) (range 10)))))))

(deftest threads
  (let [s (seque (SynchronousQueue.) (range 10))
        f1 (future (doall s))
        f2 (future (doall s))]
    (is (= (range 10) @f1 @f2))))

(defn moronic [] ; untestable
  (let [q (LinkedBlockingQueue. 100)]
    (future (dotimes [_ 1000] (Thread/yield) (.poll q)))
    (future (dotimes [_ 1000] (Thread/yield) (.offer q :foo)))
    (seque q (range 1000))))

PyMouse is looking for a new home

2011-04-16T00:00:00-07:00

First of all, thank you very much for your interest and support in PyMouse over the years.

I have some sad news however. There are some outstanding issues and pull requests that need attention, and I haven’t been giving them any recently.

I started PyMouse back in 2009 for making a remote control with my iPod Touch. Since then I haven’t used PyMouse much.

PyMouse was my first big project, and still has the most watchers and forks on Github. I put a lot of energy in figuring out all the platform specific details, and learnt a lot from it.

However, I think the time has come to say goodbye, and hand it over to the community.

I put the project on stillmaintained.com, and if you are interested in taking over, please send me a message.

On the proccess of writing a game engine in Clojure

2011-04-14T00:00:00-07:00

Or at least a set of engine-framework-commons-utils.

There have been a few games and simulations in Clojure before, a few notable ones:

The grandmother of all Clojure games is probably ants.clj by Rich himself, mostly a demonstration of concurrency in Clojure.

Ironclad is a game that deserves to be noted for its completeness; It's more than just an example.

And finally, this unnamed(and likely unfinished as well) RPG, not so much for the game as for the blog post that describes the process of writing it.

Just like Brian, I went through the process of agents and OpenGL, but let us begin with the begin.

Design 0: I want to make a game.

It's as simple as that. A nice and simple 2D game in Clojure. I started googling for game engines in Java and Clojure. The only ones I found that I liked and that did not look like they came from a past century where Slick and PulpCore.

Problem is, I did not even manage to compile Slick, and PulpCore is for applets only. Besides, they both use very OOP models that I suppose would feel awkward in Clojure.

So after reading a few opinions about Swing vs OpenGL, I decided to write my own game in Clojure based on Swing.

Design 1: Agents, agents, agents...

My initial design philosophy was very simple. It should be

multithreaded
functional
easy

So I set up an agent with a vector, made a Swing frame and overwrote its paintComponent to draw the objects in the vector to the frame.

Then I added a watcher to the agent to mark all changed objects dirty on the Swing frame, so they would be redrawn.

Design 1.1: Changed?

Okay, which object changed? Hard to tell. I turned the vector into a map, a set, and a map again, so that I could use clojure.set/difference to tell what happened.

Design 1.2: I can walk through walls!

Wait, we need collision detection. Simple, right?

Doing this the naive way would mean a quadratical increase in comparisons. I was well on my way implementing a quad tree when it hit me: For any non-huge number of objects, it's not worth the trouble.

I changed whatever I had at that moment into a sorted variant, so that I only had to check objects that where at least close in one dimension.

Design 1.999: This is getting out of hand!

At this moment I had changed every single line of code around 3 times. I had atoms, refs and agents, containing vectors, sets, maps, calling multimethods and protocols, I was knee-deep in thread pools and transactions when I remembered my second 2 goals.

This beautiful, concurrent, asynchronous design that I had was just not functional and easy at all. It had mutable state and transactions everywhere.

Design 2: I'm back in the loop.

After a few more sidetracks, I switched to active rendering and reduced my game loop to something like this pseudo-code.

(ns game)

(def world {:foo #<fooinstance> :bar #<barinstance>})

(def window (JPanel. 800 800))

(doseq [frame (iterate #(map act %) world)
        obj frame]
  (draw obj window))

Design 2.1: Reality.

After all this theorizing and experimenting, I finally got my stuff stable enough to actually write a little game; pong.

Let me say this about it: Few lines of the original pong game and my engine where left unchanged, but they where more subtle changes than before.

Next, I wrote a simple platform game. Again, changes where made.

Current design: Are we there yet?

I feel like what I have now is nearly ready to share with the world. I do believe that an event, or message processing based system could work, but what I have now does so too.

Begame currently provides:

A game loop
Time based animation
Collision detection
All the Swing stuff you don't care about
Protocols
- actor; objects that do stuff
- visible; objects that draw themselves
- solid; objects involved in collision detection

Get it.

See the examples, and check out the source from Github.

Sudoku in Logos

2011-03-17T00:00:00-07:00

Upon reading the Prolog chapter in Seven Languages in Seven Weeks, I wanted to attempt the Sudoku example in Logos.

I quickly found out that Logos had quite a few less helpers than Prolog. In Prolog, you just say fd_all_different on all the rows and columns, not so in Logos. Then I read fd_all_different is implemented as ≢'ing all elements. But then again, Logos does not yet support negation.

So rather than saying “a number can not be on the same row twice” I had to say “this row must contain all numbers from 1 to 9 once”. This is done in permutation-o.

(ns logos.sudoku
  (:refer-clojure :exclude [reify == inc])
  (:use [logos
           minikanren
           logic
           nonrel]))

(def board [(lvar) 7      (lvar) (lvar) (lvar) 8      (lvar) (lvar) 5
            (lvar) (lvar) 8      3      (lvar) 1      (lvar) (lvar) (lvar)
            1      6      3      (lvar) (lvar) 4      8      9      7
            9      8      (lvar) (lvar) (lvar) (lvar) 4      (lvar) (lvar)
            2      (lvar) (lvar) 1      (lvar) 5      (lvar) (lvar) 9
            (lvar) (lvar) 4      (lvar) (lvar) (lvar) (lvar) 3      1
            3      4      6      9      (lvar) (lvar) 1      7      8
            (lvar) (lvar) (lvar) 4      (lvar) 7      9      (lvar) (lvar)
            7      (lvar) (lvar) 8      (lvar) (lvar) (lvar) 6      (lvar)])

(def bad-board [9      4      2      5      3      7      6      1      8
                (lvar) 7      (lvar) (lvar) 8      6      9      4      3
                (lvar) 3      8      (lvar) (lvar) 1      2      7      5
                3      (lvar) 7      6      1      (lvar) 4      (lvar) (lvar)
                4      (lvar) (lvar) (lvar) 5      (lvar) 7      3      6
                (lvar) 2      6      3      (lvar) 4      1      (lvar) 9
                8      6      (lvar) (lvar) (lvar) 5      8      2      (lvar)
                (lvar) (lvar) 9      4      2      8      3      (lvar) 7
                2      8      (lvar) 7      (lvar) 3      5      9      1])

(def full-board [5 3 4 6 7 8 9 1 2
                 6 7 2 1 9 5 3 4 8
                 1 9 8 3 4 2 5 6 7
                 8 5 9 7 6 1 4 2 3
                 4 2 6 8 5 3 7 9 1
                 7 1 3 9 2 4 8 5 6
                 9 6 1 5 3 7 2 8 4
                 2 8 7 4 1 9 6 3 5
                 3 4 5 2 8 6 1 7 9])

(def almost-board [5 3 (lvar) 6 7 8 9 1 (lvar)
                   6 7 (lvar) 1 9 5 3 4 (lvar)
                   1 9 (lvar) 3 4 2 5 6 (lvar)
                   8 5 (lvar) 7 6 1 4 2 (lvar)
                   4 2 (lvar) 8 5 3 7 9 (lvar)
                   7 1 (lvar) 9 2 4 8 5 (lvar)
                   9 6 (lvar) 5 3 7 2 8 (lvar)
                   2 8 (lvar) 4 1 9 6 3 (lvar)
                   3 4 (lvar) 2 8 6 1 7 (lvar)])

(def empty-board (repeatedly 81 lvar))

(defn disj-o [x l out]
  (cond-e ;a
   ((cons-o x out l))
   ((exist [f r i]
           (cons-o f r l)
           (cons-o f i out)
           (disj-o x r i)))))

(defn permutation-o [s l]
  (cond-e
    ((== s []) (== l []))
    ((exist [a b x y z]
       (cons-o a b s)
       (cons-o x y l)
       (disj-o x s z)
       (permutation-o z y)))))

(defn blocks [board]
  (mapcat (fn [n]
            (map #(apply concat %)
              (partition 3
                (partition 3 9
                  (drop n board)))))
          [0 3 6]))

(defn slice [board]
  (let [rows (partition 9 board)
        cols (apply map list rows)
        blocks (blocks board)]
    [rows cols blocks]))

(defmacro sudoku [b]
  (let [board (repeatedly 81 gensym)
        [rows cols blocks] (slice board)
        permfn (fn [row]
                 `(permutation-o (list ~@row) (range 1 10)))]
    (concat
      `(exist [~@board]
              (== ~b (list ~@board)))
      (interleave
        (map permfn rows)
        (map permfn cols)
        (map permfn blocks)))))

The beauty of Logos and logic programming in general is that it allows you to tell the computer the data and the rules, instead of telling it what to do. You’d imagine this to be shorter as well. Furthermore, miniKanren is quite efficient, and Logos equally so.

So, how does this code compare to a regular Clojure version? To tell you the truth, not good. The plain Clojure version is both shorter and faster. By how much?

(ns sudoku
  (:use clojure.pprint)
  (:require [clojure [set :as s]]))

(def _ nil)

(def board [_ 7 _ _ _ 8 _ _ 5
            _ _ 8 3 _ 1 _ _ _
            1 6 3 _ _ 4 8 9 7
            9 8 _ _ _ _ 4 _ _
            2 _ _ 1 _ 5 _ _ 9
            _ _ 4 _ _ _ _ 3 1
            3 4 6 9 _ _ 1 7 8
            _ _ _ 4 _ 7 9 _ _
            7 _ _ 8 _ _ _ 6 _])

(def hard [_ 6 _ _ 5 _ 3 2 _ ; 3 added
           _ _ _ 3 _ _ _ 9 _
           7 _ _ 6 _ _ _ 1 _
           _ _ 6 _ 3 _ 4 _ _
           _ _ 4 _ 7 _ 1 _ _
           _ _ 5 _ 9 _ 8 _ _
           _ 4 _ _ _ 1 _ _ 6
           _ 3 _ _ _ 8 _ _ _
           _ 2 9 _ 4 _ _ 5 _]) ; can be either 7 or 9

(def from-blocks [0 9 18 1 10 19 2 11 20 3 12 21 4 13 22 5 14 23 6 15 24 7 16 25 8 17 26 27 36 45 28 37 46 29 38 47 30 39 48 31 40 49 32 41 50 33 42 51 34 43 52 35 44 53 54 63 72 55 64 73 56 65 74 57 66 75 58 67 76 59 68 77 60 69 78 61 70 79 62 71 80])
(def to-blocks [0 1 2 27 28 29 54 55 56 3 4 5 30 31 32 57 58 59 6 7 8 33 34 35 60 61 62 9 10 11 36 37 38 63 64 65 12 13 14 39 40 41 66 67 68 15 16 17 42 43 44 69 70 71 18 19 20 45 46 47 72 73 74 21 22 23 48 49 50 75 76 77 24 25 26 51 52 53 78 79 80])

(defn reduceable-board [board]
  (for [cell board]
    (if (number? cell)
      cell
      (set (range 1 10)))))

(defn prune [group]
  (let [nrs (set (filter number? group))]
    (for [cell group]
      (cond
        (number? cell) cell
        (= (count cell) 1) (first cell)
        :else (s/difference cell nrs)))))

(defn transpose [old mapping]
  (apply assoc (vec old) (interleave mapping old)))

(defn slice [board]
  (let [rows (map prune (partition 9 board))
        cols (map prune (apply map list rows))
        blocks (map prune (partition 9 (transpose (flatten cols) to-blocks)))]
    (transpose (flatten blocks) from-blocks)))

(defn sudoku [board]
  (loop [board (reduceable-board board)]
    (let [next-board (slice board)]
      (pprint (partition 9 next-board))
      (if (or (= board next-board)
              (every? number? next-board)
              (some #(= #{} %) next-board))
        next-board
        (recur next-board)))))

The Clojure code is just a tad shorter, but not significantly so. With pattern matching and negation in place(more on that later), I think the difference becomes negligible.

In speed, however, the difference is huge. The Clojure version can solve a Sudoku in a matter of seconds, I had to leave the Logos version running overnight. Truth be told, for smaller fields or very, very easy boards, Logos can do it in half a second as well. This is why most examples implement mini-sudoku.

There is light at the end of the tunnel though! A recent update to Logos adds pattern matching, which allows for a more Prolog-ish way to express relations.

(defn-e append-o [x y z]
  ([() _ y])
  ([[?a . ?d] _ [?a . ?r]] (append-o ?d y ?r)))

(defn append-o [l s out]
  (cond-e
   ((null-o l) (== s out))
   ((exist [a d res]
      (cons-o a d l)
      (cons-o a res out)
      (append-o d s res)))))

Negation is also planned, which would make it a lot faster to express the relation the numbers in a Sudoku row have.

Last but not least, cond-a and cond-u will return, allowing me to cut away useless branches. Cond-a quits after the first match, rather than searching for alternate solutions.

Pure CSS Minecraft skin previewer

2011-02-26T00:00:00-08:00

If you where expecting the second part of my bookshelf, hang on, it's nearly done.

Meanwhile, a few friends set up a Minecraft server, and I joined them to dig some holes and build a house.

A neat thing with Minecraft is that you can make your own skin, only, it doesn't really look like something when you open it in Photoshop.

There are a few skin editors floating around the web, but nothing with the power of Photoshop. So as a programmer, I decided to craft my own viewer, rather than installing some weird piece of software.

What you see here is a piece of CSS that uses transforms and background-position(tiles) to show me a preview of my new skin. Thus, my workflow consisted of editing, saving, refreshing and more editing.

Live demo, showing my current skin: http://wishfulcoding.nl/char/char.html

Recent versions of Safari, Chrome and Firefox should be able to preview you own skins.

My bookshelf 2/5: The Reasoned Schemer

2011-02-26T00:00:00-08:00

This is the second part of my bookshelf, a series of posts about a couple of books I'm reading.

Unlike the CouchDB book, The Reasoned Schemer(TRS henceforth) is a book that you don't simply read on the couch, but actually study. This is why it took weeks rather than days to come to this point, where I understand how to use(but not implement) miniKanren.

The book has a gradient on the cover, which gives it the impression of a technical book(it's from MIT press), but it actually contains drawings and the typesetting is brilliant(it is written with some sort of LaTeX I believe). It is also very suitable for vegetarians, since it uses vegetarian food rather than foo/bar/baz.

The book is written in questions and answers, both in separate columns. You could overlay the answers, and try them yourself first.

TRS describes miniKanren, a logic programming language. It is the bare essence of Prolog and Kanren, written in a few hundred lines of Scheme.

I used Logos, an implementation in Clojure by David Nolen, to follow the examples in the book. This worked out very well, but there are a couple of minor differences.

fresh is called exist in Logos.
Logos uses vectors for binding variables.
Everything is interleaving, so there are no cond-i and all-i.
Clojure does hot have pairs, and only proper conses are allowed, it uses lcons instead.
Clojure has first and rest, so car-o and cdr-o are called after their clojure.core counterparts.

Later in the book, an arithmetic system is implemented in miniKanren. I attempted to port it to Logos, though it is not complete and bug free yet.

I do not yet fully understand the last few chapters of the book, one of which describes unification. The last chapter is unique in the book in that it does not use the question-answer style, but rather explains the inner workings of miniKanren in tightly packed chunks of information.

I really enjoyed the book, and I hope to find some good uses for my newly acquired powers.

My bookshelf 1/5: CouchDB

2011-02-10T00:00:00-08:00

I recently ordered four programming books from Amazon, and one from Manning. I'm planning to share some comments, insights, thoughts and something akin to a review, as I 'resurface' on the web in between my reading spurts.

I started with CouchDB: A definitive Guide, because I expected it to be the least mind-taxing of the five, and because I am considering CouchDB for two projects of mine.

The book contains a lot of good information, but is also very messy in places. It is baginer-paced, but omits a lot of details.

I found the first chapters, the chapter about CouchApp and the appendices to be the most informative.

The first chapters take you on a high-level tour of CouchDB, with a great deal of fine information. I was pleasantly reminded of the nature of MVCC and B-trees.

Multi Version Concurrency Control means there is absolutely no locking involved, so reads are really fast. Even more so, because views are generated on writing.

Because both databases and views are stored as B-trees, a surprising number of actions are completed in logarithmic time. Surprising ones include updating a (re)reduced view, and taking a range from a large number of items.

Another thing to remember is that everything is sorted by keys, and that keys can be complex types, such as lists.

After about half of the book, I started to skip sections, and I completely skipped the chapters about scaling, which seem nearly inappropriate for a beginner-level book.

One chapter I did read thoroughly is the chapter about CouchApp. CouchApp makes it really easy to maintain and deploy complex web applications that run sans lingua(I made that up; it means "nothing but CouchDB").

That chapter was really helpful, but comes with a flip-side. The information about CouchApp on the web is really sparse:

This blog post is in response to a lot of well-deserved confusion in the community around CouchApps.

The appendices mainly go into detail about things like installation and the details about the use of B-trees.

So, that's it for now. I'd call this book a "good mess". Keep an eye on this blog or my Twitter feed for the next books, and my first CouchDB apps.

Take n distinct random items from a vector

2011-02-01T00:00:00-08:00

Another IRC hacking session.

(ns test
  (:use (incanter core charts)))

; naive, O(n+m)
(defn take-rand1 [n coll] (take n (shuffle coll)))

; lazy, O(n!@#$%m^&)
(defn take-rand2 [n coll]
  (take n (distinct (repeatedly #(rand-nth coll)))))

; reduce, reorder, subvec, O(m)
(defn take-rand3 [nr coll]
  (let [len (count coll)
        ; why doesn't rand-int take a start?
        rand-int (fn [lo hi] (+ lo (rand-int (- hi lo))))]
    (subvec (->> (range nr)
                 (reduce #(conj %1 [%2 (rand-int %2 len)]) [])
                 (reduce
                   (fn swap [a [i b]]
                      (assoc a b (get a i) i (get a b)))
                   coll))
            0 nr)))

; amalloy, O(m)
(defn take-rand4 [num coll]
  (first
   (nth
    (iterate (fn [[ret candidates]]
               (let [idx (rand-int (count candidates))]
                 [(conj ret (candidates idx))
                  (subvec (assoc candidates idx (candidates 0))
                          1)]))
             [[]
              coll])
    num)))

(defn t [f]
  (let [start (. System (nanoTime))]
    (f)
     (- (. System (nanoTime)) start)))

(defn plot-len [f n]
  (let [coll (vec (range n))]
    (t #(doall (f 1000 coll)))))

(defn plot-take [f n]
  (let [coll (vec (range 100000))]
    (t #(doall (f n coll)))))

(def x (range 1000 100000 1000))

(-> (scatter-plot x (map (partial plot-len  take-rand1) x) :series-label "shuffle" :legend true)
    (add-points   x (map (partial plot-len  take-rand2) x) :series-label "filtered")
    (add-points   x (map (partial plot-len  take-rand3) x) :series-label "reduce")
    (add-points   x (map (partial plot-len  take-rand4) x) :series-label "iterate")
    (save "len.png")) ; http://yfrog.com/5r80779474p

(-> (scatter-plot x (map (partial plot-take take-rand1) x) :series-label "shuffle" :legend true)
    (add-points   x (map (partial plot-take take-rand2) x) :series-label "filtered")
    (add-points   x (map (partial plot-take take-rand3) x) :series-label "reduce")
    (add-points   x (map (partial plot-take take-rand4) x) :series-label "iterate")
    (save "take.png")) ; http://yfrog.com/mr42647628p

Sorting obsession

2011-01-15T00:00:00-08:00

It all started when, one day, I was thinking about the little I knew about sorting. What I knew was that sorting was usually done by comparing all items and switching them around, hitting the magical O(nlogn) barrier.

Thinking about how I would sort a deck of cards, sorting seemed more like a O(n) operation. With that in mind, I started implementing what turned out to be called Pigeonhole Sort. I also learned that sorting algorithms can be classified as being a Comparison sort or not.

(ns pigeonhole-sort)

(defn int-sort [s]
  (let [listmap (reduce #(update-in
                           (update-in %1 [%2] (fnil inc 0))
                           [:max] max %2) {:max 0} s)]
    (mapcat #(repeat (get listmap % 0) %)
            (range (inc (:max listmap))))))

With that out of the way, I started to compare my sort to the sort used by Java and Clojure, which turned out to be using Timsort, originating in Python. In my quest to understand Timsort - and why it was faster than my Pigeonhole Sort - I found this great website that graphs how different sort algorithms work.

Simply put, I was intrigued by the pattern displayed by Heap Sort. So next I found myself reading the Wikipedia pages for Binary Heaps, and shortly after that, I had started implementing that in Clojure.

This is, ladies and gentleman, a persistent heap, in all its heapyness, with a heapsort implemented on top of it.

(ns persistent-heap)

(defn swap [heap idx idy]
  (assoc heap idx (get heap idy) idy (get heap idx)))

(defn children [idx]
  (let [idx (inc (* idx 2))
        idy (inc idx)]
    [idx idy]))

(defn parent [idx]
  (if (not= 0 idx)
    (/ (- idx (if (odd? idx) 1 2)) 2)
    nil))

(defn tree 
  ([heap] (tree heap 0))
  ([heap idx]
   (let [[left right] (children idx)
         node (get heap idx nil)]
     (when node
       [node (tree heap left) (tree heap right)]))))

(defn heap-up 
  ([heap] (heap-up heap >= (dec (count heap))))
  ([heap compfn] (heap-up heap compfn (dec (count heap))))
  ([heap compfn idx]
   (if-let [par (parent idx)]
     (if (compfn (get heap idx) (get heap par))
       (recur (swap heap idx par) compfn par)
       heap)
     heap)))

(defn heap-down
  ([heap] (heap-down (pop (swap heap 0 (dec (count heap)))) >= 0))
  ([heap compfn] (heap-down (pop (swap heap 0 (dec (count heap)))) compfn 0))
  ([heap compfn idx]
   (let [[left right] (children idx)
         leftv (get heap left nil)
         rightv (get heap right nil)
         node (get heap idx nil)]
     (if (and node leftv rightv)
       (cond
         (compfn leftv (max rightv node))
           (recur (swap heap idx left) compfn left)
         (compfn rightv (max leftv node))
           (recur (swap heap idx right) compfn right)
         :else heap)
       heap))))

(deftype PersistentHeap [heap]
         clojure.lang.ISeq
         (first [this] (first heap))
         (next [this] (PersistentHeap. (heap-down heap)))
         (more [this] (.next this))
         (cons [this obj] (PersistentHeap. (heap-up (conj heap obj))))
         (seq [this] (seq heap)))

(defn persistent-heap [coll]
  (into (PersistentHeap. []) coll))

(defn heapsort [coll]
  (->> (persistent-heap coll)
       (iterate rest)
       (take (count coll))
       (map first)))

Some 'benchmark' results:

user=> (time (dorun (heapsort (shuffle (range 1e5)))))
"Elapsed time: 3148.041 msecs"
nil
user=> (time (dorun (int-sort (shuffle (range 1e5)))))
"Elapsed time: 477.781 msecs"
nil
user=> (time (dorun (sort (shuffle (range 1e5)))))
"Elapsed time: 105.354 msecs"

Note that heapsort is lazy, insofar that it does realize but does not sort the sequence untill requested. There is also a bug that allows for the last 2 items to be out of order.

I'm tired of the NoSQL buzz, and bring you SlouchDB

2011-01-13T00:00:00-08:00

It has all the features your favorite NoSQL database has.

Fast
Scalable
Concurrent
Key/value store
Document orientated
MapReduce
In-memory
Persistent *
NoSQL
REST API *
Written in X, so...

* Requires a few extra lines.

The implementation looks like this.

(ns slouchdb)

; Fast scalable in-memory concurrent NoSQL database
(def db (ref {:bob  {:age 59 :sex :male}
              :bill {:age 17 :sex :male}
              :mary {:age 28 :sex :female}}))

;; Views ;;

(defn total-age []
  (reduce + (pmap (comp :age val) @db))) ; Parallel MapReduce!

;; Examples ;;

(println (apply str (interpose \newline
  ["Get an item"
  (:bob @db)
  "MapReduce"
  (total-age)
  "Add/update an item"
  (dosync (alter db assoc-in [:bill :age] 25))
  "Delete an item"
  (dosync (alter db dissoc :mary))])))

; For scaling, press CMD or CTRL and + or -

; For a persistent version, see java.io.Serializable

; For a REST API, check out Moustache and Aleph.

Cooperative concurrency in Clojure

2011-01-05T00:00:00-08:00

Today I had an interesting discussion about cheap preemptive threads in Clojure. Someone claimed Haskell can easily start 10.000 threads without getting slow, while Clojure - and native threads in general - do get slow after a couple of thousands.

After a discussion about Agents and thread pools, I proposed cooperative concurrency as a cheap solution. Cooperative concurrency isn't exactly preemptive, but it works. This is a for-fun implementation of trampoline using a hidden gem in Clojure called PersistentQueue.

Like trampoline, recursive functions return another function, which get called in a loop. Unlike trampoline, mine takes a seq of functions, puts them into a queue and executes the frontmost function, putting the result on the back of the queue.

Below is an example of the classic mutual recursive odd and even functions running in parallel, or alternating actually. Like pcalls, but on a single thread.

(ns cooperative)

(defn cooperative-tasks
  "Trampoline over a queue for
  cooperative multitasking"
  [fs]
  (if (not-any? fn? fs)
    fs
    (recur
      (doall (for [f fs]
        (if (fn? f)
          (f)
          f))))))


(comment
  (declare my-odd?)

  (defn my-even? [n]
    (if (zero? n)
      true
      #(my-odd? (dec (Math/abs n)))))

  (defn my-odd? [n]
    (if (zero? n)
      false
      #(my-even? (dec (Math/abs n)))))

  (cooperative-tasks [#(my-odd? 50009) #(my-odd? 50008) #(my-even? 50008) #(my-even? 50009)])
)

Happy Christmas!

2010-12-24T00:00:00-08:00

(ns xmas)

(doseq [line (concat (take 9 (map #(apply str
                                          (take (+ (rem % 3)
                                                   (int (/ % 3)))
                                                (repeat "*")))
                                  (range 20)))
                     (take 2 (repeat "*")))]
  (println (format "%1$30s*%1$s" line)))

;                              *
;                             ***
;                            *****
;                             ***
;                            *****
;                           *******
;                            *****
;                           *******
;                          *********
;                             ***
;                             ***

8 ways to tell how many Xes are in a list

2010-12-22T00:00:00-08:00

Inspired by a question on #clojure. There we go.

mrBliss(1):

(count (filter #{1} '(1 2 4 5 1 5 2))) ; Use this

Raynes:

(count ((group-by identity [1 2 3 4 5 1 5 2]) 1))

Me(1):

(get (frequencies [1 2 3 1 2 3]) 1) ; Or this

Me(2):

(reduce #(if (= %2 1) (inc %1) %1) 0 [1 2 3 2 3 1])

Me(3)

(count (nth (partition-by identity (sort [ 1 2 3 1 2 3])) 0))

mrBliss(2):

(count (remove (complement #{1}) '(1 2 3 1 2 3))) ; But don't ever do this

_ato(1):

((fn f [[x & xs :as coll]] (cond (= coll '(1)) 1, (nil? xs) 0, :else (apply + (map #(f (take-nth 2 %)) [coll xs])))) [1 2 3 4 5 1 5 2])

_ato(2):

(let [s (sort [1 2 3 4 5 1 5 2])] (- (.lastIndexOf s 1) (.indexOf s 1) -1))

But at least Clojure has an awesome and helpful community!

The Perfect Server

2010-12-11T00:00:00-08:00

I want a server that...

Is fast.
Easy to setup and maintain.
Fixable(at runtime, which every dynamic language promotes itself with)
Serves everything I throw at it
- PHP
- Python
- Clojure

Serving everything poses a bit of a problem here.

Tomcat proxy'd behind Apache

This is by far the most common setup I believe. But now I have two servers to configure, run and maintain!

Java implementations of everything

I believe both Quercus and Jython run all major frameworks in their languages, and run at reasonable speeds. I still fear it'll be a constant fight for the alternative route though. Consider porting all of the mod_rewrite rules Wordpress uses to servlet rules. Eew!

Standard implementations under JFastCGI

Close, but much of the same problems as the previous one. Native Python and PHP libs are a big win though.

"Wouldn't it be dreamy if there was a server that would be a breeze to setup, that runs all languages I use, but without the limitations involved with running PHP and Python in a servlet"

- Dreamy girl from Head First books.

mod_ring would be dreamy for one, running Clojure's HTTP API under Apache. But after a quick trip to Google and mod_wsgi, I gave up this idea. Doesn't exist; to complicated to write.

Ring is a nice idea though. I wonder if I can wrap JFastCGI in a nice Ring sauce.

Yes we can!

Another trip to Google and SourceForge reveals that FastCGIHandler takes adapters for the Servlet and Portlet API. So it would suffice to write an adapter for Ring, and I would be able to use Django and Wordpress in a nice Moustache wrapper.

https://github.com/pepijndevos/fastcgi-ring

So, dit I meet my original goals?

Is fast - Somewhat, see below.
Easy to setup and maintain - yes, it's all Clojure and Leiningen/Cake.
Fixable - yes, nREPL might come in handy here.
Serves everything I throw at it - yes sir!

I ran a basic `ab -n 1000 -c 50` on both Apache and Aleph running fastcgi-ring, with an identical installation of Wordpress. Both utilize 100% of my CPU, where Apache spawns tens of workers, and php-cgi a predefined set of 4 workers. Apache is a bit faster, with the gap increasing when I just test "hello world". I did not test the speed of other combinations like Tomcat+JFastCGI or Lighttpd.

$ ab -n 1000 -c 50 http://localhost:3000/index.php
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        aleph
Server Hostname:        localhost
Server Port:            3000

Document Path:          /index.php
Document Length:        8353 bytes

Concurrency Level:      50
Time taken for tests:   119.077 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      8558000 bytes
HTML transferred:       8353000 bytes
Requests per second:    8.40 [#/sec] (mean)
Time per request:       5953.868 [ms] (mean)
Time per request:       119.077 [ms] (mean, across all concurrent requests)
Transfer rate:          70.18 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       5
Processing:   337 5809 2453.4   5782   12297
Waiting:      336 5808 2453.4   5782   12296
Total:        338 5809 2453.3   5782   12297

Percentage of the requests served within a certain time (ms)
  50%   5782
  66%   6832
  75%   7467
  80%   7899
  90%   9152
  95%  10033
  98%  10848
  99%  11370
 100%  12297 (longest request)

$ ab -n 1000 -c 50 http://localhost/~pepijndevos/wp/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        Apache/2.2.15
Server Hostname:        localhost
Server Port:            80

Document Path:          /~pepijndevos/wp/
Document Length:        8353 bytes

Concurrency Level:      50
Time taken for tests:   99.862 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      8626000 bytes
HTML transferred:       8353000 bytes
Requests per second:    10.01 [#/sec] (mean)
Time per request:       4993.091 [ms] (mean)
Time per request:       99.862 [ms] (mean, across all concurrent requests)
Transfer rate:          84.35 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.8      0      10
Processing:   880 4971 1019.3   4896    9849
Waiting:      865 4953 1019.1   4881    9833
Total:        884 4972 1019.2   4897    9849

Percentage of the requests served within a certain time (ms)
  50%   4897
  66%   5053
  75%   5144
  80%   5212
  90%   5657
  95%   6543
  98%   8279
  99%   8864
 100%   9849 (longest request)

Free premium Twitter marketing tool

2010-12-06T00:00:00-08:00

A friend told me about someone who sells a Twitter product for marketeers for around €300. What this tool does is that it keeps track of hash tags you specify and mails tweets to you, so you can answer questions and get new followers.

I was surprised to say the least. I frequently write these kind of things, but I find them to trivial to sell, I told my friend. To prove my point, I wrote this Python script - which I will give away for free - that tracks keywords, makes a noise and opens tweets in a browser as it finds them.

import json
import webbrowser
import urllib2
import urllib

#http://dev.twitter.com/pages/streaming_api_methods#track
kwds = "#durftevragen photoshop"
# Twitter username
uname = ""
#Twitter password
pwd = ""

password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()

top_level_url = "http://stream.twitter.com"
password_mgr.add_password(None, top_level_url, uname, pwd)

handler = urllib2.HTTPBasicAuthHandler(password_mgr)

opener = urllib2.build_opener(handler)

data = opener.open('http://stream.twitter.com/1/statuses/filter.json?track=' + urllib.quote_plus(kwds))

for line in data:
    j = json.loads(line)
    url = 'http://twitter.com/' + j['user']['screen_name'] + "/status/" + j['id_str']
    print url
    print('\a')
    webbrowser.open(url)

Now that I have given you worth €300 of Python code for free, I'd like to ask you a favor. I recently started my own company that is specialized in Wordpress blogs, Twitter tools and other web services and I'd like to know about the Twitter tool you've always wanted and what you'd pay for it.

Some ideas of myself(and my friend):

Extend my Twitter email suit with this tracking.
Multiple keywords
Organized in nice IMAP folders.
Reply per email(No need to visit Twitter).
Assemble a FAQ/knowledge base on your Wordpress blog.
Tweet random facts from there.

The age of APIs ... and virtual ghost towns?

2010-12-02T00:00:00-08:00

Nowadays almost any self-respecting site seems to have an API. What can you do with all this data?

Clients
Analytics
Aggregation
And... a clone? Sure!

I was thinking it'd be fun to do a Reddit clone, and experiment with different voting systems. Point is, there are quite a few of them, so how am I going to get users? Answer: I don't need users!

I took this and this, and in a few hours(mostly spent updating the work of Lau), I had an autonomous Reddit clone running of the Twitter API, and I believe it to be possible to create a complete social network that runs without any user input.

How much Clojure do you know?

2010-11-29T00:00:00-08:00

user=> (count (ns-publics 'clojure.core))
546

That are a lot of functions. Unless you wrote half of them I wager you don’t know them all. Don’t worry, neither do I. But it doesn’t hurt to learn some more. That is why I wrote a Clojure quiz.

Simply do the standard stuff(make sure you have git and lein or cake)

git clone git://github.com/pepijndevos/clojure-quiz.git
cake deps
cake repl
(use 'clojure-quiz.core)

Now you can play three types of quizzes.

(doc-quiz) Find the correct doc string for the given function.
(name-quiz-easy) Find the correct function for the given doc.
(name-quiz) Same as above, but without multiple choice.

(name-quiz) is by far the hardest, and I generally score around 70% while I can usually get 100% on the others.

You can tweak the quiz by using binding to set the namespace to use and the number of options to give:

(binding [target 'clojure.set number 5] (doc-quiz))

Have fun!

Crowd sourced news with Clojure

2010-11-14T00:00:00-08:00

This snippet gets tweets, filters their links, sorts, resolves and counts them in about 30 lines Clojure, showing off the power of its Java interoperability and concurrent data structures.

(ns news
  (:refer-clojure :exclude [resolve])
  (:use clojure.contrib.json)
  (:import [java.io BufferedReader InputStreamReader]
           [java.net URL]))

(def cred "") ; username:password

(def json (let [con (.openConnection (URL. "http://stream.twitter.com/1/statuses/sample.json"))]
            (.setRequestProperty con "Authorization" (str "Basic "
                                                          (.encode (sun.misc.BASE64Encoder.) (.getBytes cred))))
            (BufferedReader. (InputStreamReader. (.getInputStream con)))))

(def urls (agent (list)))

(def resolve (memoize (fn [url]
  (try
    (let [con (doto (.openConnection (URL. url))
                (.setInstanceFollowRedirects false)
                (.connect))
          loc (.getHeaderField con "Location")]
      (.close (.getInputStream con))
      (if loc loc url))
    (catch Exception _ url)))))

(defn top [urls]
  (reduce #(if (> (val %1) (val %2))
             %1 %2)
          (frequencies @urls)))

(future (doseq [tweet (repeatedly #(read-json json))
                url (:urls (:entities tweet))]
          (send-off urls #(conj % (resolve (:url url))))))

And this is just the start. I want to see if I can use Aleph and ClojureQL to kickstart my own little news service with it.

Clojure versus Python

2010-11-04T00:00:00-07:00

Mian versus Clomian

How Clojure sat in a corner converting and boxing while Python did the work

A plot about Minecraft(pun intended)

Update: The Clojure version is now a lot faster. Thanks to the people of the Clojure mailing list. I also uploaded the map I used if you want to compare the results.

Okay, back to business. During the writing of the original Python hack I had to do a few tricks I thought would be easy to do in Clojure. So I started to wonder how the Clojure code would look and how fast it’d be.

My original hack was kind of slow, but it’s greatly improved and now renders a whole map in under 10 seconds.

4s for reading all files
3s for calculating the graph
8s total

The code to read all the files:

paths = glob(join(world_dir, '*/*/*.dat'))

raw_blocks = ''
for path in paths:
    nbtfile = NBTFile(path, 'rb')

    raw_blocks += nbtfile['Level']['Blocks'].value

The code to calculate the graph:

layers = [raw_blocks[i::128] for i in xrange(127)]

counts = [[] for i in xrange(len(bt_hexes))]
for bt_index in range(len(bt_hexes)):
    bt_hex = bt_hexes[bt_index]
    for layer in layers:
        counts[bt_index].append(layer.count(bt_hex))

Nice eh? Now the Clojure version. Clojure doesn’t have a nice blob module, so I’ll spare you the code that gives me the data. Sufficient to say is that it also runs in about 4 seconds.

My initial version for the calculating was short and sweet and looked like this:

(defn freqs [blocks]
  (->> blocks
    (partition 128)
    (apply map vector)
    (pmap frequencies)))

Now, this is twice as fast as what I currently have, but it has a problem. While Python operates on bytes the whole time, these lines of Clojure operate on a sequence of objects. These objects are just a tad bigger than the bytes in a string, so keeping 99844096 of those in memory is impossible.

So, either I had to find a way to make Clojure throw away all the objects it had already processed, or I had to make it use a more compact storage for them. I tried both, and ended up with a function to concatenate Java arrays, but working with them is a real pain, so I made my function use them wrapped in Clojure goodness and made sure the Java GC threw them out as soon as I was done.

(defn freqs [blocks]
  (->> blocks
    (partition 128)
    (reduce (fn [counts col]
              (doall (map #(assoc! %1 %2 (inc (get %1 %2 0))) counts col)))
            (repeatedly 128 #(transient {})))
       (map persistent!)))

This is not threaded like to previous example, but it works. Everything I tried to make it use all my cores either started to eat more and more memory, or was slower then the single-treaded one. Most of them where both.

So, how fast is it?

5s file reading
Over a minute of processing
Over a minute + 5s total

Wait, what? Python did this in 3 seconds, right? Yea… So even if I had used the faster function and had 10GB of RAM it’d be 10 times slower.

Why? I don’t know. All I can come up with is that that Python just acts on a string, while Clojure does boxing and converting 99844096 times. If you happen to know what’s wrong, or how to make it faster, be sure to tell me!

Where to dig in The Nether

2010-11-02T00:00:00-07:00

Spoiler: Don't

You remember my other Minecraft post? A lot happened since then. Victor Engmark took my hack and made a nice tool out of it that runs way faster as well! Notch released the Halloween update with a new world that is at the moment called The Nether. And last but not least, I ported Mian to Clojure to form Clomian, to compare it to the Python version and to analyze The Nether.

There are still some issues with the code, and I might actually have found my first bug that is actually in Clojure and not in my code. Therefore I will talk and compare in another post, and for now just show you the results.

The bug:

(reduce #(map + %1 %2) (partition 5 (range 1e6)))
java.lang.StackOverflowError

(reduce #(pmap + %1 %2) (partition 5 (range 1e6)))
(99999500000 99999700000 99999900000 100000100000 100000300000)

The graphs:

As you can see The Nether is an unfriendly place with nothing but Bloodstone, Lava, Fire and Mushrooms.

Unlike the normal world, The Nether is 100% enclosed by Bedrock and in the middle, half of everything is still Bloodstone.

Unlike the normal world, there aren't any minerals and there isn't a clear distinction between above and below.

The only valuables I see are mushrooms and Lightstone, and these are somewhat uncommon and scattered across the whole level.

Conclusion:

The Nether is an interesting world, but it's not worth getting killed for, unless you desperately need any of those special blocks or shrooms.

How to be a good programmer

2010-10-14T00:00:00-07:00

Get back to work.

Where to dig in Minecraft

2010-10-10T00:00:00-07:00

I have been playing Minecraft a lot lately, and I was wondering where to dig for certain minerals. I knew Redstone was only found in lower layers of the level, but I knew nothing concrete.

I pulled together Gnuplot and a Python lib for reading NBT files and analyzed a level. Here is the result, the code and some notes.

I suspect the sea level is around 60, because of the clay there. Coals is rare and Iron very rare above sea level.

Both Diamond and Redstone stop at 20, while gold can be found a little higher. There is no noticeable decline in Coal and Iron near the bottom, so best is to dig below level 20.

As can be seen above, there is a lot of Lava, Water and Obsidian around and below level 10, so stay above level 10 for safe mining.

The code:

from nbt import nbt
from itertools import izip_longest, repeat
import glob
import Gnuplot

files = glob.glob("/Users/pepijndevos/Library/Application Support/minecraft/saves/World2/*/*/*.dat")
output = "minecraft.png"

col = []

for file in files:
    print file
    file = nbt.NBTFile(file,'rb')
    
    col.extend(izip_longest(*[iter(file["Level"]["Blocks"].value)]*128))

    file.file.close()

layers = zip(*col)

minerals = {
    #'Bedrock': '\x07',
    #'Water': '\x09',
    #'Lava': '\x0B',
    #'Sand': '\x0C',
    #'Gravel': '\x0D',
    'Gold': '\x0E',
    'Iron': '\x0F',
    'Coal': '\x10',
    'Obsedian': '\x31',
    'Diamond': '\x38',
    'Redstone': '\x49',
    'Clay': '\x52',
}

names = minerals.keys()
hexes = minerals.values()

def count_minerals(layer):
    def filter_mineral(mineral):
        count = len([block for block in layer if block == mineral])
        return count
    
    m = map(filter_mineral, hexes)
    return m

g = Gnuplot.Gnuplot(debug=1)
g('set term png')
g('set out "%s"' % output)
g('set style data lines')

data = zip(*(count_minerals(layer) for layer in layers))

data = (Gnuplot.PlotItems.Data(list(enumerate(mineral)), title=names[index]) for index, mineral in enumerate(data))

g.plot(*data)

I regret doing this thing in Python. My coding style is strongly influenced by Clojure, and I had to do a number of hacks that would be trivial to do in Clojure. I might do a comparison in a future post if I ever do a Clojure version.

How reify works, and how to write a custom type

2010-09-25T00:00:00-07:00

Yesterday I had the idea of lazy sequences that can go in both directions. I dropped into #clojure on irc.freenode.net to ask someone about it.

Given the fact that Clojure-in-Clojure is possible, it is thus possible to define lazy seqs. Would it also be possible to define a lazy seq where the 0th element is actually in the middle of an infinite lazy seq extending in both ways? so like (iterate inc) with negative number included :)

I found Chris Houser(Author of Joy of Clojure) interested, and willing to help me along.

so... how shall we proceed? You want to see my code or should I try to drop hints or ask leading questions?

I went for the later one, and had a great learning experience. Near then end Paul deGrandis chimed in to recommend Joy of Clojure. I'm convinced!

Joy of Clojure has an awesome section on proxy, reify, defprotocol, and defrecord. I reference it often to judge if I'm abusing something

At the point you're at in Clojure (based on what I follow in here), you'd get a lot of mileage out of it.

Bellow You'll find a grep'd-over version of the log with most of the background noise canceled. The full log is at http://clojure-log.n01se.net/date/2010-09-24.html

The final code for my version

(ns biseq)

(defprotocol Spinnable (spin [_]))

(defn biseq [start fwd bwd]
  (reify
    clojure.lang.ISeq
    (first [this] start)
    (next [this] (biseq (fwd start) fwd bwd))
    (more [this] (.next this))
    (seq [this] this)
    Spinnable
    (spin [this] (biseq start bwd fwd))))

(extend-type clojure.lang.LazySeq
  Spinnable
  (spin [this] (spin (seq this))))

(defn spin [seq]
  (.spin seq))

(let [s (biseq 0 inc dec)]
  (println (take 10 s))
  (println (take 10 (spin s)))
  (println (take 20 ((fn f [s]
                       (lazy-seq (cons
                                   (first s)
                                   (f (rest (if (zero? (rem (first s) 5))
                                              (spin s)
                                              s)))))) s))))
;(0  1  2  3  4  5  6  7  8  9)
;(0 -1 -2 -3 -4 -5 -6 -7 -8 -9)
;(0 -1 -2 -3 -4 -5 -4 -3 -2 -1 0 -1 -2 -3 -4 -5 -4 -3 -2 -1)

and chousers version

(defprotocol Spinnable
  (spin [this] "Return a seq walking the opposite direction as this"))

(defn iter-bi [x f b]
  (reify
    Spinnable
    (spin [_] (iter-bi x b f))
    clojure.lang.ISeq
    (first [_] x)
    (more [_] (iter-bi (f x) f b))
    (next [this] (rest this))   ; same as rest since all iter-bi's are infinite seqs
    (seq [this] this)
    (equiv [_ _] false)))       ; cheater!

(extend-type clojure.lang.LazySeq
  Spinnable
  (spin [this] (spin (seq this))))

(->> (iter-bi 0 inc dec)
     (drop 4)
     spin
     (take 10))
;=> (5 4 3 2 1 0 -1 -2 -3 -4)

fliebel: Given the fact that Clojure-in-Clojure is possible, it is thus possible to define lazy seqs. Would it also be possible to define a lazy seq where the 0th element is actually in the middle of an infinite lazy seq extending in both ways? so like (iterate inc) with negative number included :)

chouser: ISeq only provides an API for walking forwards. You'd need a different interface for walking the other direction.

fliebel: Hrm :( that means I'll be unable to use with with existing functions, right?

chouser: well, what existing function do you intend to call to get item before 'first'?

fliebel: Dunno‚ I mean, if I'm not using ISeq, I can’t even use first, right?

chouser: well, your theoretical bidirectional lazy seq could implement ISeq for going forward

fliebel: Oh wait, Java, so I can implement ISeq and add my own interface and fns for going the other way.

fliebel: How about this‚ you define a lazy seq like iterate but with 2 fns, for going in either direction. then you call spin on it to change which direction you're seqing over :)

chouser: hm, interesting.

fliebel: So, what Clojure bits do I need to implement that?

chouser: a deftype and ... a function?

fliebel: *looks up deftype*

chouser: actually, just a deftype would do, though it's not quite as pretty.

fliebel: I don't think I get it.

chouser: (rseq (rest (drop 10 (IterBi. 0 inc dec)))) ;=> (5 4 3 2 1 0 -1 -2 -3 -4 -5 ...)

chouser: so that's what you wanted, right? interesting idea.

fliebel: I understand how I can do without the fn, but I don't understand deftype.

chouser: actually, you don't need deftype either. you could use reify or proxy if you prefer, and a normal function.

fliebel: I never worked with those 1.2 things before. So it's going to be a lot of learning :)

chouser: so... how shall we proceed? You want to see my code or should I try to drop hints or ask leading questions?

fliebel: the later one :) extended with my guessing and reading

fliebel: So I basically need to look up the ISeq interface and overwrite a few methods?

chouser: ok, so how would you use deftype or reify to implement an ISeq that just repeats the same value over and over

fliebel: Where can I find the ISeq interface?

chouser: http://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/ISeq.java

fliebel: I guess it'd be (reify clojure.lang.ISeq (first [this] 1)) but that fails

fliebel: Do I need to implement all of them? Or can I just use a concrete subclass of ISeq?

chouser: just implement the ones you need for now. What happens when you call 'first' on your reify there?

fliebel: I get java.lang.AbstractMethodError when defining it, because ISeq is an abstract interface.

chouser: are you sure that's why?

fliebel: Hm, first works, but just defining it fails.

chouser: Ah, repl evaluates

chouser: yep, and after E is P

fliebel: ?

chouser: REPL

fliebel: ah

chouser: and printing a seq means walking it an printing each of the items. we're not ready for that.

fliebel: I see

chouser: so, (first ...) works. what about (next ...)?

fliebel: let me try..

fliebel: What about more and cons?

chouser: ignore them for now.

fliebel: yea, but not doing second on it still gives me the error.

chouser: it's a shame it doesn't tell you the specific method it's trying to call that fails.

fliebel: It doesn't even give a line or file.

chouser: right, so use (.printStackTrace *e)

fliebel: But seq isn't in ISeq, or is that up in Seqable?

chouser: Yea, but you can put it under ISeq in your reify

fliebel: I know, but I need to know it’s interface.

chouser: http://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Seqable.java#L16

fliebel: right‚ Now I made it return a list. It prints that and *then* gives the same error.

chouser: yeah. so that final error is it calling equiv. not sure why

fliebel: hm, Do we need to fix that?

chouser: probably not

fliebel: (reify clojure.lang.ISeq (first [this] 1) (next [this] this) (seq [this] (list 1 2 3)))

chouser: do you have an ISeq handy already for your 'seq' to return?

fliebel: this?

chouser: what happens when you try it?

fliebel: I guess it'll print a lot of 1s

fliebel: okay, it does :)

chouser: good job!

fliebel: but take 10 doesn't work yet

chouser: take is calling your (unimplemented) rest method

fliebel: I keep forgetting the difference between next and rest. There aren’t any docstrings on ISeq :(

chouser: the difference between more (called by clojure.core/rest) and next only matters when being very careful about nearly vs. complete laziness.

ohpauleez: new rest is almost always what you want, except when you don't

chouser: that is, we don't really care much. So what happens if your 'more' is the same as your 'next'?

fliebel: Works :)

chouser: congrats!

fliebel: I know how to reverse it :D

chouser: :-P

chouser: ok, but you want to apply a function instead of just repeating, right? how would you do that?

fliebel: Hey, my ISeq doesn't implement clojure.lang.Reversible

chouser: hm, what are going to do about that?

fliebel: I don't think reify allows me to define multiple ancestors :(

fliebel: I would not have led you astray

fliebel: Can I nest reifies? :P

chouser: yes, but that's not what you want here.

fliebel: Nope :(

chouser: according to the deftype docs, "Each spec consists of..."?

fliebel: ah...

fliebel: I added clojure.lang.Reversible (rev [this] this)

fliebel: (. rev (rseq)) what is this?

chouser: ancient syntax. the same as (.rseq rev)

fliebel: lol

fliebel: Can we go somewhat more down the inheritance tree to find something that covers the basics? Or do we need everything customized anyway?

chouser: reify doesn't let you inherit from anything but pure interfaces.

fliebel: So what is next?

fliebel: I'm now struggling with where to keep state and how to modify things without an endless recursion.

chouser: yep. hint: fn args are state

== diner & work pause ==

fliebel: sure :) I nearly got it working :)

chouser: ok, back. That's great. what have you got?
fliebel:
(ns biseq) (defprotocol Spinnable (spin [_])) (defn biseq [start fwd bwd] (reify clojure.lang.ISeq (first [this] start) (next [this] (biseq (fwd start) fwd bwd)) (more [this] (.next this)) (seq [this] this) Spinnable (spin [this] (biseq start bwd fwd)))) (extend-type clojure.lang.LazySeq Spinnable (spin [this] (spin (seq this)))) (defn spin [seq] (.spin seq)) (let [s (biseq 0 inc dec)] (println (take 10 s)) (println (take 10 (spin s))) (println (take 20 ((fn f [s] (lazy-seq (cons (first s) (f (rest (if (zero? (rem (first s) 5)) (spin s) s)))))) s)))) ;(0 1 2 3 4 5 6 7 8 9) ;(0 -1 -2 -3 -4 -5 -6 -7 -8 -9) ;(0 -1 -2 -3 -4 -5 -4 -3 -2 -1 0 -1 -2 -3 -4 -5 -4 -3 -2 -1)
chouser: nice!

fliebel: thanks :) You're a good teacher :)

chouser: I think that's the same as what I have. but you said "nearly"?

fliebel: The problem I had was that first was defined as the first of this, but I couldn't do (first this) of course.
chouser: if you want to "check" your answer:
(defprotocol Spinnable (spin [this] "Return a seq walking the opposite direction as this")) (defn iter-bi [x f b] (reify Spinnable (spin [_] (iter-bi x b f)) clojure.lang.ISeq (first [_] x) (more [_] (iter-bi (f x) f b)) (next [this] (rest this)) ; same as rest since all iter-bi's are infinite seqs (seq [this] this) (equiv [_ _] false))) ; cheater! (extend-type clojure.lang.LazySeq Spinnable (spin [this] (spin (seq this)))) (->> (iter-bi 0 inc dec) (drop 4) spin (take 10)) ;=> (5 4 3 2 1 0 -1 -2 -3 -4)
fliebel: this is your version?

chouser: yep

fliebel: very close :) Only you have equiv defined and next uses more. Does it matter if you use (.more method call) or rest?

chouser: not really. yours is probably faster

but duplicates half a line of code. :-)

fliebel: yea... :(

chouser: so take your pick. what you've got is good.

fliebel: One problem I have is that drop 5 returns a lazy seq which I can't reverse.

chouser: yeah, you see how I work around that in my example?

fliebel: rest, I used seq to do the job

chouser: oh! smart!

chouser: interesting -- I hadn't considered that, and wouldn't have been sure it would work. nicely done.

fliebel: I wasn’t sure it would work, but seq is magic you know...

fliebel: One thing I haven't figured out yet is how to make a seq that goes 1 2 3 4 5 4 3 2 1, so changes direction in the middle.

chouser: ((fn f [s] (lazy-seq (cons (first s) (f (rest (if (zero? (rem (first s) 5)) (rseq (seq s)) s)))))) (iter-bi 1 inc dec))

fliebel: I'm going to try and understand :)

chouser: BTW, I wanted to call attention to my comment on rseq. I think we're abusing it here and probably ought to define a new protocol instead. (defprotocol Spinnable (spin [_])) or something.

fliebel: Okay. that simple?

chouser: yep

fliebel: So now in reify I replace reversible with spinnable?
chouser: there, now you don't have to call seq manually on lazy-seqs:
(defprotocol Spinnable (spin [this] "Return a seq walking the opposite direction as this")) (defn iter-bi [x f b] (reify Spinnable (spin [_] (iter-bi x b f)) clojure.lang.ISeq (first [_] x) (more [_] (iter-bi (f x) f b)) (next [this] (rest this)) ; same as rest since all iter-bi's are infinite seqs (seq [this] this) (equiv [_ _] false))) ; cheater! (extend-type clojure.lang.LazySeq Spinnable (spin [this] (spin (seq this)))) (->> (iter-bi 0 inc dec) (drop 4) spin (take 10)) ;=> (5 4 3 2 1 0 -1 -2 -3 -4)
fliebel: What? You can just add behavior to existing types? Am I in #ruby?

chouser: nope. in clojure we do it without monkey patching or adapter classes. :-)

chouser: I mean "nope" you're not in #ruby. Of course you can write new functions that take existing classes as args that then do their own thing with them.

fliebel: But you just added extra behavior to LazySeq, right?

dnolen: it's a properly namespaced extension tho. It's not visible to other namespaces.

chouser: it certainly looks like I did. In this case we could have written spin like: (defn spin [x] (if (instance? LazySeq x) ...))) right?

fliebel: I think so...

chouser: it's our function, we just defined it with defprotocol which allows us to add new cases on the fly

fliebel: yay

chouser: essentially. Except then rhickey sprinkled it with magic dust to make it very fast.

fliebel: :)

chouser: it's good to know not everyone knows about this yet -- it's my topic at Strange Loop

fliebel: oh, I feel special now :)

chouser: good I'm glad! But... why?

fliebel: Because I know something not everyone knows ;)

chouser: ah, good!

fliebel: Except that I'm not o sure yet what exactly you just said. Only that it's magic and fast. Two things I like especially.

chouser: Here are the basics: http://clojure.org/datatypes

fliebel: I tried to read that this morning, but maybe with my new knowledge I can understand it.

ohpauleez: Joy of Clojure has an awesome section on proxy, reify, defprotocol, and defrecord. I reference it often to judge if I'm abusing something

chouser: thanks. didn't want to say it myself. :-)

ohpauleez: At the point you're at in Clojure (based on what I follow in here), you'd get a lot of mileage out of it.

ohpauleez: and the biggest problem I had in clojure was deciding, "when"

fliebel: “when”?

ohpauleez: Joy of Clojure it helped me a lot with that

ohpauleez: In Clojure, you have this whole array of functionality. Take concurrency

we all have that little chart burned in our brains by now

we know vars and atoms and agents

but when should I be using promises over futures, or when am I abusing a future and should really be using an agent

when should I use protocols and not some other small hack

fliebel: right… I have that problem a lot indeed

ohpauleez: Joy of Clojure is structured by answering when, and for me, that pulled it all together.

chouser: that's very encouraging, thanks.

fliebel: wow, under 10 there is a whole set of "when to use … "

Say thanks to the man who makes your computer run

2010-09-20T00:00:00-07:00

Today I was browsing HackerNews and reading up on the latest cool projects including at least 3 buzzword technologies, when I suddenly realized how strange it was to go "ooh" and "aah" for weekend projects written in fancy high-level languages while we are ignorant to the people who get their hand dirty to make it possible in the first place.

That is when I decided to find the authors of my sound card driver and thank them for their amazing work. I run Ubuntu now and then on my iMac, but the sound never worked for me, until recent versions of Ubuntu shipped the correct ALSA driver.

I encourage you to do the same.

Go to a technology you use to write awesome applications or the solution to a problem you had.
Look for the dependencies, and continue to drill down until you find a project that makes you blink your eyes ands say "wow, I didn't know this was involved."
Thank the author. Really, I'm serious!

I had to find out which sound card I have, what the driver for that was, and finally who the authors are. It turns out It's the Intel HDA driver in the alsa-driver package. The source file contained their email addresses.

Did you know Node.js uses V8, which uses Scons, which uses Python, which uses readline, which uses ncurses, which are all compiled using GCC or LLVM?
Did you know you can use your package manager to see the dependencies of a project?
Did you realize your package manager installs hundreds of dependencies and dependencies of dependencies to run something like Python?
Consider who is more awesome, the guy who writes a Twitter app in Node, the guy who made Node, or the guy who made Node possible.

Twitter OAuth for Open Source clients

2010-09-14T00:00:00-07:00

After reading these two articles about how to compromise Twitter tokens from a client and about how Twitter is abusing OAuth secrets and screwing OS clients by blocking their tokens, I came up with the following plan.

key = "your key"
secret = "your secret"

try
    request = sign(twitter.com/verify_credentials, key, secret)
    request.post()
except 401
    key = "BigFish key"
    secret = "BigFish secret"

I really love Twitter and the Twitter API, but it is obvious by now that Twitter does not care about small OSS projects.

I propose small Open Source Twitter clients to use tokens from the big fish as a fallback for their own tokens. This will allow small clients to continue to work after their tokens are blocked(albeit under the big fish name), unless Twitter blocks their own clients as well or faces the fact that their approach doesn't work.

A more politically correct option would be to write an OAuth Echo service that stores your tokens in a secure place, and accepts request without a secret. Take your pick.

5 minutes Lisp in Python

2010-09-10T00:00:00-07:00

I love both Python and Clojure, so after I read this gist about a Lisp in Ruby, I decided to craft my own in Python.

The idea is basically to abuse a few Python functions, namely Generators, Decorators and grouping to have a syntax that looks like Lisp with yield at the start of every statement.

It's all still Python, so you can mix and mach however you like, I sneaked a List Comprehension in the example, which fits the Lisp syntax quite well.

At the moment of writing, the Gist below is also #3 on HackerNews.

# 5 minutes Lisp in Python
# Pepijn de Vos <http://pepijndevos.nl>
#
# Inspired by 30 minutes Lisp in Ruby
# http://gist.github.com/562017
# 
# This Lisp does not read or parse anything at all.
# A generator and a Decorator are abused to run sexps.
#
# Usage:
# 
# Define a function decorated with @lisp and start every sexp with yield.
#
# The function names should be strings.
#
# Result is stored in fn name.
#
# Example below:
#

def lisp(fn):
    code = fn()
    val = code.next()
    while True:
        try:
            try:
                newval = getattr(__builtins__, val[0])(*val[1:])
            except AttributeError:
                newval = getattr(val[1], val[0])(*val[2:])

            val = code.send(newval)
        except StopIteration:
            return getattr(val[1], val[0])(*val[2:])

@lisp
def example():
  (yield 'join',
    ", #",
    (yield '__mul__',
      [(yield 'str', i) for i in
        (yield 'range', (yield '__add__', 5, 5))],
      2))

print example

Twitter in your inbox

2010-09-07T00:00:00-07:00

I have been working on this for quite a while, with initial plans to make it a service. But I have changed plans, and am releasing it to the world.

What have I been working on, you ask? Not that I like sliced bread, but it's at least as good as that: It solves some problems while it introduces some others. It's a proxy between Twitter and your email client that sits on your computer or a server translating mail-language(POP, SMTP) to Twtter-language(REST API).

This way you can read your tweets in your mail client, store them in folders, search though them, and do all the other stuff mail clients are good at. It even allows you to reply and forward(retweet in Twitter lingo) tweets.

Here is how to get it. You go to my GitHub repo, which lives at the link below, and pull/download the code.

http://github.com/pepijndevos/Twemail

The next step is to get the dependencies, which include Twisted, OAuth2 and Twitter-Text-Python. You might try to use setup.py for that, but I'm not sure it works.

Now you simply run "twistd -ny twemail.tac" in the Twemail directory. If everything went well you'll see some messages rolling by about starting services.

Now all that remains is to configure your mail client with the following information"

email: username@twitter.com
password: your Twitter password
POP server: localhost on port 1100
SMTP server: localhost on port 2500

Here is how it works.

You can send messages by mailing post@twitter.com
Replying is done by hitting reply and replacing the subject line.
Retweets are done by forwarding to the sender of the tweet

In the future I also want to handle attachments and IMAP, but this is currently not included.

Clojure Rock-Paper-Scissors using a Markov chain

2010-09-04T00:00:00-07:00

A long time ago I saw a challenge on a Python forum about writing a Rock-Paper-Scissors bot that performs better than a random bot.

As strange as it might seem, Rock-Paper-Scissors is anything but random. As long as it's played by people, they'll always try to come up with a winning move. I've been told there are even championships with this game.

So while trying to improve my Clojure skills I came up with the idea of using a Markov chain to write a Rock-Paper-Scissors bot.

Markov chains - as I recently learnt - are a random process where every step is only based on the previous. I'm not a psychologist, but when I play Rock-Paper-Scissors, I usually go like "I just lost using paper, so lets try rock" or "I won using rock, lets do it again", so I think it suits the challenge.

I use this map to define the probability of certain actions:

{:won {:same 1, :inferior 0, :superior 0}, :lost {:same 0, :inferior 1, :superior 0}, :draw {:same 0, :inferior 0, :superior 1}}

So with this example data, If I used rock, and won. I'll choose rock again, now I lose, so I choose the item inferior to rock: scissors. Draw, so I'll choose the item superior to scissors: rock.

The roulette function takes a map like {:same 1, :inferior 2, :superior 4} and returns one of the keys with the probability of the value. Calling this 7 times should approximately return 4 times :superior, 2 times :inferior and :same only once.

The guess function uses the Markov map to predict my action, and take the item that wins from mine.

Then there is the check function, which just returns if I won or lost based on the 2 items, and the relation fn which does the same for determining which of the 2 items is better.

Then there is the play function which focuses on getting your input, comparing results, and updating the Markov data to represent your actual actions.

You should update the map with the map that is printed at the end of the game, to improve the learning process of my bot. If you collect a lot of data, it could be interesting to send it in, so I can improve my bot to have a proper data set to start with.

Have fun!

(ns markov-stone)

;{:won {:same 0, :inferior 0, :superior 0}, :lost {:same 0, :inferior 0, :superior 0}, :draw {:same 0, :inferior 0, :superior 0}}

(defn roulette [slices]
  (let [slices (sort-by val slices)
        total (reduce + (vals slices))
        r (rand total)
        cumulative (map vector (keys slices) (reductions + (vals slices)))]
    (first (first (drop-while #(< (last %) r) cumulative)))))

(defn guess [markov prev state]
  (let [inferior {:s :p, :p :r, :r :s}
        superior (clojure.set/map-invert inferior)
        pred (->> markov
                  state
                  roulette)]
    (case pred
          :same (superior prev)
          :superior (inferior prev)
          :inferior prev)))

(defn check [user cpu]
  (if (= user cpu)
    :draw
    (if (= user (cpu {:s :p, :p :r, :r :s}))
      :lost
      :won)))

(defn relation [prev cur]
  (if (= prev cur)
    :same
    (if (= prev (cur {:s :p, :p :r, :r :s}))
      :superior
      :inferior)))

(defn play [markov prev state score]
  (println "Choose your weapon[r/p/s]!")
  (if-let [user (#{:r :p :s} (keyword (read-line)))]
    (let [names {:r "ROCK" :p "PAPER" :s "SCISSORS"}
          cpu (guess markov prev state)
          result (check user cpu)
          score (case result
                      :won (inc score)
                      :lost (dec score)
                      :draw score)]
      (println (names cpu))
      (println "Game is" (name result) "- Score:" score)
      (recur (update-in markov [result (relation prev user)] inc) user result score))
    (do
      (println "Invalid input; Final score:" score)
      markov)))

(println (play {:won {:same 20, :inferior 17, :superior 16}, :lost {:same 16, :inferior 13, :superior 38}, :draw {:same 23, :inferior 18, :superior 21}} :r :draw 0))

Restore the old iTunes icon

2010-09-02T00:00:00-07:00

Since the last keynote, where Apple announced iTunes 10, I've heard a lot of complaints about the vertical close buttons and the new icons.

For the close buttons you should keep an eye on http://hints.macworld.com/, but at least I solved the problem of the ugly icon.

Before updating, save the old icon by right-clicking on iTunes, and then on "Show package contents", then browse to Content/Resources and copy iTunes.icns somewhere safe.

Now upgrade iTunes, and do the same thing in reverse, so you'll overwrite the new iTunes.icns. Below is a video and the original icon, in case you are to late to copy it yourself. You might need to start and stop iTunes a few times, or even reboot to se the "new" icon.

[[posterous-content:xcKOWzC5uTfI2Lgl3rky]]

[[posterous-content:nGbvjkjyCfDKW1Svw1lV]]

[update] Posterous doesn't like my icon. Download it from Dropbox instead: http://dl.dropbox.com/u/10094764/iTunes.icns

Also, about the control buttons: defaults write com.apple.iTunes full-window -1

http://hints.macworld.com/article.php?story=20100901223846748

A smile for web standards

2010-09-01T00:00:00-07:00

We all love web standards. While I was implementing a little easter egg(which are also very lovable) on http://coverontwerp.nl/ with a good deal of CSS3 love, I thought it would be nice to turn as many clicks as possible into a smile for web standards, a mini ACID test if you will.

Bellow you'll find a CSS snippet that, when included on your site, will flash a smile for every link clicked in a supporting browser. It is totally unobtrusive, and only blinks the moment anyone clicks a link, thus spreading some positive feelings to all supporters of web standards.

Example implementations can be found here and on http://coverontwerp.nl/. Show some standards love, by including the snippet below on your site!

a:active:after {
	display: inline-block;
	margin-left: 5px;
	font-weight: bold;
	content: ":)";
	-webkit-transform: rotate(90deg); 
	-moz-transform: rotate(90deg);
	-o-transform: rotate(90deg);
}

Other selectors, styling and ASCII arts are encouraged!

Search Google in running browser

2010-08-23T00:00:00-07:00

After reading this hint, I wanted to get it working with my previous script to open links in the currently running browser.

I did not use OnMyCommand, but instead just made an Automator service. Here is how:

Open Automator and create a service
Set the type to text
Add the ‘Run shell script’ action
Set the shell to Python and the input to ‘as arguments’
Paste the script below

The script(this text serves only to satisfy Posterous’s parsing engine, which is seriously pissing me off):

import sys
import urllib
import webbrowser
webbrowser.open("http://www.google.nl/search?q=" + urllib.quote(sys.argv[1]))

You need to have my previous script installed to do the actual opening of the url. With some more trickery, it is even possible to add this service as a keyboard shortcut. I suggest you Google around a bit, and comment if you find something revolutionary.

JS server benchmark: Node.js & Rhino; Part 2

2010-08-22T00:00:00-07:00

After having tested the pure JS speed of Node and Rhino in my previous post, I wanted to see some actual numbers for serving stuff.

Long story short, I grabbed the "Hello world!" examples from both Node.js and Rhino running on Jetty and ran 'ab' on them.

I was not able to do high-concurrency tests on my Mac, because Jetty gave up around 100, and Node somewhere around 200. I think I'm having the issue described here.

So here is the graph. I don't have much experience with benchmarking, but again, it is obvious that Node.js is in fact quite a lot faster than the Rhino/Jetty combo.

Remove the bookmark button in Safari

2010-08-20T00:00:00-07:00

A while back I found this comic, comparing Safari with a bike, and Firefox with a car with a lot of stuff attached to it.

But in reality, since Safari 5 supports extensions, it starts to look more and more like this illustration from a Dutch children book.

Now that I have lots of goodies and a nice Delicious bookmark button, I wanted to get rid of the bookmark button embedded in the navigation bar. This video shows how to do it.

[[posterous-content:XUORk3FShLrdFY9GRo7d]]

If someone knows how to embed the Delicious button in the navigation bar, I would be very pleased.

JS server benchmark: Node.js & Rhino

2010-08-19T00:00:00-07:00

Background

I’m playing with the idea of writing a little web framework in JavaScript using jQuery on both the server and the browser. To run jQuery on the server, you need — except for JavaScript — some magic, such as env.js and jsdom.

Env.js is a complete browser environment that allows you to run almost all browser JS code(like jQuery) on Rhino, a JavaScript engine written in Java. It is very complete, but tied to Rhino.

Jsdom is a younger project implementing only the DOM in CommonJS, though it is written for Node.js, an event-based network server based en Googles V8 JavaScript engine.

While I know some Java, and env.js does all I need, I got the impression that Node.js offers more in terms of speed and is more suitable for a web framework.

Since jsdom does not do AJAX and events, I’m facing the choice between using Rhino, or porting env.js to Node. I started to investigate the possibilities, and here is what I found.

V8 is fast
env.js has only ±100 lines of Rhino specific code.

Running the benchmark

Searching for benchmarks comparing server side JavaScript, I found none. I did find the JS benchmark Google uses to test V8, which might of course be skewed towards V8. I tried them anyway.

Node.js

Installing Node was a breeze, but getting the V8 benchmark to run on a V8 based framework was not so easy.

For some unknown reason the V8 benchmark uses load() to load everything together, while Node uses require(), with a slightly different meaning, which made simply substituting them impossible.

Stride on #node.js came up with the solution, consisting of concatenating the files together with some shell magic:

cat base.js crypto.js deltablue.js earley-boyer.js raytrace.js regexp.js richards.js splay.js run.js > v8bench.js

Note that the order of the files was important, otherwise a simple “cat *.js” would have sufficed.

Now all that remained was commenting out the original imports and defining a print function:

print = require('sys').log

Now this is the result:

$ node v8bench.js
19 Aug 16:13:25 - Crypto: 2434
19 Aug 16:13:26 - DeltaBlue: 3150
19 Aug 16:13:28 - EarleyBoyer: 12752
19 Aug 16:13:29 - RayTrace: 6230
19 Aug 16:13:30 - RegExp: 2287
19 Aug 16:13:31 - Richards: 1547
19 Aug 16:13:33 - Splay: 8879
19 Aug 16:13:33 - ----
19 Aug 16:13:33 - Score (version 5): 4090

Note the last number(average score) and the length of the command(2 parts).

Rhino

Installing Rhino was equally easy(just “port install rhino nodejs”)

Then the trouble started… First I had to figure out how to run Rhino, because it doesn’t come with its own command, like Node does. Turns out it’s “java -jar /path/to/js.jar”

After that, I ran ‘run.js’ from the v8 benchmark, and it just worked! Without modification! … Until it gave java.lang.OutOfMemoryError. Some Googling reveals that unless you have a memory leak in your app, you can increase the memory limit, which I did.

Here are the results for Rhino:

$ java -Xmx265m -jar js.jar run.js
Richards: 20.4
DeltaBlue: 138
Crypto: 120
RayTrace: 253
EarleyBoyer: 248
RegExp: 59.1
Splay: 279
----
Score (version 5): 120

Again note the score and the length of the command(5 parts). The -Xmx265m is for raising the memory limit.

My verdict

Sure, Node’s score is about 34 times the score of Rhino, but I talked to the author of Claypool, and he used Rhino for several production sites, without performance problems. It’s a measurement without a scale, so I can’t judge if it will matter to me.

With the speed sorted out, there is just the general style and comfort of both systems. Node.js is a very young C++ project, while Rhino has been around since the Netscape ages and is written in Java.

While I’m attracted to the speed and freshness of Node, I know more Java than I know C++, and there are tons of Java libs available to Rhino. And then there is env.js, which needs to be ported to Node, if I decide to use that.

Besides the porting and library issue, there is also hosting. Because Node is so new, you’d have to rent your own VPS, while Rhino can run in any Java servlet container, and even Google App Engine with the help of Rhino-for-webapps

So, my verdict? Undecided: As always, it depends on what you want, and I’m not yet sure what I want.

[update]

FireFox gets only 394 in the same test, so Rhino is probably not that bad.

I would be very grateful if anyone could give me some information on the CommonJS load/require and print problem or ways to speed up Rhino.

Twisted pop3 example server

2010-08-18T00:00:00-07:00

Remember last time when I showed you the Twisted SMTP server? This time it's a POP3 server. I think it is more common to write SMTP servers as a part of a project(to send notifications and stuff), but I needed a POP3 server, so I think there might be others who need a POP3 server as well.

The process is rather similar to SMTP, only you need to implement the IMailbox interface. This is usually done with some sort of file or db based solution, but I used a simple list in this case. You might want to have a look at the mailbox module for a serious implementation.

Below you'll find a bare bones POP3 server, which can be run with `twistd -ny pop3.tac` and below that a sample telnet session, by running `telnet localhost 1230` in another terminal.

Connected to localhost.
Escape character is '^]'.
+OK <20100818095058.3878.894125537.0@pepijn-de-voss-imac.local>
user guest
+OK USER accepted, send PASS
pass password
+OK Authentication succeeded
stat
+OK 20 1020
retr 1
+OK 51
From: me
To: you
Subject: A test mail

Hello world!
.
retr 40
-ERR Bad message number argument
quit
+OK 
Connection closed by foreign host.

"""
An example pop3 server
"""

from twisted.application import internet, service
from twisted.cred.portal import Portal, IRealm
from twisted.internet.protocol import ServerFactory
from twisted.mail import pop3
from twisted.mail.pop3 import IMailbox
from twisted.cred.checkers import InMemoryUsernamePasswordDatabaseDontUse
from zope.interface import implements
from itertools import repeat
from hashlib import md5
from StringIO import StringIO

class SimpleMailbox:
    implements(IMailbox)

    def __init__(self):
        message = """From: me
To: you
Subject: A test mail

Hello world!"""
        self.messages = [m for m in repeat(message, 20)]


    def listMessages(self, index=None):
        if index != None:
            return len(self.messages[index])
        else:
            return [len(m) for m in self.messages]

    def getMessage(self, index):
        return StringIO(self.messages[index])

    def getUidl(self, index):
        return md5(self.messages[index]).hexdigest()

    def deleteMessage(self, index):
        pass

    def undeleteMessages(self):
        pass

    def sync(self):
        pass


class SimpleRealm:
    implements(IRealm)

    def requestAvatar(self, avatarId, mind, *interfaces):
        if IMailbox in interfaces:
            return IMailbox, SimpleMailbox(), lambda: None
        else:
            raise NotImplementedError()

portal = Portal(SimpleRealm())

checker = InMemoryUsernamePasswordDatabaseDontUse()
checker.addUser("guest", "password")
portal.registerChecker(checker)

application = service.Application("example pop3 server")

f = ServerFactory()
f.protocol = pop3.POP3
f.protocol.portal = portal
internet.TCPServer(1230, f).setServiceParent(application)

Check multiple twisted.cred checkers for a valid login

2010-08-01T00:00:00-07:00

This is a snippet of a credential checker I wrote to authenticate a user
first via my DB, and if that fails via Twitter. It helped me to speed up
login, after the initial Twitter access tokens are stored.

In theory, this snippet could be used to authenticate against any number
of checkers, providing different interfaces. Only if all of them fail,
it returns an error.

from twisted.cred.checkers import ICredentialsChecker
from twisted.cred import error, credentials
from zope.interface import implements
from collections import deque

class CascadingChecker:
    """
    Check multiple checkers untill one succeeds.
    Else raise UnauthorizedLogin.
    """

    implements(ICredentialsChecker)
    credentialInterfaces = set()
    
    def __init__(self):
        self.checkers = []
        self.checked = []
    
    def registerChecker(self, checker):
        self.checkers.append(checker)
        self.credentialInterfaces.update(checker.credentialInterfaces)
    
    def _requestAvatarId(self, err, queue, credentials):
        try:
            ch = queue.popleft()
        except IndexError:
            raise error.UnauthorizedLogin()
        
        d = ch.requestAvatarId(credentials)
        return d.addErrback(self._requestAvatarId, queue, credentials)
    
    requestAvatarId = lambda self, credentials: self._requestAvatarId(None, deque(self.checkers), credentials)

from twisted.trial import unittest
from checkers import CascadingChecker
from twisted.cred.credentials import UsernamePassword
from twisted.cred.checkers import InMemoryUsernamePasswordDatabaseDontUse
from twisted.cred.error import UnauthorizedLogin
from twisted.cred.credentials import IUsernameHashedPassword, IUsernamePassword

class CascadingTest(unittest.TestCase):
    
    def setUp(self):
        ch1 = InMemoryUsernamePasswordDatabaseDontUse()
        ch2 = InMemoryUsernamePasswordDatabaseDontUse()
        self.cach = CascadingChecker()
        
        ch1.addUser('foo', 'bar')
        ch1.addUser('boo', 'far')
        ch2.addUser('for', 'bao')
        
        self.cach.registerChecker(ch1)
        self.cach.registerChecker(ch2)
    
    def testInterfaces(self):
        self.assertEquals(self.cach.credentialInterfaces.difference((IUsernameHashedPassword, IUsernamePassword)), set())
    
    def testLoginFirstChecker(self):
        user = UsernamePassword('foo', 'bar')
        return self.cach.requestAvatarId(user)
    
    def testLoginSecondChecker(self):
        user = UsernamePassword('for', 'bao')
        return self.cach.requestAvatarId(user)
    
    def testLoginFail(self):
        user = UsernamePassword('steve', 'pswd')
        self.assertFailure(self.cach.requestAvatarId(user), UnauthorizedLogin)

XAuth(CamelCase), xAuth(iCase), XOAUTH(ALLCAPS); 3 distinct technologies, confusing?

2010-07-29T00:00:00-07:00

While thinking up names, Unix/open source people tend to come up with names containing ‘X’. I can almost hear them scream “‘X’, the new and free(as in beer, pizza and speech) ‘i’(as in iPhone, iPod, etc.)!”

My (possibly fictive) story goes like this: Some time ago, 3 separate development teams came together to find a name for their product.

The first team was making a secure, social, easy way of authenticating browsers. Since they where developing an open authentication protocol, and read loads of developer guidelines, they came up with XAuth, using proper CamelCasing, as one was supposed to.

The second team had a nice OAuth implementation running, but some people required using good ol' passwords, so they came up with the idea of requesting tokens using regular credentials, instead of complicated token exchanges. Since they wanted to be hip, and had a nice marketing department, they came up with xAuth, in line with the iDevice casing.

The third team thought it would be nice to apply the security of OAuth to email. Since this was way to cool to be called OAuth-for-email, they decided to add the cool ‘X’ in front of it. And as we all know(don’t we?), mail servers talk to each other in ALL CAPS, so it was a logical thing to call their system XOAUTH(although they’re not all that consistent about it).

Twisted SMTP server with authentication

2010-07-22T00:00:00-07:00

This is the next post about Twisted, this time about Twisted.mail. After my Twisted project is done I'm going to contribute all my findings in some examples, docstrings and maybe even a howto. This post mainly serves as my notebook for things to remember about SMTP in Twisted.

This is a twistd app taken from the Twisted examples section, and modified to support authentication. Below you'll find all my notes.

# Copyright (c) 2001-2004 Twisted Matrix Laboratories.
# See LICENSE for details.

# You can run this module directly with:
#    twistd -ny esmtpserver.tac


"""
A toy email server with authentication.
"""

from zope.interface import implements

from twisted.internet import defer
from twisted.mail import smtp
from twisted.mail.mail import MailService
# these challengers are located in imap4
from twisted.mail.imap4 import LOGINCredentials, PLAINCredentials

from twisted.cred.checkers import InMemoryUsernamePasswordDatabaseDontUse #except in examples and testing
from twisted.cred.portal import IRealm
from twisted.cred.portal import Portal

class ConsoleMessageDelivery:
    implements(smtp.IMessageDelivery)
    
    def receivedHeader(self, helo, origin, recipients):
        return "Received: ConsoleMessageDelivery"
    
    def validateFrom(self, helo, origin):
        # All addresses are accepted
        return origin
    
    def validateTo(self, user):
        # Only messages directed to the "console" user are accepted.
        if user.dest.local == "console":
            return lambda: ConsoleMessage()
        raise smtp.SMTPBadRcpt(user)

class ConsoleMessage:
    implements(smtp.IMessage)
    
    def __init__(self):
        self.lines = []
    
    def lineReceived(self, line):
        self.lines.append(line)
    
    def eomReceived(self):
        print "New message received:"
        print "\n".join(self.lines)
        self.lines = None
        return defer.succeed(None)
    
    def connectionLost(self):
        # There was an error, throw away the stored lines
        self.lines = None

class ConsoleSMTPFactory(smtp.SMTPFactory):
    def __init__(self, *a, **kw):
        smtp.SMTPFactory.__init__(self, *a, **kw)
        # make this factory make ESMTP servers
        self.protocol = smtp.ESMTP
    
    def buildProtocol(self, addr):
        p = smtp.SMTPFactory.buildProtocol(self, addr)
        # add the challengers from imap4, more secure and complicated challengers are available
        p.challengers = {"LOGIN": LOGINCredentials, "PLAIN": PLAINCredentials}
        return p

class SimpleRealm:
    implements(IRealm)

    def requestAvatar(self, avatarId, mind, *interfaces):
        # if we are authenticating a IMessageDelivery
        if smtp.IMessageDelivery in interfaces:
            # a tuple of the implemented interface, an instance implementing it and a logout callable
            return smtp.IMessageDelivery, ConsoleMessageDelivery(), lambda: None
        raise NotImplementedError()


def main():
    from twisted.application import internet
    from twisted.application import service    
    
    portal = Portal(SimpleRealm())
    # initiate a simple checker
    checker = InMemoryUsernamePasswordDatabaseDontUse()
    checker.addUser("guest", "password")
    portal.registerChecker(checker)
    
    a = service.Application("Console SMTP Server")
    internet.TCPServer(2500, ConsoleSMTPFactory(portal)).setServiceParent(a)
    
    return a

application = main()

If you want anything beyond a toy, you'll need to use ESMTP instead of SMTP
You need to have a portal with a realm and a checker set up to use the AUTH command
challengers are what provide the protocol for logging in, note that they are located in twisted.imap4
A portal takes one checker instance per authentication method, so you can have one for handling strange tokens, and one for regular username/password authentication, but not two for the same type.
Take a look at the source for InMemoryUsernamePasswordDatabaseDontUse, it is really simple to write your own checker.

Messing with the REPL with Twisted

2010-07-22T00:00:00-07:00

I love to try out things on the REPL(read eval print loop) before I write scripts with them. The trouble with Twisted is that everything returns deferreds, and after your start the reactor, you can’t do anything anymore.

All this is solved by this one single command. It gives you a REPL powered by the Twisted event loop. To make things even better: It prints the value of a deferred once it is finished!

python -m twisted.conch.stdio

Added hint: Put this in a file, do “chmod +x” on it and put it on your $PATH.

Pepijndevos.nl moved to Posterous

2010-07-20T00:00:00-07:00

After weighting the pros and the cons of moving to Posterous completely, I decided to do it. Wishful Coding has now completely moved to Posterous.

If you see anything broken, drop me a line or try the original blog, which will be available for a while at pepijn.cqhosting.nl.

I'd like to thank the awesome people of both Posterous and CQ Hosting(who used to host my Wordpress blog). From the initial export tot the switch of the domain took only a few hours.

The only thing that didn't go as planned are he images in old posts. They where still linked to pepijndevos.nl, which meant that switching the domain would break links.

I asked CQ Hosting to make my old blog available through some sort of address, which they did almost instantly. Now I could replace every instance of pepijndevos.nl with pepijn.cqhosting.nl, and keep all images and links intact.

My beautiful Wordpress Theme is gone for now, but I'll make my own Posterous theme later on. Watch out for some crazy CSS3 and HTML5!

My thoughts on iPhone reception

2010-07-20T00:00:00-07:00

In the press conference Apple gave on July 16 about the iPhone G4 reception issues, they basically said that the 4 or 5 bars you usually see where fake, and caused by a "bug" in the way signal strength is measured.

If I remember correctly, there was another "bug" with the reception prior to iOS 4, that was solved by a software update "improving" G3 reception.

I can't help thinking they just tweaked their formula to show high bars to save their asses, and this backfired on them with iOS 4.

Bring actual image data in sync with EXIF orientation

2010-07-20T00:00:00-07:00

While sorting out all my holiday photographs, I turned them to the right orientation using F-Spot photo manager. When I came home I imported them into Adobe bridge, sorted them all out using all sorts of labels, tags and ratings.

Everything was fine so far, until I looked at them with a bad viewer, or uploaded them to Facebook. F-Spot and Bridge happily adjusted the orientation in the EXIF meta-date, but left the actual image untouched.

Not all viewers check the EXIF data before displaying an image, so they display it in the original orientation, corresponding to orientation 1 in EXIF.

Now I was stuck with over a thousand neatly sorted and rotated photos, that I could not publish without rotating them all again with another application.

Luckily I was saved by this page: http://sylvana.net/jpegcrop/exif_orientation.html

If you run Mac and probably Linux as well, jpegtran is already installed.

Just download the C app and the Bash script, but them in the same folder and run this command to compile:

gcc -o jpegexiforient jpegexiforient.c

Now run these commands to make the files executable:

chmod +x jpegexiforient
chmod +x exifautotran

I had to change exifautotran to read ./jpegexiforient in 2 places instead of just jpegexiforient. Otherwise it'd say "command not found".

Now you can run "./exifautotran file.jpg" to convert a single file, or "./exifautotran /dir/*.jpg" to convert a whole directory of jpg files.

Twitter xAuth in Python using Twisted and Twitty Twister

2010-07-19T00:00:00-07:00

This is the first post in what will most likely be a series of posts related to Twitter, POP3, SMTP and Twisted. This post will be about asynchronous authentication with Twitter using xAuth.

xAuth is a simplified form of OAuth. Instead of doing the dance, you use credentials(like basic auth), but exchange them for an access token. You need to send a request to api@twitter.com to enable it.

This technique is also applicable to getting OAuth tokens, although that is more complicated.

We are going to use Twitty Twister, which is a Twitter lib for Twisted which supports signing OAuth requests. It does not, however, include code for obtaining an access token.

To obtain the token, it is suggested to use an external library. But since my app is 100% Twisted, and most OAuth libs blocking, I had a problem right there. We don’t want to have our nice reactor sit idle while we request our token, do we?

The solution is to use python-oauth to make the request, but just before we send it, you call request.to_postdata() and use twisted.web.client.getPage to send the request in an asynchronous manner.

My main resources where: http://dev.twitter.com/pages/xauth http://github.com/simplegeo/python-oauth2#readme http://twistedmatrix.com/documents/10.1.0/api/twisted.web.client.getPage.html

The resulting code(insert your key, secret, username and password):

from oauth import oauth
from twisted.web.client import getPage
from twisted.internet import reactor
from twittytwister import twitter
import cgi

url = "https://api.twitter.com/oauth/access_token"
consumer = oauth.OAuthConsumer("key", "secret")

def respond(tokens):
    params = cgi.parse_qs(tokens, keep_blank_values=False)
    key = params['oauth_token'][0]
    secret = params['oauth_token_secret'][0]
    print "got access"

    token = oauth.OAuthToken(key, secret)
    tweet = twitter.Twitter(consumer=consumer, token=token).update("Hello world! First tweet using xauth via #Twisted and #Python")
    tweet.addCallback(posted)

def posted(message):
    print "message:", message

request = oauth.OAuthRequest.from_consumer_and_token(oauth_consumer=consumer, http_method='POST', http_url=url,
    parameters = {
        'x_auth_mode': 'client_auth',
        'x_auth_username': 'username',
        'x_auth_password': 'password'
    }
)
request.sign_request(oauth.OAuthSignatureMethod_HMAC_SHA1(), consumer, None)

page = getPage(url, method='POST', postdata=request.to_postdata())
page.addCallback(respond)

print "running..."
reactor.run()

Twitter to text message is back!

2010-07-19T00:00:00-07:00

When you click the small phone icon, you can select to receive a text message on you phone of tweets of that user.

Twitter previously offered a feature to text you for any new tweet and to allow you to tweet by text, but I assume that was shut down when people started to have thousands of followers.

Trying out Posterous

2010-07-13T00:00:00-07:00

I received a message via Twitter that Posterous now features Wordpress in its switch campaign. Although I'm in Ireland at the moment, I did not hesitate to try it out!

This is just a message sent from Gmail, that will probably end up on several other places thanks to the autoposting feature in Posterous.

I will also attach an image, to see what will happen to it.

In the future I might make pepijndevos.nl the domain for pepijndevos.posterous.com, and make it my main website, but before I do that, I need to figure out how to port a few features of my old blog to the new Posterous blog.

A better Trash for Mac

2010-06-19T00:00:00-07:00

Different at least... I bothers me that the Mac Trash leaves hidden .Trash folders everywhere, including external drives. I wrote a few AppleScripts to move all Trash in one place and delete files completely. If you only want to remove trash from external drives, see my earlier post. This approach is still far from ideal, but it's interesting and it might inspire someone to do a better job than I did. The idea is to move files to a sparse image and delete that image when you need to empty the Trash. You can do this with Disk Utility and a bit of AppleScript.

To create the image, open Disk Utility in /Applications/Utilities, click New Image and create a Sparse Image.

I think it is nice to make the image a Stationary Pad, so you get a new Trash can every day for easier selective deletion. To do this, select the image and press CMD+i and select Stationary Pad. Now all that remains are the AppleScripts. Save those as Application Bundles named Trash and Delete respectively.

property image : POSIX file "/Users/pepijndevos/Trash/Trash.sparseimage" -- edit this
property mount : POSIX file "/Volumes/Trash" -- and this

tell application "Finder"
        if not (exists mount) then
                open image
                delay 5
        end if
        open mount
end tell

on open the_files
        tell application "Finder"
                if not (exists mount) then
                        open image
                        delay 5
                end if
                repeat with the_file in the_files
                        do shell script "mv -f " & quoted form of POSIX path of the_file & " " & quoted form of POSIX path of mount
                end repeat
        end tell
end open

on open the_files
        repeat with the_file in the_files
                do shell script "rm -rf " & quoted form of POSIX path of the_file
        end repeat
end open

Now drag these to the Dock for easy access to the Trash and Delete functions. You can just drag files on them to trash or delete them. Another option of course is to have only the Delete script and a regular folder in your Dock, but that is not half as fun, is it?

Updating to Wordpress 3.0

2010-06-17T00:00:00-07:00

I'm trying to upgrade this blog to WordPress 3.0. While I skipped 2.9.2 because it was only a minor fix, this time there are some cool new features, which I'm not going to tell about, use Google! This theme used the default comment template in WordPress 2.9, but unfortunately this has been removed or broken. To fix this, I had to use the comments.php from the new and shiny twentyten theme. I copied their comments.php over to my theme, but unfortunately this only made matter worse. It uses a few functions defined in functions.php. Rather than trying to copy those specific functions over to my functions.php, I just included theirs:

include(TEMPLATEPATH . "/../twentyten/functions.php");

Can you think of a more ugly solution? I'll fix it later. There might be more bugs to be discovered. Do you see anything out-of-order on this blog? Please leave a comment.

Wordpress RSS widget in a post/page

2010-06-10T00:00:00-07:00

I'm currently writing a theme for a friend who is going to coach photographers. He wants to have a page that lists RSS feeds from all those photographers. One part of the puzzle is easily solved by using Yahoo! Pipes to combine those feeds, the other part isn't that easy to solve. WordPress comes with a standard RSS widget, but it's only a widget, not a shortcode. There are a bunch of plugins that either offer a RSS widget or turn feeds items into posts. What I did is write my own plugin in only 10 lines of actual code, utilizing the shortcode API and the built-in RSS widget.

Save and upload the snippet below to your plugins folder
Activate it in the WP admin
Add the [rsstag] shortcode to your post or page
Add the feed url like this: [rsstag url="http://pepijn.cqhosting.nl/feed/"]
Save!

You can optionally add one or more of show_summary=1, show_date=1 or show_author=1 to add the respective meta-data to every entry.

php
/*
Plugin Name: Wordpress RSS shortcode
Plugin URI: http://pepijn.cqhosting.nl/2010/06/wordpress-rss-widget-in-a-post-page
Description: This plugin fetches an RSS feed as a shortcode
Author: Pepijn de Vos
Version: 1.0
Author URI: http://pepijn.cqhosting.nl
*/

function rsstag($atts) {
        $atts = shortcode_atts(array(
                'url' => get_bloginfo('rss2_url'),
                'show_author' => 0,
                'show_date' => 0,
                'show_summary' => 0
        ), $atts);
        //var_dump($atts);
        wp_widget_rss_output($atts['url'], $atts);
}
add_shortcode('rsstag', 'rsstag');
?>

I will upload this to WordPress.org soon.

Wordpress MU in Resin with Quercus

2010-05-22T00:00:00-07:00

While researching Quercus today, I ran into some trouble running the new integrated Multi Site feature of Wordpress 3.0(previously a separate project known as Wordpress MU or WPMU for short) on Resin. Quercus is a PHP implementation written in Java that runs on any Java Servlet Container, including but not limited to Jetty, Tomcat or Resin. I took Resin for it was the easiest to setup and includes Quercus. Normally one would just use Apache and mod_php, but I also want to run Jython and Clojure projects, so I'm looking for a solution to combine these. Wordpress MU allows me to run multiple blogs on a single installation(as seen on wordpress.com), which I want to use for hosting websites for my clients. The problem with WPMU is that to map the different urls to one installation, some tricky mod_rewrite code is required for Apache, but since I'm using Resin here, I'll have to find another solution. There is a post on the Quercus wiki describing the process of setting up a Multi Site Wordpress installation, but this is for an old version of WPMU(before the merge) and an old version of Resin. After a good few hours of trying, I got it working with the latest version of Wordpress 3.0, Quercus 4.0.3 and Resin 4.0.7. Just follow the wiki, except for a few points:

Download at least Wordpress 3.0, which is currently in beta
Enable the network feature like this
Use this xml file instead:

<?xml version="1.0"?>
<web-app xmlns="http://caucho.com/ns/resin" xmlns:resin="urn:java:com.caucho.resin">
	<resin:Forward regexp="^/index\.php$" target="/"/>
	<resin:Dispatch regexp="^">
		<resin:IfFileExists/>
	</resin:Dispatch>
	<resin:Redirect regexp="^(/[_0-9a-zA-Z-]+/)?wp-admin$" target="$1wp-admin/"/>
	<resin:Forward regexp="^(/[_0-9a-zA-Z-]+/)?files/(.+)" target="/wp-includes/ms-files.php?file=$2"/>
	<resin:Forward regexp="^(/[_0-9a-zA-Z-]+/)?(wp-(content|admin|includes).*)" target="/$2"/>
	<resin:Forward regexp="^(/[_0-9a-zA-Z-]+/)?(.*\.php)$" target="/$2"/>
	<resin:Forward regexp="/.*" target="/index.php"/>
</web-app>

Note that I did not use the pro version, skipped the DNS stuff by using directory instead of subdomains in WP, I also skipped the chown stuff by removing this block from resin.xml:

<resin:if test="${resin.userName == 'root'}">
        <user-name>pepijndevos</user-name>
        <group-name>staff</group-name>
      </resin:if>

RSS feed of Twitter lists

2010-05-20T00:00:00-07:00

You want them? Me to! Here is how. The old-fashioned way:

Create a Yahoo! ID
Do some Yahoo! Pipe magic
Get the RSS feed

But this is sub-ideal. Pipes have a huge url, a delay to long for practical use with Twitter, and are unreliable at times. I searched the API for a solution but found none. The only official format for lists is XML and JSON. Just by trying anyway, I and Jaap found out that lists do not have an RSS feed, but they do have an Atom feed(which is sort of RSS on steroids)! This can be used in most cases where an RSS feed is required. The new way:

Copy the url of a list, for example http://twitter.com/erwinelling/eightmedia
Modify the url to look like this http://api.twitter.com/1/erwinelling/lists/eightmedia/statuses.atom

Have fun! One word of warning though, because this feed is undocumented, it might change or disappear at any time.

Save your date with PyMouse

2010-05-06T00:00:00-07:00

With PyMouse this would never have happened. Don't risk your date, get PyMouse right now at its GitHub page! All that it takes to wiggle your pointer with PyMouse is this:

from pymouse import PyMouse
from time import sleep
from random import randint

m = PyMouse()
w, h = m.screen_size()
while True:
    try:
        m.move(randint(0, w), randint(0, h))
        sleep(20)
    except KeyboardInterrupt:
        print "Goodby!"
        break

Open external links in running browser

2010-05-03T00:00:00-07:00

As a web developer I have like 5 browsers installed which I use regularly. That is, not on daily basis, but for occasional testing and browsing. What annoys me most about this is that when I click a link in another application, no matter what browser I'm currently running, it opens a fresh Safari window. The ideal solution would be to open any urls with the browser I'm using at that moment, no mater which browser that is and which one is set as the default browser. I have written a piece of Applescript that will do just this, you might need to edit this for the browsers which you are using though. Below you'll find a zip file containing the app bundle, and all the scripts needed to generate it yourself.

Browser Loader

To edit the browsers you use, open "/Applications/Utilities/AppleScript Editor.app" and then open the app bundle, this will reveal the script below, where you can make adjustments to the browsers used and their order. If you want do do this yourself, or the app bundle does not work for you for some reason, this is the actual Applescript I used:

on open location the_url
        tell application "System Events"
                set browser_apps to the name of every process whose visible is true
                if "Opera" is in browser_apps then
                        tell application "Opera"
                                open location the_url
                                activate
                        end tell
                else if "firefox-bin" is in browser_apps then
                        tell application "Firefox"
                                open location the_url
                                activate
                        end tell
                else if "Google Chrome" is in browser_apps then
                        tell application "Google Chrome"
                                open location the_url
                                activate
                        end tell
                else
                        tell application "WebKit"
                                open location the_url
                                activate
                        end tell
                end if
        end tell
end open location

Copy this file to the Applescript editor and hit Save, before you save, choose to save as an app bundle and make sure to check the "Stay Open" box. Now we need to modify the app bundle to make it eligible for url opening. To do this, right-click on the app and select "Show Package Contents", go to "Contents" and open "Info.plist". Edit "Info.plist" to contain this xml code within the outer dict element. For an example, download the app bundle above.

<key>CFBundleIdentifier</key>
        <string>nl.pepijndevos.urlhandler</string>
        <key>CFBundleURLTypes</key>
        <array>
                <dict>
                        <key>CFBundleURLName</key>
                        <string>Applescript urls</string>
                        <key>CFBundleURLSchemes</key>
                        <array>
                                <string>http</string>
                                <string>https</string>
                                <string>file</string>
                        </array>
                        <key>LSIsAppleDefaultForScheme</key>
                        <true/>
                </dict>
        </array>

Now, when you click an url it will just open your browser. I'm not sure what to do exactly, but here is what I did

Run the app once
Set it as the default browser in Safari

[update] Using Linux? Try this script on Humbug.in This script has also appeared on lifehacker.com download.com and maxosxhints.com

Top 5 multi-protocol IM clients for Mac

2010-04-21T00:00:00-07:00

Among the few applications I use constantly is my IM client. Just like my web browser, I constantly seek better and faster IM applications. With browsers there are a few big ones and everyone knows them, for IM on Mac however, you have to take a close look to find something that suits your needs. Since I've done a lot of this searching, I'm going to do a list of my favorites. If I missed your favorite, pleas leave a comment.

Adium

One of the most used IM clients on Mac. Adium is an application based on the same technologies that power the popular Pidgin client, but offers a native Cocoa interface and a wealth of plugins and customizations, listed at their extras pages. The 1.4 beta supports some additional cool features, but due to a lack of manpower is still not done. I recommend using the beta, as it is quite stable.

iChat

The chat client included with Mac OS X, it supports only the XMPP and AIM protocols, but I consider it one of the best if you don't need other protocols. It is possible to use other protocols via a XMPP transport, but this very is unreliable at times.

Trillian

Is a well known Windows client that has recently released an alpha version for Mac. It is not very feature rich yet, but it looks and works nice so far. One downside of Trillian is that it stores your credentials in a central account on their server, one of the upsides of Trillian is that it stores all your credentials in a central account, which is cool if you use Trillian on multiple computers/phones.

Pidgin

This app is mainly aimed at Linux and Windows users, but with a little help of my previous post you can get it to work on Mac quite nicely. It uses the same technology as Adium, but due to a broader audience it might offer some features not currently present in Adium. Only use this if you know you need it, Adium has a nicer interface on Mac.

Yahoo Messenger

Contrary to what you might think Yahoo also supports WLM, provides a good Mac look and a full set of features, unlike some other branded clients. It's not the ideal thing though for the real IM diehard.

Conclusion

I'd go with Adium if you're a regular user, or iChat if you don't need any of those networks. For the experimental people, Trillian and Pidgin might be good options. For the social network people it might be worth waiting for Raindrop or Digsby. For the WLM oriented minds, the official WLM client, Mercury and aMSN might be of interest.

display: inline-block; tricks

2010-04-14T00:00:00-07:00

In most cases where you want to have a set of items on one row, you use floats to make things work. The truth is, floats are not meant to do this and they create all sorts of strange behavior, even in good browsers if you don't know what you're doing. This article tells an interesting story on how you could use inline-block in all major browsers. I want to add a few tricks to the mix I learned in the past few years. Use case 1: You want a horizontal menu with more items than fit on the screen, but with a flexible width. Floats are not up to the job; They would just wrap to the next line instead of creating a scrollbar. Use case 2: You want to center something vertically. You could use a mix of position and margin statements to achieve this, but not if you don't know the width and height of the object. Since the elements are in the text flow now, we can apply text-specific CSS! The first example is achieved by setting whitespace: no-wrap; and overflow: auto; which makes all the items stay on one line and create a scroll bar. The second example is achieved by adding &zwnj; before the item to make it behave like a real line of text, that has vertical-align: middle; applied to it.

Examples

Control the mouse with a joystick

2010-04-12T00:00:00-07:00

When I made PyMouse I did so because I had the idea to make an iPhone mouse. Today I had another crazy idea for moving the mouse: with a joystick! It turns out it's quite easy to use the joystick via PyGame, so that is what I did this evening. I plugged in the joystick from my brother, copied some PyGame example, imported my own PyMouse module and here is the result. You might need to tweak it a little to make it work with your joystick or game controller.

import pygame
from pymouse import PyMouse
from time import sleep

# edit this to reflect your joystick axis and buttons
action = {'x':0, 'y':1, 'multiplier':3, 'left':0, 'right':1}

pygame.init()
j = pygame.joystick.Joystick(0) # first joystick
j.init()
m = PyMouse()
print 'Initialized Joystick : %s' % j.get_name()
state = [0, 0]
try:
    while True:
        pygame.event.pump()
        sleep(0.1)
        # check if any button state has changed and change mouse state accordingly
        if j.get_button(action['left']) and not state[0]:
            state[0] = 1
            print "left press"
            m.press(*m.position())
        elif not j.get_button(action['left']) and state[0]:
            state[0] = 0
            print "left release"
            m.release(*m.position())
        elif j.get_button(action['right']) and not state[1]:
            state[1] = 1
            print "right press"
            m.press(*m.position(), button=2)
        elif not j.get_button(action['right']) and state[1]:
            state[1] = 0
            print "right release"
            m.release(*m.position(), button=2)
        
        x, y = m.position()
        m.move(
            # get_axis returns a value between -1 and 1
            # fumble a bit here to reverse axis
            x + (j.get_axis(action['x']) * 50 * abs(j.get_axis(action['multiplier']) - 1)),
            y + (j.get_axis(action['y']) * 50 * abs(j.get_axis(action['multiplier']) - 1))
        )
except KeyboardInterrupt:
    j.quit()

Delicious bookmarks in Safari

2010-04-10T00:00:00-07:00

I recently started using Delicious, and while they have a nice Firefox plugin and a bunch of bookmarklets, there is no easy way to get Delicious bookmarks into the Safari bookmarks bar. To get this done I wrote an Applescript with a little piece of Python to get Delicious bookmarks right in Safari. Here is how it looks: Here is how to do it:

Get the 2 scripts below and save them to a suitable location. You might want to place it in ~/Library/Scripts/Applications/Safari
Edit the Applescript with the location of the Python script and the URL of your Delicious feed.
Run the Applescript.
Drag the resulting bookmarks folder or its content anywhere you want.

This will work for any RSS and maybe even Atom feed, you could even bookmark the feed of your Twitter account, for... *hum* easy access to individual tweets. This is the Python code:

import feedparser
print "<dl>"
for e in feedparser.parse( "http://feeds.delicious.com/v2/rss/pepijndevos" ).entries:
    print '<dt><a href="%s">%s</dt>' % (e.link, e.title.encode('ascii','ignore'))
print "</dl>"

Don't ask me why, but Safari needs broken html to import. Maybe because it is meant to import IE bookmarks? Anyway, this is the Applescript, the real thing:

do shell script "python ~/bin/feed2html.py > ~/Documents/delicious.html"
tell application "Safari"
        activate
        tell application "System Events"
                tell application process "Safari"
                        click menu item "Import Bookmarks…" of menu "File" of menu bar item "File" of menu bar 1
                        keystroke "g" using {shift down, command down}
                        keystroke "~/Documents/delicious.html"
                        keystroke return
                        keystroke return
                end tell
        end tell
end tell

PyMouse 1.0: Mouse control and events on Python

2010-04-09T00:00:00-07:00

Since the first release of PyMouse I received a lot of feature requests. A list of all requests ordered by quantity:

Receive mouse events
Keyboard support

Digging around in platform specific APIs is not my favorite hobby, but when Aldo came to me with a good start for the Unix code, and copyrights send me some Windows code, I could do nothing but finish my Mac code and adapt everything to a common API. I'm also very grateful for the help from mailing list members of PyObjC and Python Xlib. Without their help I'd probably have to reverse the current version number. Keyboard support is not yet there, and I do not currently have time to write it. But I wrote down everything I know about the APIs I worked with on the PyMouse wiki, so it shouldn't be to hard to figure out for anyone willing to do it.

Download PyMouse

Good looking X11 apps on Mac

2010-04-07T00:00:00-07:00

A while back I discovered that Pidgin is now available for Mac. The problem is that it is still a GTK app. For Mac users that means it runs in X11. It has always bothered me that all X11 applications you run on Mac look so extremely ugly. When I used Ubuntu a few years back I discovered mac4lin, a funny project that intends to make a Mac theme for Gnome. Ironically, since Gnome is based on GTK you can run mac4lin on the system it intends to emulate, which would give your GTK apps a nice Mac look. Let's see how... For this example I'm also going to install Pidgin, to show you how it looks. First, however we need to get a bunch of software. You need to have MacPorts installed for this to work.

First we need to get GTK without X11. You can leave of the variants if the build fails to get the X11 version.
```
sudo port install gtk2 +quartz +no_x11
```
Now we can get Pidgin, or any other GTK2 app.
```
sudo port install pidgin +quartz +no_x11
```
To be able to easily switch theme later we also need this little app.
```
sudo port install gtk2-chtheme
```
Now download and extract the latest version of mac4lin from their website.
Extract Mac4Lin_GTK_Aqua_v1.0.tar.gz in the GTK folder to /opt/local/share/themes/ (You'll need to authenticate)
Run /opt/local/bin/gtk2-chtheme, select the newly installed theme.
Click the font button and set it to Lucida Grande at 10pt, done!
Run /opt/local/bin/pidgin to see the effect.

Before: After: Keyboard shortcuts: You might also want to use this hint to enable the CMD key for keyboard shortcuts. Sound: By default Pidgin uses you 'system beep' to notify you, solution:

sudo port install qtplay

easy_install: no eggs found

2010-04-06T00:00:00-07:00

I blogged about this before, but I was bitten by the same issue again! When you run an old version of setuptools and easy_install, you might get an error concerning eggs not being found. If you're having the same problem, the solution is simple:

easy_install -U setuptools

Manage your social networks

2010-03-28T00:00:00-07:00

I recently had one of my "search for relief" attacks, which means I start to try out a lot of software in search for something better than I already have. This happens a few times a year, and leads to a lot of (remainders of) crap and cool software on my computer. After I went through the OS(Mac), browser(WebKit), chat(Adium), editor(MacVim) and maybe more, it was time for the social networks. I have now figured out a set of social networks I like, but that led me to a new problem: management! An ideal piece of software would be able to manage at least:

Gmail
Wordpress
Twitter
Facebook

And probably also:

Delicious
Flickr
Buzz

That ideal application would be Raindrop, but there is a small problem, it isn't finished yet. I tried very hard, but it would not work for me at this stage. Wait a minute... All those services above are websites, aren't they? So, why not use the browser? Because you'd have to open about ten tabs and handle each service separately... Unless you use Flock! Flock is a browser that is based around FireFox, but with social networks built right in the core. This might be your solution, but it is not mine; I like the speed, simplicity and integration of WebKit(developer version of Safari). My solution is quite the opposite of Flock. I try to use applications for what they are meant to do, and update as much networks as possible on the way, which in practice means linking networks together. Here is my current setup:

Let me explain that in more detail. A lot of these services include an option to either read or write to Twitter, and for those that do not write to Twitter there is TwitterFeed to parse RSS feeds into Tweets.

Buzz is kind of redundant, but when you click 'connected sites' under Buzz in Gmail it can dupe your Tweets for those that do use it.
Facebook is not just another Twitter, so it might need some extra attention, but you can cover wall updates and photos by adding the Twitter app and importing your Flickr photos. To do this
1. Go to your profile
2. Click 'Options' and then 'Settings' just above your feed
3. Select Flickr
4. Figure out the rest
Flickr can also be used as an Twitter image/video service by adding Twitter as a blog via http://www.flickr.com/account/blogs/ and then clicking 'blog this' above any photo. There are numerous Flickr uploaders to use if you want to.
Wordpress, Delicious, GitHub or any other service with a feed can be linked to your Twitter with TwitterFeed. You might choose to install additional software to facile operation of Wordpress and Delicious.
Update: By adding 'http://twitter.com/statuses/user_timeline/<username>.rss' to TwitterFeed you can publish Twitter to Identi.ca and some other services as well.

I suggest you browse around my social network a bit and check the 'via' tag to see where the content is coming from. If you are unsure how to connect a certain service, leave a comment, and I'll elaborate on that. I'm starting to get a little concerned about content duplication, any thoughts about that?

Tweet-o-Graph

2010-03-22T00:00:00-07:00

Back in 2009 I made a hackish Python script that would generate a graph like the one shown here from your Twitter account. It turned out few people are willing to fumble with Python scripts to get 'just an image'. Luckily, I'm releasing a completely rewritten web app for the same cool idea! tweetograph.pepijn.cqhosting.nl Now with support for oauth, mentions, retweets, replies, gif, svg, dot, and a lot more. It is written with Python and web.py, using this script from my previous post.

Cross-domain AJAX POST request -- Twitter client

2010-03-16T00:00:00-07:00

One of the top items on my web-project-wish-list is a client-side Twitter client written in JavaScript, without resorting to a proxy.

The problem

The technology needed to write web applications is named AJAX, but for security reasons, AJAX has one important limitation! You can not make requests to another domain. Imagine what would happen if a malicious website would request your gmail inbox while you are logged in!

Alternatives

Because of this limitation, people started to look for other ways to make cross-domain request. Bring in JSONP! JSONP allows you to request data from another domain by inserting a script tag with a src attribute referring to the data location. This is used in the Twitter widget showed in my sidebar. JSONP has two limitations.

You give the other domain scripting access to your site, opening a potential XSS vulnerability.
You can only make a GET request.

iframes

Hidden iframes are another method to make asynchronous requests, but they to are limited to GET requests and are only accessible from the same domain. I have not yet found a solution for the access problem, but I can make POST requests.

The solution

Make an iframe and set its display property to hidden.
Load a page in the frame containing a form.
Add hidden inputs to the form.
Submit it!

Example: Click here to send a tweet to Twitter about this post!

iframe removed

The only remaining problem is that we can not access the return data, but as you can see, that is not needed in all cases.

Clojure static site generator

2010-03-12T00:00:00-08:00

Ever wondered why systems like Wordpress generate the whole page every time a visitor comes around? I did, and I'm not the only one. There are numerous static site generators and caching plugins around, and this is one of the former. First I wanted to write a static site generator in Python, but there is already one. I started to learn Clojure, and met defn on the Clojure IRC channel, who also wanted to write a Clojure static site generator. We started coding, and that is how Utterson came to life. I must admit though that I haven't heard very much of defn since the start of the project. Utterson works by threading Markdown files through Clojure template files to generate static HTML files. Other files(images, etc.) are copied as-is.

java -jar utterson.jar src dest

The project is currently in a state in which it works, but it would make for an awkward blog, since there are still some base features missing. http://github.com/pepijndevos/utterson For now this blog is going to be using Wordpress, until I finish the generator and find solutions to problems like comments.

Three good reasons to run a Webkit Nightly

2010-03-08T00:00:00-08:00

You get to use the latest an coolest features around.
You can try out my 3D CSS browser game, which is finally online.
You can see this wacky CSS endless zoom effect, and other beautiful show-offs.

No, seriously... You might not need it for daily web usage, but if you are a developer, it's fun! Get it at webkit.org, even the nightly is more reliable than IE6. I'm using it right now to write this.

How to install Python/web.py on a shared host

2010-03-03T00:00:00-08:00

In the past I have written a few Python web applications, but when I asked my host about Python, they told me they where specialized in PHP and nothing but PHP(that is why this site runs Wordpress). If your are lucky and have a very expensive dedicated server you can install whatever you want, but how about my poor projects? I'm not going to re-write them in PHP obviously. I figured that as long as you have a CGI bin, you can do whatever you want. I started writing a CGI script that would install Python and Web.py for me; Check out the resulting web.py example. This is the script that I came up with. It should be doable to customize this for Pylons, TurboGears, Django, or any other Python framework. You should at least fill in your own home directory twice, we're doing a Python installation inside you home directory.

#!/bin/sh

echo "Content-type: text/html

";

curl http://www.python.org/ftp/python/2.6.4/Python-2.6.4.tgz | tar -zx
cd Python-2.6.4
./configure --prefix=/your/home/directory #change this to your home
make
make install
cd ..
curl http://www.saddi.com/software/flup/dist/flup-1.0.2.tar.gz | tar -zx
cd flup-1.0.2
/your/home/directory/bin/python setup.py install
cd ..
curl http://webpy.org/static/web.py-0.33.tar.gz | tar -zx
cp -r web.py-0.33/web .

Note that you should add .py as a CGI handler in your .htaccess file:

AddHandler cgi-script .py

Since our fresh Python installation is not on the PATH variable, you should include a shebang header in your scripts to point to the right path:

#!/your/home/directory/bin/python

Again, check the result!

How to navigate the browser ballot screen

2010-02-23T00:00:00-08:00

This is my response to the EU/Microsoft browser ballot screen. Honestly, most people don't even know what their browser is, what do you think the difference in IE market share will be? http://xkcd.com/627/ also applies here. I recommend any of the five except IE. In case you're wondering, I'm using Safari.

Wordpress.com Stats: Top Posts Widget

2010-02-22T00:00:00-08:00

This hack is so cool and so simple, I expect anyone to run up to me and yell that this has been done before. I created a 'Top Posts' sidebar widget based on the widget that comes with any Worpdpress.com blog. I know there are several widgets like this, but they all include their own statistics tracking. Mine is nothing more than an excerpt of the Dashboard widget that comes with the Wordpress.com Stats plugin. For people not interested in technical details, here is a plugin file, that works as usual; Make sure you install the Wordpress.com Stats plugin first! I said this hack was simple, here is way: The code is already there! I'm surprised Wordpress did not include the widget themselves. Lets have a look at stats.php: Line 925-926:

foreach ( $top_posts = stats_get_csv( 'postviews', "days=$options[top]$csv_args[top]" ) as $post )
        $post_ids[] = $post['post_id'];

Line 942-949:

<?php foreach ( $top_posts as $post ) : if ( !get_post( $post['post_id'] ) ) continue; ?>
<p><?php printf(
        $printf,
        '' . get_the_title( $post['post_id'] ) . '',
//                        '' . $post['post_title'] . '',
        number_format_i18n( $post['views'] )
); ?></p>
<?php endforeach; ?>

My resulting guesswork looks like this:

<h4>Top posts</h4>
<ul>
<?php foreach($top_posts = stats_get_csv('postviews', "days=7") as $post): if(!get_post($post['post_id'])) continue; ?>
        <li></li>
<?php endforeach; ?>

~~If anyone is interested, I can upload this to the Wordpress plugin repository.~~ Done: http://wordpress.org/extend/plugins/wordpresscom-stats-top-posts-sidebar-widget/

Food versus code

2010-02-21T00:00:00-08:00

Food: 279kJ/66kcal; E263, E181, E459;
Code: [a-zA-Z0-9]*; hey

Now, which one is easier to decipher? Do you think we need simple food, simple code, or both? For the record, the first food part is from coffee, the ingredients are Calcium acetate, Tannin and Beta-cyclodextrine. The first code part is a regex matching any number of alpha-numeric characters, the second part is 'hey' in html entities.

Shorten Wordpress urls in Adium

2010-02-20T00:00:00-08:00

I use Adium as my Twitter client, it includes a nice url shortener supporting multiple services, so what is my problem? Nothing, I just had some fun making use of the wp.me service. Every Wordpress blog contains a header containing the short url, so it was only 3 lines of Applescript and one line of Bash to get the short url of any wp blog and post it to Adium. If anyone with knowlegde of regex/grep comes around, he might get it to work with any shorturl/shortlink header.

The current line looks like this:

curl --head <url> -s | grep -o http://wp\.me/[a-zA-Z0-9\-]*

Update: sed seems to be able to do what I want.

curl --head http://pepijn.cqhosting.nl/ -s | \
sed -n 's/Link: <\(.*\)>; rel=shortlink/\1/p'

Use:

%_shorten{<url>}

Download the script: shorten.AdiumScripts

Cross-domain caching

2010-02-20T00:00:00-08:00

Looking at the list of HTTP headers at Wikipedia, you'll notice a few headers like If-Modified-Since, that serve to send content over only if it has changed since the last request. While working with JavaScrip frameworks like jQuery, the thought occurred to me, that maybe half of all the websites I open are using the exact same JS framework, isn't that a waste of bandwidth? The best real-life solution would be to get your JS from Google, so you at least share a cached version with others that do the same: http://code.google.com/apis/ajaxlibs/ My imaginary solution however, would allow the same file, served from different hosts to be cached only once. Look again at the list of headers... We're looking for something to identify files as equal... What about Content-MD5? It exists, right there! But as far as I can tell it's used for content-verification only. So I want to propose a new HTTP header: If-MD5-Differs What do you think about it? Cool? Useless? Security issues?

Automatically remove Trash from usb sticks

2010-02-19T00:00:00-08:00

I think most Mac users have had this problem; You delete a bunch of files from your pen drive to make room for a large file, and it keeps complaining about lack of space. The problem is that when you remove files on Mac, they're just moved to a hidden folder named '.Trashes', so they are actually still there. A commonly suggested solution is to empty the Trash while the USB stick is plugged in, but this might be problematic in some cases where you want to keep your Trash, just in case...

My new solution is to set this little script as a folder action, it will remove the trash only from the USB stick as soon as you mount it(unmount proved problematic).

on adding folder items to this_folder after receiving these_items
    do shell script "rm -rf " & POSIX path of these_items & ".Trashes/"
end adding folder items to

Save this script to /Library/Scripts/Folder Action Scripts/ and now set it as the folder action for /Volumes. You might need to show hidden files in Finder for /Volumes to be visible. To do this, follow this guide.

Write clean code!

2010-02-18T00:00:00-08:00

Nieuwe website

2009-08-26T00:00:00-07:00

Hehe, eindelijk, ik heb een eigen website opgezet! Ik had al een tijdje een wordpress.com blog, maar nu heb ik een website waar ik wel alles me kan doen wat ik wil.