Wishful Coding

Didn't you ever wish your computer understood you?

or

or

How Clojure sat in a corner converting and boxing while Python did the work

or

Update: The Clojure version is now a lot faster. Thanks to the people of the Clojure mailing list. I also uploaded the map I used if you want to compare the results.

Okay, back to business. During the writing of the original Python hack I had to do a few tricks I thought would be easy to do in Clojure. So I started to wonder how the Clojure code would look and how fast it’d be.

My original hack was kind of slow, but it’s greatly improved and now renders a whole map in under 10 seconds.

• 4s for reading all files
• 3s for calculating the graph
• 8s total

The code to read all the files:

paths = glob(join(world_dir, '*/*/*.dat'))

raw_blocks = ''
for path in paths:
nbtfile = NBTFile(path, 'rb')

raw_blocks += nbtfile['Level']['Blocks'].value

The code to calculate the graph:

layers = [raw_blocks[i::128] for i in xrange(127)]

counts = [[] for i in xrange(len(bt_hexes))]
for bt_index in range(len(bt_hexes)):
bt_hex = bt_hexes[bt_index]
for layer in layers:
counts[bt_index].append(layer.count(bt_hex))

Nice eh? Now the Clojure version. Clojure doesn’t have a nice blob module, so I’ll spare you the code that gives me the data. Sufficient to say is that it also runs in about 4 seconds.

My initial version for the calculating was short and sweet and looked like this:

(defn freqs [blocks]
(->> blocks
(partition 128)
(apply map vector)
(pmap frequencies)))

Now, this is twice as fast as what I currently have, but it has a problem. While Python operates on bytes the whole time, these lines of Clojure operate on a sequence of objects. These objects are just a tad bigger than the bytes in a string, so keeping 99844096 of those in memory is impossible.

So, either I had to find a way to make Clojure throw away all the objects it had already processed, or I had to make it use a more compact storage for them. I tried both, and ended up with a function to concatenate Java arrays, but working with them is a real pain, so I made my function use them wrapped in Clojure goodness and made sure the Java GC threw them out as soon as I was done.

(defn freqs [blocks]
(->> blocks
(partition 128)
(reduce (fn [counts col]
(doall (map #(assoc! %1 %2 (inc (get %1 %2 0))) counts col)))
(repeatedly 128 #(transient {})))
(map persistent!)))

This is not threaded like to previous example, but it works. Everything I tried to make it use all my cores either started to eat more and more memory, or was slower then the single-treaded one. Most of them where both.

So, how fast is it?