Wishful Coding

Didn't you ever wish your
computer understood you?

Clojure versus Python

or

Mian versus Clomian

or

How Clojure sat in a corner converting and boxing while Python did the work

or

A plot about Minecraft(pun intended)

Update: The Clojure version is now a lot faster. Thanks to the people of the Clojure mailing list. I also uploaded the map I used if you want to compare the results.

Okay, back to business. During the writing of the original Python hack I had to do a few tricks I thought would be easy to do in Clojure. So I started to wonder how the Clojure code would look and how fast it’d be.

plot

My original hack was kind of slow, but it’s greatly improved and now renders a whole map in under 10 seconds.

  • 4s for reading all files
  • 3s for calculating the graph
  • 8s total

The code to read all the files:

paths = glob(join(world_dir, '*/*/*.dat'))

raw_blocks = ''
for path in paths:
    nbtfile = NBTFile(path, 'rb')

    raw_blocks += nbtfile['Level']['Blocks'].value

The code to calculate the graph:

layers = [raw_blocks[i::128] for i in xrange(127)]

counts = [[] for i in xrange(len(bt_hexes))]
for bt_index in range(len(bt_hexes)):
    bt_hex = bt_hexes[bt_index]
    for layer in layers:
        counts[bt_index].append(layer.count(bt_hex))

Nice eh? Now the Clojure version. Clojure doesn’t have a nice blob module, so I’ll spare you the code that gives me the data. Sufficient to say is that it also runs in about 4 seconds.

My initial version for the calculating was short and sweet and looked like this:

(defn freqs [blocks]
  (->> blocks
    (partition 128)
    (apply map vector)
    (pmap frequencies)))

Now, this is twice as fast as what I currently have, but it has a problem. While Python operates on bytes the whole time, these lines of Clojure operate on a sequence of objects. These objects are just a tad bigger than the bytes in a string, so keeping 99844096 of those in memory is impossible.

So, either I had to find a way to make Clojure throw away all the objects it had already processed, or I had to make it use a more compact storage for them. I tried both, and ended up with a function to concatenate Java arrays, but working with them is a real pain, so I made my function use them wrapped in Clojure goodness and made sure the Java GC threw them out as soon as I was done.

(defn freqs [blocks]
  (->> blocks
    (partition 128)
    (reduce (fn [counts col]
              (doall (map #(assoc! %1 %2 (inc (get %1 %2 0))) counts col)))
            (repeatedly 128 #(transient {})))
       (map persistent!)))

This is not threaded like to previous example, but it works. Everything I tried to make it use all my cores either started to eat more and more memory, or was slower then the single-treaded one. Most of them where both.

So, how fast is it?

  • 5s file reading
  • Over a minute of processing
  • Over a minute + 5s total

Wait, what? Python did this in 3 seconds, right? Yea… So even if I had used the faster function and had 10GB of RAM it’d be 10 times slower.

Why? I don’t know. All I can come up with is that that Python just acts on a string, while Clojure does boxing and converting 99844096 times. If you happen to know what’s wrong, or how to make it faster, be sure to tell me!

Where to dig in The Nether

Spoiler: Don't

You remember my other Minecraft post? A lot happened since then. Victor Engmark took my hack and made a nice tool out of it that runs way faster as well! Notch released the Halloween update with a new world that is at the moment called The Nether. And last but not least, I ported Mian to Clojure to form Clomian, to compare it to the Python version and to analyze The Nether.

There are still some issues with the code, and I might actually have found my first bug that is actually in Clojure and not in my code. Therefore I will talk and compare in another post, and for now just show you the results.

The bug:

(reduce #(map + %1 %2) (partition 5 (range 1e6)))
java.lang.StackOverflowError

(reduce #(pmap + %1 %2) (partition 5 (range 1e6)))
(99999500000 99999700000 99999900000 100000100000 100000300000)

The graphs:

Where to dig in The NetherWhere to dig in The Nether

As you can see The Nether is an unfriendly place with nothing but Bloodstone, Lava, Fire and Mushrooms.

Unlike the normal world, The Nether is 100% enclosed by Bedrock and in the middle, half of everything is still Bloodstone.

Unlike the normal world, there aren't any minerals and there isn't a clear distinction between above and below.

The only valuables I see are mushrooms and Lightstone, and these are somewhat uncommon and scattered across the whole level.

Conclusion:

The Nether is an interesting world, but it's not worth getting killed for, unless you desperately need any of those special blocks or shrooms.

How to be a good programmer

Get back to work.

Published on