Wishful Coding

I came across a random tweet, found a Wikipedia page and bumped into some smart people and long story short, apparently compression is equivalent to general intelligence.

So, how do you build ChatGPT with data compression? What if you compress a large corpus of text to build up the encoding table, then you compress your prompt and append some random data and decompress the random data, and hope it decompresses to something sensible.

It feels vaguely similar to diffusion, but what do I know. Look, this is just a dumb idea, let’s just see what happens ok? Well, here is my progress so far. It’s kind of whack but it’s hilarious to me that it produces something resembling words.

import nltk
import lzma
import random

my_filters = [
    {"id": lzma.FILTER_LZMA2, "preset": 9 | lzma.PRESET_EXTREME},
]
lzc = lzma.LZMACompressor(lzma.FORMAT_RAW, filters=my_filters)

corp = nltk.corpus.reuters.raw().encode()
out1 = lzc.compress(corp)

corp = ' '.join(nltk.corpus.brown.words()).encode()
out2 = lzc.compress(corp)

corp = nltk.corpus.gutenberg.raw().encode()
out3 = lzc.compress(corp)

out_end = lzc.flush()

lzd = lzma.LZMADecompressor(lzma.FORMAT_RAW, filters=my_filters)

lzd.decompress(out1)
lzd.decompress(out2)
lzd.decompress(out3)
# mess around to avoid LZMAError: Corrupt input data
lzd.decompress(out_end[:-344])
# insert prompt????
print(lzd.decompress(random.randbytes(50)).decode(errors="ignore"))

Here are a few runs. Note how the start is always , and tri, usually completing it into some word. Are we doing some primitive accidental “prompting” or just flushing the buffer? Either way, not bad for mere seconds of “training”!

$ python train.py 
, and triof billioerse,
But
ht and see th,
Thy smile, in to be happy,
Wmson,
Over tout as aThy smile;t as aThyrged in
 
ent, foldehe snoion since how long,
my roomr? Is ic books 

$ python train.py 
, and triompact, sca,
Take deepcky fouy vitaliz  bodiehow there i,
Nor drummiwisibly wile of the-ations, dutway?
Yet ld woman'okesmanall whoy slow bekesmanalle me 

$ python train.py 
, and tri billions of the boftier, faie no acqutory's dazzd haOr that thpages:
(Sometimeseathe ihern, Sounte, fld Turkey n one,
Worlseathe Border Minstrelsy,ine, New-Ene Queen.

'Thelicate l

$ python train.py 
, and tri, sleepinlke babes  bent,
Abird;
Forhis fair n!
By thea mystic strangehe gifts ofhe body aering
 t: I haue a lugs whipageantuperb-fnz

$ python train.py 
, and triions of b--n the gra, with
e open the countless buAnd bid theng;
Billi  toward you.i ally undya Songs

To f Death, istas of was mar to be UY 9,30

In this digital age so much of our information is only on the internet, and it’s hard to predict how much of it would survive another Carrington event. So what if we engrave all of the information on the internet in stone? We have clay tablets dating back thousands of years, so if we want to preserve our knowledge for the millenia to come, it’s a time-proven method.

That’s it. That’s the plan. “set in stone” Wikipedia, literally. I know it sounds crazy and I don’t have any illusions about completing the project, but it seems a fun mix of art project and potentially my biggest contribution to the persistence of human kind haha

But how do I go about doing this? I want to automate it as much as possible, but it also has to be super durable and some amount of authentic. Like, if you’re going for pure efficiency and inforation density you’d maybe engrave binary into acrylic, but what’s the fun in that. There are so many remaining options from laser engraving slate to CNC routing clay tablets. Let’s just research and try some?

CNC engraving clay tablets

I started watching a bunch of youtube videos about low fire backyard pottery, where people make pottery on their own improvised coal furnaces and such. Some things I learned:

• get clay and mix in sand to make it less prone to exploding
• dry really really really well
• stick it in a bunch of coal and hope for the best

So I started making my clay tablet by mixing sand into the clay and adding water. Then I used (wet) planks to make the tablet as flat as possible.

Then I copied a wikipedia page into a text file and used hf2gcode to make gcode:

text=$(python - $1 <<END
import unidecode, sys
with open(sys.argv[1]) as f:
    print(unidecode.unidecode(f.read()))
END
)
echo "$text" | fold -w 20 | ./src/hf2gcode -s 0.3 -n 12

After letting the clay dry and with my gcode in hand I went to Tkkrlab to try to CNC it. I used bCNC for this. The first try chipped off some bits, so I sprayed some water to make it more soft. But the text was way too small, so I redid the gcode with a bigger font and only the first paragraph of the wikipedia page.

With a bigger font size and regular water spraying I was making good progress, untill the bit started cutting deeper and deeper. Despite my best efforts the tablet wasn’t flat enough, so I had to stop, readjust the CNC, and do some hacky maneuvers to start the job from that point. Success?! Some parts were barely touching, but it’s something!

So I went back home and put it in the oven to dry further. First at below 100, then at 150, and then cranking it all the way up. I guess I was too impatient because it exploded.

Lessons learned

• I need to be way more patient with drying (and possibly avoid making bubbles?)
• It needs to be way flatter (or I need to measure and compensate)
• It looks amazing
• Information density is super low

So I’m a bit torn, the first two items are probably solvable, and I really love how it looks, but at this font size it can barely fit a paragraph. With a beter process I can maybe reduce it a little bit, in particular the line height is quite excessive, but it’s just not going to fit an entire wikipedia page.

Do I continue pushing this, or do I try something else? I think I might try what laser engraving slate looks like once the laser cutter at Tkkrlab is back in operation.

Someone asked if anyone had any experience interfacing Lego Mindstorms with Julia, and I was like this is right up my alley, so I got a little bit very much nerdsniped into making this work.

You see, the sensible thing would be to just run Julia on the EV3 which runs a full Linux, but the modern approach would be to run Julia on the Robot Inventor/Spike Prime Hub, which is an STM32 running a MicroPython firmware.

Here is the game plan. Lacking a MicroJulia implementation and the resources to write a completely custom firmware, we’re going to compile Julia functions into MicroPython extension modules and run them on PyBricks.

This way you can do all your fancy ControlSystems stuff in Julia, compile it to machine code, and then write some Python glue for the sensors and motors.

We’ll basically need to do 3 things

• Compile Julia into ARM machine code
• Link the machine code into an .mpy library
• Run said library on PyBricks

Run native code on PyBricks

MicroPython has this whole page about how to write native modules in C and compile them into .mpy files. This should be easy!

So I just copied their factorial example, changed ARCH = armv7emsp, and ran make to end up with a sweet native .mpy module. Cool now just upload it. Oh. Anyway after some hacks I could use pybricksdev run ble test.py to upload a tiny test program:

import factorial
print("hello", factorial.factorial(4))

Except it threw ValueError: incompatible .mpy arch at me no matter what I tried. A little digging later, I found that I needed to enable MICROPY_EMIT_THUMB. But then I got a weird error in MicroPython, which I hacked around as well.

Then, a small victory: running C code! Now onto Julia.

Compile Julia into ARM machine code

For the first part I used the amazing AVRCompiler.jl which ties into GPUCompiler.jl which ties into LLVM. Long story short, we can abuse the machinery Julia has for running a static subset of Julia on the GPU for generating machine code for other architectures.

All I did was compile Julia from source¹ while adding ARM to the supported architectures, and changed the target triple in AVRCompiler to the one I found by by copying the CFLAGS from PyBricks and asking LLVM:

clang -print-effective-triple -mthumb -mtune=cortex-m4 -mcpu=cortex-m4 -mfpu=fpv4-sp-d16 -mfloat-abi=hard --target=arm-none-eabi
thumbv7em-none-unknown-eabihf

Once that is in place you can just do

using AVRCompiler

function factorial(x)
    if x == 0
        return 1
    end
    return x * factorial(x - 1)
end

obj = AVRCompiler.build_obj(factorial, (Int32,))
write("factorial.o", obj)

Glueing it all together

So we have a way to run native .mpy modules, and we have a way to compile Julia functions to .o files. Now we just need some glue to link the object file into .mpy and make it accessible from MicroPython.

I basically took the factorial example, ripped out the factorial_helper and replaced it with extern int julia_factorial(int); and modified the Makefile to call my Julia script to generate the object file.

# Location of top-level MicroPython directory
MPY_DIR = ../pybricks-micropython/micropython
# Name of module
MOD = factorial
# Source files (.c or .py)
SRC = wrapper.c
SRC_O = factorial.o
# Architecture to build for (x86, x64, armv6m, armv7m, xtensa, xtensawin)
ARCH = armv7emsp
# Include to get the rules for compiling and linking the module
include $(MPY_DIR)/py/dynruntime.mk
# Build .o from .jl source files
%.o: %.jl $(CONFIG_H) Makefile
	$(ECHO) "JL $<"
	julia --project=. $<

This was all looking really promising until I ran make and was greeted by

LinkError: factorial.o: undefined symbol: __aeabi_unwind_cpp_pr0

But, yolo, lemme just comment that out in mpy_ld.py and press on. This symbol appears to be related to exceptions and I don’t really care.

$ pybricksdev run ble test.py
Searching for any hub with Pybricks service...
100%|████████████████████████████| 280/280 [00:00<00:00, 2.15kB/s]
hello 24

At last! Julia code running on the Lego hub! TODO: make cool demo