But what do you test? We are reverse engineering the inner workings of complicated devices, and if you’re slightly wrong you can generate a good looking bitstream that just doesn’t work. So of course during development you test your bitstreams on real hardware to make sure they work. And on top of that we do set up CI, but it can only test that we can generate a bitstream.

This is not idle thought, just yesterday we had one FPGA silently fail due to improvements in another. It is very hard to catch these regressions reliably without constantly manually testing every single devices, which is completely infeasible.

For a long time I’ve dreamed of setting up a raspberry pi with a bunch of FPGAs and somehow running automated tests on it. But it always felt like a huge ordeal to really put everything together. Running and maintaining a raspberry pi with a bunch of FPGAs, writing useful self-tests that automatically verify correct behaviour, triggering a test run, downloading or generating bitstreams, programming the FPGAs, obtaining and verifying the results, and communicating that back to Github.

But this week I looked at the problem again and realized all the pieces had falling into place to make this not only feasible but almost trivial.

1. I am already running a Raspberry Pi with Home Assistant on my very own Mini-ITX motherboard: Sentinel Core
2. For running LLMs on my pi, I built custom Docker addons for Home Assistant
3. To build those Docker containers, I’m using a Github self-hosted runner on my VPS
4. We have femto RISC-V UART examples now based on Bruno Levy’s FPGA tutorials

So the plan is simple, plug some FPGAs into Home Assistant, run a self-hosted runner as an addon, and add a CI task that uploads an example and verifies its UART output against a reference.

For the addon I just forked an existing addon and added USB access and openFPGALoader: home-assistant-github-runner-add-on

For the CI side, it basically just adds a self-hosted step after our current bitstream generation that fetches the bitstream artifacts, uploads them, and diffs the UART output: Add hardware-in-the-loop CI test infrastructure

I’m excited to bring this new level of reliability to Apicula, and curious if other projects could do something similar.

So I thought it would be a fun idea to take two boxes and some money to Treeweek 2 and try to actually play Newcombs Problem.

Rules of Byzantine Mini Newcomb’s problem

This is a thought experiment made real, as a thought experiment about what it means to really play this game. What’s up with the name? Mini: I am not made of money so we are playing at reduced stakes and on a donation basis. Byzantine: How do you tell Omega from a charlatan? How reliable are my predictions really? Are you really precommitted? If I know that you know that I know that you know that…

The Rules

• Think carefully about the nature of the game and your strategy, this is the most important step!!
• Give me your (nick)name, twitter handle, and a minimum donation of €10
• I give this information to my reliable predictor and prepare the boxes accordingly.
• You will be offered two boxes, a transparent one containing €10 and an opaque one that contains either €100 (if I predict that you will only choose the opaque box), or €0 (if I predict that you will take both boxes).
• You take any money from your chosen box(es) and reflect on what happened. Any money in any remaining boxes goes back to the bank.

Byrules

To keep it fun for everyone:

• Don’t donate more than you’re willing to lose.
• Officially you can only play once, but if you ask nicely maybe you can go again.
• Anyone can choose to join the “ethical review board” and I’ll explain my whole spiel but then you can’t play or leak info.
• At the end of the week all will be revealed, and if you wrote down your strategy we can compare notes.

My reasoning behind all of this

First of all, why “mini byzantine”? I would argue it is the only version that can be played.

First of all. There is no scenario where I trust someone who claims to be an alien in posession of millions of earthly USD and giving it to me. In the real world you’re always dealing with the question of who is this guy, why is he giving me money, and how can he possibly know my choice?

There are only two types of scenarios where you are offered large sums of money: scams and games of chance like a lottery or casino. And both of them make money rather than lose it, by asking for a fee.

You could run this game like a scam or game of chance where there is just a vanishingly small chance there is anything in the opaque box, but that’s not really in the spirit of the problem is it?

Although you could argue that a predictor that reliably predicts two boxes could be argued to be a logical if not ethical solution to the problem. “this is obviously a scam so I should take two boxes” as a sort of self fulfilling prophecy.

So lets impose that this is a zero sum game where I’m neither scamming you out of your money or just giving away money for free, which is the only fair and reasonable way to play.

The introduction of a donation actually meaningfully alters the game under this constraint.

Your donation can be seen as the expected value you place in this game. If you donate 10 euro you obviously place low odds on there being anything in the opaque box, and should logically take two boxes to not lose money. While if you donate 100 euro you obviously expect there to be money in the opaque box that you intend to take. These are the two stable equilibrium.

If you donate 110 you expect to outsmart the predictor which is irrational to singal, and if you donate something inbetween you’re not sure what to expect. So the question is, if your expected value/donation is 50 euro, what should the prediction be?

A reasonable choice in this case depends on the past donations. If people keep donating 50 euro I can’t just keep paying out 100, so we need to look at past donations and payouts compute our account balance and historical expected value. So in order for a logical prediction to be one box, it needs to be higher than or equal to the historical average payout, and the current account balance needs to actually be enough to fill the boxes. That can be easily implemented in an excel spreadsheet, which I did.

I’m really curious if we’ll just get stuck on the 10 euro equilibrium or a trickle of higher donations and mispredictions will lower the expected value low enough for a 100 euro payout to happen. Or will there be people with sufficient trust and money who have come to the same conclusion as me.

My predictor

I put the following equation in a spreadsheet that checks if your donation is more than the average payout, and the current account balance allows a payout.

=IF(AND(C2>AVERAGE($E1:E$2), SUM($C$2:C2)-SUM($E1:E$2)>=110), "one box", "two box")

Results

Based on the conversations I’ve had and the number of actual games played (0), we can conclude:

The only winning move is not to play.

I have recently become a dad, which is sometimes amazing, and sometimes suffering, such as when the baby is in a fussy mood and refusing to sleep. Some friends were very enthusiastic about a $1.5k smart crib that makes white noise and rocks your baby to sleep.

You know what also makes noise, rocks your baby to sleep, and doesn’t cost $1.5k? That’s right, this Lego model. This is already by far my most useful Lego creation, and we’re using it regularly to great effect.

At the core of this creation are two bogies, driven by Lego motors, strapped to the wheel of a stroller with rubber bands. Each bogie is driven by two motors for a total of 4 driven wheels and 4 motors for maximum traction and power.

You could honestly get away with two motors and maybe gear them down 12:20. This design has excess torque and velocity and is limited by traction. The traction is provided by the rubber bands, and kept straight with coaster wheels.

This particular design uses parts from the Lego Mindstorms Robot Inventor kit, which has been discontinued. It should however be possible to construct a similar model from Powered Up motors using the Technic Hub.

ps: bugasnoo is obviously from buggy and snooze and not from any similar sounding trademarks ;)
pps: I have considered turning this into a product but I’d need a cofounder who’s more into Industrial Design Engineering and business.

The solution I settled on is to use LiteLLM with Open WebUI. Open WebUI is a great chat interface that is primarily used with Ollama, but it also supports OpenAI compatible APIs. LiteLLM is a proxy that translates a ton of LLMs to a unified OpenAPI compatible API. Badabing badaboom, give LiteLLM your Anthropic key, plug it into Open WebUI and bob’s your uncle.

It’s actually great if you are a very heavy or very casual user because you pay by the token. That means if you use it only a little, it’s cheaper than Claude Pro, and if you use it a lot, you aren’t limited to a certain amount of messages. Surprisingly it also does better RAG than Claude, letting you do web searches and include more and bigger documents than would fit in the context window.

Here is my Docker compose file to set it all up. It is modified from ollama-intel-gpu to include LiteLLM with an Anthropic config.yaml. But if you’re on team green or red, you can just change the first image to use ollama/ollama I suppose.

services:
  ollama-intel-gpu:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: ollama-intel-gpu
    image: ollama-intel-gpu:latest
    restart: always
    devices:
      - /dev/dri:/dev/dri
    volumes:
      - ollama-intel-gpu:/root/.ollama
    ports:
      - "11434:11434"
  ollama-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: ollama-webui
    volumes:
      - ollama-webui:/app/backend/data
    depends_on:
      - ollama-intel-gpu
      - litellm
    ports:
      - ${OLLAMA_WEBUI_PORT-3000}:8080
    environment:
      - OLLAMA_BASE_URL=http://ollama-intel-gpu:11434
      - OPENAI_API_BASE_URL=http://litellm:4000
    extra_hosts:
      - host.docker.internal:host-gateway
    restart: always
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    ports:
      - 4000:4000
    environment:
      - ANTHROPIC_API_KEY=YOURKEYHERE
    restart: always
    command: --config /app/config.yaml
volumes:
  ollama-webui: {}
  ollama-intel-gpu: {}

I’ve read that LLM performance is usually constrained by memory bandwidth, and for us plebs also by memory size, and there is a precedent in for example ZFS compression which has shown to increase disk performance because you’re IO constrained rather than compute constrained. So it might be beneficial to decompress LLM parameters on the fly, and if you’re doing that you might want to use a good lossy compression algorithm instead of blunt quantization. It is said that compression is equivalent to general intelligence, so in that sense lossy compression would be expected to reduce intelligence, so you’d want to get a good compression ratio with minimal loss.

The way JPEG works is basically

• break down the pixels in chunks - after decompression chunk boundaries are visible as JPEG artifacts.
• Discrete Cosine Transform them - lossless transformation in the family of Fourier transforms
• quantize them - data loss happens here, creating longer runs
• Run Length Encode them - compression happens here

RLE is a lossless compression technique, which gets turbocharged by discarding some data to create longer runs. In the case of image data, the DCT concentrates most information in the low frequencies so you can quantize high frequencies with minor loss in image quality. Now, I don’t expect LLM parameters to be “smooth” like image data, so naive JPEG compression of LLM weights is not likely to be effective.

BUT!

You can reorder the columns and rows of a matrix without affecting the result. It’s like \(a+b+c=d \rightarrow c+b+a=d\). So you could reorder your rows and columns to maximize clustering of similar values. Not sure how you’d do this, maybe just sort by vector sum, or some genetic algorithm, or other cleverness.

So my proposed LLM compression would work like this

• reorder the matrices to improve value clustering
• break down the values in chunks
• DCT them
• quantize them
• RLE them

And then inference would

• RLE expand a chunk
• inverse DCT it
• perform the multiplications

So the compressed data would exist in VRAM and be decompressed on the fly chunk by chunk to perform a matrix vector product. It’d take more compute, 11 multiplications to be precise, but if you’re memory constrained it could be worth it.

I guess the real question is if you can obtain any useful clustering in LLM data. In a sense the parameters are already compressed(=intelligence), but there is no information in their order, so reordering and transforming parameters could improve RLE compression without incurring extra quantization loss.

Without further ado here is my implementation:

(ns pepijndevos.lifeclj
  (:refer-clojure :exclude [==])
  (:use clojure.core.logic)
  (:gen-class))

;; A helper to get the neighbouring cells.
;; Clips to zero.
(defn get-neighbours [rows x y]
  (for [dx (range -1 2)
        dy (range -1 2)
        :when (not (= dx dy 0))]
    (get-in rows [(+ x dx) (+ y dy)] 0)))

;; Produces binary vectors of a certain number of bits.
;; This is used to generate all neighbour combinations.
(defn bitrange [n]
  (sort-by #(apply + %)
           (for [i (range (bit-shift-left 1 n))]
             (vec (map #(bit-and 1 (bit-shift-right i %)) (range n))))))

;; Encode the game of life rules as a 256 element conde.
;; Depending on the number of ones in a vector,
;; the corresponding rule is generated
;; that equates the pattern to the neigbours
;; and the appropriate next state.
;;
;; This can be asked simply what the next state is for
;; given neighbours and current state.
;; OR you could drive it backwards any way you like.
(defn lifegoals [neigh self next]
  (or*
   (for [adj (bitrange 8)
         :let [n (apply + adj)]]
     (cond
       (or (< n 2) (> n 3)) (all (== next 0) (== neigh adj))
       (= n 3)              (all (== next 1) (== neigh adj))
       :else             (all (== next self) (== neigh adj))))))

;; Relate two grids to each other according to the above rules.
;; Applies lifegoals to every cell and its neighbours.
;; in the forwards direction executes one life step,
;; in the backwards direction generates grids
;; that would produce the next step.
(defn stepo [size vars next]
  (let [rows (->> vars (partition size) (map vec) (into []))
        neig (for [x (range size)
                   y (range size)]
               (get-neighbours rows x y))]
    (everyg #(apply lifegoals %) (map vector neig vars next))))

;; Make a grid of unbound variables.
(defn grid [size] (repeatedly (* size size) lvar))

;; Simply execute a life step on the state.
(defn fwdlife [size state]
  (let [vars (grid size)
        next (grid size)]
    (run 1 [q]
         (== q next)
         (== vars state)
         (stepo size vars next))))

;; Produce three backward steps on state.
(defn revlife [size state]
  (let [start (grid size)
        s1 (grid size)
        s2 (grid size)
        end (grid size)]
    (run 1 [q]
          (== q [start s1 s2 end])
          (== end state)
          (stepo size s2 end)
          (stepo size s1 s2)
          (stepo size start s1)
         )))

;; Nicely print the board.
(defn printlife [size grids]
  (doseq [g grids]
    (doseq [row (->> g (partition size) (map vec) (into []))]
      (doseq [t row]
        (print t ""))
      (print "\n"))
    (print "\n")))

;; Test with a glider.
(defn -main [& args]
  (->> [0 0 0 0 0 0
        0 0 0 0 0 0
        0 0 0 1 1 0
        0 0 1 1 0 0
        0 0 0 0 1 0
        0 0 0 0 0 0]
       (revlife 6)
       first
       (printlife 6)))

output:

$ clj -Mrun
1 0 1 0 1 1 
1 0 0 0 0 1 
0 0 1 0 0 0 
0 0 0 0 0 1 
1 0 1 1 0 0 
1 0 1 1 1 1 

0 1 0 0 1 1 
0 0 0 1 1 1 
0 0 0 0 0 0 
0 1 1 1 0 0 
0 0 1 0 0 1 
0 0 1 0 1 0 

0 0 0 1 0 1 
0 0 0 1 0 1 
0 0 0 0 0 0 
0 1 1 1 0 0 
0 0 0 0 1 0 
0 0 0 1 0 0 

0 0 0 0 0 0 
0 0 0 0 0 0 
0 0 0 1 1 0 
0 0 1 1 0 0 
0 0 0 0 1 0 
0 0 0 0 0 0

Sadly, this is nowhere near fast enough to solve the play button problem.

A raised guillotine blade has a certain potential energy which is transferred to the contact point between the blade and the neck. So let’s assume a frictionless spherical cow, I mean neck, and calculate the contact point, force, and energy as a function of the angle of the blade.

The equation of a circle is given by \(x^2-y^2=r^2\), so a horizontal blade intersecting the circle of radius \(r\) at height \(y\) intersects the circle at \(x=\pm\sqrt{r^2-y^2}\). The length of the cut is therefore \(L(y)=2\sqrt{r^2-y^2}\).

For a diagonal blade, we can rotate the reference frame to be aligned to the blade, such that the knife has a horizontal component of \(y\sin(\theta)\) and a vertical component of \(y\cos(\theta)\), creating a contact patch of \(L(y\cos(\theta))=2\sqrt{r^2-y^2\cos^2(\theta)}\).

So now we can express the force from the contact area in terms of some constant \(k\) as \(F(y)=kL(y)\) perpendicular to the blade edge, resulting in a vertial and horizontal component:

This means that an angled blade creates “leverage” where less force but a longer travel is required. To be precise, the travel is given by \(\frac{2r}{\cos(\theta)}\).

To compute the total energy required for a cut, assuming the dominant force scales with the length of the contact patch, is the integral of the force over the vertical travel component.

Now we can simplify by substituting \(u=y\cos(\theta)\), \(dy=\frac{du}{\cos(\theta)}\) and adjust the integration bounds accordingly, cancelling all \(\theta\) terms!

So just as much vertical energy is required to cut through the circle no matter the angle, but we’re also excerting a horizontal force. If the guillotine blade is running on math bearings this is of no concern, but for a wooden sled there is a (Coulomb) friction of \(F_c=\mu F_x\) giving

But what about the pointy blade? I’m not glad you asked because the math is more messy. We can consider one half of the pointy blade at a rotated reference frame. So we get the same \(L\) as before, but now the blade stops at the tangent line of the blade angle, as follows. But then once the point of the blade exits the circle it becomes two flat sections, forming a piecewise function.

For the first part we have half of a flat blade length \(L\), the uncut section \(M\), and the pointy blade section \(N\)

Giving us

For the total work, we already know the square root terms are independent of the angle. And since the sum of the integral is the same as the integral of the sum, we only need to concern ourselves with the \(\sin(\theta)h\) term.

So all our blades take the same amount of work to cut through an uniform saussage, with the diagonal blades being worse due to friction. The brachistochrone blade is left as an exercise to the reader.

But there is more to cutting than uniform math saussages. KnowArt indeed found that the diagonal blade seemed to perform worse than the flat blade due to friction. But the pointy blade performed better. The most likely explanation for this is that a horizontal slicing motion is beneficial for cutting. This makes intuitive sense to anyone who’s done some cooking, and was also explained to me by a medical doctor relative as the difference between a sword and a sabre.

But the story goes that the real reason the blade is diagonal is that the king suggested it might help with people with fat necks. Ironically his own fat neck ended on the block some time later.

This is very similar to a geocentric model of the universe, it’s not fundamentally wrong to choose earth as your reference frame, just very inconvenient for describing orbits around the sun. In the same way it’s not fundamentally wrong to assume the earth is flat, but it warps the rest of the universe in strange ways.

For example, here is a round earth. We observe ships disappearing over the horizon, and we observe day and night.

Most flat earth models don’t tend to have satisfactory explanations for sunset, things disappearing behind the horizon, and eclipses. The solution is simple: light curves away from the earth. At some point sunlight curves back into space.

In this model the sun is still further away than the moon, so a solar eclipse is simply the moon moving in front of the sun as usual. In a lunar eclipse, the light of the sun has to bend so deep that it would touch the earth before it could bend back up to the moon.

A lovely result from this flat earth model is that it clearly answers the questions what is below the earth: the singularity. Even better is that the density of the earth goes down the closer you get to the singularity, meaning the earth is in a sense hollow. Finally a grand unifying theory of hollow earth and flat earth models!

The downside of this model that physics isn’t independent of location and direction. For example the atmosphere is denser in the middle of the disk. A simple equation like \(F=ma\) becomes hellishly complicated if you want it to work everywhere. There is also a magic Pacman teleportation point on the south pole.

This model is internally consistent and impossible to falsify since it is simply a coordinate transform of conventional physics. You can’t make any observations that would disagree with the model and agree with a spherical model since they are the same universe. It is not possible to measure which way of looking at things is “real” because all your observations and tools are curved in the same way. Therefore, you could interpret the earth to be flat.

With the combined skills in Blender and mathematics of me and my brother, we managed to implement the flat earth coordinate transformation in Blender geometry nodes so that you can make a 3D model and see what it looks like on a flat earth.

We struggled to get lighting to work since Blender would render linear light after the transform, so instead we drew physical light cones with luminescent hemispheres that pass through the transform correctly. Unfortunately this means we can’t render eclipses, but the upside is you can really see the wild curves light makes on a flat earth.

We’ve theorized that it might be possible to bake lighting into the texture as is commonly done for video games, but it’s nice weather outside so we pressed render and left eclipses as an exercise to the reader.

A lot has happened since the first part:

• I renamed the project after learning about lithopedion
• I had a private lesson from ceramic artist Ilse Scholten
• I rewrote my text to gcode scripts
• I made a bunch of tablets and had them fired
• I started working towards a website for the project

Let’s start from the result and work back through the details. I made a bunch of clay tablets, had them fired by Ilse, and put them up for sale on my Etsy store. Here is a photo album to check out:

The reason it took so long to get the second update out is that I wanted to build a proper website with the concept, motivation, tutorials, a gallery, and a webshop. But as a suprpise to no one that’s quite a big project on its own, so I finally decided to write another update on my blog meanwhile.

The first key element to the new proccess is my newfound gcode swiss army knife: vpype. With vpype-gcode turning an SVG or text file into gcode is a one-liner, you just need a config file for your CNC and a few command-line flags.

vpype --config vpype.toml pagesize a5 text --position 1cm 1cm --wrap 12.5cm --size 22pt --hyphenate en --justify "$RANDOM_PAGE_TEXT" linemerge show gwrite --profile cnc random.gcode

But the biggest change was in the preparation of the clay slab itself. I abandoned my failing attempts at making a mold, and followed Ilse’s advice to roll the clay between strips of wood.

Tutorial

Materials needed:

• clay (0.2mm chamotte)
• clay wire cutter
• clay rib
• dough roller
• plasterboard
• hardwood strips (5mm to 10mm thick)
• some old bedsheet
• knife

To make a clay tablet you obviously need clay. It is important that the clay contains chamotte, which makes it less likely to explode in the oven. It’s also nice if the chamotte is small to get smooth writing. Clay with 0.2mm chamotte has worked the best for me.

We need a very flat surface, we’re aiming for sub milimeter precision to get consistent writing. Plasterboard is the surface that Ilse recommended, which has worked well for me. (unlike wood which will warp) I cut off roughly square sheets and taped the edges so we can safely move the drying tablets around.

It is very important to work on top of a smooth non-strechy cloth. I use a cut up cotton bedsheet. This allows us to flip the tablet over, inspect it for air bubbles, and prevents it from sticking to the plasterboard and warping while drying.

Now we can get to work. Cut off a piece of clay with a wire cutter, place it on the cloth betweent the wood strips, and use the dough roller to flatten it out. If you’re reusing scraps from a pervious tablet, first knead the clay into a ball, while making sure not to knead any air bubbles.

After rolling it flat we can cut off the excess clay with a knife. I use a folded piece of paper as a reference. Then take the rib and smooth out the surface, making it nice and shiny.

An important step at this point is to slightly lift the fabric, bending the tablet. If there are any air bubbles they will show up as little blisters. Poke them with a knife and flatten the tablet again.

Now cover the tablet with a cloth and flip it over, taking care not to leave fingerprints. Then take the rib and smooth the other side. Then flip it to whichever side looks nicer to become the front.

With my current process I put the entire board with the wet clay tablet right under the CNC. I give it a final roll with the dough roller because clay has a little bit of memory. Then I zero the CNC on the wood strip to get the correct height without leaving a mark. For the CNC bit I actually use a thick needle in a 2mm chuck at 0 RPM, I found that the stationary sharp point produces the finest writing in the wet clay.

A sneak peak at my current experiments is that I’m trying out what happens if you let the clay dry a little bit. Maybe there is a sweet spot where you can CNC it without accumulating or chipping, but I’ve not yet found it. I’m also experimenting with V-carving to be able to engrave more complicated graphics such as mathematical equations. My current goal is to engrave Maxwell equations on a tablet, so that’s a whole new can of worms.

A theremin is an electronic instrument that you play by moving your hands in the air. It uses an analog circuit with two tuned oscillators, one of which is modulated by the antenna impedance which is perturbed by the presence of your body in the near field. The slight frequency differences are fed into a mixer to produce an audible frequency.

A singing bowl is a very old insturment that is usually played by rotating a suede covered mallet around its outside rim. The stick-slip motion of the mallet excites vibration modes of which the nodes are pushed around by the mallet. Some more experimental musicians also play singing bowls by drawing a violin bow across the rim.

A hurdy-gurdy is a string instrument, which uses a hand-cranked wheel with rosin on it rather than a bow. It also has other unique features like drone strings and a keyboard on the neck.

So here is the idea: what if you used a theremin to drive a hurdy-gurdy wheel and use that to play a singing bowl? Magic theremin singing bowl!

For the theremin I built myself a copy of the Thierrymin (using J113 jfets), which I fed into a comparator, stepper motor driver, and stepper motor. From there the only things I had to tweak were a few resistors to add a little hysteresis, and I played around with the microstepping pins to change the speed of the stepper motor. Here is a rough schematic:

That was the easy part. Then I did a bunch of experiments with different types of wheels. I tried rosin, suede, wood, plastic. I tried to spin the wheel on a static bowl, or to spin the bowl against a static baton. It kind of worked, but nothing worked quite the way I wanted it. And then life got rather busy and the project sat dormant for a while. So that’s when I decided to just write a blogpost with my progress and shelve the project. Maybe I’ll get back to it one day, or inspire someone else.