Given the recent popularity of #Dalle2 and #StableDiffusion, I thought it would be fun to write a thread about a different kind of generative art.

Each of the videos below is a randomly generated simulation and today I will explain about the project that created them.

A 🧵!

1/

These are all Cellular Automatons, randomly generated by my Neural-Net Controlled CA Engine. 

The goal of this thread will be to show you how the hell can we build our way up from from simple cellular automatons to the beautiful forms you see above.

Let's start! 🚀 

 2/

Every beginner programmer is likely familiar with Conway's infamous Game of Life, one of the simplest yet most beautiful examples of a CA. 

In the GoL a grid of cells with binary values evolves over time using extremely simple rules that create complex, emerging patterns. 

3/

The value of every cell at timestep t, is a function of its own value, and the values of its immediate neighbors from step t-1. So C(x, t) = F(C(x, t-1),  C(nb_xs, t-1)).

We can visualize these rules: 
(ruleset is visualized on the left, and the resulting CA is on the right)

4/

Let's start by making the CA continuous. Instead of the values being binary, 0 or 1, they can now be anywhere in between 0-1. 

And the ruleset is now a continuous function F(x,y) which (for now) looks like a smoothed version of the previous policy

5/

The next step is to upgrade the neighborhood mechanism. Instead of a cell and its immediate neighbors, we will now be looking at an inner neighborhood and an outer one. 

6/


So now for every cell, we will calculate a weighted average based on its inner and outer neighborhoods, and will apply the ruleset according to these two values.

When running with this mechanism it looks like so:
(ruleset was slightly changed to make the result prettier)

7/

So this is our basic setup, now we can start playing with the ruleset. For example, we can take a high resolution grayscale-image and use it instead of F as a lookup table. 

Here is what the twitter logo looks like as a cellular automaton!

8/

Now this next step is where things get weird, and fun!

We can use a simple, feed-forward neural net to represent F(x, y), to represent the ruleset. It will receive two values (inner, and outer neighborhood averages at timestep t) and output one value (cell value at t+1) 

9/

Since the network receives two values as inputs and outputs one value, we can visualize the ruleset as a grayscale image. Below are examples to policies from a few randomly parameterized neural networks. 

10/

As you can see, these are pretty boring. What I did to make these more interesting is use Fourier Features as described here: https://bmild.github.io/fourfeat/ which basically puts our low-dimensional inputs through a series of trigonometric functions of different magnitudes.

11/

Just take a look what a difference it makes in our policies: 

12/

Much better, these look complex and versatile.

Now let's cycle through a few of these policies and see what they actually look like.

13/

As you can see these range from very boring to quite chaotic. So now we can introduce constraints, so as to make the results a bit more interesting. 

1. Instead of s[t+1] = F(s[t]) interpolate it with a smoothing  coefficient: 

s[t+1] = s[t] * a + F(s[t]) * (1-a)


14/

These are starting to look awesome! But..

I tend to think of pixels with higher value as pixels with more mass and vice versa. With that in mind, it always bothered me that mass seems to appear and disappear out of nowhere. 

15/

So the way I made it a bit more realistic is by introducing the following changes:

1. At every frame, softmax S[t+1] and multiply it by the desired amount of "mass" in the simulation

2. At every frame, normalize S[t+1] such so that it has a mean of 0 and variance of 1

16/


We can do either one of these or both at the same time. When we do both it looks like this! 

17/

Now of course grid-global normalizations like these are not a "correct" way of enforcing a constant mass, since a surge of mass creation in one area will cause mass depletion in all other areas. I have ideas on how to implement a more realistic system but more on that later 

18/

To make these more presentable, we can make the grid larger and use pretty colormaps for visuals: 

19/

And lastly! (for now) we can make this 3-channeled! So the network will now take 6 values per cell and output 3 values per cell (rgb). This will create full-color videos, and with all the above constraints applied channel-wise, here's (some) of what we get:

20/

And that's it! 

Thanks for reading and I hope you found these as awesome as I do. Lmk if you have any questions, if there's enough interest I'll throw it in a repo for all to enjoy. 

21/

This is implemented in Pytorch, without compute shaders as is commonly used for CAs. So there is GPU utilization but probably not as good as it could've been. 

Regarding future work I have two main directions: 

22/

1. Improve the network architecture to get more interesting results. Perhaps a CPPN (Like they used in pic-breeder) seems perfect for this kind of task 

23/

2. Work on actual proper mass retaining, instead of each cell deciding how much mass it has in s[t+1] it needs to control some sort of mass attract/repel scalar and let the mass diffuse accordingly. 

Maybe something like:  https://michaelmoroz.github.io/Reintegration-Tracking/
(which is fascinating btw)

/24

It's cool to see how far this project has come. I wrote the first version when I was 15, just starting out.

It was terribly implemented, and in raw-java, but even back then it looked cool: https://www.youtube.com/shorts/2mE8Zs-9JxU

Thanks for reading! Follow for more cool project write ups.
25/25

You can read the unrolled version of this thread here: https://typefully.com/ereb0s_rl/2u345nc

Neural Network Controlled Cellular Automaton Engine

You can read the unrolled version of this thread here: https://typefully.com/ereb0s_rl/9Bh9sZg