Ever wondered how a blur filter works? 

Well, you create a matrix of weights, called a kernel, and then use those to average the pixel values surrounding your target pixel. 

We tend to use a Gaussian distribution for the kernel weights, hence Gaussian blur.

The larger the kernel (radius) the heavier the blur and the more expensive the operation. 

But because our kernel is symmetrical, we can actually split it up into two passes. We blur the vertical pixels and then the horizontal pixels for the same result but fewer operations.

If we are using a backdrop blur there's the added step of computing the area that needs to be blurred and then essentially rasterising that. No wonder they are so expensive.

Anyway, just thought that was interesting. 


You can read the unrolled version of this thread here: https://typefully.com/DanHollick/oZtKyyX

How Gaussian Blurs work

I know everyone is pissed about that iPad ad but I was too busy being excited about those new tandem OLED displays. 

Essentially they are just two OLED panels glued together - so what makes them so exciting? 

Let me explain 👇 

First we need a quick refresher on how OLEDs work. 

Instead of using a backlight like transmissive displays, OLEDs use clever physics and chemistry to emit light directly from the sub pixel.

The advantage here is that they don't have a backlight. 

This means they can display an almost pure black, making them much higher contrast.

They are also more power efficient because each pixel only consumes as much current as it needs.

But here's the bummer - they can't get very bright. There's a natural limit to how much light each pixel can emit. 

This is where tandem OLED comes in. By sandwiching another layer of emissive pixels on top you can almost double the brightness.

Weirdly, this also makes the displays use less power. 

Because OLEDs are less power efficient at higher current densities, being able to run each layer at a lower current means you can save power overall for a given luminance. Which also helps reduces burn-in.

But it's not all sunshine and rainbows.

Adding another layer causes the distance between the emitters and the top metal electrode and substrate to increase. These aren't perfectly transparent so they can start to reflect light hitting it at certain angles.

This reflection means a lot of this extra light gets lost internally. This is called Light Extraction Efficiency (LEE). 

There are a few ways to get around this and they all try to prevent the light from hitting the substrate at an angle that will reflect it back down.

All these techniques increase the manufacturing cost and complexity which has meant no-one has put these into production. Until now.

Interestingly, we don't actually know yet which technique is being used here.

Anyway, this is pretty huge - they are essentially the perfect display.

If the panel manufacturer can make these cost effective at larger sizes we might see them in the Macbook soon. 

Or even a new Pro XDR - which sucks because I just bought one a few months ago.


You can read the unrolled version of this thread here: https://typefully.com/DanHollick/JuPJtrn

Tandem OLED Displays

How does a Large Language Model like ChatGPT actually work?

Well, they are both amazingly simple and exceedingly complex at the same time.

Hold on to your butts, this is a deep dive ↓

You can think of a model as calculating the probabilities of an output based on some input.

In language models, this means that given a sequences of words they calculate the probabilities for the next word in the sequence.

Like a fancy autocomplete.

To understand where these probabilities come from, we need to talk about something called a neural network.

This is a network like structure where numbers are fed into one side and probabilities are spat out the other.

They are simpler than you might think.

Imagine we wanted to train a computer to solve the simple problem of recognising symbols on a 3x3 pixel display.

We would need a neural net like this:
- an input layer
- two hidden layers
- an output layer

Our input layer consists of 9 nodes called neurons - one for each pixel. Each neuron would hold a number from 1 (white) to -1 (black).

Our output layer consists of 4 neurons, one for each of the possible symbols. Their value will eventually be a probability between 0 and 1.

In between these, we have rows of neurons, called "hidden" layers. For our simple use case we only need two.

Each neuron is connected to the neurons in the adjacent layers by a weight, which can have a value between -1 and 1.

When a value is passed from the input neuron to the next layer its multiplied by the weight.

That neuron then simply adds up all the values it receives,  squashes the value between -1 and 1 and passes it to each neuron in the next layer.

The neuron in the final hidden layer does the same but squashes the value between 0 and 1 and passes that to the output layer.

Each neuron in the output layer then holds a probability and the highest number is the most probable result.

When we train this network, we feed it an image we know the answer to and calculate the difference between the answer and the probability the net calculated.

We then adjust the weights to get closer to the expected result.

But how do we know *how* to adjust the weights?

I won't go into detail, but we use clever mathematical techniques called gradient descent and back propagation to figure out what value for each weight will give us the lowest error.

We keep repeating this process until we are satisfied with the model's accuracy.

This is called a feed forward neural net - but this simple structure won't be enough to tackle the problem of natural language processing.

Instead LLMs tend to use a structure called a transformer and it has some key concepts that unlock a lot of potential.

First, lets talk about words.

Instead of each word being an input, we can break words down into tokens which can be words, subwords, characters or symbols.

Notice how they even include the space.

Much like in our model we represent the pixel value as a number between 0 and 1, these tokens also need to be represented as a number.

We could just give each token a unique number and call it a day but there's another way we can represent them that adds more context.

We can store each token in a multi-dimensional vector that indicates it's relationship to other tokens.

For simplicity, imagine a 2D plane on which we plot the location of words. We want words with similar meanings to be grouped closer together.

This is called an embedding.

Embeddings help create relationships between similar words but they also capture analogies.

For example the distance between the words dog and puppy should be the same as the distance between cat and kitten.

We can also create embeddings for whole sentences.

The first part of the transformer is to encode our input words into these embeddings.

Those embeddings are then fed to the next process, called attention which adds even more context to embeddings.

Attention is massively important in natural language processing.

Embeddings struggle to capture words with multiple meanings.

Consider the two meanings of 'bank'. Humans derive the correct meaning based on the context of the sentence.

'Money' and 'River' are contextually important to the word bank in each of these sentences.

The process of attention looks back through the sentence for words that provide context to the word bank.

It then re-weights the embedding so that the word bank is semantically closer to the word 'river' or 'money'.

This attention process happens many times over to capture the context of the sentence in multiple dimensions.

After all this, the contextualised embeddings are eventually passed into a neural net, like our simple one from earlier, that produces probabilities.

That is a massively simplified version of how an LLM like ChatGPT actually works. There is so much I've left out or skimmed over for the sake of brevity (this is the 20th tweet).

If I left something important out or got some detail wrong, let me know.

I've been researching this thread on/off for ~6 months. Here are some of the best resources:

Obviously this absolute unit of a piece by Stephen Wolfram which goes into much more detail than I have here:

https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/

For something a bit more approachable, this @3blue1brown video series about how neural nets learn.

There are some lovely graphics here, even if you don't follow all the maths.

https://youtu.be/aircAruvnKk?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi

And @CohereAI's blog has some excellent pieces and videos explaining embeddings, attention and transformers.

https://txt.cohere.ai/what-is-attention-in-language-models/
https://txt.cohere.ai/sentence-word-embeddings/


How ChatGPT works: a deep dive

Why are some typefaces harder to read than others at the same font-size?

Well, it has a lot to do with x-height but of course it's a bit more complicated than that: ↓

You probably know this already but the x-height of a typeface is the difference between the baseline and the height of the lowercase letters.

We can also think of x-height as a ratio of the total cap height or body height.

Typically we use the letter x to determine this, hence the name.

Interestingly, curved lowercase letters like a, c, and e are often slightly taller than the x-height. They purposefully overshoot so that they appear the same visual height as x, v, w, and z.

Anyway...

Researchers have studied how x-height impacts reading speed and found that there is an optimal x-height size.

This optimal x-height is around 0.3° of visual arc (i know wtf) after which reading speed begins to decrease.

Now what the fuck does arc mean?

Arc is a way of measuring how much of your field of view an object takes up, in degrees.

This depends on both how big *and* how far away something is - removing all the variance caused by pixel size and density.

Practically, this means that typefaces with x-heights outside of this optimal rage need to be either larger or closer to have the same legibility as typefaces within this range.

If I assume a viewing distance of 40cm from my LG 4K monitor I can roughly work out the arc size of some typefaces and you can see that they are in the ballpark:

Of course, there is a lot more to legibility than just x-height.

Font-weight and contrast matter as much, if not more but I thought this was an interesting topic.

There is a great article on this here:
https://www.imarc.com/blog/best-font-size-for-any-device

And the original paper is a surprisingly gripping read:
https://jov.arvojournals.org/article.aspx?articleid=2191906

If you want to go down the rabbit-hole of pixel sizing I wrote about that here:

https://twitter.com/DanHollick/status/1501584351842111488?s=20

I also wrote about font rasterisation, which is kinda unrelated but interesting nonetheless:

https://twitter.com/DanHollick/status/1441110053818302474

You can read the unrolled version of this thread here: https://typefully.com/DanHollick/9p1b5IsMqv8X

The Role of X-Height in Typefaces

Why is making a dark mode greyscale so hard to get right?

Well, of course it has to do with the weird way humans perceive colour and contrast. 👇

The main issue is we we perceive contrast as higher between lighter colours.

Even though these two sets of colours have the same mathematical contrast we perceive the left as having more contrast than the right.

So in designs that use a lot of subtle light greys (containers, borders etc) it isn't as simple as just inverting the colour scale.

The contrast between darker greys will be lower as you can see below.

To fix this, you need to space the colours further apart to achieve the same contrast. This is tricky if your grey scale was set up for light mode.

One tactic I use is to create the scale based on the APCA contrast, which is much more reflective of human perception.

This ensure that contrast remains pretty consistent across modes even when you just flip the scale.

Not perfect but close.

So why do we tend to tint dark backgrounds blue?

Well, our perception of contrast isn't linear across hues. For instance, we perceive green as much brighter than blue.

This means we are much more sensitive to a green tint than we are to a blue tint.

Adding green or red just alters the tone much more than blue does, and pure black feels unnatural to some.

Blueish-blacks are more common in nature too (night sky).

But, of course this complicates your grey scale.

As the scale moves from light to dark the hue component will also need to move gradually towards blue.

The blue tinting looks very weird in the lighter shades (just look at my profile picture)

In summary, this shit is hard.

Below you'll find a bunch of threads I've done about some of this colour stuff before.

This one is about the difficulties of modelling human perception of contrast.

https://twitter.com/DanHollick/status/1417895151003865090?s=20

If you want to learn more about APCA, there is more detail in this thread.

https://twitter.com/DanHollick/status/1468958644364402702?s=20

You can use APCA in figma right now:
https://www.figma.com/community/plugin/806578669827234193/zebra

This one is about perceptually uniform colour spaces.

https://twitter.com/DanHollick/status/1488889983058382848?s=20

Another one about why sRGB sucks.

https://twitter.com/DanHollick/status/1471186282810277893?s=20

Unevenly spaced color systems:

https://twitter.com/DanHollick/status/1481350977466900484?s=20

Why some gradients are muddy:
https://twitter.com/DanHollick/status/1463209495044104193?s=20


You can read the unrolled version of this thread here: https://typefully.com/DanHollick/UceA0ne

Darkmode

Ever heard of a shader but too afraid to ask what it even means at this point?

Lets fuck around and find out 👇

First, we need to understand the difference between a CPU and GPU and why we need a separate unit just to draw graphics anyway.

(Don't worry, it's not going to be too technical.)

The thing that makes a CPU great is that it can perform a wide variety of tasks relatively quickly.

But the limitation is that each CPU core is optimised to do one thing at a time and so tasks are largely performed sequentially.

This isn't ideal for rendering. Drawing something 30+ times a second starts to occupy a lot of CPU capacity.

For high-res graphics changing in real time the CPU on it's own will struggle to keep up - plus it's got other CPU shit to do.

The GPU by contrast is specifically designed to render graphics  by being able to do thousands of things at once.

Picture it as thousands of tiny units all strapped together, each just figuring out the colour of a pixel.

This parallel processing is really, really fast.

But the consequence of parallelism is that each separate thread has no idea what the others are doing - making programming tricky.

You can't change one pixel value based on the result of a separate pixel.

In fact, not only does each thread have no idea what the other threads are doing, they have no idea what they have just done.

They are memoryless, so you can't compute an input based on a previous output.

This is kinda what shaders are - code designed to produce graphics on the GPU within these constraints.

We call a 2D shader a fragment (or pixel) shader because it's a piece of code that calculates the colour of a single fragment of the screen - often a single pixel.

So with all these constraints, how do you actually do anything useful in a shader?

Well, you can do a lot with a little. Just knowing the fragment position and the resolution of the canvas can get you quite far.

Here's a simple example on a 40x40 grid.

If we take the fragment position and divide it by the resolution of the whole canvas we can then render a different colour at every fragment.

It creates this gradient across the canvas.

If you add time as a variable you can start to animate the gradient.

And if you throw some more maths at it you can create some really amazing shit.

https://www.shadertoy.com/view/XslGRr

So why am I telling you about this specialised graphics programming?

Well they are starting to be used more on the web, thanks to improvements in hardware and web technology.

This animated gradient on stripe.com is a shader.

Same with this liquify effect:

https://designembraced.com/case/emblem-rings

Anyway, this thread only scratches the surface of shaders - we haven't even discussed 3D and vertex shaders.

(Checkout this article by @MaximeHeckel )
https://blog.maximeheckel.com/posts/the-study-of-shaders-with-react-three-fiber/

Most of this was taken from the amazing resource:
The Book of Shaders by @patriciogv and @_jenlowe_.

It's the best place to get started with programming shaders.

https://thebookofshaders.com/


If you wanna play with shaders in Figma, @rogie's Noise and Texture plugin has a playground just for that.

https://www.figma.com/community/plugin/1138854718618193875

You can read the unrolled version of this thread here: https://typefully.com/DanHollick/gpnhhud

Shaders

If you are used to Figma, there's a quirk of Framer components that might catch you out.

In Figma, variants can have completely separate layers. But in Framer, each variant contains all the layers from all other variants hidden inside of it.

Let me explain.

Take this accordion component.

There are two states: collapsed and expanded. The expanded state reveals a frame containing the answer.

If we look at the collapsed state we can see that the answer is also a child of this variant - just with the visibility set to hidden.

You can think of components in Framer as big group of layers and each variant is just a configuration of the properties of all those layers.

They are conceptually similar to boolean properties in Figma, where you are just configuring the visibility of a specific layer.

So why does it work like this? I'm not 💯 sure but I think it's for Framer Motion.

It needs to know what all the possible states are to work out how to make layout transitions between them in CSS.

(more details here https://www.nan.fyi/magic-motion via @nandafyi)

But here's where things get a little complicated.

Because you can edit layers that belong to other variants from anywhere, Framer links them.

This means you don't have to manually make changes to each variant.

The basic rule is: layers inherit the properties of the variant they were copied from.

If you change a property on the original, that change rolls out to the copy.

Once you override it on the copy, the link is broken for that specific property.

Here's an example.

I added hover variants to each state. If I change the background colour you can see how inheritance works and how you can break it.

You can also relink a property (right click -> reset overrides)

There is a weird exception that catches me out.

If a layer is hidden in the original but visible in the copy any changes you make to the copy will propagate back to the original.

I think this might be to make sure it animates in smoothly but I'm not sure.


Some bonus reading from @MaximeHeckel about layout animations with Framer Motion.

https://blog.maximeheckel.com/posts/framer-motion-layout-animations/

You can read the unrolled version of this thread here: https://typefully.com/DanHollick/Y1XT7N4

Framer Components

Did you know these are called Boolean Operations?

That's because they use booleans to determine which part of the shapes should be visible.

Let me explain 👇

To keep it simple lets use two overlapping circles.

Imagine a boolean for each circle called "inside" - for the area inside each circle the boolean is true, for the area outside it is false.

Any point within the bounding box can be represented in terms of these booleans.

Boolean operations are just a way of combining these booleans together to make a new shape.

Take Union for example- the new shape will consist of any area that is inside the red circle OR the blue circle.

Because we have combined the two booleans, there are now 4 possible states (2x2) :

- Outside Red, Outside Blue
- Outside Red, Inside Blue
- Inside Red,  Outside Blue
- Inside Rad, Inside Blue


You can see this more clearly with Intersect which creates a new shape where the two shapes overlap.

Put differently, anywhere "inside" is true for BOTH shapes.

Difference/Exclude is almost the inverse of the Intersect.

It creates a shape anywhere "inside" is true for one but NOT BOTH of the shapes.

Subtract works differently in that it matters which shape is the primary shape. In Figma's case the top layer is subtracted from the bottom.

The new shape is anywhere "inside" is true for the primary shape AND NOT the secondary shape

This example is pretty simple but Boolean Operations are extremely powerful and work on complex shape combinations.

The math required to determine the overlapping areas of complex shapes deserves a whole thread on its own.

If you think these operations look familiar it's because they are essentially the logical operators we use in programming:

AND - Intersect
OR - Union
NAND - Exclusion (kinda of)
NOT - Subtract (kind of)

Wanna play with vector boolean operations? Check out paper.js which has some really helpful primitives for this.

http://paperjs.org/examples/boolean-operations/

You can read the unrolled version of this thread here: https://typefully.com/DanHollick/T3dNRtq

Vector Boolean Operations

Do you just click different blending modes until it sort of looks right?

Well, that probably won't change after you read this but at least you probably won't use Lighten or Darken again.

(you should bookmark this thread and use it as a reference)

Put simply, blending modes are a way of creating a new colour based on two input colours.

Hierarchy matters to the way we work out the new colour so the input colours are split into background and foreground.

Let's start with Darken and Lighten

These are the simplest blending modes and the produce a fairly predictable darkening or lightening.

Darken compares each colour's RGB value and selects the darkest value - creating a new colour.

Lighten does exactly the same thing but it selects the lightest value.

Next up are Multiply and Screen.

Although they produce similar results, you can think of these as more elegant versions of Darken and Lighten that result in a bit more contrast.

When you have Screen and Multiply, there is no good reason to use Lighten or Darken.

Multiply does exactly what you'd think, it simply multiplies each channel together.

Because the values are normalised between 0 and 1 the product is always darker than the original.

Screen is actually similar to Multiply but it inverts both the inputs and the result, which is why the result is always lighter.

Again, we need to normalise the values between 0 and 1 to work this out.

The last pair of darkening/lightening modes are Color Burn and Color Dodge.

They have a pronounced step so they can produce quite harsh results compared to the previous modes.

Color Burn works by inverting the background, dividing it by the foreground and inverting the result.

Color Dodge is similar but we invert the foreground divide the background by it.

Overlay, Soft Light and Hard Light are known as contrast blending modes because they make the lights lighter and the darks darker.

These are the most versatile blending mode, especially for images.

Overlay is interesting.

If the background value is light (> 127.5 ) then it applies Screen at half strength making the foreground lighter.

If the background value is dark it applies Multiply at half strength making the foreground darker.

Hard Light is exactly the same as Overlay except it uses the foreground layer.

If the foreground is > 127.5 then apply Screen at half strength, else apply Multiply at half strength.

Soft Light is the most complicated of the standard blending modes.

It produces similar results to Overlay but is more subtle.

Multiply the inverted foreground by the squared background and add it to two times the product of the foreground and background. Simple.

Difference and Exclusion are known as inversion blending modes as they create a color that's almost the opposite of one of the inputs.

Difference does exactly what it sounds like - finds a color inbetween the input colors.

Subtract the bottom colour from the top colour. Any negative numbers are made positive.

Exclusion produces a similar result but in a much more complicated way.

Subtract double the product of each channel from the sum of each channel.

To do this, we need to normalise the values between 0 and 1.

Hue, Saturation, Color and Luminosity are different from the other modes because they manipulate the Hue, Saturation and Luminosity components rather than the RGB values.

They derive a new color by combining the HSL values of the two inputs.

Hue takes the Hue value from the foreground and the Saturation and Luminosity values from the background.

Saturation takes the Saturation value from the foreground and the Hue and Luminosity values from the background.

Color takes the Saturation and Hue values from the foreground and the Luminosity values from the background.

Lastly, Luminosity takes the Luminosity value from the foreground and the Hue and Saturation values from the background.


You can read the unrolled version of this thread here: https://typefully.com/DanHollick/KrBa0JP

Blending Modes

I've always wanted to know why some colours "vibrate" next to each other.

I did some digging and it's actually the same reason good camera lenses are so expensive. 👇

There are two related visual effects happening here and they both have insane names:

- Axial Chromatic Aberration - causing the vibration
- Chromostereopsis - causing the weird 3D effect

You've probably heard of chromatic aberration before:

Essentially, each wavelength of light (colour) has a slightly different focal length within your eye.

In this case red has a shorter focal length than blue.

This is what causes the "vibration" - your eye struggles to focus on the border between the two because they literally can't both be in focus at the same time.

It also causes that halo effect as your eyes blur the two colours together.

Chromostereopsis is when this focal length difference is opposite in each eye and it creates the impression of depth.

This is why that red donut appears closer to your eye than blue background.

Trippy optical illusions, like this, layer these effects to make it appear like parts of the image are actually moving.

So what does this have to do with camera lenses?

Well, chromatic aberration affects cameras too as light is dispersed through glass in a similar way to the lenses in our eyes.

Lens manufacturers reduce it by adding extra elements and special low dispersion glass to focus the wavelengths at the same point.

This glass is doped with all sorts of chemicals that change the way light refracts through it.

Obviously, this is expensive.

It's very difficult to do this in zoom lenses, where the focal length changes all the time.

That's why they end up looking like this inside and costing $$$$.

These days a lot of chromatic aberration correction happens in software, especially on smart phones.

Just another way in which digital images are made up.

https://twitter.com/DanHollick/status/1491436859238412288?s=20&t=ZJ8hTVxhr5N7mks8N9HrvA

Fun photography fact for anyone who made it down here.

The way we describe aperture (f/2.8) is actually an equation where f equals the focal length of the lens.

Aperture diameter = f(50mm)/2.8 = 17.85mm