Dither

I keep coming back to John Provencher's work. He takes a photo and strips out almost all of the detail, and you can still tell what it is from a few hundred marks.

I wanted to understand how that works well enough to control it myself, so I basically put everything on a slider. There are five ways of breaking a picture down, and they all run through one pipeline in a single file with no dependencies.

How it works

Each tool cuts the image into a grid and averages every cell down to one colour and one brightness, and then draws a mark in that cell from those two values. The character map draws a letter, the mark grid draws a shape, halftone draws a dot of a certain size and ordered dither picks a colour from the palette. That last step is the only part that changes, so they all sit behind one function, and switching between them feels a bit like picking up a different tool.

Drawing the marks

My first go at the mosaic drew everything with fillText. I thought a set of solid and hollow square characters would cover most of what I needed, but it didn't. A letter sits wherever the font's metrics put it in the cell, so filled squares never quite meet up. Outlines come with whatever stroke weight is built into the typeface and dots are always the same size, and all of that is set by the text renderer, so you can't really change it.

So I switched to drawing the marks as vectors, which gave me all of that control back. I kept the characters as names for the shapes, and that also meant I could add triangles and diamonds without worrying about whether a font has them.

// Each character is a token that picks a mark. The glyph only has to be
// typeable and suggestive of what it draws.
const MARKS = {
  '■': { kind: 'fill', size: 1 },
  '□': { kind: 'ring', size: 1 },
  '●': { kind: 'dot',  size: 0.86 },
  '▲': { kind: 'tri',  size: 1, up: true },
}

// A filled mark tiles the whole cell, an outline holds a real stroke
// weight, and a dot's radius is free to track tone. Drawing text can't
// do those, so each mark is drawn as a shape.
const inner = cell - gutter * 2
drawMark(ctx, MARKS[token], mx, my, inner, lineWidth, colour)

Halftone

My first version sized each halftone dot so its radius matched how dark the cell was, which seems like the obvious way to do it, but the midtones came out too heavy and the whole image went dark. Ink coverage is about area, and area goes up with the square of the radius, so a cell that's half dark needs a radius of the square root of a half. Adding the square root fixed it.

const coverage = 1 - luminance / 255

// Area proportional to coverage, so the tonal ramp stays linear.
// Using coverage directly as the radius makes every midtone read
// a stop too dark.
const r = cell * 0.5 * Math.sqrt(coverage) * dotScale

Blur

The halftone dots merge into blobs through a blur and then a threshold. The blur softens the dots so the ones next to each other join up, and the threshold turns them back into hard edges. My first version set the blur radius in pixels, which was fine until I made the screen coarser. With big spacing, a fixed radius starts averaging across whole cells when it should only be merging neighbouring dots, so the midtones lose their detail and the image ends up as a couple of big shapes.

So I set it as a fraction of the screen spacing instead, and now the control means the same thing at every dot size. Since then I've started checking anything I measure in pixels at both ends of its slider.

Video

Error diffusion passes each pixel's rounding error on to the pixels next to it, so every pixel depends on all the ones processed before it, and if the input changes by even one value, everything after that point comes out differently. You'd never notice on a still image. On video though, a flat wall flickers with noise, because two frames that look identical to me are slightly different in their actual pixel values.

Ordered dither and halftone don't have that problem, because a pixel's threshold comes from its own position. The same input in the same place gives the same result every frame, so flat areas stay completely still. That's mostly why I use those two for motion work, and why there's a warning inside the error diffusion panel, where you'd actually run into it. I'd only use Floyd-Steinberg on video if I wanted that texture on purpose.

Exporting

Recording the canvas live is one click, but it drops frames as soon as the filter can't keep up. Stepping through the clip frame by frame and saving a numbered sequence is exact but slow. I needed both at different times, so I built both, and the sequence goes straight into ffmpeg and comes out as ProRes.

Drawing tens of thousands of letters straight onto the on-screen canvas took fifteen seconds, and doing the same thing on a separate canvas and copying the finished frame across once took a second and a half, with exactly the same algorithm.