Image Compositing

Shaders are a quick and effective way to blend, or composite, images together.

Blend Modes

Exercise

Let’s start with an exercise, where you will write a p5.js shader to blend two images together.

You can use the same wolf and moon images I am using for my awesome design, or you can use any other two images you want.

Once you’ve figure it out, try different blend modes, similar to Photoshop blending modes. Can you mimic a multiply, additive, dodge, or burn blending mode?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
/* eslint-disable no-undef, no-unused-vars */

let compShader;
let moonImg;
let wolfImg;

function preload() {
  compShader = loadShader(shaderUrl(sketchFile0), shaderUrl(sketchFile1));

  moonImg = loadImage(sketchFile2);
  wolfImg = loadImage(sketchFile3);
}

function setup() {
  createCanvas(744, 481, WEBGL);
}

function draw() {
  background("#292f33");

  noStroke();

  shader(compShader);

  compShader.setUniform("uResolution", [width, height]);
  compShader.setUniform("uImageMap0", moonImg);
  compShader.setUniform("uImageMap1", wolfImg);

  beginShape(TRIANGLE_STRIP);

  vertex(0, 0, 0, 0, 0);
  vertex(width, 0, 0, 1, 0);
  vertex(0, height, 0, 0, 1);
  vertex(width, height, 0, 1, 1);

  endShape();
}

function windowResized() {
  //resizeCanvas(windowWidth, windowHeight);
}

import "p5";
import sketchFile0 from "./composite.vert";
import sketchFile1 from "./composite.frag";
import sketchFile2 from "./moon.jpg";
import sketchFile3 from "./wolf.jpg";

function shaderUrl(source) {
  return "data:text/plain;charset=utf-8," + encodeURIComponent(source);
}

Object.assign(window, { preload, setup, draw, windowResized });

Drawing Rectangles

When drawing rectangles, we can use the rect() drawing function instead of beginVertex(), vertex(), and endVertex(). The drawn shape will have its positions and texture coordinates automatically set in normalized space [0, 1], so we need to update the vertex pass to reflect this. Note that we don’t need to use the model view projection matrices anymore, we just need to convert the coordinates to the clip space range [-1, 1].

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
// Mesh attributes.
attribute vec3 aPosition;
attribute vec2 aTexCoord;

// Passed attributes.
varying vec2 vTexCoord;

void main()
{
  // Copy the vec3 position into a vec4.
  vec4 position = vec4(aPosition, 1.0);

  // Move the shape for the origin in the center.
  position.xy = position.xy * 2.0 - 1.0;

  // Set the clip space position.
  gl_Position = position;

  // Pass data to the fragment shader.
  vTexCoord = aTexCoord;
}

This is much simpler to write so we can use this for any fullscreen 2D drawing. Notice that the images are coming in flipped vertically, due to a mismatch in the texture coordinates. This can easily be flipped back in either the vertex or fragment shader. We want to invert the range from [0, 1] to [1, 0], and can do this by subtracting the y texture coordinate from 1.0. We can also flip the x texture coordinate to mirror the image horizontally.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
#ifdef GL_ES
precision mediump float;
#endif

// Passed attributes.
varying vec2 vTexCoord;

// Custom uniforms.
uniform sampler2D uCaptureMap;

void main()
{
  vec2 texCoordFlip = vec2(1.0) - vTexCoord;
  vec4 captureCol = texture2D(uCaptureMap, texCoordFlip);

  gl_FragColor = captureCol;
}

Background Subtraction

Shaders can be very useful and fast for computer vision. One very common operation we will encounter is background subtraction.

A simple implementation is to save one frame of video as the background reference, and then calculate the difference between the current video frame and the background frame to get a “foreground” frame.

We will use p5.Image.get() to save the background frame as a p5.Image and pass it into the shader as a second sampler2D uniform variable.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
#ifdef GL_ES
precision mediump float;
#endif

// Passed attributes.
varying vec2 vTexCoord;

// Custom uniforms.
uniform sampler2D uCaptureMap;
uniform sampler2D uBackgroundMap;

void main()
{
  vec2 texCoordFlip = vec2(1.0) - vTexCoord;

  vec4 captureCol = texture2D(uCaptureMap, texCoordFlip);
  vec4 backgroundCol = texture2D(uBackgroundMap, texCoordFlip);

  vec3 foregroundCol = captureCol.rgb - backgroundCol.rgb;
  //foregroundCol = clamp(foregroundCol, 0.0, 1.0);
  gl_FragColor = vec4(foregroundCol, 1.0);
}

This does not look quite right. We are using the result of our pixel subtraction directly, but instead we should use it as a mask for the foreground image.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#ifdef GL_ES
precision mediump float;
#endif

// Passed attributes.
varying vec2 vTexCoord;

// Custom uniforms.
uniform sampler2D uCaptureMap;
uniform sampler2D uBackgroundMap;

void main()
{
  vec2 texCoordFlip = vec2(1.0) - vTexCoord;

  vec4 captureCol = texture2D(uCaptureMap, texCoordFlip);
  vec4 backgroundCol = texture2D(uBackgroundMap, texCoordFlip);

  vec3 deltaCol = abs(captureCol.rgb - backgroundCol.rgb);
  float deltaLum = max(deltaCol.r, max(deltaCol.g, deltaCol.b));
  if (abs(deltaLum) > 0.1)
  {
    gl_FragColor = captureCol;
  }
  else
  {
    gl_FragColor = vec4(0.0, 0.0, 0.0, 1.0);
  }
}

Adaptive Background

You may have noticed in the previous example that as soon as the light changes or the camera moves even slightly, the background frame is no good anymore.

One solution is to manually recapture the background, but this is not ideal (especially for an installation!) because we would need to keep checking if our background is valid or not, and would need to move all foreground elements out of the frame to recapture the background.

Another option is to use a running background: a background that gradually adapts itself over time to the changing conditions. This works by adding a little bit of the current frame to the background on every loop. If the light or background content changes, this will gradually make its way to the background frame. Any foreground element won’t be present in the video long enough to make it into the background frame.

Let’s implement this in two parts. First we will build our running background, and once that is working properly we can use it as input in our background subtraction code.

Offscreen Buffer

We will render this first part to something called an offscreen buffer. This is similar to the canvas we have been using so far, but it doesn’t get drawn to the screen. It just lives in GPU memory and can be used as a sampler for the second part.

  • In p5.js, we can create an offscreen buffer using createGraphics(). This call looks a lot like createCanvas() and will also require WEBGL as the third parameter.
  • A p5.Graphics object can be used like a canvas. We can enable a shader by calling shader() on the graphics object, we can draw a rectangle by calling rect() on the graphics object, etc.
  • Nothing will be draw on screen since we are rendering our rect inside the offscreen buffer. We can treat this buffer as a p5.Image and draw it to the screen using the image() function.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
/* eslint-disable no-undef, no-unused-vars */

let backAddShader;
let webcam;
let backImage;

function preload() {
  backAddShader = loadShader(shaderUrl(sketchFile0), shaderUrl(sketchFile1));
}

function setup() {
  createCanvas(windowWidth, windowHeight, WEBGL);

  webcam = createCapture(VIDEO);
  webcam.size(1280, 720);
  webcam.hide();

  backImage = createGraphics(webcam.width, webcam.height, WEBGL);
}

function draw() {
  // Wait until the webcam is ready.
  if (!webcam.loadedmetadata) return;

  backImage.shader(backAddShader);
  backAddShader.setUniform("uCaptureMap", webcam);
  backImage.rect(0, 0, webcam.width, webcam.height);

  background("#292f33");

  noStroke();
  fill(255);

  image(backImage, width * -0.5, height * -0.5, width, height);
}

function windowResized() {
  resizeCanvas(windowWidth, windowHeight);
}

import "p5";
import sketchFile0 from "./shaders/backAdd.vert";
import sketchFile1 from "./shaders/backAdd.frag";

function shaderUrl(source) {
  return "data:text/plain;charset=utf-8," + encodeURIComponent(source);
}

Object.assign(window, { preload, setup, draw, windowResized });

Mixing Images

Our shader is similar to our exercise where we blended two images together, except we will use a uniform to control the blending weight in the GLSL mix() function. mix() takes 3 parameters, the first two can be numbers or vectors (or colors) and the third is a blending weight, indicating how much of each number or vector to use in the output. The blending weight is a float in the [0.0, 1.0] range.

Whenever we want to reset the background (or the first time the loop runs), we want to copy the current frame as-is into the background. Therefore, the blending weight should be 1.0. The rest of the time, we will want to keep most of the current background and only add a bit of the current frame. For example, a blending weight value of 0.1 will only blend in 10% of the current frame (into 90% of the background frame).

Note that we use backImage both as a uniform texture as input, and as our render buffer as output.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
/* eslint-disable no-undef, no-unused-vars */

let backAddShader;
let webcam;
let backImage;
let resetBackground;
let mixSlider;

function preload() {
  backAddShader = loadShader(shaderUrl(sketchFile0), shaderUrl(sketchFile1));
}

function setup() {
  createCanvas(windowWidth, windowHeight, WEBGL);

  webcam = createCapture(VIDEO);
  webcam.size(1280, 720);
  webcam.hide();

  mixSlider = createSlider(0.0, 1.0, 0.1, 0.001);
  mixSlider.position(10, 10);
  mixSlider.style("width", width - 40 + "px");

  backImage = createGraphics(webcam.width, webcam.height, WEBGL);

  resetBackground = true;
}

function draw() {
  // Wait until the webcam is ready.
  if (!webcam.loadedmetadata) return;

  if (resetBackground) {
    backImage.shader(backAddShader);
    backAddShader.setUniform("uBufferMap", webcam);
    backAddShader.setUniform("uCaptureMap", webcam);
    backAddShader.setUniform("uMixPct", 1.0);
    backImage.rect(0, 0, webcam.width, webcam.height);

    resetBackground = false;
  }

  backImage.shader(backAddShader);
  backAddShader.setUniform("uBufferMap", backImage);
  backAddShader.setUniform("uCaptureMap", webcam);
  backAddShader.setUniform("uMixPct", mixSlider.value());
  backImage.rect(0, 0, webcam.width, webcam.height);

  background("#292f33");

  noStroke();
  fill(255);

  image(backImage, width * -0.5, height * -0.5, width, height);
}

function mousePressed() {
  resetBackground = true;
}

function windowResized() {
  resizeCanvas(windowWidth, windowHeight);
}

import "p5";
import sketchFile0 from "./shaders/backAdd.vert";
import sketchFile1 from "./shaders/backAdd.frag";

function shaderUrl(source) {
  return "data:text/plain;charset=utf-8," + encodeURIComponent(source);
}

Object.assign(window, { preload, setup, draw, mousePressed, windowResized });

Oddly enough, the mirroring does not seem to match up. I’m not exactly sure what’s going on here, but my guess is that custom graphics (our background frame) are coming in flipped, and built-in graphics (our webcam image) are coming in un-flipped. This is likely a bug, but we can easily fix it by flipping the UVs (another name for texture coordinates) for the custom image.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
#ifdef GL_ES
precision mediump float;
#endif

// Passed attributes.
varying vec2 vTexCoord;

// Custom uniforms.
uniform sampler2D uBufferMap;
uniform sampler2D uCaptureMap;
uniform float uMixPct;

void main()
{
  vec2 texCoordFlip = vTexCoord;
  texCoordFlip.y = 1.0 - texCoordFlip.y;
  vec4 bufferCol = texture2D(uBufferMap, texCoordFlip);
  texCoordFlip.x = 1.0 - texCoordFlip.x;
  vec4 captureCol = texture2D(uCaptureMap, texCoordFlip);

  gl_FragColor = mix(bufferCol, captureCol, uMixPct);
}

Better Subtraction

Now that our background image is properly rendered, we can use it in our original background subtraction code. We just need to bring it in as a uniform variable instead of the p5.Image. We are back to drawing into our main canvas, or on-screen buffer, so we can make calls to shader() and rect() directly in the sketch.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
/* eslint-disable no-undef, no-unused-vars */

let backAddShader;
let backSubShader;
let webcam;
let backImage;
let resetBackground;
let mixSlider;

function preload() {
  backAddShader = loadShader(shaderUrl(sketchFile0), shaderUrl(sketchFile1));
  backSubShader = loadShader(shaderUrl(sketchFile0), shaderUrl(sketchFile2));
}

function setup() {
  createCanvas(windowWidth, windowHeight, WEBGL);

  webcam = createCapture(VIDEO);
  webcam.size(1280, 720);
  webcam.hide();

  mixSlider = createSlider(0, 1.0, 0.1, 0.001);
  mixSlider.position(10, 10);
  mixSlider.style("width", width - 40 + "px");

  backImage = createGraphics(webcam.width, webcam.height, WEBGL);

  resetBackground = true;
}

function draw() {
  if (!webcam.loadedmetadata) return;

  if (resetBackground) {
    backImage.shader(backAddShader);
    backAddShader.setUniform("uBufferMap", webcam);
    backAddShader.setUniform("uCaptureMap", webcam);
    backAddShader.setUniform("uMixPct", 1.0);
    backImage.rect(0, 0, webcam.width, webcam.height);

    resetBackground = false;
  }

  backImage.shader(backAddShader);
  backAddShader.setUniform("uBufferMap", backImage);
  backAddShader.setUniform("uCaptureMap", webcam);
  backAddShader.setUniform("uMixPct", mixSlider.value());
  backImage.rect(0, 0, webcam.width, webcam.height);

  background("#292f33");

  noStroke();

  shader(backSubShader);
  backSubShader.setUniform("uCaptureMap", webcam);
  backSubShader.setUniform("uBackgroundMap", backImage);

  rect(0, 0, width, height);
}

function mousePressed() {
  resetBackground = true;
}

function windowResized() {
  resizeCanvas(windowWidth, windowHeight);
}

import "p5";
import sketchFile0 from "./shaders/backPass.vert";
import sketchFile1 from "./shaders/backAdd.frag";
import sketchFile2 from "./shaders/backSub.frag";

function shaderUrl(source) {
  return "data:text/plain;charset=utf-8," + encodeURIComponent(source);
}

Object.assign(window, { preload, setup, draw, mousePressed, windowResized });

Note that we can use the same code for the vertex pass of both shader programs, so both are using the same file for the vertex input.

We still have a flipped image issue here, which we can solve by flipping the UVs on the webcam image again.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#ifdef GL_ES
precision mediump float;
#endif

// Passed attributes.
varying vec2 vTexCoord;

// Custom uniforms.
uniform sampler2D uBackgroundMap;
uniform sampler2D uCaptureMap;

void main()
{
  vec2 texCoordFlip = vTexCoord;
  texCoordFlip.y = 1.0 - texCoordFlip.y;
  vec4 backgroundCol = texture2D(uBackgroundMap, texCoordFlip);
  texCoordFlip.x = 1.0 - texCoordFlip.x;
  vec4 captureCol = texture2D(uCaptureMap, texCoordFlip);

  vec3 deltaCol = abs(captureCol.rgb - backgroundCol.rgb);
  float deltaLum = max(deltaCol.r, max(deltaCol.g, deltaCol.b));
  if (abs(deltaLum) > 0.1)
  {
    gl_FragColor = captureCol;
  }
  else
  {
    gl_FragColor = vec4(0.0, 0.0, 0.0, 1.0);
  }
}