Project 4: Digital Visioning Webcam Art

In this project, we're going to make Processing "see" the world through a webcam and alter the resulting image in real time.

Artists Adrien M & Claire B use this technique in their "XYZT" installation to create a surreal funhouse mirror.

Camille Utterback uses code to make live video interactive in "Text Rain".

On the web, Olivia Jack's Selfie Apocalypse plays a wild variety of interactive tricks with your webcam.

TUTORIAL

1. Before we load in a camera image, we're going to learn about loops in code. Loops let you repeat a command multiple times, while only typing it once, and are thus really useful for long, repetitive tasks like, say, going through every pixel of an image and changing its color.

We're going to start by using a 'for loop' to draw a line of white circles on the screen. For loops are stuctured like this:

for (
A starting point, usually a counter variable at 0.
,
A condition. The code will keep looping while this is true. Usually this sets a max the counter will reach before it stops.
,
An instruction to do each loop, usually just adding one to the counter.
)

{
Code to repeat each loop
}

For example, the code below sets a counter variable (named 'counter') to '0', then repeats, adding 1 to the counter each time ('counter++' is shorthand for 'add 1 to counter'), as long as the counter is smaller than the screen width. Each repetition, it draws an ellipse on screen, at the x position of the 'counter' variable. This will end up drawing ellipses across the screen, from x=0 to x=500.

function setup(){

createCanvas(400, 400);

var counter;

//draws ellipses across screen
for (counter=0; counter <=width; counter++){
fill(255, 100);
noStroke();

//draw each ellipse, using 'counter' for the x position
ellipse(counter, 50, 20, 20);
}

}

2. Very basic, right? We just drew a fat line. Let's space out those ellipses into a chain of polka-dots. We're going to tweak the code to only draw a dot every twenty pixels. (We'll also add random colors for pizazz!)

To space out our dots, we're going to use the modulus operator (%). The modulus (%) operator gives us the remainder when dividing two numbers, e.g.:

5 % 2 = 1
◉◉|◉◉|◉

6 % 4 = 2
◉◉◉◉|◉◉

10 % 5 = 0
◉◉◉◉◉|◉◉◉◉◉

This means we can look for a modulus (%) of zero to check if a number is divisible by another.

Below, we've added an 'if' statement that uses the modulus operator (%) to only draw ellipses at x positions that are divisible by 20 (e.g. 20, 40, 60...).

function setup(){

createCanvas(400, 400);

var counter;

//draws ellipses across screen
for (counter=0; counter <=width; counter++){
	fill(random(255), random(255), random(255));
	noStroke();

	//draw an ellipse if 'counter' is divisible by 20
	if(counter%20 == 0) ellipse(counter, 50, 20, 20);
}

}

3. Now, by 'nesting' a for loop inside of another, we can traverse the screen horizontally AND vertically, filling the whole screen with ellipses.

To do this, we'll also create two counter variables, 'counterx' and 'countery', that each loop will use. The outer loop will use 'countery' to move vertically down the screen. And each time it loops, it will run the second loop inside of it, which will use counterx to draw ellipses across the screen. Together, it will move down and across, filling the screen wth dots.

P5 has a system for trying to catch poorly-coded loops that go on forever and slow down computers. However, at the moment the system is over-sensitive and will cause an error with any nested loop. By adding //noprotect at the top of my program, I can turn that system off.

//noprotect

function setup(){

createCanvas(400, 400);

background(0);

var counterx;
var countery;

//loop that moves down screen vertically
//to draw each line of dots

for (countery=0; countery <=height; countery++){
	
	//loop that moves across each line, drawing dots
	for (counterx=0; counterx<=width; counterx++){
		if(counterx%20 == 0 && countery%20==0){
			fill(random(255), random(255), random(255));
			noStroke();
			ellipse(counterx, countery, 20, 20);
		}
	}
}
}

4. We can use mouseX and mouseY here, like we would any other code, to affect things like shape size, position, and color. Here, we’ll use a function called dist() to measure the distance between the mouse and each ellipse, and change the ellipse size based on mouse closeness.

Note that, now that things will be moving and interacting, we've had to organize the code into setup() and draw() functions.

(We also added counterx and countery to the fill color values to make things more visually interesting! Now the red and green values of each ellipse are determined by its horizontal and vertical placement.)

//noprotect

function setup(){
createCanvas(400, 400);
}

function draw(){
background(0);

var counterx;
var countery;

for (countery=0; countery<=height; countery++){
	for (counterx=0; counterx<=width; counterx++){
	
		if (counterx%20 == 0 && countery%20==0){
		
		fill(counterx, countery, 0, 100);
		noStroke();
		
		//create a variable called 'distance' that
		//contains the distance between the mouse
		//and the current ellipse being drawn...
		
		var distance = dist (counterx, countery, mouseX, mouseY);
				
		//...and use that value to change the value of
		//a variable called 'size' which we'll use as
		//the width and height of the ellipse
				
		var size = map(distance, 0, 500, 50, 5);
		ellipse(counterx, countery, size, size);
		}
	}
}
}

5. Now we're going to set that code aside, and introduce the code for loading images from a webcam, and displaying them on the screen. From this point forward, the images on the right will just be still images, not functional programs, since we can't have multiple P5 windows all pulling from the webcam on the same website.

The below program simply grabs frames from the webcam and displays them onscreen. Read the comments (after the "//" lines ) to see what each line of code does.

//create a new variable named 'capture'
//which will store a video-capturing 'object'
//that grabs frames from the camera

var capture;

function setup() {
  createCanvas(640, 480);
  //create a video-capturing 'object'
    //stored in our 'capture' variable
  capture = createCapture(VIDEO);
  //set 'capture' to capture a 640x480 frame
  capture.size(640, 480);
  //If we don't "hide" the capture, P5 will just
    //display it onscreen wherever it wants.
	  //We'll 'hide' it so we can control where/how it's displayed
  capture.hide();
}

function draw() {

//draw frame to screen, aligned to top-left (0,0)
//and as wide and tall as our canvas (640, 480)
  image(capture, 0, 0, 640, 480); 
}

6. Next, we'll add a single line of code, a filter() command, to add a simple visual effect to our video. We've made a super-basic Instagram filter!

var capture;

function setup() {
  createCanvas(640, 480);
  capture = createCapture(VIDEO);
  capture.size(640, 480);
  capture.hide();
}

function draw() {
  image(capture, 0, 0, 640, 480);
  filter(POSTERIZE,3); // add a visual effect
}

7. Now we're going to get a little more technical. We're going to use a for loop to travel through every pixel of the video, and replace any dark pixels with neon green, to create a sort of solarized pop-art effect.

This is a pretty basic pixel manipulation, but once you've done it you can go further, altering the colors in different ways.

To do this, we're going to use a function called createGraphics(), which lets us create an image to store the video frames in for tweaking. We'll also use a function called loadPixels(), which pulls the pixels out of an image and puts them into an array. An array is basically a list in computer code. Each item in the array has a number, called an index, we can use to find it. Indices start at 0 and count up. For example, the array that P5 uses to hold the color information for an image is like this:

index	0	1	2	3	4	5	6	7	8	9	10	11
contents	pixel #0's red value	pixel #0's green value	pixel #0's blue value	pixel #0's alpha value	pixel #1's red value	pixel #1's green value	pixel #1's blue value	pixel #1's alpha value	pixel #2's red value	pixel #2's green value	pixel #2's blue value	pixel #2's alpha value

("alpha" means transparency)

P5 (unlike Processing), makes it a little tricky to find the pixel you want to change, since each pixel's color information is spread across four indices. Which means pixel #2's color information isn't in index [2], it's in index [8], [9], [10], and [11]!

index	0	1	2	3	4	5	6	7	8	9	10	11
contents	pixel #0's red value	pixel #0's green value	pixel #0's blue value	pixel #0's alpha value	pixel #1's red value	pixel #1's green value	pixel #1's blue value	pixel #1's alpha value	pixel #2's red value	pixel #2's green value	pixel #2's blue value	pixel #2's alpha value
	Pixel #0				Pixel #1				Pixel #2

Luckily, you might see a pattern there we can use to find the index for the color info we want to tweak:
pixel's red value = (pixel #) * 4
pixel's green value = ((pixel #) * 4) + 1
pixel's blue value = ((pixel #) * 4) + 2

You can access any item in that array using square brackets [] like in the example below:

//I have an array named 'image.pixels,' which contains the color data of an image named 'image'
image.pixels[0] = 255; //set the red value for the first pixel, pixel #0, to 255
image.pixels[1] = 0; //set the green value for the first pixel, pixel #0, to 0
image.pixels[2] = 100; //set the blue value for the first pixel, pixel #0, to 100

int counter=0; // we'll use this variable in a loop below to move through each pixel
for (counter = 0; counter < image.width*image.height; counter+=4){ //a for loop that moves through every pixel in 'image'
	image.pixels[counter*4] = random(255); //Set the current pixel's red value to a random number
	image.pixels[counter*4+1] = random(255); //Set the current pixel's green value to a random number
	image.pixels[counter*4+2] = random(255); //Set the current pixel's blue value to a random number
}

Look at the code below, and read the comments for each line as you add them to your code sketch. We're going to step through each pixel in the image and change any dark pixels to neon green.


//noprotect

var capture;

//creates a variable to store the current video frame
var vidframe;

function setup() {
  createCanvas(640, 480);
  capture = createCapture(VIDEO);
  capture.size(640, 480); 
  capture.hide();
  
  //set the size of the 'vidframe' to store frames captured by 'capture'
  vidframe = createGraphics(640, 480); 
}

function draw() {
  
  //draws the captured frame into our 'vidframe' image
  vidframe.image(capture, 0, 0, 640, 480);

  //creates an array of pixel color information that we can tweak and change
  vidframe.loadPixels();

  var counter;// this variable will take us through the loop to edit each pixel
  
  //use a for loop to travel through the array of pixel data and make changes
  for (counter = 0; counter < vidframe.width*vidframe.height; counter++){
    //grab the colors for the current pixel
    var r = vidframe.pixels[counter*4]; //store the red value of this pixel
    var g = vidframe.pixels[counter*4+1]; //store the green value of this pixel
    var b = vidframe.pixels[counter*4+2]; //store the blue value of this pixel

    //check the total color values of the current pixel
    //If it's dark, replace the pixel with neon green!
    if (r+g+b < 150){
       vidframe.pixels[counter*4] = 0; //the red value
       vidframe.pixels[counter*4+1] = 255; //the green value
       vidframe.pixels[counter*4+2] = 0; //the blue value
    }	
 }

  //updatePixels() locks in any changes we made to the pixels
  vidframe.updatePixels();

  //draw our "vidframe" image onto our canvas
  image (vidframe, 0, 0, 640, 480); 
}

8. The below example combines everything we've done with loops and video.

It extracts the color values of the pixels, like above, but uses them to color shapes, which are drawn using a loop and modulus, like the earlier loop examples.

One new thing the below code needs to do is use the vidframe.pixels array, which is basically just a long list of colors in a single line, to color and place shapes on a 2D screen where the pixels are arranged in several rows. We can use division (/) to find our y value, and modulus (%) to find our x. Here's a detailed description of how that works, but if it loses you, feel free to just scroll down to the code and test it out.

Imagine our screen was just 6 pixels wide and 5 tall. The pixels[] array would look like 30 pixels in a line:

0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29

These correspond to 5 rows, each containing 6 pixels:

0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29

Which make this "screen" of pixels:

	0	1	2	3	4	5
0
1
2
3
4

Imagine we were using a loop to traverse this array. We get to pixel 8, and find that it is orange. But we need to figure out where on the screen, in x and y, to draw the orange pixel (or ellipse, in the below program).

0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29

If we divide 8 (the current pixel) by 6 (the width of the screen), and ignore any remainder, we get the y value (or row) of the pixel:
8 / 6 = 1 ~~r 2~~

If we use the modulus, we get just the remainder, which is the x value (or column) of the pixel:
8 % 6 = 2

Which would place the orange pixel here on our little screen, at (2, 1):

	0	1	2	3	4	5
0
1
2
3
4

Okay, all pixel talk aside, let's look at the example. Read the comments to see what each line of code exactly does to create the effect on the right.

//noprotect

var capture;
var vidframe;

function setup(){
  createCanvas(640, 480);
  capture = createCapture(VIDEO);
  capture.size(640, 480);
  capture.hide();
  vidframe = createGraphics(640, 480); 
}

function draw(){

  background(0);
  //draws the captured frame into our 'vidframe' image
  vidframe.image(capture,0,0,640,480);
  
  //creates an array of pixel color information that we can tweak and change
  vidframe.loadPixels();
  
  var counter;

  //Use a loop to travel through every pixel of the video image.
  //Notice we're counting up 10 at a time in these loops
  //That's because we don't need to look at every pixel,
  //just every 10th pixel, so we can draw a circle.
  for (counter = 0; counter < width*height; counter +=10){
    
      //Use the modulus in an 'if' statement to make sure
      //we draw ellipses only every 10 rows
      if (int(counter/width)%10==0){ //IF we're on row 0, 10, 20, 30, etc...
         var r = vidframe.pixels[counter*4]; //store red value of this pixel
         var g = vidframe.pixels[counter*4+1];//store green value of this pixel
         var b = vidframe.pixels[counter*4+2];//store blue value of this pixel
  
         // use that color to draw next shape
         fill(color(r,g,b));
        
         // calculate what ‘column’ we’re on
         // by taking the modulus of our current
         // position in the pixel array over
         // the width of the screen
         var column = int(counter % width); 

         // calculate what ‘row’ we’re on
         // by dividing our current
         // position in the pixel array by
         // the width of the screen
         var row = int(counter / width); 
        
         //draw the 10x10 circle at the pixel position
         ellipse(column, row, 10, 10);
     }
  }
}

9. By adding just a few tweaks to how we draw the ellipses, we can make a more stylized image. Here we'll randomize the size of the ellipses every frame, and make them semitransparent, so they'll overlap and jitter to create a kind of animated painting effect.

//noprotect

var capture;
var vidframe;

function setup(){
  createCanvas(640, 480);
  capture = createCapture(VIDEO);
  capture.size(640, 480);
  capture.hide();
  vidframe = createGraphics(640, 480); 
}

function draw(){

  background(0);
  //draws the captured frame into our 'vidframe' image
  vidframe.image(capture,0,0,640,480);
  
  //creates an array of pixel color information that we can tweak and change
  vidframe.loadPixels();
  
  var counter;

  //Use a loop to travel through every pixel of the video image.
  //Notice we're counting up 10 at a time in these loops
  //That's because we don't need to look at every pixel,
  //just every 10th pixel, so we can draw a circle.
  for (counter = 0; counter < width*height; counter +=10){
    
      //Use the modulus in an 'if' statement to make sure
      //we draw ellipses only every 10 rows
      if (int(counter/width)%10==0){ //IF we're on row 0, 10, 20, 30, etc...
         var r = vidframe.pixels[counter*4]; //store red value of this pixel
         var g = vidframe.pixels[counter*4+1];//store green value of this pixel
         var b = vidframe.pixels[counter*4+2];//store blue value of this pixel
  
         // use that color to draw next shape, first adding an alpha value
         fill(color(r,g,b, 100));
		 
		 noStroke();
        
         // calculate what ‘column’ we’re on
         // by taking the modulus of our current
         // position in the pixel array over
         // the width of the screen
         var column = int(counter % width);

         // calculate what ‘row’ we’re on
         // by dividing our current
         // position in the pixel array by
         // the width of the screen
         var row = int(counter / width);
        
         //draw the circle at the pixel position, with randomized size
         ellipse(column, row, random(20,50), random(20,50));
     }
  }
}

Now that we've played with several different ways of manipulating the image from a webcam, it's time for you to come up with your own webcam artwork! Look below for some examples, and a specific assignment/prompt.

EXAMPLES

ASSIGNMENT

Project 4: Digital Visioning Webcam Art

Resources: