Project 2: Make a Picture that Responds to Sound

We're going to learn to add some basic interactivity to our face, first by making it respond to mouse movement, then by tweaking that code so it responds to sounds picked up by the microphone.

This project requires your computer to have a microphone attached. If not, that's okay. You can either do the mouse-movement part of this unit, or move onto the next unit, making a "mischievous" painting program using mouse and keybord input.

TUTORIAL

1. We're going to start with a basic Processing sketch that draws a face. Feel free to copy and paste this code into Processing to follow along, or use your own face project as the basis.

  size(400, 300); //sets the size for the window
  background(0, 200, 100);
  strokeWeight(4);
  fill(255, 0, 100);
  stroke(100, 0, 50);
  rect(100, 250, 200, 100, 35); //shoulders
  ellipse(200, 150, 150, 200); //face
  strokeWeight(1);
  stroke(0,0,0);
  fill(255,255,255);
  ellipse(250, 150, 20, 20); //eye
  ellipse(150, 150, 20, 20); //eye
  fill(255,255,0);
  stroke(150,100,0);
  ellipse(200, 180, 20, 40); //nose
  strokeWeight(2);
  stroke(100, 0, 50);
  line(150, 210, 200, 205);//mouth part 1
  line(200, 205, 250, 210);//mouth part 2

2. To make our project interactive, we need to divide the code into two 'functions.' A function is a named chunk of code that does a specific job. Every interactive Processing project has at least these two functions:

void setup() - Runs once at the beginning. This is where we set the size, and generally "set things up."
void draw() - Runs over and over again, constantly. This is generally where we "draw" shapes and things. Since its constantly drawing over and over, we can change the drawing to make things animate and move!

Look at the lines added to the code below. The only command in our setup() function is the size() command, and we put everything else in the draw() function. Notice how every function name ends with "()" parentheses, and the contents of every function is contained within "{}" curly braces.

void setup(){
  size(400, 300); //sets the size for the window
}

void draw(){
  background(0, 200, 100);
  strokeWeight(4);
  fill(255, 0, 100);
  stroke(100, 0, 50);
  rect(100, 250, 200, 100, 35); //shoulders
  ellipse(200, 150, 150, 200); //face
  strokeWeight(1);
  stroke(0,0,0);
  fill(255,255,255);
  ellipse(250, 150, 20, 20); //eye
  ellipse(150, 150, 20, 20); //eye
  fill(255,255,0);
  stroke(150,100,0);
  ellipse(200, 180, 20, 40); //nose
  strokeWeight(2);
  stroke(100, 0, 50);
  line(150, 210, 200, 205);//mouth part 1
  line(200, 205, 250, 210);//mouth part 2
}

3. Now we're going to create our first variable in Processing. A variable is a value that can change, which we give a name to. The name can be as simple as 'x' or as complex as 'iceCreamSandwichesEatenOnWednesday' though we try to keep them short.
Our variable will be called 'eyemove,' since we're going to store a number in it that will change, and will cause our face's eyes to change. The code "float eyemove;" tells Processing to make a new variable, the type of which is 'float' (which just means a number that can have decimals).

We'll add the code at the very beginning of our program, outside of any function, so both functions can use it.

Inside the draw() function, we'll add a line of code to give eyemove a value:

eyemove = mouseX;

This code sets eyemove equal to the horizontal position ('x') of the mouse in the screen. Lower down, we'll change the code where we drew the eyes, and replace the '20' we put for width and height with 'eyemove.' Now the eye size will change based on mouse position!
Try adding the following code to your sketch and trying it out:

float eyemove;

void setup(){
  size(400, 300); //sets the size for the window
  }

void draw(){
  eyemove = mouseX;
  background(0, 200, 100);
  strokeWeight(4);
  fill(255, 0, 100);
  stroke(100, 0, 50);
  rect(100, 250, 200, 100, 35); //shoulders
  ellipse(200, 150, 150, 200); //face
  strokeWeight(1);
  stroke(0,0,0);
  fill(255,255,255);
  ellipse(250, 150, eyemove, eyemove); //eye
  ellipse(150, 150, eyemove, eyemove); //eye
  fill(255,255,0);
  stroke(150,100,0);
  ellipse(200, 180, 20, 40); //nose
  strokeWeight(2);
  stroke(100, 0, 50);
  line(150, 210, 200, 205);//mouth part 1
  line(200, 205, 250, 210);//mouth part 2
}

4. It moves! But maybe a bit too much. Since our window is 400 pixels wide, mouseX ranges from 0 to 400. So when we wiggle our mouse, the eyes go from 0 to 400 pixels wide. We're going to use a command called 'map()' to adjust the effect the mouse has on the eyes. 'map()' takes a value between two numbers, and 'maps' it onto a value between two different numbers. For example, this sketch maps the mouse's X position between 0 and 500 onto a new number, named 'redBarX,' between 200 and 300. You can see the effect - when you move the mouse from 0 to 500, the red bar only moves from 200 to 300:

Here is the actual map() code we're going to add to our program. eyemove = map(mouseX, 0, width, 20, 50);

It maps 'mouseX' between 0 and the width of our screen, onto 'eyemove' between 20 and 50. So when we move our mouse across the screen, the eyes will just grow from 20 pixels wide to 50.
Try adding the following code to your sketch and trying it out:

float eyemove;

void setup(){
  size(400, 300); //sets the size for the window
  }

void draw(){
  eyemove = map(mouseX, 0, width, 20, 50);
  background(0, 200, 100);
  strokeWeight(4);
  fill(255, 0, 100);
  stroke(100, 0, 50);
  rect(100, 250, 200, 100, 35); //shoulders
  ellipse(200, 150, 150, 200); //face
  strokeWeight(1);
  stroke(0,0,0);
  fill(255,255,255);
  ellipse(250, 150, eyemove, eyemove); //eye
  ellipse(150, 150, eyemove, eyemove); //eye
  fill(255,255,0);
  stroke(150,100,0);
  ellipse(200, 180, 20, 40); //nose
  strokeWeight(2);
  stroke(100, 0, 50);
  line(150, 210, 200, 205);//mouth part 1
  line(200, 205, 250, 210);//mouth part 2
}

5. Much better! Now, we can use these same techniques to add more variation to our face's movement, all tied to our MouseX value.
Below, I've added a few more 'float' variables, and 'mapped' the mouse movement onto them, then put into different lines of code in my drawing. You not only can change the size of shapes, but their positions - and by adding variables to the fill() commands, you can also make colors change, too!
Try adding the following code to your sketch and trying it out, or play around with making other changes:

float eyemove;
float headcolor;
float headsize;
float mouthmove;

void setup(){
  size(400, 300); //sets the size for the window
}

void draw(){
  eyemove = map(mouseX, 0, width, 20, 50);
  headcolor = map(mouseX, 0, width, 0, 100);
  headsize = map(mouseX, 0, width, 1, 1.5);
  mouthmove = map(mouseX, 0, width, 0, 50);
  background(0+headcolor, 200+headcolor, 100-headcolor);
  strokeWeight(4);
  fill(255, 0, 100-headcolor);
  stroke(100, 0, 50);
  rect(100, 250, 200, 100, 35); //shoulders
  ellipse(200, 150, 150*headsize, 200*headsize); //face
  strokeWeight(1);
  stroke(0,0,0);
  fill(255,255,255);
  ellipse(250, 150, eyemove, eyemove); //eye
  ellipse(150, 150, eyemove, eyemove); //eye
  fill(255,255,0);
  stroke(150,100,0);
  ellipse(200, 180, 20, 40); //nose
  strokeWeight(2);
  stroke(100, 0, 50);
  line(150, 210, 200, 205+mouthmove);//mouth part 1
  line(200, 205+mouthmove, 250, 210);//mouth part 2
}

6. Now the face reacts to our mouse. But lets add another dimension of interaction. To do this, we'll use some pre-made code you can copy and paste into your program, and then tweak our code a little to work with it.

NOTE: This code works as-is in Processing 2, but to use it in Processing 3, you will need to install an additional 'sound library.' This is very easily done from within Processing 3 - CLICK HERE FOR DIRECTIONS ON INSTALLING THE MINIM SOUND LIBRARY FOR PROCESSING 3.

Adding the 'Minim' sound library to Processing 3:

a.) In the menu, go to "Sketch->Import Library->Add Library"

b.) In the Add Library window that pops up, type 'minim' into the search bar. Select the Minim audio library.

c.)Click the 'install' button, and wait a moment while Processing installs the code library.
Click here to close this and resume the tutorial

To begin, we're going to just copy some pre-made code into our program. This code will make our Processing sketch listen to the microphone, so that we can use its input in our code. Don't worry about how it works, just copy and paste it into the bottom of your program:

float eyemove;
float headcolor;
float headsize;
float mouthmove;

void setup(){
  size(400, 300); //sets the size for the window
}

void draw(){
  eyemove = map(mouseX, 0, width, 20, 50);
  headcolor = map(mouseX, 0, width, 0, 100);
  headsize = map(mouseX, 0, width, 1, 1.5);
  mouthmove = map(mouseX, 0, width, 0, 50);
  background(0+headcolor, 200+headcolor, 100-headcolor);
  strokeWeight(4);
  fill(255, 0, 100-headcolor);
  stroke(100, 0, 50);
  rect(100, 250, 200, 100, 35); //shoulders
  ellipse(200, 150, 150*headsize, 200*headsize); //face
  strokeWeight(1);
  stroke(0,0,0);
  fill(255,255,255);
  ellipse(250, 150, eyemove, eyemove); //eye
  ellipse(150, 150, eyemove, eyemove); //eye
  fill(255,255,0);
  stroke(150,100,0);
  ellipse(200, 180, 20, 40); //nose
  strokeWeight(2);
  stroke(100, 0, 50);
  line(150, 210, 200, 205+mouthmove);//mouth part 1
  line(200, 205+mouthmove, 250, 210);//mouth part 2
}

// Don't worry about understanding the stuff below!
import ddf.minim.*;
  
Minim minim;
AudioInput in;
float volume = 0;
float volumeF = 0;
  
//-------------------------------------
void setupAudio(){
  minim = new Minim(this);
  in = minim.getLineIn(Minim.MONO, 512);
}
  
//-------------------------------------
void getVolume(){
  volumeF = in.right.level()*1000;
  volume = 0.8*volume + 0.2*volumeF;
}
  
//-------------------------------------
void stop(){
  in.close();
  minim.stop();
  super.stop();
}

7. Now that we've pasted in that code, we're going to make a few small tweaks to our code. We'll add "setupAudio();" to our setup() function to start the program listening to the microphone.
We'll add "getVolume();" to the draw() function. This takes the current volume the microphone is receiving, and puts it in a variable called "volume."
Finally, we'll tweak our "map" functions so instead of taking in the mouse's x movement from 0 to the width of the screen, it will instead read the volume between 0 and 255.

...and that's it! When you run your program, the image should respond to the sounds heard by the microphone.

float eyemove;
float headcolor;
float headsize;
float mouthmove;

void setup(){
  size(400, 300); //sets the size for the window
  setupAudio();
}

void draw(){
  getVolume();
  eyemove = map(volume, 0, 255, 20, 50);
  headcolor = map(volume, 0, 255, 0, 100);
  headsize = map(volume, 0, 255, 1, 1.5);
  mouthmove = map(volume, 0, 255, 0, 50);
  background(0+headcolor, 200+headcolor, 100-headcolor);
  strokeWeight(4);
  fill(255, 0, 100-headcolor);
  stroke(100, 0, 50);
  rect(100, 250, 200, 100, 35); //shoulders
  ellipse(200, 150, 150*headsize, 200*headsize); //face
  strokeWeight(1);
  stroke(0,0,0);
  fill(255,255,255);
  ellipse(250, 150, eyemove, eyemove); //eye
  ellipse(150, 150, eyemove, eyemove); //eye
  fill(255,255,0);
  stroke(150,100,0);
  ellipse(200, 180, 20, 40); //nose
  strokeWeight(2);
  stroke(100, 0, 50);
  line(150, 210, 200, 205+mouthmove);//mouth part 1
  line(200, 205+mouthmove, 250, 210);//mouth part 2
}

// Don't worry about understanding the stuff below!
import ddf.minim.*;
  
Minim minim;
AudioInput in;
float volume = 0;
float volumeF = 0;
  
//-------------------------------------
void setupAudio(){
  minim = new Minim(this);
  in = minim.getLineIn(Minim.MONO, 512);
}
  
//-------------------------------------
void getVolume(){
  volumeF = in.right.level()*1000;
  volume = 0.8*volume + 0.2*volumeF;
}
  
//-------------------------------------
void stop(){
  in.close();
  minim.stop();
  super.stop();
}

THAT'S IT! You now know all the code you need to do this assignment, and make your Processing face sound-responsive. Look at the examples below, then look at the "Assignment" link for the resources and requirements.

EXAMPLES

Because of browser security reasons, the microphone-responsive Processing examples can't be posted in a website. So the examples included here respond instead to mouse movement.

Click to view code

ASSIGNMENT

Resources:

Reference "Cheat Sheet" for drawing different shapes in Processing
Processing Language Reference for more complex shapes, check "arc" "vertex" and "curves"
Rubric - Expectations and scoring for this assignment

Code template (Non-moving face. You may use this as a basis, or come up with your own totally original design):

void setup() {
  size(500, 400); //sets the size for the window
  background(0, 200, 100);
}

void draw()
{
  background(0, 200, 100);
  rect(150, 300, 200, 100); //shoulders
  ellipse(250, 200, 150, 200); //face
  ellipse(300, 200, 20, 20); //eye
  ellipse(200, 200, 20, 20); //eye
  ellipse(250, 230, 20, 40); //nose
  line(200, 260, 250, 255);//mouth part 1
  line(250, 255, 300, 260);//mouth part 2
}

'Helper code' for listening to the microphone input (copy and paste at the end of your file):

// Don't worry about understanding the stuff below!
import ddf.minim.*;
  
Minim minim;
AudioInput in;
float volume = 0;
float volumeF = 0;
  
//-------------------------------------
void setupAudio(){
  minim = new Minim(this);
  in = minim.getLineIn(Minim.MONO, 512);
}
  
//-------------------------------------
void getVolume(){
  volumeF = in.right.level()*1000;
  volume = 0.8*volume + 0.2*volumeF;
}
  
//-------------------------------------
void stop(){
  in.close();
  minim.stop();
  super.stop();
}