Week 3: Empathy Machine vol. 1
Hatching
So, to start this part, I wanted to clarify a few things. First of all, creating projects in Arduino environment is hard. And it is not hard just because of the syntax and what you could do to program in this environment but just that you don't always get what you want and what you usually end up with is something that just kinda works or doesn't work and you have no idea why. Life in programming, in short. Then, think about adding animals into the mix, how do you really design for them? How do you design to imitate them? Now, the first part is tricky enough but designing to empathise with them is some form of an abstraction, that I love to think about but actually executing a meaningful project? Hell, that is not easy at all. And to throw on top of it all, my sparkfun kit just does not cut it. The 5$ ultrasonic sensor, is bad. Many other modules I have? Are bad too. Harbouring the kit across the world, I figured would be a good idea but then it turns out it all is just simply not enough. Also, I am still doing orientations for the fabrication labs (at that time) So come to think of it all, I had a mission impossible, especially that I read 'no presentations in powerpoint/keynote' as a challenge (and I loveeee challenges!), so I decided - to do a presentation but in front-end with a js library. Which was fun but rrrrreally time consuming and naturally it deterred me from doing what needed to be done. And what needed to be done? Well. The rest!
So there were two approaches I have undertaken. First was to address the assignment due to the constraints I was bound by, primarily: equipment and surprisingly at that point also imagination. Therefore I quickly thought of some phenomena that captured my interest in the past. In this case, a moose.
Moose
The thing is that some differences between us and animals have its basis also in how our metabolical systems work. For instance: humans can get intoxicated in a variety of ways, one is alcohol, casualy drank primarily in the form of ethanol as a result of other metabolical processes undertaken by yeast, which by eating sugar, processes it and releases among some chemical compounds - alcohol. The very same thing does not quite happen for humans, though, as carbohydrates are used as primary fuel for, well, life. And so eating an apple, which is primarily carbohydrates in the forms of sugars, does not quite generate much of a response, except for the joy of how (often) crunchy and sweet it is. That doesn't quite go for some species like Moose, though, which get intoxicated by eating apples. After digging into the subject, I discovered this was not quite exactly the case but still I was looking forward to exploring why that was happening and I figured ever since I have an alcohol sensor, perhaps I could make use of it.
Moose also have a different body temperature, many animal species vary in these terms from humans who's normal body temperature is set at 36.6 degrees Celsjus. I had a temperature sensor so perhaps I could emulate some responses made in case of: what if the air temperature plummets, how does this affect the animal in question?
Issues
Then after a while I discovered some interesting things about my Arduino setup. Not only did I not have enough modules but some of them seemed to not work quite properly, like the temperature sensor. Tight deadlines never go in pair with problems, especially those of programming nature that you'd just keep bouncing back and forth for hours but seeing how I could not get the project through, I just decided to drop this idea. One that kind of went beyond just any specific animal. I just decided to target the idea from the perspective of communication, something essential to us all (and which I often have problems with but that's a different subject matter).
And so I started by showcasing some ways that humans communicated back in the days before internet was here, by smoke signals. I believe Indians used it and it was good enough. These kinds of forms, more in an auditory manner were utilized in Game of Thrones, a popular HBO TV Show based on a series of books, in which horns blown in a specific manner would communicate a particular event taking place. And then I figured that, hm, I've also seen Meerkats in Planet Earth and it was explained that they have very elaborate ways and means of communicating, where particular nuances in how they speak matter. Also they are very social species. I'm not sure why I wanted to mention polar bears, though. Now that I think about it, me not remembering why I threw it in bothers me. Perhaps I wanted the audience to bear with me? Or just showcase the great rift between how animals communicate and how humans developed a multitude of languages that go into a variety forms of complexity. For instance a morse code, which in some way I was looking forward to emulating.
Obvious Approach
So I decided to take an obvious approach, by incorporating a set of code that I worked for, I think it wouldn't be an overstatement if I said over hundred(s) of hours, at the very least trying to get a specific form of communication working - Bluetooth. So I restudied the code and extracted parts specific to what I was trying to achieve and made a very basic setup in which accordingly to the horizontal touch position, I would generate specific piezo buzzer sound frequencies. This way, I wanted to emulate communication. Cause you could do something with it, send morse codes, elicit specific forms of emotions. Yeah, I know it was a bit far-fetched but potentially it was doable and there was an idea hatched in my head that took reality within an extremely pressing amount of time.
And so I improvised...
Reflection
The prototype was not successful. It was praised for the value of portraying communication but many among the audience just did not simply understand what I was trying to achieve. The use of Bluetooth communication was valued here as well. But ever since the prototype was bare basic, all I had was the idea that I did my best to present but there are limits to how much you can depict with a minimal presentation slidedeck and a prototype which on its own just simply does not convey what it is.
Nevertheless, I felt that I should show off some skills and figuring out the bluetooth communication was not so simple as it may seem. Especially that I essentially hacked a bluetooth module dedicated for mBot robots, this stuff takes some time to figure out (although I managed 2 semesters ago). I reference a method that I researched also these 2 semesters ago, created by Prof. Takaya and I think this is pretty much it from the referenced code. Oh, of course there is the Leaf project source code that I keep on Github based on which I have got this setup up to full speed.
Arduino Code
const int buzzer = 8;
byte incoming;
void setup()
{
pinMode(buzzer, OUTPUT);
Serial.begin(115200);
}
void loop()
{
incoming = Serial.read();
if (incoming==255){
noTone(buzzer);
}
else{
Serial.println(incoming, DEC);
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
tone(buzzer,map(incoming, 0, 127, 100, 900));
}
}
Processing Code
import android.content.Intent;
import android.os.Bundle;
import ketai.net.bluetooth.*;
import ketai.ui.*;
import ketai.net.*;
import android.os.Environment;
import java.util.*;
byte checkmate;
float lastX;
void setup() {
background(0);
frameRate(30);
fullScreen(P2D);
strokeWeight(4);
orientation(PORTRAIT);
//start listening for BT connections
bt.start();
//at app start select device…
isConfiguring = true;
//font size
fontMy = createFont("SansSerif", 40);
textFont(fontMy);
//selectionUpdate(1);
}
void draw() {
if (isConfiguring)
{
ArrayList names;
background(78, 93, 75);
klist = new KetaiList(this, bt.getPairedDeviceNames());
isConfiguring = false;
} else
{
lastX = mouseX;
float na = map(constrain(map(lastX, width/12, width*11/12, 0, 100), 0, 100), 0, 100, 0, 127);
byte naa = (byte)na;
if (checkmate != naa) {
checkmate=naa;
bReleased = true;
byte[] data = {naa};
bt.broadcast(data);
//first tap off to send next message
bReleased = false;
println(naa);
delay(50);
}
background(50);
fill(#7ED321);
textAlign(CENTER);
textSize(width/20);
text("Empathy Machine", width/2, height/16);
}
}
PFont fontMy;
boolean bReleased = true; //no permament sending when finger is tapped
KetaiBluetooth bt;
boolean isConfiguring = true;
String info = "";
KetaiList klist;
ArrayList devicesDiscovered = new ArrayList();
//Following void appropriated from Prof. Takaya's Blog: http://prof-takaya.blogspot.com.au/2013/02/connecting-android-and-arduino-by.html
void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
bt = new KetaiBluetooth(this);
}
//Following void appropriated from Prof. Takaya's Blog: http://prof-takaya.blogspot.com.au/2013/02/connecting-android-and-arduino-by.html
void onActivityResult(int requestCode, int resultCode, Intent data) {
bt.onActivityResult(requestCode, resultCode, data);
}
//Following void appropriated from Prof. Takaya's Blog: http://prof-takaya.blogspot.com.au/2013/02/connecting-android-and-arduino-by.html
void onKetaiListSelection(KetaiList klist) {
String selection = klist.getSelection();
bt.connectToDeviceByName(selection);
//dispose of list for now
klist = null;
}
//Following void appropriated from Prof. Takaya's Blog: http://prof-takaya.blogspot.com.au/2013/02/connecting-android-and-arduino-by.html
//Call back method to manage data received
void onBluetoothDataEvent(String who, byte[] data) {
if (isConfiguring)
return;
//received
info += new String(data);
//clean if string to long
if (info.length() > 16)
info = "";
}
Week 3: Empathy Machine vol. 2
Processing Code (Reworked)
import ketai.camera.*;
import ketai.cv.facedetector.*;
import ketai.ui.*;
//import blobDetection.*;
KetaiCamera cam;
KetaiVibrate vib;
//BlobDetection theBlobDetection;
boolean newFrame= false;
String[] img = {"roo1.png", "emu1.png", "quokka1.png"};
PImage[] images = new PImage[img.length];
PImage myImage;
int MAX_FACES = 1;
KetaiSimpleFace[] faces = new KetaiSimpleFace[MAX_FACES];
int handle;
int k = -1;
long[] pat = {5, 125};
void setup() {
for (int i=0; i< img.length; i++) {
String imgs = img[i];
images[i] = loadImage(imgs);
}
orientation(PORTRAIT);
//cam.setCameraID(0);
//destimg = createImage(640, 320, RGB);
vib = new KetaiVibrate(this);
cam = new KetaiCamera(this, 640, 320, 30);
println(cam.list());
cam.setCameraID(1);
// img =new PImage(80,60);
// theBlobDetection = new BlobDetection(img.width, img.height);
// theBlobDetection.setPosDiscrimination(true);
// theBlobDetection.setThreshold(0.2f); // will detect bright areas whose luminosity > 0.2f;
//theBlobDetection.setBlobMaxNumber(1);//Max blob detected = 1;
cam.start();
}
void draw() {
imageMode(CENTER);
background(0);
pushMatrix();
translate(0,640);
rotate(-PI/2);
image(cam, 0, width/2, 640,480);
faces = KetaiFaceDetector.findFaces(cam, MAX_FACES);
popMatrix();
current();
FaceDetection();
imageMode(CORNER);
//cam.loadPixels();
roo();
emu();
quok();
if (k==-1){
textAlign(CENTER);
textSize(width/20);
text("How will an animal react to you?", width/2, height/16);
}
}
//void onPause()
//{
// super.onPause();
// //Make sure to releae the camera when we go
// // to sleep otherwise it stays locked
// if (cam != null && cam.isStarted())
// cam.stop();
//}
void onCameraPreviewEvent()
{
cam.read();
// cam.manualSettings();
newFrame = true;
}
int vale;
int vale1;
int vale2;
boolean valeCounter = true;
boolean valeCounter1 = true;
boolean valeCounter2 = true;
void roo() {
if (valeCounter==true){
vale++;
vale++;
vale++;
vale++;
vale++;
if (k==0){
tint(255,0,0);
}
image(images[0], width/7,height*6/8,100+vale,100+vale);
noTint();
if (vale>150){
valeCounter=false;
}
}
else if (valeCounter==false){
vale--;
vale--;
vale--;
vale--;
vale--;
if (k==0){
tint(255,0,0);
}
image(images[0], width/7,height*6/8,100+vale,100+vale);
noTint();
if (vale<150){
valeCounter=true;
}
}
//fill(#7ED321);
//rect(width/7,height*6/8,width/7,height/8);
//textAlign(CENTER);
//textSize(width/20);
//text("Kangaroo", width/7, height/8);
}
void emu() {
if (valeCounter1==true){
vale1++;
vale1++;
vale1++;
vale1++;
vale1++;
if (k==1){
tint(255,255,0);
}
image(images[1], width*3/7,height*6/8, 100+vale1, 100+vale1);
noTint();
if (vale1>150){
valeCounter1=false;
}
}
else if (valeCounter1==false){
vale1--;
vale1--;
vale1--;
vale1--;
vale1--;
if (k==1){
tint(255,255,0);
}
image(images[1], width*3/7,height*6/8, 100+vale1, 100+vale1);
noTint();
if (vale1<150){
valeCounter1=true;
}
}
//fill(#7ED321);
// rect(width*3/7,height*6/8,width/7,height/8);
//textAlign(CENTER);
//textSize(width/20);
//text("Emu", width*3/7, height/8);
}
void quok() {
if (valeCounter2==true){
vale2++;
vale2++;
vale2++;
vale2++;
vale2++;
if (k==2){
tint(255,0,255);
}
image(images[2], width*5/7,height*6/8, 100+vale2, 100+vale2);
noTint();
if (vale2>150){
valeCounter2=false;
}
}
else if (valeCounter2==false){
vale2--;
vale2--;
vale2--;
vale2--;
vale2--;
if (k==2){
tint(255,0,255);
}
image(images[2], width*5/7,height*6/8, 100+vale2, 100+vale2);
noTint();
if (vale2<150){
valeCounter2=true;
}
}
//fill(#7ED321);
// rect(width*5/7,height*6/8,width/7,height/8);
//textAlign(CENTER);
//textSize(width/20);
//text("Quokka", width*5/7, height/8);
}
void current(){
if (pmouseX > width/7 && pmouseX < width/7+200 && pmouseY > height*6/8 && pmouseY < height*6/8+200){
k = 0;
}
else if(pmouseX > width*3/7 && pmouseX < width*3/7+200 && pmouseY > height*6/8 && pmouseY < height*6/8+200){
k = 1;
}
else if (pmouseX > width*5/7 && pmouseX < width*5/7+200 && pmouseY > height*6/8 && pmouseY < height*6/8+200){
k = 2;
}
else if (pmouseX > width/2-320 && pmouseX < width/2+320 && pmouseY > 240 && pmouseY < 720){
k = -1;
fill(255,255,255);
text("Nah, that's you, click an animal", width/2, height*8/16);
}
}
void FaceDetection(){
if (k >= 0 && k<= 2){
for (int i=0; i < faces.length; i++)
{
imageMode(CORNER);
//We only get the distance between the eyes so we base our bounding box off of that
//rect(faces[i].location.x, faces[i].location.y, 2.5*faces[i].distance, 3*faces[i].distance);
image(images[k], faces[i].location.x+320, faces[i].location.y+240, 2.5*faces[i].distance, 3*faces[i].distance);
println("Face Detected!");
if(k==0){
vib.stop();
vib.vibrate(1);
textAlign(CENTER);
textSize(width/20);
fill(255,0,0);
text("Kangaroo?", width/2, height/16);
text("IT WILL FIGHT YOU", width/2, height*8/16);
}
if(k==1){
vib.stop();
vib.vibrate(100);
textAlign(CENTER);
textSize(width/20);
fill(255,255,255);
text("Emu?", width/2, height/16);
text("IT WILL RUN LIKE HELL", width/2, height*8/16);
}
else if(k==2){
vib.stop();
vib.vibrate(10);
textAlign(CENTER);
textSize(width/20);
fill(255,0,255);
text("Quokka?", width/2, height/16);
text("It will wanna HUG you!", width/2, height*8/16);
}
}
}
}
Resources:
https://idangero.us/swiper/api/
Ketai Detector: https://forum.processing.org/one/topic/
ketaifacedetector-in-portrait-mode-on-tablet.html
Rotating Image: https://forum.processing.org/two/discussion/16641/how-to-only-rotate-an-image-without-changing-the-background
PImage Resize: https://forum.processing.org/one/topic/problem-with-pimage-resize.html
Eye of the Tiger Piezo: http://forum.arduino.cc/index.php?topic=215687.0
Face Detect with Front Camera: https://github.com/ketai/ketai/issues/62
Video Recording with Ketai: https://forum.processing.org/two/discussion/3643/is-it-possible-to-do-a-video-recording-using-ketai-lib
Ketai syntax for Face Detection: http://ketai.org/examples/facedetector
OpenCV and Android: https://forum.processing.org/one/topic/computer-vision-libraries-and-android.html