In 2006, Katherine Isbister and Kia Höök developed the Sensual Evaluation Instrument (SEI), a tool that evaluates affect and aids in nonverbal communication.  Created with artist Rainey Straus while the professors of human-machine interaction were working on a European Union project on emotion and technology, the ‘blobjects’ were tested with a goal of enabling conversations with designers and computer scientists about emotion.  

The instruments provoked an unexpected outcome of universality during testing: subjects kindredly selected the same objects to express similar emotions (e.g., a spiky object to express anger or fear). We spoke with Katherine and Rainey over Skype about how these objects came into being, their emotional resonance, and how they’ve developed a life of their own over the years.

Katherine Isbister: What happened for us was Kia and I, we got interested in doing this nonverbal reporting of emotion; like empowering people to convey how they felt without having to use words. This is when I was working with her at her lab in Sweden; we first did some primitive experiments with things like color.  So anyway, it didn’t work at all because people had really culturally specific associations with color and a lot of times  they would come up with narratives about things that were more real object based—there’s a lot of sort of socio-cultural overlay people do onto objects all the time. That was slightly discouraging. But at the time, I knew Rainey had done her MFA project on these incredible I don’t know what to call them, Rainey, nubbins?

Rainey Straus: Blobjects. And actually there was a whole show [on them] down in San Jose. I think the show was called Blobjects which happened. A lot of it focused on product design but it’s definitely a kind of aesthetic that’s in the visual vocabulary currently.

Early Sensual Evaluation objects

KI: Yeah, so we basically said, look, we’re in over our aesthetic depth here and we know you have the skill set. Here’s what we’re trying to do and here’s what kind of didn’t work and here’s our instinct about what maybe could work and she really generated these objects out of that brief essentially. The results that you saw in the paper, to some extent, came as a surprise to us. I mean, we were just trying to enable discussion with designers about emotion. We weren’t even necessarily aiming for universality but what ended up happening was this pretty consistent mapping of aspects of the shapes themselves to emotion which was really fascinating. So I don’t know, Rainey, if we even had that conversation about how conscious that was on your part?

RS: Well, I mean I think I’d step back and say that one of the things I’m really interested in in terms of form making is the idea of synesthesia. So, a lot of times that’s more oral: hearing and seeing color.But, I think there’s also a way in which for me intuitively, emotions also have a certain sense of form. And I think that’s really what I was kind of drawing from as we were crafting these objects and it’s already something that I had brought into my practice maybe in a more organic way without naming it. But this really was sort of a crystallization of ways in which I had already been working, and then bringing it into a more codifable system, or attempting to.

KI: Kia and I were working on this big European Union project on emotion, understanding emotion and how to work with emotion with technology, and we were sort of reacting to this idea of ‘oh, just hook up biosensors to people and you will know all’ as if your heart rate tells everything about how you’re feeling. We wanted to implement an engaging system to communicate about it, but maybe not have to communicate in this very dry ‘circle a number between one and seven’ way. And, we didn’t want to force articulation of things that maybe weren’t even ready to be articulated. But then, I wanted to tether it to the rigor of social science because to some extent computer scientists are reluctant to take up things that seem more ephemeral or more aesthetic. They don’t know how to fit that into their practice. 

And so I was like, I know! We can do this calibration exercise—I knew about this thing called the international affective picture set which these psychologists in Florida have shown these pictures to like thousands of people and gotten their self reported ratings of how the pictures made them feel. So, we used some of the pictures from that to essentially calibrate people’s use of the objects and that’s where we got this interesting result that wow, there’s a lot of consistency in which object a person chooses depending on what’s put in front of them.  

In terms of the actual use of the objects, there was the kind of range we wanted which was some people would gesture wildly with the objects. Some people would stack them. Some people used the whole range of objects in the course of interaction and then other people would only use just a few and kind of, which seemed to kind to reflect our own level of expressivity in interesting ways.  

RS: The original pieces were cast in a really soft vinyl and they were so much about the touching and manipulation of them. So, that’s sort of historically where they came from. They were absolutely meant to be held as part of the experience of interacting with them or viewing them.

KI: Yes, so we got the kind of dialogue we wanted to happen, but then we had this other result of wow! this seems to actually be working across cultures in an interesting way.  That ties into what Rainey was saying about synesthesia and the notion that emotion travels. It makes perfect sense. Like when you look at the objects you’re like, oh yeah, something that’s spiky is kind of scary.

I think it also speaks to a lot philosophical thinking about the wisdom of physicality and where intelligence is really rooted and beings (sort of whole beings, as opposed to the mind body split), and that there is an inherent intelligence to our physical and tactile understanding. I think a really powerful piece of this project is that it articulates and kind of illustrates that in a very concrete way.

What we were interested in was uncovering what’s unfolding for a person as they experience a system. What the designer really wants to know is am I evoking the sorts of things I had hoped to as it unfolds for the person and they need to get in sync with what’s happening for the person in a different way than, say, a psychologist. So, I feel like for us it opened up this realization that we may have to really come up with very different tools if we’re trying to facilitate this kind of communication that can be more immediate and maybe not readily summarizable. And maybe that’s just fine.