Salon with Marco Donnarumma #

First of all, thanks for having me and for the invitation. I’m very happy I can contribute a little to the research that is ongoing with the whole team there at Microsoft Research. And yeah, I’ve been collaborating with Baptiste since a while now. That’s also why he’s one of the few people who can pronounce my surname correctly.

And I will talk about the body a lot and specifically how the body can interface with technology and AI in ways that possibly are not the most conventional ones. And I’ve been doing this for a couple of decades now. And in the last decade, I’ve been working with machine learning, AI, physiological computing, robotics, biomechanics, all different ways that would allow me to explore the body through technology and explore technology through the body. So I will share my slide. I’m sorry, I didn’t share the audio. Let me do that again. So you should be seeing my slide. Please let me know if it’s not the case.

Yeah, so for example, here in this picture, you can already see a quite illustrative image of my work. And I think what this image also does is to suggest sort of radical imaginations of what bodies can be through technology, but not only thanks to that, also thanks to different ways of imagining embodiment, often starting from what is considered normal, no label, and so on.

So just quickly about my background. So I essentially grew up in media arts, working with sound and video. In the beginning, the work became much more interactive. And until the point I started working exclusively on the ideas of liveness in technological mediated performance and also installation. This then was combined with long term passion that I have for performance art, in particular feminist performance art of the second wave, the 70s. And through the years, this combination of technology, interactivity and performance brought me to start working also in theatre and dance. So what would be most contemporarily called contemporary performance.

So you will see this, what I do is a hybrid of performative practices, technological creation and theoretical reflection. So some key points in my practice also as a theorist are transdisciplinarity, DIY culture, I’m a Linux advocate since a couple of decades. Then the combination of art science and technology, all wrapped in feminist theories of the body, critical disability studies. And then I try to inform the theory through practical experiments that then become artworks and these take place across different media, including robotics, AI, prosthetics, biosensor interactive music.

I’m going to start with what I will not talk about in this talk, which is I will not touch much upon generative AI, large language models. I do not use them in my artistic practice as a choice, as a principle. I do have work with them in order to write about them because I do find them quite interesting, those and all the culture that is around it, especially when seen in the longer tradition of machine learning, autonomous computation.

So I do not use them in my work because I think it is this particular ways of thinking about machinic creativity as being able to reconstitute human creative ideas through digital data is essentially flawed. So I think a good way of understanding how a generative image model work is really seeing them as technologies that are void of any intention. They do have signs of that intention, which are to be found in the data points, in the multi-dimensional data sets that they inhabit. But to my view, at the end of the day, they are completely detached and detached in a kind of eerie way from the actual experience of creation.

So yes, they do create something, obviously, and also technically they are amazing breakthrough. But the most important question for me is like, what do we do with what they create? What do we think about what we create, what they create? So it’s interesting to see the generative models in this way, as I said, like eerily detached from actual experience of creation. And then it kind of makes sense when you look at the outputs of these machines, which in very little time became immediately recognizable with this kind of gradient aesthetics, very rich and posh colors, amazing compositions, which somehow all ends up looking like a generative AI image.

So this made me think of it as a sort of creating indifference through repetition, which is obviously a paraphrasing of Derrida. But yeah, I find it interesting how this incredible super complex mechanism of computing data and computing artworks, then at the end of the day, what we have is this stuff that all looks more or less the same. We even have a word now for it, which is AI slop.

So on one hand, we have great PR marketing campaign telling us that these machines now have the same creativity or humor. On the other, we all see that that’s pretty much not the case. But that’s why the whole business is quite eerie in a way. We know that something it’s not like it’s being sold, but we are forced to think so. So then my question is, OK, if this is the technology that we have at hand, and this is the kind of cultural impact they’re having, basically trying to redefine creativity.

And what do we do with the relations that are at the core of creativity, relation between people, relation between histories, relation between artistic tradition? Where is all of that in a generative, creative model? How do we get back to it? Not by just thrashing technology, thrashing the whole AI thing, but by working with it, working through it in a critical way.

So of course, there are many ways. I’m not here to give you the answer, but I’ll just show you some of the methods that I found useful to explore this kind of question. And so there is a point that became about 10 years ago very important for me. How do we think of how machine affect the body? And once we think and reflect about that, how does that question change the way we think about other people affecting our bodies and our bodies affecting the bodies of other people?

So all of this is very nicely wrapped in this quote by Vinciane Desprez. It’s all about learning to be affected. Affected here refers to the study of affect and in cultural studies. For those who don’t know, affect is a sort of immaterial but very visceral and contagious psychosomatic way of defining relations between humans, between machines. It’s a real reflection about how we influence one another, but not only psychologically, also at the level of the flesh, in the level of the guts. And now that changes our experience.

So the first time I started thinking systematically about this was with this piece entitled Corpus Nil, this accidentally is also the first piece where I collaborated with Baptiste. And here I was coming from several years of working in the field of new interfaces for musical expression, where a lot of the work done there is about control, control of the interface, control of the music. And a lot of that control happens through hands, obviously, like most of musical instruments, even traditional ones. So there was this very peculiar idea to me, like what happens if I try to play an instrument without hands? Of course it didn’t work.

So I had to define an instrument that did not need to be played, but would play along with me without using scores, without using patterns or anything like that, but establishing a relationship between my body directly and the machine learning software, the music software, which was the instrument.

And so what happens here is I have two physiological sensors on my arms. You can see these black bands on my upper arms. And they pick up from my body the sounds of muscle contraction, which you can think of, for example, as the sound of the heart beating, as the muscle of the heart that produces the sound. So you have that sound in every part of your body. You can amplify it with an instrument that I created. It’s called X-Sense. And the other sensor instead captures electronic voltages, micro voltages from skin surface. And these data are related, obviously, to muscle activity.

And therefore, one can infer from them also information about movement, about movement as it happens inside the body rather than outside in space. And so I performed this choreography where I deconstruct my own body, that’s me in the picture, in a way that it still looks human but doesn’t look human enough. So the whole choreography happens with my hands completely erased, my head completely erased and the rest of the body too. So you have this optical illusion of only seeing this lump of flexion, say, which evolves through different stages. And as this happens, the machine learning system captures this data from these muscle sensors and tries to imitate the sound of my muscle by using a bank of digital synthesizers.

And so the result is essentially sort of drone like very rich in overtones and bars of noises, a sort of metaphorical sound representation of the muscle activity inside the body. Then what’s interesting is that I said in the beginning, I cannot really play this instrument. The instrument just plays along with me and it’s a duet, but it’s not very much a synchronized duet. There is no direct mapping between my gestures and the sound, but it all happens in this kind of shifting fields of activity influencing one another, affecting one another. I will show you a little trailer.

So making this piece and also working with Baptiste was quite illuminating for me. You know, I was still, it was 10 years ago, still fairly young. And this was the first time that also that I tried to use my own body in a meaningful way. Meaning, I, that was the time where I was starting familiarize with feminist body theory and disability studies and I started asking questions to myself about what kind of body am I staging in my biggest pieces? How do I make that body say something about itself? Because otherwise the most performances even today, especially without the technology, the body is always like a given.

Oh yeah, there is a body performing there, but very rarely that body is questioned or questions around that body are asked. The few cases are those that are explicitly dealing with topics of identity or disability. But everything else, just take it as a given. So it was an interesting process of getting through the technical methods to reach this kind of interaction stage. And having that interaction saying something about the understanding of the body as well. It was quite a spontaneous research and work. I was doing my PhD at the time. But then it was quite illuminating for all my future work.

And in particular, it was the strategy of appropriating hardware and software in order to unsettle the body, like literally. So physically in one sense, using this sensor to conduct the interactive work with the machine I cannot control. And therefore I always have to be on the edge, I always have to be alert. So it’s not making my life easier. And appropriating, why? Because the sensor that I’ve been using, especially my own Xthsense, it’s also a technology that comes from the medical field. And it is used also in prosthetic control, for example, to detect muscle movement of amputated limbs. So you see, there is a whole sort of very large circle coming, spiraling onto itself between the body technology and so on.

Now this kind of work also led me somewhere else, a pretty related field, that of objection. So there is something about objection that I find very interesting, specifically in connection with technology. Because most of the time we are solving the idea of technology as being the cleanest and the shiniest thing that we can ever possess. We call server farm clouds for a reason. Which I find is always quite uncanny, but interesting at the same time.

So starting from this quote by Julia Kristeva, I then began developing a series of robots that have much to do with objection through movement and through neural networks. So I created this machine, which is called Amigdala, which is essentially a robotic limb, again, kind of human looking, with a steel knife. And it cuts its own skin. Having to learn how to cut it day by day, because the skin itself becomes harder and harder throughout the exhibition. So the neural network has to keep updating itself, because otherwise it wouldn’t be able to cut the skin. Which also means that the neural network never really achieves a point of resolution. It’s always left there hanging, trying to learn what it needs to, but never arriving at a confluence, at the final point of reaching it and finishing its task.

I’ll show you a video while I speak over it. So this was an interesting excursus that then again brought me somewhere else. So this whole idea of objection connected with technology makes even more sense these days. We would never imagine a technology to be so abject as a body can be. And that’s interesting.

We do have humanoid robots, sort of. They kind of work, sometimes. But they’re also very shiny things. The body is there, but it’s not there, really. It’s just a very idealized version of what the body is. But all the wetness of the body, all the different forms that the body can have, or all the different parts of the body that the body can miss. There is not that possibility in the kind of technology that are developed in that sense.

And there is a reason, of course, because abjection is scary. It can also be fascinating. But it points at something that nobody really wants to see. Which is, for example, the reason why always disabled bodies are so hidden or completely idealized. And I say this as a disabled person myself. I have hearing aid because I’m almost profoundly deaf, but I have a little bit of hearing. I can read lips and amplify what I have with my hearing aid. That also informs my work and informs everything that I’m saying now.

But again, I don’t think one needs to be disabled to understand what I’m talking about. And yeah, so what I liked about working with Amigdala was that I could create a system that based on neural networks, especially neural tractors, that could be also interacted with without having to make any changes to the software. So what I programmed for the machine is the way the machine “perceives” its own body and the skin that it’s cutting.

So it uses servo motors that sends data back to the computer board that is running the machine. And then this data are fed to various algorithms, including some reinforcement learning, in order to allow the machine to sense the different consistency of the skin and also understand whether it’s stuck somewhere, why it got stuck and try to avoid it the next time, for example. It was very interesting to collaborate with a laboratory for this project. It’s called the Neurorobotic Research Laboratory. It was interesting because I could get a crash course in neurorobotics in a couple of years that allowed me to do this project and the other that I’m going to show, but also to see the clash of cultures between the neuroroboticists in the lab and the way I was trying to work with the robots, which nevertheless was resolved very gracefully because the scientists were so open to allow me to do something like that with their tools. And their tools looked like that.

So for the nerds in the audience, this is how I had to program this machine onto an SD card over blocks. Not a lot of fun, but still. So this then brought me to the second strategy, and I will kind of speed up now because I see I’m running out of time, which is to invent prosthetics to train into different somatic modes. So the somatic is the experience of the body in relation to movement, in relation to others. And so starting from amygdala, then I created another set of prostheses that more or less work in the same way, but they are completely different in the way they are engineered because they’re made for the stage. So here you see a piece that’s called Langeweide that I co-performed and co-directed with Margherita Pevere, who is a bio artist and a performer as well.

And so here I have this partial prosthesis, which is an arm jutting out of my body, of my face, and then Margherita is in the other prosthesis, which is a mask made of biofibre biocellulose. So here, one of the points of the performance was how to co-exist on stage with this organic material mask in the face of Margherita and with this machinic autonomous robots on my face and what kind of choreography, what kind of improvisation could come about. I will show you quickly the excerpt as I talk about it.

So again, the whole performance is not really about AI per se, but it’s about how that kind of technology influences the way we relate to one another and also to other non-human entities. Of course, as you’re seeing the image, the piece is not data-scholic about this, or my work tends to be evocative of something and then hopefully the political message or meaning that I try to convey is well hidden, but still accessible to everyone.

That’s the kind of work that I do. There is a reason to that also because I do believe in the power of sensation. I think it’s a lot stronger to have an experience that leaves something, a trace in your body, like physical sensation that stays with you after a show or after seeing an artwork rather than reading 10 pages about it and trying to understand the concept.

That’s of course my personal view, but that’s what I try to do with my work. I do feel that this is also a fascinating approach when working with technology because not having to explain about offering an experience that is self-explanatory about a piece of technology I think can be quite powerful. So from this work, then I went on and did other performances, this is Alia Tsutai with a Shanghai choreographer, Nunu Kong, and the actress Lingling Chen, and kept exploring this way of working with robotics and prosthesis.

But then also started thinking about something else. How Margaret Shildrake puts it in this quote is quite apt. So prosthesis exceeds their instrumentality and constructs the very thing that they prefer to announce, which is the body. So this question, how prosthesis, including algorithms, including AI, that power prosthesis, how do they construct the body as a concept, as an understanding? This is also a question that I feel is very much overlooked these days, but it’s unsurprising given the trajectory of completing research in AI these days. I mean, most of it, of course, there are other branches that study that.

But this led me then to the more recent work that I’m doing, and this is the last thing I’m showing. Sorry about this, that would be a couple of minutes late. I hope it’s fine. So I started thinking, okay, I’m working with prosthesis. Somehow I was still playing within this safe field of putting something on the body and it’s a sort of limbs that is doing something and it’s autonomous and everybody’s seeing it moving. And while I’m still very fond of that work, I realize now that it would unfortunately lend itself to also different kind of understanding.

For example, a transhumanist understanding of man evolution through technology, which is the exact opposite that I’m trying to say in the work. So it’s not that my work then became perceived in that way, but there was something uneasy with me or maybe something I wanted to explore further. And so that was a good moment for me to start working on deafness.

I have this condition since more than 10 years now. That’s the time that it took me to decide to work with it. So I started thinking how to work with the hearing prosthesis and also with the cultural concept that are around it and with the idea of deafness as a mode of listening. Because it’s not that deaf people don’t listen. It’s just that they listen through the skin, through the bodies rather than through the cochlea. So this led me also to another strategy that I want to share with you today, which is repurposing normative designs to explore what they discard in the first place.

And so for this work, I made an open call for deaf collaborators, where deaf means a range of different listening capabilities. One can be profoundly deaf, one can be native deaf, one can become deaf later in life, one can be hard of hearing and everything in between.

So I created a group of five deaf collaborators, six plus me, and then we worked together for five months exploring how deaf embodiments allow us for listening and also how technology, in particular AI, because hearing aid, cochlear implants, they are mostly running on AI-powered systems. How this technology also influences the way we think about sound as deaf people, but also how hearing people think about deaf people.

Because as you remember, a prosthesis constructs a body. So eventually after this five months together, regular session, phenomenological analysis, sound diaries, brainstorming together, mind mapping, and then eventually I came up with this idea of turning the notion of prosthetics completely upside down. So normally a prosthesis is for an individual, it is for fixing something that that individual is lacking. And I thought, okay, so let’s start from the basic. Let’s make a prosthesis that is for a collective, for a collectivity, that is to be shared. And let’s make a prosthesis that does not augment anything, but rather take deaf listening, so bodily and visceral listening, and bring it to everyone in an audience. Could be other deaf people, it could be hearing people.

So you can see this prosthesis here. There are actually this thing on my torso and belly. And they are, the mechanical part is very simple. It’s based on the biomechanics of bone conduction and how it makes sense. It can be attached to the body, but also detached from it. And then there is an engine, an audio engine, that takes either sounds from my body with the sensor instrument I mentioned earlier, and transmits it to the body of other people when I place the prosthesis on them during the performance. Or it uses a system based on timber decomposition, using an algorithm that is called non-negative matrix factorization. What basically it is, audio systems take data sets of field recordings and sound from the body, pre-recorded, and then restructure the sound in a way that are much clearer in a bodily form of listening. So through bone conduction, resonances of the bodily cavities, rather than through the cochlea.

So what I’m doing here, I’m subverting, or at least trying to subvert the algorithms that are used in the normative prosthesis, and use them instead to celebrate and make available for everyone the way how a deaf body listens and experiences sound. So I like to think of the performance as a sort of attunement through communal sensory reconfiguration.

This also allows me to think of the stage in a different way. For example, as you can see here, I’m there like sitting with people, and we are producing sounds together by holding hands and making resistance and force with one another. And I can get in very intimate relations with people because I need to put the prosthesis on them, or I need to put the sensors that I use to capture sound from my body on their body. And I found this quite fascinating. People let you do it as long as you win their trust, which is also part of the performance that I like very much.

But also it’s an interesting way of getting people to experience the implication of certain technology through their bodies directly. And even if they don’t understand what algorithms are running what, that’s not really the point. It’s about what the experience of the mediation of that technology gives to them. And then it’s up to them to interpret that. Surely once you feel sound, not anymore through your ear, but inside your body, when you feel my heartbeat inside your body pulsating, then that gives you something. And then again, it’s subjective how one reacts to it, but I like this integration of experience, technology, algorithms, and bodies all in one place and all in one experience. All right, I’m finished here. Sorry, I’m a little long here, but you got my links here and thanks everybody for listening.

Thank you very much, Marco. That was great. Thanks a lot. And yeah, we have a few minutes for questions and there are a lot of questions actually on the document. And so we’ll try to pick a few of them and let’s see where we can get at.

I think there are a lot of questions about agency and prosthesis. So I will start by a question, then Jenny, Gonzalo, feel free to jump in if you have any specific questions to ask. I think I will start with this first question about or I’ll try to put several questions together. So there is one question: are technologies truly devoid of intention? And how we could discuss this point of intention related to agency that actually has a strong, I mean, it seems to be very important in your work because you perform with the technology and the way the technology reacts live and the way you interact with it actually change the path of the journey of the performance sometimes.

You talk about configuration. So maybe it’s a very confusing question, but there are this concept about intention, agency or reconfiguration that I think are going in the same direction.

Well, I, yeah, okay, that’s an interesting question. Let me see how we can keep it short. Intention, in my view, and this is already super arguable, but in my view is about a willingness. So it’s about the intent. I don’t think we can talk of AI and machine learning as being willing of anything.

They do have agency, though, a lot. And the agency is also reflected and becomes even more powerful when it is reflected across the different layers of culture. So for sure, there is machinic agency. I’m not even questioning that. That’s actually one of the things I try to play with, as you already noted. I do think it’s interesting then to undermine human intention through machinic agency and vice versa, undermine machinic agency through human intention. And that’s perhaps something that is common through my research in all these years, I guess.

Thanks. And yeah, actually, related to that, you talk about duets with technology in Corpus Nil, but again, I guess it comes back after. I think it’s always like this thing of performing live with technology that changed the journeys of the performance. And I know that in the field, I mean, we are a few researchers in human computer interaction here and in this field of research, there is often this word collaboration, collaboration with the machine, you know, human-AI collaboration or human-AI co-creation. And so do you do you see any form of human-AI co-creation or collaboration in your work or is that something else that’s happening?

Okay, so maybe all the people will hate me now, but no, I don’t see it. Sorry. Again, I think this comes back to the question of intention. So I can collaborate, I can co-create with anything that has a willing in, can produce a willing action. So anything that has a will. So my dog has a will, an AI system does not. So I can collaborate more with my dog rather than with AI. And I know this is very arguable, I’m opening a can of worms here. But yeah, that’s the way I see it. Also I think, you know, a lot of this terminology, and I’m not talking about research and scientists here, but in the market, a lot of this terminology is over abused in order to sell their products. And I wish there was a much more open conversation where researchers and scientists could actually explain what they mean by that, rather than leading this marketing campaign for trading something in a way that it’s not. Of course, we can work with these tools and these work have their own agencies, but there are more subtle way of talking about it.

Yeah, I found fascinating the wording, I mean, in your work, but also in the work of scientists, researchers, also in the work of communication team, I think it’s where lies a lot of representation and projection of cultural images. And it’s something that we worked on actually with Jenny, Gonzalo, Vera and Kate. And I think there’s here something very, very relevant, especially related to AI.

Maybe Gonzalo or Jenny, if you want to jump in, because we don’t have to have a chance to ask a question to Marco. Jenny, you asked a question at the end about the decision, why Marco decided to not use gen AI, we can take that one.

Yeah, I would actually love to ask that Marco. You know, one of the things that we’re exploring actually right now is this question of artist resistance to generative AI or artist refusal. And you mentioned early on that you have chosen not to use generative AI in your practice on principle. And I’m curious if you can speak a little bit more about why you’ve made that choice.

Yeah, thank you for that question. It’s pretty essentially a question of resistance. And now a general audience wouldn’t even understand the difference whether I’m using it or I’m not using it. But other artists, especially young artists, would see whether I’m using it or not, would see whether other artists are using it or not. And curators, other professionals in the field would also see the same. So again, it might be a little thing, but it does have an influence or a tiny bit of influence in the way that the professional network around me see and understands the cultural climate around this technology.

And I think it’s good, though, that I also write about this technology so that it’s not just a plain refusal. But then if somebody asks me the same question that you did, then I can also point them to something that I wrote about where there is an elaborate answer to that question. So I think it’s good to operate on different fields.

And then why am I resisting it? Well, because it’s going all the wrong ways in which it can. And apart from the questions of stealing artworks from other people and stealing styles, there is also a major question of not only about biases and social justice, but in general, for me, it’s quite impacting what this technology is doing to culture and the idea of creativity per se, the idea of imagination. This is very impacting, in my opinion, and it is going to have consequences over the next 10, 20 years unless something changes. And that scares me, because that means that as things stand, the significance, the worth of culture as it’s made by people, it’s decreasing step by step. The more these technologies become disseminated. So yeah, I think it’s useful to do something against that.

Thank you so much for that answer. OK, yeah, Gonzalo, I don’t know if you wanted to ask one of your questions. Yeah, I guess it would be a conclusion. I really love the artists and the artists that we follow, but a lot of what I see of your art, Marco, and your work is this push and pull of adoption, rejection, objection, incorporation. And I wonder as you, as a craft person, interact with technology, what have you find, what aspects of it you find that were pushing against you and that actually, that in turn, informed the direction in which you took your work?

You mean when the work was pushing against me or the technology?

That’s right. When the material was pushing against you. You know, clay can be resistive of you in some form or steel or metal, but these technologies have their own resistances, I will say. I wonder if you can share with us when you find moments of resistance from the technology to you and that shaped the direction in which you took it.

Yeah, no, totally. I have lots of that moment with the amygdala. Yeah, also because that’s engineering-wise, that was the most sophisticated thing I’ve done. And it was often a pain because it just wouldn’t behave. And then here I’m going against everything I said in the past hour. Oh, technology should behave. No, that’s not the point. But yeah, I think it’s interesting to work with robotics and machine learning or AI because you get into a lot of those pushes against you.

That’s also what I like about engineering my own machines, that you try to have them doing something, but they just won’t at some point. So for example, I mentioned, to make a concrete example, I mentioned that amygdala uses also reinforcement learning on top of a very basic neural network. Well, I didn’t want to call the reinforcement learning in force, but I had to because the machine kept getting stuck in the same exact way. And there was no engineering, no mechanical way that I could avoid that. Also because the machine is made to cut something that is hardening over time. That’s not making it easier for it.

But for example, I tried all kinds of mechanical engineering ways to solve that problem and I couldn’t. And then I ended up having to spend months trying to understand how to code a reinforcing learning algorithm in force. Eventually it worked. But as you say, also in performance, when I use these machines in performance, it’s all really an open bet. But that’s what I like. I think the push from the machine is also the interesting part of doing this kind of work. And it’s interesting, in my more recent work with this prosthetic bone on the body, they are actually made on this very simple idea of injecting sound in the body of someone else. But for anyone in a general audience, that’s already pushing over a certain threshold of intimacy, of personality as well. So that’s another interesting occasion for that. Which then again opens up different ways for me to thinking about this performance interaction on stage and so forth.

Yeah, I think here there is also a very interesting difference from when you build algorithms and when you build robotics devices that actuate in the real world that can be harmful, that can harm you. And also that create behaviours that maybe sometimes you are more sensitive to the behaviours of, like to the movements than from the image. You know, it might be more and more selective as an audience. So I guess here also there is this thing of acting in the real world through robotics and through this algorithm also brings, I guess, other dimensions of resistance that you don’t have when you’re just designing an algorithm that creates maybe sound or images.