Synthetic Media

I... had a hard time choosing an example of synthetic media because there are the following two works that I really like so I'll try to discuss both of them I guess.

‍

Learning to see by Memo Akten (2018)

This video is one of my favorite artworks and it was one of the reasons I became interested in creative coding and just art & tech practices generally. I saw it for the first time around October 2018, when machine learning algorithms started getting widely used for artistic purposes. It completely blew my mind at the time.

‍

‍

This piece was made by ML models that were trained on several datasets: ocean, fire, clouds, and flower images (probably pulled from Google images or Flickr — sourcing "ethical" datasets wasn't being discussed as much back then). When Akten is feeding it new footage it "sees" it as if it were whatever it had in its current dataset. It can only output what it has been taught — nothing else. Similarly but on a much much higher scale — our brains interpret what our senses perceive by looking at what we already know and fitting it into those patterns. Another example would be fish eating plastic in the ocean — they just don't know what plastic is, and it's similar enough in color, shape and size to things they're used to perceiving as food unfortunately.

‍

Back in 2018 the output footage wasn't as realistic as it could have gotten with the tools we have at our disposal today. But as the technology advances so does the conversation about it changes. The things that would be interesting to create shift and raise new questions. So as for the ethical ramifications — it just wasn't something that was widely talked about and taken into consideration at the time. It was more about just successfully making it work in the first place.

‍

Later on, with NFT's coming along, the conversation shifted to talking about the ecological consequences of minting and mining and all that (of course, some or all of these conversations overlapped). Today, as the ability to produce extremely realistic footage, that is pretty much indistinguishable from authentic one, becomes so easy and accessible to most, that the process in and of itself is understandably raising some ethical concerns.

I don't think there's one right answer to how we should deal with and regulate these technologies, but I do think this is an opportunity to create artworks and projects that give us a framework to discuss it and evolve the way we perceive it as the tools themselves continue to evolve. Once a moral standard will have been established (and that feels pretty close if you ask me), new questions will be made and new projects that cause us to ask these questions, and discuss and frame them, will arise. We ought to keep it interesting.

‍

Lifeshapes/Bryły życiorysów by Norman Leto (2010)

‍

I also chose Norman Leto's Lifeshapes (which is a fragment from the full-feature movie "Sailor") to discuss Synthetic Media. This was made way before image synthesis tools were available to artists (or to anyone for that matter), but computational modeling was (and still is, at least to me) a valid way of synthesizing imagery.

Leto takes answers from 150 very personal questions, ranging on topics from people's professional careers to their sex lives. This data is fed into a homemade algorithm, which shapes the content that, depending on the information itself, can take a few or many hours to render. (Given that some of these people include Google co-founder Larry Page and Michael Jackson, presumably many of these answers were gathered from published interviews.)

‍

‍

The results are explosive, 3-D portraits with a linear progression. Childhoods usually seem to start out as a simple string. Something happens, and the portrait balloons out in an unpredictable, bulbously multifaceted way, displaying remarkably personal information about public figures that we have no hopes to process beyond our own guttural response. (By comparison, a man in a coma since he was 12 has the lifeshape of a perfect sphere with a dangling string.)

‍

‍

Leto himself is introducing each Lifeshape to the viewers and explaining the logic behind the sculptures' structure as well as the process itself, while also talking about getting the data, and about how he could barely render some of these Lifeshapes.

‍

‍

Similarly to Learning to See, the possible ethical ramifications don't really come into play in this piece. While Leto is trying to depict these people's lives in a somewhat mathematical and objective way, almost as if was some form of data-viz project, he's basing his data on public knowledge and he's seemingly aware of the fact that it's not all necessarily true. This movie is almost about him as much as it is about the lifeshapes, and so it becomes about him showing his work more than about the work itself.

He does mention at some point that his great grandfather was very famous, but he cannot say what made him famous as it would hurt the whole film's chances of getting accepted to film festivals (one could speculate it was probably something unethical).

‍

‍

Synthetic Media project

Group project by Matthew Lau & I.

‍

Straight out of class both of us mentioned Jean Baudrillard’s Simulacra and Simulation and realized we wanted to create something that’s about the medium itself, but in a self-aware manner. We thought about these image synthesis tools as these simulacra machines, creating things that resemble what the models were being taught but that are stripped away from the original meaning.

‍

We started coming up with concepts by looking at texts that could be interesting to work with. We’ve looked through A Pattern Language, The Whole Earth Catalog, The Futuristic Manifesto and a few short stories by renowned writers. Eventually we settled on Julio Cortazar’s Instructions on How to Climb a Staircase. The text goes as follows:

‍

“No one will have failed to observe that frequently the floor bends in such a way that one part rises at a right angle to the plane formed by the floor and then the following section arranges itself parallel to the flatness, so as to provide a step to a new perpendicular, a process which is repeated in a spiral or in a broken line to highly variable elevations. Ducking down and placing the left hand on one of the vertical parts and the right hand upon the corresponding horizontal, one is in momentary possession of a step or stair. Each one of these steps, formed as we have seen by two elements, is situated somewhat higher and further than the one prior, a principle which gives the idea of a staircase, while whatever other combination, producing perhaps more beautiful or picturesque shapes, would be incapable of translating one from the ground floor to the first floor.

You tackle a stairway face on, for if you try it backwards or sideways, it ends up being particularly uncomfortable.

The natural stance consists of holding oneself upright, arms hanging easily at the sides, head erect but not so much so that the eyes no longer see the steps immediately above, while one tramps up, breathing lightly and with regularity. To climb a staircase one begins by lifting that part of the body located below and to the right, usually encased in leather or deerskin, and which, with a few exceptions, fits exactly on the stair. Said part set down on the first step (to abbreviate we shall call it "the foot"), one draws up the equivalent part on the left side (also called "foot" but not to be confused with "the foot" cited above), and lifting this other part to the level of "the foot," makes it continue along until it is set in place on the second step, at which point the foot will rest, and "the foot" will rest on the first. (The first steps are always the most difficult, until you acquire the necessary coordination. The coincidence of names between the foot and "the foot" makes the explanation more difficult. Be especially careful not to raise, at the same time, the foot and "the foot.")

Having arrived by this method at the second step, it's easy enough to repeat the movements alternately, until one reaches the top of the staircase. One gets off it easily, with a light tap of the heel to fix it in place, to make sure it will not move until one is ready to come down.”

The text itself is very literal and seemingly silly. It reminded us of how we write prompts for Chat-GPT or for the various image synthesis tools. We looked up Boston Dynamic’s dog-like robot learning how to walk through natural terrain, and all those rendered physics simulations that use reinforcement learning to teach humanoids how to walk like humans.

‍

‍

However, they almost always end up coming up with their own ways of performing the task at hand that are very much different from what actual humans would’ve intuitively done. We talked about how these neural networks are modeled after our brains, and probably do work similarly, but not quite the same way, because they're not actual humans. And so in that sense we thought it would be interesting to break down our instructions text into a set of steps a robot should follow, using Chat-GPT, and then generate prompts out of those to create footage that’s based on that, and possibly also on images we’ll train it on. Here's a link to the Chat-GPT chat with all the generated text.

‍

We’re trying to figure out the styling of the footage we generate. We’re looking at either going with a 1950’s vintage cinema style or maybe the scientific humanoid simulation style. We’re currently experimenting with the different image synthesis tools and trying to see what feels right.

‍

‍

We think there’s an interesting tension to work with here because while the text is very poetic it is somewhat robotic at the same time. And however intuitive and obvious it might be to actual humans, climbing up stairs is not an easy task for a robot. It makes the people building it have to think about all these intuitive steps we must make in order to be able to do that, and lay it all out in computational terms that the robot could “understand”.

‍

Thanks for reading this far!