Vortex. Visual Aural Textual. One Language.


Vortex. Visual Aural Textual. One Language.Robert Appleton interview by Majid Abbasi (published in Farsi, in Neshan Magazine, Tehran)"Sound text and image integrate together no matter what random order they're placed in. Our minds synchronize the experience. We invent the meaning. And this forms the basis of human interaction."M_ What is 'Vortex' - as it relates to the title of your recent performance?R_ Vortex is my development of a single language of the visual, aural and textual. In our interview today for example, the camera you are using captures the sounds, we speak to each other in three languages - the visual, aural and textual. The full title of my work is 'Vortex. Visual Aural Textual. One Language.' The term Vortex is represented by the sounds which form it in English: V (for visual) + or (for aural) + tex (for textual) = Vortex. It also exists as an 80 page thesis which earned my Masters degree in Interdisciplinary Art, Media and Design at Ontario College of Art and Design University in Toronto last October and as the outline of my book which will have a foreword by Noam Chomsky - probably our greatest living linguist.



Music Gallery Toronto, 2010 with Bob Brough saxophone and Pieter Coussement, programming and electronics


M_ Why "one language"? Where does it come from? What is the background? R_ Perhaps the oldest form in history to express the singularity of sound text and image is the ancient OM chant - which is at once a word, a picture and a sound. For centuries however, we have preferred to develop three separate languages - the visual (image), the aural (sound) and the textual (word), for which we have created individual aesthetics, vocabularies and histories as well as thousands of dialects, alphabets and sounds. Language has become too closely associated with only one vocabulary – the word. Changes in philosophy, society and technology have made new communications concepts essential. Vortex is the result of my investigation into expanding our idea of language. Improvising the visual, aural and textual simultaneously in real-time uncovers new meanings not expressed by any one medium alone. When Gyorgy Kepes published his Language of Vision in 1938, he said "As the eye is the agent of conveying all impressions to the mind, the achieving of visual communication requires a fundamental knowledge of the means of visual expression. Development of this knowledge will generate a genuine 'language of the eye,' whose 'sentences' are created images and whose elements are the basic signs, line, plane, halftone gradation, colour, etc." (Gyorgy Kepes, 1938 in Bauhaus, Hans M. Wingler, MIT Press, 1969, p 197)And 50 years later in 1989, Armin Hoffman wrote "Among semioticians, information theorists and media experts it has been commonly held for some years that reality can no longer be captured and described with linguistic means alone. It is changing too quickly and growing too complex. Clearly language as our most important medium of communication has reached an impasse... A language of pictures, drawings, diagrams and photographs is in the process of supplanting language, or at least extending and enriching its scope." (Notes on the Language of Signs and Symbols, Birkhauser)With Vortex, I have set out to create a single language of all three components – words, pictures and sounds. I have written several earlier publications on the subject: A Word is a Picture became a design class in which I did ethnographic research with students in China, Turkey, the US and Canada. This was followed by A Picture is a Sound which became another class. About Time: A Theory of Design, Music and the Play Instinct was published in 2009 (in Tempus Fugit, Index Books, p 11). And Vortex is the latest in the series.


Event 2, Text 1 

M_ My feeling on your work when I was at your performance at The Music Gallery is that it is closer to art than design? Is this correct? R_ I practice design as an art form – in the way I think all good designers do – attempting to reach its ultimate level, as the art of design. I've also been called an artist by some designers and a designer by some artists. In New York City in the early 90's when the debate over whether someone was a graphic designer, an artist or a commercial artist was at it's height, new media people would say "I don't care what you call it as long as it's interesting". And now that I'm also working with music and language, I feel that I've taken this full circle. It's not design, it's not art, it's not music and it's not literature. It's all of this. And for me that's the measure of the work. It's by not attaching to any one subject that new things begin to emerge. I see this project going beyond my personal work – it becomes public, a new medium if you will, which many can contribute to. M_ You mean that it can be communicated to an audience? R_ It can be communicated. In the Music Gallery, the audience was essential since they interpreted and contributed to everything we did. I made this announcement at the beginning of my piece "Tonight we will all make this event. Everything you do, will be projected into the performance, influencing sound, image and text." When the audience applauded, their sound made the images and text shimmer and the performance began. My Music Gallery videos are here  


 Event 2, Screen 1


M_ This is the spirit of design. So it's a kind of visual communication, and it's between art and design?R_ Yes. And you know, the first piece that I played at the performance on Monday is influenced by Leo Lionni's Little Blue and Little Yellow, about the relationship between two colors, yellow and blue, which represent the mixing of cultures or the connecting of ethnicities. My piece (Event 1) is dedicated to Leo. It's about a mixture of the three 'colors' of image, text and sound into a new way of speaking. We became friends after I presented him for the AIGA/NY American Masters Series at Cooper Union. An informal video interview I made of him is in Roger Remington's collection in the Vignelli Center for Design Studies at RIT. My artistic references are very broad. They include: designers Moholy-Nagy, Jan Tschichold, Gyorgy Kepes, Armin Hoffman, Karl Gerstner, Paul Rand, Leo Lionni, Wolfgang Weingart and Muriel Cooper; artists Marinetti, Kandinsky, Klee, Robert Irwin, James Turrell and Christian Marclay; musicians Arnold Schoenberg, John Cage, George Russell, Thelonious Monk, John Coltrane, Miles Davis, Iannis Xenakis, David Tudor and Robert Ashley; film makers Oskar Fischinger, Norman McLaren, Michael Snow, John Whitney and Stanley Kubrick; philosophers Walter Benjamin, Jacques Lacan, Fredric Jameson, Slavoj Zizek, Nicolas Bourriaud and linguists Ferdinand de Saussure and Noam Chomsky.M_ You mentioned that your work is at the "intersection of improvised sound, text and image." How does this happen? R_ That's a broad question, so I'll try to answer it in two parts: Physically it happens in the computer because I use the software Max/MSP/Jitter, which translates sound text and image into mathematics. And the mathematics is what makes everything work together. So I physically create the responses between sound text and image by speaking and playing my keyboards. And there's also a response from other performers and the audience. Everything which takes place during a performance influences the totality of the performance and what happens. The question is also interesting as philosophy – what are the relationships between sound text and image? In Event 5, I say "Sound, text and image integrate together no matter what random order they're placed in. Our minds synchronize the experience..." Different meanings are created by the juxtaposition of sound text and image during improvised performance. And with Vortex, invention in the moment using three vocabularies can be made by one single person or by as many people as desired.



Event 2, Book 1, 2009


M_ So maybe it's important that I ask this question: How does information help art and design, and what is its role? R_ Information informs. I believe we have not yet identified all the information we possess. M_ How important is technology in your work? And also in art & design generally?R_ Technology is essential to my work and appears in most of what I do. When I produce a performance, or a movie, then publish it on the internet or in print, it becomes mediated by technology, whether it began life in that form or not. I still love analog media however – and I always will. Digital music, design and art are relatively new, as photography and film were in the early 20th century. Their significance is their currency, the new forms which are created with them, and the intermingling of media which use mathematics as a base (almost everything today). What has also grown with technology is language. New vocabulary has been developed to describe the effects of technology in art, design and music. 'Crossfading' is one example. In film and music it means moving between or mixing two sources such as video tracks in a movie. In art and design it can mean mixing two images, or two subjects, or two typographic treatments. 'Pitch Control' allows us to adjust the speed of a recording while the pitch (or scale) can remain constant. In Douglas Gordon's art piece 24 Hour Psycho, Hitchcock's film is pitch controlled – or slowed down to run for 24 hours. In design typography can be 'Pitch Controlled' by adjusting its kerning while maintaining its scale. Nicolas Bourriaud has written extensively of these influences in art (Postproduction, Lukas & Sternberg, 2002).



Immersion diagram


M_ You mentioned an experiment in China when you and DouDou were improvising together. What was the relationship between music made by this parakeet and your playing on a keyboard? Were both of you doing your own thing? R_ Just before arriving in Toronto I spent a year teaching at CAFA in China. I had an electric piano in my apartment and was looking after our landlady's parakeet called DouDou. One morning I was playing piano as usual and my wife was making breakfast. Suddenly I realized that DouDou and I were improvising together – she was singing and chirping and we were following each other musically as improvising musicians. It lasted for over an hour. Cathe set up a video camera and made a recording. DouDouBob can be seen here [ ]Interspecies communication is very interesting. When making music with other humans, we can practice our vocabulary and become better at it; we can compare this with other vocabularies and decide how we might change it to make it better than it was. And we can present that. When communicating in an unknown environment however – with a creature which is not using these human traits – the rules are gone. There is no way to 'practice' playing with a bird, except by doing it. The bird doesn't get better, and we don't get better. We just do or don't do. And what's interesting in this situation is the coincidences that occur. And the subconscious communication between one mind and another. One very small creature and one much larger, listening to and responding to one another. Proof that language beyond words is very powerful (note: I made a presentation of this work to the Product Design Team at the Lenovo Research Center in Beijing in 2007 and they were also interested in the idea).M_ You explained a little about the software program you used. Please tell me more.R_ In the simplest terms Max/MSP/Jitter acts like a series of electronic diagrams on music which is translated into mathematics by Midi. It operates like a switch which allows a course of action and is turned on or off with something called a 'bang.' The operator can also modify this course of action with additional 'switches' - controlling speed, scale, volume, etc., with infinite flexibility (using any known scale of measurement). It was written by Miller Puquette at IRCAM in Paris, which was founded by the composer Pierre Boulez. Puquette recreated an electronic music environment in software. It's quite well designed and others have added functionality so that as Max/MSP/Jitter, it now controls audio, video and image. And it can operate objects outside the computer using wireless technology and sensors like Arduino.In my use of Max, we have tailored the software to work with ideas about immersive language and space. We created a test-environment, a 360 degree cube in which almost anything can happen – and be measured as research. Projection can take place on multi-surfaces – in front, above, below, to the side, in a state of immersion. Some people make anatomically correct models of the human skull – on which they perform measurements. This is an anatomically correct model of the human imagination – for use in pure science, installation or live performance.


Max/MSP software patch (above) 

M_ How can you discover new meaning in vocabulary?R_ We are used to viewing things through particular lenses. The lens through which we view literature for example is a literary lens. And actually there are many more things going on at the same time as we are reading. In John Cage's idea that everything we hear is music – there is no silence and we can simply be quiet to hear everything that is. What I'm saying is that within this same 'silence' there's a lot more than sound. There are a lot of pieces of information we haven't yet identified, and we may not be aware of them until they're pointed out to us. In my work, information and meaning can come from anywhere within the perceivable vicinity of an event. It is not restricted to the body of the event itself. For example, a sound that happens across the road, outside my vision and within my hearing is included. A person not involved in an event, though moving through or past the event is also included. There are multiple layers of information which we tune in and out of. All of these layers have meaning. M_ And also tell me please about the installation on the performance stage. So many things were there – a balloon, a three-projector system, with only one screen, a computer, three keyboards, three performers with no apparent conducting by you...R_ At The Music Gallery there were three performers, myself on electronics and voice projected on to 3 screens, Bob Brough on saxophone and Pieter Coussement on electronics and vibrator with an inflated balloon. I was conducting invisibly and composing in the moment. My computer controlled the electronic imagery and my sound. Bob and Pieter were playing independently. I was composing in the moment. They were composing in the moment. My conducting was created with the score Performance Score.pdf which everyone played from and then with all the imagery sound and text we created collectively – performers and audience. The space was also important – one in which the full range of human interactions could occur. My ideal performance space has projection above, below, behind and to the side. All our senses are represented, and its as if you're inside your own head.   M_ What is your feeling about new concepts in design? Or to put this more clearly, how is design changing in the 21st century?R_ Graphic Design (to be specific) is no longer limited to two or three dimensions. It's not limited to printing or manufacturing. It's not just words and pictures. It's not restricted by anything other than our imaginations and the willingness of society to accept it. The design subjects are becoming broader for me, rather than more specialized. The principles of quality, creativity and communication remain, even as they are modified by individual artistic and philosophical points of view.


Performance ScoreMusic Gallery, 2010


M_ Do you believe that we have reached the end of desktop publishing? And old media have been replaced with new ones? R_ Not the end, the beginning. My father-in-law reached the age of 90 last week and purchased a new 13" MacBook Pro. I'm very familiar with Macs so I helped him set it up – and I was shocked at how unnecessarily difficult this has become. Hopefully, new computers will take into account our different cognitive abilities. Then our lives will become infinitely better. We have just begun with computing. New media and environments such as Vortex go beyond the 'desktop.' The best is yet to come if we can embrace human need.M_ How does information enhance art and design? What is its role? What differences are there in design process between now and the past? R_ Information is knowledge. It informs design as much as it informs art or any other subject. It's role is to allow access to knowledge and to allow new knowledge to be developed. The difference between process in the past and now is the change in tools and perception caused by new conditions in contemporary life. The pace of change on the surface seems faster all the time. Yet it's only when we stop, and study the present, that we begin to understand what has changed. My perceptions change each time I enter the academic world or change direction in my career. Then I learn, and new knowledge, passing through me, is returned to my work and its audience.


AGI Istanbul, 2009 with Yahya Dai, saxophone


M_ If Vortex is a language, what is it's grammar? And what are the relationships between sound text and image?R_ In English grammar, adjectives modify nouns. With a statement like "water is hot" 'water' is a noun, and 'hot' is an adjective that modifies it. The adjective tells us what kind of water this is. In Vortex grammar an adjective can be a visual, aural or textual segment (not necessarily a word) with the power to modify whatever is around it. In Sketch for Performance (which can be viewed here ) my animated typography from Fredric Jameson's essay on postmodernism loops 3 times. The text is edited together with a randomly chosen soundtrack from Mickey Mouse, Karnival Kid, 1929. The audio segment by chance is precisely the same length as the entire animation. It is played once and each time the animation repeats, its meaning is subtly changed by the sound. In the opening scene the title appears with a fanfare (a comic though real suggestion of significance). Initial screens which present Jameson's well-accepted argument play over an opening sigh followed by the hushing sound associated with the announcement of important remarks in a speech or an opening at the Opera. Rising tension is maintained by the sound of a needle crackling on a scratched LP or 78 disk. A brief explosion after the appearance of 'premonitions of the future,' another hush at 'catastrophic or redemptive' and 'the end of this or that' is followed by a dog bark suggesting happy agreement with the premise? A creaky spring, a munching sound and a whistle introduce 'post,' followed by 'modernism' which fades to a strumming guitar and the beginning of the love song Sweet Adeline which introduces the second playing of the animation. Now the song's lyrics are converted into approximations of words as their meaning becomes forgotten by the singers. When 'catastrophic or redemptive' appears for the second time it begins with a whistle and cork-pop (a celebration?). The forgotten lyrics continue briefly until at 'the end of this or that' we hear a snore and whistle. The final rendition of the text begins out of sequence, as the music begins to speed to a rousing finish, and the animated image immediately zooms out ultimately disappearing in reference to a shrinking spotlight – as the words 'post' and 'modernism' now a fraction of their former size – vanish forever and the piece ends. Through our interpretation of Jameson's text about the state of postmodernism and the synchronicity of visual, aural and textual matter, this piece demonstrates how much sound with image and text can change meaning. By playing this game of improvising three languages as one (Vortex), we discover new vocabulary, where three senses contribute equally.1 1     Recent workshops and performancesAGI BeijingGrafist IstanbulCAFA Beijing Lenovo BeijingAGI IstanbulAXA BarcelonaFuturePlaces PortoMusic Gallery Toronto


AXA Barcelona, 2010 with JP Carrascal guitarand Sergi Felipe Fernandez saxophone

M_ Any final thoughts?R_ Anyone with an interest in communicating with 'One Language' should feel free to communicate with me. There have been several titles for this current work. In New York "A Word is a Picture" was a course at Parsons School of Design and a small book. Then came "A Picture is a Sound" and "A Word is a Picture is a Sound". This is my life's work. It is performance, installation, publishing, pure research – and all the things which flow from that. I'll end with a quote from George Russell who's Lydian Chromatic Concept of Tonal Organization was responsible for the notion of Tonal Gravity and the development of Modal Jazz.             "Something unseen was working. It didn't come             from me. But this knowledge used me as its...        vehicle. When you take a road that hasn't been         followed, you're really obligated to see it         through... What's a life for... if not to explore?"22             Design, image, concept © Robert Appleton 2009-Contact All Rights reserved