A Rough Guide to the Theory of Semiotics

How communication works as series of signs

In past posts, I have discussed an approach to communication called discourse theory, which connects communication to social processes and asks how people with different interests contribute to and shape commonly-accepted truths in a society. Such a framework is useful for exploring truth claims and knowledge construction, particularly when the focus lies on who has the power to make certain statements, but it does not tell the full story of how communication works. What happens, for instance, when discourse is not expressed through language, but through gestures, images, or sounds? What happens when discourse becomes mass-communicated? This is where another approach to communication provides helpful answers: the theory of semiotics.

Where discourse theory is interested in macro processes, like the flow of knowledge through time, semiotics is interested in micro elements. It is concerned with questions like: How does information become transmitted? What makes up a message? To answer such questions, semiotic theorists look at the building-blocks of communication processes and examine how these building-blocks interact. In other words, they are interested in signs. As Ellen Seiter puts it (1992: 31): “Semiotics is the study of everything that can be used for communication: words, images, traffic signs, flowers, music, medical symptoms, and much more. Semiotics studies the way such ‘signs’ communicate and the rules that govern their use.”

In this post, I will introduce you to this science of the sign, and will explain why I think it offers a useful way of conceptualizing political communication, whether in East Asia or elsewhere. Semiotics can at times be highly technical and complicated, which is why I will try to keep things as simple as possible. If I have done my job well, then by the end of this post you will have a better idea of what a signal is, what the difference between an icon and an index is, what a code does, and what semiosis means. I’ll discuss all of these terms, and will use practical examples from China to make them come to life. So if you are wondering what semiotics might have to say about images of Mao Zedong, or Internet censorship, or cosmetics, you will find out below.

To start us off, I will first look at the smallest though arguably most important element of semiotics: the sign. How have different thinkers conceptualized signs and the ways in which these units transport meanings? I’ll then move up to the next level of complexity, where signs come together to form messages. Finally, I will move up once more, to the level of broader social dynamics. This will lead me to my conclusion: that semiotics connects beautifully with theories of discourse, offering a rich account of how complex communication processes work.

Things that stand for other things

So what is a sign? In essence, a sign can be anything that stands for something else. The classic example is of course the traffic sign, which stands for a specific behavioural imperative, such as “stop” or “no U-turn allowed”. However, signs come in all shapes and forms. Dark clouds on the horizon can stand for a coming rain storm, and a photo of those clouds may still stand for that same rain storm many years later. As these seemingly trivial examples already illustrate, there are different kinds of signs, and the rules that structure how they work are quite different.

One of the scholars who is most famous for having studied signs is Ferdinand de Saussure. Saussure was a French linguist who was concerned with the structural features of language. He explored what the relation might be between a word and the thing it stands for. His view of such relations forms one of the founding pillars of semiotics. Saussure defined the sign as a “two-sided psychological entity” that links a mental concept of a thing (the signified) to a sound pattern (the signifier; Saussure 1916/1983: 66). For instance, if I say the word “rain”, then my verbal expression connects to a mental idea of water falling from the sky. Saussure famously argued that even though the link between a signified and its signifier becomes a matter of “collective habit” (ibid.: 68), and is consequently governed by rules, the nature of the sign is itself arbitrary. The English word “rain” has nothing to do with the actual phenomenon it describes – it does not sound like the actual fall of rain, and neither does it look like it. What is more, words for rain in different languages demonstrate that the signifier for this phenomenon can easily be something entirely different (Seiter 1992: 14): the French word “pluie”, the Dutch and German word “regen”, or the Chinese word “yu” don’t have anything in common.

Once we move away from linguistic expressions, however, the situation becomes much more complicated. Saussure’s view has consequently been criticized for not taking into account how other types of signs work, and how the relation between a signifier and its signified are often not at all arbitrary (Eco 1979: 15). If I draw you a picture of falling rain, is the relation between the event and my drawing really random, or do all drawings of rain have something in common beyond social conventions of how to draw? Take another example: one night, you are woken up from your sleep by the sounds of heavy rain against your bedroom window. You won’t need to take a look outside to know it is raining. The sound already tells you so. Is it then really accurate to say that the sound is arbitrarily linked to the phenomenon?

Peirce’s semiotic triangle

At roughly the same time that Saussure was developing his theory of semiotics, an American logician named Charles Sanders Peirce was working independently on a similar project. He was convinced that there were different types of signs, and that the relation between a sign and its object was not necessarily arbitrary (cf. Hoopes 1991: 11). To make this case, Peirce argued that a sign consisted of three elements that form a triangle: an object, its representation, and its meaning. For instance, the sound of rain on your window (what Peirce called the “referent”) is linked to the actual rain falling outside (the “object”), but it is your interpretation (the “interpretant”) that connects the two, thereby completing the triangle and turning the sound into a “sign” for rain. In fact, without the interpretant, you might still have a signal, but you wouldn’t have any meaning.

This is the semiotic theorist’s answer to the classic philosophical question of whether a tree that falls in the woods makes any noise if there is no one there to witness it. The answer is both yes and no: the fall of the tree produces a signal, which we might call “sound”, but without someone to interpret that signal it does not become a meaningful entity, or sign.

For Peirce, what crucially defines a sign is the link in the triangle that is most important in the relationship. This led him to define three different kinds of signs (ibid.: 239-240). The first is the icon, which is a sign in which the representation resembles the object strongly enough to be recognizable as the “real thing”. This is the case for drawing, or for photos, or a video recording. The important link in the triangle is between the object and its meaning, but the referent is less relevant. Think of some of your past experiences with photo journalism: when you see a news picture of a terrible explosion, your attention goes to the object, on the assumption that you are indeed seeing the real event “as it is”. The fact that you are looking at an artificially created and edited representation is probably not your main concern… unless, of course, you are critically analysing how the image was constructed.

The second kind of sign is the index. Here, the referent is the consequence of the object, and the meaning is not crucial to the relationship. For instance, smoke is an index that something is burning, regardless of whether anyone is there to interpret the smoke. This is then also the category in which the sound of rain on your window would belong. Whether you are waking up from it or not, the rain is still falling, and it is still causing a signal to be transmitted through your room: the tapping sound on your window.

Finally, there is the type of sign that Saussure was writing about: the kind that does not have a strong relation to its object. Peirce refers to such signs as symbols. The relation that matters is the one between the referent and its meaning, as is the case with words, or company logos, or national flags, or the kinds of rude hand-gestures that people make when you just stole their parking space. The representation and its meaning are important, but the link to the object is purely a social convention. Imagine, for a second, what would happen if national flags were not flags at all, but something entirely different, say: national bouncy balls. If you were able to convince people that a specific colour of bouncy ball stands for their nation, then national holidays could easily consist of masses of people bouncing rubber balls around. It would be no less logical than waving a flag.

In most practical cases, a sign will not neatly fall into one of these three categories, but will share properties of each. For instance, is an emoticon smiley an icon or a symbol? It resembles a smiling face, but the resemblance can be quite weak, for instance when I “wink” at you by typing: ; ) Would anyone who is untutored in the use of emoticons recognize this character combination as a representation of a winking face? The more the icon becomes stylized, the more it starts functioning as a symbol. A very illustrative example is the Chinese character script. Many characters started their life as drawing of real things, or as linguists would say: as pictograms. Over the centuries, these drawings become more and more abstract, until they no longer resembled the thing they stood for in any meaningful way. The character for “horse” may have started as a picture of a horse, but show it to a non-Chinese speaker now, and they are unlikely to recognize it as such. It has moved from being an icon to being a symbol.

In that sense, the triangular relationship that Peirce introduced does not need to be understood in rigid terms. Specific signs can be located anywhere between the extremes, and they can move from one category to another over time. The semiotic triangle is a useful heuristic device that reminds us that the elements of communication behave according to certain patterns, and that these patterns can be explored.

Signs within signs within signs

One of the fascinating things about semiotic triangles like the one that Peirce conceptualized is that they can be stacked. For any object, I can create countless amounts of triangles that link the object to a referent and an interpretant. Take the example of Mao Zedong, the famous founder of the People’s Republic of China. Mao has been the object of endless representation. His portrait decorates the Gate of Heavenly Peace in Beijing. His statue decorates city squares, while paintings depicting the great chairman adorn government offices, not to mention many Chinese living-rooms. Then there are the famous Andy Warhol pop-art paintings of Mao, artworks from China that deploy Mao’s image, the stylized silhouettes that adorn many Mao pop paraphernalia, and of course the various movies that tell the story of Mao’s life. The list goes on. In semiotic terms, each instance of representation is its own sign triangle, even though all of them start with the same object.

The referent of a sign can also be the starting point of a new sign triangle. Think of the tourists who take pictures of Mao’s famous painting above the Gate of Heavenly Peace. Each goes home with their own, new sign triangle – only now the object of the triangle is itself a triangle. I am doing something similar in this post when I show you images of Mao-related art works.

Finally, the interpretant of a sign triangle can likewise form the starting point of another semiotic relation. The meaning that Mao has for someone could be the focus of a verbal statement, or the starting point for an artistic expression, or the cause of a salute. All of these examples produce new signs, which can in turn be the starting point for new sign relations, and so on. This process of creating chains of signs is called semiosis.

An important distinction in this regard is between a single sign, for instance a Mao picture, and a sign within another sign, for instance what you make of the Mao picture that I show you. The simple depiction of Mao in the form of a painting is called a denotation. I can show the picture to anyone who is familiar with Mao and they will be able to point at the picture and tell me what the object of the picture is. Yet each person I show the picture to will take that first step (interpreting the referent as standing for Mao) and make it the starting point of a second step (relating the image of Mao to what Mao means to them). Such signification at one level of remove is a connotation.

The semiotician Umberto Eco, more commonly known for his famous novels, has provided a detailed account of how processes of denotation and connotation work. To Eco (1979: 56), a connotation is a semiotic relation (or sign-function) that consists of another semiotic relation. Eco explored what the rules of such relations were and argued that “a sign-function is the correlation between an expression and a content based on a conventionally established code (a system of correlational rules)” (ibid.: 191). Eco’s concept of code is crucial to understanding how signs can be combined to form messages, and why coding, transmitting, receiving, and decoding such messages follows systematic patterns.

The coding and decoding of messages

Why is it that a red light at an intersection will compel you to stop driving, yet a red LED lamp on your computer screen does not compel you to stop surfing the internet? The answer is that “red light” is coded differently by social conventions in each of these contexts.

A code governs how signs should be interpreted in a given situation. Think of the booming industry for women’s cosmetics, which creatively cashes in on the fact that “skin tone” connotes different things in different cultures. In East Asia as well as in Europe or the US, skin tone is coded to stand for prosperity. Yet in East Asia, the “sub-codes” of this sign-function are different from the US or Europe. In the so-called “West”, prosperity is reflected in the degree to which people have leisure time. If you have the option of being outside a lot, playing golf, or going skiing, or lying on a beach, then you are “well off”. An index of this is that your skin is likely to become tanned. Conversely, if you are pale, you might be overworked and sick, which is not a good sign. In East Asia, however, tanned skin is an index of manual labour, and is consequently associated with work on a construction site or in a field. Prosperity, on the other hand, is associated with the luxury of staying in-doors. If you are a white-collar worker, or are able to work from home, your skin-tone will remain light.

Cosmetic giants have exploited these layered sign-functions to make a profit. In London or Berlin or Boston, where social codes associate prosperity with tanned skin, you will find self-tanning lotions and spray-on tans on supermarket shelves. In Shanghai or Tokyo or Seoul, where the code works slightly differently, you will instead find whitening creams and other bleaches on those shelves. Note that these products are mostly designed to appeal to women – an outcome of the way that gender is coded in our societies.

As these examples show, social conventions and the communication codes they create can have broad implications. In fact, codes can be highly political. Think of the way that the Chinese state monitors the Internet in China for controversial keywords, and the strategic responses from those who try to avoid such censorship. For instance, discussions about the ruling Party’s policies can be quite heated on the Chinese web, which is why key slogans like “harmonious society” or “the three represents” often become so contentious that they are censored. But what do you do when you still want to discuss the object and its meaning, but your referents keep getting erased? Well, you design new referents and recode your message. This is exactly what Chinese netizens regularly do, and the Chinese character script lends itself to such practices: the characters for “harmony” can be replaced by other characters that sound the same, but that have a completely different meaning, such as “river crab”. A blogger can thus code and transmit a message about “river crab society” without getting censored. Of course, once this code becomes widely known, the new referent will also be censored, leading to a kind of semiotic arms race: censorship becomes a battle over who gets to code certain referents in certain ways.

The disambiguated message

A major challenge of communication is to make sure that messages are decoded the way they were intended. If an object can connote different things in different contexts, depending on the code that is applied, how can I make sure that the recipient of my message uses the right code? For example, I could show you the picture of an apple, but how can I assure that you understand that I am using the apple as a reference to the computer manufacturer of the same name, rather than as a Biblical reference to original sin? What if the picture of the apple is badly lit, so that you can’t even tell whether it is an apple or a pear?

The French semiotician Roland Barthes has answered these kinds of questions by showing how signs act together to disambiguate meanings. Barthes was concerned with French advertisements, and how the images worked together with the words. His views remain highly controversial, but what I find useful about his work is his argument that words can play two functions in conjunction with images (Barthes 1977: 38-40). They can either clarify what the image shows, or they can clarify what it means. If the words state what the image shows, then they disambiguate a denotation and “anchor” the viewer’s perception. If they state what the image means, then they disambiguate a connotation and “relay” a specific interpretation. In either case, by juxtaposing visual and verbal signs, I can significantly influence what kind of code a viewer is likely to apply when decoding the image.

Yet the disambiguating element does not have to come in the form of language. I don’t have to create a caption to let you know that an image of an apple stands for sin. I could combine the apple with other visual signs, such as a naked couple, a tree, and a snake. If you are familiar with Biblical codes, then you’ll get the message. Similarly, music can be an excellent way to guide the perception of images. Music scores for movies aptly demonstrate this. Here is a useful video experiment that shows how different kinds of music can radically change the meaning of images, even though the video footage is exactly the same in all examples. In other words, relay and anchoring effects can be achieved by layering signs in different modes (visual, acoustic, verbal, etc.), which is why complex communication processes can also be called “multi-modal” (Kress & van Leeuwen 2001: 22).

This is essentially how mass communication works. It is why you and I can watch the same mass-communicated video, for instance this car commercial, and not have hugely diverging views about what the video is about: the combination of gestures, costumes, camera actions, and sound clarifies what codes we need to adopt to understand the clip, and these codes assure that we understand the message as it was intended – provided, of course, that we have actually previously learned these code. In this particular example, someone with no cultural knowledge of commercials would probably have an equally hard time decoding the message as someone who had never seen StarWars.

Conclusion: how the theory of semiotics connects to discourse

As I hope these examples have demonstrated, semiotics is an exciting perspective for thinking about political communication. It not only captures the “micro” elements of communication processes, it also highlights how these processes follow certain rules, opening them up to scientific analysis. What is more, semiotics places a strong emphasis on the fact that it is human beings who create the sign functions of human interactions, and that these interactions therefore follow social conventions. This is where semiotics overlaps with theories of discourse. Both can help us understand how meanings are subject to change, and how such change might be not only a matter of time, but also of human effort and of power relations. In other words: sign functions, like the broader discourses that they form, are the outcome of continuous creative impulses, revisionist activities, and complicated negotiations.

For those of you who want to get into the subject of semiotics in more detail, I have provided a list of further readings below, and can also recommend Daniel Chandler’s online introduction Semiotics for Beginners. I hope to write additional posts in the future on how shifts of meanings take place, and on how signs travel across society. If you are interested in methodological tools for studying multi-modal communication, take a look at my practical tips on how to conduct a visual communication analysis. For now, I hope to have inspired you to take notice of the various signs that you are confronted with every day, and to start thinking about political communication as a series of signs that you can decode and analyse.

References

Barthes, Roland. 1977. “Rhetoric of the Image”, in Heath, Stephen (ed.): Roland Barthes – Image, Music, Text. 11th ed., London: Fontana Press: 32-51.

Eco, Umberto. 1979. A Theory of Semiotics. Bloomington: Indiana University Press.

Hoopes, James (ed.). 1991. Peirce on Signs – Writings on Semiotic by Charles Sanders Peirce. Chapel Hill & London: The University of North Carolina Press.

Kress, Gunther & Van Leeuwen, Theo. 2001. Multimodal Discourse – The Modes and Media of Contemporary Communication. London: Arnold.

Saussure, Ferdinand de. 1916/1983. Course in General Linguistics, transl. Roy Harris. London: Duckworth.

Seiter, Ellen. 1992. “Semiotics, Structuralism, and Television”, in Allen, Robert C. (ed.), Channels of Discourse, Reassembled: Television and Contemporary Criticism. 2nd ed., London: Routledge: 31-66.