Multimodality

Multimodality is an inter-disciplinary approach that understands communication and representation to be more than about language. It has been developed over the past decade to systematically address much-debated questions about changes in society, for instance in relation to new media and technologies. Multimodal approaches have provided concepts, methods and a framework for the collection and analysis of visual, aural, embodied, and spatial aspects of interaction and environments, and the relationships between these.

Three interconnected theoretical assumptions underpin multimodality.

First, multimodality assumes that representation and communication always draw on a multiplicity of modes, all of which contribute to meaning. It focuses on analyzing and describing the full repertoire of meaning-making resources that people use (visual, spoken, gestural, written, three-dimensional, and others, depending on the domain of representation) in different contexts, and on developing means that show how these are organized to make meaning.

Second, multimodality assumes that resources are socially shaped over time to become meaning making resources that articulate the (social, individual/affective) meanings demanded by the requirements of different communities. These organized sets of semiotic resources for making meaning (with) are referred to as modes which realize communicative work in distinct ways – making the choice of mode a central aspect of interaction and meaning. The more a set of resources has been used in the social life of a particular community, the more fully and finely articulated it will have become. In order for something to ‘be a mode’ there needs to be a shared cultural sense within a community of a set of resources and how these can be organized to realize meaning.

Third, people orchestrate meaning through their selection and configuration of modes, foregrounding the significance of the interaction between modes. Thus all communicational acts are shaped by the norms and rules operating at the moment of sign making, and influenced by the motivations and interests of people in a specific social context.

Multimodal research  undertaken to date can be classified as having four major foci:

  1. The systematic description of modes and their semiotic resources;
  2. Multimodal investigation of interpretation and interaction with specific digital environments;
  3. Identification and development of new digital semiotic resources and new uses of existing resources in digital environments; and
  4. Contribution to research methods for the collection and analysis of digital data and environments within social research.

There is considerable debate as to whether multimodality can truly be considered a theory or whether it is more appropriate to view it as a method. In comparison with ethnography, it can be argued that multimodality can be used as theory, as perspective, or as method, and that these different degrees of engagement with multimodality help to make sense of what is seen to count as multimodal.

Editor: Carey Jewitt

Key References:

Jewitt, C. (ed.) (2009) The Routledge Handbook of Multimodal Analysis. London: Routledge.

Norris, S. (2004) Analyzing Multimodal Interaction. London, RoutledgeFalmer.

O’Halloran, K. L. & Smith, B. A. (eds.) (2011) Multimodal Studies: Exploring Issues and Domains. New York & London: Routledge.