The relationships across and between modes in multimodal texts and interaction are a central area of multimodal research, and multimodal research often investigates the relationship between a given context and the configuration of modes in a text or situated interactions. This helps to both better understand the modal resources in use and to address substantive questions. This focus on multimodal orchestration has been a focus of this work, for instance understanding how multimodal cohesion is realized (or not) through the integration of different semiotic resources in multimodal texts and communicative events via rhythm, composition, information linking, and modal density or intensity.
How people draw on different embodied modes when they interact has been the subject of studies within multimodality. Bourne and Jewitt (2003), for example, analysed how a teacher’s orchestration of modes in a classroom drew together the semiotic and social resources of texts and students to create a gendered debate. Flewitt’s (2006) multimodal study of preschool classroom interaction demonstrates the strong link between the communicative demands of a context (in this case home and preschool playgroup) and the modes in use. Focusing on all modes of communication (speech, gesture, movement, gaze and so on) she is able to scrutinize young children’s multifunctional uses of different modalities as “…intentional, socially organized activity in the construction of meaning” and to argue against ‘pathologizing the absence of talk’ (Flewitt, 2006: 47). These studies show the potential of multimodal research to offer a different account of classroom communication by locating the analysis of speech in the broader context of children’s total multimodal resources and examining their inter-semiotic relations.
The ways in which contemporary digital texts are organized via textual features such as digital layering and hyper-linking and the impact of this on how people navigate multimodal digital texts has also been examined as a part of multimodal orchestration. This term is potentially useful when thinking about the take up of designed resources and exploring the dynamics of the interaction between modes in a text or interaction.
See also: coherence, transduction, multimodal orchestration, multimodal ensemble
Key References
Bourne, J. and Jewitt, C. (2003)
Orchestrating debate: a multimodal approach to the study of the teaching of higher order literacy skills
Reading: literacy and language, UKRA July: 64 – 72.
Flewitt, R.S. (2006)
Using video to investigate preschool classroom interaction: education research assumptions and methodological practices
Visual Communication, 5(1): 25-50.
Glossary of multimodal terms 