Multimodality

In its most basic sense, multimodality is a theory of communication and social semiotics. Multimodality describes communication practices in terms of the textual, aural, linguistic, spatial, and visual resources - or modes - used to compose messages. Where media are concerned, multimodality is the use of several modes (media) to create a single artifact. The collection of these modes, or elements, contributes to how multimodality affects different rhetorical situations, or opportunities for increasing an audience's reception of an idea or concept. Everything from the placement of images to the organization of the content creates meaning. This is the result of a shift from isolated text being relied on as the primary source of communication, to the image being utilized more frequently in the digital age. While multimodality as an area of academic study did not gain traction until the twentieth century, all communication, literacy, and composing practices are and always have been multimodal.

Although discussions of multimodality involve medium and mode, these two terms are not synonymous.

Gunther Kress's scholarship on multimodality is canonical in writing studies, and he defines mode in two ways. In the first, a mode “is a socially and culturally shaped resource for making meaning. Image, writing, layout, speech, moving images are examples of different modes.” In the second, “semiotic modes, similarly, are shaped by both the intrinsic characteristics and potentialities of the medium and by the requirements, histories and values of societies and their cultures.”

Thus, every mode has a different modal resource, which is historically and culturally situated and which breaks it down into its parts, because “each has distinct potentials [and limitations] for meaning.” For example, breaking down writing into its modal resources would be syntactic, grammatical, lexical resources and graphic resources. Graphic resources can be broken down into font size, type, etc. These resources are not deterministic, however. In Kress’s theory, “mode is meaningful: it is shaped by and carries the ‘deep’ ontological and historical/social orientations of a society and its cultures with it into every sign. Mode names the material resources shaped in often long histories of social endeavor.” Modes shape and are shaped by the systems in which they participate. Modes may aggregate into multimodal ensembles, shaped over time into familiar cultural forms, a good example being film, which combines visual modes, modes of dramatic action and speech, music and other sounds. Multimodal work in this field includes van Leeuwen; Bateman and Schmidt; and Burn and Parker's theory of the kineikonic mode.

...
Wikipedia