Elements of Multimodal Design

Multimodal design is the art and science of creating a user interface that includes multiple touchpoints using different modalities. The concept is to produce an interface that combines these different modalities that organically fit together, such as using voice as an input mechanism and a graphical user interface as the output for the user. Instead of relying on one modality, multimodal design doesn’t exclude any type of input or output.

Input Versus Output

Let’s start with the basics: input and output modalities. Think of an input modality as any way of interacting with a system. This could be through your mouse, keyboard, or touchpad. However, there are many other options, such as speech, gestures, and physiological events. In summary, the user is providing information to a system, ideally in the most efficient and effortless way possible.

With a Voice User Interface (VUI) it’s fairly straightforward from a user’s perspective. The user wants something and simply asks for it. This is commonly referred to as the intent. The user then expects the system to come back with a relevant response, either as an action or as information.