Multimodal Design: An Overview

Design generally entails multiple kinds, or modalities, of representation and reasoning. For example, designers reason with different kinds of representations, including both imagistic (e.g., drawings, sketches, and diagrams) and propositional (e.g., function, behavior, causality, and structure). This multimodal nature of design representation and reasoning raises several issues for artificial intelligence (AI) research on design. For example, what types of knowledge are captured by various modalities of representation? What kinds of inferences are enabled and constrained by different representation modalities? How might we couple a representation in one modality with a representation in another or transform a representation in one modality to another? AI researchers have long been interested in these issues, although not necessarily in the context of design.

AI research on multimodal representations and reasoning relevant to design has generally followed several important threads. In one thread, AI research has sought to understand the various modalities in terms of the types of knowledge they capture and the inferences they enable. For example, Davis (1984) describes an early effort to declaratively represent and then reason about the structure and behavior of physical systems, and Sembugamoorthy and Chandrasekaran (1986) describe an early attempt to declaratively represent functions of physical systems and relate them to their structure via their behaviors. Both efforts focused on diagnostic problem solving. In contrast, Glasgow and Papadias (1992) present an analysis of imagistic representations and use symbolic arrays to represent spatial knowledge.

Another thread of AI research on multimodal representations and reasoning pertains to interpreting imagistic representations of a system by reasoning about its structure and behavior. For example, Stahovich, Davis, and Shrobe (1998) describe an attempt at abstracting the behaviors of a physical system from its schematic sketch. A third research thread is concerned with the coupling of reasoning across different representation modalities. For example, Funt (1980) describes an early effort in which a diagrammatic reasoner answered questions posed by a propositional problem solver, and Chandrasekaran (2006) presents a recent attempt at a multimodal cognitive architecture in which propositional and diagrammatic components cooperate to solve problems.

AI research on design per se has pursued similar threads. For example, Gero (1996) has analyzed the role of imagistic representations in creative design and describes cognitive studies of imagistic representations and reasoning in design (Gero, 1999). Gebhardt et al. (1997) describe a computer-aided design system that used both diagrammatic design cases and propositional design rules. Yaner and Goel (2006) describe an organizational schema for combining functional, causal, spatial, and diagrammatic knowledge about design cases.

The five papers selected for this Special Issue push the envelope of research on multimodal design further. The research contexts, goals, and methods of the first two papers are similar. “Modality and Representation in Analogy” by Linsey, Wood, and Markman describes a cognitive study that examines the effect of the modality of external representations on the retrieval and use of analogies in the context of biologically inspired design. “The Effect of Representation of Triggers on Design Outcomes” by Sarkar and Chakrabarti describes a cognitive study on the effects of the modality and ordering of external representations on the number and quality of designs generated by analogy in the context of biologically inspired design. Linsey et al. find that verbal annotations on external diagrams significantly improve retrieval and use of analogies, and Sarkar and Chakrabarti determine that imagistic external representations (e.g., videos) improve the quality of generated design ideas when compared with verbal (e.g., textual descriptions of function, behavior, and structure) representations. The issue of the modality of external representations is critical in building computational environments that can foster design by analogy.

“Analogical Recognition of Shape and Structure in Design Drawings” by Yaner and Goel describes a computational technique for constructing structural models from two-dimensional vector-graphics line drawings of physical systems. The technique, called compositional analogy, constructs a structural model of an input design drawing by analogical transfer of the structural model of a similar known drawing. The technique reasons about both imagistic representations (the drawings) and propositional representations (the structural model).

“A Grammar-Based Multiagent System for Dynamic Design” by Slusarczyk develops a semiformal approach to multifunctional design of spatial layouts, for example, the layout of furniture in a house. The paper addresses the design task in a multiagent framework, using a hypergraph grammar for design actions and a set grammar for design states. The technique apparently can succeed not only in placing objects in a space but also in adjusting their locations.

“A Review of Function Modeling: Approaches and Applications” by Erden, Komoto, van Beek, D’Amelio, Echavarria, and Tomiyama surveys research on functional modeling of physical systems. Although, strictly speaking, this paper does not deal with multimodal design explicitly, it is clear that functional representations and reasoning play an important role in much of multimodal design and different researchers appear to have different notions of “function” and the use of functional models in design. This paper provides a useful service by pulling together multiple threads of AI research on functional representations and reasoning in design.

These five papers were selected for this Special Issue after two rounds of reviews. In the first round all submitted papers were peer reviewed by multiple reviewers; in the second round the Guest Editors reviewed the revised manuscripts. We thank the authors and reviewers of all submissions for their hard work. We also thank Prof. David Brown, the Editor-in-Chief of AIEDAM, for his support and guidance throughout the review process. We hope that this Special Issue will lead to new research on multimodal design.