Body, Language and Mind Vol.1- Embodiment

Body, Language and Mind Volume 1: Embodiment

≥

Cognitive Linguistics Research 35.1

Editors Dirk Geeraerts Rene´ Dirven John R. Taylor Honorary editor Ronald W. Langacker

Mouton de Gruyter Berlin · New York

Body, Language and Mind Volume 1: Embodiment Edited by Tom Ziemke Jordan Zlatev Roslyn M. Frank

Mouton de Gruyter Berlin · New York

Mouton de Gruyter (formerly Mouton, The Hague) is a Division of Walter de Gruyter GmbH & Co. KG, Berlin

앝 Printed on acid-free paper 앪

which falls within the guidelines of the ANSI to ensure permanence and durability.

Library of Congress Cataloging-in-Publication Data Body, language, and mind. Volume 1, Embodiment / edited by Tom Ziemke, Jordan Zlatev, Roslyn M. Frank. p. cm. ⫺ (Cognitive linguistics research ; 35.1) Includes bibliographical references and index. ISBN 978-3-11-019327-5 (hardcover : alk. paper) 1. Language and languages ⫺ Philosophy. 2. Mind and body. 3. Semiotics. I. Ziemke, T. (Tom), 1969⫺ II. Zlatev, Jordan. III. Frank, Roslyn M. P107.B63 2007 401⫺dc22 2007028708

Bibliographic information published by the Deutsche Nationalbibliothek The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data are available in the Internet at http://dnb.d-nb.de.

ISBN 978-3-11-019327-5 ISSN 1861-4132 쑔 Copyright 2007 by Walter de Gruyter GmbH & Co. KG, D-10785 Berlin All rights reserved, including those of translation into foreign languages. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the publisher. Printed in Germany

Table of contents

List of contributors Introduction: The body eclectic Tom Ziemke and Roslyn M. Frank

VII

1

Section A: Historical roots We are live creatures: Embodiment, American Pragmatism and the cognitive organism Mark Johnson and Tim Rohrer

17

Bringing the body back to life: James Gibson's ecology of embodied agency Alan Costall

55

From the meaning of embodiment to the embodiment of meaning: A study in phenomenological semiotics Goran Sonesson

85

Embodiment and social interaction: A cognitive science perspective Jessica Lindblom and Tom Ziemke

129

Section B: Body and mind Representing actions and functional properties in conceptual spaces Peter Giirdenfors

167

From pre-representational cognition to language Takashi Ikegami and Jordan Zlatev

197

vi

Table ofcontents

Making sense of embodied cognition: Simulation theories of shared neural mechanisms for sensorimotor and cognitive processes Henrik Svensson, Jessica Lindblom and Tom Ziemke

241

Phenomenological and experimental contributions to understanding embodied experience Shaun Gallagher

271

Section C: Body, language and culture Embodiment, language, and mimesis Jordan Zlatev

297

The body in space: Dimensions of embodiment TimRohrer

339

On the biosemiotics of embodiment and our human cyborg nature Claus Emmeche

379

Embodiment and self-organization of human categories: A case study of speech Luc Steels and Bart de Boer

411

Communication as situated, embodied practice Wolff-Michael Roth

431

Index

457

List of contributors

Bart de Boer did a Master's degree in computer science at the Rijksuniversiteit Leiden (1994) and a PhD in artificial intelligence at the AI lab of the Vrije Universiteit Brusse1 (1999) under professor Luc Steels. He has worked as a postdoc in Brussels and at the University of Washington under professor Patricia Kuhl. He has also performed linguistic fieldwork in Nepal. His main research interest is in computer mode1ing the evolution of speech. He is currently working as an assistant professor in cognitive robotics at the Rijksuniversiteit Leiden. e-mail: [email protected] Alan Costall is Professor of Theoretical Psychology at the University of Portsmouth, England. His research interests are wide, but are held together by a commitment to interdiscip1inarity, and a broadly ecological or mutualist perspective. He has been increasingly involved in work on the history of modem psychology and its relations (or lack of them) to other disciplines. Recent publications include: A. Costa11 and o. Dreier, (eds.), Doing Things with Things. (London: Ashgate, 2006); A. Costa11, I. Leudar, and V. Reddy. (2006) "Failing to see the irony in 'mind-reading. '" Theory & Psychology 16(2): 163-167; Bard, K.A., M. Myowa-Yamakoshi, M. Tomonaga, M. Tanaka, A. Costa11, A., and T. Matsuzawa. (2005) "Group differences in the mutual gaze of chimpanzees (Pan Troglodytes)." Developmental Psychology 41: 616-624; Rogers, S. D., E. E. Kadar and A. Costa11, A. (2005) "Gaze patterns in the visual control of straight road driving and braking as a function of speed and expertise." Ecological Psychology 17: 19-38; Costa11, A., and I. Leudar. (2004). "Where is the 'theory' in theory of mind?" Theory and Psychology 14: 625-648; Costa11, A., M. Sinico and G. Parovel. (2003) "The concept of 'invariants' and the problem of perceptual constancy." Rivista di Estetica, n.s. 24(3), 49-53; Ost, J., and A. Costa1l. (2002) "Misremembering Bart1ett: A study in serial reproduction." British Journal ofPsychology 93: 243-255. e-mail: [email protected] Claus Emmeche is a theoretical biologist, Ph.D., Associate Professor and Head of the Center for the Philosophy of Nature and Science Studies, 10-

viii

List ofcontributors

cated at the Niels Bohr Institute, University of Copenhagen. The Faculty of Science founded the center in 1994 to explore a new and more sciencerelated way to do philosophy of nature, yet keeping a notion of science as more than natural science. Emmeche has taught courses in philosophy of biology and philosophy of science and his current research interests include biosemiotics, artificial life, ontology, organism/body/cyborg relations, and philosophy of nature. He is active in the Copenhagen biosemiotics school (cf. Reading Hoffmeyer: Rethinking Biology, with Kalevi Kull and Frederik Stjemfelt) and in developing a cluster of mandatory science studies courses for the bachelor programmes in Denmark. e-mail: [email protected] Roslyn M. Frank is Professor Emeritus in the Department of Spanish and Portuguese at the University of Iowa. She is co-editor of Cognitive Models in Language and Thought: Ideology, Metaphors and Meaning (2003); Language and Ideology, Vol. 2. Cognitive Description Approaches (2001) and has published extensively in the field of cognitive linguistics as well as in ethnoscience, most particularly in ethnomathematics and ethnoastronomy. Her research on the Basque language has taken her to Euskal Herria, the Basque Country, where she has done extensive fieldwork and given numerous seminars. In addition she has given presentations on these research topics throughout Europe. e-mail: [email protected] Shaun Gallagher is Professor and Chair of Philosophy and Cognitive Sciences at the University of Central Florida; he has been occasional Visiting Professor at the University of Copenhagen (2004-2006) and Visiting Scientist at the Medical Research Council's Cognition and Brain Sciences Unit at Cambridge University (1994). He is co-editor of the interdisciplinary journal Phenomenology and the Cognitive Sciences. His research interests include phenomenology and philosophy of mind, cognitive sciences, hermeneutics, theories of the self and personal identity. His most recent book, How the Body Shapes the Mind, is published by Oxford University Press (2005). He is co-editor of the forthcoming Does Consciousness Cause Behavior? An Investigation ofthe Nature of Volition (MIT Press, 2006). He is currently working on several projects, including a co-authored book, The Phenomenological Mind: Contemporary Issues in Philosophy ofMind and the Cognitive Sciences (Routledge, 2007). His previous books include: Hermeneutics and Education (1992) and The Inordinance of Time (1998).

List ofcontributors

ix

He has edited or co-edited volumes including: Ipseity and Alterity: Interdisciplinary Approaches to Intersubjectivity (2004),· Models of the Self (1999); Hegel, History, and Interpretation (1997). Home page: http:// pegasus.cc.ucf.edu/"-Jgallaghr e-mail: [email protected] Peter Gardenfors is professor of cognitive science at Lund University (Sweden). He leads the Ph.D. program in Cognitive Science there (LUCS). He has published numerous books and articles on decision theory, epistemology, belief revision, concept formation and the evolution of cognition (see http://www.lucs.lu.se/People/Peter.Gardenfors/bibli02000.html). The most important books are Knowledge in Flux: Modeling the Dynamics of Epistemic States (Bradford Books, MIT Press, 1988); Conceptual Spaces (Bradford Books, MIT Press, 2000); How Homo Became Sapiens: On the Evolution of Thinking (Oxford University Press, 2003); and The Dynamics of Thought (Springer, 2005). e-mail: [email protected] Takashi Ikegami earned his Ph.D in Physics (1989). He works on Artificial Life and Complex Systems by simulating computational models. His publications range from self-reproduction, ecological systems, embodied cognition to cognitive linguistics. Some of his recent articles are: Ikegami, T. (2005). "Neutral phenotypes as network keystone species." Population Ecology 47: 21-29; Iizuka, H. and T. Ikegami. (2004) "Adaptability and diversity in simulated turn-taking behavior." Artificial Life 10: 361-378; Ikegami, T., and G. Morimoto. (2003) "Chaotic itinerancy in coupled dynamical recognizers." CHAOS 13: 1133-1147; Ikegami, T. (1999). "Evolvability of machines and tapes." J. Artificial Life and Robotics 3( 4): 242245; Ikegami, T. and M. Taiji. (1998). "Structures of possible worlds in a game of players with internal models." Acta Polytechnica Scandinavica 91: 283-292. e-mail: [email protected] Mark Johnson is Professor of Philosophy and Knight Professor of Liberal Arts and Sciences at the University of Oregon. His research has focused on the philosophical implications of the role of human embodiment in meaning, conceptualization, and reasoning. He is co-author, with George Lakoff, of Metaphors We Live By (1980) and Philosophy in the Flesh (1999) and author of The Body in the Mind (1987) and Moral Imagination (1993). He

x

List ofcontributors

is currently completing a book on the aesthetic dimensions of meaning, drawing on evidence from cognitive science, phenomenology, neuroscience, and the arts that reveals the origins of meaning in felt qualities, sensorimotor patterns, and emotions. e-mail: [email protected] Jessica Lindblom is a cognitive science Ph.D. candidate working at the School of Humanities and Informatics, University of Skovde, Sweden. She previously received a master's degree in computer science (2001) and bachelor's degree in cognitive science (2000). Her main research interests are social aspects of embodied and situated cognition, and their implications to interactive technology. Some of her publications, together with her supervisor professor Tom Ziemke, are Lindblom and Ziemke (2003) "Social situatedness of natural and artificial intelligence. Vygotsky and beyond." Adaptive Behavior 11(2): 79-96, and Lindblom and Ziemke (2005) "Body-in-motion: broadening the social mind." In: Bruno G. Bara, Lawrence Barsalou and Monica Bucciarelli (eds), Proceedings of the XXVII Annual Conference ofthe Cognitive Science Society, 1284-1289. Mahwah, NJ: Lawrence Erlbaum. e-mail: [email protected] Tim Rohrer has published extensively on metaphor and embodiment in diverse disciplines for over fifteen years. His work has ranged from experimental cognitive neuroscience to information technology policy and from the politics of conflict resolution to the philosophy of language. In addition to the lines of investigation reflected in this volume, he is also researching how metaphors shape attitudes toward wildfire mitigation in the western United States. He is perhaps best known as the founder and maintainer of the Center for the Cognitive Science of Metaphor Online, a collection of formative articles in metaphor theory and cognitive semantics (http://zakros.ucsd.edu;'--.Jtrohrer/metaphor/metaphor.htm). He holds a PhD in philosophy from the University of Oregon and has been a Thomas J. Watson scholar, a Fulbright researcher at the Center for Semiotic Research in Aarhus, Denmark, and a NllI fellow at the Institute for Neural Computation at the University of California at San Diego. At present he directs the Colorado Advanced Research Institute in Boulder, Colorado. e-mail: [email protected]

List ofcontributors

xi

Wolff-Michael Roth is Lansdowne Professor of Applied Cognitive Science at the University of Victoria, Canada. His research focuses on cultural-historical, linguistic, and embodied aspects of scientific and mathematical cognition and communication from elementary school to professional practice, including, among others, studies of scientists, technicians, and environmentalists at their work sites. The work is published in leading journals of linguistics, social studies of science, sociology, learning sciences, and education and various subfields of education (curriculum, mathematics education, science education). His recent books include Toward an Anthropology of Science (Kluwer, 2003), Rethinking Scientific Literacy (Routledge, 2004, with A. C. Barton), Talking Science (Rowman and Littlefield, 2005), and Doing Qualitative Research: Praxis of Method (SensePublishers, 2005). e-mail: [email protected] Goran Sonesson is Professor of semiotics and Director of the Department of Semiotics at Lund University. He holds a doctorate in general linguistics from Lund University, as well as a doctorate in semiotics from the Ecole des Hautes Etudes en Sciences Sociales, Paris. Between 1978 and 1983, he was involved in Paris with the semiotics of gesture, and then worked on Mayan language and culture in Mexico, after which he has occupied different research positions in semiotics in Lund. Sonesson's main work is the monograph Pictorial Concepts (Lund: Lund University Press 1989), which is a critical survey of different contributions to pictorial semiotics, reviewed in the context of findings in perceptual psychology and cognitive science. An important part of the book is devoted to a critical assessment of the theories of iconicity presented by, among others, Goodman and Eco. This work has subsequently been extended in numerous articles, published in Semiotica, RSSL Zeitschrift fur Semiotik, VISID, Degres, Sign System Studies, Current Anthropology, etc. His most recent publications are concerned with bringing a semiotic perspective to the study of evolution. e-mail: [email protected] Luc Steels is a professor of Computer Science at the Vrije Universiteit Brussel (VUB). He graduated in linguistics at the University of Antwerp and in computer science at the Massachusetts Institute of Technology, working in the MIT AI Laboratory. After that he worked in the domain of geophysical measurement interpretation as a project leader for geological expert systems at Schlumberger. In 1983 he founded the VUB Artificial

xii

List ofcontributors

Intelligence Laboratory, which he still directs this day. He was cofounder and chairman (from 1990 until 1995) of the VUB Computer Science Department (Faculty of Sciences) and also founder and director of the Sony Computer Science Laboratories in Paris. His scientific research interests cover the whole field of artificial intelligence, including natural language, vision, robot behavior, learning, cognitive architecture, and knowledge representation. His publications can be found in major AI/cogsci journals such as Behavioral and Brain Sciences, Trends in Cognitive Science, Artificial Intelligence Journal, etc. He also has edited a dozen books. At the moment his research focus in on fundamental research into the origins of language and meaning. e-mail: [email protected] Henrik Svensson is currently pursuing a Ph.D. in cognitive science at the School of Humanities and Informatics, University of Skovde, supervised by professor Tom Ziemke. He received his B.S. degree (2001) and M.Sc. degree (2002) from the University of Skovde. His main research interest concerns the relation between agent-environment interaction and higherlevel cognition. Some of his recent publications are: Svensson, H. and T. Ziemke (2005) "Embodied representation: What are the issues", in: B Bara, L. Barsalou, and M. Buccarelli (eds.), Proceedings ofthe 27th Annual Meeting of the Cognitive Science Society, 2116-2121. Mahwah, NI: Lawrence Erlbaum; and Svensson and Ziemke (2004) "Making sense of embodiment", In: K. Forbus, D. Gentner and T. Regier (eds.), Proceedings of the 26th Annual Conference of the Cognitive Science Society, 1309-1314. Mahwah, NI: Lawrence Erlbaum. e-mail: [email protected] Tom Ziemke is Professor of Cognitive Science in the School of Humanities and Informatics at the University of Skovde, Sweden. His research is mainly concerned with embodied and distributed cognition, i.e. theories and models of how cognition is shaped by the living body and its interaction with the material and social environment. He is coordinator of a largescale European project on robotic models of embodied cognition, called "Integrating Cognition, Emotion and Autonomy" (www.his.se/icea). and member of the executive committee of euCognition - The European Networkfor the Advancement ofArtificial Cognitive Systems. He is also asso-

List ofcontributors

xiii

ciate editor of the journals New Ideas in Psychology and Connection Science. e-mail: [email protected] Jordan Zlatev is Associate Professor at the Centre for Languages and Literature, Lund University, Sweden. His PhD thesis (Stockholm University, 1997) is the monograph Situated Embodiment: Studies in the Emergence of Spatial Meaning, in which he formulates a synthetic socialcognitive framework for the study of language, and applies this to the typology and acquisition of spatial semantics. He is the co-founder of the annual international workshop series Epigenetic Robotics: Modelling Cognitive Development on Robotic Systems, first held in Lund, 2001 and the bi-annual conference Language, Culture and Mind, first held in Portsmouth 2004. Zlatev collaborates extensively with semioticians, cognitive scientists and philosophers within the project Language, Gestures and Pictures in Semiotic Development and more recently within the highly interdisciplinary EU-project Stages in the Evolution and Development of Sign Use (SEDSU) (www.sol.lu.se/sedsu). His own work concentrates on differences in primary intersubjectivity between great apes and humans, and on a cross-cultural study of the ontogeny of gestural communication. The key theoretical concept for his current work is that of bodily mimesis, understood (following Donald 1991) as the conscious use of the body for representational means. Zlatev has published on these topics extensively over the past years in refereed journals and books, and is currently working on the monograph Bodily Mimesis and the Grounding ofLanguage and coediting the book The Shared Mind: Perspectives on Intersubjectivity. e-mail: [email protected]

Introduction: The body eclectic

Tom Ziemke and Roslyn M Frank

1.

Background

This is the first volume of a two-volume set with the title Body, Language and Mind. While this volume focuses on the concept of embodiment, i.e. the bodily and sensorimotor basis of phenomena such as meaning, mind, cognition and language, the second volume addresses social situatedness, i.e. the ways in which individual minds and cognitive processes are shaped by their interaction with sociocultural structures and practices. Naturally, the volumes overlap significantly, and in fact they have both to some degree emerged out of a one-day theme session on embodiment held at the 8th International Cognitive Linguistics Conference in Logrofio, Spain, in the summer of 2003. Some of the contributors to these volumes also participated in the original theme session, whereas others have been invited later to complement the range of perspectives represented. The concept of embodiment has received a great deal of attention in the cognitive sciences during the last twenty years, and as a result terms like embodied mind, embodied action, embodied cognition are now commonly used, often in juxtaposition to concepts like situated action (Suchman 1987), situated cognition (e.g. Clancey 1997), distributed cognition (Hutchins 1995) or the extended mind hypothesis (Clark and Chalmers 1998). In fact, by the 1990s several authors had already declared embodied cognitive science, which is often taken to more or less include all of these concepts, to be a new paradigm in cognitive science (e.g. Varela, Thompson and Rosch 1991; Clark 1997, 1999; Pfeifer and Scheier 1999). All of this might give you the impression: (a) that there is a clear notion of what embodiment is, and subsequently, a consensus concerning in what sense cognitive processes (or perhaps certain types of cognition) are embodied, and (b) that embodied cognitive science in fact is a theoretical framework that is more or less established and agreed upon by researchers working in the field. Somewhat surprisingly, however, neither (a) nor (b) are actually true.

2

Tom Ziemke and Roslyn M. Frank

Although it is by now widely agreed that cognition is embodied, in the sense that it is shaped by the body and sensorimotor interaction with the environment, it is less clear exactly what this means (cf. e.g., Chrisley and Ziemke 2003; Clark 1999; Wilson 2002; Ziemke 2001a, 2003, 2007). Is it the physical, the biological, the animate, the phenomenal (experienced), or the social body that shapes cognition, or perhaps all of these? And, exactly how does the body shape cognition; is it, for example, only involved in actual sensorimotor interaction with the environment, i.e. in the grounding of mental representations in the traditional sense (cf. e.g. Harnad 1990; Ziemke 1999), or does its influence go further, i.e. is the body also crucially involved in thought, language, and other supposedly abstract activities, such as mathematics (cf. e.g. Lakoff and Nufiez 1999)? The following brief overview provides several useful distinctions in conceptions of embodiment that might help to clarify differences in theoretical frameworks and commitments in the field that sometimes remain hidden under a superficial agreement on 'embodiment'. First, we have Nufiez (1999) who distinguished between trivial, material, and full embodiment. Trivial embodiment simply is the view that "cognition and the mind are directly related to the biological structures and processes that sustain them". Obviously, this is not a particularly radical claim, and consequently few cognitive scientists would reject it (dualist philosophers of consciousness, on the other hand, might). According to Nufiez, this view further "holds not only that in order to think, speak, perceive, and feel, we need a brain - a properly functioning brain in a body - but also that in order to genuinely understand cognition and the mind, one can't ignore how the nervous system works". Material embodiment makes a stronger claim, but it only concerns the interaction of internal cognitive processes with the environment, i.e. the issue of grounding, and thus considers reference to the body to be required solely for accounts of low-level sensorimotor processes. In Nufiez's terms: "First, it sees cognition as a decentralized phenomenon, and second it takes into account the constraints imposed by the complexity of real-time bodily interactions performed by an agent in a real environment". Full embodiment, finally, is the view that the body is involved in all forms of human cognition, including seemingly abstract activities, such as language or mathematical cognition (e.g. Lakoff and Nufiez 1999). In Nufiez's own words: Full embodiment explicitly develops a paradigm to explain the objects created by the human mind themselves (i.e., concepts, ideas, explanations, forms of logic, theories) in terms of the non-arbitrary bodily-experiences

Introduction: The body eclectic

3

sustained by the peculiarities of brains and bodies. An important feature of this view is that the very objects created by human conceptual structures and understanding (including scientific understanding) are not seen as existing in an transcendental realm, but as being brought forth through specific human bodily grounded processes.

In a similar vein, Clark (1999) distinguished between the positions of weak embodiment and radical embodiment. According to the fonner, traditional cognitive science can roughly remain the same; i.e. theories are merely constrained, but not essentially changed by embodiment. This is similar to Nufiez's view of material embodiment. The position of radical embodiment, on the other hand, largely compatible with Nufiez's notion of full embodiment, is, as Clark fonnulated it, "radically altering the subject matter and theoretical framework of cognitive science". More recently, Wilson (2002) distinguished between six views of embodied cognition, of which only the last one requires full or radical embodiment whereas the first five might be considered variations or aspects of material embodiment: (1) cognition is situated, i.e. it occurs "in the context of task-relevant inputs and outputs", (2) cognition is timepressured, (3) cognition is for the control of action, (4) we off-load cognitive work onto the environment, e.g. through epistemic actions (Kirsh and Maglio, 1994), i.e. manipulation of the environment 'in the world', rather than 'in the head', (5) the environment is actually part of the cognitive system, e.g. according to Clark and Chalmers' (1998) notion of the 'extended mind', and (6) 'off-line' cognition is body-based, which according to Wilson is the "most powerful claim" (cf. Svensson, Lindblom and Ziemke this volume). Complementary distinctions and classifications are proposed in several of the contributions to this volume. Emmeche (this volume), for example, points out that different disciplines have different perspectives on the body. He therefore distinguishes between physical embodiment (physics), organismic embodiment (biology), animate embodiment (zoology), and anthropic embodiment (anthropology, sociology). However, Emmeche also points out: "The point is not the exact number of levels (these are contingent upon a historically relative state of science) but the fact that irreducible levels do exist". Rohrer (this volume) takes a different approach and catalogues what he considers twelve important senses or dimensions of embodiment in current research on the topic: philosophy, socio-cultural situation, phenomenology, perspective, development, evolution, the cogni-

4

Tom Ziemke and Roslyn M Frank

tive unconscious, neurophysiology, neurocomputational modelling, morphology, directionality ofmetaphor, and grounding. It should be noted that, naturally, not all of these views, types or dimensions of embodiment are equally relevant for all disciplines and perspectives. For example, the question of whether or not an embodied mind actually needs to be physical or biological might not be a meaningful question to a neuroscientist, given that real brains and bodies obviously are both physical and biological. However, for the philosopher wrestling with dualist or functionalist conceptions of mind, or for the artificial intelligence researcher trying to build, or at least model, minds, these are still burning questions (cf. e.g. Ziemke 2001a, 2007; Zlatev 2001, 2003; Johnson and Rohrer this volume; Lindblom and Ziemke this volume). Similarly, the phenomenal experience of the lived body is perhaps not the primary interest for the cognitive linguist interested in syntax or metaphor, but it certainly is relevant to many other aspects of language and mind (cf. e.g. Sonesson this volume; Gallagher this volume; Zlatev this volume; Roth this volume). To some degree the diversity of perspectives on embodied cognition can of course be attributed to the inter- or multidisciplinary nature of cognitive science as such. After all, it is not surprising that, for example, linguists, neuroscientists and artificial intelligence (AI) researchers would be interested in different aspects of embodiment, for the same reasons that they offer different, but hopefully complementary perspectives on the study of mind and language. True complementarity, however, rather than mere coexistence of different disciplines and perspectives, would seem to require some kind of common framework and terminology. For traditional cognitive science, the common ground was provided by theories of functionalism, computationalism and representationalism, the basic assumptions of which were summarized by Gardner (1987: 6) as follows: First of all, there is the belief that, in talking about human cognitive activities, it is necessary to speak about mental representations and to posit a level of analysis wholly separate from the biological or neurological, on the one hand, and the sociological or cultural, on the other.

Consequently, the complementarity and division of labor between different disciplines was, perhaps, relatively clear in the old, pre-embodiment days when everybody agreed that thought was the operation of computational processes on mental, presumably symbolic representations. Linguists were mostly concerned with language as a symbol system, without much interest in its biological implementation, while neuroscientists tended to be inter-

Introduction: The body eclectic

5

ested in how and where in the brain the relevant computations and representations were implemented, and AI researchers were striving to synthesize the very same computations and representations in artificial implementations. Apart from the fact that the above picture of traditional, computationalist cognitive science is of course a bit too rosy (the exact nature of those mental representations, for example, caused, and still causes, considerable debate), we have to admit that there is no equivalent consensus yet in embodied cognitive science. We could say that is because, as a discipline or paradigm, embodied cognitive science is still relatively young, but then again it is 20-30 years old by now, depending on exactly how you situate its origins in time. Moreover, one could rightly point out that this diversity of perspectives is due to the expanding nature of the community of researchers concerned with this approach. In short, since the embodied cognition perspective is still growing and gaining ground, it might be too early to hope for a convergence of perspectives, indeed, at this stage such a convergence of opinions might be premature. However, the current situation might also provoke comparisons with soap bubbles that are destined to burst sooner or later if there is nothing under the surface to hold them together. One could also say that things were simpler for traditional cognitive science, because that approach was able to define itself from the start both with respect to what it rejected, behaviorism, and with respect to a common vision, the computer metaphor for mind. But this, in turn, raises the question of exactly what the common vision might be in the case of embodied cognitive science (as well as its common unifying metaphor). Some people would say that simulation theories (cf. Svensson, Lindblom and Ziemke this volume) are what constitute the common vision or framework in embodied cognitive science - very roughly speaking, the idea that the same neural mechanisms are used for both sensorimotor interaction and abstract thought. Nonetheless these ideas certainly still need to be worked out in ~ much more detail. Whichever way you look at it, although embodied cognitive science undeniably is an exciting and vibrant area of research, it is hard not to agree with Wilson's (2002: 625) assessment of what might be perceived as a conceptual muddle: While this general approach is enj oying increasingly broad support, there is in fact a great deal of diversity in the claims involved and the degree of

6

Tom Ziemke and Roslyn M. Frank

controversy they attract. If the term "embodied cognition" is to retain meaningful use, we need to disentangle and evaluate these diverse claims.

In fact, that is exactly, what we hope to offer readers with the collection of

chapters making up this volume, namely, a means to disentangle and evaluate the different perspectives, and not least to understand which of them are actually competing or rather complementary. The papers selected for inclusion in this first volume have been written by some of the leading cognitive scientists, cognitive linguists, psychologists, philosophers, AI researchers, semioticians, and phenomenologists working on embodied cognition today. That means, the perspective of cognitive linguistics is here put into the context of a number of related disciplines. The abovementioned second volume, on the other hand, focuses on social situatedness and contains more directly linguistically oriented contributions. Admittedly, we cannot promise that after reading all of the contributions to this volume you will feel much wiser in terms of exactly which notion of embodiment is the "correct" one. But you will have learned a great deal about the most important current perspectives on embodiment and their historical backgrounds as well as the overlaps and controversies existing between them. That, we believe, is the best deal anybody can offer at the moment when it comes to understanding the multidisciplinarity of modem embodiment research. In that sense, this book can be read as an update to some of the "classical", highly influential books on embodiment that appeared in the 1990s, such as Varela, Thompson and Rosch's (1991) The Embodied Mind, Clark's (1997) Being There, or Lakoff and Johnson's (1999) Philosophy in the Flesh, as well as other multi-author collections on the topic (e.g. Nunez and Freeman 1999; Ziemke 2002; Robbins and Aydede in press). Other books provide more detailed accounts on specific topics, such as Damasio's (1994, 1999,2003) books on the role of the body in emotion and consciousness, Goldin-Meadow's (2003) book on gesture as embodied thought, Pfeifer's books on embodied AI (Pfeifer and Scheier 1999; Pfeifer and Bongard 2006), or Maturana and Varela's (1980, 1987) classical books on the biology of cognition. There are also several recent books that do a very good job at integrating some of the perspectives, such as Gallagher's (2005) How the Body Shapes the Mind, Gibbs' (2006) Embodiment and Cognitive Science, and Thompson's (2007) Mind in Life. However, we believe that these works need to be complemented with a broad spectrum of perspectives, from different disciplinary and historical backgrounds, as collected in this multi-author volume.

Introduction: The body eclectic

2.

7

Overview of the contributions to this volume

This volume is divided into three parts or sections, each of which consists of four or five chapters. The first section consists of four chapters that explore in detail the historical roots underlying current discussions of embodiment in cognitive science and linguistics. First, Johnson and Rohrer contrast the dualistic view of mind and body as two ontologically different entities, connected through internal representations of external reality, with what they call embodied realism, according to which cognition and language must be understood as arising from organic processes. They trace the rejection of mind-body dualism from early American Pragmatists, such as James and Dewey, forward to recent embodied cognitive science, which they argue needs to be pragmatically-centered. Costall's chapter points out that the move from mechanistic behaviorism to cognitive psychology and the computer metaphor for mind, through its focus on internal information processing, in fact simply maintained the view of the mechanical body as a mere shell or container for cognitive processes. As an alternative, he elaborates on the contributions that Gibson's "ecology of embodied agency" and his concept of affordances can make to our understanding of embodiment and "being in the world". Sonesson's chapter addresses the contributions that phenomenology and semiotics, with their focus on consciousness and meaning, can provide to current discussions of embodiment in cognitive linguistics, biosemiotics, and other disciplines, discussions which, in his view, tend to ignore relevant distinctions between different types and forms of meaning. Furthermore, building on Piaget's concept of differentiation, he suggests a developmental sequence going from schemas to signs and external representations. Lindblom and Ziemke, in the last chapter of the "historical roots" section, point out that, despite much recent emphasis on the bodily and social basis of cognition, the role of embodiment in social interaction is still relatively poorly understood. They therefore trace the role of biological and sociocultural factors in explanations of cognition from Darwin to modem cognitive science, and discuss further steps and conceptual clarifications that will be required in the development of a science of embodied cognition and social interaction.

8

Tom Ziemke and Roslyn M. Frank

The second section of this volume consists of four chapters that address from different perspectives the relation between body and mind as well as the actual mechanisms underlying embodied cognition. Gardenfors elaborates his theory of conceptual spaces, i.e. representational spaces built up from "quality dimensions". While earlier work on conceptual spaces focused on perceptual concepts, here it is shown how this analysis can be extended to actions as spatiotemporal patterns of forces and functional concepts as regions of affordances in an action space. The discussion extends work in cognitive linguistics/semantics by Johnson and Talmy on "force dynamics" and discusses the transition from an embodied, first-person notion of force to a third-person perspective. Ikegami and Zlatev argue that there is a tendency in embodiment theories to "resolve" oppositions between, for example, interaction and representation, or procedural and declarative lmowledge, by simply ignoring the differences, and thus, in effect, typically eliminating the more "disembodied" side of such oppositions. They therefore explore how mechanisms of pre-representational cognition such as dynamical categories, internal meaning spaces, and synaesthesia can play a role in the grounding of concepts, representations, and language. Drawing on experimental evidence from neuroscience, psychology and other disciplines, Svensson, Lindblom and Ziemke propose that a key to understanding the embodiment of cognition is the "sharing" of neural mechanisms between sensorimotor processes and higher-level cognitive processes. The latter are argued to be embodied in the sense that they make use of partial simulations or emulations of sensorimotor processes through the re-activation of neural circuitry also active in bodily perception and action. Gallagher, in the last chapter of the "body and mind" section, clarifies the distinction between body image and body schema from a phenomenological perspective, and then discusses how this distinction can contribute to our understanding of embodied experience and intersubjectivity. Relating these concepts to empirical research concerning issues such as intentional action, imitation and mirror neurons, he connects the phenomenological insights of Husserl and Merleau-Ponty to the most recent neuroscience research on social cognition. The third and final section contains five chapters that address the relation between body, language and culture from a range of different perspectives.

Introduction: The body eclectic

9

Zlatev argues that much recent work on embodiment tends to undervalue concepts such as representation, consciousness and convention/normativity, which he claims makes it difficult for current "embodiment theories" to account for human language and cognition. To illustrate these difficulties, he critically examines the approach presented in Lakoff and Johnson's (1999) highly influential book Philosophy in the Flesh, and develops his own alternative notion of mimetic schema as a mediator between the individual human body and collective language. Rohrer reviews the variety of standard usages that the term "embodiment" currently has in cognitive science and contrasts notions of embodiment and experientialism at different levels of investigation. With the goal of developing a broad-based theoretical framework for embodiment he examines examples of these usages and in the process brings into view related research issues such as mental imagery, mental rotation, spatial language and conceptual metaphor across several levels of investigation, with a focus on whether and how different conceptualizations can form a cohesive research program. Emmeche presents biosemiotics as a new perspective on living systems inspired by von Uexklill and Peirce. He suggests necessary distinctions between physical, biological, animate, phenomenal and social body, and develops a Neo-Aristotelian evolutionary emergentist perspective which he argues to be necessary in order to coherently account for these dimensions of embodiment. Furthermore, he characterizes humans as techno-culturally embedded beings within a space of meanings that are not only symbolic, but also socially empowered by different kinds of sociocultural systems. From the perspective of artificial intelligence, Steels and de Boer consider the kinds of categories that have been found to be involved in human behavior, and how they are shaped by embodiment as well as the collective dynamics generated by social interactions. Using a computational case study developed for speech sounds, they show how a group of autonomous agents equipped with a sufficiently realistic perceptual and auditory apparatus can arrive at a shared repertoire of vowels that exhibits the same universal trends as found in human vowel systems, although not biased by innate a priori categories. Roth, finally, in the last chapter of the volume, argues that many researchers are pre-occupied with written language as the main paradigm of communication. A large part of human communication, however, involves participants in face-to-face conversation, attending to semiotic resources such as gestures, body positions, and material structures in the environ-

10

Tom Ziemke and Roslyn M. Frank

ment. He provides a dialectical account of human activity and analyses an episode from a science classroom to exemplify the central role of the body in communication and to illustrate his view of communication as materially and socially situated and embodied practice.

3.

Conclusion

Embodied cognitive science is still a relatively young approach to the study of mind and language. Critics of the approach might argue that it has so far failed to produce a coherent theoretical framework that integrates and unifies different approaches and disciplines. However, one could also argue that it was in fact the premature convergence of traditional cognitive science on functionalist/computationalist and representationalist/symbolic theories of mind that for a long time disconnected cognitive science from many of its historical roots which are now being rediscovered (cf. e.g. Lindblom and Ziemke this volume; Johnson and Rohrer this volume). Hence, we are convinced that a fuller understanding of embodied cognition will emerge from the type of broad multidisciplinary interaction that has led to the production of this multi-author volume, rather than from the more narrow insights that any single discipline or approach could produce on its own. At the same time we are also convinced that the road towards a unified theoretical framework in embodied cognitive science is neither short nor straightforward. Indeed, there still is much work ahead for theorists and modelers of embodied cognition. But, as they say, even the longest journey starts with the first steps. And one of the most important steps in the development of a truly interdisciplinary science of embodied cognition certainly is to acquire a clear comprehensive understanding of the different perspectives that characterize this field today.

References Chrisley, Ron and Tom Ziemke 2003 Embodiment. In: Encyclopedia of Cognitive Science, 1102-1108. London: Macmillan. Clancey, William J. 1997 Situated Cognition: On Human Knowledge and Computer Representations. New York: Cambridge University Press.

Introduction: The body eclectic

11

Clark, Andy Being There - Putting Brain, Body and World Together Again. 1997 Cambridge, MA: MIT Press. An embodied cognitive science? Trends in Cognitive Sciences 3 (9): 1999 345-351. Clark, Andy and David Chalmers 1998 The extended mind. Analysis 58 (1): 7-19. Damasio, Antonio R. 1994 Descartes' Error: Emotion, Reason, and the Human Brain. New York: Avon Books. The Feeling of What Happens: Body and Emotion in the Making of 1999 Consciousness. New York: Harcourt Inc. Looking for Spinoza: Joy, Sorrow and the Feeling Brain. New York: 2003 Harcourt Inc. Gallagher, Shaun 2005 How the Body Shapes the Mind. Oxford: Oxford University Press. Gardner, Howard 1987 The Mind's New Science. New York: Basic Books. Gibbs, Ray Embodiment and Cognitive Science. Cambridge: Cambridge Univer2006 sity Press. Goldin-Meadow, Susan 2003 Hearing Gesture: How Our Hands Help Us Think. Cambridge, MA: Harvard University Press. Hamad, Stevan 1990 The Symbol Grounding Problem. Physica D, 42: 335-346. Hutchins, Edwin 1995 Cognition in the Wild. Cambridge, MA: MIT Press. Kirsh, David and Paul Maglio 1994 On Distinguishing Epistemic from Pragmatic Action. Cognitive Science, 18 (4): 513-549. Lakoff, George and Mark Johnson 1999 Philosophy in the Flesh: The Embodied Mind and its Challenges to Western Thought. New York: Basic Books. Lakoff, George and Rafael Nunez 1999 Where Mathematics Comes From. New York: Basic Books. Maturana, Humberto and Francisco J. Varela 1980 Autopoesis and Cognition: The Realization of the Living. Dordrecht, The Netherlands: D. Reidel Publishing. 1987 The Tree of Knowledge - The Biological Roots of Human Understanding. Boston: Shambalaya.

12

Tom Ziemke and Roslyn M. Frank

Nuiiez, Rafael 1999 Could the Future Taste Purple? Reclaiming Mind, Body and Cognition. Journal ofConsciousness Studies, 6 (11-12): 41-60. Nuiiez, Rafael and WaIter J. Freeman (eds.) 1999 Reclaiming Cognition - the Primacy of Action, Intention and Emotion. Exeter: Imprint Academic. Pfeifer, Rolf and Christian Scheier 1999 Understanding Intelligence. Cambridge, MA: MIT Press. Pfeifer, Rolf and Josh Bongard 2006 How the Body Shapes the Way We Think: A New View of Intelligence. Cambridge, MA: MIT Press. Robbins, Philip and Murat Aydede in press Cambridge Handbook on Situated Cognition. Cambridge, U.K.: University of Cambridge Press. Suchman, Lucy 1987 Situated Action. Cambridge, MA: MIT Press. Svensson, Henrik, Jessica Lindblom and Tom Ziemke this vo!. Making sense of embodied cognition: simulation theories of shared neural mechanisms for sensorimotor and cognitive processes. Thompson, Evan 2007 Mind in Life: Biology, Phenomenology, and the Sciences of Mind. Cambridge, MA: Harvard University Press. Varela, Francisco J., Evan Thompson and Eleanor Rosch 1991 The Embodied Mind: Cognitive Science and Human Experience. Cambridge, MA: MIT Press. Wilson, Margaret 2002 Six views of embodied cognition. Psychonomic Bulletin and Review 9 (4): 625-636. Ziemke, Tom 1999 Rethinking Grounding. In: Alexander Riegler, Markus Peschl and Astridvon Stein (eds.), Understanding Representation in the Cognitive Sciences: 177-190. Plenum Press: New York. 2001 a Are robots embodied? In: Christian Balkenius et a!. (eds), Proceedings ofthe First International Workshop on Epigenetic Robotics, 7583. Lund, Sweden: Lund University Cognitive Studies, vo!. 85. 2001 b The construction of 'reality' in the robot. Foundations of Science 6 (1): 163-233. 2003 What's that thing called embodiment? In: Richard AIterman and David Kirsh (eds.), Proceedings of the 25 th Annual Meeting of the Cognitive Science Society, 1305-1310. Mahwah, NJ: Lawrence Erlbaum.

Introduction: The body eclectic 2007

13

What's life got to do with it? In: Antonio Chella and Riccardo Manzotti (eds.), Artificial Consciousness: 48-66. Exeter: Imprint Academic. Ziemke, Tom (ed.) 2002 Special issue on situated and embodied cognition. Journal of Cognitive Systems Research 3 (3). Zlatev, Jordan 1997 Situated Embodiment. Studies in the Emergence ofSpatial Meaning. Stockholm: Gotab Press. 2001 The epigenesis of meaning in human beings, and possibly in robots. Minds and Machines, 11 (2): 155-195. 2003 Meaning = Life (+ Culture). An outline of a unified biocultural theory of meaning. Evolution ofCommunication 4 (2): 253-296.

Section A Historical roots

We are live creatures: Embodiment, American Pragmatism and the cognitive organism Mark Johnson and Tim Rohrer

Abstract The philosophical tradition mistakenly asks how the inside (i.e., thoughts, ideas, concepts) can represent the outside (i.e., the world). This trap is a consequence of the view that mind and body must be two ontologically different entities. On this view the problem of meaning is to explain how disembodied "internal" ideas can represent "external" physical objects and events. Several centuries have shown that given a radical mind-body dichotomy, there is no way to bridge the gap between the inner and the outer. When "mind" and "body" are regarded as two fundamentally different kinds, no third mediating thing can exist that possesses both the metaphysical character of inner, mental things and simultaneously possesses the character of the outer, physical things. Embodied Realism, in contrast to Representationalist theories, rejects the notion that mind and body are two ontologically distinct kinds, and it therefore rejects the attendant view that cognition and language are based on symbolic representations inside the mind of an organism that refer to some physical thing in an outside world. Instead, the terms "body" and "mind" are simply convenient shorthand ways of identifying aspects of ongoing organism-environment interactions - and so cognition and language must be understood as arising from organic processes. We trace the rejection of this mind-body dualism from the philosopher-psychologists known as the early American Pragmatists (James and Dewey) forward through recent cognitive science (such as Varela, Maturana, Edelman, Hutchins, Lakoff, Johnson, Brooks). We argue that embodied realism requires a radical reevaluation of the classical dualistic metaphysics and epistemology - especially the classical Representationalist theory of mind - and we conclude by investigating the implications for future investigations for a new, pragmatically-centered cognitive science. Keywords: cognitive linguistics, cognitive science, embodiment, image schema, metaphor, neurobiology, pragmatism, Representationalism, semantics.

18

1.

Mark Johnson and Tim Rohrer

Introduction: What difference does embodied realism make?

When a young child crawls toward the fire in the hearth and a mother snatches up the child before the child gets burned, is that cognition? When a team of British mathematicians decodes enemy ciphers during wartime, is that cognition? When ants carrying food back to their nest lay down chemical signals and thereby mark trails to a food source, is that cognition? Note the commonalities among these situations. In each case the body (both individual and social) is in peril. First, the well-being and continued successful functioning of the organism is at risk. Action must be undertaken to ensure the continued flourishing of the living, physical, individual body of the organism. To survive and flourish, the organism must make adjustments in its way of acting, both within its current environment and in its relations with other creatures. The child must be snatched from the imminent danger of the flames, the mathematicians desperately work to prevent their country from being overrun by the enemy, and the ants must find food and bring it back to the queen in order for the colony to survive. Second, note that in each case the cognition is social, composed of multiple organisms co-operating in response to current or anticipated problems posed by the environment. That environment is not merely physical but also includes the social "body" - whether the family, the nation or the ant colony - whose survival and flourishing is at risk. And finally, note that each of these situations have been taken by theorists as emblematic of cognition par excellance (Dewey 1925; Hodges 1983: 160-241; Deneubourg et al. 1983; Brooks and Flynn 1989). The importance of embodiment in cognition is now widely appreciated in the cognitive sciences, yet there remains considerable debate as to what the term "embodiment" actually means (Rohrer 2001a; in press; Ziemke 2003; Anderson 2003). Is "the body" merely a physical, causally determined entity? Is it a set of organic processes? Is it a felt experience of sensations and movement? Is it the individual physical body, or does it include the social networks such as families without which it would cease to exist? Or is the body a socially and culturally constructed artifact? In this chapter, we argue that each of these views contributes something important to an adequate theory of embodied cognition, and that a proper understanding of embodiment can be found within the philosophical context first elaborated in early American Pragmatism in the works of thinkers such as William James and John Dewey. As we see it, embodiment theory inherits several key tenets of how these Pragmatist philosophers viewed cognition:

We are live creatures

(1) (2) (3) (4)

(5)

19

Embodied cognition is the result of the evolutionary processes of variation, change and selection. Embodied cognition is situated within a dynamic ongoing organismenvironment relationship. Embodied cognition is problem-centered, and it operates relative to the needs, interests, and values of organisms. Embodied cognition is not concerned with finding some allegedly perfect solution to a problem, but one that works well enough relative to the current situation. Embodied cognition is often social and carried out cooperatively by more than one individual organism.

Note that the Pragmatists advance a radically different view of cognition than the one we are most familiar with from "classical" cognitive science, where it is assumed that cognition consists of the application of universal logical rules that govern the manipulation of "internal" mental symbols, symbols that are supposedly capable of representing states of affairs in the "external" world. Fodor summarizes this theory as follows: What I am selling is the Representational Theory of Mind ... At the heart of the theory is the postulation of a language of thought: an infmite set of 'mental representations' which function both as the immediate objects of propositional attitudes and as the domains of mental processes. (Fodor 1987: 16-17)

These internal representations in the "language of thought" acquire their meaning by being "about" - or referring to - states of affairs in the external world. Fodor aclmowledges that his Representationalist theory of meaning requires "a theory that articulates, in nonsemantic and nonintentional terms, sufficient conditions for one bit of the world to be about (to express, represent, or be true of) another bit" (Fodor 1987: 98). Typically the first "bit" would be a symbol in the internal language of thought while the second "bit" that it represents might be either some thing or event in the external world or else a brain state underlying a conception of some fictive entity or scene. The internal/external split that underlies this view presupposes that cognition could be detached from the nature and functioning of specific bodily organisms, from the environments they inhabit and from the problems that provoke cognition. Given this view, it would follow that cognition could take place in any number of suitable media, such as a human brain or a machine. This theoretical viewpoint, functionalism, was instru-

20

Mark Johnson and Tim Rohrer

mental in the developing the first electronic calculating machines and general-purpose computers. In fact, these machines were originally developed by the British military to reduce the tedious workload of military mathematicians (or human "computers" - in the sense of humans who compute). But this thought experiment did not end merely with offloading the tedium of calculation onto electronic machines. From its original conception in the work of Alan Turing (1937), the idea of a universal computing machine became the metaphor of choice for future models of the brain. For example in Newell and Simon's (1976) conception of the brain as a physical symbol system, they consider the human brain to be just a specific instance of a Turing-style universal machine. In short, for classical cognitive science cognition is defined narrowly as mathematical and logical computation with intrinsically meaningless internal symbols that can supposedly be placed in relation to aspects of the external world. The Pragmatist challenge to classical cognitive science should come as no surprise, since one of the Pragmatists' chief targets was the tendency within the philosophical tradition to assume that what demarcates "rational" humans from "lower" animals is the supposedly unique ability of humans to engage in symbolic representation between internal thoughts/ language and the external world. The remedy offered by the Pragmatists is based on their view that cognition is action, rather than mental mirroring of an external reality. Moreover, cognition is a particular kind of action - a response strategy that applies some measure of forethought in order to solve some practical real-world problem. During World War II the practical problem of breaking the German codes was of utmost importance to the British war effort, and this led to the development of a series of machines (the Bombes) which could try a vast number of possible cipher keys against intercepted German communications. These decoding machines were among the predecessors of the modem computer. Early computers were designed to model human action - computing possible cipher keys so that machines would replace human labour (Hodges 1983: 160-241). However, this success in the modelling of a very specific intellectual operation was soon mistakenly regarded as the key to understanding cognition in general. If one thinks that mathematical and logical reasoning is what distinguishes human beings from other animals, one might erroneously assume that any computational machine that could model aspects of this peculiarly human trait could also be used to model cognition in general. Hence the MIND As COMPUTER metaphor swept early (firstgeneration) cognitive science. This is a disembodied view of rationality. By

We are live creatures

21

contrast, on the Pragmatist view, our rationality emerges from, and is shaped by, our embodied nature. Thus, Dewey famously asserted that "to see the organism in nature, the nervous system in the organism, the brain in the nervous system, the cortex in the brain is the answer to the problems which haunt philosophy" (Dewey 1925: 198). In the following sections we show how the Pragmatist view of cognition as action provides an appropriate philosophical framework for the cognitive science of the embodied mind. We begin by describing the nondualistic, non-Representationalist view of mind developed by James and Dewey. Their understanding of situated cognition is reinforced by recent empirical research and developments within the cognitive sciences. We cite evidence from comparative neurobiology of organism-environment coupling ranging from the amoeba all the way up to humans, and we argue that in humans this coupling process becomes the basis of meaning and thought. We describe the patterns of these ongoing interactions as image schemas that ground meaning in our embodiment and yet are not internal representations of an external reality. This leads to an account of an emergent rationality that is embodied, social and creative.

2.

James and Dewey: The continuity of embodied experience and thought

In many ways the American Pragmatist philosophers James and Dewey provide us today with exemplary non-reductionist and non-Representationalist models of embodied mind. Their models combined the best biology, psychology and neuroscience of their day with nuanced phenomenological description and a commitment that philosophy should address the pressing human problems of our lives. James and Dewey understood something taken for granted in contemporary biological science: cognition emerges from the embodied processes of an organism that is constantly adapting to better utilize relatively stable patterns within a changing environment. One problem for such a naturalistic account of mind is to explain how meaning, abstract thinking, and formal reasoning could emerge from the basic sensorimotor capacities of organisms as they interact with the environment and each other. The fundamental assumption of the Pragmatists' naturalistic approach is that everything we attribute to "mind" - perceiving, conceptualizing, imagining, reasoning, desiring, willing, dreaming - has emerged (and con-

22

Mark Johnson and Tim Rohrer

tinues to develop) as part of a process in which an organism seeks to survive, grow and flourish within different kinds of situations. As James puts it: Mental facts cannot be properly studied apart from the physical environment of which they take cognizance. The great fault of the older rational psychology was to set up the soul as an absolute spiritual being with certain faculties of its own by which the several activities of remembering, imagining, reasoning and willing, etc. were explained, almost without reference to the peculiarities of the world with which these activities deal. But the richer insight of modem days perceives that our inner faculties are adapted in advance to the features of the world in which we dwell, adapted, I mean, so as to secure our safety and prosperity in its midst. (James 1900: 3)

This evolutionary embeddedness of the organism within its changing environments, and the development of thought in response to such changes, ties mind inextricably to body and environment. The changes entailed by such a view are revolutionary. From the very beginning of life, the problem of knowledge is not how so-called internal ideas can re-present external realities. Instead, the problem of knowledge is to explain how structures and patterns of organism-environment interaction can be adapted and transformed to help deal constructively with changing circumstances that pose new problems, challenges and opportunities for the organism. On this view, mind is never separate from body, for it is always a series of bodily activities immersed in the ongoing flow of organism-environment interactions that constitutes experience. In Dewey's words: Since both the inanimate and the human environment are involved in the functions of life, it is inevitable, if these functions evolve to the point of thinking and if thinking is naturally serial with biological functions, that it will have as the material of thought, even of its erratic imaginings, the events and connections of this environment. (Dewey 1925: 212-213)

Another way of expressing this rootedness of thinking in bodily experience and its connection with the environment is to say that there is no rupture in experience between perceiving, feeling and thinking. In explaining ever more complex "higher" functions, such as consciousness, self-reflection and language use, we do not postulate new ontological kinds of entities, events, or processes that are non-natural or super-natural. More complex levels of organic functioning are just that - levels - and nothing more, although there are emergent properties of "higher" levels of functioning. Dewey names this connectedness of all cognition the principle of continu-

We are live creatures

23

ity, which states that "there is no breach of continuity between operations of inquiry and biological operations and physical operations. 'Continuity' [...] means that rational operations grow out of organic activities, without being identical with that from which they emerge" (Dewey 1938: 26). What the continuity thesis entails is that any explanation of the nature and workings of mind, even the most abstract conceptualization and reasoning, must have its roots in our organismic capacities for perception, feeling, object manipulation and bodily movement. Furthermore, social and cultural forces are required to develop these capacities to their full potential, including language and symbolic reasoning. Infants do not speak or discover mathematical proofs at birth; Dewey's continuity thesis requires both evolutionary and developmental explanations. For James and Dewey, this meant that a full-fledged theory of human cognition must have at least three major components:

(1)

(2)

(3)

There must be an account of the emergence and development of meaningful patterns of organism-environment interactions - patterns of sensorimotor experience shared by all organisms of a certain kind and meaningful for those organisms. Such patterns must be tied to the organism's attempts to function within its environment. There must be an account of how we can perform abstract thinking using our capacities for perception and motor response. There would need to be bodily processes for extending sensorimotor concepts and logic for use in abstract reasoning, as well as an account of how the processes embodying such abstract reasoning capacities are learned during organismic development. This story has at least two parts: (a) an evolutionary and physiological account explaining how an adult human being's abstract reasoning utilizes the brain's perceptual and motor systems, and (b) a developmental and anthropological account of how social and cultural behaviors educate the sensorimotor systems of successive generations of children so that they may speak and perform abstract reasoning. There must be an account of how values and behavioural motivations emerge from the organism's ongoing functioning. This explanation will include (a) the physical and social makeup of organisms, (b) the nature of their emotional responses, and (c) the kinds of environments (e.g., material, social, cultural) they inhabit. In the present space we are able to offer only a very compressed and partial treatment of such an account.

24

Mark Johnson and Tim Rohrer

3.

Organism-environment coupling

3.1.

Maturana and Varela: From chemotaxis to the nervous system

Dewey's principle of continuity states that there are no ontological gaps between the different levels of an organism's functioning. One way to see what this entails is to survey a few representative types of organismenvironment couplings, starting with single-cellular organisms and moving up by degrees to more complex animals. In every case we can observe the same adaptive process of interactive co-ordination between a specific organism and recurring characteristics of its environment. But does that mean that we can trace human cognition all the way back to the sensorimotor behavior of single-cellular organisms? On the face of it, this seems preposterous - viewed from an evolutionary biologist's perspective, there are clear differences in the size, complexity and structural differentiation of human beings as compared with single-cellular organisms like bacteria. Single-cellular organism behaviour is not ordinarily relevant to the behaviour of multi-cellular organisms - except insofar as there might be structural morphological analogies between the sensorimotor activity of singlecellular organisms and particular sensorimotoric cells within the multicellular body. Just this sort of morphological analogy plays a key role in Maturana and Varela's argument that central nervous systems evolved in multi-cellular organisms to co-ordinate sensorimotor activity (Maturana and Varela 1998: 142-163). In a single-cellular organism locomotion is achieved by dynamically coupling the sensory and motoric surfaces of the cell membrane. When an amoeba engulfs a protozoan, its cell membranes are responding to the presence of the chemical substances that make up the protozoan, causing changes in the consistency of the amoeba's protoplasm. These changes manifest as pseudopods - digitations that the amoeba extends around the protozoan as it prepares to feed upon it. Similarly, certain bacteria have a tail-like membrane structure called a flagellum that is rotated like a propeller to move the bacterium. When the flagellum is rotated in one direction the bacterium simply tumbles, while reversing the direction of rotation causes the bacterium to move. If a grain of sugar is placed into the solution containing this bacterium, chemical receptors on the cell membrane sense the sugar molecules. This causes a membrane change in which the bacterium changes the direction of rotation of its flagellar propeller and gradually moves toward the greatest concentration of the sugar molecules (che-

We are live creatures

25

motaxis). In both cases, changes in the chemical environment cause sensory perturbations in the cellular membrane, which invariably produces movement. The key point here is that, without anything like an internal representation, single-cellular organisms engage in sensorimotor coordination in response to environmental changes. Even at this apparently primitive level, there is a finely tuned ongoing coupling of organism and environment. Multi-cellular organisms also accomplish their sensorimotor coordination by means of changes in their cell membranes. However, the cellular specialization afforded by a multi-cellular organism means that not every cell needs to perform the same functions. Maturana and Varela (1998) discuss the example of an evolutionarily ancient metazoic organism called the Hydra (a coelenterate). The Hydra, which lives in ponds, is shaped like a two-layered tube with four or six tentacles emanating from its mouth. On the inside layer of the tube, most cells secrete digestive fluids, while the outside layer is partly composed of radial and longitudinal muscle cells. Locomotion is accomplished by contracting muscle cells along the body of the organism: some of these contractions cause changes in the hydrostatic pressure within the organism, changing its shape and direction of locomotion. Between the two layers of cells, however, are specialized cells - neurons - with elongated membranes that can extend over the length of the entire organism before terminating in the muscle cells. These tail-like cellular projections are the axons, and evolutionarily speaking they are the flagella of the multi-cellular organism.! Changes in the electrochemical state in other, smaller cellular projections of the cells (the dendrites) cause larger changes in the electrochemical state of the axonal membrane, which in turn induces the muscle cells to contract. These neural signals typically originate in either the tentacles or the "stomach" of the Hydra, such that their electrochemical state responds to the molecules indicating the presence or absence of food and/or excessive digestive secretions. These neurons consistently terminate in the longitudinal and radial muscles that contract the Hydra body for locomotion or for swallowing. The topology of how the nerve cells interconnect is crucially important: when touched, a chain of neurons fire sequentially down a Hydra tentacle toward its mouth

1. Recent research shows that this may be more than a surface morphological analogy: all microtubular cellular projections stem from a common ancestor (Erickson et al. 1996; Goldberg 2003).

26

Mark Johnson and Tim Rohrer

and cause the muscle cells to curl the tentacle about its prey even as its mouth begins to open. The Hydra does not "represent" an external world; instead, the structural coupling between organism and environment allows the Hydra to contract the correct muscles to swallow, or to move up and left, or right and down. Like the Hydra opening its mouth as a reflexive part of bringing food to it with its tentacles, we humans think in order to act and we act as part of our thinking - cognition is action. But how is it that we humans can learn new behaviors, while the Hydra generally cannot?

3.2.

From neural maps to neural plasticity

Although still surprisingly continuous with the Hydra, human cognition is a little more similar to what happens in frogs, owls and monkeys in that all of these organisms have nervous systems that include neural maps and adaptive neural plasticity. Frogs have a certain regularly occurring pragmatic problem - they need to extend their tongues to eat a fly - which was the subject of a classic experiment in the early history of neurobiology (Sperry 1943). When a frog is still a tadpole, it is possible to rotate the frog's eye 180 degrees, making sure to keep the optic nerve intact. The tadpole is then allowed to develop normally into a frog. The frog's tongue extends to exactly the opposite point of the frog's visual field from where the fly is located. No amount of failure at catching the fly will teach the frog to move its tongue differently; the frog acts entirely on the basis of the rewired neural connections between the retinal image and the tongue muscles. Maturana and Varela conclude thatfor thefrog "there is no such thing as up or down, front and back, in reference to an outside world, as it exists for the observer doing the study" (1998: 125-126). The frog has no access to our notion of the external world and our 180-degree rotation of its eye; it has only its experience of the world found in the neurons comprising its (experimentally inverted) retinal map. One of the most profound findings in neuroscience is that nervous systems exploit topological and topographic organization. In other words, organisms build neural "maps". In neural maps, adjacent neural cells (or small groups of neural cells) fire sequentially when a stimulus in adjacent positions within a sensory field moves. For example, scientists have stimulated the frog's visual field and measured the electrical activity of a region of its brain to show that as one stimulates the frog's visual field, the

We are live creatures

27

neurons of its optic tectum will fire in co-ordination with the visual stimulus. Fraser (1985) covered the frog's optic tectum with a 24-electrode grid, with each electrode recording electrical activity that was the sum of the signals from a receptive field containing many optic nerve fiber terminals. When a point of light was moved in a straight line from right to left and then from bottom to top in the frog's right visual field, the electrode grid recorded neuronal activity in straight lines, firing sequentially, first from the rostral (front) to the caudal (back) and then from the lateral to the medial. We call this the frog's retinal (or retinotectal) map because it encodes environmental visual stimuli in a topographically consistent manner. The spatial orientation of this topography is rotated in various ways. Thus visual right-to-Ieft has become front-to-back and so on, but the topographic mapping between movement in the vertical visual plane and the plane of the retinotectal neural map remains consistent. Even though there is considerable spatial distortion in the neural map, the key relational structures are preserved. In some other cases, such as some auditory maps and colour maps, where the correspondences can be less about shape and position, the organization is more properly called topologic than topographic, but the organizing principle of the neural mapping of sensation still holds. The degree to which such neural maps might be plastic has been the subject of much recent study. In the case of rotating the eye of the frog, Sperry performed a radical and destructive intervention that is outside the realm of "normal" Darwinian deviation - in other words, if this were to occur by natural selection such a frog would die quickly without passing on its genes. However, interventions which are less radical and perhaps more likely to occur in nature, such as cutting the optic nerve and destroying part of the optic tectum of a goldfish, result in a recovery of function in which the optic nerve axons regenerate to make a complete retinal map in the remaining part of the tectum (Gaze and Sharma 1970). Although radical interventions can "break" the neural maps, even the more evolutionarily determined neural networks exhibit some range of adaptive neural plasticity to environmental factors. Plasticity is particularly profound in cross-modal neural maps. Consider another subtle environmental intervention: suppose we were to have an owl wear glasses that changed its perception of the visual field. Similar to the frog, owls have developed an extremely accurate method of attacking prey. The owl hears a mouse rustling on the ground and locates the mouse using the tiny difference in time it takes for a sound to reach one ear versus the time it takes the sound to reach the owl's other ear. This establishes the

28

Mark Johnson and Tim Rohrer

mouse's approximate position in the owl's retinotectal map, and the diving owl then visually confirms the exact location of its prey before it strikes. Knudsen and colleagues (Knudsen 1998, 2002) put prismatic glasses on adult and juvenile owls which distorted the owls' vision by 23 degrees. After 8 weeks with glasses, adults raised normally never learned to compensate, but juveniles were able to learn to hunt accurately. Moreover, when the glasses were reintroduced to the adult owls who had worn them as juveniles, they were then able to readjust to the glasses in short order; in other words, the prism-reared owls could successfully hunt with or without glasses. These behavioural adaptations have anatomical underpinnings in the plasticity of the neural maps. When injected with an anatomical tracing dye, comparison of the neural arbors from normally-reared and prismreared owls revealed a different pattern of axonal projections between auditory and spatial neural maps, "showing that alternative learned and normal circuits can coexist in this network" (Knudsen 2002: 325). In other words, in order to deal with wearing glasses, the owl brain had grown permanent alternative axonal connections in a cross-modal neural map of space located in the external nucleus of the inferior colliculus (ICX). The ICX neural arbor of prism-reared owls was significantly denser than in normally developing owls, with neurons typically having at least two distinct branches ofaxons (DeBello, Feldman and Knudsen 2001). By contrast, the retinotectal maps of the visual modality alone do not exhibit the same plasticity, either in owls (whose retinotectum did not change) or in frogs. Analogous anatomical research on frogs reared and kept alive with surgically rotated eyes has shown that after five weeks, the retinotectal neural arbors initially exhibited a similar pattern of "two-headed axons" that is, they had two major axonal branches. However, after ten weeks the older axonal connections are starting to decay and disappear, while after sixteen weeks no two-headed axons could be traced (Guo and Udin 2000). Apparently, the frog's single-modal retinotectal maps do not receive enough reentrant neural connections from other sensory modalities to sustain the multiple branching neural arbors found in the cross-modal map of the prism-reared owls. Working on neural plasticity in adult squirrel and owl monkeys, Merzenich and colleagues (Merzenich et al. 1987; reviewed in Buonomano and Merzenich 1998) have shown that it is possible to dynamically reorganize the sensorimotor cortical maps subject to certain bodily constraints. Similar to the owls and frogs that grew dual arborizations, these monkeys exhibited

We are live creatures

29

a plasticity based on their brains' ability to select which parts of their neural arbors to use for various kinds of input. In a series of studies, Merzenich and colleagues altered the monkey's hand sensory activity by such interventions as (1) cutting a peripheral nerve such as the medial or radial nerve and (la) allowing it to regenerate naturally or (lb) tying it off to prevent regeneration; (2) amputating a single digit; and (3) taping together two digits so that they could not be moved independently. The results show that cortical areas now lacking their previous sensory connections (or independent sensory input in the third condition) were "colonized" in a couple of weeks by adjacent neural maps with active sensory connections. In other words, the degree of existing but somewhat dormant neural arbor overlap was large enough to permit reorganization. And in the case of (la), where the nerve was allowed to regenerate, the somatosensory map gradually returned to occupy a similar-sized stretch of cortex, albeit with slightly different boundaries. Learning in adults is accomplished in part by neural gating between redundant and overlapping neural arbors. All of these examples of ontogenetic neural change suggest that there is a process of neural arbor selection akin to natural selection taking place in concert with specific patterns of organism-environment interactions. On precisely these grounds the neurobiologist Gerald Edelman (1987) has proposed a theory of "Neural Darwinism", or "neuronal group selection", to explain how such neural maps are formed in the organism's embryonic development. Different groups of neurons compete to become topological neural maps as they migrate and grow during neural development. Successful cortical groups, driven primarily by regularities in the environment passed on from those neurons that are closer to a sensory apparatus, will fire together and wire together in a process of axonal sprouting and synaptogenesis. Some neuronal groups will fail to find useful topological connections, and they eventually die and are crowded out by the successful neuronal groups, while others will hang on in something of an intermediate state of success (Edelman 1987: 127-140). In the adult organism, the latent axonal arbors from only partly successful attempts to wire together lay dormant, ready to reorganize the map as needed by means of further synaptogenesis. Edelman (1987: 43-47) calls these latent reorganizations of the neuronal groups secondary repertoires, as distinguished from their normal primary repertoires. Like frogs, owls and monkeys, we humans have sets of visual, auditory and somatosensory neural maps. The more obvious of these map perceptual space in fairly direct analogs - preserving topologies of pitch, the retinal

30

Mark Johnson and Tim Rohrer

field, colour, the parts of the body and so on - but subsequent maps preserve increasingly abstract topological structure (or even combinations of structure) such as object shape, edges, orientation, direction of motion and even the particular degree of the vertical or horizontal. Like the frog, we live in the world of our maps. Topologically speaking, our bodies are in our minds, in the sense that our sensorimotor maps provide the basis for conceptualization and reasoning. We perceive the patterns of our daily organism-environment interactions in image-like fashion, constantly seeking out various topological invariances in those patterns that prove useful to us. In the following section we will show how our imagination and our reason are constituted by patterns of activation within these neural maps. But before proceeding to human cognition, we must first address why neural "maps" are not classical Representations.

3.3.

Neural maps are not internal representations

Some people might suppose that talk of neural "maps" would necessarily engender Representationalist theories of cognition. On this view, the map would be construed as an internal representation of some external reality. But the account we have been giving does not entail any of the traditional metaphysical dualisms that underlie Representationalist views - dichotomies such as inner/outer, subject/object, mind/body, self/world. Such dichotomies might describe aspects of organism-environment interactions from an observer's perspective, but they do not indicate different ontological entities or structures. According to our interactionist view, maps and other structures of organism-environment co-ordination are prime examples of non-representational structures of meaning, understanding and thought. 2 2. We are certainly not suggesting that neuroscientists should purge the term "representation" from their vocabulary. Nor are we suggesting that there is no sense in which it would be appropriate to say that some neuronal structure is a representation from the perspective of the scientist who is studying cognitive processes. For example, we do not object to neuroscientists saying that a particular neural map in the auditory cortex can "represent" various pitch relations among musical tones, though we prefer to employ more enactive terms such as "map" and "activation contours". However, such casual usage doesn't necessarily entail the Representational Theory of Mind that we are challenging here. Instead, we argue that Representationalism is based on a mistaken philosophical analogy

We are live creatures

31

Maturana and Varela (1998: 125-126) make this important philosophical point quite clear. We must not read our scientific or philosophical perspectives (i.e., our theoretical stance) on cognition back into the experience itself that we are theorizing about. We must not uncritically assume that distinctions we make in explaining a certain cognitive experience are thereby part of the person's experience. To do so is to fall prey to what James termed the "Psychologist's Fallacy". In observing something scientifically, one must always consider the standpoint of the scientist in relation to the object of study. When we use terms such as "retinal map", "pitch maps", "sensorimotor maps", "color maps" and so forth to describe the operations of various neural arrays in a frog's nervous system, or in human nervous systems, we are doing so from our standpoint as observers and theorists who can see mappings between those neural structures and our own experience of the "external world". But for the frog, and for the human in the act of perceiving, that map is the basis for its experience of the world. The map constitutes the sensorimotor experience of a certain part of the frog's world. The frog's neural map itself has its origin not in the immediate mappings that we observers see in the moment, but in a longitudinal evolutionary and developmental process during which those neural connections were "selected for" by Darwinian or neo-Darwinian mechanIsms. In short, what we (as scientists) theoretically recognize and describe as an organism's "maps" are not for that organism internal representations. Rather, what we' call sensorimotor and somatosensory maps (whether in multi-cellular organisms, monkeys, or humans) are for that organism precisely the structures of its experienced world! Consequently, we must be careful not to be misled by philosophers of mind and language who would treat these maps as internal representations of external realities, thereby surreptitiously introducing an "inner/outer" split that does not exist in reality for the organism. (namely "the language of thought" framework in which a mental state refers to the world much as a word supposedly simply refers to an object or a state-ofaffairs in the world). In order to undermine such Representationalist theories, we argue that actual neural representations are perpetually situated in dynamic organism-environment interactions that are continually changing along experiential, developmental and phylogenetic timelines. Hence, it is a mistake to think that neural maps are representations in virtue of an immediate word-world referential mapping, whether that word is a linguistic entity or a mental entity in a "language of thought".

32

4.

Mark Johnson and Tim Rohrer

Ontological continuity and human thought: Image schemas and amodal perception

Since the earliest episodes of ancient Greek philosophy, humans have been distinguished from "brute" animals and all lower organisms by their supposedly unique capacity for abstract conceptualization and reasoning. According to this view, human reason is what makes it possible for us to form abstract mental representations that stand for and point to states of affairs that are either external to us or are not currently present in our experience (i.e., are past or future). But the Pragmatists' Continuity Thesis denies the inner/outer dichotomy upon which Representationalist theories are based. Consequently, the problem for an embodied view of cognition is how to explain our marvellous human feats of abstraction, reasoning and symbolic interaction, yet without positing an ontological rupture between "lower" animals and humans. The key, once again, is the coupling (the interactive co-ordination) of an organism (here, a human one) and its environment. Recurring adaptive patterns of organism-environment interaction are the basis for our ability to survive and flourish. In humans, these patterns are no more "internal" representations than they are in other creatures. Let us consider briefly some of the most basic kinds of structural couplings that make up a human being's experience of its world.

4.1.

Image schemas and cross-modal perception

The character of our experience is delineated in large part by the nature of our bodies and brains, the kinds of environments we inhabit, and the values and purposes we have. The patterns of our ongoing interactions (or "enactions" as Varela, Rosch and Thompson (1991) have called them, to stress their active, dynamic character) define the contours of our world and make it possible for us to make sense of, reason about, and act reliably within this world. Thousands of times each day we see, manipulate and move into and out of containers, so containment is one of the most fundamental patterns of our experience. Because we have two legs and stand up within a gravitational field, we experience verticality and up-down orientation. Because the qualities (e.g., redness, softness, coolness, agitation, sharpness) of our experience vary continuously in intensity, there is a scalar vector in our world. For example, lights can grow brighter or dimmer, stoves get

We are live creatures

33

hotter or cooler, iced tea gets sweeter as we add sugar. We are subject to forces that move us, change our bodily states and constrain our actions, and all of these forces have characteristic patterns and qualities. We are bound inextricably to our world interactively (enactively) by means of these recurring patterns that are the very conditions for us to survive, grow and find meaning. Without such patterns, and without neural maps of such characteristic patterns, each moment of our experience would be utterly chaotic, as though we had to make sense of our world from scratch, over and over again as each new moment arose. What Johnson (1987) and Lakoff (1987) called "image schemas" are precisely these stable recurring patterns of sensorimotor experience by which we engage a world that we can understand and act within to further our purposes. There are numerous sources of evidence for the existence of image schemas, ranging from experimental psychology to linguistics to developmental psychology. We hypothesize that these image schemas are neurally embodied as patterns of activation in and between our topological neural maps. Image schemas are thus part of our non-representational coupling with our world, just as barn owls and squirrel monkeys have image schemas that define their types of sensorimotor experience. Image schematic structure is the basis for our understanding of all aspects of our perception and motor activities. An example from Lakoff and Nufiez (2000) illustrates this image-schematic basis of spatial concepts in humans. What we call our concept IN is defined for us by a CONTAINER image schema that consists generically of (1) a boundary that demarcates (2) an interior from (3) an exterior. When we say The car is in the garage, we understand the garage as a bounded space, we profile (Langacker 1986) the interior of that space, and we regard the car as what cognitive linguists call a trajector within that space, with the garage (as container) serving as a landmark in relation to which the trajector is located. Similarly, when we hear the sentence Grandpa walked from the outhouse to the garage, we understand that situation via a SOURCE-PATH-GOAL schema that consists of (a) a starting point, (b) a destination (endpoint), and (3) a path from the starting location to the destination. In other words, the from-to construction is imageschematic. The English word into is understood via a superimposition of the SOURCE-PATH-GOAL schema on the CONTAINER schema, as follows:

34

Mark Johnson and Tim Rohrer

in activates a CONTAINER schema with the interior profiled. to activates a SOURCE-PATH-GOAL schema with the GOAL (endpoint) profiled. The GOAL (endpoint) is mapped onto the interior of the CONTAINER schema. We thus understand Grandpa's (as trajector) movement as beginning outside the garage (CONTAINER) and terminating inside the garage (as landmark), as a result of motion along a path from the exterior to the interior. into in English is thus an elementary composition of two image schemas. Image schemas are realized as activation patterns (or "contours") in human topological neural maps. As with much interdisciplinary research in the neurosciences, the evidence for this first emerged from intracranial neuronal recordings on monkeys and was later extended to humans via analogous neuroimaging studies. When Rizzolatti and colleagues (Fogassi et al 2001; see review in Rizzolatti, Fogassi and Gallese 2002) showed macaque monkeys visual imagery of another monkey grasping a banana with their hands, they were able to record activity from "mirror" neurons in the same areas of secondary somatomotor cortex that would be implicated if the monkey himself were performing the particular grasping action. Analogous human neuroimaging experiments (Buccino et al. 2001) in which participants watched a video clip of another person performing an action showed increased activation in the human secondary somatomotor cortices that are known to map human hand and arm grasping motions. Along with Rizzolatti's colleague Gallese, we interpret these and related results as having shown that these neural maps contain image schematic sensorimotor activation patterns for grasping (Gallese and Lakoff 2005). An explicit attempt to model image schemas using known facts about our neural maps can be found within the neurocomputational modelling literature. Regier (1996) has developed what he calls "structured" or "constrained" connectionist neural models for a number of image schemas. "Constrained" neurocomputational connectionism builds into its neural models a small number of structures that have been identified in research on human visual and spatial processing. These include center-surround cell arrays, spreading activation, orientation-sensitive cells and neural gating.

We are live creatures

35

Regier has shown how these constrained connectionist models of image schemas can learn spatial relations terms. 3 There is also a growing body of research from developmental psychology suggesting that infants come into the world with capacities for experiencing image-schematic structures. Stem (1985) described certain types of experiential structures that infants are able to detect, and he argues, first, that these capacities form the basis for meaning and the infant's sense of self; and, second, that these capacities continue to play a central role in meaning, understanding and thinking even in adults who are capable of propositional thinking. Let us briefly consider two of these basic structures: (1) cross-modal perception, and (2) vitality affect contours. Stem begins with a well-known experiment (Meltzoff and Borton 1979) in which blind-folded infants were given one of two pacifiers to suck. One was the typical smooth pacifier, while the other had protruding nubs. When the blindfolds were removed and smooth and nubbed pacifiers were placed on either side of the infant's head, most of the time (roughly 75%) the infant would attend to the nipple of the pacifier just sucked. Based on this and other studies (e.g., Lewkowicz and Turkewitz 1981), Stem suggests that Infants thus appear to have an innate general capacity, which can be called amodal perception, to take information received in one sensory modality and somehow translate it into another sensory modality. [...] These abstract representations that the infant experiences are not sights and sounds and touches and nameable objects, but rather shapes, intensities and temporal patterns - the more "global" qualities of experience. (Stem 1985: 51)

Although he speaks of these structures of cross-modal perception as amodal, abstract "representations", Stem also makes it clear that these perceptual structures are not inner mirrorings of external things but rather are the contours of the infant's experience: the cross-modal shapes, intensities and temporal patterns that we call image schemas. Like infants, we adults have a ROUGH/SMOOTH image schema, which we can use as we anticipate the change in surface texture as we walk. For 3. This does not, of course, prove that human cognition necessarily works this way, but Regier's use of computational neural models built on known human neural architectures offers a number of advantages over traditional PDP connectionist models. Moreover, Regier's models can be appropriated into programs that allow robots to perform certain bodily movements.

36

Mark Johnson and Tim Rohrer

example, we can see where we will step from the rough carpet of the hallway onto the slippery tile of the bathroom, and we transfer this information from the visual to the somatomotor system so that our feet will not slip. Such patterns of cross-modal perception are especially clear examples of how image schemas differ from being just a topographically mapped image in a neural map; they are sensorimotoric patterns of experience which are instantiated in and coordinated between unimodal neural maps. Our image schematic experience may, as in the case of the owl, become instantiated in its own cross-modal neural map; or, as in the case of monkeys, it might consist of coordinated activation patterns between a network of more modal neural maps, including possibly calling on the secondary rather than primary repertoires of those maps. We predict that cases analogous to each will be observed in human neuroanatomical studies. A second type of pattern that makes up the infant's (and adult's) imageschematic experience is what Stem (1985) calls "vitality affect contours". Stem illustrates this with the notion of a "rush", or the swelling qualitative contour of a felt experience. We can experience an adrenaline rush, a rush of joy or anger, a drug-induced rush, or the rush of a hot-flash. Even though these rushes are felt in different sensory modalities, they are all characterizable as a rapid, forceful building up or swelling contour of the experience across time. Stem notes that understanding how such affect contours are meaningful to creatures like us gives us profound insight into meaning generally, whether that meaning comes via language, vision, music, dance, touch, or smell. We crave the emotional satisfaction that comes from pattern completion, and witnessing even just a portion of the pattern is enough to set our affect contours in motion. The infant just needs to see us begin to reach for the bottle, and she already begins to quiet down - the grasping image schema does not even need to be completely realized in time before the infant recognizes the action. When as adults we hear a musical composition building up to a crescendo, this causes increasing emotional tension that is released at the musical climax. The emotional salience of the vitality affect contours in image schemas shows that image schemas are not mere static "representations" (or "snapshots") of one moment in a topographic neural map (or maps). Instead, image schemas proceed dynamically in and through time. To summarize, image schemas can be characterized more formally as:

We are live creatures

(1) (2) (3) (4) (5) (6)

37

recurrent patterns of bodily experience, "image"-like in that they preserve the topological structure of the perceptual whole, as evidenced by pattern-completion, operating dynamically in and across time, realized as activation patterns (or "contours") in and between topologic neural maps, structures which link sensorimotor experience to conceptualization andlanguage,and structures which afford 'normal' pattern completions that can serve as a basis for inference.

Image schemas constitute a preverbal and pre-reflective emergent level of meaning. They are patterns found in the topologic neural maps we share with other animals, though we as humans have particular image schemas that are more or less peculiar to our types of bodies. However, even though image schemas typically operate without our conscious awareness of how they structure our experience, it is sometimes possible to become reflectively aware of the image-schematic structure of a certain experience, such as when I am consciously aware of my cupped hands as forming a container, or when I feel my body as being off balance.

4.2.

Abstract conceptualization and reasoning

Pragmatism's Continuity Thesis claims that we must be able to move, without any ontological rupture, from the body-based meaning of spatial and perceptual experience that is characterizable by image schemas and affect contours, all the way up to abstract conceptualization, reasoning and language use. Although there is not yet any fully worked out theory of how all abstract thought works, some of the central mechanisms are becoming better understood. One particularly important structure is conceptual metaphor (Lakoff and Johnson 1980; 1999). The most sweeping claim of Conceptual Metaphor Theory (CMT) is that what we call "abstract" concepts are defined by systematic mappings from bodily-based sensorimotor source domains onto abstract target domains. These metaphor mappings are found in patterns motivated by image schematic constraints - for example, if we map an interior from the source domain, we can expect to map the exterior as well; if we have source and destination mappings, we can expect a path mappIng.

38

Mark Johnson and Tim Rohrer

Consider the sentence We have a long way to go before our theory is finished. Why can we use the phrase a long way to go, which is literally about distance in motion through space, to talk about the completion of a mental project (i.e., developing a theory)? The answer is that there is a conceptual metaphor PURPOSEFUL ACTIVITIES ARE JOURNEYS, via which some cultures understand progress toward some nonphysical goal as progress in moving toward a destination. The metaphor consists of the following conceptual mapping: The PURPOSEFUL ACTIVITIES ARE JOURNEYS METAPHOR Source (motion in space) Target (mental activity) Starting point A -7 Initial state Ending location B -7 Final State Destination -7 Purpose to be achieved Motion from A to B -7 Process of achieving purpose Obstacles to motion -7 Difficulties in achieving goals This conceptual mapping also makes use of one of our culture's most basic metaphors for understanding the passage of time, in which temporal change is understood metaphorically as motion along a path to some location. In this metaphor, the observer moves along a time line, with the future arrayed as the space in front of her and the past as the space behind. Consequently, when we hear We have a long way to go until our campaign fund drive is finished, we understand ourselves metaphorically as moving along a path toward the destination (completion of the fund drive), and we understand that there can be obstacles along the way that would slow our progress. Conceptual Metaphor Theory proposes that all abstract conceptualization works via conceptual metaphor, conceptual metonymy and a few other principles of imaginative extension. To date there is a rapidly growing body of metaphor analyses of key concepts in nearly every conceivable intellectual field and discipline, including the physical and biological sciences, economics, morality, politics, ethics, philosophy, anthropology, psychology, religion and more. For example, Lakoff and Nlinez (2000) have carried out extensive analyses of the fundamental metaphorical concepts that underlie mathematics, from simple models of addition all the way up to concepts of the Cartesian plane, infinity and differential equations. Winter (2001) analyzes several key metaphors that define central legal concepts and are the basis for legal reasoning. Grady (1997) examines

We are live creatures

39

"primary metaphors" (such as PURPOSES ARE DESTINATIONS) that are combined systematically into more complex metaphors (such as PURPOSEFUL ACTIVITIES ARE JOURNEYS). The reason that conceptual metaphor is so important is that it is our primary means for abstract conceptualization and reasoning. Pragmatism's principle of continuity claims that abstract thought is not disembodied; rather, it must arise from our sensorimotor capacities and is constrained by the nature of our bodies, brains and environments. From an evolutionary perspective this means that we have not developed two separate logical and inferential systems, one for our bodily experiences and one for our abstract reasoning (as a pure logic). Instead, the logic of our bodily experience provides all the logic we need in order to perform every rational inference that we do. In our metaphor-based reasoning, the inferences are carried out via the corporeal logic of our sensorimotor capacities, and then, via the sourceto-target domain mapping, the corresponding logical inferences are drawn in the target domain. For example, there is definite spatial or bodily image-schematic logic of containment that arises in our experience with containers: (a) (b)

An entity is either inside the container or outside it, but not both at once. If I place an object 0 within a physical container C and then put container C inside of another container D, then 0 is in D.

In other words, our bodily encounters with containers and objects that we observe and manipulate teach us the spatial logic of containers. Next, consider the common conceptual metaphor CATEGORIES ARE CONTAINERS, in which a conceptual category is understood metaphorically as an abstract container for physical and abstract entities. For example, we may say: The category 'human' is contained in the category 'animals,' which is contained in the category 'living things. ' Similarly, we may ask Which category is this tree in? Based on the inferential image-schematic structure of the source domain, and via the source-to-target mapping, we then have corresponding inferences about abstract concepts: (a')

An entity either falls within a given category, or falls outside it, but not both at once [e.g., Charles cannot be a man and not a man at the same time, in the same place, and in the same manner]. (The Law of the Excluded Middle).

40

(b')

Mark Johnson and Tim Rohrer

If an entity E is in one category C', and C' is in another category D', then that entity E is in category D' [For example, All men are mortal (C' is in D~ and Socrates is a man (E is in C~, therefore Socrates is mortal (E is in D~].

Thus, according to CMT we would then predict that the abstract inferences are "computed" using sensorimotor neural maps, and those inferences are activated as target-domain inferences because there are neural connections from sensorimotor areas of the brain to other areas that are responsible for so-called "higher" cognitive functions. The hypothesis is that human beings don't run an inferential process at the sensorimotor level and then perform an entirely different inferential process for abstract concepts; rather, human beings utilize the inference patterns found in the sensorimotor brain regions to perform "abstract" reasoning. Just as the Pragmatist Principle of Continuity requires, there is no need to introduce a new kind of reasoning (with a different ontological basis) to explain logical reasoning with abstract concepts.

4.3.

Evidence for conceptual metaphor and abstract reasoning using conceptual metaphors

Recently several new sources of evidence have become available to explain the possible neural bases for the image-schematic mappings that operate in conceptual metaphors. The new evidence comes from both the patientbased neurological literature and neuroimaging studies of normal adults. While we have long known that patients can develop anomias reflecting selective category deficits for animals, tools and plants (Warrington and Shallice 1984), several recent studies have reported a selective category deficit for body part terms (Suziki, Yamadori and Fujii 1997; Shelton, Fouch and Caramazza 1998; Coslett, Saffran and Schwoebel 2002). The deficit work suggests that lesions in the secondary motor cortices, in regions which likely contain both somatotopic and egocentric spatial maps, can cause difficulties in tasks such as body part naming, naming contiguous sections of the body, and so on. This finding suggests that the comprehension of body part terms requires the active participation of these neural maps. Two other neuroimaging studies also show that we can drive the human somatomotor maps with both metaphoric and literal linguistic stimuli re-

We are live creatures

41

lating to the body. In an fMRI study, Hauk, Johnsrude and Pulvermiiller (2004) have shown that single word terms such as "smile", "punch" and "kick" differentially activate face, armlhand and leg regions within the somatomotor maps, suggesting that literal language can differentially activate body-part related somatomotor neural maps. Similarly, an fMRI neuroimaging study by Rohrer (2001 b, 2005) shows that both literal and metaphoric sentences using hand terms (e.g., She grasped the apple and He grasped the theory) activate primary and secondary hand regions within the primary and secondary sensorimotor maps. After the presentation of the linguistic stimuli, Rohrer also mapped the hand somatic cortex of each study participant using a tactile hand-stroking task. A comparison between the tactile and the sentential conditions shows a high degree of overlap in the primary and secondary somatomotor cortex for both language tasks, cf. Figure 1.

Metaphoric Hand Sentences

Figure 1.

fMRI activation courses in response to literal and metaphoric action sentences. Areas active and overlapping from a hand somatosensory task were outlined in white (Rohrer 2001b).

There is also evidence from neurocomputationally inspired models of conceptual metaphor and abstract reasoning. Building on Regier's work on modelling the image-schematic character of spatial relation terms, Naraya-

42

Mark Johnson and Tim Rohrer

nan (1997; Feldman and Narayanan 2004) developed a constrained connectionist network to model how the bodily logic of our sensorimotor systems enables us to perform abstract reasoning about international economics using conceptual metaphors. For example, the system was able to successfully interpret both In 1991, the Indian government deregulated the business sector and In 1991, the Indian government loosened its stranglehold on business. Narayanan's model can perform inferences either entirely within the sensorimotoric domain or in the linguistic domain using common conceptual metaphor mappings. Taken together with the neurophysiological and neuroimaging evidence for image schemas and conceptual metaphors, these neurocomputational models support the image-schematic and metaphoric basis of our language and abstract reasoning.

5.

The continuity of embodied social and cultural cognition

In this chapter, we have been presenting evidence for the embodied char-

acter of cognition, and we have suggested an appropriate Pragmatist philosophical framework for interpreting that evidence. Contra Representationalism, we have argued that cognition is not some inner process performed by the "mind", but rather is a form of embodied action. We argued this by giving examples of how cognition is located in organism-environment interactions, instead of being locked up in some allegedly private mental sphere of thought. However, an exclusive focus on the organism's engagement and coupling with its environment can lead to the mistaken impression that thought is individual, not social. Therefore, we must at least briefly address the crucial fact that language and abstract reasoning are socially and culturally situated activities. Thus far, we have discussed only one socio-cultural dimension, albeit a crucially important one, namely, development. Our brief discussion of development was framed more within the context of nervous systems than within socio-cultural interactions. We stressed the point that epigenetic bodily interactions with the world are what shape our neural maps and the image schemas in them. For humans, a very large and distinctive part of that involves interacting with other humans. In other words, human understanding and thinking is social. This raises the question: How do socially and culturally determined factors come to play a role in human cognition? Perhaps a sceptic might say that the locus of the distinctively human lies in a socially and culturally learned capacity for classical Representational-

We are live creatures

43

ism. Once again, however, the Representationalist proposal rests on two mistakes. First, there is not a radical ontological break from the rest of the animal kingdom with respect to socially and culturally transmitted behaviors, both in general and specifically in the cases of linguistic and symbolic communication. Second, having challenged the "inner mind" versus "outer body" split, we must not then proceed to replace it with another equally problematic dichotomy - that between the "individual" and the "social". We must recognize that cognition does not take place only within the brain and body of a single individual, but instead is partly constituted by social interactions and relations. The evidence to which we now turn comes from cognitive ethology and distributed cognition. Of course there are ways in which our socio-cultural behaviors are peculiarly human, but the story is once again much more complex and multi-dimensional than classical Representationalists suppose. Following Maturana and Varela (1998: 180-184) we would define social phenomena as those phenomena arising out of recurrent structural couplings that require the co-ordinated participation of multiple organisms. They argue that just as the cell-to-cell interactions in the transition from single to multi-cellular organisms afford a new level of intercellular structural coupling, so also recurrent interactions between organisms afford a new level of inter-organism structural coupling. The social insects are perhaps the most basic example of this kind of recurrent inter-organism behaviors. For example, ants must feed their queen for their colony to remain alive. Individual workers navigate their way to and from the nest and food sources by leaving trails of chemical markers, but these markers are not distinctive to the individual ant. When seeking food, an individual ant moves away from markers dropped by other ants. Naturally the density of such markers decreases in proportion to the distance from the nest. But when one finds food they begin to actively seek denser clusters of markers, thus leading them back to the nest. Furthermore, whenever a worker ant eats, their chemical markers change slightly. These chemical markers attract, rather than repel, other ants. Thus the ants gradually begin to form a column leading from a food source to a nest. Note that the ants' cognition is both social, in that it takes place between organisms, and distributed, in the sense that it offloads much of the cognitive work onto the environment. No single ant carries around an "internal representation" or neural map of where the ant colony is. Ant cognition is thus nonrepresentational in that it is both intrinsically social and situated in organism-environment interactions.

44

Mark Johnson and Tim Rohrer

The evolutionarily programmed social cognition of insects, however, does not include the capacity for spontaneous imitation which is so central to human cognition. For a social behavior to become a learned behavior and then continue across generations, a capacity for spontaneous imitation is crucial. However, zoological ethologists have long known that this imitative capacity is not unique to humans. Researchers studying macaques left sweet potatoes on the beach for a colony of wild monkeys who normally inhabit the jungle near the beach. After gradually becoming habituated to the beach and becoming more familiar with the sea, one monkey discovered that dipping the potatoes in a tidepool would cleanse them of the sand that made them unpalatable. This behavior was imitated throughout the colony in a matter of days, but the researchers observed that older macaques were slower to acquire the behavior than the younger ones (Kawamura 1959; McGrew 1998). Maturana and Varela (1998: 203) define cultural behavior precisely as this kind of relatively stable pattern of such transgenerational social behavior. The culturally acquired behavior most often held up by classical Representationalists as the hallmark of the distinctively human is language. However, even here there is not a clear break from the animal kingdom in terms of basic cognitive capabilities, as we see when considering the results of researchers who have been trying to teach symbolic communication to other primates. Instead, their observations are consonant with our theory of how language and image schemas emerge from bodily processes involving cross-modal perception. In experiments done by SavageRumbaugh and colleagues (1988), three chimpanzees who had been trained in symbolic communication were able to make not only cross-modal associations (i.e., visual to tactile), but were able to make symbolic to sensorymodal associations. For example, Kanzi was able to hear a spoken English word and accurately (100% of the time) choose either the corresponding visual lexigram or a visual picture of the word. Sherman and Austin were able to choose the appropriate object by touch when presented with a visuallexigram (100% correct), and conversely they were also able to choose the appropriate visuallexigram when presented with a tactile-only stimulus (Sherman: 96% correct, Austin: 100%) or olfactory-only stimulus (Sherman: 95% correct, Austin 70%: correct). Their ability to perform such symbol to sensory-modality coordination enhanced their performance on tasks measuring solely cross-modal coordination; as Savage-Rumbaugh et al. observe: "these symbol-sophisticated apes were able to perform a variety of cross-modal tasks and to switch easily from one type of task to an-

We are live creatures

45

other. Other apes have been limited to a single cross-modal task" (1988: 623). Although these chimpanzees will never approach the linguistic capabilities of humans, these results show that the continuity of our human capacity for abstract cross-modal thought is shared by at least some members of the animal kingdom. 4 In fact, related recent research on primates suggests that it is the distinctively human socio-cultural environment (and not some great zoological discontinuity in comparative cognitive capacity) that facilitates the cross-modal cognitive capabilities underlying language and abstract reason. We have already noted the neural development of the cross-modal maps of juvenile owls can be modified by epigenetic stimulation, but it is equally important to realize that the cross-modal basis for many of our image schemas require epigenetic stimulation of the kind presented by human parents. Tomasello, Savage-Rumbaugh and Kruger (1993) compared the abilities of chimpanzees and human children to imitatively learn how to perform novel actions with novel objects. They tested 3 conspecific (mother-reared) chimpanzees and 3 enculturated chimpanzees, along with 18 and 30 month-old human children. They introduced a new object into the participant's environment, and after observing the participant's natural interactions with the object, the experimenter demonstrated a novel action with the object with the instruction "Do what I do". Their results showed that the mother-reared chimpanzees were much poorer imitators than the enculturated chimpanzees and the human children, who did not differ from one another. A human-like sociocultural environment is an essential component not only for the development of our capacity for imitation, but also for the development of our capacities for the cross-modal image schemas that underlie language and abstract reasoning (see also Fouts, Jensvold and Fouts 2002). Finally, there is also considerable evidence from cognitive anthropology that adult humans do not think in a manner consistent with the dichotomies posed by classical Representationalism. Like the social insects, we tend to offload much of our cognition onto the environments we create. We tend to accomplish this in two ways - first, we make cognitive artifacts to help us engage in complex cognitive actions, and, second, we distribute cognition among members of a social organization. As an example of the first,

4. This conclusion is further supported by results showing that human children with specific language impairments show deficiencies in their ability to perform cross-modal tasks (Montgomery 1993).

46

Mark Johnson and Tim Rohrer

Hutchins (1995: 99-102) discusses how medieval mariners used the 32point compass rose to predict tides. By superimposing onto the compass rose the 24-hour day (in 45-minute intervals), the mariners could map the lunar "time" of the high tide (the bearing of the full moon when its pull causes a high tide) to a solar time of day. As long as we know two facts the number of days since the last full moon and the lunar high tide for a particular port - we simply count off a number of points on the compass rose equal to the days past the full moon to compute the time of next high tide. Without the schema provided by the cognitive artifact, computing the next high tide is a much more laborious cognitive task. As an example of the second, Hutchins (1995: 263-285) discusses how the partially overlapping knowledge distributions of a group of three navy navigation personnel function cognitively within the team considered as a team. Although no single team member is expected to constantly maintain a complete internal representation of all the navigational data, Hutchins shows how the social distribution of the cognitive tasks functions as a brake on serious navigational errors that could imperil the ship, because the participants each know some of the spatial relations and procedures immanent to another team member's job. In short, the offloading of some of the cognitive load onto the environment, as found both in cognitive artifacts and the social distribution of cognitive tasks, is crucial to many of our daily cognitive activities. A fully adequate treatment of the social dimension of thought would require substantially more evidence and analysis than we can provide here. We have only attempted to suggest that sociocultural cognition in general is not unique to humankind, that the common bases for cross-modal cognition and symbolic/linguistic communication are not unique to humans, and that human cognition cannot be locked up within the private workings of an individual mind. Since thought is a form of co-ordinated action, it is spread out in the world, co-ordinated with both the physical environment and the social, cultural, moral, political and religious environments, institutions and shared practices. Language - and all forms of symbolic expression - are quintessentially social behaviors. Dewey nicely summarizes the intrinsically social character of all thought in his argument that the very idea of thinking as a kind of inner mental dialogue is only possible because of socially established and preserved meanings, values and practices: When this introspectionist thinks he has withdrawn into a wholly private realm of events disparate in kind from other events, made out of mental stuff, he is only turning his attention to his own soliloquy. And soliloquy is

We are live creatures

47

the product and reflex of converse with others; social communication not an effect of soliloquy. If we had not talked with others and they with us, we should never talk to and with ourselves. Because of converse, social give and take, various organic attitudes become an assemblage of persons engaged in converse, conferring with one another, exchanging distinctive experiences [...]. Through speech a person dramatically identifies himself with potential acts and deeds; he plays many roles, not in successive stages of life but in a contemporaneously enacted drama. Thus mind emerges. (Dewey 1925: 135)

HThus mind emerges!" It emerges as, and is enacted through, social cognition. There is no radical rupture with our bodily experience of meaning; instead, that meaning is carried forward and given voice through language and other forms of social symbolic interaction and expression.

6.

Embodied meaning, thought and language

We have been arguing against disembodied views of mind, concepts and reasoning, especially as they underlie Representationalist theories of mind and language. Our alternative view - that cognition is embodied - has roots in American Pragmatist philosophy and is being supported and extended by recent work in second-generation cognitive science. Pragmatists like James and Dewey understood that philosophy and empirical science must develop in mutual cooperation and criticism, if we are ever to have an empirically responsible understanding of the human mind and all of its marvelous capacities and acts. Pragmatism is characterized by (1) a profound respect for the richness, depth and complexity of human experience and cognition, (2) an evolutionary perspective that appreciates the role of dynamic change in all development (as opposed to fixity and finality), and (3) recognition that human cognition and creativity arise in response to problematic situations that involve values, interests and social interaction. The principle of continuity encompasses the fact that apparently novel aspects of thought and social interaction arise naturally via increased complexity of the organismenvironment interactions that constitute experience. Pragmatists thus argue that all of our traditional metaphysical and epistemological dualisms (e.g., mind/body, inner/outer, subject/object, concept/percept, reason/emotion, knowledge/imagination and theory/practice) are merely abstractions from the interactive (enactive) process that is experience. Such distinctions are not absolute ontological dichotomies. Sometimes they serve us well, but

48

Mark Johnson and Tim Rohrer

oftentimes they serve us quite poorly, depending on what problems we are investigating, what values we have, and what the socio-cultural context is. In recent years the number of researchers engaged in some variation of "embodied cognition" has swelled prodigiously. Once upon a time, cognitive science seemed defined by the Representationalist view that the body is inconsequential to the study of the mind. But that has changed dramatically. Some Representationalists have recently argued for a very limited sense of embodiment that would keep intact much of the first generation of cognitive science's representational baggage (Clark 1998). Today we are witnessing a new generation of cognitive science emerging which defines "embodied cognition" as a fundamentally non-representational project. Contributions to a radical theory of embodied cognition are being made by dynamic systems theorists who argue that cognition, though amenable to mathematical description, is not computational (van Gelder 1995), by neurobiologists whose experiments show us how metaphors of information transfer mislead us in understanding the population dynamics behind neural organization (Edelman 1992), and by cognitive roboticists who understand that having a body is perhaps not such a bad thing after all (Brooks 1991; Brooks and Stein 1994). Even Alan Turing, a leader among that lost first generation who so errantly steered cognitive science toward disembodiment, was willing to admit he might be wrong when it came to how we might teach a robot language: It can also be maintained that it is best to provide the machine with the best sense organs that money can buy, and then teach it to understand and speak English. That process could follow the normal teaching of a child. Things would be pointed out and named, etc. Again, I do not know what the right answer is, but I think both approaches should be tried. (Turing 1950: 460)

We have already tried the disembodied Representationalist approach, and its failures have breathed new life into the Pragmatist approach to embodied cognition. The themes we have been tracing throughout this chapter - our animal engagement and cognition, our ongoing coupling and our falling in and out of harmony with our surroundings, our active value-laden inquiry to reestablish harmony and growth, and our community of social interactions are beautifully encapsulated by Dewey in his attempt to recover the value of the aesthetic dimensions of meaning in human life: At every moment, the living creature is exposed to dangers from its surroundings, and at every moment, it must draw upon something in its sur-

We are live creatures

49

roundings to satisfy its needs. The career and destiny of a living thing are bound up with its interchanges with environment, not externally but in the most intimate needs. The growl of a dog crouching over his food, his howl in time of loss and loneliness, the wagging of his tail at the return of his human friend are expressions of the implication of a living in a natural medium which includes man along with the animal he has domesticated. Every need, say for hunger for fresh air or food, is a lack that denoted at least a temporary absence of adequate adjustment with surroundings. But it is also a demand, a reaching out into the environment by building at least a temporary equilibrium. Life itself consists of phases in which the organism falls out of step with the march of surrounding things and then recovers unison with it - either through effort or some happy chance [... ]. These biological commonplaces are something more than that [mere biological consequences]; they reach to the roots of the esthetic in experience. (Dewey 1934: 13-14)

We humans are live creatures. We are acting when we think, perhaps falling in and out of step with the environment, but never are our thoughts outside of it. Via our bodily senses the environment enters into the very shape of our thought, sculpting our most abstract reasoning from our embodied interactions with the world.

Acknowledgements

The authors would like to acknowledge the many insightful comments by the reviewers and the editors of this volume. Tim Rohrer would like to acknowledge the stimulating intellectual environment of the Sereno and Kutas cognitive neuroscience laboratories at UCSD as he conducted the fMRI research outlined here, as well as the generous research support of a Nlli fellowship.

References Anderson, Michael 2003 Embodied cognition: A field guide. Artificial Intelligence 149: 91130.

50

Mark Johnson and Tim Rohrer

Brooks, Rodney A. 1991 Intelligence without representation. Artificial Intelligence Journal 47: 139-159. Brooks, Rodney A. and Anita M. Flynn 1989 Fast, cheap and out of control: A robot invasion of the solar system. Journal ofthe British Interplanetary Society 42: 478-485. Brooks, Rodney. A. and Lynn A. Stein 1994 Building brains for bodies. Autonomous Robots 1 (1): 7-25. Buccino, Giorgio, F. Binkofski, Gereon R. Fink, Luciano Fadiga, Leonardo Fogassi, Vittorio Gallese, Riidiger J. Seitz, Karl Zilles, Giacomo Rizzolatti and Hans-Joachim Freund 2001 Action observation activates premotor and parietal areas in a somatotopic manner: An fMRI study. European Journal of Neuroscience (2): 400-4004. Buonomano, Dean V. and Michael M. Merzenich 1998 Cortical plasticity: From synapses to maps. Annual Review ofNeuroscience 21: 149-186. Clark, Andy Being There: Putting Brain, Body and World Together Again. Cam1998 bridge, MA: MIT Press. Coslett, H. Branch, Eleanor M. Saffran and John Schwoebel 2002 Knowledge of the human body: a distinct semantic domain. Neurology 59: 357-363. DeBello, William M., Daniel E. Feldman and Eric I. Knudsen 2001 Adaptive axonal remodeling in the midbrain auditory space map. Journal ofNeuroscience 21 (9): 3161-3174. Deneubourg, Jean-Louis, Jacque Marie Pasteels and Jean-Claude Verhaeghe 1983 Probabilistic behavior in ants: A strategy of errors. Journal of Theoretical Biology 105: 259-271. Dewey, John 1925 Experience and Nature. In: John Dewey, The Later Works, 19251953, vol. 1, Jo Ann Boydston (ed.). Carbondale: Southern Illinois University Press, 1981. 1934 Art as Experience. New York: Penguin-Putnam Inc., reprinted 1980. 1938 Logic: The Theory of Inquiry, In: John Dewey, The Later Works, 1925-1953, vol. 12, Jo Ann Boydston (ed.). Carbondale: Southern Illinois University Press, 1981. Edelman, Gerald 1987 Neural Darwinism. New York: Basic Books. 1992 Bright Air, Brilliant Fire: On the Matter of Mind. New York: Basic Books.

We are live creatures

51

Erickson, Harold P., Dianne W. Taylor, Kenneth A. Taylor and David Bramhill 1996 Bacterial cell division protein FtsZ assembles into protofilament sheets and minirings, structural homologs of tubulin polymers. Proceedings of the National Academy of Sciences USA. 93 (1): 519523. Feldman, Jerome and Srini Narayanan 2004 Embodied meaning in a neural theory of language. Brain and Language 89 (2): 385-392. Fodor, Jerry Psychosemantics: The Problem of Meaning in the Philosophy of 1987 Mind. Cambridge, MA: MIT Press. Fogassi, Leonardo, Vittorio Gallese, Giorgio Buccino, Laila Craighero, Luciano Fadiga and Giacomo Rizzolatti Cortical mechanism for the visual guidance of hand grasping move2001 ments in the monkey: A reversible inactivation study. Brain 124 (3): 571-586. Fouts, Roger S., Mary Lee A. Jensvold and Deborah H. Fouts 2002 Chimpanzee signing: Darwinian realities and Cartesian delusions. In: Marc Bekoff, Colin AlIen and Gordon M. Burghardt (eds.), The Cognitive Animal: Empirical and Theoretical Perspectives in Animal Cognition. Cambridge, MA: MIT Press. 285-291. Fraser, Scott E. 1985 Cell interaction involved in neural patterning: An experimental and theoretical approach. In: Gerald Edelman, W. E. Gall and W. M. Cowan (eds.), Molecular Bases of Neural Development, 581-507. New York: Wiley. Gallese, Vittorio and George Lakoff 2005 The brain's concepts: The role of the sensory-motor system in reason and language. Cognitive Neuropsychology 22: 455-479. Gaze, R. M. and Sansar C. Sharma 1970 Axial differences in the reenervation of the goldfish optic tectum by regenerating optic nerve fibers. Experimental Brain Research 10: 171-181. Goldberg, Jeffrey L. 2003 How does an axon grow? Genes and Development 17 (8): 941-958. Grady, Joseph 1997 Foundations of meaning: primary metaphors and primary scenes. Ph.D. Dissertation, University of California. Guo, Yajin and Susan B. Udin 2000 The development of abnormal axon trajectories after rotation of one eye in Xenopus. Journal ofNeuroscience 20 (11): 4189-4197.

52

Mark Johnson and Tim Rohrer

Hauk, Olaf, Ingrid Johnsrude and Friedemann Pulvermfiller 2004 Somatotopic representation of action words in human motor and premotor cortex. Neuron 41 (2): 301-307. Hodges, Andrew 1983 Alan Turing: The Enigma. New York: Walker and Co. Hutchins, Edwin 1995 Cognition in the Wild. Cambridge, MA: MIT Press. James, William 1900 Psychology (American Science Series, Briefer Course). New York: Henry Holt and Company. Johnson, Mark 1987 The Body in the Mind: The Bodily Basis of Meaning, Imagination and Reason. Chicago: University of Chicago Press. Kawamura, Syunzo 1959 The process of subculture propagation among Japanese macaques. Primates 2: 43-60. Knudsen, Eric 1. 1998 Capacity for plasticity in the adult owl auditory system expanded by juvenile experience. Science 279 (5356): 1531-1533. 2002 Instructed learning in the auditory localization pathway of the barn owl. Nature. 417 (6886): 322-328. Lakoff, George 1987 Women, Fire and Dangerous Things. Chicago: University of Chicago Press. Lakoff, George and Mark Johnson 1980 Metaphors We Live By. Chicago: University of Chicago Press. 1999 Philosophy in the Flesh: The Embodied Mind and Its Challenge to Western Thought. New York: Basic Books. Lakoff, George and Rafael Nunez 2000 Where Mathematics Comes From: How the Embodied Mind Brings Mathematics Into Being. New York: Basic Books. Langacker, Ronald 1986 Foundations of Cognitive Grammar. 2 vols. Stanford: Stanford University Press. Lewkowicz, David J. and Gerald Turkewitz 1981 Intersensory interaction in newboms: modification of visual preferences following exposure to sound. Child Development 52 (3): 827832. Maturana, Humberto R. and Francisco J. Varela 1998 The Tree of Knowledge: The Biological Roots of Human Understanding, revised edition. (Robert Paolucci, trans. 1987 Arbol del conocimiento.) Boston: Shambhala Press.

We are live creatures

53

McGrew, William C. 1998 Culture in nonhuman primates? Annual Review ofAnthropology 27: 301-328. Meltzoff, Andrew N. and Richard W. Borton 1979 Intermodal matching by human neonates. Nature 282 (5737): 403404. Merzenich, Michael M., Randall J. Nelson, Jon H. Kaas, Michael P. Stryker, Max S. Cynader, A. Schoppmann and John M. Zook 1987 Variability in hand surface representations in areas 3b and 1 in adult owl and squirrel monkeys. Journal Comparative Neurology 258 (2): 281-296. Montgomery, James W. 1993 Haptic recognition of children with specific language impairment: effects of response modality. Journal Speech Hearing Research 36 (1): 98-104. Narayanan, Srini 1997 KARMA: Knowledge-based active representations for metaphor and aspect. Ph.D. Dissertation, University of California at Berkeley. Newell, Alan and Herbert Simon 1976 Computer science as empirical inquiry: Symbols and search. Communications ofthe ACM 19: 113-126. Regier, Terry 1996 The Human Semantic Potential Spatial Language and Constrained Connectionism. Cambridge, Mass.: MIT Press. Rizzolatti, Giacomo, Leonardo Fogassi and Vittorio Gallese 2002 Motor and cognitive functions of the ventral premotor cortex. Current Opinion in Neurobiology 12 (2): 149-154. Rohrer, Tim 2001 a Pragmatism, Ideology and Embodiment: William James and the philosophical foundations of cognitive linguistics. In: Rene Dirven, Bruce Hawkins and Esra Sandikcioglu (eds.), Language and Ideology: Cognitive Theoretic Approaches: Volume 1, 49-81. Amsterdam: John Benjamins. 2001 b Understanding through the Body: fMRI and of ERP studies of metaphoric and literal language. Paper presented at the 7th International Cognitive Linguistics Association Conference, July 2001. 2005 Image schemata in the brain. In: Beate Hampe (ed) From Perception to Meaning: Image Schemas in Cognitive Linguistics, 165-196. Berlin: Mouton de Gruyter. in press Embodiment and experientialism. In: Dirk Geeraerts and Hubert Cuyckens (eds), Handbook of Cognitive Linguistics. New York: Oxford University Press.

54

Mark Johnson and Tim Rohrer

Savage-Rumbaugh, Sue, Rose A. Sevcik and William D. Hopkins 1988 Symbolic cross-modal transfer in two species of chimpanzees. Child Development 59 (3): 617-625. Shelton, Jennifer R., Erin Fouch and Alfonso Caramazza 1998 The selective sparing of body part knowledge. A case study. Neurocase 4: 339-351. Sperry, Roger W. 1943 Effect of a 180-degree rotation of the retinal field on visuomotor coordination. Journal ofExperimental Zoology 92: 263-279. Stem, Daniel 1985 The Interpersonal World ofthe Infant. New York: Basic Books. Suziki, K., A. Yamadori and T. Fujii 1997 Category specific comprehension deficit restricted to body parts. Neurocase 3: 193-200. Tomasello, Michael, Sue Savage-Rumbaugh and A. C. Kruger 1993 Imitative learning of actions on objects by children, chimpanzees and enculturated chimpanzees. Child Development 64 (6): 1688-1705. Turing, Alan M. 1937 On computable numbers, with an application to the Entscheidungsproblem. Proceedings of London Mathematical Society, Series 2, 42:230-265 1950 Computing machinery and intelligence. Mind 59 (236): 433-460. van Gelder, Tim 1995 What might cognition be, if not computation? Journal ofPhilosophy 92 (7): 345-381. Varela, Francisco, Evan Thompson and Eleanor Rosch 1991 The Embodied Mind: Cognitive S~ience and Human Experience. Cambridge: MIT Press. Warrington, Elizabeth K. and Tim Shallice 1984 Category specific semantic impairments. Brain 107: 859-854. Winter, Steven 2001 A Clearing in the Forest: Law, Life and Mind. Chicago: University of Chicago Press. Ziemke, Tom 2003 What's that thing called embodiment? Proceedings of the 25th Annual Meeting of the Cognitive Science Society, 1305-1310. Lawerence Erlbaum.

Bringing the body back to life: James Gibson's ecology of agency

Alan Costall " [...] the self, or rather the soul, by which I am what I am, is entirely distinct from the body, is indeed easier to know than the body, and would not cease to be what it is, even if there were no body." (Descartes 1637 [1960]: 61).

Abstract In its supposed revolutionary move beyond "mechanistic behaviourism", modem psychology replaced the image of the passive subject with that of a highly active information processor modelled upon the general purpose computer. But such activity has been envisaged as essentially subcutaneous. For, in addition to the explicit and modem metaphor of the mind as computer, modem psychology simply retained the traditional mechanistic metaphor of the body as a stimulus-response machine. Thus, even though the ghost inside the machine has been replaced by a highly active "processor of information", the passive, mechanical body that harboured the ghost continues to structure psychological theory. Gibson's early work was an explicit attempt to develop a pure stimulusresponse psychology, but he eventually came to realize that this schema should not be merely repaired by invoking mediating cognitive processes but had, instead, to be rejected in its entirety. He replaced the stimulus-response formula still so fundamental to mainstream psychology with an "ecology of embodied agency": an exploration of the material conditions - affordances and information - that support our "being in the world". Keywords: embodiment, James Gibson, affordances, proprioception, information, nature-culture dualism, mutualism.

56

1.

A/an Costall

Introduction

Psychophysical dualism, as formulated by Rene Descartes (1596-1650) excluded the mind from the realm of natural science, and at the same time promoted the ambitious claim of the new mechanistic science to include everything else. Indeed, it was precisely by excluding mind from the physical order of things that the universal claim of the new science could be protected. For, as Margaret D. Wilson (1980: 42) has put it, an "immaterial principle is by definition not a part of 'nature''', and so does not count as a proper object for science. Thus the apparent intractability of mind to mechanistic treatment could hardly be deemed a failure on the part of science, but a reflection, instead, upon the "non-scientific" nature of mind itself. Descartes did, however, leave our bodies to science. And, since within Cartesian theory the mind was supposed to be the sole source of activity the sole mover - the body had to be regarded as not only separate and alien from ourselves, but also entirely inert. Falling squarely on the physical side of the Cartesian psychophysical divide, the body - along with the rest of extended matter - was relegated to the realm of the essentially passive. The body thus ends up in Cartesian theory much as it had first made its appearance within modem science, inert and alien - as a corpse. Modem psychologists would insist that psychological theory has come a very long way since Descartes. His dualisms of mind and body, and of mind and world, are, they would patiently explain, very much part of the prescientific history of our discipline, and nothing to do with present-day psychology. In this chapter, I will argue that a dualism between body and mind persists in modem cognitive psychology in a blindingly obvious way, and which amounts to a dualism more extreme and more systematic than anything to be found in the writings of Descartes. In a crucial sense, the body has largely disappeared from mainstream psychological theory, figuring as nothing more than a passive channel of "stimuli" and "responses". I will then turn to the work of one of psychology's most centrally placed and interesting misfits, James Gibson (1904-1979), who over a long career not only subjected mainstream psychology to sustained criticism, but sought to construct a non-dualistic alternative, which I have come to describe as an ecology of agency. Gibson is important to the current debates about embodiment because, by challenging the "stimulus-response" thinking that still constrains much of modem psychological theory, he has helped to

Bringing the body back to life: James Gibson's ecology ofagency

57

bring psychology's moribund body back to life - a revived body that is active, social and lived.

2. Was Descartes a "Cartesian"? So far, I have presented Descartes as a good, straightforward Cartesian. But, in fact, he was much more complicated and sensible than he appears in most of the standard textbook accounts. For example, it turns out that he was never very much bothered by the "problem of other minds" that would seem to ensue from a stark division of body and mind (Avramides 2001). Maybe he was not even a dualist (Baker and Morris 1996). Sometimes he even entertained the possibility that the mind might ultimately be understood in mechanistic terms (Cottingham 1992). Perhaps he was even a "proto-phenomenologist" (Tibbets 1973). My purpose here in this section is not to provide a review of the revisionist accounts of Cartesian philosophy that have appeared over the last fifty years or so. But I do want to draw attention to some important and surprising differences between "Cartesian dualism" as presented in the official potted histories, and the more subtle, possibly inconsistent, things Descartes himself had to say about the relation of mind and body. Descartes has most often been criticized for failing to provide an explanation of the relation of body and mind, where, by explanation, is meant a mechanical account of how the two relate. But, of course, that criticism completely misses his fundamental point, that the mind is not part of the physical order of things (Richardson 1982). Indeed, he makes very clear that he does not see that there is any general problem that needs explanation. The relation of body to mind is - and perhaps has to be - taken for granted. Mind and body have to be intimately related if we are to respond to injury and also to hunger and thirst with urgency and immediacy. As he strikingly put it in his Meditations: For nature also teaches me by these feelings of pain, anger, thirst, and so on, that I am not just lodged in my body like a pilot in his ship, but that I am intimately united with it, and so confused and intermingled with it that I and my body compose, as it were, a single whole. If this were not so, I should feel no pain when my body was injured; I should simply note the injury with my understanding, as a pilot sees with his eyes any damage done to his ship. (Descartes 1641 [1960]: 161)

58

Alan Costall

Evidently, Descartes was, at the very least, in two minds about dualism. Indeed, he eventually discloses that he did not regard the union of body and mind as a real problem at all, but one created by reflective thought. It cannot be understood in an external way, but can only be lived: [...] what belongs to the union of soul and body can be understood only in an obscure way either by pure intellect or even when the intellect is aided by imagination, but is understood very clearly by means of the senses. [...] it is just by means of ordinary life and conversation, by abstaining from meditating and from studying things that exercise the imagination, that one learns to conceive the union of soul and body. (Descartes 1643 [1971]: 279-280)

Clearly, there is a need to be wary about our supposed Cartesian past, but not simply to put it aside. It is precisely the distorted, mythical, "official" history of psychology, even about distant figures such as Descartes, that really counts when trying to understand historically how the current discipline of psychology has been structured. After all, agenda setting, including self-congratulation and the induction of new students, is what "official" disciplinary history is primarily about. The dodgy history is powerful historical stuff. So, here is my attempt to distil the already potted essence of the Cartesian dualism of mind and body as it exists within the collective memory of psychologists: 1. 2. 3.

3.

There is a stark ontological dualism between mind and body: they are entirely different kinds of "stuff'. The body is a passive stimulus-response mechanism.} The mind is the only active principle, and mediates between stimulus and response (in the case of voluntary action).

Psychology's missing body

Modem psychologists would certainly not identify themselves as Cartesians in the above sense. They would argue that they have emphatically rejected ontological dualism by developing naturalistic accounts of mind, primarily based upon analogies with reassuringly substantial devices, such as computers. Furthermore, given their rejection of stimulus-response be1. Descartes' place in the history of the reflex and of "stimulus-response thinking" within psychology is, again, a good deal more complex than the standard accounts imply (cf. Danziger 1983).

Bringing the body back to life: James Gibson's ecology ofagency

59

haviourism, they would insist they have also rejected the old mechanistic psychologies based on the concept of the reflex. And if this were all true, you would have to conclude that modem psychologists have undermined the old mechanistic psychology by a thoroughly new mechanistic psychology, based on analogies with new kinds of machines never envisaged by the early proponents of a mechanistic psychology. As Stephen Toulmin (1993: 146) has put it: If Descartes, Newton or Leibniz had been shown a late 20th century computer, [...] they could only have reacted by declaring: "That's not [what I would have called] a 'machine' at all."

Yet, it is not the case that mainstream cognitive psychology entirely replaced traditional mechanism. It retains the old mechanistic image of the body. The new mechanism of mind has been merely assimilated to the old dualism of mind and body, along with the existing conception of the body as a passive machine. This dualism is now, however, reformulated in terms of two radically different kinds of machines - a machine within a machine, a new mechanical mind implanted with the old mechanical body. However, we have all been so fixated upon how to theorize the mind in terms of the new mechanism that this retention of the old mechanistic schema of the body has been systematically overlooked. Ever since the rise of cognitive psychology, with its self-conscious rhetoric of Kuhnian revolution, its proponents have claimed to be rejecting stimulus-response psychology, whilst in the same breath reinstating its basic schema: The past few years have witnessed a noticeable increase in interest in an investigation of the cognitive processes. [...] It has resulted from a recognition of the complex processes that mediate between the classical "stimuli" and "responses" out of which stimulus-response learning theories hoped to fashion a psychology that would by-pass anything smacking of the "mental". (Bruner, Goodnow and Austin 1956: vii [emphasis added]) It seems obvious to us that a great deal more goes on between the stimulus and response than can be accounted by a simple statement about associative strengths. (Miller, Galanter and Pribram 1960: 9 [emphasis added]) [...] the dramatic shift away from behaviorism, which dominated the field for over thirty years, to cognitivism, .. [has] allowed one to study not just learning but memory, not just speech but language, and notjust stimulus and response but the processes that mediate them. (Hirst 1988: vii [emphasis added])

60

Alan Costall [...] in contrast to early behaviorists, but like their intellectual descendants such as cognitive psychologists and ethologists [... ], I believe that scientists can and should construct models for elucidating the knowledge and cognitive processes that connect stimulus and response. (Schiffer 1999: 8 [emphasis added])

In an early assessment of the cognitive revolution, Donald Hebb, in his presidential address to the American Psychological Association, claimed that the stimulus-response formula was indeed essential to the new cognitive approach because it served to define what psychologists could possibly mean by "cognitive": [...] the whole meaning of the term "cognitive" depends on [the stimulusresponse idea], though cognitive psychologists seem unaware of the fact. The term is not a good one, but it does have meaning as a reference to features of behavior that do not fit the S-R formula; and no other meaning at all as far as one can discover. The formula, then, has two values: fIrst, it provides a reasonable explanation of much reflexive human behavior, not to mention the behavior of lower animals; and secondly, it provides a fundamental analytical tool, by which to distinguish between lower (noncognitive) and higher (cognitive) forms of behavior. (Hebb 1960: 737)

Here is a very recent formulation of the modem "alternative" to stimulusresponse psychology by Rom Harre (2002: 104), an influential critic of mainstream psychology. He is using the following example of word recognition to make a much more general point about how we should theorize in psychology: Instead of the behaviorist pattern: Stimulus (retinal sensation) ~ Response (perception of word) we must have Observable stimulus (retinal sensation) together with unobservable Cognitive process ('knowledge utilization') ~ Observable response (recognition of word)

Cognitive psychologists keep presenting their own position as an alternative to stimulus-response behaviouristic psychology, when what they are actually offering is no more than an elaboration of that traditional framework. As Edward Reed has put it, cognitivism has largely ended up as the "flip-side" of stimulus-response behaviourism, where "mental processes" are defined as whatever "is left over after one tries to stuff all psychological phenomena into the S-R box" (Reed 1997: 267). And because of this

Bringing the body back to life: James Gibson's ecology ofagency

61

legacy, psychologists keep failing to recognize real alternatives when they come along, because they cannot understand alternatives to their own position as anything other than reversions to the traditional "pure" version of the stimulus-response framework. Thus, in their otherwise insightful overview of theory and research on visual perception, Bruce and Green (1990: 389) claim that the newer developments (including Gibson's ecological psychology) are giving rise to an important new debate "about the necessity for a cognitive or psychological level of theory to stand between the "stimulus" and the "response"". But why have cognitive psychologists been so slow to notice their retention of the stimulus-response schema? First of all, the very rhetoric about "cognitive revolution" and liberation from behaviourism has given rise to a remarkably long-lasting bout of intellectual complacency. In fact, modem cognitive psychology has retained a number of the central features of the behaviourism it claims to have replaced: the attempt to repair stimulus-response psychology by appeal to mediating processes (a central feature of neo-behaviourism), a commitment to the "hypothetico-deductive method", and also "methodological behaviourism", the assumption that the data of psychology are confined to meaningless "behaviour" (cf. Still and Costa111991; Leahey 1992; Neisser 1997: 248). Secondly, the stimulus-response schema also conforms perfectly to the procedures by which psychologists still conduct most of their research, and it is enshrined within those procedures. Thus the participants in experiments are "presented" with a stimulus or an "independent variable" and are required to respond to that stimulus. They are not expected to transform the "conditions" imposed upon them, nor take themselves off to quite a different one. For the duration of the experiment, participants agree, in effect, to "lend" their agency to the investigator. Admittedly, such a state of passivity is not purely an artifact of the laboratory - it is commonplace in our lives at work and school. But, fortunately, it is not our sole mode of being in the world. As George Kelly (who, like Gibson, was another member of psychology's "awkward squad") once warned: Behaviour is man's way of changing his circumstances, not proof that he has submitted to them. What on earth, then, can present-day psychology be thinking about when it says it intends only to predict and control behaviour scientifically? Does it intend to halt the human enterprise in its tracks? (cited by Westland 1978: 69).

It is, however, the standard metaphor of the computer that has given the stimulus-response formula (now reformulated in the language of input and

62

Alan Costall

output) its new lease of life, and, hence, made the body largely disappear from psychological theory (see also Costall 1991). First of all, the person/computer is purely an information processor, passively receiving an "input" (in other words, a stimulus), operating upon it, and producing output. Such a person/computer does not get about, and certainly does not have much of a life, and so has no need of a body, other than to support "cognitive processing" and to provide an "interface" to the outer world - a passive "channel" for both input and output. 2 There is also a surprisingly neglected theoretical aspect of the standard computer metaphor that has led to the disappearance of the body. This metaphor gives rise to a two-part division between the software (the program) and hardware (the actual computing machine). According to this reassuringly substantial metaphor of the computer, with its tidy division of program and machine, the business of the psychologist is neatly and conveniently circumscribed: to infer the software of the mind without reference to the "machinery" of the body. The software, after all, exists in quite a different order from the computer, and cannot be reduced to it, and so psychologists need no longer be anxious that they are engaged in a spurious and unscientific undertaking, nor fear that they might eventually be taken over by the biochemists or the neurologists. As one of its many enthusiasts proudly proclaimed, cognitivism provides us with "a science of structure and function divorced/rom material substance" (Pylyshyn 1986: 68 [emphasis added]). On the face of it, the current obsession within cognitive psychology about the neural localization of psychological functions would seem to represent a radical move against disembodied theorizing, yet this new kind of phrenology has done nothing to bring the body seriously back into psychological theory, since the "body" in question is nothing more than bits of the brain.

2. This standard computer metaphor has been succeeded by other computer models, but not, within cognitive psychology, superseded by them. Connectionism, which has been widely taken up within cognitive psychology, is generally presented as complementary to the classical, symbol-based, approach to information processing, and, in any case, also treats the body as passive in relation to the world. In contrast, the more recent research upon robotics and "autonomous agents" has made remarkably little impact within mainstream cognitive psychology, and is seldom mentioned in the textbooks. However, we cannot safely assume that even "the new AI" has escaped from the schema of the-machinewithin-the-machine (cf. Ziemke 2001).

Bringing the body back to life: James Gibson's ecology ofagency

63

But let us now look more closely at the other side of the standard computer metaphor, the hardware or computer. What precisely is the hardware supposed to represent? Presumably not the mind. But is it the entire body, or the brain, or, indeed, just those structures that specifically support cognitive processing? Psychologists have been remarkably unconcerned about this side of the analogy, and about the kind of theoretical work this side of the computer analogy is supposed to be doing. For, according to the ideal of the computer as a general purpose machine, the computer itself (assuming it is switched on and operating properly) imposes absolutely no constraints on how it actually functions. It is the program that entirely specifies how the machine will operate and this is why it offers the obvious and sufficient explanation of what is going on when the program is "running" a computer. 3 According to the standard computer metaphor, therefore, the body as computer ends up doing no explanatory work at all, any more than does the paper on which physicists write down their equations. And so, to the extent that theorists wish (within the terms of the traditional computer metaphor) to theorize the body as anything other than as a transparent, completely unconstraining entity, they would have to represent it as part of the software. To repeat, it is the program, according to the ideal of the computer as a general purpose machine, that is the only source of constraint! Finally, the theoretical focus of the cognitivist research project itself turns our attention resolutely away from the body and bodily activity. Even if what people are actually doing necessarily provides the data for psychology, their behaviour is not supposed to constitute the real object of inquiry: To take behavior as the focus of attention for psychology is as big an error as to take tracks in cloud chambers as the main object of study in particle physics. Such tracks are interesting only as clues to the existence of certain particles and to their properties. (Macnamara 1999: 241.)

What people can be observed to be doing can thus only be regarded as merely an indirect "clue" or index of an underlying mental apparatus, and 3. The virtue of programs as a language in which to formulate psychological theories lies precisely in the fact that they have to do all the theoretical work, just as, in astronomy, differential equations do all the explanatory work in accounts of planetary motion - though it does not follow that, to the extent that programs provide an effective language for theorizing in psychology, the objects of our theorizing are themselves necessarily "running programs", any more than the planets are staying on course by solving differential equations.

64

Alan Costall

is not, in itself, of any focal psychological interest. And, when you think about it, what people are required to do in most psychological experiments could be of no other interest, given that what they are usually doing is nothing more than just pushing buttons or else keys on computer keyboards. Such "responses" do not have any significance at all, except as indices with a meaning tied to the particular experiment in question. And, thus, once again, the structure of the psychology experiment not only represents but in turn reinforces the dualism of modem cognitivism: that behaviour, as bodily movement, has no intrinsic or manifest meaning, its meaning deriving, instead, from underlying and unobservable mental structures which are deemed to be "the main object of study" in the particular experiment in question. The disappearance of the body is also reflected in the illustrations that accompany the textbooks. For example, when Descartes describes how the body responds to injury, he provides us with a figure of a rather chubby naked boy with his toe dangerously close to a fire (Descartes 1664 [1966]: 271). When Warren (1922: 4), in his textbook, Elements of Human Psychology, discusses the stimulus-response arc, he presents the well-defined figure of a man dressed in "plus-fours", the fashionable leisurewear of the time. When Thurstone (1923: 355) discussed the "stimulus-response fallacy" he included an illustration of "our minds", but this mind still takes a definite bodily form, even if merely that of some amoeba-like creature. So, how does the body (dis)appear in the modem textbooks? As an outline box, of course: a schematic interface to the outside world, and a container of many other, much more interesting, boxes representing various cognitive modules and the connections between them. In modem psychology, the body, construed as a receptive stimulus-response machine, has atrophied through many years of intellectual neglect to a shapeless and abstracted container. Psychologists have been so busy mechanizing the mind, they forgot about the other side of their mechanistic theorizing, the stimulus-response body. James Gibson was an important exception.

4.

James Gibson and the ecology of agency

As a student at Princeton, Gibson was greatly influenced by Edwin B. Holt, who had, in turn, been taught and inspired by William James. Gibson was thus familiar with the Darwinian adaptationist orientation of American psychology. But, remarkably, during his early career at Smith College, he

Bringing the body back to life: James Gibson's ecology ofagency

65

was also in close contact with the Gestalt psychologists, Kurt Koffka, Fritz Heider and Kurt Lewin, who had emigrated from Europe. As a consequence, Gibson's approach brought together the functionalist emphasis upon the coordination of animal and environment with the Gestaltist reaction against atomistic analysis in favour of a relational, holistic approach. Gibson also drew upon many other more "exotic" intellectual sources, including Merleau-Ponty and Marx, though he could have been a good deal more forthcoming about such influences (Heft 2001; Reed 1988). His career was a continual project towards undermining the many dualisms within the human sciences, and his work a remarkably productive fusion of American functionalism, behaviourism, Gestaltism, phenomenology and his own remarkable obstinacy. As his wife, Eleanor Gibson approvingly put it, "he adored arguments". (cited in Szokolosky 2004: 277). He was not prepared to take "taken-for-granted assumptions" for granted, not even those once central to his own work. It may seem odd to introduce Gibson in this chapter, and at this point, since according to his most influential critics he was surely a throw-back to pure stimulus-response behaviourism (e.g. Gregory 1997). If they were referring to the Gibson of the 1940s and 1950s, their claim would have some justification. But, as I will explain, like Wittgenstein, there were at least two Gibsons: an "early" and a "later" one. Before the "cognitive revolution" (variously dated from the late 1950s to the early 1970s), Gibson (e.g. 1950, 1958, 1959) had already taken exception to the self-contradictory nature of psychological theorizing, with, on the one hand, its fundamental commitment to a mechanistic stimulusresponse psychology, and then, on the other hand, its attempt to dream up possible processes intervening between the stimulus and response that might possibly explain the lack of any lawful causal relation between the so-called stimuli and responses. He thought this whole business of invoking efficient causality between stimulus and response and then invoking a deus ex machina to "explain" why such efficient causality seldom held true was scientifically disreputable. As he put it, he had "no patience with the attempts to patch up the S-R formula with hypotheses of mediation. In behavior theory as well as psychophysics you either find causal relations or you do not" (Gibson 1967: 132). The "early Gibson" attempted to reinstate stimulus-response theory, or a "perceptual psychophysics", both by redefining the stimulus in a more holistic, Gestalt way, and also by explaining how the higher-order structure of the stimulus was, in turn, determined by the very structure of the envi-

66

Alan Costall

ronment itself. For example, in the case of a terrestrial environment, there is a ground surface extending around the animal which gives rise to a gradient of texture at the retina, and, in an uncluttered environment, also a "visible horizon". Even in his early writings, however, you can already find Gibson beginning to notice something fundamentally wrong with the stimulus-response formula. Having been concerned in the Second World War with aviation, he had come to place great emphasis upon what he later called "optic flow", the streaming of optical texture primarily as a result of our own activity in the world: The normal human being [..] is active. His head never remains in a fixed position for any length of time except in artificial situations. If he is not walking or driving a car, or looking from a train or airplane, his ordinary adjustments of posture will produce some change in the position of this eyes in space. Such changes will modify the retinal images in a quite specific way. (Gibson 1950: 117 [emphasis added])

Once we allow for the active nature of human beings and other animals, the "stimulus" can no longer figure as an efficient cause, nor be considered as the "starting point" of perceiving. After all, if we are to retain the old static language of stimulus and response, it is typically the "response" that precedes the "stimulus", and which gives rise to it. In other words, "one could say that the behavior is the first cause of all the stimulations" (MerleauPonty 1942 [1965]: 13). By the late 1950s, Gibson had rejected the stimulus-response schema entirely. People, he insisted, are not passive recipients of stimuli, except under conditions such as those of the psychological laboratory, where immobility is imposed upon us, either through the instructions or else more intrusively through the use of clamps: The headrest of the laboratory prevents the observer from turning his head and looking around [...]. It also, of course, prevents him from getting up and walking around. (Gibson 1979: 1)

However, when we are doing things, and even when just "observing" our surroundings, we are active not just in our heads (as much of modem theory still insists), but bodily. We are acting upon and exploring our surroundings. Thus, according to Gibson, the visual system, for example, does not just involve the eyes and a brain (cf. Gregory 1997), but must be defined functionally rather than anatomically. The eyes, which themselves are under muscular control, are part of a moving head, which, in turn, is set

Bringing the body back to life: James Gibson's ecology ofagency

67

on top of a body that gets around in the world. Thus, as Gibson liked to put it, the visual system also has legs. Indeed, when we bring an object to our eyes to inspect it more closely, our hands, from this functionalist perspective, should also be regarded as part of the visual system (Cowie 1993). Perversely, the textbooks routinely force Gibson into the category of a "bottom-up" theorist. In such theories, the "processing" of the stimulation is supposed to be completely "data-driven", whereas in "top-down" theories, the input is assumed to be subject to active interpretation or hypothesis construction (Cavanagh 1999). This modem-sounding distinction can, in fact, be found in Kepler's proposal, published in 1604, that the retinal image is the starting point of vision: In what manner this image or picture is brought together by the visual spirits which reside in the retina or in the nerves, and whether it is made to appear before the soul or tribunal of the faculty of vision by a spirit within the cerebral chambers, or whether the faculty ofvision, as a magistrate sent by the soul, goes out from the council chamber ofthe brain to meet this image in the optic nerves and retina descending to a lower court, these things I leave to the natural philosophers [.. ] for disputing. (cited by Straker 1976: 20 [emphasis added])

The underlying continuity between Kepler's account and most modem theories of vision concerns precisely the assumption that the body is a passive recipient of external stimulation. Now, if we really did spend all our lives just waiting for things to happen to us - as the participants in psychology experiments are typically required to do - then whatever "activity" is involved in perceiving would necessarily be confined to "internal processing" (cf. Ben Zeev 1984). Activity would necessarily be "subcutaneous". But this was the very assumption Gibson was rejecting. And this is why the very distinction between "bottom-up" and "top-down" is irrelevant to Gibson's later theory given he rejected the very concept of the stimulus. If we really must talk of Gibson in terms of "ups" or "downs", then Gibson is best regarded (according to a very nice slip of a student's pen) as a "bottom-down theorist". After all, Gibson regarded perceiving as a way of making "contact" with our surroundings, a "reaching out" into the world. 4 4. Arnheim (1956 [1969]: 33) held a rather similar view: "[...] vision is anything but a mechanical recording device. [...] Rather, in looking at an object, we reach out for it. With an invisible finger we move through the space around us, go out to the distant places where things are found, touch them, catch them, scan their

68

Alan Costall

This tactile conception of perceiving was captured in, and probably encouraged, by an important study conducted upon active touch (Gibson 1962; cf. Ikegami and Zlatev this volume). Participants were required to recognize various objects either when these objects were simply placed into their motionless hands, or else when they were given permission to explore them actively in their hands. In passive touch, there is the dull sense of "something" sitting on the surface of the hand: in active touch, there is the vivid sense of a coherent object passing between the palm and fingers. Gibson's contrast between active versus passive touch nicely encapsulates one of the radical shifts in his theoretical position. Having started from an explicit commitment to stimulus-response theory, Gibson was, in fact, one of the few experimental psychologists to reject entirely the mechanistic framework of traditional perceptual theory. After all, the very ideal of experimental investigation in psychology has surely become that of imposing conditions upon our "subjects" and then determining how they react. Seldom are they allowed to explore, let alone change, the situations in which they are placed. Yet, as Gibson came to realize, perceiving is an embodied activity, one involving skill and intelligence.

5.

Resources for agency

The two key concepts developed by Gibson in relation to his ecological approach are information and affordance. Consistent with Gibson's rejection of stimulus-response framework, these should not to be regarded as "efficient causes" but as resources for an embodied and active subject.

5.1.

Affordances

Let us start with Gibson's concept of affordances, a concept that was mainly set out, and then rather sketchily, in his final writings (e.g. Gibson 1979). Its purpose was to undermine the dualisms of the mental and physical, of meaning and materiality, of the world and us, that continue to structure that last outpost of scientism and individualism, modem cognisurfaces, trace their boundaries, explore their texture. It is an eminently active occupation."

Bringing the body back to life: James Gibson's ecology ofagency

69

tivist theory. Yet it is not as though its implications are restricted to that target, for, even when the human sciences, as in social constructivism and postmodernism, try to go their own way, they often manage to retain these traditional dualisms through a failure to engage seriously in a radical examination of all this "modernist" metaphysics. Thus, we find the anthropologist, Roy Ellen (1996: 31), having argued for a discursive view of nature to replace a scientistic one - and in a book devoted precisely to "redefining nature" - coming to the remarkably traditional conclusion that culture "emerges from nature as the symbolic representation of the latter." And we have the social constructivist, Stuart Hall, insisting that meaning is confined to a representational realm of symbols, in opposition to the material world: [...] it is not the material world which conveys meaning: it is the language system or whatever system we are using to represent our concepts. (Hall 1997: 25)

Gibson's concept of affordances attempted to undermine the dualisms of subject and object, matter and meaning by treating animal and environment as mutually defining: The affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill. [...] I mean by it something that refers to both the environment and the animal in a way that no existing term does. It implies the complementarity of the animal and the environment. (Gibson 1979: 127)

Affordances are, in essence, relationa1. 5 They concern the meanings of "things" in terms of what could be done with them, and hence implicate an agent. But these are not meanings that are "projected" onto things; they very much have to do with the nature of the objects involved. Take the example of stairs. Stairs are not stimuli. They do not force us to climb them. And there are a limitless number of things we can do with them: sit on them, break them up for firewood, and make a grand entrance down them. Woodworm can even live in and eat them. But the fact that we can do some many different things with things does not imply that we can do anything with anything. Thus to use stairs in their canonical way - for go5. Gibson was blatantly inconsistent on this point, sometimes insisting that affordances (and information) were independent of animals and thus undermining his attempt to go beyond subject-object dualism (cf. Costall 1995). I will return to this point.

70

Alan Costall

ing up and down them - requires (among many other things) that their risers and treads are of the appropriate dimensions. Beyond a certain critical point the stair no longer affords climbing. To ascend a staircase, we need to be able to reach the next step with our foot, and, furthermore, then be able to lift our body so that its weight is centered on that step. Going down stairs is more precarious, since we also need to check that we do not overdo things and end up in a painful fall. Normally, young children do have serious problems going up and down stairs. In fact, one can find age "norms" in the textbooks on motor development and developmental psychology, where the failure or success to achieve that norm is taken to reflect the intrinsic developmental condition of the child (Gesell, Ilg and Ames 1977). There are striking differences in the age norms for going down as opposed to going up stairs, and also for alternating the feet between steps as opposed to moving one foot forward and then gingerly following through by putting the other foot onto that same step. With the normal staircases that children normally encounter, it is not until they are around the late age of four and a half years that they begin to risk alternating their feet when descending. However, although the "climbability" of a staircase is a function of its dimensions, it also depends upon the size of the person in question. People do not come in standard sizes, and, in particular, young children are generally much smaller than adults. Yet these age norms have been based on "normal" staircases - stairs, in other words, designed for adults, not children. It is very curious that in our schools there are child-sized chairs and tables, but not child-sized stairs. In an inspired study, Josep Roca and his colleagues simply checked to see how children would cope with a scaleddown staircase where the steps were just 10 cm high and 20 cm. deep (Roca et al. 1986). The children coped remarkably well. The mean age at which they could climb either up or down the staircase was about twelve months, and alternating the feet between steps was achieved only slightly later at around eighteen months. Even this study did not make allowances for the fact that some of the participants were smaller than others, and not least because of their different ages. Yet it is the precise relation between the dimensions of the stair and of the individual user that is crucial. For example, according to Warren (1995), when the ratio of height of a step to leg length is greater than .88, it is simply no longer possible to step up onto it, and one must then resort to climbing with ones hands and knees. There is also a definite optimum ratio

Bringing the body back to life: James Gibson's ecology ofagency

71

where energy expenditure in climbing is least, and this, again, is a conjoint function of both the stair and the user. Under the specific conditions used in Warren's research, where the diagonal distance between successive stairs was held constant at 14 inches, the optimum ratio of riser height to leg length was .26. Let us take just one more example. The ways in which young children (and indeed other primates) are able to grasp objects has been of great interest to developmental psychologists, paediatricians and primatologists, especially the precision grip where the object is held between thumb and forefinger. Presumably in the sake of the serious pursuit of "scientific control", many of the main studies on grasping have used a target object with a single, standard size. Primates, however, including human children and indeed even adults, do not come in a standard size, and the graspability of the object surely depends on the relation between the object and the agent. Karl Newell and his colleagues presented both young children and adults with a range of cubes of different sizes and recorded the different kinds of ways they picked them up, for example between the flat of the palms, within the palm of one hand with all of the fingers grasping the object, and the prestigious precision grip (NewelI et al. 1989). When these investigators related the frequency of these various grip patterns, not to the absolute size of the cubes, but body-scaled to the individual participants, the transitions between different kinds of grip corresponded to definite ratios common to both the children and adults. Indeed, it was the bodilyscaled dimensions of the cubes, rather than age that accounted for most of the variability between the different grip patterns. Although the children in this study were older than the age at which children have been recorded as first being able to use the precision grip, the results suggest we need to be wary about age norms that make no reference to the child's relation to the test situation. There are critical points, therefore, concerning "body-scale" that define the limits within which we can act upon something in a particular way and beyond which we simply cannot. But scale is not the only issue, and nor does it function in isolation. For example, whether - and how - we might grasp an object depends upon a host of its other characteristics: its fragility, its slipperiness, its mass and also the distribution of its mass, its value, and so on. And what "holds" scale together with all of these other characteristics together and gives them meaning is the animal or person in question, though not so much as a "perceiver" (cf. Gibson 1979: 137; Heft 2001: 132; Ingold 2000: 168) but, much more fundamentally, as an agent:

72

Alan Costal!

[Affordances] have unity relative to the posture and behavior of the animal being considered. So an affordance cannot be measured as we measure in physics. (Gibson 1979: 127-128 [emphasis added])

This, then, is how the concept of affordances helps to undermine, rather than merely "bridge", the old psychophysical dualism. Affordances constitute the material resources for action but they do not fall on the far side of the material-mental divide. They are, as Gibson put it, both physical and mental, because they already implicate the needs and purposes of an agent, who, in turn, is envisaged as existing within - rather than beyond - the natural order of things (Heft 1989). There can be no question that Gibson intended affordances to include the culturally specific, as in his much discussed example of a postbox, which "affords letter-mailing to a letter-writing human in a community with a postal system" (Gibson 1979: 139). Yet for many in the human sciences, the culturally specific - the "conventional" - is somehow supposed to crowd out materiality. Yet, even though a postal system constitutes a highly specific human practice, materiality still matters. Postboxes, for example, as part of this system, have to perform the function of accepting and temporarily storing letters, but they also need to be distinguishable visually from litter bins and other kinds of things. And, of course, we do not discover what postboxes mean just by peering at them! It is through being a member of a community in which postboxes are actually used that we come to understand what they are supposed to afford, including being instructed by other people about how to use them, seeing other people using them, and, if all else fails, consulting a manual on how to use them. Indeed, if, as in the case of autism, we are somehow excluded from connecting with other people, the normal use of objects can be seriously disrupted (Williams, Costall and Reddy 1999).

5.2.

Information

Gibson's concept of information points to a second essential resource for effective agency: experiencing the world in relation to ourselves, including our bodily capacities. According to Gibson, the information available to perceivers is limitless and sufficient to specify the important features of our environments and our relations to it. Traditional perceptual theory, in contrast, not only stresses the poverty and unreliability of the available

Bringing the body back to life: James Gibson's ecology ofagency

73

information but also regards the perception of the world and of ourselves as quite separate. Gibson often quoted, approvingly, Bishop Berkeley's claim that vision enables animals "to foresee the damage or benefit which is like to ensue upon the application of their own bodies to this or that body which is at a distance" (Berkeley 1707 [1975]: 24; cf., for example, Gibson 1950: xiv, 1966: 156, 1979: 232).) There is now a substantial body of research with adults and children showing that we can be very effective in "foreseeing" affordances, for example, whether a staircase is climbable and even the optimum riser height for one" s own height, whether one can walk through a gap either with or without turning one's shoulders, or ducking one's head, and so on (e.g. Mark 1987; Warren 1984, 1995; Warren and Whang 1987). And we typically do this not by first visually "measuring" the object and only then comparing it to our own body. Rather, we experience the object in relation to ourselves. This is obviously the case, for example, when we are about to pick up an object. We do not just see the object nor just our hands, but the object and our hands in relation to one another. The relation of the object to our bodies is also specified in more subtle ways. Thus our eye-height is specified by the "visible horizon" (Gibson 1979: 162-164). Objects extending above the horizon are higher than our own eye-level, and the proportion in which it intersects the horizon corresponds to the height of that object relative to one's eye-height. Like the critical ratios for stepping on stairs, and picking up cubes, these horizon relations relative to the height of the point of observation, lack an objective metric, they are dimensionless and body-scaled. They directly relate to us, and are all the better for that: [...] the "knowledge" of his height that comes to the observer simply from living in his body is both more fundamental and more meaningful to him than the knowledge communicated to him by a statement such as "X is Y feet long". (Sedgwick 1973: 47)6

Complementarily, we develop, and sustain, a relatively stable sense of the limits and capacities of own bodies-in-relation-to-the-world in the very course of our activities (Stins, Kadar and Costall 2001). Interestingly (if

6. The variety of "anthropocentric" measurements of length is remarkable, being based on the size of the hand, the length or width of the fingers, the thumbnail, the fist and outstretched thumb, the foot, the pace of the legs, and so on (Klein 1975; Connor 1987).

74

Alan Costall

not surprisingly), this sense of our own bodies can be disrupted during periods of rapid growth (Heffernan and Thomson 1999).

5.3.

Proprioception.

In traditional theory, perceiving the world and perceiving oneself have been regarded as quite separate issues. Perceiving the world was supposed to be mediated by "exteroceptors" in the eyes, ears, nose and skin, and perceiving the self was supposed to be achieved through specialized "interoceptors" within our muscles, joints and the semicircular canals within the inner ear. Gibson regarded this division - based on a dualism of the objective and subjective - as simply mistaken: "The awareness of the world and one's complementary relations to the world are not separable" (Gibson 1979: 141). As we have seen, this complementarity is fundamental (if often tacit) in Gibson's discussion of the perception of affordances: perceiving what an object affords implicates a "subject" with certain motives and capacities. This matter of "reciprocity" of object and subject is focal, however, in Gibson's treatment of the proprioceptive function of vision: "our awareness of being in the world" (Gibson 1979: 239). The "external sense" of vision is richly informative about ourselves, and our relation to our surroundings (cf. also Butterworth 1995; Neisser 1994).7 The point is central to Gibson's account of "optic flow", the deformations of optical structure that derive from our own movements within the world. The flow structure is, Gibson argued, the basis for our "awareness of movement or stasis, of starting and stopping, of approaching or retreating, of going in one direction or another, and the imminence of an encounter" (Gibson 1979: 236). Similarly, the "visible horizon" expresses the reciprocity of self and environment: it "is neither subjective nor objective" (Gibson 1979: 164). Then, of course, we can literally see ourselves in the world. As I am typing this chapter, I can see my hands and arms, trunk and legs extending in front of me, and also, though less distinctly, the frames of my glasses, eye-brows and nose. Gibson included the famous illustration from Ernst

7. Compare Merleau-Ponty (1993: 37): "Every localization of objects in the world presupposes my locality. In a sense, an object of perception continuously speaks to us of ourselves. As incarnate subjects, we are expressed by the object."

Bringing the body back to life: James Gibson's ecology ofagency

75

Mach's Analysis ofSensations (1885 [1959]) in all of his three main books (Gibson 1950, 1966, 1979). It is a "view" of a man's office that includes the "visible ego" of the observer himself, including his nose and the end of his impressively long moustache. Now, as Hans Lubbe has noted, the implications of this simple image were, indeed, radical: This drawing is an illustrated criticism of the divorce of subject from object which renders theory of perception quite incapable of relating one to the other. The drawing demonstrates how, even in the simple process of seeing, there is no phenomenal world in which the subject itself is not already present, and that there is no subject which is not already present in the world. (Liibbe 1960 [1978]: 115.)

In a neat switch, Gibson also drew our attention to what we cannot see namely, ourselves restricting our own view: Ask yourself what it is that you see hiding the surroundings as you look out on the world - not darkness, surely, not air, not nothing, but the ego! (Gibson 1979: 112)

6.

Conclusion

In this chapter, I have tried to explain the importance of Gibson's critique of the passive and atrophied body behind much of psychological theory, and his attempt to provide a radically different kind of psychology based, not on "stimuli" and "responses", but on an ecology of agency - in other words, on the material resources for our effective and collective being in the world. Many of those who have become discontented with the restrictive cognitivist or representationalist thinking (cf. Johnson and Rorher this volume) that pervades the human sciences have been attracted to Gibson's challenge to the dualisms of subject and object, and of matter and meaning, that constrain much of modem - and post-modem - thought. But, as I have been arguing for some time now, we need to be wary of a fundamental inconsistency in Gibson's work: a vacillation between his insistence upon the mutuality of organism and environment and his retreat into a standard kind of realism that would exclude us from the world to be known (e.g. Costa111981, 1989, 1995,2003,2004; Costall and Still 1989). It is the realist version of Gibson that dominates not just the introductory textbooks (Costall and Morris 2004), but also the advanced literature. On this view, Gibson is indeed committed to a universalistic account of the

76

Alan Costal!

environment, including affordances and information. Thus, even Lakoff (1987: 216), someone highly appreciative of Gibson's emphasis upon embodiment, has claimed that "the Gibsonian environment is monolithic and self-consistent and the same for all people", and the ecological approach cannot make sense of "experiential or cultural categories". But these supposed limitations only follow if we lose sight of the principle of mutuality, and suppose, as many of Gibson's closest followers do, that "animals or humans do not enter the picture, except as a scale/actor" (Mace 1977: 50 [emphases added]). The mutualist alternative is to bring the animated body, the embodied agent, squarely into the picture. To repeat the crucial passage from Gibson that I used earlier: [Affordances] have unity relative to the posture and behavior of the animal being considered. So an affordance cannot be measured as we measure in physics. (Gibson 1979: 127-128 [emphases added])

This principle of relativity holds no matter how historically-specific or indeed idiosyncratic the activity happens to be. Indeed, it is likely that a good deal of what Gibson himself had to say about the visual control of locomotion could be most relevant to forms of high speed movement never experienced by humans before the relatively recent introduction of trains, cars and planes. Nevertheless, materiality still matters, and what the "later Gibson" in his mutualist mode was opening up was a view of materiality and culture as interpenetrating, rather than in opposition (see also Ingold 1996, 2000). Gibson was certainly important for his critique of the dualisms fundamental to so much of psychology and social theory. But as I have been trying to explain, he has also provided us with promising conceptual resources for going beyond those dualisms. There would be no point however in taking our existing unworldly conceptions of the social, the cultural and the symbolic and trying to tag them onto the body as presented to us by the realist Gibson. Although that body is no longer the passive stimulus-response machine of standard psychological theory, it is abstracted from any specific circumstances or history or identity. We would be left with more of the same: the opposition of the "human world" and the "natural world" that keeps leading the human sciences to suppose that the social and the cultural exist solely within our individual or collective heads, and that symbolism could be nothing more than the representation of a separate and inherently meaningless world (e.g. Ross 2004: 65).

Bringing the body back to life: James Gibson's ecology ofagency

77

The idea that we do not belong to the natural order of things has a very long past, extending well beyond Descartes, and it has been central to the metaphysics of modem science. Gibson is important because he has presented us with an alternative mutualist scheme in which we can at last see ourselves as part of nature - part of what nature has, and will, become. Affordances and information are, as Gibson claimed, "in" the world, but only in the sense that we are too. It is not just a question of spatial location. As John Dewey (1958: 295) insisted, we do not exist in the world "as marbles are in a box" but rather "as events are in history, in a moving, growing never finished process".

References Arnheim, Rudolph 1969 Art and Visual Perception. (Rev. ed.) London: Faber. [First published in 1956.] Avramides, Anita 2001 Other Minds. London: Routledge. Baker, Gordon and Morris, Katherine J. 1996 Descartes' Dualism. London: Routledge. Ben Zeev, Aaron 1993 The Perceptual System: A Philosophical and Psychological Perspective. New York: Peter Lang. Berkeley, George 1709 [1975] Philosophical Works including the Works on Vision. [Introduction and notes by M.R. Ayers.] London: Dent. Bruce, Vicki and Patrick Green 1990 Visual Perception: Physiology, Psychology and Ecology. (2nd edition) Hove, UK: Erlbaum. Bruner, Jerome, Jacqueline Goodnow and George Austin 1956 A study ofthinking. New York: Wiley. Butterworth, George 1995 An ecological perspective on the origins of self. In: Jose L. Bermudez, Naomi Eilan and Anthony Marcel (eds.), The Body and the Self, 87-105. Cambridge, MA: MIT Press. Cavanagh, Patrick 1999 Top-down processing in vision. In: Robert A. Wilson and Frank C. Keil (eds.), The MIT Encyclopedia of the Cognitive Sciences, 844845. Cambridge, MA: MIT Press.

78

Alan Costall

Connor, Robert D. 1987 The Weights and Measures of England. London: Her Majesty's Stationery Office (Science Museum). Costall, Alan 1981 On how so much information controls so much behaviour. In: George Butterworth (ed.), Infancy and Epistemology, 30-51. New York: St. Martin's Press/Brighton: Harvester Press. A closer look at 'direct perception'. In: Angus Gellatly, Don Rogers 1989 and John A. Sloboda (eds.), Cognition and Social Worlds, 10-21. Oxford: Clarendon Press. Graceful degradation: Cognitivism and the metaphors of the com1991 puter. In: Arthur Still and Alan Costall (eds.), Against Cognitivism: Alternative Foundations for Cognitive Psychology, 151-170. New York: Harvester-Wheatsheaf. Socializing affordances. Theory and Psychology 5: 467-481. 1995 From direct perception to the primacy of action: A closer look at 2003 James Gibson's ecological approach to psychology. In: Gavin J. Bremner and Alan M. Slater (eds.), Theories ofInfant Development, 70-89. Oxford: Blackwell. From Darwin to Watson (and Cognitivism) and back again: The 2004 principle of animal-environment mutuality. Behavior and Philosophy 32: 179-195. Costall, Alan and Paul Morris 2004 The Textbook Gibson: Misrepresentations of an anti-representationalist. European Workshop on Ecological Psychology, Torri del Benaco, Lake Garda, Verona, Italy, 26-29 June, 2004. Costall, Alan and Arthur Still 1989 Gibson's theory of direct perception and the problem of cultural relativism. Journalfor the Theory ofSocial Behavior 19: 433-441. Cottingham, John 1992 Cartesian dualism: Theology, metaphysics and science. In: John Cottingham (ed.), Cambridge Companion to Descartes, 236-257. Cambridge: Cambridge University Press. Cowie, Roddy 1993 On acting in order to see things better. Paper presented at the Workshop on "The Primacy of Action", International Society for Ecological Psychology, University of Manchester, 13-14 September, 1993. Danziger, Kurt 1983 The schema of stimulated motion: Towards a pre-history of modem psychology. History ofScience 21: 183-210.

Bringing the body back to life: James Gibson's ecology ofagency

79

Descartes, Rene 1960 Meditations. In: Discourse on Method. [A. Wollaston, trans.] London: Penguin Books. Original 1637. 1960 Discourse on Method. [A. Wollaston, trans.] London: Penguin Books. Original 1641. 1971 Correspondence with Princess Elizabeth. In: Elizabeth Anscombe and Peter Geach (eds.), Descartes: Philosophical Writings, 274-282. Indianapolis: Bobbs-Merrill. Original 1643. 1966 On mechanism in human action [translated by Tamar March from L'Homme]. In: Richard Herrnstein and Edwin Boring (eds.), A Source Book in the History of Psychology, 266-272. Cambridge, MA: Harvard University Press. Original 1664. Dewey, John 1958 Experience and Nature. New York: Dover. [Based on the Paul Cams lectures of 1925.] Ellen, Roy Introduction. In: Roy Ellen and Katsuyoshi Fukui (eds.), Redefining 1996 Nature: Ecology, Culture and Domestication, 1-36. Oxford: Berg. Gesell, Amold, Frances L. Ilg and Louise Bates in collaboration with Glenna E. Bullis 1977 The Childfrom Five to Ten. (Rev. ed.) New York: Harper and Row. Gibson, James 1950 The Perception ofthe Visual World. Boston: Houghton Mifflin. 1958 An interpretation of Woodworth's theory of perceiving. In: S. Georgene and John P. Seward (eds.), Current Psychological Issues: Essays in Honour ofRobert s. Woodworth, 39-52. New York: Holt. Perception as a function of stimulation. In: Sigmund Koch (ed.), 1959 Psychology: A Study of a Science, Vol. I, 456-501. New York: McGraw-Hill. Observations on active touch. Psychological Review 69: 477-491. 1962 1966 The Senses Considered as Perceptual Systems. Boston: Houghton Mifflin. Autobiography. In Edwin Boring and Gardner Linzey (eds.), A His1967 tory of Psychology in Autobiography, Vo!. 5, 127-143. New York: Appleton-Century-Crofts. 1979 The Ecological Approach to Visual Perception. Boston: Houghton Mifflin. Gregory, Richard 1997 Eye and Brain: The Psychology of Seeing, 5th ed. Princeton, NJ: Princeton University Press.

80

Alan Costall

Hall, Stuart 1997

Representation: Cultural Representations and Signifying Practices. London: Sage.

Harre, Rom 2002 Cognitive Science: A Philosophical Introduction. London: Sage. Hebb, Donald 1960 The American revolution, American Psychologist 15: 735-745. Heffeman, Dorothy and Thomson, James 1999 Gone fishin': perceiving what is reachable with rods during a period of rapid growth. In: Dorothy Heffeman and James A. Thomson (eds.), Studies in Perception and Action V, 223-228. Mahwah, NJ: Erlbaum. Heft, Harry Affordances and the body: An intentional analysis of Gibson's eco1989 logical approach to visual perception. Journal for the Theory of Social Behaviour 19: 1-30. Ecological Psychology in Context: James Gibson, Roger Barker and 2001 the Legacy of William James's Radical Empiricism. Mahwah, NJ: Lawrence Erlbaum Associates. Hirst, William 1988 Preface. In: William Hirst (ed.) The Making of Cognitive Science: Essays in Honor of George A. Miller, i-xi. Cambridge: Cambridge University Press. Ikegami, Takashi and Jordan Zlatev this vo!. From pre-representational cognition to language. Ingold, Tim Situating action V: The history and evolution of bodily skills. Eco1996 logical Psychology 8: 171-182. The Perception of the Environment: Essays in Livelihood, Dwelling 2000 and Skill. London: Routledge. Johnson, Mark and Tim Rorher this vo!. We are live creatures: Embodiment, American Pragmatism and the cognitive organism Klein, H. Arthur 1975 The World ofMeasurements. London: George AlIen and Unwin. Lakoff, George 1987 Women, Fire and Dangerous Things: What Categories Reveal about the Mind. Chicago: University of Chicago Press. Leahey, Thomas 1992 The mythical revolutions of American psychology. American Psychologist 47: 308-318.

Bringing the body back to life: James Gibson's ecology ofagency

81

Liibbe, Hans 1978 Positivism and phenomenology: Mach and Husserl. [Trans. A. L. Hammond.] In: Thomas Luckman (ed.), Phenomenology and Sociology, 90-118. London Penguin Books. [Article fust published in 1960.] Mace, William 1977 James J. Gibson's strategy for perceiving: Ask not what's inside your head, but what your head's inside of. In: Robert Shaw and John Bransford, (eds.), Perceiving, Acting and Knowing: Toward an Ecological Psychology, 43-65. Hillsdale, NJ: Erlbaum. Mach, Ernst The Analysis of Sensations and the Relation of the Physical to the 1885 Psychical. New York: Dover. Reprint 1959. Macnamara, John 1999 Through the Rearview Mirror: Historical Reflections on Psychology. Cambridge, MA: MIT/Bradford Books. Mark, Leonard 1987 Eyeheight-scaled information about affordances: A study of sitting and stair climbing. Journal of Experimental Psychology: Human Perception and Performance 13: 361-370. Merleau-Ponty, Maurice 1965 The Structure of Behaviour. [trans. Alden L. Fisher.] London: Methuen. [Original French edition published in 1942.] 1993 The experience of others (1915-1952). (F. Evans and H. J. Silverman, trans.) In: Keith Hoeller (ed.), Merleau-Ponty and Psychology, 33-63. New Jersey: Humanities Press. [First English translation of Merleau-Ponty's 1951-1952 lecture course L'Experience d'autrui, given at the University of Paris (Sorbonne).] Miller, George, Eugene Galanter and Karl Pribram 1960 Plans and the Structure of Behavior. New York: Holt, Rinehart and Winston. Neisser, Ulrich 1994 Self-perception and self-knowledge. Psyke and Logos 15: 392-407. 1997 The future of cognitive science: An ecological analysis. In: David M. Johnson and Christina E. Emeling (eds.), The Future ofthe Cognitive Revolution, 247-260. New York: Oxford University Press. Newell, Karl, Diedre Scully, F. Tenenbaum and S. Hardiman 1989 Body scale and the development of prehension. Developmental Psychobiology 22: 1-13. Pylyshyn, Zenon 1986 Computation and Cognition. Cambridge, MA: MIT Press.

82

Alan Costal!

Reed, Edward 1988 James Gibson and the Psychology of Perception. New Haven: Yale University Press. 1997 The cognitive revolution from an ecological view. In: David M. Johnson and Christina E. Emeling (eds.), The Future ofthe Cognitive Revolution, 261-273. New York: Oxford University Press. Richardson, Robert C. 1982 The "scandal" of Cartesian interactionism. Mind 91: 20-37. Roca, Josep, Martinez, Lizandra Mireia, Anna Fabregas Mireia and Anna Cordoner 1986 Registres evolutius motors: Una observaci6 critica. Apunts 6: 61-64. Ross, Norbert 2004 Culture and Cognition: Implications for Theory and Method. Thousand Oaks, CA: Sage. Schiffer, Michael Brian (with Andrea R. Miller) 1999 The Material Life of Human Beings: Artifacts, Behavior and Communication. London: Routledge. Sedgwick, Howard 1973 The visible horizon: A potential source of visual information for the perception of size and distance. Unpublished doctoral dissertation, Comell University, January 1973. Still, Arthur and Alan Costall (Eds.) 1991 Against Cognitivism. New York: Harvester Wheatsheaf. Stins, John, Endre Kadar and Alan Costall 2001 A kinematic analysis of hand selection in a reaching task. Laterality 6: 347-367. Straker, Stephen 1976 The eye made "other"; Durer, Kepler and the mechanisation of sight and vision. In: Louis A. Knafla, Martin S. Staum and T. H. E. Travers (eds.), Science, Technology and Culture in Historical Perspective, 7-25. University ofCalgary Studies in History, No. 1. Szokolosky, Agnes 2004 An interview with Eleanor Gibson. Ecological Psychology 15: 271281. Thurstone, Louis 1923 The stimulus-response fallacy in psychology. Psychological Review 30: 354-369. Tibbetts, Paul 1973 Historical note on Descartes' psychophysical dualism. Journal of the History ofthe Behavioral Sciences 9: 162-165.

Bringing the body back to life: James Gibson's ecology ofagency

83

Toulmin, Stephen 1993 From clocks to chaos: Humanizing the mechanistic world view. In: Herman Haken, Anders Karlqvist and Uno Svedin (eds.), The Machine as Metaphor and Tool, 139-153. Berlin: Springer-Verlag Warren, Howard 1922 Elements ofHuman Psychology. Boston: Houghton Mifflin. Warren, William and S. Whang 1987 Visual guidance of walking through apertures: Body-scaled information for affordance. Journal of Experimental Psychology: Human Perception and Performance 13: 371-383. Warren, William, Jr. 1984 Perceiving affordances: Visual guidance of stair climbing. Journal of Experimental Psychology: Human Perception and Performance 10: 683-703 1995 Constructing an econiche. In: John Flach, Peter Hancock, Jeff Caird and Kim Vicente (eds.), Global Perspectives on the Ecology of Human-Machine Systems, Vo!' 1,210-237). Hillsdale, NJ: Erlbaum. Westland, Gordon 1978 Current Crises in Psychology. London: Heinemann. Williams, Emma, Alan Costall and Vasu Reddy 1999 Children with autism experience problems with both objects and people. Journal of Autism and Developmental Disorders 29: 367378. Wilson, Margaret 1980 Body and mind from the Cartesian point of view. In: Robert W. Rieber (ed.), Body and Mind: Past, Present and Future, 35-55. New York: Academic Press. Ziemke, Tom 2001 The construction of 'reality' in the robot. Foundations of Science 6 (1): 163-233.

From the meaning of embodiment to the embodiment of meaning: A study in phenomenological selDiotics

Goran Sonesson A Qualisign /---I cannot actually act as a sign until it is embodied; but its embodiment has nothing to do with its character as a sign. A Sinsign / ---I involves a qualisign, or rather, several qualisigns. But these qualisigns are of a peculiar kind and only form a sign through being actually embodied. Charles S. Peirce, Nomenclature and Divisions of Triadic Relations

Abstract Unlike much of the contemporary discussion of embodiment, phenomenology is really involved with the body as a kind of meaning appearing to consciousness; and it does not only attend to the body of the biological organism, but also to the kind of organism-independent artefacts which are required by some sign systems. Because it is concerned with meaning, phenomenology is akin to semiotics. From the point of view of the latter discipline, however, signs must be distinguished from other meanings, and clear criteria are needed for doing so. At least one such criterion can by found in the work of Piaget: differentiation. Meaning in the more general sense of organisation and selection is at the basis of the common sense world, and thus accounts for what is known in Cognitive Linguistics as "image schemas". Cognitive Linguistics, just as biosemiotics, ignores this important distinction. Moreover, some cognitive linguists seem to deny the distinction between organism and environment, which must prevail if "image schemas" are to be acquired, along the lines of earlier conceptions of schematisation. On the basis of these considerations, a developmental sequence can be suggested going from schemas to signs and organism-independent artefacts. Keywords: body, ecology, embodiment, evolution, Lifeworld, memory, phenomenology, picture, semiotic function, semiotics, sign.

86

Goran Sonesson

1.

Introduction

In our time, in which the term "embodiment" is put to quite new (and, to my mind, either fuzzy or redundant) uses, authors such as Johnson (1987) and Varela, Thompson and Rosch (1991) have not failed to suggest a continuity with an earlier discussion of embodiment, taking place about a century ago, notably within phenomenological philosophy (e.g., Husserl 1973b; 1976). Yet these references to phenomenology seem to me to be fairly superficial, and the grasp of the phenomenological notion of embodiment shown often appears to be incomplete, if not inadequate. This is why I will start out by explaining the emergence of the problem of embodiment within phenomenological philosophy. Taking a cue from the phenomenologists themselves, I will also suggest that phenomenology may be interpreted as a branch of psychology, and thus serve as an ingredient of cognitive science as well as a basis for semiotic theory. From there on, my search for the multiple "bodies of the mind" will follow a somewhat spiralling movement: first, I will argue that the concept of sign or representation, which I take to be indispensable for our understanding of human consciousness, supposes something of a body of its own. Then we will see how meaning, which is not specifically embodied in signs, is a requisite, in both a systematic and an evolutionary sense, for the attainment of the sign function (Piaget 1945; Sonesson 1992b). I will go on to suggest that what is elsewhere known as "image schemas" (e.g. Johnson 1987; Lakoff and Johnson 1999, Johnson and Rohrer this volume) do indeed constitute a level of meaning prior to the sign but, for that very reason, are not directly involved in metaphors, which, in my view, and that of the tradition of classical rhetoric, must be construed as signs, and indeed signs standing for other signs (cf. Sonesson 1989, 1998b). Finally, we will look at embodiments of meaning in a rather different sense, of the kind which develops, phylogenetically and perhaps also ontogenetically, after the attainment of the (linguistic) sign, such as pictures, writing, and theories, that is, organism-independent sign-vehicles spanning time and/or space. My aim is not to exhaust the repertory of embodiments of meaning, but merely to expound some of their varieties, and to pinpoint their different evolutionary import.

From the meaning ofembodiment to the embodiment ofmeaning

2.

87

The Cartesian divide: Where angels fear to tread

In the philosophical tradition, embodiment emerges as a problem within the philosophy of consciousness, which aims to reconstruct the world as given to a (generic) subject. In this sense, embodiment gives rise to two separate strands in the particular version of the philosophy of consciousness inaugurated by Russerl, known as phenomenology: in relation to the physical body of the subject itself and/or his or her counterpart in perceptual space, the generic other; in relation to signs and other overarching structures, which, like the physical body, appear in the mind, without being of the mind, and seem to require some kind of physical substratum in order to exist.

2.1.

Phenomenology from the phenomenological point of view

The justification for a philosophy of consciousness is of course that in the common sense world, which Russerl later was to baptise the Lifeworld, everything there is is accessible to us through consciousness. The paradox is that, at the same time, the body, our own, as well as that of the other, cannot be a mere figment of consciousness. To paraphrase the classical dictum of 19th century psychology reemerging in the modem discussion of consciousness (cf. Dennett 1991), the body is not a mere epiphenomenon of consciousness. Indeed, this transcendence of our physical being to consciousness is itself part of the Lifeworld. As Max Scheler (quoted by Gurwitsch 1985) nicely put it, "we know that we are no angels", that is, no free-floating sprits without bodies. The second strand is quite different: genuine semiotic structures such as mathematical concepts, logic, and even language appear to transcend consciousness much in the mode of a Regelian "absolute spirit". They are, in Russerlean terms, "idealised" in order to be detached from their dependence on individual subjects - which is why they may harbour what Deacon (2003) has recently called "semiotic constraints", whose origin is independent of both nature and nurture. And yet, as Russerl (1962a: 365-386) recognised in his study of the origin of geometry, for the idealisation to be complete, its products have to be "embodied" in some kind of notational system, because only in that way can they gain a stable, public existence in a domain completely separate from their instantiations in the practical

88

Goran Sonesson

situations of the Lifeworld. More recently thinkers from separate traditions such as Ivins (1953), Innis (1950), and Donald (1991), have regained this insight in some form or other. The task of phenomenology, as Russerl saw it, was to explain the possibility of human beings having knowledge of the world; as a philosophical endeavour, phenomenology is about the way the world of our experience is "constituted". As a contrast, psychology is not about the world, but about the subject experiencing the world. Rowever, every finding in phenomenological philosophy, Russerl claims, has a parallel in phenomenological psychology, which thus could be considered a tradition within psychological science (cf. Russerl 1962b; Gurwitsch 1974). If consciousness is a relation connecting the subject and the world, then phenomenology is concerned with the objective pole and psychology is about the subjective one. It is often forgotten that Russerl not only inspired but himself was inspired by the Gestalt psychologists. Close followers of Russerl such as, most notably, Gurwitsch (1957, 1966), were as much involved with phenomenological psychology as with philosophy and discussed the findings not only of the psychology of perception but of contemporary contributors to neurobiology such as Gelb and Goldstein. Also the early Merleau-Ponty (1942, 1945),1 was, in this respect, an exponent ofphenomenological psychology. Many of those who are concerned with embodiment today appear to come from the diametrically opposite camp. Edelman (1992), for instance, clearly does not discover the body from the horizon of consciousness, but quite the opposite, he implies that the mind cannot be divorced from the body. In a sense, this is hardly controversial: unlike those hypothetical angels, human beings can only boast a mind as long as they have a body. But, if this is true in the order of existence, it is not necessarily so from the point of view of investigation. After all, Brentano (1885) did not use a scalpel, much less fMRI, to discover the property of intentionality (in the sense of directedness), which Edelman recognises as an irreducible characteristic of consciousness; nor did James (1890) find any of those "Jamesian properties" of consciousness repeatedly mentioned by Edelman in such a way. Indeed, far from being "a deliberately non-scientific set of reflections on consciousness and existence" (Edelman 1992: 159), phenomenology started out from the fact of intentionality and attempted to probe ever

1. Who may not quite deserve the hero status given to him by Varela, Thompson and Rosch (1991); See also Gallagher, this volume.

From the meaning ofembodiment to the embodiment ofmeaning

89

deeper into its ramifications, in order to rediscover and amplify those very Jamesian properties of consciousness mentioned by Edelman. Russerl and Gurwitsch may have been wrong to think of phenomenology as a discipline completely separate from biology and psychology, but the relative disconnection of phenomenological reflections, like those of Brentano and James, from biologicallmowledge has no doubt borne rich intellectual fruit. If "a biologically based theory of mind" can in some respects "invigorate" phenomenology, the opposite is certainly just as true. Interestingly, Edelman (1992; Edelman and Tonini 2000) claims that consciousness as such cannot be a spurious occurrence, because that would not have made evolutionary sense. That is, consciousness is not an epiphenomenon. But we have seen that, to classical embodiment philosophy, the problem is to show that the body is not an epiphenomenon.

2.2.

The science of common sense and its operations

The apparent paradox arises because, in the two cases, the point of view is entirely different. Phenomenology, like the science of semiotics, takes as its point of departure the way things make sense to us, that is, how they mean. In this very broad sense phenomenology accomplishes a semiotical reduction: things are considered only from the point of view of their having meaning to us (where "we" might be people of a particular culture or subgroup, or humankind in general)? From a phenomenological point of view, there is, in a sense, no way of overcoming the divide formulated by Descartes, for Descartes did not invent it: it is intrinsic to that phenomenon which, in Descartes' own words, is the most widely distributed one in the world, common sense. Common sense is not notorious for being right, but if we ask ourselves how the body (and the rest of the world) makes sense to us, then common sense is our very subject matter. Even so, common sense gives rise to an apparent contradiction: my body is necessarily experienced through my consciousness, but in my consciousness it is experienced as

2. Elsewhere (Sonesson 1989 26ft), I have opposed, in this sense, the qualitative reduction to the more familiar quantitative one, characteristic of the traditional natural sciences. There are similarities, but also differences, to the series of "reductions" distinguished by Husserl: the phenomenological and eidetic reductions, notably.

90

Goran Sonesson

being outside of it. 3 All post-Cartesian meditations, including those of Russerl (1973a) and those of Merleau-Ponty (1945), have been concerned to account for this paradox. To do so, it is necessary to accomplish a painstaking analysis (of which there can be no better example than the posthumous papers of Russerl himself, together with the - also largely posthumous - works of Peirce) of all those structures of the mind that are normally at the margin of consciousness (cf. below 5.2). In this sense, all human and social sciences which aspire to discover regularities, such as linguistics and other semiotic sciences, necessarily start out from phenomenology - and we should be happy if those phenomenological investigations sometime manage to be as meticulous as those of Russerl and Gurwitsch. Saussure famously observed that "linguistics and the other semiological sciences" are so difficult, because they are not concerned with anything material: indeed, he continued, their subject matter is the point of view we take on material things. Starting from this principle, Prieto (1975a: 140ff, 1975b: 215ft) has claimed, that, contrary to what is ordinarily taken for granted, it is natural science which is subjective, since it has to take a stand on physical reality, which as such is indifferent, whereas semiotics is capable of objectivity, in so far as it describes the subjective point of view of individuals and communities. According to another formulation, the object of linguistics is the knowledge common to the speaker and hearer (1975a: 110), i.e. it produces knowledge about knowledge, not, as the natural sciences, about the material world (1975a: 140ft). Prieto thus postulates a simple coincidence between the object and the discourse of semiotics. It is, however, less the phoneme, than the features defining it, which are relevant to linguistics, and these are not ordinarily identified by the speaker. In more recent linguistics, it is the "deep structure" or the "image schemas" which are claimed to be relevant for linguistic knowledge, not the particular syntactic form or stylistic turn, of which the speaker is usually aware. We therefore conclude that the linguist, and the semiotician generally, may have to descend at least one level of analysis below the ultimate level of which the user is aware. Put into traditional epistemological terms, we may say that after coinciding with the user in his understanding of the phoneme, the semiotician

3. Strictly speaking, this is not the problem of our own body, nor of the other, but the more general one of the external world, as pointed out by Gurwitsch (1979: 26f). Still, it is quite sufficient for us to note that it also applies to the body.

From the meaning ofembodiment to the embodiment ofmeaning

91

goes on to explain the conditions of possibility of this understanding on the level of distinctive features. In this case, semiotics contains the knowledge of the user and something more, and, quite apart from the problem of obtaining the correct understanding, this explicative part introduces an element of subjectivity. We shall say that what is of primary importance to semiotics is operative knowledge, i.e. knowledge that must exist at some, probably low, level of awareness, in order to render behaviour understandable (and thus explainable); thus, it is not discursive knowledge, the spontaneous theories of the user, which might be what we first tend to identify with common sense. The operation of ideation, familiar to the phenomenologist, the commutation text of structuralist linguistics, the grammaticality or acceptability judgement of the grammarian, and some varieties of psychological experimentation are all techniques for attaining these layers, bringing that which is at the margin of consciousness into its centre (cf. Sonesson 1989: 27ff; and see Zlatev, this volume, for a similar argument). In phenomenological semiotics, then, we are concerned, in the first place, with the figure of the body as it appears on the horizon of consciousness. Once we have described this figure - better than James, Russerl, and so on - we may try to explain it, delving ever deeper into the margins of consciousness. We can of course try to search for explanations outside of consciousness, but we must be aware that this is a complete change of direction. Most contemporary theories of embodiment do not appear to pose the question of meaning. Varela, Thompson and Rosch (1991) start out from the phenomenology of Merleau-Ponty, but, after the first few pages, it is not really clear how the issues they discuss relate to the phenomenological problem of the body, i.e. the body as it appears to consciousness. Lakoff and Johnson (1999: 102) distinguish three different levels of embodiment, which they refer to as "the neural level, phenomenological conscious experience and the cognitive unconscious", none of which, in the end, seems to have anything to do with meaning, as opposed to neurobiology.4 Both senses of embodiment characterised from a phenomenological perspective at the beginning of this section involve a process by which something not recognized as a body presents itself as a being one: in the first case, a mind is being situated in the world; in the second case an idea

4. See Zlatev this volume for a discussion of whether these levels can reasonably be separated, and, in particular, of the problematic character of the "cognitive unconscious".

92

Goran Sonesson

is being reified into an object publicly accessible to all. By denying the distinctions both between body and mind and expression and content, scholars such as Lakoff and Johnson deprive themselves of the very foundations needed by their own notion of "image schemas". To see this, however, we have to start by specifying the concepts of sign and schema.

3.

Meaning embodied in signs

It is true of both main traditions of semiotics, the Saussurean and the Peircean, that they have never really offered any specific definition of the sign - by which I mean a set of criteria permitting us to separate meanings which are signs from other meanings. The same thing appears to apply to the notion of representation in cognitive science (cf. Sonesson 1992b, 2003a, 2003b, 2006, forthcoming). This goes a long way to explaining why many semioticians (such as Greimas, Eco, etc.) have rejected the sign, without much of an argument, and why the second generation of adepts to cognitive science (e.g. Lakoff and Johnson 1999; Johnson and Rohrer this volume) now seem to be doing the same thing with reference to the notion of representation. So before we can pose any questions about the psychological and evolutionary role of the sign concept, we have to be clear about what it is. This involves not only deciding the criteria for analysing a phenomenon of meaning into two separate parts, but also those allowing us to posit an asymmetrical relation between these parts: not only does the expression have to be separate from the content, but the former should stand for the latter, not the reverse.

3.1.

From pebbles to feathers: The notion of differentiation

When Peirceans and Saussureans quarrel over the presence of two or three entities in the sign, they seldom pause to ask themselves what kind of objects, defined by what type of features, are involved. The whole question becomes moot if there is no reason to analyse meaning into two parts, as suggested by both contemporary cognitive scientists and old-time existentialists and Lebensphilosophen. What, then, is it that permits us to determine that an object endowed with meaning is made up of an expression, or "representamen", and a content, or "object" (where further instances of the Peircean version are not relevant)? Peirceans and Saussureans alike would

From the meaning ofembodiment to the embodiment ofmeaning

93

no doubt agree that signs have something to do with the classical formula, often quoted by Jakobson (1975), aliquid stat pro aliquo ("something in the place of something else"), or, as, Jakobson also puts it, more simply, with renvoi, or reference. But this formula itself is vague or ambiguous. Before we can separate signs from other meanings, we have to spell out those criteria for something being a sign that are simply taken for granted, both in the Peircean and in the Saussurean tradition. This can be done by combining what Russerl says about appresentation (something which is directly present but not thematic refers to something which is indirectly present but thematic) and what Piaget says about the semiotic function (there is a differentiation between expression and content in the double sense, I take it, that they do not go over into each other in time and/or space, and that they are perceived to be ofdifferent nature). Phenomenology, which is not afraid of spelling out the self-evident, may offer some help here. Saint Augustine, who has often (as so many others) been hailed as the first semiotician, defined the sign as "a thing which, over and above the impression it makes on the senses, causes something else to come into thought as a consequence" (as translated by Deely 1982: 17ff). Russerl's (1913, 1939) own definition of the sign, which describes the expression as something which is directly perceived but not in focus, and the content as being indirectly perceived while at the same time being the focus of the relation, could be taken as a way of specifying the Augustinean suggestion. 5 Piaget certainly abides by Saussure opposing the sign to the symbol (where the latter is the motivated sign). What Piaget added to Saussure was most obviously a developmental perspective, in particular on the level of ontogeny. But, just as importantly, though it has seldom been observed (cf. Sonesson 1992b, etc.), he realised that not all meanings are signs or symbols, and he even began groping for a definition of that which accounts for the specificity of the sign. According to Piaget the sign function (which Piaget himself called first the symbolic, and then the semiotic function) is a capacity acquired by the child at an age of around 18 to 24 months, which enables him or her to imitate something or somebody outside the direct presence of the model, to use language, make drawings, play "symboli5. These observations could be taken to imply that the content is "embodied" in the expression. Expression would stand to content as body to soul. This was explicitly suggested by Cassirer (1957: 100), but it is also hinted at in some passages by Peirce. The parallel is nonetheless, in my view, seriously flawed (as will be discussed in Section 3).

94

Goran Sonesson

cally", and have access to mental imagery and memory. The common factor underlying all these phenomena, according to Piaget, is the ability to represent reality by means of a signifier, which is distinct from the signified. Indeed, Piaget argues that the child's experience of meaning predates the sign function, but that such meaning does not suppose a differentiation of signifier and signified (see Piaget 1945, 1967, 1970). In several of the passages in which he refers to the sign function, Piaget goes on to point out that "indices" and "signals" are possible long before the age of 18 months, but only because they do not suppose any differentiation between expression and content. The signifier of the index, Piaget (1967: 134ft) says, is "an objective aspect of the signified"; thus, for instance, the visible extremity of an object which is almost entirely hidden from view is the signifier of the entire object for the baby, just as the tracks in the snow stand for the prey to the hunter. But when the child uses a pebble to signify candy, he is well aware of the difference between them, which implies, as Piaget tells us, "a differentiation, from the subject's own point of view, between the signifier and the signified" (ibid.) Piaget is quite right in distinguishing the manifestation of the sign function from other ways of "connecting significations", to employ his own terms. Nevertheless, it is important to note that, while the signifier of the index is said to be an objective aspect of the signified, we are told that in the sign and the "symbol" (i.e. in Piaget's terminology, the conventional and the motivated variant of the sign function, respectively) expression and content are differentiated from the point of view of the subject. Curiously, this distinction between the subjective and objective points of view is something Piaget seems to lose track of in his further discussion. We can, however, imagine this same child that in Piaget's example uses a pebble to stand for a piece of candy having recourse instead to a feather in order to represent a bird, or employ a pebble to stand for a rock, without therefore confusing the part and the whole: then the child would be employing a feature, which is objectively a part of the bird, or the rock, while differentiating the former form the latter from his point of view. Only then would he be using an index, in the sense in which this term is employed in semiotics, that is, in (what this semiotician takes to be) the Peircean sense of the term. Contrary to what Piaget implies, the hunter, who identifies the animal by means of the tracks, and then employs them to find out which direction the animal has taken, and who does this in order to catch the animal, does not, in his construal of the sign, confuse the tracks with the animal itself, in which case he would be satisfied with the former. Both the

From the meaning ofembodiment to the embodiment ofmeaning

95

child in our example and the hunter are using indices, or indexical signs, where the "real" connection is transformed into a differentiation. On the other hand, the child and the adult will fail to differentiate the perceptual adumbration in which they have access to the object from the object itself; indeed, they will identify them, at least until they change their perspective on the object by approaching it from another vantage point. And at least the adult will consider a branch jutting out behind a wall as something that is non-differentiated from the tree, to use Piaget's example, in the rather different sense of being a proper part of it. 6 In the Peircean sense an index is a sign, the relata of which are connected, independently of the sign function, by contiguity or by that kind of relation that obtains between a part and the whole (henceforth termedfactorality). But of course contiguity and factorality are present everywhere in the perceptual world without as yet forming signs: we will say, in that case, that they are mere indexicalities. Perception is perfused with indexicality.7 An index, then, must be understood as indexicality (an indexical relation or ground, to use an old Peircean term) plus the sign function. Analogously, the perception of similarities (which is an iconic ground) will only give rise to an icon when it is combined with the sign function. Deacon (1997: 76ft) must therefore be wrong when he claims that camouflage in the animal world such as the moth's wings being seen by the bird as "just more tree" are essentially of the same kind as those "typical cases" of iconicity we are accustomed to call pictures. As always, there are passages in Peirce's work which may be taken in different ways, but it makes more systematic and evolutionary sense to look upon iconicity and indexicality as being only potentials for something being a sign which still have to be "embodied" (cf. Section 4). While the introduction of the notion of differentiation is a substantial accomplishment on the part of Piaget, he unfortunately never spells out its import. He defines differentiation in terms of the subject's point of view, but then uses examples in which the disconnection already exists objectively, as pointed out above. Objectivity can here, I take it, be identified

6. About proper parts, perceptual perspectives, and attributes as different ways of dividing an object and thus different indexicalities, cf. Sonesson 1989: 1.2.). 7. I am using "indexicality" here (just as "iconicity") in the sense of something which is necessary for a sign being an index (or an icon), but which cannot function "as a sign until it is embodied". See, in particular, Sonesson (1993a, 1998a, forthcoming)

96

G6ran Sonesson

with the common sense world, which the child, in Piagetean terms, is in the process of "constructing". Differentiation should not be identified with displacement as defined by Hockett (1977), which (rightly, no doubt) appears as one of the "design features" of language in most introductory textbooks. As in the case of the tracks left by the hunted animal, displacement may be a consequence of differentiation. But differentiation only comes on its own when the sign is in presence of its referent, for then it allows us to construe reality in different ways ("subjectively", as Piaget would have said), picking out that which is relevant, and ignoring, or downplaying other features. We must be careful not to confuse different relationships involving the sign. Differentiation, in Piaget's sense, must pertain to the signifier and the signified, which are always equally present in the here and now of the sign usef, since they are mental (or, in some cases, intersubjective) entities. To the hunter, both the signifier and the signified of the tracks are present here on the ground (Of, to be precise, on the ground as he perceives it). But the signified contains the information that is itself only part of a larger whole (or rather something once contiguous to a larger whole) which was present here at an earlier time, but which is now elsewhere, more precisely in the direction indicated by the tracks. And the displacement, in Hockett's sense, has taken place between that signified whole and the real animal, which is now present somewhere else. When the sign, whether it is a stretch of discourse, a picture, or an animal track, is present along with the referent, however, the signified allows us to refocus the referent, in other words, to present it in a particular perspective. For this the sign requires independence: that is so say, a "body" of its own.

3.2.

Some other ways of "connecting significations"

As presented here, the concept of sign (or representation) does not include ordinary perception: our way of being in the world is not to be likened to the presence at some kind of private theatre. Second generation cognitive scientists (cf. Johnson and Rohrer this volume) are therefore quite right in rejecting the notion of representation of their forbears. They are wrong, however, to reject all kinds of representation (to the extent that it corresponds to the sign function). More fundamentally, they commit a serious error by not attending to the definition of representation before rejecting it

From the meaning ofembodiment to the embodiment ofmeaning

97

altogether. A few notions of history may help us to disengage ourselves from the present-day conceptual muddle. As was noted above, Augustine seems to have been responsible for making explicit the common sense notion of sign on which later thinkers such as Saussure and Russerl (and, at least in his definitions, Peirce) are tacitly building: it is, he tells us (in the convenient paraphrase of Deely 1994: 58) "something which, on being perceived, brings into awareness another besides itself'. Thomas Aquinas already had some misgivings about this definition, without ever daring to reject it outright. The followers of Aquinas in Paris may have been somewhat bolder. In a written form which has come down to us, however, we first know this criticism from the works of Pedro da Fonseca, who was active in Coimbra, Portugal, in the 16th century. To Fonseca and his followers, the definition of the sign must be considerably broader: a sign is anything which serves to bring into awareness something different from itself, whether the sign (in the sense of the signifier) itself becomes subject to awareness in the process or not. If the sign itself does not have to be perceived in order for us to come to an awareness of that which is signified, Fonseca described it as beingformal; but if the sign cannot lead to the awareness of anything at all unless it is itself perceived, he called it instrumental (cf. Deely 1982: 52ff, 1994: 58ff). Thus, Fonseca pointed to a distinction, which seems to have been lost by latter-day semioticians and cognitive scientists. What is here called an instrumental sign clearly is that which we, following Russerl and Brentano, but also Edelman, have described as the fundamental trait of consciousness, intentionality, that is, the property of being directed to that which is outside of consciousness. In fact, when closely considered, Fonseca's observations really go against the grain of the identification of our awareness of the world with the sign. It echoes Russerl's as well as Gibson's description of the perceptual act as something which points beyond itself without itself being present to consciousness (cf. Sonesson 1989: ill.3.2). Indeed, when Gibson (1978: 228) observes that, when we are confronted with the cat from different points of view, etc., what we really see is all the time the same invariant cat, he actually recovers the central theme of Russerlian phenomenology, according to which the object is entirely, and directly, given in each of its perspectives or noemata (see Russerl 1939, 1962a, 1962b, 1973b; Sonesson 1989: 1.2.2). In a similar fashion, Russerl's favourite example is a cube which can be observed from different sides. In Gibsonean terms, these are "the surfaces of the world that can

98

Goran Sonesson

be seen now from here" (Gibson 1978: 233). Husserl's cube and Gibson's cat instantiate the same phenomenal fact. Just as Husserl called into question the conception of his contemporary Helmholtz, according to which consciousness is like a box within which the world is represented by signs and images, from whose fragmentary pieces we must construct our perceptions (cf. Kiing 1973), so Gibson's strawmen are the followers of Helmholtz, the so-called "constructionists" (who have recently re-emerged within cognitive science; cf. Hoffman 1998), who claim that hypotheses are needed to build up perceptions from the scattered pieces offered us by sensation (cf. Sonesson 1989: 111.3.3).8 Husserl rejected the picture metaphor of consciousness, showing Brentano and Helmholtz to be in error in their very conception of pictures and other signs, because they ignored the transparency of the expression to the content. Gibson (1978) instead emphasises the dissimilarity of the picture from a real-world scene, thus showing numerous experiments using pictorial stimuli to study normal perception to be seriously misguided. To both Husserl and Gibson, normal perception gives direct access to reality; pictures, however, constitutes a kind of indirect perception to Gibson, while to Husserl (1980) they are "perceptually imagined" (cf. Sonesson 1989: ill.3.6, forthcoming). To perceive surfaces is a very different thing from perceiving marks on surfaces, Gibson (1980) maintains. Depth is not added to shape, but is immediately experienced. In fact, the perception of surfaces, of their layout, and of the transformations to which the latter are subjected, is essential to the life of all animal species, but the markings on these surfaces have only gained importance to man, notably in the form of pictures (Gibson 1980: xii, 1978: 229). Surfaces have the kind of meaning which Gibson elsewhere calls "affordances"; the markings on surfaces, however, have "referential meaning". Without discussing the exact import that should be given to the term "affordance" (cf. Costall this volume), we may safely conclude that "referential meaning" is a property of what we have called the sign function. That is, surfaces do not standfor other surfaces, but the markings on surfaces may possibly do so. The pattern of a surface and the pattern on a surface are different, and can usually be distinguished by an adult. The

8. Reed (1996) notes some parallels between Gibson and the American pragmatists (without, however, referring to Peirce). On Gibson's sources, also see Costall this volume.

From the meaning ofembodiment to the embodiment ofmeaning

99

surface on which a "graph" has been executed can be seen underneath the "graph". To Gibson, then, the picture is a surface among other surfaces before becoming a sign. Gibson (1978: 231) observes that, besides conveying the invariants for the layout of the pictured surfaces, the picture must also contain the invariants of the surface that is doing the picturing: those of the sheet of paper, the canvas, etc., as well as those of the frame, the glass, and so on. Although Gibson does not use the term, he clearly describes the picture as a sign, in the strict, Augustinean sense of the word: as a surface which, on being perceived, brings into awareness something besides itself. Gibson never specifies what he means when he claims that surfaces are only seen to stand for something else by (adult) human beings, in contradistinction to animals and infants. If he meant to suggest that surfaces can never be taken to be something else than surfaces by animals and children he was clearly wrong: we know that even doves may react the same way to a picture as to that which is depicted (cf. Sonesson 1989: ill.3.1). The difficulty, clearly, consists in seeing, at the same time, both the surface and the thing depicted. We should grant Fonseca the insight that there is some kind of analogy between signs and intentional acts. However, to use the term sign in both cases dangerously suggests that there is no important distinction to be made. In his late life, Peirce realised that all his notions were too narrow: instead of "sign", he reflected, he really ought to talk about "medium" or "mediation" (manuscript quotations given in Parmentier 1985). In the following, we will use the term mediation for this general sense of meaning which Fonseca called sign and to which Peirce sometimes also may be hinting. In some respects, at least, it seems to correspond to Gibson's "affordances", and to Piaget's notion of "connecting significations".

4.

On the way to the human Lifeworld

If there is meaning before signs, then even the immediate experience of perception is in some very general sense "mediated". The semiotician A. J. Greimas (1970: 49) once suggested that there could be a cultural science of nature, a semiotics of the natural world - which was concerned, then, with the world that is natural to us, just as a particular language is our "natural language". But Greimas was not the first to conceive of a cultural science of nature. His semiotics of the natural world, together with Husserl's sci-

100

Goran Sonesson

ence of the Lifeworld, and "ecological physics" as invented by Gibson are all sciences of normality, of that which is so much taken for granted that it is ordinarily not considered worthy of study (cf. Sonesson 1989, 1994, 1996,). It may seem strange to put together ideas and observations made by a philosopher, a psychologist, and a semiotician; yet these proposals are largely the same; indeed, there are indications that both Greimas and Gibson took their cue from Russer!' Greimas, Gibson, and Russerl all felt the need for such a science because they realised that the "natural world", as we experience it, is not identical to the one known to physics but is conceived from the standpoint of human consciousness. Russerl's Lifeworld as well as Gibson's ecological physics, but not Greimas' natural world, take this level to be a privileged version of the world, "the world taken for granted", in Schiitz's (1967) phrase, from the standpoint of which other worlds, such as those of the natural sciences, may be invented and observed (cf. Sonesson 1989: 26-29, 30-34, and passim).

4.1.

The ecology taken for granted: the Lifeworld

Every particular thing encountered in the Lifeworld is referred to a general type. According to Schiitz ([1974] 1932, 1967), other people, apart from family members and close friends, are almost exclusively defined by the type to which they are ascribed, and we expect them to behave accordingly. Closely related to the typifications are the regularities that obtain in the Lifeworld, or, as Russerl's says, "the typical way in which things tend to behave". This is the kind of principles tentatively set up which are at the foundation of Peircean abductions. Many of the "laws of ecological physics", formulated by Gibson (1982: 217ft), and which are defied by magic, are also such "regularities [that] are implicitly known": that substantial objects tend to persist, that major surfaces are nearly permanent with respect to layout, but that animate objects change as they grow or move; that some objects, like the bud and the pupa transform, but that no object is converted into an object that we would call entirely different, as a frog into a prince; etc. Some of the presuppositions of these "laws", such as the distinction between "objects that we would call entirely different", are also at the basis of the definition of the sign function (cf. Sonesson 1992a, 2000, 2001).

From the meaning ofembodiment to the embodiment ofmeaning

101

It has been suggested (notably by Smith and Varzi 1999) that the Lifeworld, in this sense, is simply the niche, in the sense of (non-Gibsonean) ecology, in which the animal known as the human being stakes out his life (cf. Sonesson 2001: 99). The niche, then, in this sense, is the environment as defined by and for the specific animal inhabiting it. In Husserlean language, the niche is subjective-relative - relative to the particular species. The precursor of the niche, understood in this way, is the notion of Umwelt introduced by von Uexkiill (1956, 1973), which is one of the key concepts of the field known as biosemiotics (see Emmeche this volume). Uexkiill's notion of meaning centres on the environment, the Umwelt, which is differently determined for each organism. As opposed to an objectively described ambient world, the Umwelt is characterised for a given subject, it terms of the features of the world which the subject perceives (Merkwelt) and the features which it impresses on the world (Wirkwelt), which together form a functional circle (Funktionskreis). According to a by now classical example, the tick hangs motionless on a branch until it perceives the smell of butyric acid emitted by the skin glands of a mammal (Merkzeichen), the effect of which is to send a message to its legs to let go of the support (Wirkzeichen). When the tick drops onto the body of the mammal, a new cycle is started, because the tactile cue of hitting the mammal's hair incites the tick to move around in order to find the skin of its host. Finally, a third circle is initiated when the heat of the mammal's skin triggers the boring response, which permits the tick to drink the blood of its host. Together, these different circles consisting of perceptual and operational cue bearers make up the interdependent wholes of the subject, corresponding to the organism, and the Umwelt, which is the world as it is determined for the subject in question. Scholars involved with biosemiotics tend to take this model, immensely enlightening as it is in itself, and simply project onto it the sign conception suggested by Peirce. The first difficulty with this approach, of course, resides in finding out the real import of the Peircean sign conception. Since this is in itself an infinite task, any scrutiny of the parallel risks getting bogged down very early on. If we confront the sign conception defined in this chapter with the world of the tick, however, it will be easy to see that the two are entirely distinct. Not only is there no distinction between expression and content to the tick; there is no separation of sign and reality. At least in part, this is also an opposition between the Umwelt and the Peircean SIgn.

102

Goran Sonesson

4.2.

From Umwelt to Lebenswelt: the thematic field

Pending the invention of biosemiotics, Cassirer (1942: 29ff, 1945: 23ft) was no doubt the first thinker outside of biology to take von Uexkiill's ideas seriously. After pointing out that, to human beings, all experience is mediated (a case of Vermittlung), he observed that this is also true of animals, as described by von Uexkiil1. But he makes no mention of the fact that, to von Uexkiill (1956, 1973), the Funktionskreis is a "theory of meaning" (Bedeutungslehre). In fact, he opposes "animal reactions" to "human responses". Cassirer may be wrong in not seeing the similarity between signs and other meanings (though he suggests it in passing using the term 'Vermittlung'), but he is quite right, I submit, in insisting on the difference. Very tentatively, let us suppose that, in the biosemiotic conception, the features of the world observed by the animal correspond to the sign-vehicle or expression (Peirce's "representamen"); the object or referent would then be that which causes theses features to be present to the animal; and the Peircean interpretant or content would in turn correspond to the pieces of behaviour which tend to make up the reaction of the animal to the features in question. There is no point getting lost here in Peircean exegesis: if anything, we are faced with a "formal sign", as conceived in the Fonseca tradition. As we are using the terms, we would have some kind of mediation (Cassirer's Vermittlung), but not a sign. As Ziemke and Sharkey (2001: 709) point out, it is hard to find the object of the sign, in the ordinary sense of its referent in the "outside world". Indeed, that which is for us, as observers, three cues to the presence of a mammal, the smell of butyric acid, the feel of skin, and the warmth of the blood, do not have to be conceived, in the case of the tick, as one single entity having an existence of its own (a "substance", in Gibson's terms), but may more probably constitute three separate episodes producing each its own sequence of behaviour. In fact, Ziemke and Sharkey go on to quote an early text by von Uexkiill, in which he says that "in the nervous system the stimulus itself does not really appear but its place is taken by an entirely different process" (von Uexkiill 1909, quoted here from Ziemke and Sharkey 2001, my italics). Uexkiill calls this a "sign", but it should be clear that it does not in any way fulfil the requirements of the sign function. Indeed, expression and content are not differentiated, already because they do not appear to the same consciousness. The butyric acid is there to the tick; the mammal is present only to us.

From the meaning ofembodiment to the embodiment ofmeaning

103

What is lacking here - to the tick - is real Thirdness: the reaction to the primary reaction, that is, the reaction which does not respond to a simple fact (Firstness), but to something which is already a reaction, and thus a relation (Secondness; see Table 1). Without having to enter into the earlier discussion of differentiation, we see that, even from a strictly Peircean point of view, there is no Thirdness for the tick: it does not respond to any relationship, since it is not aware (even in the most liberal sense of the term) of any second term (the mammal) to which the first term (the butyric acid) stands in a relation. Table 1.

The relationship between principles, grounds, and signs, from the point of view ofPeirce.

Firstness

Secondness

Principle

Iconicity

Ground

Iconic ground

Indexicality = indexical ground

Iconic sign (icon)

Indexical sign (index)

Sign

Thirdness

Symbolicity = symbolic ground = symbolic sign (symbol)

In fact, things are even more complicated. In a true sign relation, the mammal is not really the object, in the Peircean sense, for which the butyric acid is the representamen (the expression). Or, to be more precise, it is not the dynamical object. At the very most, it is the immediate object. In Peirce's conception, while the immediate object is that which directly induces the sign process, the dynamical object is something much more comprehensive, which includes all those things which may be known about the same object, although they are not present in the act of inducing. Indeed, the dynamical object is that which corresponds to the potentially infinite series of different interpretants resulting from the same original immediate object. It should be clear that, for the tick and similar beings, there could be no distinction between direct and dynamical object, because there is no room for any further development of the chain of interpretants. In this

104

Goran Sonesson

sense, Deacon's (1997: 63) idiosyncratic reading of Peirce, according to which only signs such as those found in human language (his "symbols") give rise to chains of interpretants seem to have some justification - in reality, ifnot in Peircean theory (cf. Sonesson 2003a, 2006).9 To account for the distinction between the "immediate object" and the "dynamical object", we need the concept of ground. lo In one of his wellknown definitions of the sign, a term which he here, as so often, uses to mean the sign-vehicle, Peirce (1931-58, 2: 228) describes it as something which "stands for that object not in all respects, but in reference to a sort of idea, which I have sometimes called the ground ofthe representation" (my italics; see Table 1). Some commentators have claimed that Peirce is here talking about some properties of the expression, whereas others favour the content. In fact, however, the ground must concern the relation between them. Such an interpretation seems to be born out by Peirce's claim that the concept of ground is indispensable, "because we cannot comprehend an agreement of two things, except as an agreement in some respect". (1.551). In another passage, Peirce himself identifies ground with an abstraction exemplifying it with the blackness of two black things (1.293). It therefore seems that the term "ground" must stand for those properties of the two things entering into the sign function by means of which they get connected, i.e. both some properties of the thing serving as expression and some properties of the thing serving as content. In case of the weathercock, for instance, which serves to indicate the direction of the wind, the contentground merely consists in this direction, to the exclusion of all other properties of the wind, and its expression-ground is only those properties which makes it turn in the direction of the wind, not, for instance, the fact of its being made of iron and resembling a cock (the latter is a property by means of which it enters an iconic ground, different from the indexical ground making it signify the wind). If so, the ground is really a principle of relevance, or, as a Saussurean would say, the "form" connecting expression and content: that which must necessarily be present in the expression for it to be related to a particular content rather than another, and vice-versa (cf. Sonesson 1989: ill. 1, 1995, forthcoming). The butyric acid, the hairiness, and the warmth form the immediate objects of the tick, while the mammal as such is the dynamical object. The

9. The problem, however, is that true indices and icons, as experienced as least by human beings, have as many interpretants as symbols. 10. This was independently noted by S0ren Brier (2001).

From the meaning ofembodiment to the embodiment ofmeaning

105

difference, however, is that there is no way that the tick, unlike human beings, may learn more about the "dynamical object" than that which is given in the immediate one. Meaning here appears as a kind of "filter": it lets through certain aspects of the "real world" which, in is entirety, is unknowable, though less so for human beings than for the tick. The Kantian inspiration of von Uexkiill is of course unmistakable. Indeed, in the terms of another thinker with a Kantian inspiration, Biihler (1934), the filter model is based on "abstractive relevance", the neglect of such physical properties which are not endowed with meaning, similar to those properties of the physical sound which vary a lot without the units of meaning (the phoneme, the word, etc.) being changed, which Saussure and Hjelmslev characterised as "substance" in opposition to "form". Returning to modem day biosemiotics, it can be easily shown that what these authors are involved with has little to do with meaning as sign function, but very much concerns meaning as relevance, organisation, configuration andlor filtering. In their early joint paper, Emmeche and Hoffmeyer (1991: 4), criticising the concept of information in information theory, point out (paraphrasing Bateson), that they are interested in "a difference that makes a difference to somebody". They go on to say that living beings "respond to selected differences in their surroundings" (their italics in both cases). The formulation clearly invokes relevance, and even some kind of filtering device. Later on in the paper, however, when the Peircean sign concept is introduced, the DNA-sequence of the gene is said to be the representamen, the protein its object, and the interpretant the cellularbiochemical network. It is, however, difficult to detect any sign function here. According to Emmeche and Hoffmeyer, the contribution of Peircean semiotics is to show us that "the field of genetic structures, or a single gene, cannot be seen in isolation from the larger system interpreted" (1991: 34). This certainly suggests meaning in the sense of a whole or a configuration. In a later paper, Emmeche (2002) sets out to show that in the living being function and meaning are the same. This can also be demonstrated, because Emmeche understands meaning in the sense of function: the relation of the part to the whole. But even in this article, there are traces of the filtering concept of meaning: we learn that "the whole operates as a constraint". Saying that cytochorme c means something to the cell is the same as saying that is has a function. It is not just any molecule. We could well synthesise small proteins and artificially introduce them into the cell. They would be

106

Goran Sonesson

without importance or they would be dysfunctional or, with certain fortuitous strokes of luck, they would actually fulfil some function in the cell. (Emmeche 2002: 19)

This implies that the meaning of the enzyme "is structural" in the sense that "the cell's molecules form a system of dissimilarities (like the elements of language in Saussure)" (Emmeche 2002: 20). This parallel is correct to the extent that there are relevancies in cells, in particular if these relevancies result from a system of oppositions, like those of Saussurean language. From this point of view, everything that is in the cells is also in language. But the opposite cannot be true. There is, of course, no sign function as we have defined it. It is useful to distinguish relevance from filtering, although they have something in common: picking up a limited set of features from the totality of the environment. However, relevance, strictly speaking, does not exclude anything: it merely places some portions of the environment in the background, ready to serve for other purposes. Thus, in the case of language, properties that are not relevant for determining the meaning of words and sentences, still may serve to inform about the dialect, or even identify the person speaking (Hjelmslev's "connotational language"; cf. Sonesson 1989). Indeed, relevance lets the difference between "immediate object" and "dynamical object" subsist, in the vague sense which they retain in the "scholastic" interpretation of Peirce (see above): that which is directly given, in contrast with that which is potentially given for further exploration. Thus, Biihler (1934), added to the principle of "abstractive relevance" that of "apperceptive supplementation", which explains the projection of properties not physically present in perception to the meaningful experience. In contrast,jiltering simply crosses out that which is not let through the filtering device.}} The difference between relevance and filtering no doubt has something to do with the capacity to be aware of the borders of one's Umwelt. It requires some kind of "metacognition": to the tick, to paraphrase Wittgenstein, the limits of its language are the limits of its world, but not so (in spite of Wittgenstein) to human beings. Or rather, the limits of any particular Umwelt are not the limits of our Lebenswelt. Schiitz (1967) sug11. It can now be seen that Btihler's principles of abstractive relevance and apperceptive supplementation go much further than the sign. They have been found in the studies of the systems of cooking and clothing realised by Levi-Strauss, Barthes, and others (as demonstrated by Sonesson 1989.

From the meaning ofembodiment to the embodiment ofmeaning

107

gested there are really "multiple provinces of meaning", such as dreaming, religious experience, the art world, the play world of the child, and that esoteric practise we lmow as science. The peculiarity of the Lifeworld, in this context, is that is offers access to the other worlds, and is accessible to all of them. In this sense, the human Lebenswelt is different from the Umwelt of other animals. Or at least is has the capacity for being different. In Peircean terms, human beings may reach for the dynamical objects beyond the immediate ones. They may try to transform Nature into Culture. However, as Wittgenstein (1971) observed, even if we had a common language game, we would perhaps not have so much to discuss with a lion. The lion, presumably, does not try to go beyond his own Umwelt to grasp the properties of the objects that lie behind it. There is, so to speak, no "dynamical object" beyond the immediate one to him. If the Umwelt is an organised network offilters and/or relevancies, as I suggested above, it seems that maturing in the child consists in breaking out of one Umwelt and going on to another, broader one, until reaching the human Lifeworld. Between each Umwelt and the next, which encompasses it, there is, to borrow a famous expression from Vygotsky (1978) a "zone of proximal development". In this sense, ontogenesis itself forces us to go through a series of "finite provinces of meaning", in the sense of Schiitz. A temporal dimension is thus added. It might therefore be said that what most perspicuously differentiates the tick from the human being (without prejudging for the moment on the question where the exact border is to be placed) is the structure of the field of consciousness: in Gurwitsch's (1957, 1964, 1985) terms, human consciousness is made up of a theme which is the centre of attention, a thematic field around it consisting of items which are connected to the present theme by means of intrinsic links permitting it to be transformed into a theme in its own right, as well as other items present "at the margin" at the same time, without having any other than temporal relations to the theme and its field. 12 The tick of course has access neither to the thematic field nor to the margin. In a way, this is simply another way of saying that the tick cannot reach beyond the immediate object. But Gurwitsch's analysis breaks up that of Peirce: it implies that, not only is there no way for the tick 12. Gurwitsch is right, I believe, in suggesting that this thematic structure translates to language (and no doubt also to other semiotic resources), as most clearly illustrated in the transposition of the functioning of pronouns from the perceptual world to discourse (cf. Gurwitsch 1985); it is unfortunate, however, that he fails to attend to the difference in structuring occasioned by the sign function.

108

Goran Sonesson

to "go on from here" (the Husserlean "etcetera principle"), its experience of the here and now is also very limited. In other words, there is no real "immediate object" to the tick, not only because it is not opposed to a future more extensive dynamical object, but also because even in the here and lmow, what is immediately experienced does not appear as a thematic structuring, or perspective, on such a dynamical object. I have suggested, then, that an important difference between human beings and (some) other animals consists in the thematic structure of consciousness, or, in other words, the function of attention. Some similar difference in the structure of attention have been discussed in very different quarters lately, separating human beings and apes, as well as children of different ages (cf. Tomasello 1999; Tomasello et al. 2005; Zlatev 2002, 2003, this volume). A discussion of such a progression in the development of attention presupposes an analysis of our awareness of the other's body and mind, which would take us out of the limits of this chapter. Something will be said, nevertheless, on the attention to one's own body in the next section. Before that, however, it will be necessary to take stock. I suggested above that there were really two differences between the way in which ticks and other lower animals have access to meaning and the human way. The first of these is the thematic structure: there is no immediate object, because there is no dynamical object in relation to which it may be seen as an adumbration. But there is more to it: there is no representamen (expression), either, because no distinction can be made between such a representamen and the object, either immediate or dynamic. Taking into account the Fonseca tradition, we earlier noted that one kind of mediation (for which I reserve the term sign) consists of a signifier (expression) which has to be perceived as such in order to usher into the perception of the corresponding signified (content); and another one (which following the Brentano-Husserl tradition, can be called intentionality) which may consist in a signifier which is not ordinarily perceived as such but still somehow serves to mediate the perception of a signified. It will be remembered that, according to von Uexkiill, "in the nervous system the stimulus itself does not really appear but its place is taken by an entirely different process" (my italics). If so, this is not even mediation in the broad sense of the term. As Husserl and Gibson have insisted, we are alternatively confronted with different view of the cube or the cat, etc., but what we really see is all the time the same invariant cube or cat. The tick smells the same invariant butyric acid, period. In the world of the tick,

From the meaning ofembodiment to the embodiment ofmeaning

109

there are no signs, as distinct from the world itself. Differentiation has not even started. In other words, signification has not acquired a "body" of its own.

5.

The body in the Lifeworld

In the previous section I suggested that in identifying the functional cycle with the Peircean concept of sign, biosemiotics conflates meaning in the most general sense of organisation and relevance with the more specific sign function. Inversely, contemporary embodiment theorists such as Lakoff and Johnson reduce the sign to the more general model of the pick-up of features from the environment. If Lakoff and Johnson engage in one form of reductionism, biosemiotics seems to accomplish its inversion. The result, however, is the same: distinctions, which are important, both theoretically, and from the point of view of phylogeny and ontogeny, can no longer be maintained. The hybrid term "image schema" has many antecedents, at least as to its latter part. Before the advent of Lakoffs and Johnsons's work, the most familiar usage was no doubt that of Piaget (1970: 41): as a kind of "abstraction from action" taking place at different stages through child development. Schiitz ([1974] 1932), however, used the term to refer to all kinds of fossilized (or, in his words, "sedimented") sequences of action, which could be used to make sense of new actions within the common sense world. The idea of a spatial, if not specifically bodily, projection, is important to the notions of schema in the psychology and sociology of Janet (1928), Halbwachs (1925, 1950) and Bartlett (1932; cf. Sonesson 1988). In all these conceptions, schemas are the result of earlier actions. This seems to accord with the definition by Johnson (1987,2005) of image schemas as being abstractions from the interaction of organism and environment. If so, as I will suggest below, image schemas should require some kind of separation of the human Umwelt into body and world.

5.1.

The body as the axis of the world

It is not surprising that the figure of the body looms large at the horizon of consciousness. After all, the body is our condition of access to all possible experience of the world. It is at the origin of one fundamental characteristic

110

Goran Sonesson

of the Hussserlean Lifeworld: that everything in it is given in a subjectiverelative manner. This means that the access in question is not a merely physical fact: it amounts to the insertion of the mind in the meaningful whole, which is the common sense world. That is, the body appears (also) as meaning. The same observations apply to language. The body (though often presented as a faceless Ego) is at the centre of language, in the I-here-now. Many pronouns and adverbs serve as marks of what Benveniste (1966) has called the "taking into charge" of the language system by the subj ect: these marks can only be understood with reference to the position in space and time of the person doing the speaking. Just as the perceptual world, language is adumbrated from the position of the subject, whose insertion in the world can only be accomplished by the body. Proxemics is concerned with the subject as a body occupying the central position of space. According to Hall (1966) all cultures define their public, social, personal and intimate, spheres, but the distances that characterise each one of these spaces are different in different cultures. When subjects coming from different cultures meet, their respective spaces tend to clash. According to one of Hall's classical examples, a person from an Arab culture, who posits himself within what is from his point of view the personal sphere, the distance from which it is comfortable to have a chat, inadvertently enters the intimate sphere of a Westerner, the sphere in which it is proper to "fight or make love". From a proxemic point of view, the subject could thus be seen as a topological construction: a series of concentric circles demarcating the public, social, personal and intimate, spaces (in relation to another subject), within which is found the bodily envelope, all of which are defined by the fact that they may be penetrated and thus produce an effect of meaning (see Figure 1 and Sonesson 1993b; 2001). This is to say that these "protective shells",' as Hall calls them, are more or less permeable. In topological terms, they possess the property of being open or closed. More exactly, in merotopological terms, some parts of them have the property of being open and others that of being closed. They produce a meaning when their borders are overstepped. This is of course the case with the Arab conversationalist stepping into the sphere of fighting and love of the Westerner. The case of the bodily envelope is however more easily illustrated: it possesses a series of openings (mouth, nostrils, etc.), but it may also be penetrated elsewhere, with more serious consequences, such as injury.

From the meaning ofembodiment to the embodiment ofmeaning

111

Public

sphere Figure 1.

The body envelope and its surrounding proxemic spheres (cf. Hall 1966, Spiegel and Machotka 1974). The arrows illustrate entries through designated openings and through the closed borders, respectively.

The final protective shell of the body, the skin, did not form part of Hall's original model. It was added later, by Spiegel and Machotka (1974), who also pointed out the difference between orifices permitting penetration, and other places where entry can only be forced. In this respect, their contribution connected to another tradition, the Freudian one, whose model of the body is reminiscent of some of the image schemas suggested by Lakoff and Johnson, if we generalise the sexual interpretation to a more general bodily practice (cf. Sonesson 1989).

5.2.

The bucket theory of the body

The function of the image schemas, as conceived by some cognitive linguists (cf. Hampe 2005), seems to be to project our experience of the body (even if experience per se sometimes seems to be dispensed with) to the interpretation of the world, thus accounting for pervasive linguistic phenomena such as metaphor, metonymy and polysemy. In the same vein, Gardner (1970: 360ft), elaborating on an idea of the psychoanalyst Erik

112

Goran Sonesson

Erikson, claims that certain holistic properties are given a particular import from being first experienced in the relationship between one's own body and the field of objects outside the body, sometimes in relation to the keeping of portions of the environment inside the body, and sometimes in relation to the release of what was once part of the body. Since each bodily zone has a characteristic mode, and since each mode possesses several vectorial properties, the modal/vectorial properties can be seen to form a system: to the oral-sensory zone (mouth and tongue), there corresponds passive and active incorporation; in relation to the anal-secretory zone, retention is experienced; in respect to the sphincter, there is expulsion; and, finally, to the genital zone (penis/vagina), there corresponds intrusion and inclusion. The result of this Freudian parti pris is not only an insistence on the primacy of sexual interpretations, but the neglect of some essential bodily relationships. Just as, according to Piaget, conceptual schemas are abstracted from actions through the many stages of intellectual development, the modal/vectorial properties, as Gardner presents them, may also be conceived to take their origin in the actions of one's own body, but rather than being abstracted, they are immediately seen as global characteristics, and while they may be transposed to other objects than the body, and other relationships than that of the body to the world, as is the case in "symbols", they somehow remain bound up with the body in all their further applications as being the deeper source of their sense (cf. Sonesson 1989). However, Arnheim (1966: 215ff) is right in arguing against Freudian pansexualism that a piece of pottery and a womb have the common class meaning of being containers, rather than the first signifying the second, and that the predominance of the sexual interpretation is due to cultural factors. It seems more probable that bodily experience of a more general kind, including that of enclosing an apple in one's hand and sticking the hand into a hollow in the ground, is the primary basis of modal/vectorial properties. When Arnheim suggests that, going up the tree of Porphyry, both the womb and the piece of pottery will be found to be containers, he is certainly not making the kind of analysis that Porphyry or his followers (as for instance Eco 1984: 46ff) would accept, since the womb does not meet the necessary and sufficient conditions for being a container ordinarily conceived; nor is it referable to the container prototype in a strict sense. Rather it is a member of the extended class of containers acceptable in "symbolism". On this interpretation, of course, the womb would be a deviant piece of pottery, rather than the reverse.

From the meaning ofembodiment to the embodiment ofmeaning

113

In making this kind of argument (cf. Sonesson 1989: 1.4.5), I found myself in a terrain very close to Cognitive Linguistics without knowing it. The first step was to identify the modal-vectorial properties as being topological. In the Piagetean conception, the geometry of the child's first space is topological, that is, it contains the kind of relations that would be preserved in a figure drawn on a piece of rubber (cf. Vuipillot 1967: 104ff). Properties of this kind are neighbourhoodness or proximity, separation, succession, inclusion or interiority/exteriority, and continuity. If we now merely introduce a distinction between two instances, the ego and the world, or the other, it will be possible to derive all of Gardner's "modes" from the topological property of inclusion, to which another topological property, that of succession, is applied. Clearly, intrusion and inclusion are opposites, as are incorporation and expulsion, but rentention seems to call for some corresponding term: this must be resistance (postulated in Sonesson 1989, in complete ignorance of force-dynamical theories, for which cf. Talmy 1988; Gardenfors this volume), of which there are two variants, the resistance of the world to us, and of us to the world. Actually, there is nothing very new about resistance as a fundamental concept: it has been the basis of the definition of reality in philosophical epistemology, from Berkeley over the ideologues Destutt de Tracy and Maine de Biran to Sartre. Indeed, "this sense of being acted upon, which is our sense of the reality of things" is the definition of Secondness in the conception of Peirce (1998: 4): A door is slightly ajar. You try to open it. Something prevents. You put your shoulder against it, and experience a sense of effort and a sense of resistance. These are not two forms of consciousness; they are two aspects of one two-sided consciousness. It is inconceivable that there should be any effort without resistance, or any without a contrary effort. This double-sided consciousness is Secondness. (Peirce 1998: 268)

Since Peirce goes on immediately to note that "all consciousness, all being awake, consists in a sense of reactions between ego and non-ego", it is curious that he should not recognise the difference between the case in which the ego takes the active part, and the case in which the non-ego has the initiative and the ego is reduced to resistance. However, it is clear that from a Peircean point of view, the Freudian interpretations of incorporation, retention, expulsion, and so on, are only special cases of more general bodily processes. For they no doubt continue to be bodily based: it is the body of the ego which first enters into a clash with the non-ego.

114

Goran Sonesson

These operations obviously serve more humble purposes than proving the existence of the outside world. Just as such image schemas as PATH and CONTAINER, they are generalizations of "a recurrent pattern, shape, and regularity in, or of, [ ...} ongoing ordering activities" as actions, perceptions, and conceptions (Johnson 1987: 29, original italics). In spite of what is suggested by this definition, Lakoff and Johnson often do not seem to have any use for the body as an experienced meaning, as opposed to the way it appears to the biological sciences. What is at issue is the exact role played by the body. Whether the actions which sediment to form "images schemas" are accomplished in relation to the inside of the body, or to something outside of it, a minimum requirement for their schematisation must be the existence of the bodily envelope as a relevant level of analysis. It is difficult to understand how such schemas may even come into existence if human beings are as tightly embedded in their Umwelt as the tick, entirely merged with their environment. Yet, in introducing the theory of image schemas, this is precisely the view propounded by Johnson (2005). In defining image schemas, Johnson (1987, 2005, Johnson and Rohrer this volume) uses expressions that are clearly reminiscent of Piaget, although the latter is never quoted. However, in order to acquire sensorymotor schemas, let alone "symbols" (Piaget 1945) or "mimetic schemas" (Zlatev 2005), some sense of a distinction between the acting subject and the world resisting him or her is clearly required. In order to arrive at schemas of a higher order (including the sign function), the subject must somehow take cognizance of the more basic schemas themselves. Johnson and Rohrer (this volume) who do not believe in "the supposedly unique ability of humans to engage in symbolic representation", consider it only an illusion resulting from our seeing the interaction of organism and environment "from our standpoint as observers and theorists". But this leaves unexplained the fact that we are ever able to reach such a standpoint. No doubt there must be a number of progressive stages leading from sensorymotor experience to our capacity for engaging in theory.

6.

The body as portable memory

Is has been suggested by Donald (1991, 2001) that there are several discontinuities in hominid evolution, all involving the acquirement of a distinct kind of memory, considered as a strategy for representing knowledge.

From the meaning ofembodiment to the embodiment ofmeaning

115

Although Donald's model concerns phylogeny, parallels in ontogeny are readily suggested (cf. Zlatev 2002, 2003, 2005, this volume; Ikegami and Zlatev this volume). Without necessarily taking Donald's model at face value, I am going to make use of it here since it permits a productive integration with semiotic theory. Indeed, the Tartu school characterizes signs as memory devices and defines culture as collective memory. According to Lotman et al. (1975), material objects and information are similar to each other, and at the same time differ from other phenomena, in two ways: they can be accumulated, whereas for example, sleep and breathing cannot be accumulated, and they are not absorbed completely into the organism, unlike food, but remain separate objects after the reception. The interesting thing, however, not discussed by the Tartu school, is how material artefacts and signs come to work together. According to Donald's conception, many mammals are already capable of episodic memory, which amounts to the representation of events in terms of their time and place of occurrence. The first transition, which antedates language and remains intact at its loss (and which Donald identifies with Homo erectus) brings about mimetic memory, which is required for such abilities as the construction of tools, miming, imitation, coordinated hunting, a more complex social structure and simple rituals. This stage thus in part seems to correspond to what we have called the attainment of the sign function (though Donald only notes this obliquely, in talking about the use of intentional systems of communication and the distinction of the referent). Yet, it should be noted already at this point that while all abilities subsumed in this stage seem to depend on iconic relations (perceptions of similarity), only some of them are signs because they do not involve any asymmetric relation between an expression and the content for which it stands. Only the second transition, occurring with Homo sapiens, brings about language with its semantic memory, that is, a repertory of units that can be combined. This kind of memory permits the creation of narratives, that is, mythologies, and thus a completely new way of representing reality. Interestingly, however, Donald does not think semiotic development stops there, although further stages are no longer based on any biological changes. However, the third transition obviously would not have been possible without the attainment of the three earlier stages. What Donald calls theoretical culture presupposes the existence of external memory, that is, devices permitting the conservation and communication of knowledge independently of face-to-face interaction between human beings. The first

116

Goran Sonesson

apparition of theoretical culture coincides with the invention of drawing. For the first time, knowledge may be stored externally to the organism. The bias having been shifted to the visual modality, language is next transferred to writing. It is this possibility of conserving information externally to the organism that later gives rise to science (cf. Figure 2).

----\...----------""'.~ 'V'" o +:i

~ ~

Sign function • Toolmakjng

-Iolitation

J5

~

• Language

- Gesture:E

iconicity indexicality

Figure 2.

.2 ~

0

~

I-

· Pictures • Writing • Theory

symbolicity iconicity indexicality

Donald's model of evolution related to the notion of sign function.

There are two remarkable features in Donald's analysis. The stage preceding the attainment of the language capacity requires memory to be located in the subject's own body. But, clearly, it can only function as memory to the extent that it is somehow separable from the body as such. The movement of the other must be seen as distinct from the body of the other in its specificity, so that it can be repeated by the self. This supposes a distinction between token and type (that is, relevance) preceding that of the sign function. The stage following upon language supposes the sign to acquire a "body" of its own, that is, the ability to persist independently of human beings. Language only seems to require the presence of at least two human beings to exist: they somehow maintain it between themselves. But it is not enough for two persons to know about a picture for it to exist: there must be some kind of organism-independent artefact on which it is inscribed.

From the meaning ofembodiment to the embodiment ofmeaning

117

The picture must be divorced from the bodies (and minds) of those making use of it. 13 Writing is of course, by definition, the transposition of language to independent artefacts. The case of "theory" may be less obvious: why should not two persons be able to entertain a theory between them? As Husserl (1962a) noted well before Donald, complex sign systems, such as mathematics and logic, only seem to function as such when given an existence independent of human organisms. In the case of pictures, Ivins (1953) has observed that it is their reproducibility (as in Floras, for instance) that makes them into scientific instruments. In their capacity of being permanent records, pictures are not, as art historians are wont to say, unavoidably unique, but, on the contrary, are destined for reproduction. Indeed, they permit repeated acts of perception, as do no earlier memory records. Students of prehistoric pictures such as White (2000) often suggest that creators of such works must have been capable of language. In fact, not much can be concluded on the basis of the depictions having come down to us: even though pictures, by their nature, must have been made on material which conserves the markings on the surface, they might at first have been created on surfaces (such as sand) which only preserve them for a short time. And it is not easy to establish any clear-cut relation between linguistic capacity and the sophistication of the depictions (whatever that is). There are, however, more fundamental reasons for supposing pictures to be later in phylogeny than language: they suppose a record which is independent of the human body; and they require us to see a similarity within an over-arching dissimilarity. Posner (1989) distinguishes two types of artefacts: the transitory ones (as the sound of a woman's high heeled shoes against the pavement) and enduring ones (as the prints that the woman's shoes may leave in clay, in particular if the latter is later dried). The transitory artefa~ts, in this sense, also have a material aspect, just as the lasting ones; they only have the particularity of developing in time, which is why they cannot be accumulated without first being converted. Strictly speaking, the sound sequence produced by high heels against the pavement, and other transitory artefacts, can of course be accumulated (as opposed to being converted into an enduring artefact, which is the case of the sound tape), in the form of the

13. This is of course what is known, mainly in Marxist literature, as the process of reification. As shown by Cassirer (1942: 113ft), this process, far from being only a "tragedy of culture", is the prerequisite for (huma)n culture.

118

Goran Sonesson

(typical) leg movements producing this sound, that is, as a mimetic record, accumulated in the body, but still distinct from it, since the movements can be learnt and imitated, and even intentionally produced as signs of (traditional) femininity. Posner's example of an enduring artefact is interesting in another way: the cast of prints left by the woman's high heels is of course an organism-independent record, just as the marks of a Roman soldier's sandals found in prehistoric caves, and the hand-prints on cave walls. Another case in point may very well be the so-called Berekhat Ram figure (250000-280000 BP), which, if it is not the likeness of a woman, as has been claimed with very little justification, could be the result of abrasion produced by regular movements indicating the intervention of a human agent (that is, "anthropogenic" movements). This suggests that the first organism-independent records may be indexical, rather than iconic, in character. However, even if objects like these were independent objects already in prehistory, there is nothing to prove they were perceived as signs, that is, as expressions differentiated form contents, before pictures were so perceived. Episodic memory, in Donald's sense (which should not be confused with earlier uses of the term) is most clearly "disembodied" memory: it only goes as far as the attention span does. It may refer to a bodily act, such as going in or out of a container-type object, but it is unable to generalise this movement beyond a particular moment and place, and thus it does not give rise to any kind of independent embodiment (cf. Table 2). Mimetic memory still accumulates in the subject's own body, but it only becomes such, to the extent that what is recorded in the body also exists elsewhere, in at least one other body (or perhaps, in same cases, in other moving artefacts), which supposes generalisation or, more exactly, typification: the creation of a type referring to different tokens instantiated in different bodies. As tokens, then, they are in the body; as types they are shared by different uses. Typification, in this sense, does not require the sign function, but is no doubt a prerequisite for it: indeed, it is during this stage, most likely, that the sign function emerges. Mythic memory (which I would prefer to call linguistic memory or perhaps, as Donald sometimes does: semantic memory) is different again: it has a separate existence, but, like some kind of real-world ectoplasm, is requires the collaborative effort of a least two consciousnesses (which no doubt have to be embodied) for this existence to be sustained. Transitory artefacts, as spoken language or (as Posner would have it) the sound of high-heeled shoes on the pavement, acquire a body only to the extent that a

From the meaning ofembodiment to the embodiment ofmeaning

119

sender and a receiver agree roughly on what they are. Only theoretic memory has a distinct "body" of its own: it subsists independently of the presence of any embodied consciousness, because it is itself embodied. Of course, without anybody able to perceive it, organism-independent records are not of any use. Without any human beings present, they are really worse off than the famous acorn falling from a tree without anybody around to hear its sound. Table 2. Donald's memory types analysed in relation to the nature of accumulation (in the sense of Lotman et al. 1975) 1.1. Type of memory

1.3. Type of embodiment

Episodic

Attention span (event in time/space)

Mimetic

Action sequence co-owned by Ego and Alter

Own body

Mythic

Transient artefact co-produced by Ego and Alter

In the interac-

tion between Ego and Alter

Enduring artefact co-externalised by Ego and Alter

External in relation to Ego and Alter

Theoretic

7.

1.2. Type of accumulation

Conclusions

In this chapter I have tried to relate different notions of embodiment stemming from the phenomenological tradition, and contemporary conceptions of embodiment, taking their origin in Cognitive Linguistics, biology, and cognitive science. My aim has been to show that the various forms of embodiment in both traditions are very different, but that once they are properly analysed, they may be connected with each other, and placed on something like an evolutionary scale similar to the one proposed by Donald

120

Goran Sonesson

(1991). Indeed, the whole point of making these distinctions has been to show the complexity of the "ladder" from (non-human) animals to human beings, a ladder that requires a series of very different steps, only one of which is the capacity for language. Another goal of the essay has been to suggest the way in which the sign function, the general faculty for conceiving signs, emerges out of one kind of embodiment and constitutes a requirement for attaining another one. In the process, I have suggested that we must distinguish meaning in a very general sense, akin to organisation and/or selection, from the sign function, which requires the peculiar property of differentiation. I have claimed that this distinction is not observed in either biosemiotics nor in some parts of Cognitive Linguistics. Moreover, I have argued that what is called "image schemas" by cognitive linguists is basically a kind of bodily meaning, resulting from the position of the human body at the centre of the common sense world, known in phenomenology as the Lifeworld.

Acknowledgements

The author wishes to thank Jordan Zlatev and Tom Ziemke for their detailed comments on an earlier draft of this chapter.

References Arnheim, Rudolf 1966 Towards a Psychology ofArt. London: Faber and Faber. Bartlett, Fredrick C. 1932 Remembering. A Study in Experimental and Social Psychology. Cambridge: Cambridge University Press, Reprint 1967. Benveniste, Emile 1966 Problemes de linguistique generale. Paris: Gallimard. Brentano, Franz 1885 Psychologie von empirischen Standpunkt. Leipzig: Mener. Reprint 1924. Brier, Seren 2001 Cybersemiotics and Umweltslehre. On the cybersemiotic integration of Umweltlehre, enthology, autopoiesis theory, second order cybernetics and Peircean biosemiotics. Semiotica. Special issue on Jakob von Uexkall, 134(1/4): 779-814.

From the meaning ofembodiment to the embodiment ofmeaning

121

Biihler, Karl 1934 Sprachtheorie. Fischer, Frankfurt/M: Ullstein. Reprint 1978. Cassirer, Emst 1942 Zur Logik der Kulturwissenschaften. Goteborg: Elanders. 1945 An Essay on Man. New Haven: Yale University Press. 1957 The Philosophy of Symbolic Forms. Ill. The Phenomenology of Knowledge. Translated by Ralph Mannheim. New Haven: Yale University Press. Costall, Alan this vol. Bringing the body back to life: James Gibson's ecology of embodied agency. Deacon, Terry 1997 The Symbolic Species: The Co-Evolution ofLanguage and the Brain. New York: Norton. 2003 Universal grammar and semiotic constraints. In: Christiansen, Morton H. and Simon Kirby (eds.), Language evolution, 111-139. Oxford: Oxford University Press. Deely, John Introducing Semiotic. Its History and Doctrine. Bloomington: Indi1982 ana University Press. New Beginnings. Early Modern Philosophy and Postmodern 1994 Thought. Toronto: University of Toronto Press. Dennett, Daniel 1991 Consciousness Explained. Boston: Little, Brown, and Co. Donald, Merlin 1991 Origins of the Modern Mind. Three Stages in the Evolution of Culture and Cognition, Cambridge, Mass.: Harvard University Press. A Mind So Rare. The Evolution of Human Consciousness. New 2001 York: Norton. Eco, Umberto 1982 Semiotics and the Philosophy of Language. Bloomington: Indiana University Press. Edelmann, Gerald 1992 Bright Air, Brilliant Fire. On the Matter of the Mind. London: PenguinBooks. Edelmann, Gerald and Guilio Tonini 2000 A Universe ofConsciousness. New York: Basic Books. Emmeche, Claus 2002 The chicken and the Orphean egg. On the function of meaning and the meaning of function. Sign System Studies 30 (1): 15-32. this vol. On the biosemiotics of embodiment and our human cyborg nature.

122

Goran Sonesson

Emmeche, Claus and Jesper Hoffmeyer 1991 From language to nature - the semiotic metaphor in biology. Semiotica 84 (1/2) 1--42. Gallagher, Shaun this vol. Phenomenological and experimental contributions to understanding embodied experience. Gardenfors, Peter this vol. Representing actions and functional properties in embodied conceptual spaces. Gardner, Howard 1970 From mode to symbol. The British Journal ofAesthetics 10 (3): 357375. Gibson, James 1978 The ecological approach to visual perception of pictures. Leonardo, 4 (2): 227-235. 1980 A prefatory essay on the perception of surfaces versus the perception of markings on a surface. In: Margaret Hagen (ed.), The Perception of Pictures. Volume I: Alberti's Window, xi-xvii. New York: Academic Press. 1982 Reasons for Realism. Selected Essays of James J Gibson. Edward Reed and Rebecca Jones (eds.). Hillsdale, New Jersey: Lawrence Erlbaum. Greimas, Algirdas J. 1970 Du sens. Paris: Seuil. Gurwitsch, Aron 1957 Theorie du champ de la conscience. Bruges: Desclee de Brouver. 1964 The Field ofConsciousness. Pittsburgh: Duquesne University Press. 1966 Studies in Phenomenology and Psychology. Evanston: Northwestern University Press. 1974 Phenomenology and the Theory of Science. Evanston: Northwestern University Press. 1979 Human Encounters in the Social World. Pittsburgh: Duquesne University Press. 1985 Marginal Consciousness. Athens, Ohio: Ohio University Press. Halbwachs, Maurice 1925 Les cadres sociaux de la memoire. Paris: PUF. Reprint 1952. 1950 La memoire collective. Paris: PUF. Reprint 1968. Hall, Edward 1966 The Hidden Dimension. London: Bodley Head Ltd. Hampe, Beate (ed.) 2005 Perception to Meaning: Image Schemas in Cognitive Linguistics, Berlin: Mouton de Gruyter.

From the meaning ofembodiment to the embodiment ofmeaning

123

Hoffman, Donald 1998 Visual Intelligence. How We Create What We See. New York: Norton. Hockett, Charles 1977 The View from Language. Athens: University of Georgia Press. Husserl, Edmund 1913 Logische Untersuchungen. Tiibingen: Niemeyer. Reprint 1968. 1939 Erfahrung und Urteil. Prag: Academia Verlagsbuchhandlung. Die Krisis der europiiischen Wissenschaften und die transzendentale 1962 a Phiinomenologie. Husserliana VI. The Hague: Nijhoff. 1962 b Phiinomenologische Psychologie. Husserliana IX. The Hague: Nijhoff. 1973 a Cartesianische Meditationen. Husserliana I. The Hague: Nijhoff. 1973 b Ding und Raum. Husserliana XVI. The Hague: Nijhoff. 1976 Ideen zu einer reinen Phiinomenologie und phiinomenologische Psych 0 logie. Husserliana Ill. The Hague: Nijhoff. 1980 Phantasie, Bildbewusstsein, Erinnerung. Husserliana XXIII. The Hague: Nijhoff. Ikegami, Takashi and Jordan Zlatev this vo!. From pre-representional cognition to language. Innis, Harold 1950 Empire and Communication. Toronto: University of Toronto Press. Reprint 1972. Ivins, William M. 1953 Prints and Visual Communication. Cambridge, Mass.: Harvard University Press. Jakobson, Roman 1975 Coup d'adl sur le developpment de la semiotique. Bloomington: Indiana University Press. James, William 1890 The Principles ofPsychology. New York: Dover Publications. Janet, Pierre 1928 L 'evolution de la memoire et la notion du temps. Paris: Chahine. Johnson, Mark 1987 The Body in the Mind. Chicago and London: University of Chicago Press. 2005 The philosophical significance of image schemas. In: Beate Hampe (ed.), From Perception to Meaning: Image Schemas in Cognitive Linguistics., 15-33. Berlin: Mouton de Gruyter. Johnson, Mark and Tim Rohrer this vo!. We are life creatures: Embodiment, American Pragmatism, and the cognitive organism.

124

Goran Sonesson

Kiing, Guido 1973 Husserl on pictures and intentional objects. Review of metaphysics. (June) XXXVI (4): 670-680. Lakoff, George and Mark Johnson 1999 Philosophy in the Flesh. New York: Basic Books. Lotman, Jurij M., Boris A. Uspenskij, Vjaceslav. V. Ivanov, V. N. Toporov and A. M. Pjatigorski 1975 Thesis on the Semiotic Study of Culture. Lisse: The Peter de Ridder Press. Merleau-Ponty, Maurice 1942 La structure du comportement. Paris: PUF. 1945 La phenomenologie de la perception. Paris: PUF. Parmentier, Richard J. 1985 Signs's place in medias res: Peirce's concept of semiotic mediation. In: Elizabeth Mertz and Richard J. Parmentier (eds.) Semiotic Mediation. Sociocultural and Psychological Perspectives, 23-48. Orlando: Academic Press. Piaget, Jean La formation du symbole chez l'enfant. Neuchatel: Delachaux and 1945 Niestle. Reprint 1967. 1967 La psychologie de I 'intelligence. Paris: Armand Colin. 1970 Epistemologie des sciences de l'homme. Paris: Gallimard. Peirce, Charles Sanders 1931-58 Collected Papers I-VIII. C. Hartshome, P. Weiss and A. Burks (eds.). Cambridge, MA: Harvard University Press. 1998 The Essential Peirce, Volume II. Ed. by the Peirce Edition Project. Bloomington and Indianapolis: Indiana University Press. Posner, Roland 1989 What is culture? Toward a semiotic explication of anthropological concepts. In: Waiter Koch (ed.), The Nature of Culture, 240-295. Bochum. Brochmeyer. Prieto, Luis J. 1975 a Pertinence et pratique. Paris: Minuit. 1975 b Essai de linguistique et semiologie generales. Geneve: Droz. Reed, Edward S. 1996 The Necessity ofExperience. New Haven: Yale University Press. Schiitz, Alfred 1932 Der sinnhafte Aufbau der sozialen Welt. Vienna: Springer. Reprint 1974. 1967 Collected Papers I: The Problem of Social Reality. Nijhoff: The Hague.

From the meaning ofembodiment to the embodiment ofmeaning

125

Smith, Barry and Achille Varzi 1999 The niche. Nous 33 (2): 198-222. Sonesson, Goran 1988 Methods and models in pictorial semiotics. Semiotics Project: Lund University. 1989 PictorialConcepts. Inquiries into the Semiotic Heritage and its Relevance for the Analysis of the Visual World. Lund: Aris/Lund University Press. Bildbetydelser. Inledning till bildsemiotiken som vetenskap. Lund: 1992 a Studentlitteratur. 1992 b The semiotic function and the genesis of pictorial meaning. In: Eero Tarasti (ed.), Center/Periphery in representations and institutions. Proceedings from the Conference of The International Semiotics Institute, Imatra, Finland, July 16-21, 1990, 211-156. Imatra: Acta Semiotica Fennica. 1993 a Pictorial semiotics, Gestalt psychology, and the ecology of perception. Semiotica 99 (3-4): 319-399. 1993 b The multiple bodies of man. Project for a semiotics of the body. Degres 74: d-d42. 1994 Prolegomena to the semiotic analysis of prehistoric visual displays. Semiotica 100 (3): 267-332. 1995 On pictorality. The impact of the perceptual model in the development of visual semiotics. In: Thomas Sebeok and Jean Umiker-Sebeok (eds), .The Semiotic Web 1992/93: Advances in Visual Semiotics, 67-108. Berlin and New York: Mouton de Gruyter. 1996 An essay concerning images. From rhetoric to semiotics by way of ecological physics. Semiotica 109 (1/2): 41-140. 1998 a /entries/. In: Paul Bouissac (ed.) Encyclopaedia of Semiotics. New York and London: Oxford University Press. 1998 b That there are many kinds of pictorial signs. VISIO 3 (1): 33-54. 2000 Iconicity in the ecology of semiosis. In: Troels Deng Johansson, Martin Skov and Berit Brogaard (eds.), Iconicity - A Fundamental Problem in Semiotic, 59-80. Aarhus: NSU Press. 2001 From Semiosis to Ecology. On the theory of iconicity and its consequences for the ontology of the Lifeworld. In: Andrew W. Quinn (ed.), Cultural Cognition and Space Cognition/Cognition culturelle et cognition spatiale. VISIO 6 (2/3): 85-110. 2003 a The Symbolic Species revisited. Considerations on the semiotic turn in cognitive science and biology, SGBWP3. Lund: Lund University. 2003 b Why the mirror is a sign - and why the television picture is no mirror. Two episodes in the critique of the iconicity critique. In S. European Journalfor Semiotic Studies 15(2-4): 217-232.

126

Goran Sonesson

2006

The meaning of meaning in biology and cognitive science. A semiotic reconstruction. In Trudy po znakonym sistemam - Sign Systems Studies 34.1, 135-214. forthc. From iconicity to pictorality. A view from ecological semiotics. To appear in VISIO: thematic issue: Iconicity revisitedlIconicite revisite, Sonesson, Goran (ed.). Spiegel, John and Pavel Machotka 1974 Messages ofthe Body. New York: The Free Press. Talmy, Leonard 1988 Force dynamics in language and cognition. Cognitive Science 12: 49-100. Tomasello, Michael 1999 The Cultural Origins ofHuman Cognition. Cambridge, MA: Harvard University Press. Tomasello, Michael, Malinda Carpenter Joseph Call, Tanya Behne, and Henrike Moll 2005 Understanding and sharing intentions: The origins of cultural cognition. Behavioral and Brain Sciences 28: 675-735. Varela, Francisco, Evan Thompson and Eleanor Rosch 1991 The Embodied Mind. Cognitive Science and Human Experience. Cambridge, Mass.: MIT Press. Von Uexkiill, Jakob 1909 Umwelt und Innenwelt der Tiere. Berlin: Springer. 1956 Streifzuge durch die Umwelten von Tieren und Menschen Bedeutungslehre. Hamburg: RowoWt. 1928 Theoretische Biologie. Frankfurt/M.: Suhrkamp. Reprint 1973. Vuipillot, Eliane 1967 La perception de l'espace. In: Paul Fraisse and Jean Piaget (eds.), Traite de psychologie experimentale VI: La perception, 101-186. Paris: PUF. Vygotsky, Lev 1978 Mind in Society. Cambridge. Mass.: Harvard University Press. Wittgenstein, Ludwig 1971 Philosophische Untersuchungen. Suhrkamp: Frankfrut/M. White, Randall 2000 Prehistoric Art. New York: Harry N. Abrahams, Inc. Ziemke, Tom and Noel E. Sharkey 2001 A stroll through the worlds of robots and animals: Applying Jakob von Uexkiill's theory of meaning to adaptive robots and artificial life. Semiotica 134 (1-4): 701-746.

From the meaning ofembodiment to the embodiment ofmeaning

127

Zlatev, Jordan 2002 Mimesis: The "missing link" between signals and symbols in phylogeny and ontogeny. In: Anneli Pajunen (ed.), Mimesis, Sign and the Evolution ofLanguage, 93-122. Turku: University of Turku Press. 2003 Meaning = Life (+ Culture). An outline of a unified biocultural theory of meaning. Evolution ofCommunication 4 (2): 253-296. 2005 What's in a schema? Bodily mimesis and the grounding of language. In: Beate Hampe (ed.), From Perception to Meaning: Image Schemas in Cognitive Linguistics, 313-342. Berlin: Mouton de Gruyter. this vol. Embodiment, language, and mimesis.

Embodiment and social interaction: A cognitive science perspective Jessica Lindblom and Tom Ziemke We respond to gestures with an extreme alertness and, one might almost say, in accordance with an elaborate and secret code that is written nowhere, known to no one and understood by all. Sapir 1928.

Abstract Much recent work in cognitive science has focused on the embodied, situated and distributed nature of mind. This is a radical shift away from the computer metaphor for mind that characterized traditional cognitive science. However, although much attention is paid nowadays to the biological and bodily basis of cognitive processes as well as their sociocultural embedding, the role of embodiment in social interaction is still relatively little understood. In this chapter we trace the role of biological and sociocultural factors in explanations of cognitive processes from Darwin to modem cognitive science. We discuss different degrees of commitment to embodiment and point out further steps and conceptual clarifications that will be required in the further development of a science of embodied cognition and social interaction. Keywords: embodied cognition, history of cognitive science, interactive technology, radical embodiment, social interaction.

1.

Introduction

Embodiment has become a much discussed concept in cognitive science in recent years, and many take it, together with situatedness, to be the defining feature of a new approach to the study of mind commonly referred to as embodied cognitive science or embodied cognition (e.g., Clark 1997, 1999; Varela, Thompson and Rosch 1991). Embodied cognition offers a radical shift in explanations of the human mind - a Copernican revolution in cog-

130

Jessica Lindblom and Tom Ziemke

nitive science, you might say - which emphasizes the way cognition is shaped by the body and its sensorimotor interaction with the world. This is a reaction against the computer metaphor of traditional cognitive science, which views cognition as symbol manipulation, centralized and taking place inside the skull while the body only serves as some kind of input and output device, i.e., a physical interface between internal program (cognitive processes) and external world.! That means, embodiment only played a marginal, if any, role in traditional cognitive science which instead focused on mental representations and computational processes. Gardner (1987: 6), for example, stressed the major assumption for the then "new science of mind" as follows: First of all, there is the belief that, in talking about human cognitive activities, it is necessary to speak about mental representations and to posit a level of analysis wholly separate from the biological or neurological, on the one hand, and the sociological or cultural, on the other.

Consequently, the computer and its syntax-driven way of manipulating symbols became the general model of how the human mind functions. Accordingly, the traditional view of social interaction has been that agents relate to each other much in the same way they relate to other parts of the external world, i.e., by having more or less explicit internal representations of each other. That means, the "secret code" that Sapir referred to in the above introductory quote was quite literally taken to be an actual symbolic/representational code. For example, one agent might encode her mental states into some form suitable for transmission via some communication channel, such as language or gestures, and another would receive and decode the transmitted message, and thus come to an understanding of the first agent's mental states and actions. That means, traditional cognitive-scientific explanations of both individual and social cognition very much focused on internal, individual processes and representations, whereas the interaction with the environment - physical, social or cultural - was treated as fairly peripheral. Gardner (1987), for example, argued that such "murky concepts" as context, history and culture only could cause problems for efforts to find the "essence" of individual cognition, and therefore should not be addressed and integrated until later on. Today's proponents of embodied, situated and/or distributed cognition, on the other hand, paint a much more complex picture of the mind, its reIa1. In cognitive science the traditional view is referred to as "cognitivism" which, however, should not be confused with the cognitivist approach in linguistics.

Embodiment and social interaction: A cognitive science perspective

131

tion to the body, and its interaction with the physical and sociocultural environment. Hutchins (1995), for example, argued that there are unnoticed costs involved when we disregard culture, context and history, which he considered important factors in the development of individual intelligence. The body's role in cognitive processes has also received much attention in recent discussions in cognitive science (cf. e.g., Clark 1997, 1999; Varela, Thompson and Rosch 1991; Ziemke 2002) and a large variety of notions of embodiment and embodied cognition have been developed (cf. e.g., Chrisley and Ziemke 2003; Clark 1999; Rohrer this volume; Svensson, Lindblom and Ziemke this volume; Wilson 2002; Ziemke 2001a, 2003; Ziemke and Frank this volume). From a psychological perspective the interaction between individual agent and environment has been addressed, while from the perspectives of philosophy and linguistics the role of body and world in the constitution of meaning has received much attention. Moreover, from the AI perspective it has been discussed what kind of body an artificial system might need to count as an embodied cognizer. However, despite the fact that embodied cognitive science pays much attention to both the sociocultural embedding of cognitive processes and their bodily basis, there still is surprisingly little work on the role of embodiment in social interaction, as recently recognized by several researchers, who have argued that current theories of embodiment need to be developed further to include the social dimension (e.g., Lindblom and Ziemke 2005; Riegler 2002; Semin and Smith 2002; Sinha and Jensen 2000; Ziemke 2002, 2003). Traditionally, the study of the body in social interaction in cognitive science has been limited to non-verbal communication, commonly viewed as a trivial form of "body language" or an "appendage" to the real intellectuallanguage and mind. However, it has been estimated that nearly two thirds of the meaning in a social situation is "received" from these nonverbal signs (Burgoon, Buller and Woodall 1996). Famell (1999) pointed out that the widespread neglect of the body in the social sciences is a consequence of the Platonic-Cartesian heritage that has resulted in the view of the mind as an internal locus of rationality, thought, language and knowledge (cf., Damasio 1995, 1999; Johnson and Rohrer, this volume). The body, on the other hand, is still widely regarded as the sensate locus of irrationality and feeling, a view that has been supported by the Christian

132

Jessica Lindblom and Tom Ziemke

disregard2 of the flesh as the locus of sinful desire and irrationality. Theories of cultural and social cognition therefore mainly tend to overlook the bodily aspects of social interaction (cf. Tomasello 1999). Varela (1992), for example, argued that "[s]ocial scientists are body-dead because they are conceptually brain-dead to signifying acts within the semiotics of body movements". Trevarthen (1977; cf., Hendriks-Jansen 1996) pointed out that one reason for the neglect of the body in psychological research was that the bodily movement patterns of humans were difficult to observe with current technology, and cognitive science therefore became more of a static science of perception, cognition and action than a science of embodied dynamic interactions. On the other hand, when researchers actually paid attention to embodied movement, it often appeared that, as Famell (1995) formulated it, the moving body has lost its mind. Another reason why researchers commonly overlook the role of the body is perhaps the widespread fear of slipping into biological reductionism which is also why most social scientists tend to, or prefer to, view mind as superior to and independent of the body (Famell 1999). Hence, it should be noted that investigating the role of embodiment in social interaction is not the same as relapsing into (socio-) biological reductionism. In fact, the underlying supposed dichotomy between nature and culture, i.e., biological and sociocultural factors, is highly misleading in the first place, despite the fact that it has a long tradition in scientific discussions. Ingold (2000) used the example of learning to walk as an illustration of the traditional conception of the relation between "nature" and "culture". It is commonly argued that walking is an innate human capability, but Ingold does not categorize human walking as either innate or an acquired. A child learns to walk according to the standard manner of its social and cultural environment, and there is no one "natural" or "pure" biological way of walking, as one might assume. That means, the human skill of walking can be viewed as "biological" in the sense of being a part of the functions of the human organism, but it is also a result of the child's involvement in a social and physical world during normal development. Ingold (2000) therefore pointed out that instead of speaking of embodiment, the term "enmindment" could be used as well, since body and mind are not separate, but in some sense two sides of one coin, i.e., two ways of

2. See also Barbour (1999) who points out that the dichotomous concept of man in Christianity is a result of the Greek dualism of body and soul and actually not supported by the biblical view.

Embodiment and social interaction: A cognitive science perspective

133

describing the same process, namely the activity of the (human) organism in its social and physical environment. Currently, the increased interest in both bodilylbiological and sociocultural processes in cognitive science has led to a partial rediscovery of psychological, philosophical and biological theories that predated cognitive science. Yet these theories received very little attention during the first decades of cognitive science which at the time was dominated by the computer metaphor. This applies, for example, to the works of the Russian scholar Vygotsky, who already in the 1930s suggested that social interaction is not an add-on to, but in fact a necessary requirement for individual cognitive development, an idea revived in recent humanoid robotics research (cf., Lindblom and Ziemke 2003), and the French philosopher Merleau-Ponty, who already in the 1940s argued that what he called intercorporeality constituted the basis of intersubjectivity and social interaction, an idea that has been rediscovered or confirmed in recent neuroscientific findings involving so-called mirror neurons (cf., Gallagher this volume). These examples illustrate that modem cognitive science still might have much to gain from re-evaluating and possibly incorporating some of these "old" ideas. The next section therefore provides some historical perspectives, ending with the "cognitive revolution" and the criticisms thereof. Section 3 then further discusses the role of embodiment and social interaction in cognitive science today. Next, Section 4 discusses the positions of "simple" vs. "radical" embodiment in current theories of embodiment, and finally, Section 5 discusses implications for embodied cognitive science and its possible contributions to the development of interactive technology.

2.

Historical perspectives

Cognitive science, which as a discipline only has been around for a few decades, can be said to have a short history but a long past, given that its philosophical roots can be traced back to ancient Greece (Gardner 1987). Even the view of the human being as a reasoning device, calculating according to rules, has its origin in the work of Plato, who argued that phenomena that could not be formalized explicitly, such as bodily skills and feelings, should not count as knowledge. Consequently, he distinguished between the rational mind and the body with its emotions and skills. This was the starting point of the Western philosophical tradition, assuming that reckoning is the "language of the mind" (Dreyfus 1979; cf., Johnson and

134

Jessica Lindblom and Tom Ziemke

Rohrer this volume; Lakoff and Johnson 1999). This conceptual separation of mind and body was further developed by Aristotle who divorced the practical from the theoretical and defined the human being as the "rational animal". Descartes then confirmed this view, rather than adding anything radically new, and formulated the dualist viewpoint of mind and body as two different substances (Cisek 1999; Dreyfus 1979; see also Costall this volume). That means, while other animals, according to Descartes, are mere mechanisms, or bodies without minds, humans alone possess a soul which he also described as the "pilot of the corporeal boat" (Dreyfus 1979; Freeman and Nufiez 1999). Vitalists, on the other hand, disagreed with this mechanistic view of life and argued that all living creatures had some sort of "vital energy" comparable to a spirit or soul. The issue never really got resolved, but new life was brought into the debate through the work of Darwin (cf., Ziemke 2001b). After Charles Darwin's 1859 book, The Origin of Species, many researchers began to search for unified theories that would explain the behavior of humans as well as other organisms. Interestingly, this resulted in two almost opposite lines of research, which nevertheless can be traced back, at least partly, to Darwin himself. Darwin tried to explain how different species had evolved by assuming that patterns of behavior were the product of evolution and that there was a mental linkage between animals and humans (Cosmides et al. 1997; McFarland 1993). In modem terms, Darwin certainly viewed the mind as embodied and did not believe in a mind separate from the body. For example, he wrote in his personal notebook (1836-1844) that "experience shows the problem of the mind cannot be solved by attacking the citadel itself - the mind is function of [the] body - we must bring some stable foundation to argue from" (Sheet-Johnstone 1999: 435). However, instead of studying mind as "function of the body", many later researchers reduced the body and its interactions with the environment to only include the brain itself (cf., Sheets-Johnstone 1999). Darwin's 1872 book, The Expression of the Emotions in Man and Animal, can be considered the first modem book on behavior, although he focused primarily on facial expressions and did not pay much attention to gestures and body language. Darwin himself combined psychology and biology, but he could also be said to have caused the split between the two. Although his work was a serious attack on dualism in the sense that he tried to bridge the presumed gulf between body and soul, his fear of public opinion resulted in two parallel lines of explanations for the development

Embodiment and social interaction: A cognitive science perspective

135

of the human mind, namely a phylogenetic line that stressed descent, and an adaptationist line that emphasized selection (Cosmides et al. 1997). The adaptationist perspective has generally been ignored within psychology, while the phylogenetic branch has been further explored, partly following Darwin, who mainly tried to explain the human mind from a phylogenetic perspective (Cosmides et al. 1997). Given that Darwin believed in a single universal line of development, in order to make this claim work, he had to run counter to his earlier ideas which stressed the role of environmental conditions and adaptations, resulting in several lines of evolutionary development. Consequently, he had to argue for some kind of supremacy of reason in reference to some general inherited endowment. The adaptationist perspective, however, would not fully support this view, since it focused on the differences between separate species. Darwin was worried that this perspective could be interpreted as supporting the dualist view, in the sense that different qualities in separate species actually could lead to a uniquely human soul (Cosmides et al. 1997). Darwin's work had great general impact and brought about increased interest in the study of behavior (AlIen and Bekoff 1997; Cosmides et al. 1997; McFarland 1993). Subsequently, two contrary lines of behavioral research emerged. Firstly, there were those who attributed human-like subjective mental qualities to other vertebrates and even invertebrates. Darwin (1872) himself, for example, described dogs as feeling pleasure when doing what they considered their duty, and ants as feeling despair at having lost their home to a spade (cf. Sparks 1982). He was, nevertheless, well aware that many of these behaviors were purely instinctive. His protege Romanes (1882) was less careful and provided anthropomorphic explanations for a large range of animal behaviors, attributing to animals human-like degrees of intelligence, consciousness and emotions, based solely on his own observations (cf. Ziemke 2001b). Secondly, there were the mechanists who also believed in common behavioral mechanisms in humans and other organisms, but considered the attribution of subjective mental qualities as unscientific. Instead they viewed animals - unlike Descartes, often including humans - as Cartesian puppets, i.e., mere objects guided by their environment. Darwin himself in his 1880 book, The Power ofMovement in Plants, compared the downward digging behavior of earthworms and moles to the downward growth of roots in response to gravity. Von Sachs (1882) further elaborated the view of plants as guided by "forced movements", which he termed tropisms, and demonstrated the orientation of various plant organs towards or away from

136

Jessica Lindblom and Tom Ziemke

stimuli such as gravity, light or moisture. Loeb (1890, 1918) extended von Sachs' theory of tropisms to animal behavior to stress what he saw as the fundamental identity of the curvature movements of plants and the locomotion of animals. Loeb's own experiments revealed many different kinds of tropisms in caterpillars, moths, shrimps, etc., directed towards or away from light, gravity, chemicals, etc. This led him to apply his theory to human intelligence and to claim even humans to be mere mechanisms "compelled to react to physical conditions like corn waving in the wind" (Sparks 1982). Thus, his theory became the opposite of Romanes' view of feeling, conscious animals. At about the same time, Sherrington identified the reflex as the elementary unit of behavior and described the reflex arc as consisting of three elements: an effector organ, a conducting nervous pathway leading to that organ, and a receptor to initiate the reaction. As a consequence, experi- . mental psychology made the supposedly distinct elements of stimulus and response to its units of analysis and treated them as largely separate (cf. Sherrington 1906; Ziemke 2001b). Sherrington's own initial description of the reflex arc, nonetheless, implied an integrated neural circuitry of "coordinated action". Dewey (1896) was also very critical of the more restricted interpretation of the reflex arc and argued that this psychological model was a legacy of dualism. He rejected the linear model of stimulus and response, proposing instead the concept of circular sensorimotor coordination (Dewey 1896; cf. Clancey 1997). Hence, he put forward an alternative explanation of how the mind works, without an intervening "consciousness" controller as a kind of soul, while at the same time stressing the embodied nature of mind, with sensorimotor coordination as the key underlying mechanism (cf. Johnson and Rohrer this volume). Cisek (1999) has pointed out two important issues in the development of behavioral research at the beginning of the 20 th century. Firstly, there was a split in the study of living organisms, resulting in one line that concentrated on studying behavior, whereas the other focused on physiology. Secondly, the behavioral sciences as a whole suffered from a certain envy of formal approaches. Many researchers therefore tried to explain behaviors and mental phenomena in a mathematical fashion in order to validate the study of behavior "scientifically", worried that otherwise their research might not be considered serious science. Behaviorism, which became the dominant approach in psychology in the beginning of the 20th century, in particular in the US, argued that only observations of overt behavior should be the object of study (AlIen and

Embodiment and social interaction: A cognitive science perspective

137

Bekoff 1997). The approach had its roots in mechanistic theories and the animal learning work of Pavlov, who viewed the biological body as a machine (McFarland 1993). Behaviorists aimed to explain behavior as the result of learning, in terms of classical and operant conditioning. Antimenta/ism and equipotentia/ity, the assumption that there was no inherent bias in organisms towards some stimulus-response associations over others, became the two pillars of behaviorism (Cosmides et al. 1997). This was the first rigorous attack against the dualistic assumption in the sense that it totally ignored mental content. Accordingly, little, if any, interest was directed towards embodiment other than in a mechanistic sense. All along, while psychology was dominated by mechanistic and behaviorist ideas, there were researchers working on alternative, constructivist and phenomenological conceptions of behavior and mind. Some of these can be viewed as the forerunners or roots of modem embodied and situated theories of cognition. Researchers like von Uexldill, Heidegger and Merleau-Ponty, for example, like behaviorists, emphasized, in different ways, the situated and embodied nature of mind and the crucial role of agentenvironment interaction. However, completely unlike behaviorists, they focused on the subjective, phenomenal dimensions of "being-in-theworld", as Heidegger (1962) called it. At about the same time, Vygotsky and Piaget developed their highly influential theories of cognitive development which, in different ways, emphasized the embodied, (socially) situated and constructive nature of cognition. Due to their relevance to modem theories of embodiment and/or its role in social interaction, the contributions of these researchers will be discussed in more detail in the following. Jakob von Uexldill (1864-1944), a German biologist strongly inspired by the Kantian insight that all knowledge is determined by the knower's subjective ways of perceiving and conceiving, considered it the task of biology to expand Kant's research by investigating the role of the body and the relationship between subjects and their objects (von Uexldill 1928; cf. Ziemke 2001b). According to von Uexldill, the main problem with mechanist and behaviourist ideas was that they overlooked the organism's subjective nature. In his own words: The mechanists have pieced together the sensory and motor organs of animals, like so many parts of a machine, ignoring their real functions of perceiving and acting, and have gone on to mechanize man himself. According to the behaviorists, man's own sensations and will are mere appearance, to be considered, if at all, only as disturbing static. But we who still hold that

138

Jessica Lindblom and Tom Ziemke

our sense organs serve our perceptions, and our motor organs our actions, see in animals as well not only the mechanical structure, but also the operator, who is built into their organs as we are into our bodies. We no longer regard animals as mere machines, but as subjects whose essential activity consists of perceiving and acting. (von Uexkii1l1957: 6)

It should be noted that von Uexkiill's view is not vitalistic (cf. Emmeche 2001; Ziemke and Sharkey 2001). He described animal behavior as guided by "successive reflexes" each of which is "elicited by objectively demonstrable physical or chemical stimuli"; at the same time he also emphasized that the organism's components are forged together to form a coherent whole, i.e., a subject that acts as a behavioral entity which forms a "systematic whole" with its Umwelt, namely, its subjective world of perception and action. The French philosopher Maurice Merleau-Ponty (1908-1961), strongly inspired by Gestalt psychology and also partly by Heidegger and von Uexkiill, argued that the mind was essentially embodied and interacting with the surrounding world (e.g., Merleau-Ponty 1963, 1962; cf. Dreyfus 1979). More specifically, he argued that it is actually the body which provides meaning or intentionality for the mind. His core concept was the idea that "I am my body" as a kind of "embodied cogito", which means that it is not only the brain that does the thinking, but the whole body (Priest 1998; cf. Roth this volume). He argued that it is the body that has the necessary "knowledge" to perform tasks present at hand, since it "knows how to act" and "how to perceive" through the history of its phylogenetic and ontogenetic interactions with the environment. Merleau-Ponty (1969) tried to find the fundamental primitives of the mutual interdependence between the organism and its world, and he concluded that both are different aspects of the same primary whole of "brute being", which he referred to as the flesh (cf. Mingers 2001). That means, the lived body, according to Merleau-Ponty, has a twofold character, being able to see or touch on the one hand, and being seen or touched on the other. In his own words, the flesh is "not matter, is not mind, is not substance. To designate it, we should need the old term "element" [... ] in the sense of a general thing, midway between the spatio-temporal individual and the idea" (Merleau-Ponty 1969, quoted in Mingers 2001: 116). In addition to questions concerning low-level functions and bodily intentions, Merleau-Ponty tried to situate his theory in a cultural context, arguing that bodily behavior constitutes the most basic form of communication and cooperation among organisms (cf. Gallagher this volume). Even

Embodiment and social interaction: A cognitive science perspective

139

human language is an extension of these bodily acts, and when humans have acquired a language, it will control human cognition and communication (Loren and Dietrich 1997; Neuman 2001). Merleau-Ponty (1969) treated the relation between language and thought as an intertwined process, arguing that the use of language actually is the process of thinking; either we speak to ourselves or to other persons (Mingers 2001). To summarize, the main characteristics of Merleau-Ponty's work are his nondualistic and anti-behavioristic view of embodied experience, his argument that bodies in fact are intentional in themselves, and the view of cognition as a biological phenomena, rather than a mental one. The Swiss biologist/psychologist Jean Piaget (1896-1980) similarly stressed the importance of sensorimotor activity for the emergence of intelligent behavior (e.g., Piaget 1952). He first and foremost viewed "cognition as an instrument of adaptation, as a tool for fitting ourselves into the world of our experiences" (von Glasersfeld 1995: 14). That means, cognition in Piaget's sense is about the organization of an agent's sensorimotor experiences and interactions with the environment - a view very similar to that of von Uexkiill (cf. Ziemke 2001b). Hence, the relationship between action and lmowledge is central in Piaget's genetic epistemology, which posited lmowledge as constructed by the child in interaction with its environment. The basic organizational force of intellectual development, according to Piaget, is logic, and this is highlighted in his theory which characterizes different forms of logical thinking. Consequently he claimed that his theory of cognitive development was universal (cf. Sinha and Jensen 2000; Wadsworth 1996). It has been argued that Piaget neglected the social dimension, but Sinha and Jensen (2000) have pointed out that this is a common misunderstanding (cf. Cole and Wertsch 1996). For example, Piaget stated that "[h]uman lmowledge is essentially collective and social life constitutes an essential factor in the creation and growth of lmowledge, both pre-scientific and scientific" (Piaget 1995: 30). He actually stressed the social dimension as an essential factor for the development of cognition, supposing that sociocultural factors could either accelerate or retard the developmental process, but perhaps not change its direction towards the terminal point of (logical) cognitive development (Sinha and Jensen 2000). Piaget's most significant contribution probably is the claim that cognitive changes are the outcome of a constructive developmental process that occurs in sensorimotor interaction with the environment. However, it should also be mentioned that Piaget's theory has been criticized heavily

140

Jessica Lindblom and Tom Ziemke

(cf. e.g., Miller 1983; Wadsworth 1996). Boden (1994), for example, noted that Piaget actually underestimated children's innate capabilities, both initially and later on during development. Furthermore, Piaget's theory was based on intellectual ideals of the Western tradition, and did not pay much attention to cultural differences in cognitive development, as illustrated by his claim that his theory was universal (cf. Cole and Wertsch 1996). The role of culture was strongly emphasized, on the other hand, in the development theory of Russian scholar Lev Vygotsky (1896-1934). He rejected mechanistic and behaviorist theories because they disregarded what he considered essential differences between human and animal intelligence (cf. Lindblom and Ziemke 2003). He wanted a psychological theory that would describe the development of what he considered exclusively human abilities, and, as a result, proposed that individual cognitive development requires a sociocultural embedding through certain transformation processes. These transformations, from elementary to higher mental functions, take place via signification, i.e., the shift of control from the environment in elementary mental functions, to the individual's voluntary regulation of her behavior in higher mental functions. This voluntary regulation of individual behavior, according to Vygotsky (1978), is accomplished through the use of mediating artificial or self-generated stimuli, functioning as a link between external stimulus and response. He argued that the incorporation of such psychological tools was the key difference between animal and human behavior, and summarized this position in the statement that "the central fact about our psychology is the fact of mediation" (Vygotsky 1933, quoted from Wertsch 1985:15). Psychological tools, according to Vygotsky, also function as regulators of human social behavior, and, among these tools, language is an especially important "organizer", both in the form of speech and written text. The other transformation process of cultural cognitive development is internalization, as stated in Vygotsky's (1978) general law of cultural development, which states that every function in the child's cognitive development appears twice, first between humans (at a social interaction level) and then in the child's mind (becomes internalized at the individual level). Hence, the cognitive abilities of an "enculturated" adult are the product of developmental processes, in which "primitive" and "immature" humans are transformed into cultural ones through social interactions, leaving room for different forms of intelligence. A common criticism (and misunderstanding) ofVygotsky's work is that he focused almost exclusively on the sociocultural aspects, but neglected

Embodiment and social interaction: A cognitive science perspective

141

the biological line of development (e.g., Davydov and Radzikhovskii 1985). However, as Wertsch (1985) pointed out, Vygotsky considered biological factors as necessary, but not sufficient conditions, stating for example that "culture creates nothing; it only alters natural data in conformity with human goals" (Vygotsky 1960, cited in Wertsch 1985:47). Elsewhere he stated: Both planes of development - the natural and the cultural - coincide and mingle with each other. The two lines of change interpenetrate each other and essentially form a single line of sociobiological formation of the child's personality.3 (Vygotsky 1960, quoted from Wertsch 1985:41)

In summary, despite the fact that Vygotsky's and Piaget's theories are

commonly contrasted as focused on the "child in society" and the "child as a solitary thinker" (cf. Cole and Wertsch 1996), respectively, they are in fact largely compatible and agree in viewing knowledge as constructed through the interaction of biological and sociocultural factors in the course of cognitive development. Despite serious criticisms and the development of the above competing theories, behaviorism remained the dominant approach in psychology, particularly in the US, throughout the first half of the 20 th century. Its impact started to decrease in the mid-1950s, mainly for two reasons: firstly, ethologists challenged the equipotentiality assumption (Breland and Breland 1961; Garcia and Koelling 1966); secondly, the advent of the computer seemed to challenge anti-mentalistic assumptions and paved the way for the so-called "cognitive revolution" (Gardner 1987). Cognitive science in general, from its beginnings in the mid-1950s, and its focus on internal representations in particular, were largely conceived as reactions against the anti-mentalistic behaviorist stance. Little attention, if any, was therefore directed to studies in biology and animal behavior. This was partly because cognitive science aimed for supposedly higher mental processes which animals were assumed to lack, and partly because studies of animal behavior were associated with the "horror" of behaviorism (Gardner 1987). As a result, other theories that stressed agent-environment interaction, such as the work of von Uexki.ill, Heidegger, Vygotsky, Merleau-Ponty and Piaget, were largely ignored during the era of computationalist cognitive science. In a nutshell, while behaviorists had treated mind as an opaque box in a transparent world, computationalist cognitive science treated it as 3. It should be noted that the translation "sociobiological" should not be confused with the later use of the term "sociobiology" by Wilson (1975).

142

Jessica Lindblom and Tom Ziemke

a transparent box in an opaque world (Lloyd 1989; cf. Sharkey and Ziemke 1998). According to the computer metaphor in cognitive science, cognition takes place inside the skull in the form of abstract symbol manipulation, while the body only serves as an input and output device, i.e., a physical interface between internal program (cognitive processes) and external world (cf. e.g., Pfeifer and Scheier 1999). According to this view, the essence of cognition is that minds or brains, just like computers, "accept information, manipulate symbols, store items in "memory" and retrieve them again, classify inputs, recognize patterns and so on" (Neisser 1976: 5). Representations, viewed as based on a correspondence or mapping between elements of the external world and their internal correspondents, played a crucial role in this conception of mind, since in some sense they constituted the only link between agent and environment. Agents, and their environments, were of course also still considered to be physical, but the relation between body and mind was considered as similar to the relation between hardware and software in a computer. That means, the body was viewed a mere physical implementation of the mind, which obviously needs some physical instantiation (cf. Chrisley and Ziemke 2003), but apart from that is largely implementation-independent. This functionalist view (e.g., Putnam 1975), often summarized by the slogan that "[m]inds are simply what brains do" (Minsky 1975), might sound similar to Darwin's statement that "mind is function of the body". Nonetheless, it left out the body and was interpreted entirely differently. Pfeifer and Scheier summarized the functionalist view as follows: Functionalism [... ] means that thinking and other intelligent functions need not be carried out by means of the same machinery in order to reflect the same kinds of processes; in fact, the machinery could be made of Emmental cheese, so long as it can perform the functions required. In other words, intelligence or cognition can be studied at the level of algorithms or computational processes without having to consider the underlying structure of the device on which the algorithm is performed. From the functionalist position it follows that there is a distinction between hardware and software: What we are interested in is the software or the program. (Pfeifer and Scheier 1999:43)

The reason for the success of the computer metaphor for mind was that it offered convincing answers to major questions that confronted cognitive science during its initial decades (Cisek 1999). Computationalism offered non-dualistic explanations of internal states, such as "memory". Further-

Embodiment and social interaction: A cognitive science perspective

143

more it offered the exciting metaphor of mental states and processes acting as the software running on the brain's hardware. This seemed to be an elegant solution to the mind-body problem, bridging the gulf between body/biology (hardware) and mind/psychology (software). Finally, computationalist psychology also provided a longed-for mathematical formalism for the mechanisms underlying behavior. It is therefore hardly surprising that the computer metaphor became the dominant model of how the mind works (Cisek 1999). However, in the late 1970s several criticisms of computationalism, in particular the resulting research in artificial intelligence (AI), emerged (Dreyfus 1979; Searle 1980; cf. Ziemke 2001b). The overall concern in these criticisms was in fact the lack of embodiment and situatedness. Dreyfus (1979), strongly inspired by the Heideggerian notion of being-in-theworld, questioned traditional AI's use of explicit, symbolic representations, arguing that the resulting computer programs lacked situatedness since they represented descriptions of isolated bits of human knowledge "from the outside". Dreyfus' (1979) explanation of human situatedness and a computer program's lack thereof is still worth quoting at length: Humans [... ] are, as Heidegger puts it, always already in a situation, which they constantly revise. Ifwe look at it genetically, this is no mystery. We can see that human beings are gradually trained into their cultural situation on the basis of their embodied pre-cultural situation [... ] But for this very reason a program [... ] is not always-already-in-a-situation. Even if it represents all human knowledge in its stereotypes, including all possible types of human situations, it represents them from the outside [... ]. It isn't situated in anyone of them, and it may be impossible to program it to behave as if it were. [... ] It seems that our sense of our situation is determined by our changing moods, by our current concerns and projects, by our long-range selfinterpretation and probably also by our sensory-motor skills for coping with objects and people - skills we develop by practice without ever having to represent to ourselves our body as an object, our culture as a set of beliefs, or our propensities as situation-action rules. All these uniquely human capacities provide a "richness" or a "thickness" to our way of being-in-theworld and thus seem to play an essential role in situatedness, which in turn underlies all intelligent behavior. (Dreyfus 1979: 52-53)

From this very brief overview of the historical roots, predecessors and critiques of traditional cognitive science, it should be clear that the idea of an embodied mind is by no means new. Although theorists have sought to

144

Jessica Lindblom and Tom Ziemke

understand the mind from different perspectives, the rational and formalized view was the dominating approach in cognitive science for a long time, and consequently the role of the body and the environment - physical as well as social- was largely disregarded. As Dreyfus (1979) pointed out, the computer, as well as the computer metaphor for mind, is the product of over 2,500 years of traditional thinking in Plato's footsteps. In that sense, as Cisek (1999) noted, the cognitive revolution was just "old wine in new bottles". Since the late 1980s, however, cognitive science has re-discovered or re-invented many, although not all, of the "pre-revolutionary" ideas and revived them in theories that acknowledge the embodied, situated, distributed and sociocultural nature of the human mind, a topic which will be discussed in more detail in the following section.

3.

Embodiment and social interaction: Current perspectives

Today, there is a growing interest in embodiment in many areas of cognitive science. Indeed, many researchers consider embodiment a necessary requirement for intelligence and mind. However, among proponents of embodied cognition there are nowadays a large number of notions and conceptions of embodiment and embodied cognition (e.g., Chrisley and Ziemke 2003; Ziemke 2003; cf. Ziemke and Frank this volume), a fact also amply illustrated by the diversity of perspectives collected in this volume. This diversity is also somewhat characteristic of the current state of cognitive science, which during the last twenty years has developed from a fairly unanimous agreement on the centrality of computation and (symbolic) representation to a pluralistic inter-discipline that constantly challenges and re-evaluates its own foundations and conceptions. As already mentioned in the introduction, in recent years both the neuroscientific and the anthropological perspective have steadily gained ground in cognitive science. This has lead to new sub-disciplines such as social cognitive neuroscience (e.g., Blakemore, Winston and Frith 2004) which, also for technical reasons would have been inconceivable only ten years ago. Thus, it is becoming increasingly accepted that cognition does not, as Gardner (1987) described it, constitute or require "a level of analysis wholly separate from the biological or neurological, on the one hand, and the sociological or cultural, on the other" (cf. above). In fact, the opposite is true: many cognitive scientists today are actively researching the dependence of cognitive

Embodiment and social interaction: A cognitive science perspective

145

processes on their neurobiological realization, their sensorimotor embodiment as well as their sociocultural situatedness. Somewhat surprisingly, however, despite the rich discussion of different aspects and forms of embodied cognition, the social dimension of embodiment has not yet received much attention. However, recent work in cognitive science and related disciplines indicates that the body has several important roles in social interactions and cognition. Barsalou et aI. (2003), for example, have identified a number of different empirical findings, mostly from social psychology experiments, which illustrate the role of embodiment and social cognition. More specifically, they identified four types of social embodiment effects which are discussed in more detail in the following. Firstly, perceived social stimuli do not only produce cognitive states, but also bodily states. For example, it has been reported that high school students receiving good grades in an exam adopted a more erect posture than students that received poor grades (cf. Barsalou et aI. 2003). As Barsalou et aI. pointed out, it is unclear whether the bodily reaction is triggered by the social event/stimulus directly, or by mediating mechanisms. In this example it could be the case that an emotional state was triggered first which in turn resulted in a bodily state. Another example is that subjects primed with concepts related to elderly people (e.g., "gray", "bingo", "wrinkles") exhibited embodiment effects such as slower movement when leaving the experimental lab, as compared to a control group primed with neutral words. Several other studies also show similar effects (cf. Barsalou et aI. 2003). Secondly, the observation of bodily states in others often results in bodily mimicry in the observer. Studies show, for instance, that experimental subjects often mimic an experimenter's actual behavior, e.g., rubbing the nose or shaking a foot, and that subjects also tend to mimic observed facial expressions, a fact documented widely in the literature (cf. Barsalou et aI. 2003). Thirdly, bodily states produce affective states, which mean that embodiment not only facilitates a response to social stimuli but also produces tentative stimuli. For example, evidence offacial elicitation or facial feedback is well documented in the literature. According to Barsalou et aI., many studies demonstrate that the shaping of the face to an emotional expression tends to produce the corresponding affective state. For example, it has been demonstrated that subjects rated cartoons differently when holding a pen between their lips than when holding it between their teeth. The

146

Jessica Lindblom and Tom Ziemke

latter triggered the same musculature as smiling, which made the subjects rate the cartoons as funnier, whereas holding the pen between the lips activates the same muscles as frowning and consequently has the opposite effect. That means, the expressions associated with the musculature affected the evaluation of the cartoon although the subjects were not even aware of the fact that their musculature had been manipulated into a facial expression. Similarly, evidence of bodily elicitation has been provided in studies showing that bodily positions or postures actually influenced the subjects' affective state, for example, that subjects in an upright position experienced more pride than subjects in a slump position (cf. Barsalou et al. 2003). Finally, compatibility between bodily and cognitive states enhances performance. For instance, several motor performance compability effects have been reported, and it has been demonstrated that subjects actually responded faster to "positive" words (e.g., "love") than "negative" words (e.g., "hate") when asked to pull a lever towards them. Other studies have shown that embodiment-cognition compatibility reduces the required processing resources in secondary task performance. Taken together, these examples and other studies demonstrate that there is a strong relation between embodied and cognitive states. Barsalou et al. concluded that embodiment is fundamental in human cognitive processing, and that a common mechanism seems to produce such effects across various domains. Barsalou et al. (2003) offered a theoretical framework of social embodiment to explain these phenomena based on internal simulations (cf. Svensson, Lindblom and Ziemke this volume), which can be considered as a mix composed of the traditional view of symbolic information processing, mental imagery and embodiment. Even though their framework is an important step towards highlighting the importance of sensorimotor aspects in cognitive processing, it still constitutes an example of the position that Clark (1999) referred to as "simple embodiment". According to this view, traditional cognitive science can roughly remain the same; or, in Clark's words, "facts about embodiment and environmental embedding" can be treated as mere "constraints upon a theory of inner organization and processing". In Barsalou et al.' s case, for example, it could be said that their theory adds an embodied icing to the traditional information-processing cake in the sense that it acknowledges embodiment by taking as its starting point perceptual/modal representations. Yet it continues to explain cognition largely in terms of internal representations and the computational processes manipulating them.

Embodiment and social interaction: A cognitive science perspective

147

The position of radical embodiment, on the other hand, is, as Clark (1999:348) formulated it, "radically altering the subject matter and theoretical framework of cognitive science" by challenging traditional conceptions of representation and computation and questioning the conceptual separations between perception and action as well as between brain, body and world. Work on so-called mirror-neurons as the neurobiological mechanism of "intercorporeality" as the basis of social interaction (cf. Gallagher this volume) as well as simulation theories of mind-reading (cf. Gallese and Goldman 1998) are good examples of more radically embodied views of social cognition. Dautenhahn (1997), for example, hypothesized that a phenomenological dimension of social understanding might be founded in embodied mechanisms that allow humans to read social signs and other agent's minds. She suggested that an agent's own body can be used as the point of reference for "simulating" another agent's emotional stance. Such a mechanism might be found in a particular type of visuo-motor neurons in premotor cortex, so-called mirror neurons, which are active in both the execution of goal-related motor actions (e.g. grasping movements) and their observation in conspecifics (e.g. Gallese and Goldman 1998). Although the existence of mirror neurons so far has been experimentally confirmed only in monkeys4, there is strong empirical evidence that such a system actually is present in human beings as well (e.g., Arbib 2006; Gallese et al. 2002). This suggests that the function of this matching system might be a part of, or a precursor to, a general mind-reading capability that allows you to adopt the point of view of other conspecifics by matching or simulating their mental states with a resonant state of your own, i.e., putting yourself in another's "shoes" in order to understand or predict mental states and behavior. Empirical findings support the general idea (cf. Svensson, Lindblom and Ziemke this volume). For example, it has been found that observers undertake motor facilitation in the same muscles as used by the observed individual. That means, even while only observing actions of another individual, a neural "triggering" event in fact takes place in the observer. Gallese et al. (2002: 459) therefore suggested "that the capacity to empathize with others [...] may rely on a series of matching mechanisms that we just have started to uncover". Although Gallese and Goldman (1998) have stressed

4. Single neuron recordings of the type used in these experiments eventually destroy the neurons recorded from, and thus for ethical reasons cannot be used in humans.

148

Jessica Lindblom and Tom Ziemke

that imitation behavior has not been observed in connection to mirror neuron activity, Rizzolatti et al. (2002) have hypothesized that various lowand high-level resonance mechanisms may be used under the banner of "imitation", ranging from response facilitation to emulation and "true" imitation. Currently the role of mirror neurons in embodied simulation processes as the basis of social interaction in general (e.g., Gallese, Keysers and Rizzolatti 2004), and language in particular (cf. Arbib 2005; Rizzolatti and Arbib 1998), is receiving a great deal of attention. Nevertheless, these mechanisms still are far from being well understood, and there is a need to further integrate them into embodied/situated cognitive theories to develop a more thorough understanding of the mechanisms that facilitate the human sensitivity to social cues from the perspective of radical (social) embodiment.

4.

Discussion: Simple vs. radical (social) embodiment

As discussed in the introduction, as well as in several other contributions to this volume, the role of the body in cognitive processes has received an increasing amount of attention in recent years from different perspectives and disciplines. While this certainly is a step, or in fact several steps, in the right direction, i.e., towards an interdisciplinary understanding of the embodied and socioculturally situated mind, it might be worth keeping in mind that there also is a substantial risk of premature superficial agreement. As several authors have pointed out, there currently is a wealth of diverse notions, definitions and conceptions of embodiment (cf. e.g., Chrisley and Ziemke 2003; Clark 1999; Rohrer this volume; Svensson, Lindblom and Ziemke this volume; Wilson 2002; Ziemke 2001a, 2003; Ziemke and Frank this volume). On the one hand, this can be interpreted positively since it shows that the embodiment of mind is studied from a number of perspectives which eventually might converge to form a fuller understanding than any individual discipline or approach could have produced on its own. On the other hand, after two decades of research on embodied cognition, the apparent lack of coherence, could be interpreted as confirmation of the recurrent criticism that in fact all that embodied cognitive theories have in common is their rejection of traditional cognitive science (cf. Chrisley and Ziemke 2003).

Embodiment and social interaction: A cognitive science perspective

149

That not everybody means the same thing by "embodiment" should be clear by now. As discussed in the previous section, one distinction particularly relevant to cognitive science is that made by Clark (1999) between the positions of "simple embodiment" and "radical embodiment". A typical example of the former is the position of robotic functionalism (Harnad 1989), according to which cognition is computational after all, but mental representations need to be "grounded" in sensorimotor interaction with the environment (Hamad 1990). The body's role, according to this view, is still very much that of the (software) mind's physical (hardware) interface to the world it represents. That means, although the view of grounded mental representations rejects behaviorism, the mechanistic view of the body as such is largely maintained (cf. Ziemke 2001b). While this view of embodiment and representation grounding is often not formulated explicitly, it nevertheless implicitly underlies many current conceptions of embodied cognition, in particular in artificial intelligence (cf. Ziemke 2004). This fact is also reflected in the way that some of the theories discussed in Section 2 have been incorporated into today's research. Von Uexkiill's notion of Umwelt, for example, i.e., the idea that each creature necessarily has its own subjective, perception- and action-dependent view of the world, has received much attention in' embodied artificial intelligence and robotics (e.g., Brooks 1986; Clark 1997; Emmeche 2001; Prem 1997; Sharkey and Ziemke 1998; Ziemke and Sharkey 2001). However, the organismic roots (cf. Damasio 1995, 1999) and the phenomenal nature of subjective experience, which were strongly emphasized by von Uexkiill and Merleau-Ponty, are still largely ignored. Similarly, the developmental psychology of Vygotsky and Piaget has received much attention in humanoid robotics. Nonetheless in these cases it is also the functional perspective on agent-environment interaction that has been adopted, while the biological dimension has been largely ignored or abstracted in the form of computationalleaming mechanisms (cf. Lindblom and Ziemke 2003). It should be noted that, while the examples from artificial intelligence are perhaps the most obvious cases (cf. Ziemke 2004), instantiations of the simple embodiment position are of course equally common in other areas of cognitive SCIence. A prime example of "radical embodiment", on the other hand, is the work of Maturana and Varela (1980, 1987) on autopoiesis and the biology of cognition (cf. Johnson and Rohrer this volume). As we have discussed in detail elsewhere (Ziemke and Sharkey 2001; Ziemke 2001b), the notion of organisms as autopoietic, i.e., self-creating and maintaining, is very similar

150

Jessica Lindblom and Tom Ziemke

in spirit to von Uexkiill's distinction between organisms as autonomous subjects and mechanisms as heteronomous physical objects. Particularly noteworthy in the context of radical embodiment is also Varela, Thompson and Rosch's (1991) work towards an enactive cognitive science (cf. also Thompson and Varela 2001), that has drawn much inspiration from the works of Merleau-Ponty and Piaget, among others, and in its turn inspired much current work seeking to further integrate phenomenology with embodied cognitive science and neuroscience (e.g., Dreyfus 2002; Gallagher this volume; Thompson and Varela 2001). Thus, many aspects of precognitive-science theoretical biology, psychology and philosophy have found their way into current theories of embodied cognition, although their interpretation can vary significantly depending on the interpreter's own commitment to simple or radical embodiment. It should also be noted that the distinction between simple and radical embodiment should neither be overrated nor underestimated. On the one hand, it really describes a continuum of positions rather than a binary distinction. Here, for example, we chose to categorize the position of Barsalou and colleagues as simple social embodiment because it maintains elements of computationalism and representationalism in the traditional sense, whereas we consider much work on mirror neurons to fall into the radical embodiment category because it pays more attention to the phenomenal nature of subjective and intersubjective experience (cf. Gallagher, this volume). But these distinctions are not as clear-cut as they might seem; for example, there are also purely computational interpretations of mirror neurons. On the other hand, it is useful, or in fact essential for developing an embodied cognitive science, to keep in mind that there are distinctions to be made and that there might be much theoretical tension hidden under a possibly premature, superficial agreement on "embodiment" and the rejection of traditional, computationalist ideas. It should be noted that traditional cognitive science "missed out" on much by simply rejecting behaviorism while ignoring the competing theories discussed here that indicated the importance of embodiment long before cognitive science picked up on it. Likewise, current embodied cognitive science would be ill advised to simply frame itself in opposition to the computationalism of traditional, supposedly disembodied cognitive science.

Embodiment and social interaction: A cognitive science perspective

5.

151

Implications for cognitive science and interactive technology

We would like to close this chapter by pointing out some issues that need to be addressed in order for current theories of embodied cognition, radical embodiment in particular, to develop into a "mature science of the mind" (Clark 1999: 349). Firstly, although embodiment has been much discussed, the role of the body in social interaction and cognition has been oversimplified. Much of the discussion has focused on what you might call the "static" body. For example, in AI it has been discussed what kind of physical realization, or implementation, of the "body" is necessary for cognitive processes, ranging from very simple agents with a minimum of sensors and motors to very complex organism-like, e.g., humanoid, robot bodies (cf. Ziemke 2001 a, 2003). However, the crucial aspect of the body in motion has received relatively little attention, although research in child development and anthropology has shown the relevance of locomotion experience for human cognition (cf. Campos et al. 2000; Famell 1995, 1999; Sheets-Johnstone 1999). We have elsewhere discussed in more detail (Lindblom and Ziemke 2005, 2006) the example of the so-called "nine month revolution" (Tomasello 1999), which is the time when Euro-American infants begin to develop triadic joint attention abilities and a basic understanding of a "self'. How and why this transition occurs is still little understood, but we have argued that it is no coincidence that this "revolution" occurs at around the same point in time as the onset of self-produced locomotion behavior, i.e., when infants start to creep or crawl (Lindblom and Ziemke 2005, 2006). In other words, when children begin to locomote by themselves, they acquire an individual experience of the surrounding world and how it is affected through their own actions and perceptions. It should be stressed that locomotion itself might not be the crucial factor. Instead, the child's cognitive and emotional development emerges from the dynamics of the bodily experience that result from its own locomotion behavior. As a result, the child begins to distinguish between itself and its surrounding world, i.e., the child quite literally experiences different perspectives on the world, and how these perspectives depend on its own actions, and thus a primitive "self' emerges. This emerging understanding is bootstrapped through socially scaffolded bodily experience, which gives the child access to the actual meaning of the social-communicative situation. Consequently, that understanding of perspective-taking might be used during embodied simulations of the type discussed in previous sections, allowing the child to

152

Jessica Lindblom and Tom Ziemke

simulate "off-line" what it would be like to be in the other person's situation, i.e., seeing the world from the other's physical and social perspective. This example illustrates once more how firmly "culture" and "nature", as well as "body" and "mind", are seam1ess1y intertwined and grounded in the dynamics of socially situated embodied experience. Secondly, side-stepping the dualist traps of "mind" vs. "body" as well as "nature" vs. "culture" (cf. Famell 1995; Rogoff 2003; Varela 1995) requires certain methodological consequences. A promising starting-point is to take a dialectical stance, following Vygotsky, and stress that the human mind is shaped through a course of "natural cultural development" (Rogoff 2003). Moreover, alternative analytic tools for studying the body and its movements are needed, which go beyond dista1 observers' "objective" descriptions of behavior. The latter are flawed from the perspective of embodied social interaction, since they do not consider the non-observable social situation at hand, which in many cases actually provides the meaning to the visible embodied actions. For example, it is necessary to investigate so-called "body language" or "non-verbal behavior" as one dynamically embodied system that unites mind and body in action (Famell 1995). By using the term "action" instead of "behavior", it is emphasized that socially embodied actions are a set of movements that have agency, meaning or intentions for the actual person or agent, since "bodies do not move and minds do not think - people just do" (Famel1 1995). That means, the "enactment" of the body is a social act, and in order to direct yourself, you have to consider how others will act and react in response to your own actions. In that sense, our biological embodiment constrains while cultural customs affect, but do not determine, the organization of socially embodied interactions (Fame11 1999). Consequently, there are some obvious methodological problems when observing, ana1yzing and illustrating socially embodied actions. For example, if you suppose that there is no split between verbal and non-verbal communication, that implies that there are two ways of expressing the same thought, verbal and gestura1 (non-verbal), that might be appropriate in different contexts (cf. Go1din-Meadow 2003). Hence, the analysis will have to consider both the spoken words and gestural movements. The dynamic nature of socially embodied actions causes problems when it comes to representation and illustration. Common media such as verbal descriptions, pictures or photographs, are quite "static" and therefore more dynamic representational forms might need to be developed.

Embodiment and social interaction: A cognitive science perspective

153

Thirdly, and closely related to the previous point, there is a need for an integrated framework of radical social embodiment that addresses the role and relevance of the body and its sensorimotor processes in social interaction. In doing so, one should not intend to bridge the 'gap' between, e.g., verbal vs. nonverbal interaction. Many crucial questions have been addressed only partly and/or only from one of the above perspectives. For example, what role(s) do bodily movement and gesture play in cognitive processes? How do the body and its sensorimotor processes affect social interactions and, more specifically, which social cognitive processes are affected, and what functional roles does the body serve? Such a framework of the fundamental roles of embodiment in social interaction is developed by Lindblom (2006, 2007). The developed framework explains and illustrates how embodiment is the part and parcel of social interaction and cognition in the most general and specific ways, in which dynamically embodies actions themselves provide meaning and agency. Furthermore, as the multitude of perspectives briefly described here indicates, there is also a need to investigate and analyze the different notions, perspectives and aspects of social embodiment addressed in the area, and disentangle their different roles in social interaction. For the moment, different perspectives on social interaction range from environment-mediated cooperation in social insects over the "expressiveness" of different bodily communication abilities in animals, humans and robots to theory of mind in humans (and to some degree in other primates and robots). But, from a comparative perspective, it still remains unclear to what degree different kinds and levels of embodied social interactions are grounded in bodily and/or cognitive differences but see Lindblom 2007. Last, but not least, it should be noted that investigating the role of embodiment in social interaction and cognition is not just yet another variation of the old philosophical mind-body problem. It also is highly relevant to and has strong implications for the development of interactive technology for human-computer interaction (HCI) and computer-supported cooperative work (CSCW) (cf. also Emmeche this volume). This is due to the fact that with the increasing use of technology in practically all areas of human life, social aspects have become more and more relevant. This is equally true for technology-supported social interaction between humans, i.e., the increasing use of advanced communication technology in social interaction or collaborative work. Moreover, human users interact with technology that increasingly tends to adopt (partly) social modes of interaction, e.g., between humans and humanoid robots (cf. Fong, Nourbakhsh

154

Jessica Lindblom and Tom Ziemke

and Dautenhahn 2003) or so-called "embodied" conversational computer interfaces (e.g., Cassell et al. 2000). Much of this technology is developed to better meet the strong human sensitivity to social stimuli and interactions. However, it has been noted, for example in the area of computersupported cooperative work, that there is a gap, the so-called socialtechnical gap, between what technology ideally should support socially and what currently it actually can support (cf. e.g., Ackerman 2002; Erickson and Kellogg 2002). That means, there still is a stark contrast between our embodied social interactions in the lived human world and the level of support that is offered by contemporary technology. At least partly, this is due to the fact that there is no sufficient understanding of the mechanisms underlying human social interaction which would allow a more seamless use or integration of technology. In particular the role of embodiment in human social interaction is still far too little understood, and current designs of information and communication technology are still dominated by outdated information processing models of human communication. In conclusion, we suggest that further re-evaluation and integration of some of the pre-cognitive-science theories discussed in this chapter, as well as other contributions to this volume, into the framework of modem theories of embodied cognition, along the lines pointed out above, can significantly contribute to both the further theoretical development of embodied cognitive science as such and its capacity to aid the development of future interactive technology.

References Ackermann, Mark S. 2002 The intellectual challenge of CSCW: The gap between social requirements and technical feasibility. In: John M. Carroll (ed.), Human-Computer Interaction in the New Millennium, 303-324. New York: Addison Wesley. AlIen, Chris and Marc Bekoff 1997 Species ofMind: The Philosophy and Biology of Cognitive Ethology. Cambridge, MA: MIT Press. Arbib, Michael A. 2005 From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics. Behavioral and Brain Sciences 28: 105-167.

Embodiment and social interaction: A cognitive science perspective

155

Barbour, Ian G. 1999 Neuroscience, artificial intelligence and human nature: theological and philosophical reflections. Zygon 34 (3): 361-398. Barsalou, Lawrence W., Paula M. Niedenthal, Aron K. Barbey and Jennifer A. Ruppert 2003 Social embodiment. In: Brian H. Ross (ed.), The Psychology of Learning and Motivation 43: 43-92. San Diego, CA: Academic Press. Blakemore, Sarah-Jayne, Joel Winston and Uta Frith 2004 Social cognitive neuroscience: Where are we heading. Trends in Cognitive Sciences 8 (5): 216-222. Boden, Margaret A. 1994 Piaget. London: Fontana Press. Breland, Keller and Marian Breland 1961 The misbehavior of organisms. American Psychologists 16: 661664. Brooks, Rodney A. 1986 Achieving artificial intelligence through building robots. MIT AI Lab Memo 899. Burgoon, Judee K., David B, Buller and William G Woodall 1996 Nonverbal Communication: The Unspoken Dialogue (2 nd ed). New York: McGraw-Hill. Campos, Joseph J., David I. Anderson, Marianne A. Barbu-Roth, Edward M. Hubbard, Matthew J. Hertenstein and David Witherington 2000 Travel broadens the mind. Infancy 1 (2): 149-219. Cassell, Justine, Joseph Sullivan, Scott Prevost and Elizabeth Churchill (eds.) 2000 Embodied Conversational Agents. Cambridge, MA: MIT Press. Chrisley, Ron and Tom Ziemke 2003 Embodiment. In: Encyclopedia of Cognitive Science, 1102-1108. London: Macmillan. Cisek, Paul 1999 Beyond the computer metaphor. In: Rafael Nunez and WaIter J. Freeman (eds.), Reclaiming Cognition - the Primacy of Action, Intention and Emotion, 125-142. Exeter: Imprint Academic. Clancey, William J. 1997 Situated Cognition: On Human Knowledge and Computer Representations. New York: Cambridge University Press. Clark, Andy 1997 Being There - Putting Brain, Body and World Together Again. Cambridge, MA: MIT Press. 1999 An embodied cognitive science? Trends in Cognitive Sciences 3(9): 345-351.

156

Jessica Lindblom and Tom Ziemke

Cole, Michael and James V. Wertsch 1996 Beyond the individual-social antimony in discussions of Piaget and Vygotsky. Human Development 39 (5): 250-256. Cosmides, Leda, John Tooby, Jonathan H. Turner and Boris M. Velichkovsky 1997 Looking back: Historical context of present practice - biology and psychology. In: Peter Weingart, Sandra D. Mitchell, Peter J. Richerson and Sabine Maasen (eds.), Human by Nature - between Biology and the Social Sciences, 52-64. Mahwah, NJ: Lawrence Erlbaum. Costall, Alan this vol. Bringing the body back to life: James Gibson's ecology of embodied agency. Damasio, Antonio R. 1995 Descartes' Error: Emotion, Reason and the Human Brain. New York: Avon Books. 1999 The Feeling of What Happens: Body and Emotion in the Making of Consciousness. New York: Harcourt Inc. Darwin, Charles 1859 The Origin ofSpecies. London: Murray. 1872 The Expression ofEmotions in Man and Animals. London: Murray. 1880 The Power ofMovements in Plants. London: Murray. Dautenhahn, Kerstin 1997 I could be you: The phenomenological dimension of social understanding. Cybernetics and Systems 25 (8): 417-453. Davydov, V. V. and L. A. Radzikhovskii 1985 Vygotsky's theory and the activity-oriented approach to psychology. In: James V. Wertsch (ed.), Culture, communication and cognition, 35-65. New York: Cambridge University Press. Dewey, John 1896 The reflex arc concept in psychology. Psychological Review 3: 357370. Dreyfus, Hubert L. 1979 What Computers Can't Do - a Critique of Artificial Reason. New York: Harper and Row. 2002 Intelligence without representation - Merleau-Ponty's critique of mental representation: The relevance of phenomenology to scientific explanation. Phenomenology and the Cognitive Sciences 1 (4): 367383. Emmeche, Claus 2001 Does a robot have an Umwelt? Reflections of the qualitative biosemiotics of Jakob von Uexkiill. Semiotica 134 (1/4): 653-693. this vol. On the biosemiotics of embodiment and our human cyborg nature.

Embodiment and social interaction: A cognitive science perspective

157

Erickson, Thomas and Wendy A. Kellogg 2002 Social translucence: Designing systems that support social processes. In: John M. Carroll (eds.), Human-Computer Interaction in the New Millennium, 325-330. New York: Addison Wesley. Famell, Brenda 1995 Do You See What I Mean? Plains Indian Sign Talk and the Embodiment ofAction. Austin: University of Texas Press. Famell, Brenda 1999 Moving bodies, acting selves. Annual Review of Anthropology 28: 341-373. Freeman, WaIter J. and Rafael NUflez 1999 Restoring to cognition the forgotten primacy of action, intention and emotion. In: Rafael Nunez and Waiter J. Freeman (eds.), Reclaiming Cognition - the Primacy of Action, Intention and Emotion, ix-xix. Exeter: Imprint Academic. Fong, Terrence, Illah Nourbakhsh and Kerstin Dautenhahn 2003 A survey of socially interactive robots. Robotics and Autonomous Systems 42: 143-166. Gallagher, Shaun this vol. Phenomenological and experimental contributions to understanding embodied experience. Gallese, Vittorio and Alvin Goldman 1998 Mirror neurons and the simulation theory of mind-reading. Trends in Cognitive Sciences 2 (12): 493-501. Gallese, Vittorio, Christian Keysers and Giacomo Rizzolatti 2004 Mirror neurons and the simulation theory of mind-reading. Trends in Cognitive Sciences 8: 396-403. Gallese, Vittorio, Pier F. Ferrari, Evelyn Kohler and Leonardo Fogassi 2002 The eyes, the hand and the mind: behavioral and neurophysical aspects of social cognition. In: Marc Bekoff, Colin AlIen and Gordon M. Burghardt (eds.), The Cognitive Animal - Empirical and Theoretical Perspectives on Animal Cognition, 451-461. Cambridge, MA: MIT Press. Gardner, Howard 1987 The Mind's New Science. New York: Basic Books. Garcia, J. and R. A. Koelling 1966 Relation to cue to consequence in avoidance learning. Psychonomic Science 4: 123-124. Goldin-Meadow, Susan 2003 Hearing Gesture: How Our Hands Help Us Think. Cambridge, MA: The Belknap Press of Harvard University Press.

158

Jessica Lindblom and Tom Ziemke

Hamad, Stevan 1990 The symbol grounding problem. Physica D 42:335-346. 1989 Minds, Machines and Searle. Journal ofExperimental and Theoretical Artificial Intelligence 1: 5-25. Heidegger, Martin 1962 Being and Time. New York: Harper and Row. Originally published in German in 1927. Hendriks-Jansen, Horst 1996 Catching Ourselves in the Act: Situated Activity, Interactive Emergence, Evolution and Human Thought. Cambridge, MA: MIT Press. Hutchins, Edwin 1995 Cognition in the Wild. Cambridge, MA: MIT Press. Ingold, Tim Evolving skills. In: Hilary Rose and Steven Rose (eds.), Alas, poor 2000 Darwin: Arguments against Evolutionary Psychology, 273-297. New York: Harmony Books. Johnson, Mark and Tim Rohrer this vo!. We are live creatures: Embodiment, American pragmatism and the cognitive organism. Lakoff, George and Mark Johnson 1999 Philosophy in the Flesh: The Embodied Mind and its Challenges to Western Thought. New York: Basic Books. Lindblom, Jessica 2006 Embodied action as a 'helping' hand in social interaction. In: Ron Sun and Noami Miyake (eds.), Proceedings of the 28th Annual Conference of the Cognitive Science Society, 477-482. Mahwah, NJ: Lawrence Erlbaum. Minding the Body: Interacting Socially through Embodied Actions. 2007 Doctoral dissertation, University of Linkoping, University of Skovde, Sweden. Lindblom, Jessica and Tom Ziemke 2003 Social situatedness of natural and artificial intelligence: Vygotsky and beyond. Adaptive Behavior 11 (2): 79-96. 2005 Embodiment-in-motion: Broadening the social mind. In: Brono G. Bara, Lawrence Barsalou and Monica Bucciarelli (eds), Proceedings of the XXVII Annual Conference of the Cognitive Science Society, 1284-1289. Mahwah, NJ: Lawrence Erlbaum. 2006 The social body in motion: Cognitive development in infants and androids. Connection Science 18 (4): 333-346. Lloyd, Dan E. 1989 Simple Minds. Cambridge, MA: MIT Press.

Embodiment and social interaction: A cognitive science perspective

159

Loeb, Jacques 1918 Forced Movements, Tropisms and Animal Conduct. Philadelphia: Lippincott. 1890 Der Heliotropismus der Thiere und seine Ubereinstimmung mit dem Heliotropismus der Pjlanzen. Wiirzburg, Germany: Georg Hertz. Loren, Lewis A. and Eric Dietrich 1997 Merleau-Ponty, embodied cognition and the problem of intentionality. Cybernetics and Systems 28 (5): 345-358. Maturana, Humberto and Francisco J. Varela 1980 Autopoesis and Cognition: The Realization of the Living. Dordrecht, The Netherlands: D. Reidel Publishing. 1987 The Tree of Knowledge - the Biological Roots of Human Understanding. Boston: Shambalaya. McFarland, David J. 1993 Animal Behavior: Psychology, Ethology and Evolution. Harlow: Longman. Merleau-Ponty, Maurice 1962 Phenomenology of Perception. London: Routledge and Kegan Paul. Originally published in French in 1945. 1963 The Structure of Behavior. Boston, MA: Beacon Press. Originally published in French in 1942. 1969 The Visible and the Invisible. Evanston: Northwestern University Press. Originally published in French in 1964. Miller, Patricia H. 1983 Theories of Developmental Psychology. New York: W. H. Freeman and Company. Mingers, John 2001 Embodying information systems: the contribution of phenomenology. Information and Organization 11: 103-128. Minsky, Marvin 1975 A framework for representing knowledge. In: Patrick Winston (ed.), The Psychology ofComputer Vision, 211-277. McGraw Hill. Neisser, Ulric 1976 Cognition and Reality: Principles and Implications of Cognitive Psychology. San Francisco: C. A. Freeman. Neuman, Yair 2001 On Turing's carnal error: some guidelines for a contextual inquiry into the embodied mind. Systems Research and Behavioural Sciences 18: 557-564. Pfeifer, Rolf and Christian Scheier 1999 Understanding Intelligence. Cambridge, MA: MIT Press.

160

Jessica Lindblom and Tom Ziemke

Piaget, Jean The Origin of Intelligence in the Child. New York: Basic Books. 1952 Originally published in French in 1936. Sociological Studies. London: Routledge. 1995 Prem, Erich Epistemic autonomy in models of living systems. In: Phil Husbands 1997 and Inman Harvey (eds.), Fourth European Conference on Artificial Life, 48-73. Cambridge, MA: MIT Press. Priest, Stephen 1998 Merleau-Ponty. London: Routledge. Putnam, Hilary 1975 Philosophy and our mental life. In: Hilary Putnam (ed.), Mind, Language and Rationality, 48-73. Cambridge: Cambridge University Press. Riegler, Alexander 2002 When is a cognitive systems embodied? Cognitive Systems Research 3(3): 339-348. Rizzolatti, Giacomo and Arbib, Michael A. 1998 Language within our grasp. Trends in Neurosciences 21: 188-194. Rizzolatti, Giacomo, Vittorio L. Fadiga, Leonardo Fogassi and Vittorio Gallese 2002 From Mirror Neurons to Imitation: Facts and Speculations. In: Andrew Meltzoff and Wolfgang Prince (eds.), The Imitative Mind: Development, Evolution and Brain Bases, 247-266. Cambridge, MA: Cambridge University Press. Rogoff, Barbara 2003 The Cultural Nature of Human Development. New York: Oxford University Press. Rohrer, Tim this vol. The body in space: Dimensions of embodiment. Romanes, George 1882 Animal Intelligence. London: Kegan Paul. Roth, Wolff-Michael this vol. Communication as situated, embodied practice. Sapir, Edward 1928 The unconscious patterning of behavior in society. In: E. S. Dummer (ed.), The Unconscious, 114-142. New York: Knopf. Searle, John 1980 Minds, brains and programs. Behavioral and Brain Sciences 3: 417457. Semin, Giin R. and Eliot R. Smith 2002 Interfaces of social psychology and situated and embodied cognition. Cognitive Systems Research 3 (3): 385-396.

Embodiment and social interaction: A cognitive science perspective

161

Sharkey, Noel E. and Tom Ziemke 1998 A consideration of the biological and psychological foundations of autonomous robotics. Connection Science 10 (3-4): 361-391. Sheets-Johnstone, Maxine 1999 The Primacy ofMovements. Amsterdam: John Benjamins. Sherrington, Charles S. 1906 The Integrative Action of the Nervous System. New York: C. Scribner's Sons. Sinha, Chris and Kristine Jensen de Lopez 2000 Language, culture and the embodiment of spatial cognition. Cognitive Linguistics 11: 17-41. Sparks, John 1982 The Discovery ofAnimal Behaviour. London: Collins. Svensson, Hemik, Jessica Lindblom and Tom Ziemke this vol. Making sense of embodied cognition: simulation theories of shared neural mechanisms for sensorimotor and cognitive processes. Thompson, Evan and Francisco J. Varela 2001 Radical embodiment: Neural dynamics and consciousness. Trends in Cognitive Sciences 5: 418-425. Tomasello, Michael 1999 The Cultural Origins ofHuman Cognition. Cambridge, MA: Harvard University Press. Trevarthen, Colwyn 1977 Descriptive analysis of infant communicative behaviour. In: H. R. Schaffer (ed.), Studies in Mother-Infant Interaction Proceedings of Loch Lomond Symposium, 227-270. New York: Academic Press. Varela, Charles R. 1992 Harre and Merleau-Ponty: beyond the absent moving body in embodied social theory. Journal for the Theory ofSocial Behaviour 24 (2): 167-185. 1995 Cartesianism revisited: The ghost in the moving machine or the lived body. In: Brenda Famell (ed.), Human Action Sign in Cultural Contexts: the Visible and Invisible in Movement and Dance, 216-293. New Jersey: Scarecrow Press. Varela, Francisco J., Evan Thompson and Eleanor Rosch 1991 The Embodied Mind: Cognitive Science and Human Experience. Cambridge, MA: MIT Press. von Glasersfeld, Emst 1995 Radical Constructivism - a Way ofLearning and Knowing. London: Falmer Press.

162

Jessica Lindblom and Tom Ziemke

von Sachs, Julius 1882 Vorlesungen aber Pflanzenphysiologie. Leipzig, Germany: W. Engelmann. von Uexkiill, Jakob 1928 Theoretische Biologie. Berlin: Springer. 1957 A stroll through the worlds of animal and men a picture book of invisible worlds. In: Claire H. Schiller (ed.), Instinctive BehaviourThe Development of a Modern Concept, 5-80. New York: International Universities Press. Also appeared in 1992 in Semiotica 89(4): 319-391. Originally published in German in 1934. Vygotsky, Lev S. 1978 The Mind in Society: The Development ofHigher Mental Processes. Cambridge, MA: MIT Press. Original work published in Russian in 1934. Wadsworth, Barry J. 1996 Piaget's Theory of Cognitive and Affective Development (5 th ed.). New York: Longman Publishers. Wertsch, James V. (ed.) 1985 Vygotsky and the Social Formation of Mind. Cambridge, MA: Harvard University Press. Wilson, Edward O. 1975 Sociobiology - the New Synthesis. Cambridge, MA: Harvard University Press. Wilson, Margaret 2002 Six views of embodied cognition. Psychonomic Bulletin and Review 9 (4): 625-636. Ziemke, Tom 2001 a Are robots embodied? In: Christian Balkenius et al. (eds), Proceedings of the First International Workshop on Epigenetic Robotics: Modelling Cognitive Development in Robotic Systems, 75-83. Lund, Sweden: Lund University Cognitive Series, vol, 85. 2001 b The construction of 'reality' in the robot. Foundations of Science 6 (1): 163-233. 2002 Introduction to the special issue on situated and embodied cognition. Cognitive Systems Research 3 (3): 271-274. What's that thing called embodiment? In: Richard Alterman and 2003 David Kirsh (eds), Proceedings of the 25 th Annual Meeting of the Cognitive Science Society, 1305-1310. Mahwah, NJ: Lawrence Erlbaum. 2004 Embodied AI as science: Models of embodied cognition, embodied models of cognition, or both? In: Fumiya Iida et al. (eds), Embodied AI, 27-36. Heidelberg: Springer Verlag.

Embodiment and social interaction: A cognitive science perspective

163

Ziemke, Tom and Noel E. Sharkey 2000 A stroll through the worlds of robots and animals: Applying Jakob von Uexkiill's theory of meaning to adaptive robots and artificial life. Semiotica 134 (1-4): 701-746. Ziemke, Tom and Roslyn M. Frank this vol. Introduction: The body eclectic.

Section B

Body and mind

Representing actions and functional properties in conceptual spaces Peter Giirdenfors

Abstract The book Conceptual Spaces (Gardenfors 2000) presents a theory of concepts based on geometrical and topological structures in spaces that are built up from "quality dimensions". Most of the examples in the book deal with perceptual concepts based on dimensions such as colour, size, shape and sound. However, many of our everyday concepts are based on actions and functional properties. For instance most artefacts, such as chairs, clocks and telephones, are categorized on the basis of their functional properties. After giving a general presentation of conceptual spaces, I suggest how the analysis in terms of conceptual spaces can be extended to actions and functional concepts. Firstly, I will argue that "action space" can, in principle, be analysed in the same way as e.g. colour space or shape space. One hypothesis is that our categorization of actions to a large extent depends on our embodied "perception" of forces. In line with this, an action will be described as a spatio-temporal pattern of forces. When it comes to functional properties, the key idea is that the function of an object can be analysed with the aid of the actions it affords. Functional concepts can then be described as convex regions in an appropriate action space. Within Cognitive Semantics, image schemas are mainly based on perceptual and spatial dimensions (e.g. Langacker 1987; Lakoff 1987). Two exceptions are JoOOson's (1987) and Talmy's (1988) work on "force dynamics" that shows the importance of forces, and metaphorical uses of forces, for the semantics of many kinds of linguistic expression. I shall argue that a more developed understanding on "action space" would allow us to extend the semantic analyses pioneered by JoOOson and Talmy. In particular, I shall make a distinction between fIrst-person and third-person perspectives on "forces". My hypothesis is that we start out from an embodied notion of force or "power" that is then extended to forces that are exerted by other individuals and to forces that act on objects outside our control. Keywords: conceptual spaces, force dynamics, image schemas.

168

Peter Giirdenfors

1.

The problem of modelling concepts

A central problem for cognitive science is how representations should be modelled. There are currently two dominating approaches to this problem. The symbolic approach starts from the assumption that cognitive systems can be described as Turing machines. On this view, cognition is seen as essentially being computation involving symbol manipulation (e.g. Fodor 1975; Pylyshyn 1984; Pinker 1997). The second approach is associationism, where associations between different kinds of information elements carry the main burden of representation. Connectionism is a special case of associationism that models associations using artificial neuron networks (e.g. Rumelhart and McClelland 1986; Quinlan 1991). Both the symbolic and the associationist approaches have their advantages and disadvantages. They are often presented as competing paradigms, but since they are used to analyse cognitive problems on different levels of granularity, they should rather be seen as complementary methodologies. However, there are several aspects of concept formation for which neither symbolic representation nor connectionism seem to offer appropriate modelling tools. In this chapter, I will advocate a third way to represent information that is based on using geometrical structures rather than symbols or connections between neurons. Using these structures, similarity relations can be modelled in a way that accords well with human (and animal) judgments. The notion of similarity is crucial for the understanding of many cognitive phenomena. I shall call this way of representing information the conceptual form since I believe that such representations can account for more of the essential aspects of human concept formation than symbolic or connectionist theories. Based on my recent book (Gardenfors 2000), I shall first present a theory of conceptual spaces as a particular framework for representing information on the conceptual level. A conceptual space is built up from geometrical representations based on a number of quality dimensions. Most of the examples I discussed in my book deal with perceptual concepts based on dimensions such as colour, size, shape and sound. However, there is strong evidence that many of our everyday concepts are based on actions and functional properties. For instance, most artefacts, such as chairs, clocks and telephones, are categorized on the basis of their functional properties (Nelson 1986; Mandler 2004). In this chapter, I shall outline how the analysis in terms of conceptual spaces can be extended to functional concepts. Firstly, I will argue that

Representing actions andfunctional properties in conceptual spaces

169

"action space" can, in principle, be analysed in the same way as e.g. colour space or shape space. One hypothesis is that our categorization of actions to a large extent depends on our "perception" of forces. In line with this, an action will be described asa spatio-temporal pattern of forces. I shall also argue that the most cognitively fundamental forces are those that act upon or emanate from one's own body. In this sense my analysis will be based on an embodied perspective. When it comes to functional properties, the key idea is that the function of an object can be analysed with the aid of the actions it affords. Functional concepts can then be described as convex regions in an appropriate action space. I shall outline a research programme indicating that action space should be seen as a special case of a conceptual space.

2.

Quality dimensions

As introductory examples of quality dimensions one can mention temperature, weight, brightness, pitch and the three ordinary spatial dimensions height, width and depth. I have chosen these examples because they are closely connected to what is produced by our sensory receptors (Schiffman 1982). The spatial dimensions of height, width and depth as well as brightness are perceived by the visual sensory system, pitch by the auditory system, temperature by thermal sensors and weight, finally, by the kinaesthetic sensors. However, since there are also quality dimensions that are of an abstract non-sensory character, one aim of this chapter is to argue that force dimensions are important for the analysis of action concepts and functional categories. Quality dimensions correspond to the different ways stimuli are judged to be similar or different. In most cases, judgments of similarity and difference generate an ordering relation of stimuli (Clark 1993: 114). For example, one can judge tones by their pitch that will generate an ordering of the perceptions. The general assumption is that the smaller the distance is between the representations of two objects, the more similar they are. In this way, the similarity of two objects can be defined via the distance between their representing points in the space. The dimensions form the "framework" used to assign properties to objects and to specify relations between them. The coordinates of a point within a conceptual space represent particular instances of each dimension, for example, a particular temperature, a particular weight, etc.

170

Peter Giirdenfors

The notion of a dimension should be understood literally. It is assumed that each of the quality dimensions is endowed with certain geometrical structures (in some cases they are topological or orderings). As a first example, Figure 1 illustrates such a structure, the dimension of "weight" which is one-dimensional with a zero point, and thus isomorphic to the half-line of non-negative numbers. A basic constraint on this dimension, commonly made in science, is that there are no negative weights.

o I Figure 1.

The weight dimension.

A psychologically interesting example of a domain involves colour perception. In brief, our cognitive representation of colour can be described by three dimensions. The first dimension is hue, which is represented by the familiar colour circle going from red via yellow to green and to blue and then back to red again. The topological structure of this dimension is thus different from the quality dimensions representing time or weight which are isomorphic to the real line. The second psychological dimension of colour is saturation, which ranges from grey (zero colour intensity) to increasingly greater intensities. This dimension is isomorphic to an interval of the real line. The third dimension is brightness that varies from white to black and is thus a linear dimension with end points. Together, these three dimensions, one with circular structure and two with linear, constitute the colour domain which is a subspace of our perceptual conceptual space. This domain is often illustrated by the so-called colour spindle (see figure 2). Brightness is shown on the vertical axis. Saturation is represented as the distance from the centre of the spindle. Hue, finally, is represented by the positions along the perimeter of the central circle. The circle at the centre of the spindle is tilted so that the distance between yellow and white is smaller than the distance between blue and white.

Representing actions andfunctional properties in conceptual spaces

171

White

Green

Black Figure 2.

The colour spindle.

A conceptual space can now be defined as a collection of quality dimensions. However, the dimensions of a conceptual space should not be seen as totally independent entities, rather they are correlated in various ways since the properties of the objects modelled in the space co-vary. For example, in the domain of fruits the ripeness and the colour dimensions covary. It is impossible to provide a complete list of the quality dimensions involved in the conceptual spaces of humans. Some of the dimensions seem to be innate and to some extent hardwired in our nervous system, as, for example, colour, pitch, force and probably also ordinary space. Other dimensions are presumably learned. Learning new concepts often involves expanding one's conceptual space with new quality dimensions (Smith 1989). Two-year-olds can represent whole objects, but they cannot reason about the dimensions of the object. Goldstone and Barsalou (1998: 252) note: Evidence suggests that dimensions that are easily separated by adults, such as the brightness and size of a square, are treated as fused together for children [...]. For example, children have difficulty identifying whether two obj ects differ on their brightness or size even though they can easily see that

172

Peter Giirdenfors

they differ in some way. Both differentiation and dimensionalization occur throughout one's lifetime.

Still other dimensions may be culturally dependent. Finally, some quality dimensions are introduced by science. Witness, for example, Newton's distinction between weight and mass, which is of pivotal importance for the development of his celestial mechanics, but which has hardly any correspondence in human perception. To the extent we have mental representations of the masses of objects in distinction to their weights, these are not given by the senses but have to be learned by adopting the conceptual space of Newtonian mechanics in our representations. In order to separate different uses of quality dimensions it is important to introduce a distinction between a psychological and a scientific interpretation. The psychological interpretation concerns the cognitive structures (perceptions, memories, etc) of human beings and other organisms. The scientific interpretation, on the other hand, treats dimensions as a part of a scientific theory. The distinction is relevant when the dimensions are seen as cognitive (psychological) entities, in which case their structure should not be determined by scientific theories which attempt to give a "realistic" description of the world, but by psychophysical measurements that determine how our concepts are represented. The conceptual space of Newtonian particle mechanics is, of course, based on scientific (theoretical) quality dimensions and not on psychological dimensions. The quality dimensions of this theory are ordinary space (3-D Euclidean), time (isomorphic to the real numbers), mass (isomorphic to the non-negative real numbers), and force (3-D Euclidean space). In this theory, an object is thus represented as a point in an 8-dimensional space. Once a particle has been assigned a value for these eight dimensions, it is fully described as far as Newtonian mechanics is concerned. I want to make it clear that the dimensions I consider in my analysis of concepts should be given the psychological interpretation. This applies in particular to the dimension of "force" that will be analysed in the latter sections of this chapter (5-8). A problem for my distinction may be that in Western cultures, the psychological concept of "force" has been tainted by the Newtonian world-view. I will return to this topic in Section 7.

Representing actions andfunctional properties in conceptual spaces

3.

173

Concept formation described with the aid of conceptual spaces

The purpose of this section is to show how conceptual spaces can be used to model concepts. I will focus on concepts that are "natural" in the sense that they can, in principle, be learned without relying on linguistic descriptions and, when described, have simple expressions in most languages. A first rough idea is to describe a natural concept as a region of a conceptual space S, where "region" should be understood as a spatial notion determined by the topology and metric of S. For example, the point in the time dimension representing "now" divides this dimension, and thus the space of vectors, into two regions corresponding to the concepts "past" and "future". But the proposal suffers from a lack of precision as regards the notion of a "region". A more precise and powerful idea is the following criterion where the geometric characteristics of the quality dimensions are utilized to introduce a spatial structure on concepts:

Criterion C: A "natural concept" is a convex region ofa conceptual space A convex region is characterized by the criterion that for very pair of points v1 and v2 in the region all points in between v1 and v2 are also in the region. The motivation for the criterion is that if some objects which are located at v1 and v2 in relation to some quality dimension (or several dimensions) both are examples of the concept C, then any object that is located between v1 and v2 on the quality dimension(s) will also be an example of C. Criterion C presumes that the notion of betweenness is meaningful for the relevant quality dimensions. This is, however, a rather weak assumption which demands very little of the underlying dimensional structure. Most concepts expressed by basic words in natural languages are natural concepts in the sense specified here. For instance, I conjecture that all colour terms in natural languages express natural concepts with respect to the psychological representation of the three colour dimensions. In other words, the conjecture predicts that if some object 01 is described by the colour term C in a given language and another object 02 is also said to have colour C, then any object 03 with a colour that lies between the colour of 01 and that of 02 will also be described by the colour term C. It is wellknown that different languages carve up the colour circle in different ways, but all carvings seem to be done in terms of convex sets. Strong support for this conjecture has been presented by Sivik and Taft (1994). Their study

174

Peter Giirdenfors

can be seen as a follow-up of the investigations of basic color terms by Berlin and Kay (1969) who compared and systematized color terms from a wide variety of languages. Sivik and Taft (1994) focused on Swedish color terms, while Taft and Sivik (1997) compared color terms from Swedish, Polish, Spanish and American English. On the other hand, the reference of an artificial colour term like "grue" (Goodman 1955) will not be a convex region in the ordinary conceptual space and thus it is not a natural concept according to Criterion C. l Another illustration of how the convexity of regions determines concepts and categorizations is the phonetic identification of vowels in various languages. According to phonetic theory, what determines the quality of a vowel are the relations between the basic frequency of the sound and its formants (higher frequencies that are present at the same time). In general, the first two formants F] and F 2 are sufficient to identify a vowel. This means that the coordinates of two-dimensional space spanned by F] and F 2 (in relation to a fixed fundamental frequency F0) can be used as a fairly accurate description of a vowel. Fairbanks and Grubb (1961) investigated how people produce and recognize vowels in "General American" speech. Figure 3 summarizes some of their findings. As can be seen from the diagram, the preferred, identified and selfapproved examples of different vowels form convex sub-regions of the space determined by F] and F2 with the given scales? As in the case of colour terms, different languages carve up the phonetic space in different ways (the number of vowels identified in different languages varies considerably), but I conjecture again that each vowel in a language will correspond to a convex region of the formant space. Criterion C provides an account of concepts that satisfies the desideratum, formulated by Stalnaker (1981: 347), that a concept "[...] must be not just a rule for grouping individuals, but a feature of individuals in virtue of which they may be grouped". However, it should be emphasized that I only view the criterion as a necessary but perhaps not sufficient condition on a natural concept. The criterion delimits the class of concepts that are useful for cognitive purposes, although it may not be sufficiently restrictive. 1. For an extended analysis of this example, see Gardenfors (1990). 2. A selfapproved vowel is one that was produced by the speaker and later approved of as an example of the intended kind. An identified sample of a vowel is one that was correctly identified by 75% of the observers. The preferred samples of a vowel are those which are "the most representative samples from among the most readily identified samples" (Fairbanks and Grubb 1961: 210).

Representing actions and functional properties in conceptual spaces

175

3K

2KI---

2

IK

500 200

Figure 3.

4.

250

500

IK

The vowel space of American English (from Fairbanks and Grubb 1961). The scale of the abscissa and ordinate are the logarithm of the frequencies of F] and F2 (the basic frequency of the vowels was 130 cps).

Relations to prototype theory

Describing concepts as convex regions of conceptual spaces fits very well with the so called prototype theory of categorization developed by Rosch and her collaborators (Rosch 1975, 1978; Mervis and Rosch 1981; see also Lakoff 1987). The main idea of prototype theory is that within a category of objects, like those instantiating a concept, certain members are judged to be more representative of the category than others. For example, robins are judged to be more representative of the category "bird" than are ravens, penguins and emus; and desk chairs are more typical instances of the cate-

176

Peter Giirdenfors

gory "chair" than rocking chairs, deck-chairs, and beanbag chairs. The most representative members of a category are called prototypical members. It is well-known that some concepts, like "red" and "bald" have no sharp boundaries and for these it is perhaps not surprising that one finds prototypical effects. However, these effects have been found for most concepts including those with comparatively clear boundaries like "bird" and "chair". In traditional philosophical analyses of concepts based on truthconditions or functions from possible worlds to extensions (Montague 1974), it is very difficult to explain such prototype effects (see Gardenfors 2000, section 3.3).3 Either an object is a member of the class assigned to a concept (relative to a given possible world) or it is not and all members of the class have equal status as category members. Rosch's research has been aimed at showing asymmetries among category members and asymmetric structures within categories. Since the traditional definition of a concept neither predicts nor explains such asymmetries, something else must be gOIng on. In contrast, if concepts are described as convex regions of a conceptual space, prototype effects are indeed to be expected. In a convex region, one can describe positions as being more or less central. For example, if colour concepts are identified with convex subsets of the colour space, the central points of these regions would be the most prototypical examples of the colour. In a series of experiments, Rosch has been able to demonstrate the psychological reality of such "focal" colours. For another illustration, we can return to the categorization of vowels presented in the previous section. Here the subjects' different kinds of responses show clear prototype effects. For more complex categories like "bird" it is perhaps more difficult to describe the underlying conceptual space. However, if something like Marr and Nishihara's (1978) analysis of shapes is adopted, we can begin to see how such a space would appear. 4 Their scheme for describing biological forms uses hierarchies of cylinder-like modelling primitives. Each cylinder is described by two coordinates (length and width). Cylinders are combined by determining the angle between the dominating cylinder and the 3. Indeed, the approach to semantics in truth-functional semantics is antipsychological in the sense that the goal is to provide an analysis of the meaning of words and sentences that is independent of human cognition. 4. This analysis is expanded in Marr (1982, Ch. 5). A related model, together with some psychological grounding, is presented by Biederman (1987).

Representing actions and functional properties in conceptual spaces

177

added one (two polar coordinates) and the position of the added cylinder in relation to the dominating one (two coordinates). The details of the representation are not important in the present context, but it is worth noting that on each level of the hierarchy an object is described by a comparatively small number of coordinates based on lengths and angles. Hence, the object can be identified as a hierarchically structured vector in a (higher order) conceptual space. Figure 4 provides an illustration of the hierarchical structure of their representations. It should be noticed that this representation of animal concepts is purely shape-based. Animal concepts depend on many other domains, some of which may be of the functional character that will be analysed in Section 6.

_--_.

..------_._-_.__ _

cylioder ....--..-------------_ __-_

.

cr.{1·······_""". '{ i 1\, ~. ·....········1]"J

ill

U biped

limb

bird

~h·········~)G::::.1

~l

human

thick limb

deferred imitation (the imitated action is removed in time) > representative imitation - in which "the interior image precedes the exterior gesture, which is thus a copy of an "internal model" that guarantees the connection between the real, but absent model, and the imitative reproduction of it." (ibid: 279) Two important aspects of Piaget' s account of the rise of representations or "the symbolic function" should be emphasized in the present context. The first is that they arise from an overt, public activity - imitation - which with time becomes internalized. This is reminiscent of Vygotsky's (1978) "law of cultural development" stating that interpersonal forms of higher cognition precede their "intrapersonal" realizations (cf. Lindblom and Ziemke this volume). Second, as pointed out above, this makes possible the acquisition of language, which both consolidates and conventionalizes these representations, leading to a new level of cognitive structure: "Verbal representations constitute, in fact, a new type of representation, the conceptual." (Piaget 1945: 280) In other words these "symbols", i.e. internalized imitations serve as a "missing link" in the acquisition of language. The analogy to the role of bodily mimesis in phylogeny should be now obvious. On this basis, as well as a wealth of empirical data provided by Piaget, but also by many others who have studied infant imitation and ges-

324

Jordan Zlatev

ture since then (Bates et al. 1979; Acredolo and Goodwyn 1994; Zlatev 2002), I have proposed a more fitting term for the structures that Piaget is (rather confusingly for the modem reader) calling "symbols", namely, mimetic schemas (Zlatev 2005). If we refer to the definition of bodily mimesis provided above, we notice that in the case of representative imitation the first three properties: Cross-modality, Volition and Representation are fulfilled. Thus the covert imitation of a child following its "internal model" in executing an action is at least a case of dyadic mimesis. In order to become triadic, in e.g. pantomime ("baby signs") what is necessary is to understand communicative intentions. This can be seen as a wish to induce others to "activate" in consciousness schemas similar to one's own. In other words, while Piaget writes of "symbols" (mimetic schemas) as the "signifier" and the actual model as the "signified", the relation can be reversed, so that a communicative gesture becomes the signifier, while the (shared) mimetic schemas are the "signified" or perhaps in Peircian terms the "interpretant" (cf. Sonesson this volume). Let us now summarize some of the properties of mimetic schemas. Mimetic schemas can be used either dyadically (in thought) or triadically (in communication). Mimetic schemas are experiential: each schema has a different emotional-proprioceptive "feel", or affective tone (Thompson 2001) to it. For example, consider the affective contrast between the mimetic schemas KICK and KISS. Thus, mimetic schemas can be regarded as an (important) aspect of phenomenological embodiment. Mimetic schemas are representational: the "running" of the schema is differentiated from the "model event" which is represented - unlike the most common explication given to "image schemas" (Johnson 1987; Johnson and Rorher this volume; see Hampe 2005). Mimetic schemas are, or at least can be pre-reflectively shared: since my and your mimetic schemas derive from imitating culturally salient actions and objects, as well as each other, both their representational and experiential content can be "shared" - though not in the strong sense of being known to be shared in the manner of (true) symbols or conventions. They could also be called egocentric: "Imitation, with the help of images, provides the essential system of ' signifiers' for the purpose of individual or egocentric representation" (Piaget 1945: 279-280). However, it should be remembered that for Piaget, this formulation does not imply that mimetic schemas are private, but rather the contrary: "on the social plane the child

Embodiment, language, and mimesis

325

is most egocentric at the age in which he imitates most, egocentrism being failure to differentiate between the ego and the group, or confusion of the individual view-point and that ofothers" (ibid: 290, my emphasis). Mimetic schemas can serve as the basis for the acquisition of language in two ways: (a) they constitute the first form of (conscious) internal representation and help lead to the "insight" that others have internal models - a prerequisite for communicative intentions and (b) they constitute prelinguistic concepts, and in this respect correspond to Mandler's (2004) characterization of "image schemas" but not to that of Johnson and Rohrer (this volume; cf. Zlatev 2005). These properties of mimetic schemas, and particularly the last, can allow us to bridge (or at least minimize) the gap between language and embodiment, as discussed in the next section, which also retraces the argument presented in this chapter.

6.

Embodiment regained? Mimetic schemas and language

I started by pointing out three essential properties of (the knowledge of) language: conventionality, representationality and conscious accessibilityand proceeded to see if, and if so how, they can be made compatible with the currently popular conception that the (human) mind is an "embodied mind". In one of the most influential accounts of "embodiment theory", especially within Cognitive Linguistics, that of Lakoff and Johnson (1999), we saw that these three properties were essentially absent. In what followed I subjected this version of "embodiment" to criticism, and in particular its central concept of the Cognitive Unconscious. While this criticism does not automatically generalize to other accounts, it gives us reasons to worry if embodiment and language can be made compatible, not the least because of the lack a coherent concept of representation. The quest for a more adequate notion of embodiment led us to the work of Donald (1991), and the concept of (bodily) mimesis, which was explicated and related to Piaget's developmental theory. In particular, I argued for the need to acknowledge the concept of mimetic schemas, which among other things: are structures of the "lived" (phenomenal, experiential) body, meaning that they are accessible to consciousness;

326

Jordan Zlatev

are representational structures: they are differentiated from what they stand for, and can be enacted overtly (as pantomime and gesture) or covertly (as mental images); can be pre-reflectively shared with others since they (usually) arise from imitation. But notice that these three characteristics of mimetic schemas correspond to - without being identical - to the three properties of language under focus. Thus, the following hypothesis concerning the "embodiment" of language can be formulated: Public linguistic symbols are Hembodied" in the sense that part of their meaning is constituted by underlying mimetic schemas. If this hypothesis holds true, bodily mimesis can serve not only as a "missing link" between sensorimotor and linguistic cognition in evolution as envisioned by Donald (1991) and in ontogenesis as argued by Piaget and in rather different ways proposed by Nelson (1996) and Zlatev (2001, 2002) - but as a conceptual, meta-theoretical link between embodiment and language. Since language is a central aspect of human sociocultural situatedness, mimetic schemas can help integrate the two major factors that define the human mind - embodiment and situatedness - in a coherent framework. What else can we offer in support of this hypothesis? A proper treatment of this question would require a separate chapter, so here I only mention the following considerations, to be explored in more detail in the future (cf. also Zlatev 2005): First, the existence of pre-linguistic but representational mimetic schemas can help solve the puzzle how "socially shared symbolic systems" (Nelson and Shaw 2002) emerge in pre-linguistic children. Since young children lack the meta-linguistic capacity for establishing full-fledged conventions, it is still a mystery how they come from the sensorimotor to the symbolic (i.e. conventional and systematic) level. Mimetic schemas, with their implicit sharing, suggest a way out of this impasse. Second, a particular difficulty in explaining language acquisition is to account for the learning of actions terms ("verbs"). After having traditionally been considered to follow object terms ("nouns") in child language (Macnamara 1982), action words have during the past years been shown to arise simultaneously (Tomasello 1992; Nelson 1996), and if they are prominent in parental speech, even to precede nouns in some cases (Gopnik, Choi and Baumberger 1996). It is obvious how mimetic schemas

Embodiment, language, and mimesis

327

for concrete, imitable actions (e.g. RUN, EAT, SEAT... ) can serve as a basis for the acquisition of the corresponding "verbs". Furthermore, the development of shared representations for objects that can be manipulated such as cups, balls, toys, books, food etc. will be also facilitated, and thus underlie the acquisition of the corresponding "nouns".12 Notice that if mimetic schemas ground the acquisition of the first words in childhood, the prediction is that the child's early vocabulary will consist of terms such as run, sit, eat, cat and toy ... and this is indeed the case (Nelson 1996; Bloom 2000). Third, and conversely to helping explain the ease with which children acquire language, and in line with Donald's (1991) original proposal, mimetic schemas may help explain why language acquisition is so difficult even for "enculturated" apes: evolution has given us an adaptation for triadic mimesis supporting advanced imitation and gesture that is beyond the capacities of our nearest relatives in the animal kingdom. Forth, mimetic schemas as a ground for public symbols can help explain how both "cognitive" (representational) and "affective" (experiential) meaning can be communicated through language, since both aspects can be - to various degrees - shared by communicators, even if the two can be decoupled in abnormal conditions. Fifth, the close connection of linguistic symbols and mimetic schemas is consistent with the accumulating evidence from experimental psychology and neuroscience showing that language use engages motor representations, as well as the corresponding brain regions (Glenberg and Kaschak 2003; Svensson, Lindblom and Ziemke this volume). At the same time, neither this evidence, nor the present proposal implies a stronger form of "language embodiment" in which (practically) all symbolic and inferential processing is carried out by sensorimotor categories and brain regions (Lakoff and Johnson 1999; Johnson and Rohrer this volume). If that were the case it would be very hard to explain the qualitative difference between animal and human cognition, in particular with respect to language skills. To emphasize again, according to the present hypothesis, mimetic schemas ground, but do not constitute linguistic meaning - which as pointed out in

12. In the case of objects there is also another means to achieve shared reference, e.g. joint attention (Tomasello 1999), and this would serve to pick out shared perceptual attributes. But there are problems in explaining how this is done, conceptual (Quine 1960) as well as empirical (Bloom 2000); mimetic schemas for acting on the objects can help pick out the relevant properties.

328

Jordan Zlatev

Section 2 is conventional in the strong sense: not just shared, but mutually known to be shared. Sixth, the hypothesis is consistent with the recent enthusiasm surrounding "mirror neurons", which are assumed to support action recognition and imitation, and their role in the evolution of language (Rizzolati and Arbib 1998; Arbib 2003). Since there appears to be a homology between area F5 of the monkey brain where mirror neurons for grasping were originally discovered and Broca's area, it is reasonable to suppose that a developed mirror neuron system constitutes a (partial) "neural correlate" of the ability to form and entertain mimetic schemas. Seventh, and finally, a long lasting debate in the study of spontaneous co-speech gestures (e.g. McNeil1 1992) is whether they are primarily "communicative" or "cognitive", i.e. whether they are performed for the benefit of the speaker, or for the speaker himself (given that even blind people gesture to each other, as well as more mundanely, people talking on the telephone). Considering gestures to be realizations of mimetic schemas allows them to be both. The work of Kita and Ozyiirek (2003), showing the existence of non-linguistic "spatio-motoric representations" that are to some extent influenced by the language of the speaker, fits in naturally with the present proposal.

7.

Conclusion

In this chapter I have argued for the following set of interrelated theses: Language is fundamentally a socio-cultural phenomenon, based of grammatical and semantic conventions, and therefore it cannot be reduced to individual minds, and even less so to brains. However, apart from conventionality, language also presupposes representationality and conscious accessibility and these imply subjectivity. Qualitative experience is a subjective, "first-person" phenomenon as well as an interpersonal one, involving emotion and affective tone. Thus a truly experiential theory of language needs to account for the ability to communicate through linguistic signs which are shared both representationally and phenomenologically. Theories of embodiment such as that of Lakoff and Johnson (1999) which ignore these characteristics cannot satisfactorily account for language. Since language plays an important role in shaping the human mind, such theories are not capable of accounting for human cognition as well.

Embodiment, language, and mimesis

329

The concepts of bodily mimesis, and its derivative concept: mimetic schemas, can help resolve the contradiction between embodiment and language, and thus assist us in the long-term project of (re)integrating body, language and mind.

Acknowledgments

In writing this chapter, I have benefited from interactions with other members of the project Language, Gesture and Pictures in Semiotic Development at Lund University and the EU-project Stages in the Evolution and Development of Sign Use (SEDSU): Goran Sonesson, Peter Gardenfors, Tomas Persson, Ingar Brinck, and Sara Lenninger. I would also wish to thank Jorg Zinken, Gorel Sandstrom, Roslyn Frank, Alex Kravchenko; Lars-Ake Henningsson and two anonymous reviewers for comments on various earlier drafts. Finally, I wish to dedicate this essay to my friend Esa Itkonen, for his brave fight for the true nature of language against varieties ofbio-physical reductionism over the past 30 years, i.e. half his life.

References Acredolo, Linda and Susan Goodwyn 1994 Sign language among hearing infants: The spontaneous development of symbolic gestures. In: Virginia Volterra and Carol J. Erting (eds.), From Gesture to Language in Hearing and Deaf Children, 68-78. Washington, DC: Gallaudet University Press. Arbib, Michael 2003 The evolving mirror system: A neural basis for language readiness. In: Morten Christiansen and Simon Kirby (eds.) Language Evolution, 182-200. Oxford: Oxford University Press. Armstrong, David M. 1968 A Materialist Theory of the Mind. London: Routledge and Kegan Paul. Badcock, Christopher 2000 Evolutionary Psychology. A Critical Introduction. Cambridge: Polity Press. Baker, Gordon P. and P.M.S Hacker 1984 Language, Sense and Nonsense: A Critical Investigation into Modern Theories ofLanguage. Oxford: Basil Blackwell.

330

Jordan Zlatev

Bates, Elizabeth, Laura Benigni, Inge Bretherton, Luigia Camaioni and Virginia Volterra The Emergence of Symbols. Cognition and Communication in In1979 fancy. New York: Academic Press. Blackmore, Susan 1999 The Meme Machine. Oxford: Oxford University Press. Bloom, Paul 2000 How Children Learn the Meaning of Words. Cambridge, Mass.: MIT Press Brooks, Rodney 1999 Cambrian Intelligence: The Early History ofthe New AI. Cambridge, Mass.: MIT Press. Churchland, Paul M. 1992 A Neurocomputational Perspective: The Nature of Mind and Structure ofScience. Cambridge, Mass.: MIT Press. Clark, Andy 1997 Being There: Putting Brain, Body, and World Together Again. Cambridge, Mass.: MIT Press. Clark, Herbert 1996 Using Language. Cambridge: Cambridge University Press. Clark, Herbert and Catherine R. Marshall 1981 Defmite reference and mutual knowledge. In: Arivind K. Joshi, Bonnie L. Webber and Ivan A. Sag (eds.), Elements ofDiscourse Understanding, 10-63. Cambridge: Cambridge University Press. Corballis, Michael C. 2002 From Hand to Mouth: The Origins of Language. Princeton, NJ.; Princeton University Press. Custance, Deborah M., Andrew Whiten and Kim A. Bard 1995 Can young chimpanzees (Pan troglodytes) imitate arbitrary actions? Hayes and Hayes (1952) revisited. Behavior 132: 839-858. Cuyckens, Hubert, Dominiek Sandra and Sally Rice 1997 Toward and empirical lexical semantics. In: Birgit Smieja and Meike Tasch (eds.), Human Contact Through Language and Linguistics, 35-54. Frankfurt: Peter Lang. Damasio, Antonio 1994 Descartes' Error. Emotion, Reason and the Human Brain. New York: Grosset/Putnam. 2000 The Feeling of What Happens. Body, Emotion and the Making of Consciousness. New York: Harvester. de Lafuente, Victor and Ranulfo Romo 2004 Language abilities of motor cortex. Neuron 41: 178-180.

Embodiment, language, and mimesis

331

Deacon, Terry 1997 The Symbolic Species: The Co-Evolution ofLanguage and the Brain. New York: Norton. Donald, Merlin 1991 Origins of the Modern Mind. Three Stages in the Evolution of Culture and Cognition. Cambridge, Mass.: Harvard University Press. 2001 A Mind So Rare. The Evolution of Human Consciousness. New York: Norton. Dreyfus, Hubert 1972 What Computers (Still) Can't Do. A Critique of Artificial Reason. Third revised edition. Cambridge, Mass.: MIT Press. Reprint 1993. Edelman, Gerald 1992 Bright Air, Brilliant Fire: On the Matter ofthe Mind. London: Basic Books. Elman, Jerry, F. 1990 Finding structure in time. Cognitive Science, 14: 179-211. Evans, Vyv The Structure ofTime. Language, Meaning and Temporal Cognition. 2003 Amsterdam: Benjamins. Fodor, Jerry A. 1981 Representations. Cambridge, Mass.: MIT Press. Frege, Gottlob 1882 [1997] On Sinn and Bedeutung. In: Michael Beaney (ed.), The Frege Reader, 151-171. Oxford: Blackwell. Freud, Sigmund 1949 An Outline ofPsycho-analysis. London: Hogarth. Gallagher, Shaun 1995 Body schema and intentionality. In: Jose Bermudez, Naomi Eilan, and Anthony Marcel (eds.), The Body and the Self, 225-244. Cambridge: MIT/Bradford Press. 2005 How the Body Shapes the Mind. Oxford: Oxford University Press. this vol. Phenomenological and experimental contributions to understanding embodied experience. Gallup, Gordon G. 1982 Self-awareness and the emergence of mind in primates. American Journal ofPrimatology 2: 237-248. Garrett, Brain, J. 1995 Non-reductionism and John Searle's The Rediscovery of Mind. Philosophy and Phenomenological Research 55 (1): 209-215. Giv6n, Tom 2001 Syntax, VoI1-2. Amsterdam: Benjamins.

332

Jordan Zlatev

Glenberg, Arthur M. and Michael P. Kaschak 2003 The body's contribution to language. In: Brian H. Ross (ed.), The Psychology ofLearning and Motivation 43: 93-126. San Diego, CA: Academic Press. Gopnik, Alison, Soonja Choi and Therese Baumberger 1996 Cross-linguistic differences in early semantic and cognitive development. Cognitive Development 11 (2): 197-227. Gurwitsch, Aron 1964 The Field ofConsciousness. Pittsburgh: Duquesne University Press. Hampe, Beate (ed.) 2005 From Perception to Meaning: Image Schemas in Cognitive Linguistics. Berlin: Mouton de Gruyter. Itkonen, Esa Grammatical Theory and Metascience. Amsterdam: Benjamins. 1978 Causality in Linguistic Theory. A Critical Investigation into the 1983 Philosophical and Methodological Foundations of "Nonautonomous" Linguistics. Bloomington: Indiana University Press. 1991 Universal History ofLinguistics. Amsterdam: Benjamins. 2003 What is Language? Turku: University of Turku Press. Jackendoff, Ray 1987 Consciousness and the Computational Mind. Cambridge, MA: MIT Press. Johnson, Mark 1987 The Body in the Mind. Chicago: University of Chicago Press. Johnson, Mark and George Lakoff 2002 Why cognitive linguistics requires embodied realism. Cognitive Linguistics 13 (3): 245-263. Johnson, Mark and Tom Rohrer this vol. We are live creatures: Embodiment, Pragmatism and the cognitive organism. Kita, Sotaro and Asli 6zyiirek 2003 What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48: 16-32. Lakoff, George 1987 Women, Fire and Dangerous Things: What Categories Reveal About the Mind. Chicago: University of Chicago Press. Lakoff, George and Mark Johnson 1980 Metaphors We Live By. Chicago: University of Chicago Press. 1999 Philosophy in the Flesh: The Embodied Mind and its Challenge to Western Thought. New York: Basic Books.

Embodiment, language, and mimesis

333

Langacker, Ronald 1987 Foundations of Cognitive Grammar, Vol 1. Stanford, CA: Stanford University Press. Lewis, David K. 1969 Convention: A Philosophical Study. Cambridge MA: Harvard University Press. Lindblom, Jessica and Tom Ziemke this vol. Embodiment and social interaction: A cognitive science perspective. ~acnamara,John

Names for Things. Cambridge, MA: ~IT Press. Brian 1999 The emergence of language from embodiment. In: Brain ~acWhinney (ed.), The Emergence of Language, 213-256. ~ahwah, NJ: Lawrence Erlbaum. ~andler, Jean 2004 The Foundations of Mind: Origins of Conceptual Thought. Oxford: Oxford University Press. Marcus, Gary F. 2001 The Algebraic Mind: Integrating Connectionism and Cognitive Science. Cambridge, Mass.: MIT Press. ~aslin, Keith 2001 An Introduction to the Philosophy of Mind. MaIden, Mass.: Polity Press. McNeill, David 1992 Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press. ~erleau-Ponty, Maurice 1945 Phenomenology of Perception. London: Routledge and Kegan Paul. Reprint 1962. Nelson, Katherine and Lea Kessler Shaw 2002 Developing a socially shared symbolic system. In: James Byrnes and Eric Amseli (eds.) Language, Literacy and Cognitive Development, 27-57. Mahwah, NJ: Lawrence: Erlbaum. Nelson, Katherine 1996 Language in Cognitive Development. The Emergence of the Mediated Mind. Cambridge: Cambridge University Press. Ogden, C.K. and LA. Richards 1923 The Meaning ofMeaning. London: Routledge and Kegan Paul. Palmer, Gary 1996 Toward a Theory of Cultural Linguistics. Austin: The University of Texas Press. 1982

~acWhinney,

334

Jordan Zlatev

Patterson, Francis 1980 Innovative use of language in a gorilla: A case study. In Katherine Nelson (ed.) Children's Language, Vol 2,497-561. New York: Garden Press. Piaget, Jean 1945 La formation du symbole chez l'enfant, Neuchatel-Paris: Delachaux et Niestle; English translation: G. Gattegno and F. M. Hodgson. Play, Dreams, and Imitation in Childhood. New York: Norton, 1962. 1953 The Origin of Intelligence in the Child. London: Routledge and KeganPaul. 1954 The Construction ofReality in the Child. New York: Basic Books. Pinker, Steven 1994 The Language Instinct. New York: William Morrow. Quine, Willard V. O. 1960 Word and Object. Cambridge, Mass.: MIT Press. Regier, Terry 1996 The Human Semantic Potential: Spatial Language and Constrained Connectionism. Cambridge, Mass.: MIT Press. Rizzolatti, Giacomo and Michael Arbib 1998 Language within our grasp. Trends in Neurosciences 21: 188-194. Rohrer, Tim 2001 Pragmatism, Ideology and Embodiment: William James and the philosophical foundations of cognitive linguistics. In Rene Dirven, Bruce Hawkins and Esra Sandikcioglu (eds.), Language and Ideology: Cognitive Theoretic Approaches: Volume 1, 49-81. Amsterdam: Benjamins. this vol. The body in space: Dimensions of embodiment. Russell, Bemard 1905 On denoting. Mind 14: 479-93. Saussure, Ferdinand de 1916 Cours de Linguistique Generale [Course in General Linguistics]. Paris: Payot. Schmidt,Richard 1990 The role of consciousness in second language learning. Applied Linguistics 11: 17-46. Searle, John 1969 Speech Acts. Cambridge: Cambridge University Press. 1992 The Rediscovery ofthe Mind. Cambridge, Mass.: MIT Press. 1999 Mind, Language and Society. Philosophy in the Real World. London: Weidenfeld and Nicolson. 2002 Consciousness and Language. Cambridge: Cambridge University Press.

Embodiment, language, and mimesis

335

Senghas, Ann, Sotaro Kita and Asli OZyiirek 2004 Children creating core properties of language: Evidence from an emerging sign language in Nicaragua. Science 305: 1779-1782. Sinha, Chris Language and Representation. A Socio-naturalistic Approach to 1988 Human Development. New York: Harverster Press. Grounding, mapping and acts of meaning. In: Theo Janssen and 1999 Gisela Redeker (eds.), Cognitive Linguistics: Foundations, Scope and Methodology, 223-255. Berlin: Mouton de Gruyter. Blending out of the Background: Play, props and staging in the mate2005 rial world. Journal of Pragmatics (Special issue on Conceptual Blending Theory, guest eds. Seana Coulson and Todd Oakley), 37: 1537-1554. Sonesson, Goran 1989 Pictorial Concepts. Lund: Lund University Press. this vol. From the meaning of embodiment to the embodiment of meaning: A study in phenomenological semiotics. Stokoe, William C. 2001 Language in Hand. Why Sign Came before Speech. Washington D.C.: Gallaudet University Press. Strawson, Peter F. 1950 On referring, Mind 59: 320-344. Svensson, Patrik 1999 Number and Countability in English Nouns: An Embodied Model. Uppsala: Swedish Science Press. Svensson, Henrik, Jessica Lindblom and Tom Ziemke this vol. Making sense of embodied cognition: Simulation theories of shared neural mechanisms for sensorimotor and cognitive processes. TalmY,Len 2000 Toward a Cognitive Semantics, Vol I and Vol If. Cambridge, Mass.: MIT Press. Tanner, Joanne E. and Richard W. Byrne 1999 The development of spontaneous gestural communication in a group of zoo-living lowland gorillas. In: Sue T. Parker, Robert W. Mitchell and H. Lyn Miles (eds.), The Mentalities of Gorillas and Orangutans - Comparative Perspectives, 211-239. Cambridge: Cambridge University Press. Thompson, Evan 2001 Empathy and consciousness, Journal of Consciousness Studies 8 (5/7): 1-32.

336

Jordan Zlatev

Thompson, Evan and Francisco Varela 2001 Radical embodiment: Neural dynamics and consciousness. Trends in Cognitive Sciences 5 (10): 418-425. Tomasello, Michael 1992 First Verbs: A Case Study of Early Grammatical Development. Cambridge: Cambridge University Press. 1999 The Cultural Origins of Human Cognition. Cambridge, Mass.: Haryard University Press. Tomasello, Michael, Joseph Call and Brian Hare 2003 Chimpanzees understand psychological states - the question is which ones and what extent. Trends in Cognitive Sciences 7 (4): 153-156. Trubetzkoy, Nikolay S. 1939 Grudzuge der Phonologie. Gottingen: Vandenhoeck and Ruprecht. Reprint 1958. Tufvesson, Sylvia, Jordan Zlatev and Joost van de Weijer 2004 Idiomatic entrenchment and semantic priming, In: Augusto Soares da Silva, Amadeu Torres, Miguel Gonyalves (eds.), Linguagem, Cultura e Cogni9iio: Estudos de Linguistica Cognitiva, Vol 1: 309-334. Coimbra: Almedina. Varela, Francisco 1996 Neurophenomenology: A methodological remedy for the hard problem. Journal ofConsciousness Studies 3 (4): 330-350. Varela, Francisco, Evan Thompson and Eleonor Rosch 1991 The Embodied Mind. Cognitive Science and Human Experience. Cambridge, Mass.: MIT Press. Violi, Patrizia 2001 Meaning and Experience. Translated by Jeremy Carden, Bloomington: Indiana University Press. 2003 Embodiment at the crossroads between cognition and semiosis. Reserches en communication 19: 199-217. Vygotsky, Lev S. 1934 Thought and Language. Cambridge, Mass.: MIT Press. Reprint 1962. 1978 Mind in Society. The Development of Higher Psychological Processes, Cambridge, Mass.: Harvard University Press. Wittgenstein, Ludwig 1923 The Tracatus Logico Philosophicus. London: Routledge. Reprint 1961. 1953 Philosophical Investigations. Oxford: Basil Blackwell.

Embodiment, language, and mimesis

337

Ziemke, Tom 2003 What's that thing called embodiment? In: Richard Alterman and David Kirsh (eds), Proceedings of the 25 th Annual Meeting of the Cognitive Science Society, 1305-1310. Mahwah, NJ: Lawrence Erlbaum. Zlatev, Jordan 1997 Situated Embodiment. Studies in the Emergence ofSpatial Meaning. Stockholm: Gotab Press. 2001 The epigenesis of meaning in human beings, and possibly in robots. Minds and Machines 11 (2): 155-195. 2002 Mimesis: The "missing link" between signals and symbols in phylogeny and ontogeny. In: Anneli Pajunen (ed.), Mimesis, Sign and the Evolution of Language, 93-122. Publications in General Linguistics 3. Turku: University ofTurku Press. 2003 Meaning = Life (+ Culture). An outline of a unified biocultural theory of meaning. Evolution ofCommunication 4 (2): 253-296. 2005 What's in a schema? Bodily mimesis and the grounding of language. In: Beate Hampe (ed.), From Perception to Meaning: Image Schemas in Cognitive Linguistics, 313-342. Berlin: Mouton de Gruyter. Zlatev, Jordan, Tomas Persson and Peter Gardenfors 2005 Bodily mimesis as "the missing link" in human cognitive evolution. LUCS 121. Lund: Lund University Cognitive Studies.

The body in space: Dimensions of embodiment

Tim Rohrer

Abstract Recent research from a large number of fields has recently come together under the rubric of embodied cognitive science. Embodied cognitive science attempts to show specific ways in which the body shapes and constrains thought. I enumerate the standard variety of usages that the term "embodiment" currently receives in cognitive science and contrast notions of embodiment and experientialism at a variety of levels of investigation. The purpose is to develop a broad-based theoretic framework for embodiment which can serve as a bridge between different fields. I introduce the theoretic framework using examples that trace related research issues such as mental imagery, mental rotation, spatial language and conceptual metaphor across several levels of investigation. As a survey piece, this chapter covers numerous different conceptualizations of the body ranging from the physiological and developmental to the mental and philosophical; theoretically, it focuses on questions of whether and how all these different conceptualizations can form a cohesive research program. Keywords: cognitive neuroscience, Cognitive Linguistics, Embodiment, frames of

reference, mental rotation, space.

1.

Introduction: Embodiment and experientialism

1.1.

Embodiment: The return of the absent body to cognitive science

HUMAN BEINGS HAVE BODIES. Academics of every variety, so often caught up in the life of the mind, find that simple truth altogether too easy to forget. Imagine working late into the night, hotly pursuing another bit of perfect prose. But now let there be a power outage and, in the absence of electric light or the pale glow of the computer screen, imagine how we grope and fumble to find our briefcase, locate the door, and exit the building. In such circumstances, the body returns. Whenever we are unexpect-

340

Tim Rohrer

edly forced to move about in the dark, we are forcibly reacquainted with our bodily sense of space. Problems ordinarily solved beneath the level of our conscious awareness become dominant in our cognition; we find ourselves noticing subtle changes in the floor texture underfoot, carefully reaching out for the next step in the stairwell. It is a most peculiar experience, one that may well remind us of being young and just learning to walk down stairs. Unfortunately for cognitive science, many academics of that particular variety haven't simply forgotten that human beings have bodies cognitive scientists have deliberately theorized the body away. For most of its first fifty years, cognitive science was in the throes of a peculiarly devilish axis between information theory in computer science and functionalism in the philosophy of mind and psychology. Within computer science and information theory, the problem of building a thinking machine was identified with just one narrowly specified field of human cognition - computing mathematical functions (Turing 1950; Hodges 1983). Under the functionalist paradigm, the mind was treated as if it were a series of modular computer programs - or "black boxes" - whose inputs and outputs could be specified in symbolic terms. While no one would have argued that the physical architecture of vacuum tubes and transistors making up the early computers were identical to the physical architecture of the neural systems making up the brain, the functionalists did argue that the specific physical details of how such thinking systems computed were irrelevant to what they computed. In fact, they argued that as computation could take place not only in electrical and neural systems but also in mechanical systems such as a loom or Babbage's steam-powered analytical engine, cognition was independent of its physical medium. From this perspective, the only thing that mattered to simulating cognition was getting the inputs of these black boxes to compute the correct outputs (Cummins 1977). The physical body - whose architecture was seen as largely irrelevant to cognition - was redefined as a series of black boxes that computed mathematical functions. The disembodied computer was the analogical origin of the disembodied mind. In recent years however, another strain of cognitive scientists have begun to take their inspiration from the contrarian vision - embodied cognitive science. Unlike the computationalist-functionalist hypothesis, embodiment theorists working in various disciplines argue that the specific details of how the brain and body embody the mind do matter to cognition. This broad theoretical approach has been the result of many parallel devel-

The body in space: Dimensions ofembodiment

341

opments in diverse fields ranging from neurobiology and linguistics to robotics and philosophy. While there are undoubtedly many touchstones and origins of this approach, Johnson and I (Johnson and Rohrer this volume) have given a detailed account of some of the neurobiological and philosophical roots of the embodiment hypothesis in cognitive science with a particular emphasis on how the contributions of American Pragmatism anticipated modem cognitive neuroscience. By contrast, in this paper I intend only to survey the wide variety of manners in which embodied cognitive science is done, including these among others, in order to develop a general theoretic framework as a backdrop against which these research projects can be situated. One of the most central examples of how embodied cognitive science has revolutionized the field lies in the details of how the mind, brain and body interact to construct our experience of space. Tracing this example across the different disciplines of cognitive science will require the whole of this article, but as a beginning recall the basic finding of the work on mental rotation (Shepard and Metzler 1971). In their renowned experiment wherein participants were asked to determine whether one two-dimensional drawing of a three-dimensional object was identical to or a mirror image of another, they found that subjects mentally rotated the object at a linear rate - about 60 degrees per second. In other words, participants were manipulating such images as wholes, preserving their topologies while rotating them through a series of intermediate depictions. At the time of its publication, their finding was surprising because the then prevailing computationalist and functionalist view held that the mind operated in a symbolic rather than depictive fashion, and therefore argued that any such mental imagery would be merely epiphenomenal (Pylyshyn 1973). Over the ensuing thirty years, a variety of convergent evidence has established not only the fact that mental images are rotated in the brain as perceptual wholes (Kosslyn et al. 1995), but have also specified how that fact impacts our understanding of exactly what our minds are "computing" (reviewed in Kosslyn 1994; Kosslyn, Ganis and Thompson 2002). Consider, for example, how the body - and not just the brain - plays a role in modifying the rate at which mental rotations take place. Extend your left arm in front of you and hold your left hand straight out, palm upwards. Now try to rotate your hand 180 degrees to the left and then to the right. Notice that the rightward (inward) rotation is relatively easy, while the leftward is quite difficult, requiring additional shoulder and arm joint movements. A series of experiments by Parsons (1987ab, 1994; Par-

342

Tim Rohrer

sons et al. 1995) showed that when subjects were asked to perform mental rotations of images which consisted of line drawings of human hands instead of Shepard-Metzler 2D/3D block diagrams, subjects were quicker and better at identifying those rotations of the hand that were easier to perform, given the kinds of bodily constraints on joint movements we have as humans. Furthermore, Parsons found that subjects were quicker and better at judging which hand - left or right - was pictured when imagining rotating the hand that did not require difficult bodily movements. Given the details of the way the body works, the motor imagery system actively constrains how fast mental imagery is performed. 1 Even more dramatically, consider how patients with chronic arm pain in one limb perform similar mental hand rotation tasks. For their affected arm as compared to their uninjured arm patients are much slower to perform the necessary mental rotations in those conditions where the bodily movements that would be required for the actual hand rotation involve large arm movements (Schwoebel et al. 2001). A group of non-patient controls also showed no such differences between their left and right arms. Not only does the body affect how our mind works, but the body in pain affects how the mind works. Of course, this last insight should come as no surprise to anyone except those cognitive scientists who believe that our minds work just like disembodied computers. As a fallback position, a committed computationalist could simply jettison the functionalist claim that our cognition is independent of our neurophysiological architecture. One could argue that embodiment means only that "computations" of a particular kind - analog and iconic, not symbolic; physiologically embodied and perspectivally situated, not abstractly universal - are being performed as the body and brain pass topologypreserving structures forward and backward between the visual and motor systems. Admittedly, these topologies are not the "outputs" of the computations of a rigidly modular functionalist architecture, but rather dynamic activation patterns which imagistically map the perceptual contours of experience, rippling back and forth through multiple reentrant neuroanatomical connections within a web of functionally interrelated neural regions. These embodied neural "computations" compete to become the

1. Interestingly, if one adds a cylindrical "head" to Shepard-Metzler cube stimuli, one can produce similar embodied facilitation and inhibition effects to those produced by more naturalistically "embodied" stimuli such as Parsons' line drawings of hands (see Amorim, Isableu and Jarraya 2006).

The body in space: Dimensions ofembodiment

343

most salient and pragmatically useful mental constructs to address the current problem for the organism, whatever that is. While such revisions to our conception of what counts as cognitive "computations" are certainly warranted by the evidence and are important steps toward an embodied cognitive science, we might also inquire whether the focus on embodiment leads to additional constraints that are not purely physiological. Suppose that the current problem for the organism is once again the mental rotation of images. Given the results concerning how the rate of the mental rotation varies when the stimulus is a hand, are there multiple strategies for rotating mental objects that could compete to solve such problems? As one obvious difference between the Parsons stimuli and the Shepard and Metzler stimuli is that the former are line drawings of body parts while the latter are line drawings of 3D blocks, it might be possible that the motor imagery effects Parsons observed are limited to body-part images. While Kosslyn et al. (1998) had initially argued that there was this sort of stimulus-determined choice between two separate neural systems that could perform mental rotations, namely the motor imagery (hand stimulus) and visual imagery (object stimulus) systems, Kosslyn et al. (2001) now argue that there are two possible perspectives - or spatial frames of reference - that influence which strategy for mental rotation is chosen. In one such frame of reference - a viewer-centered perspective - it is possible to imagine oneself physically grasping and rotating a 3D object; while in the other frame of reference - an object-centered frame - it is possible to imagine viewing something else rotating the 3D object. Kosslyn and colleagues built wooden 3D constructs of a Shepard-Metzler block figures, and just prior to the neuroimaging had the subjects either turn by hand the wooden blocks or observe the blocks rotating on a motor-driven spindle. Participants were then instructed to imagine rotating the visual stimuli presented during the neuroimaging task in precisely the same manner. The neuroimaging results showed that the differences in the strength of activation in the motor imagery (or visual imagery) brain regions correlated with which perspective was obtained on the model via the participant's socially instructed interaction with it. Their results show that participants could voluntarily choose to adopt a particular strategy based on the frame of reference in which they were told to interact with the object and not based solely on the type of stimulus image, i.e., body parts or blocks. In other words, it is not the case that only the details of our physiology matter, such as the constraints of our joints as we imagine rotating

344

Tim Rohrer

our hands. Instead, the socially instructed choice of perspective also matters to how the embodied mind works. The Kosslyn group's experiments demonstrate why embodiment in cognitive science should never be construed as an exclusively physiological phenomenon. Even when researchers are measuring physiological changes such as changes in the blood flow or glucose uptake within the brain, both socially and environmentally induced factors can play a theoretically significant role as to what brain activity is being measured. Instructing a participant in a neuroimaging experiment to imagine using one spatial frame of reference or the other - that is, to imagine manipulating the blocks themselves as opposed to imagining the blocks spinning on their own - demonstrates how the social context influences the physiological response. Similarly, constructing a 3D physical version of a heretofore visually presented 2D stimulus predisposes the participant to interact with the stimulus using a slightly different mix of sensory modalities - resulting in a different physiological response. Note that it is not the case that the "body" enters into the measured response only in the condition where the participant physically manipulates the 3D object. In each case the body interacts with the stimuli in different ways (visually or motorically), and the resulting environmental predispositions to imagine using either the visual or the motor system are carried into the PET scanner. Unfortunately, the Kosslyn group has not yet investigated whether the social and environmental influences are separable, but it is reasonable to predict that they are. One could test this hypothesis using an experimental variation derived from semantic priming; if some participants were instructed to imagine operating in the opposite frame of reference during the scan than the one induced by their pre-scan bodily interaction with the stimulus, one would expect that their responses would be weaker in activation (inhibited) when compared to those participants for whom the social instructions coincided with the embodied environmental interaction.

1.2.

Experientialism: "The body" of cognitive science expands

The Kosslyn et al. (2001) experiment is particularly revealing because it shows that even for those of us who use methodologies dedicated to measuring the body, embodiment means not just the physiological body - or worse yet, just the physiological brain - but the body-in-space, the body as it interacts with the physical and social environment. Many of the objects

The body in space: Dimensions ofembodiment

345

we interact with every day are in fact cognitive artifacts we have designed with our bodies in mind. Consider one last set of experiments on mental rotation, one which compares the mental rotation of hands with the mental rotation of tools. Vingerhoets et al. (2002) compared the fMRI activation patterns of right-handed male subjects who were asked to decide whether a pair of pictures were different or identical (except for being rotated) when presented with either two pictures of hands (either right or left) or two pictures of hand tools (a monkey wrench, a pencil sharpener, a can opener and a soup ladle). While they found pre-motor and motor activation in both experimental conditions, their key finding was that tools, unlike hands, activated only the left hemisphere premotor and motor hand cortices contralateral to the subject's dominant hand. In other words, when we think about rotating tools - as opposed to Kosslyn's abstract shapes or Parsons' pictures of hands - we are mentally "grasping" those tools and rotating them with the same hand that we would ordinarily use to rotate them in the physical world. Thus, the body-in-the-brain is not just shaped by the body, but by the habitual interactions of the body with the environment. The point is not just that the body shapes the embodied mind, but that the experiences of the body-in-the-world also shape the embodied mind. But the experiential worlds with which we interact are more than simply physical; we are born into social and cultural milieus which transcend our individual bodies in time. Tools are an excellent example of the elements of our physical world that come to us already shaped by socio-cultural forces which predate each individual's body, if not the human body in general - for there has certainly been a long process of cultural refinement in the design of hand tools. Like tools, language is another part of the sociocultural milieu within which we exist. Can we investigate how sociocultural factors (such as the language into which we are born) shape our cognition? Let us begin by considering matters of prepositional structure, perspective and frames of reference in a linguistic context. In English we can speak metaphorically about features of the landscape in terms of the body, such as the face of a mountain, the mouth of a river, the foothills, and on. Peninsulas can be construed as fingers of land, or as heads (as in Hecata Head). In other words we understand features of the landscape metaphorically, using our bodies as the grounding frame of reference. Lakoff and Johnson (1999, 1980) have called such systematic patterns of metaphoric projection "conceptual metaphors", and have argued that they exhibit a general tendency to conceptualize more abstract entities in terms of the

346

Tim Rohrer

more bodily ones. We now know that both literal and metaphorical uses of body-part terms exhibit mental imagery effects similar to those described in the experimental lines already discussed. In an fMRI study which included instances of the LANDSCAPE IS A BODY metaphors, participants' primary and secondary hand sensorimotor cortices were active during the comprehension of both literal and metaphoric hand sentences (Rohrer 2001b, 2005). However, languages vary in how they construct space; is it possible that what is a metaphoric usage in English is the basic frame of reference habitually used by members of another culture? Linguists have documented a number of Mayan languages such as Mixtec, Tzeltal and Zapotec whose prepositional structure is entirely composed of body-part morphemes. For example, saying the stone is under the table requires saying the stone is proximal to the table's belly (yuu wa hiyaa cii-mesa / stone the be-located table-belly) (Lakoff 1987: 313). Within Cognitive Linguistics Brugman (1985) and Lakoff (1987; see also related work in MacLaury 1989) have claimed that such languages require projecting the names for body parts onto objects in the world. They argue that Mixtec speakers start off with a viewer-centered frame of reference2 and then take up the perspective of the object. This change in perspective yields an object-centered frame of reference for Mixtec spatial relation terminology, where tables metaphorically acquire bellies located where a human belly would be. From a purely neurophysiological conception of the body and one strongly influenced by an overly narrow conceptualization of the brain in terms ofjust the visual system (and not the sensorimotor system)one could conclude that this order of events was inevitable, given that in the visual system we first construe the world in our visual system in viewer-centered neural maps, and only later in object-centered maps.3 Given the cognitive neuroscience available at that time, Lakoff (1987) plausibly argued that speakers of such languages were metaphorically projecting the viewer-centered frame of reference to form another, objectcentered frame of reference.

2. Even though the topic has somewhat shifted to language, I am still using the term "frame of reference" primarily in its spatial sense, as would be found in cognitive psychology and cognitive neuroscience. I discuss the complex relation between linguistic and non-linguistic frames of reference in Section 4. 3. Current studies of the sensorimotor system reveal that there are separable frame of reference maps for body-centered (i.e., viewer-centered) mental rotation and object-centered mental rotation (see review in Parsons 2003).

The body in space: Dimensions ofembodiment

347

However, related evidence gathered in cross-cultural language acquisition studies reveals that the embodied mind is being shaped here not simply by the neurophysiology but by the particular socio-cultural practices that accompany language acquisition. The metaphoric projection hypothesis predicts that such terms would be learned first as names for the body parts, and only later extended to spatial relations terms. In a cross-cultural study of Danish- and English-speaking children on one hand and Zapotecspeaking children on the other, Jensen de Lopez and Sinha (1998; Sinha and Jensen de Lopez 2000; Jensen de Lopez 2002) investigated whether each culture's children acquire body-part morphemes first as body-part terms and then only later metaphorically project them as spatial relations terms. Their results show that Zapotec-speaking children acquire the bodypart morphemes first as spatial relations terms and only later - and seemingly independently - as names for the body parts, while Danish and English children acquire them first as body-part names and only later use them to indicate spatial relations. Furthermore, Jensen de Lopez and Sinha hypothesize that the difference derives from differing cultural practices of child-rearing. They note that Zapotec infants spend most of their first two years in a sling on the mother's back, sharing her spatial perspective, while Danish and English infants are placed in cribs and encouraged more to move about on their own. Consequently, joint attentional episodes during which the child's body parts are named may be less frequent in Zapotec child-rearing practices than in Danish or English child-rearing practices. In short, Jensen de Lopez and Sinha suggest that what might have looked like a projection of viewer-centered body-part terms in order to form an objectcentered frame of reference is instead simply the acquisition of an objectcentered frame of reference through joint attentional episodes focused on the spatial characteristics of such objects. The work of Jensen de Lopez and Sinha, along with other cross-cultural language acquisition work (Bowerman and Choi 2003), is an example of why embodied cognitive science must include the socio-cultural milieu as one dimension of variability. Note how similar the findings of Jensen de Lopez and Sinha are to the Kosslyn et al. (2001) finding concerning how the actions that directly precede the mental rotation experiment influence which neural system is chosen to perform the task. In both cases, the social context ofjoint attentional episodes, whether between caregiver and child or experimenter and participant, influences what frame of reference is chosen. To be embodied as a human being means in part that we are born into a socio-cultural milieu

348

Tim Rohrer

within which we have particular problem-solving strategies reinforced through experience. Deeply habitual experiences or an immediately prior atlentional experience can alter which strategy we choose to employ. The body of embodied cognitive science is not limited to physiological and neurophysiological influences on mind, nor to that plus the physical body's interactions with the physical world, but also incorporates the experiences of the social and cultural body as well. In other words, it has to take account of the socio-cultural context within which a particular body is situated. At this point it is clear that there are a number of different, if interrelated, senses of the term "embodiment" at play in the literature. I began this chapter with the image of how the phenomenological body intrudes upon the mind when the lights unexpectedly go out and one must fumble to find the way out of a building, and then traced some examples of how such experiential considerations motivate experiments that investigate the physiological and neurophysiological responses to the experienced body (as in imagining rotating the hand for both normal participants and participants with chronic arm pain). Continuing to examine the literature on imagination and mental rotation, I showed how even a focus on measuring the neurophysiology leads to the realization that both the physical environment and the socio-cultural context are factors which impact the embodied mind. Now it is time to begin addressing the meta-theoretic picture explicitly. How can all these senses of the term hang together as a framework for research on embodied cognition? What are the dimensions of embodiment that different theorists think it is important to measure and address? How many dimensions are there, and how do they interact to form different research clusters?

2.

Surveying the dimensions of embodiment

Like most scientists, linguists usually acknowledge that it is a difficult but admirable goal to begin as descriptively as possible before proceeding prescriptively. By my latest count the term "embodiment" can be used in at least twelve different important senses with respect to our cognition. Because theorists often (and sometimes appropriately, given their specific purposes) conflate two or more of these dimensions, it is important to get a clear picture of as many of the different dimensions of variability as possible. This list is not intended to be entirely exhaustive of the term's current

The body in space: Dimensions ofembodiment

349

usage, nor are the dimensions necessarily entirely independent of each other nor even entirely distinct from one another. Thus it is important to note that, and unlike the argumentative analyses given in Anderson (2003) or Wilson (2002), this initial survey is not intended to be a prescriptive definition ofthe term, but instead is intended only to catalogue some of the contemporary usage of the term in a way that reveals the most relevant dimensions to which one must be responsive in order to develop a general theoretic framework for embodiment theory in cognitive science. However, I do note where some theorists have used the term in several of these senses simultaneously, and in several cases I argue for making finer distinctions and recognizing more dimensions than do the original theorists themselves.

2.1.

Dimension 1: Philosophy

First - and perhaps most broadly - "embodiment" is used as a shorthand term for a counter-Cartesian philosophical account of mind, cognition and language. Descartes took problems within geometric and mathematical reasoning (such as the meaning of the term "triangle") as model problems for study, and concluded that knowledge par excellance is "disembodied" - that is, fundamentally independent of any particular bodily sensation, experience, or perspective - all of which are roots of uncertainty. In arguing that the meaning of the term "triangle" consists in the reference relationship between the word and an object that exists not in the physical, embodied world but in thought alone, Descartes' thought experiments set the stage for many thinkers within analytic philosophy, formal semantics, and early cognitive science. Broadly speaking, such philosophers of language typically construe the two central problems of meaning to be (i) mapping the reference relations between idealised objects of knowledge, their counterpart symbolic expressions in language and the objects or "states of affairs" in the real world (as in Fregean semantics), and (ii) explaining the internal logical structure of the relations which hold between these idealised objects or their corresponding linguistic symbols (as in theories of "autonomous syntax"). While Descartes was by no means unique nor alone within Western philosophy in claiming this position, his extraordinary clarity has garnered him the laurel of becoming metonymic for this package of philosophical assumptions (Lakoff and Johnson 1980, 1999; Geeraerts 1985; Johnson 1987; Rohrer 1998). Most such embodi-

350

Tim Rohrer

ment theorists, while perhaps somewhat favouring the empiricist side of the rationalist-empiricist split, generally try to dissolve such philosophical problems as hangovers from a bad metaphysics by recasting the problems in a different metaphysical basis. Many, although not all, of the theorists objecting to such Cartesian treatments of language, meaning and representation use "embodiment" in this broadly philosophical sense even as they work explicitly in one or more of the somewhat narrower dimensions of the term that follow.

2.2.

Dimension 2: The socio-cultural situation

"Embodiment" is also used to refer to the social and cultural practices within which the body, cognition and language are perpetually situated. In this sense, "embodiment" is often used to emphasize the particularistic, rather than the universalistic, tendencies of human cognition; e.g., how a particular mind in a particular body is shaped by the particular culture within which it is embedded. One example of a cognitive cross-cultural language acquisition study would be the previously discussed research by Sinha and Jensen de Lopez (2000). The cultural variation in child-rearing practices might well account for differing acquisition sequences of spatial language terms in English-, Danish-, and Zapotec-speaking children. Such socio-cultural practices can be given material form in the material artifacts that aid and manifest cognition - many of which are extensions of the body (Hutchins 1995, 1999; Fauconnier and Turner 2002). In assessing the differences between Micronesian and Western traditions of navigation, Hutchins observes that the [...] physical artifacts became repositories of knowledge, and they were constructed in durable media so that a single artifact might come to represent more than any individual could know. Furthermore, through the combination and superimposition of task-relevant structure, artifacts came to embody kinds of knowledge that would be extremely difficult to represent mentally. (1995: 96)

Hutchins cites the example of the medieval astrolabe, a set of rotating disks that embody the spatial relationships of the celestial bodies at different latitudes with much greater precision than would be possible from only the individual navigator's memory. These are set into a frame that represents the horizon, which is itself inset with a scale marking out the 24-hour day and/or the 360 degrees of the compass. He notes that the astrolabe embod-

The body in space: Dimensions ofembodiment

351

ies socio-cultural practices in two important ways. First, the astrolabe is an extension of the body in that a skilled navigator physically manipulates it by rotating its disks in order to predict celestial movements. Second, in its design the astrolabe is "a physical residuum of generations ofastronomical practice" (Hutchins 1995: 96-97). Any particular navigator using the astrolabe is the intellectual heir of a wide set of social practices which have been designed into the instrument. In contrast, a Micronesian navigator eschews such material artifacts, relying successfully instead on cultural artifacts such as chants that encode the relevant celestial relations for voyages between particular islands (Hutchins 1995: 65-92, 111). However, such cultural artifacts perform a similar function in that they also embody generations of knowledge gleaned from navigational practices.

2.3.

Dimension 3: Phenomenology

"Embodiment" has a phenomenological sense in which it can refer to the things we consciously notice about the role of our bodies in shaping our self-identities and our culture through acts of conscious introspection and deliberate reflection on the lived structures of our experience (Brandt 2000, 1999). The conscious phenomenology of cognitive semiotics can be profitably contrasted with the cognitive unconscious of cognitive psychology (see dimension 7). For example, Gallagher (this volume) traces how the work of phenomenological philosophers such as Husserl and MerleauPonty has contributed to the distinction between the conscious body image and the largely automatic body schema now emerging in cognitive neuroscience. For Husserl and Merleau-Ponty, embodiment refers not only to the lived experience of our own bodies but also to the ways in which our experience of other animate bodies moving differs from our experience of other moving objects in the physical world. This has found theoretic support from cognitive neuroscience in the discovery of the "mirror neuron" system in the premotor cortex (Gallese et al. 1996; Rizzolatti and Craighero 2004), in which primates have been shown to have neural systems which are activated not only by their own motor actions but also by witnessing another's motor action. Gallagher suggests that the emergent sense of "intercorporeality" from mirror neuron activity could be a basis of human intersubjectivity.

352

Tim Rohrer

2.4.

Dimension 4: Perspective

"Embodiment" can also refer to the particular subjective vantage point from which a particular perspective is taken, as opposed to the tradition of the all-seeing, all-lmowing, objective and panoptic vantage point. While this sense of the term can be seen as at least partly philosophical (as in Nagel 1979: 196-213; Geeraerts 1985; Johnson 1987; Rohrer 1998), the idea of considering the embodied viewpoint of the speaker has linguistic implications for the role of perspective in subjective construal (Langacker 1990; MacWhinney 2003), as well as a myriad number of psychological implications (e.g., Carlson-Radvansky and Irwin 1993; Kosslyn et al. 2002). For example, consider how the embodied perspective of the subject can interact with the canonical orientations implicit to construing spatial situations. When we give directions, we ordinarily assume that one is facing in the direction of the travel. However, in many subway trains the seats face in both directions. Should we give directions such as "after the subway goes above ground, look to your left. When you pass the automobile dealership, exit at the next stop...", imagine the confusion if the addressee should choose a seat facing opposite in the direction of travel and not make the adjustment to look to the right. Similarly, not only our bodies but also many of the objects of our world have canonical orientations that our determined by the ways in which we - that is our bodies - interact with them. Cups and trash-cans stand upright, while mattresses lie flat. When we say "the fly is over the trash can", but the trash can is lying on its side, is the fly above the side of the trash-can or adjacent to its mouth? Buildings such as cathedrals or ski lodges also have canonical orientations as to their fronts and backs; one can say "1'11 meet you in the restaurant to the right of the cathedral at noon", and inadvertently fail to specify if that is the perspective of the tourist facing the cathedral or the perspective of the cathedral as it faces the city square. We routinely project the canonical orientation of our embodiment onto the objects in the world; sometimes we take up the perspective of inanimate objects, sometimes we take up the perspective of animate bodies. Of course, problems of co-aligning frames of reference are of practical import in areas such as ship navigation practices and the internal maps built up by robots; as such this dimension frequently interacts with the other practical senses of the term as well as the metatheoretic ones.

The body in space: Dimensions ofembodiment

2.5.

353

Dimension 5: Development

In yet another important sense "embodiment" is used to refer to the developmental changes that the organism goes through as it transforms from zygote to fetus, or from infant to adult. There are at least three ways in which the developmental sense interacts with the other dimensions of "embodiment". First, certain events in the development of an organism open windows for the acquisition of a particular skill. Babies are not born speaking, nor can they handle objects or self-Iocomote at birth. As the infant acquires additional sensorimotor skills, additional patterns become available to be incorporated into its cognitive functioning. Second, and perhaps counterintuitively, such events may not expand but instead constrain the mappings between the possible patterns of embodied perceptual structures and the resulting conceptual structures of later developmental stages. For example, Bowerman and Choi (2003) have shown that while nine-month old Korean and English speaking infants can make the same spatial discriminations, at eighteen months their acquisition of language has solidified their spatial categories enough so that they are no longer able to make the discriminations which their language does not. Note that such developmental changes are not purely physiological, but take place within the relevant socio-cultural and linguistic contexts.

2.6.

Dimension 6: Evolution

An equally important temporal sense of the term "embodiment" refers to the evolutionary course the species of organism has undergone throughout the course of its genetic history. For example, an account of the gradual differentiation of the cortex into separate neural maps each representing a different frame of reference in the visual and tactile systems of mammals might provide an evolutionary explanation for which multiple frames for spatial reference were universally found by the typological studies of spatial language and cognition (Majid et al. 2004). Or on an even grander scale: humans have not always had the capacity for language and so evidence from studies on the evolutionary dimension of embodiment may often prove crucial to understanding why, for example, language processing in the brain does not appear to be exclusively concentrated as an autonomous module but instead draws on numerous subsystems from the

354

Tim Rohrer

perceptual modalities (see for treatments Edelman 1992; Donald 1991; Deacon 1997; MacWhinney 1999).

2.7.

Dimension 7: The cognitive unconscious

Additionally "embodiment" can mean those routine cognitive activities that ordinarily operate too quickly and too automatically for the conscious mind to focus on them. Lakoff and Johnson (1999: 9-15) have recently called these the cognitive unconscious. In this sense "embodiment" refers to the ways in which our conceptual thought is shaped by many processes below the threshold of our ordinary conscious awareness. As such they are generally inaccessible to introspection, though they may be measured indirectly using methods from cognitive and social psychology. Lakoff and Johnson cite examples ranging from mental imagery to semantic processing to processing sound into phonemes. They intend that this sense of the term "describes all unconscious mental operations concerned with conceptual systems, meaning, inference and language" (1999: 12). While their primary source of evidence for the cognitive unconscious is cognitive psychology, Lakoff and Johnson also intend for this sense of embodiment to include the neural modelling at least some aspects of our neurophysiological embodiment, noting that these "are obviously not independent of one another" (1999: 104). However, I would argue that these two aspects can be profitably separated out from the cognitive unconscious of experimental cognitive and social psychology as two additional dimensions of embodiment.

2.8.

Dimension 8: Neurophysiology

In a neurophysiological sense, the term "embodiment" can refer to meas-

uring the activity of the particular neural structures and cortical regions that accomplish feats like object-centered versus viewer-centered frames of reference in the visual system, metaphoric projection, and so on (Rohrer 2001b, 2005; Coulson and Van Petlen 2002). Such methods would include single-neuron recording, electroencephalography (EEG) and derivative measures (EMG and ERP), positron emission tomography (PET), functional magnetic resonance imaging (fMRI), and magneto-encephalography (MEG), as well as the neuroanatomical organization of the brain and nervous system. This dimension would comprise a portion of, but not be syn-

The body in space: Dimensions ofembodiment

355

onymous with Lakoff and Johnson's use of their term "neural embodiment" (1999: 102-103), in which they lump neurophysiologically-based methods together with the neurocomputational modelling of both highlevel cognitive tasks (such as temporal aspect in language) and low-level cognitive tasks (spatial perception). Together with observations on human physiology, the relevant neurophysiology is sometimes advanced as explaining certain constraints on the patterns exhibited in linguistic systems, such as in the regularities in the cross-cultural typology of color words (Lakoff 1987).

2.9.

Dimension 9: Neurocomputational modelling

"Embodiment" can sometimes also refer to research using neurocomputational models. Such neural networks may be said to be "embodied" in at least four different ways. First, they may more or less closely model the actual neurophysiology of the neural circuitry whose function they seek to emulate. Second, some kinds of neural networks build on better-understood neurocomputational models of the actual neurophysiology to provide "existence proofs" that a series of neural nets could in principle account some kind of cognitive behaviour - as in "structured connectionism". Neurocomputational models such as those in Lakoff and collaborators' "neural theory of language" thus are not explicit models of the underlying neurophysiology, but instead (and by using as their input structures the output from better understood perceptual neural structures) they seek to demonstrate how the known computational facts about the neurophysiology could produce certain kinds of observable linguistic behaviours (such as the metaphoric structuring of more abstract experience in terms of perceptual experience) (Lakoff and Johnson 1999: 569-83; see also Regier 1992, 1995; Feldman and Narayanan 2003). Third, and most often without any explicit reference to any intermediate structures in the underlying neuroanatomy, connectionist neural networks are taken to be models of experiential activity at the conceptual and/or psychological levels of processing, as in the stochastically-based arguments that there is no "poverty of stimulus" but instead plenty of experience to account for the acquisition of syntax by children - (Elman et al. 1996; MacWhinney 2003). Fourth, neural networks can be seen as models of how socio-cultural norms can be internalized within a specific "mind" (Zlatev 1997, 2003; Howard 2001). For example, by developing neural models of how the age/gender contrasts are

356

Tim Rohrer

marked in English and Spanish, Howard argues that biased socio-cultural norms of age and gender are partly the result of predispositions of how the human brain and nervous system learn. It is important to note that the neurocomputational use of this sense of "embodiment" is partly motivated by the fact that some other physiological, experiential and/or socio-cultural dimension of that term is explicitly being modeled by the neural network. Yet such models seek to ground their models not only in what they seek to model, but also in the fact that "neurocomputational embodiment" is explicitly anti-functionalist. All neural networks are anti-functionalist in that the particular shape of the neural model is at least partly determined by analogizing some of its computational properties to the underlying neurophysiology, rather than presuming that the cognition or behavior to be modeled is computationally independent of any such bodily constraints (as in functionalist models).

2.10. Dimension 10: Morphology The terms "embodiment" and "embodied cognition" are now also widely used in robotics (Chrisley and Ziemke 2002) where any computational modelling necessarily requires a body of some type for interaction with the world. While in robotics it is perhaps most saliently associated there with humanoid robot projects (Brooks and Stein 1994), it can also refer to cases where the work done by the robot depends on the particular morphological characteristics of the robot body (Pfeifer and Scheier 1999). For example, Comell University's Passive Dynamic Walker uses no motors and no centralized computation but instead relies on gravity, mechanical springs and cleverly designed limb morphology to "walk". By exploiting the capacities of the morphology, cognition is offloaded onto the body - a design principle that is consonant with both evolutionary theory and embodied cognitive science (Collins, Wisse and Ruina 2001; Bertram and Ruina 2001). The morphology of the physiological body also yields important constraints for measuring cognition in cognitive psychology or cognitive neurophysiology, as in the already discussed studies by Parsons (1987ab, 1994; Parsons et al. 1995) on the mental rotation of line drawings of the hand and in the Vingerhoets et al. (2002) fMRI study of the activation courses in response to images of tools.

The body in space: Dimensions ofembodiment

357

2.11. Dimension 11: Directionality of metaphor Within Cognitive Linguistics, the term "embodiment" has two often conflated senses that stem from Lakoff and Johnson's (1980: 112) initial formulation of the embodiment hypothesis as a constraint on the directionality of metaphoric structuring. More accurately, this sense of "embodiment" could be termed the directionality of metaphor mappings. In this strong directionality constraint Lakoff and Johnson claim that we normally project image-schematic patterns of knowledge unidirectionally from a more embodied source domain to understand a less well-understood target domain. In other words, they claim that each and every mapping between the elements of the source and the elements of the target is unidirectional; the logic of the image-schema is projected from the source to the target, and not from target to source. For example, in their analysis of the metaphors shaping various theories of visual attention in cognitive psychology, Fernandez-Duque and Johnson claim that: [...] each submapping is directional, going from source to target. We understand aspects of the target domain via the source domain structures and not the reverse. Such unidirectionality shows itself clearly in the reasoning we do based upon conceptual metaphors. (Femandez-Duque and Johnson 1999: 85)

This constraint has been the source of much controversy within Cognitive Linguistics. Fauconnier and Turner (1995, 2002) and others have argued that there is a much greater role for feedback between target and source to the extent that they have proposed an alternate theory in cognitive semantics, conceptual blending, which is in part a response to the unidirectionality constraint. Non-unidirectional and blending-like phenomena can be observed with respect to the same theories of visual attention analyzed by Fernandez-Duque and Johnson. For example, they correctly argue that a VISUAL ATfENTION IS A SPOTLIGHT metaphor shapes research questions in cognitive psychology, such as for experiments designed to measure the speed at which the attentional spotlight moves across the visuo-spatial field when experimental participants shift the focus of their attention (Shulman, Remington and McLean 1979) and whether the subject would attend to intermediary objects in the path of the attentional spotlight (Tsal 1983). However, the cognitive psychologist Miisseler (1994) observed that although relatively proximal shifts in the focus of visual attention seemed to behave as if the attentional spotlight did follow an analog path across the visual field illuminating everything in its path, larger shifts in the focus of

358

Tim Rohrer

visual attention did not follow an analog path. This observation about the target domain (visual attention) initiated a re-examination of the source domain. Miisseler initially proposed that the attentional spotlight was "reset" during large attentional shifts. However, convergent evidence from both other psychological studies of attention and from cognitive neuropsychology on colour, motion, shape and other visual subsystems caused an even more radical shift in the source domain of the metaphor - visual attention began to be understood as an array of multiple spotlights, as would be found in a theatre (Rohrer 1998). In no case was this process of feedback, revision, and accommodation strictly unidirectional; it was always motivated by observed changes in the target domain of visual attention that required making changes to the source domain of the spotlight(s).

2.12. Dimension 12: Grounding Finally, "embodiment" can be used to refer to a particular hypothesis as to how we might explain how abstract symbolic behaviour is grounded in experience. Within Cognitive Linguistics even Lakoff and Johnson's original formulation (1980: 112) of the embodiment hypothesis contained the germ of a broad generalisation about the kinds of basic conceptual domains which were typically serving as source domains for conceptual metaphors, rather than as explicitly referring to the directionality of projection for each and every element mapped within a particular metaphor. We might call this additional sense of embodiment the directionality of explanation in order to distinguish it from the directionality of metaphor mappings. Lakoff and Turner specifically acknowledge this sense of embodiment in their Hgrounding hypothesis ", in which they argued that meaning is grounded in the sense that we must choose from a finite number of semantically autonomous source domains to understand more abstract experiences (Lakoff and Turner 1990: 113-120). This sense of the term is related to the symbol-grounding problem in cognitive science generally (Harnad 1990), though it is important to note that many embodiment theorists would want to address that issue without conceding any sort of Cartesian-like split between words, thoughts or symbols and the worldly things to which they refer. This descriptive list is meant to illustrate that the embodied cognitive science requires thinking through evidence drawn from a multiplicity of perspectives on embodiment, and therefore drawn from multiple method-

The body in space: Dimensions ofembodiment

359

ologies. Of course almost no researcher or research project can attend to all these different senses of the term at once and still produce scientific findings, but research projects that build bridges or perform parallel experiments across these differing dimensions are of particular interest. However, once the descriptive work has been done it can be seen that many of these senses cluster about at least two poles of attraction. Critiques of embodied cognitive science from within have often given voice to two broad senses in which the term "embodiment" is used. These two could be well described as "embodiment as broadly experiential" and "embodiment as the physical substrate". In one cluster the term refers to dimensions that focus on the specifically subjective, contextual, social, cultural and historical experiences of language speakers. Dimensions (2) through (4) of my enumeration of the term's usage would typically cluster in this realm, while dimensions (7) through (10) would often cluster about the pole which emphasizes the physiological and neurophysiological bodily substrate that is typically associated with supposedly more "objective" methodologies. Such a division is at best rough and provisional however. Clearly not all the dimensions of the term can be so clustered, given that the attention to temporal character which characterizes the developmental (5) and evolutionary (6) dimensions can place them about either pole. Similarly, there are many interesting studies which bridge the gap between the experiential and the physiological poles even while largely measuring a dimension typically construed as mostly one or the other, such as the Kosslyn group's neuroimaging research into alternate strategies of mental rotation (2001). Depending on the behaviour modelled, embodiment as neurocomputational modelling (9) can also cross the line from the physical substrate to more experiential matters. Finally, the more explicitly meta-theoretical dimensions of the term, (1), (11) and (12), have much traffic with both the experiential and physical substrate poles and also do not lend themselves easily to such a rough and ready distinction. Given such considerations, at least two more poles of attraction emerge - temporal and metatheoretical studies. In the end, an adequate theoretic framework for embodiment theory in cognitive science will have to acknowledge all of the wide variety of senses in which the term "embodiment" is being used and provide a nonreductionistic framework for reconciling research across all these different dimensions.

360

Tim Rohrer

3.

The levels of investigation theoretic framework

The rough and ready distinction between experiential and physiological embodiment does have the virtue, however, of illuminating how we might assess the utility of different approaches to embodied cognition. Two other recent attempts to clarify the uses of the term "embodiment" have come to diametrically opposed conclusions as to how embodiment theory can contribute to future work. On one tack, the computer scientist Anderson (2003) proposes evaluating embodiment theory in terms of its practical utility in reconceiving how our work and our lives can be enhanced. His objectives are fundamentally technological, inquiring whether embodiment theory has any import for efforts to offload difficult cognitive tasks onto the social and cultural environment, such as teamwork or the design of intelligent material artifacts such as embedded computers. On another tack, theorists such as the psychologist Wilson (2002) propose an experimentally-based evaluation of embodiment theory, arguing that we need more investigations of how offline, subconscious bodily processes structure real-time cognition while explicitly rejecting efforts to explain cognition as being situated in and distributed across socio-cultural practices. Clearly, these approaches differ not only as to what direction future research should take but also in terms of the physical scale of their research scope. Wilson's concerns are primarily with how the individual physiological organism interacts with its environment, arguing that distributed social and cultural patterns of cognition are too impermanent to constitute a unit of analysis having explanatory force (2002: 630-631). While Anderson also agrees that insights from how the embodied individual organism interacts with the physical world are relevant, he argues that embodied cognition has "social significance, for the construction of meaning, of the terms through which we encounter the world, is not generally private, but is rather a shared and social practice" (2003: 125). In considering whether and how a material artifact such as patient's medical chart might be extended by embedded computers, he notes that such efforts should not seek to obliterate important facets of how the material artifact encodes the fact that cognition is distributed across the social group caring for the patient, such as how the handwriting indicates what member of the group made a particular observation or where the chart is kept indicates what member of the medical team has responsibility for the patient. On Wilson's argument the scope of the research would be limited to the size of space within which the individual interacts with the artifact, but for those cognitive scientists interested in improving the func-

The body in space: Dimensions ofembodiment

361

tioning of such social groups, the scope of the physical scale expands to encompass the entire cultural and communicative space of the team. However, from my own vantage as a cognitive scientist originally trained in the philosophical tradition of American Pragmatism, both of these theorists are sailing in the right direction - given that the relevant operational scale of the phenomena they are studying has changed. The first challenge for developing a theoretical framework in which we can address such differing approaches is to propose the adoption of a simple and well-understood organising criterion. Unfortunately, most previous proposals have generally accorded an ontological status, rather than an epistemological or methodological status, to the organisation of their theoretic framework. Thus most such frameworks postulate "higher" and "lower" levels of cognition in ways which imply that the higher levels may be reduced to operations at the lower levels, ultimately arguing for the elimination of higher-levels of description in favour of lower levels of description (Churchland 1981, 1989). One exception is Posner and Raichle's (1994) schematisation of the levels of investigation in cognitive neuroscience, in which the primary emphasis is given to the methodologies used to investigate the phenomena rather than their ontological status. Similarly, Edelman (1992) points out that in the physical sciences, the phenomena are operationally grouped in levels according to the physical scale of the methodology with which the phenomena are being studied. Thus the most basic organising criterion of this theoretic framework is the scale of the relative physical sizes of the embodied phenomena which produce the different kinds of socio-cultural, cognitive or neural events to be studied. In Figure 1, physical size is mapped on the y-axis, providing a relative distribution of the "higher to lower" methodological levels of cognitive processes. A general name for each level is indicated by boldface type in the first column. To provide clarification, the next column provides examples of what the relevant physiological structures are at a given physical scale. For example, at the communicative, cultural and social level we study spatial language as it used between people, and hence multiple central nervous systems; alternatively, it is possible to measure one individual's (and hence one central nervous system's) performance on a similar set of linguistic tasks. Similarly we can examine, with even more granularity, relative changes in cerebral blood flow to regions of the brain in response to spatial linguistic tasks; or we can construct neurocomputational models of those brain regions. However, Posner and Raichle's key

362

Tim Rohrer

Sample Operative Theoretic Constructs Viewer-centered, Multiple Communicative Cross-cultural lm investigations of object-centered, and cultural Central and mental rotation geo-centered systems in Nervous up frames of referand frames of Systems anthropology, ence in language; language, science reference; language child-rearing and philosophy acquisition; practices; conceptual norms as to which metaphor; gesture spatial frame used Individual Spatial frames of Performance .5m Central reference, speed performance on domain; Nervous to Cognitive, frames of referof mental 2m Systems ence and mental rotation; conceptual, morphological rotation tasks; gestural and linguistic systems measuring ability constraints to gesture in as performed by individual subjects direction-giving situations contrived to inhibit it IO-lm Gross to Neural systems Activation course Body-image, in somatosensory, motor and visual medium size to IO-2 m neural cortices, whatauditory, and visual processing where pathway; Regions areas when (anterior processing spatial cingulate, frames of referparietal lobe, ence tasks or etc.) mental rotation tasks IO-lm Neural Neuroanatomy; Neuroanatomical Motor and visual Neural circuitry in connections from cortices, parietal networks, to IO-4 m maps and topographic neural maps, pathways, visual, auditory, somatosensory maps sheets pathways regions to language areas IO-J m Individual Fine neuroOrientation-tuning Neurocellular systems; Cellular anatomicalorgani- cells; ocular neurons, to IO-6 m cortical sation of particular dominance and very small structures recolumns columns intercellular cruited in lanstructures Iguage processing None-beyond Neurotransmitter, Subcellular Less Neurotheoretical scope sYnapse, ion Than transmitters, systems; IO-6 m ion channels, subcellular, mochannels lecular and sYnapses electrophysical

Size

Physiological Level of Investigation Structures

Figure 1.

Typical Cognitive Science Tasks

Sample Methods o/Studyand Measurement Linguistic analysis, cross-linguistic typology, videotaped interview, cognitive ethnography

Verbal report, observational neurology, discourse analysis, cognitive and Developmental studies examining reaction time (RT)

Lesion analysis, neurological dissociations, neuroimaging using fMRI and PET, ERP methods, neurocomputational simulations

Electrocellular recording, anatomical dyes, neurocomputational simulations Electrocellular Recording, anatomical dyes, neurocomputational simulations Neuropharmacological, neurochemical, and neurophysical methods

Theoretic framework for embodied cognitive science.

The body in space: Dimensions ofembodiment

363

insight is that it is important to consider how the basic inquiry changes given the different tasks and methods at various levels of investigation. All methodologies have constraints and freedoms which limit or enhance their scope of investigation and define the theoretical constructs that they develop, and these are a product of the physical scale at which the measurement is taking place. The final two columns acknowledge this by specifying some of the relevant theoretic constructs and the various methodologies operative at each level of investigation. This framework can be used to structure studies of various topics of interest to cognitive scientists, such as mental imagery, frames of reference, metaphor and so on. While this type of theoretic framework is becoming commoner within much of cognitive neuroscience, most embodiment theorists have been slow to give explicit attention to the problem of how we are to theoretically situate and reconcile these different levels of investigation, perhaps due to a fear of appearing to favor reductionism. I have included just a single level of cultural and communicative analysis, but by no means should this be taken as indicative of its importance relative to other the other levels. Of course, one could argue for a multiplicity of levels embedded within this one, though they might not be clearly differentiated from one another in terms of physical scale. In choosing to include a general level situated at a meter and up on the physical size axis I mean to emphasize only that human beings should be considered not simply in terms of physiological size, but also in terms of the standard scale of their interactional distance in speaking and interacting with one another. At this level of the chart the "physiological structures" column reads "Multiple Central Nervous Systems", but that awkward term is intentionally inadequate so as to emphasize that the physiology is less relevant here - what primarily matters on this level are the social and cultural interactions between human beings. Investigations at the cultural level are occasionally given short shrift by some versions of embodied cognitive science, but generally this has been and should remain a strong thrust of future research in the field. Note also that difficult phenomena such as cultural and linguistic norms, or individual consciousness and awareness, are situated at the physical scale at which they are measured and observed, rather than attempting to place them on (or reduce them to) a lower level of investigation. Nonetheless, it is certainly possible and sometimes useful to ask, for example, how long the neural processing of a visual experience takes before impacting conscious decision-making, or how the linguistic norm of forming the English past tense might be performed in a neurocomputa-

364

Tim Rohrer

tional model. However, research in embodied cognitive science should not seek to reduce such phenomena to another level but should instead bridge across these levels in important ways - for example, the linguistic corpora used to train the neurocomputational model should be based on naturalistic recordings of an actual child's utterances rather than text harvested from internet newsgroups, and so on. While the chart depicting the theoretic framework is designed to give an overview of the relationship between body, brain and culture, this representation is not as illustrative for issues pertaining to evolutionary and developmental time scales, which may be considered at any of these levels. However, this failing is more a limitation of the imagery of a twodimensional chart than of the theoretic framework itself. If we were to add another axis for time perpendicular to the surface plane of the chart, we could imagine this framework as a rectangular solid. I have omitted representing this dimension because such an illustration would make it difficult to label the levels, but I make the point explicit here because both the developmental and evolutionary time courses of these phenomena are a central dimension to understanding them, and their bearing on the embodied mind.

4.

Applications of the theoretic framework

This theoretic framework can help link related research from one level of investigation to another, providing opportunities to test similar hypotheses and incorporate insights originally developed at one level of investigation at another. As in the introduction I have already given several examples of embodied cognition linking physiological and neurophysiological experiments on cognition, consider three examples from the cultural and performative levels of investigation involving mental rotation and frames of reference. First, in a series of cross-cultural and cross-linguistic typological studies on spatial cognition, Pederson et al. (1998) have found that the linguistic frame of reference4 which predominates in a language strongly influences 4. As I summarize their typological work I use Levinson's (2003) terminology for linguistic frames of reference, though I try to indicate a rough alternative term from the broader literature on spatial frames of reference in the cognitive sciences where their nomenclature may be uninformative to the naIve reader. However, such indications are intended only as clarifying approximations, and

The body in space: Dimensions ofembodiment

365

the spatial cognition problem-solving strategy chosen. In one of several tasks they use, experimenters place three animal figurines in a row on a table and ask participants to memorize the scene. The participants are then rotated 180 degrees and asked to reconstruct the scene on a second table. Speakers of languages which predominately use a relative frame of reference normally recreate the scene relative to their own body position, i.e., the animal to their left in the original rotation remains the animal on their left in the new 180-degree rotation. However, speakers of languages in which an absolute (geo-centric) frame of reference predominates recreate the same scene relative to the position of the animals with respect to invariant features of the landscape, i.e., the animal to the north is still placed on the north side even though the participant's body position has been rotated 180 degrees from the original scene. Despite numerous challenges, they have successfully replicated their findings in a variety of experimental environments and across a wide variety of languages (Majid et al. 2004). A typological finding, gathered at the cultural level of investigation, has been transposed onto the performative level of investigation. Their experimental research has been extended by a second source of evidence concerning spatial frames of reference which is of substantial interest to researchers working on embodiment - studies of gesture. By cleverly manipulating the body position of his experimental subjects relative to the directions the experimenter requests, Kita (2003) has shown that in giving directions speakers of languages in which the relative frame of reference predominates will often make difficult, torso-twisting gestures across their bodies in an attempt to co-align their current bodily perspective with the perspective that they would need to have in order to tell if a landmark at that point in the directions will be on the right or left. Just as with the work on mental rotation and mental imagery, in direction-giving the embodied mind simulates following the path of the directions being given. This process of mental imagery - the mental gymnastics done by the speaker's mind as one visualizes how the hearer will need to follow the directions - surfaces in the form of physical gymnastics - that is, the gestures involving the rotation of the torso. Once again, these experiments link the cultural and performative levels of investigation. Third, evidence of the ways in which language and culture embody spatial frames of reference has also been found in studies linking gesture

readers should refer to Levinson's own work for detailed defmitions of his terms for the linguistic frames of reference.

366

Tim Rohrer

and conceptual metaphor. In Indo-European languages such as English and Spanish, time is typically conceptualized using two basic TIME IS SPACE metaphors - one in which the observer stands still while times pass by (e.g., "The end of the year is coming up on us soon"; and another in which the observer is moving through a landscape of times (e.g., "We're coming up on the end of the year") (Lakoff and Johnson 1980, 1999). However, in both metaphor systems the observer faces the future, while the past is behind the observer. Nufiez and Sweetser (2006) have shown that for speakers of Aymara, who almost exclusively use a stationary version of the TIME IS SPACE metaphor, the past is in front and the future comes from behind. In Aymara, the orientation of the spatial frame of reference is reversed. Using videotaped interviews with bilingual Spanish and Aymara speakers both recounting Aymara legends and talking about their own communities' immediate future, they demonstrate how speakers gesture to areas in front of them when referring to the past, while gesturing to future events with over the shoulder motions. Furthermore, the gestures reveal that more recent past events are closer to the speaker's point of view than events in the more distant past. For example, as he contrasts ancient times with current events, an informant gestures by pointing outward and upward as opposed to pointing four times closer to his body. Together with their linguistic research on the Aymara conceptual metaphors for time, the research of Nufiez and Sweetser shows that the Aymara map the temporal frame of reference onto a viewer-centered frame of reference in an inverse direction to that found in English or Spanish. These three studies, largely situated at the "top" levels of the theoretic framework, are excellent examples of how the cultural level of embodiment is expressed in the performances of individual experimental participants. Not only that, but they also provide clues of what researchers might ask next at other levels of investigation. For example, one might ask what sorts of analogous linguistic stimuli would show neural responses similar to spatial stimuli. One could design neuroimaging experiments investigating whether navigating directions on a visual display required mental imagery rotation in either the visual or motor imagery system. Analogously, one could also investigate whether or not sentence stimuli concerning a similar navigational task requiring either a viewer-centered, objectcentered, or geo-centered set of coordinates would activate distinct brain regions. There are several convergent neural studies which indicate that this is a hypothesis worth pursuing. In a recent review article, Parsons (2003) ar-

The body in space: Dimensions ofembodiment

367

gues that we can already distinguish between superior parietal cortical regions active when mentally rotating objects about their own axis (i.e., in an object-centered frame) and superior parietal regions active when mentally rotating one's own body (a viewer-centered frame). Several neuroimaging studies using spatial relations terms as linguistic stimuli also show activation of either left (Damasio et al. 2001) or both (Emmorey et al. 2005; Carpenter et al. 1999) superior parietal cortical areas, though none of those tasks were designed to elicit distinctions between the frames of reference. Similarly, there is also convergent evidence from the literature on attention in visual hemi-neglect (a syndrome typically resulting from damage to the right superior parietal cortex) suggesting that viewer-centered and object-centered neglect are dissociable phenomena in experimental and observational neurology (Behrmann and Tipper 1999; see Rohrer 2001a for discussion). Halligan et al. (2003) reviews the neurological evidence that neglect of near and far space occurs independently, suggesting that they are likely to be mapped in separable parietal areas - evidence which suggests that the geocentric frame of reference in language could be correlated with the map of far space. Furthermore, while most studies of neglect do not report language deficits on standard naming or visuo-auditory picture matching tasks, a recent study by Eden et al. (2003) shows that when dyslexic children were asked to draw a clock face they neglected to fill in the numbers on the left side of the clock face in much the same manner as observed for the standard clock drawing task given visual neglect patients. Their results suggest that there may be a common underpinning for spatial representation and language in right superior parietal cortical areas. Yet more evidence of parietal involvement in spatial language can be found in Coslett and Lie (2000), who have reported on parietal patients who exhibit spoken language and comprehension deficits correlated with the spatial orientation of their body (see also Chatterjee 2001). In sum, while there are not yet any definitive studies showing that linguistic stimuli can drive distinct regions of the parietal cortex thought to be responsible for maintaining viewer-centered, object-centered and geo-centered frame of reference maps, studies of both parietal involvement in language and the parietal processing of different spatial frames of reference provide good reason to investigate the possibility. Although the mapping between the frames of reference observed in linguistic typologies (Majid et al. 2004) and the frames of reference used in neural processing is not likely to be isomorphic, experiments bridging the two lines of investigation could prove fruitful.

368

Tim Rohrer

One could also ask about the role of the parietal regions in understanding metaphoric expressions of spatial relations. If linguistic stimuli can drive the brain regions demonstrably involved in tasks involving the different spatial frames of reference, then would metaphoric linguistic stimuli drive those same brain areas given in tasks involving a temporal frame of reference? Similar to the Aymara gestural results, other studies of how temporal reasoning and spatial reasoning interact (see Matlock, Ramscar and Boroditsky 2005) have shown that temporal problem-solving uses the resources of spatial problem-solving. One could design linguistic comprehension and linguistic problem-solving tasks using the TIME Is SPACE metaphor that might differentially activate those parietal areas. It is also important to see that the dialogue between the levels of investigation in this theoretic framework is not a one-way street. Similar questions can also be taken to the cultural and communicative level from the neurophysiological level. For example, and with respect to multimodal user interface design, as we know that altering the viewer-centered versus the object-centered perspective influences which neural system performs mental rotations (Kosslyn et al. 2001), can we take advantage of that fact to design better hand controls and visual displays for real-world tasks that involve real-time mental rotation, such as airplane piloting or air traffic control? The appropriate display and control interface should prompt for the appropriate frame of reference. How might such controls help mitigate errors in frames of reference problems that result from the finding that people find it more difficult to rotate object-centered frames than egocentered frames (Wraga, Creem and Proffitt 1999)? If the speakers of some languages prefer to use a certain frame of reference, how does that alter how one might teach technical skills (such as navigation) that involve mental rotations in a cross-cultural setting? The list of such questions is endless, for the details of embodiment play an enormous role in every cognitive activity.

5.

Conclusions

Human beings have bodies, and those bodies shape and constrain how we think. At the outset of this chapter, I asked the reader to imagine how an event as mundane as a power outage can reawaken an awareness of the body. However, and as I have argued throughout this chapter, the body is not some other thing to which the mind returns when thinking is inter-

The body in space: Dimensions ofembodiment

369

rupted - thinking itself is shot through and through with the body. In the words of the early American Pragmatist philosopher-psychologist William James: [... ] our own bodily position, attitude, condition, is one of the things of which some awareness, however inattentive, invariably accompanies the knowledge of whatever else we know. We think; and as we think we feel our bodily selves as the seat of the thinking. If the thinking be our thinking, it must be suffused through all of its parts with that peculiar warmth and intimacy that make it come as ours. (James 1900, 1: 241-242, emphases original)

When we think about rotating a hand in space, we feel that it is more difficult to rotate in ways which conflict with the natural motions of our joints; when we have pain in the appropriate joints, we rotate it slower. As James notes, we often may be inattentive to this sort of awareness, especially when we are physically fit and habitually lost in thought; at such times we may need experiments to reacquaint ourselves with the body. But for those of us who feel pain in their wrists as they imagine gripping and rotating a hammer, no argumentative reintroduction is necessary - for them, the awareness that our own thinking is embodied is inescapable. James' felt sense of the experienced body, or what I have called the phenomenological dimension of embodiment, is only one prominent dimension among the many dimensions of the term discussed in this article. Over the first fifty or so years of cognitive science the field deliberately theorized away the many contributions of the body to thinking, a theoretical failing that is only now beginning to be corrected. Interdisciplinary research, ranging from linguistics and neuroscience to philosophy and psychology, using methods ranging from the experimental and physiological to the phenomenological and sociological, has begun to show that there are multiple dimensions along which the human body shapes human thought. For example, in classical cognitive science our mental representations were assumed to be logical and symbolic; however, cognitive neuroscience has shown them to be embodied and image-like. Like the Pragmatist philosophers, the new approaches see human cognition as action situated within a practical context, and mental representation as instrumental rather than absolute. Our ordinary experience of space is not one in which we seek to discover some absolute external frame of reference, but one in which we are obsessed with coordinating multiple frames of reference - body-, head-, or other-centered - in order to solve the practical problem at hand. That practical problem may be getting directions to an unfamiliar place, helping

370

Tim Rohrer

a patient recover from and cope with visual neglect, a team of navigators piloting a ship, file management within a computer's graphical user interface, or tracking a lost child through the forest; but all of these examples share a common thread of understanding how the body establishes frames of reference and moves in space. As the neuroscientist Antonio Damasio puts it: The body, as represented in the brain, may constitute the indispensable frame of reference for neural processes that we experience as the mind; that our very organism rather than some absolute external reality is used as the ground reference for the constructions we make of the world around us and for the construction of the ever-present sense of subjectivity that is part and parcel of experiences; that our most refmed thoughts and best actions, our greatest joys and deepest sorrows, use the body as a yardstick. (Damasio 1994: xvi)

Embodied cognitive science begins with the realization that the body, along all of the dimensions I have outlined in this chapter, grounds and shapes human cognition. Wakened from its long dormant slumber, the body has returned to cognitive science. In this survey chapter I have traced how our physiological, neurophysiological, interactional and sociocultural embodiment impinges on how we think. The embodied mind is not something which should be narrowly identified with anyone of these levels of investigation, nor with anyone of the dimensions of variability that I have noted. The embodied mind cannot be reduced only to the brain any more than it can be reduced to culture. Nor is the embodied mind merely a computer, in any traditional sense of that term; it can be said to perform "computations", but the substance and structure of these computations are imagistic due to the particular kind of bodies we have and the environments we inhabit. We are just at the beginning phase of understanding the myriad ways in which the body is in the mind.

References Amorim, Michel-Ange, Brice Isableu and Mohammed Jarraya 2006 Embodied spatial transformations: "Body analogy" for the mental rotation. Journal of Experimental Psychology: General 135: 247327.

The body in space: Dimensions ofembodiment

371

Anderson, Michael L. 2003 Embodied cognition: A field guide. Artificial Intelligence 149: 91130. Behrmann, Marlene and Steven P. Tipper 1999 Attention accesses multiple reference frames: Evidence from visual neglect. Journal ofExperimental Psychology 25: 83-101. Bertram, John E. A. and Andrew Ruina 2001 Multiple walking speed-frequency relations are predicted by constrained optimization. Journal ofTheoretical Biology 209: 445-453. Bowerman, Melissa and Soonja Choi 2003 Space under construction: Language-specific spatial categorization in fITst language acquisition. In: Dedre Gentner and Susan GoldinMeadow (eds.), Language and Mind, 387-427. Cambridge, MA: MIT Press. Brandt, Per Aage 1999 Domains and the grounding of meaning. In: Jose Luis Cifuentes Honrubia (ed.), Estudios de Lingiiistica Cognitiva, 467-478. Alicante: Department de Filologia Espaiiola, Lingiiistica General y Teoria de la Literatura, Universidad de Alicante. 2000 Metaphor, catachresis, simile: A cognitive and semiotic approach and an ontological perspective. Ms., Aarhus, Denmark: Center for Semiotic Research, University of Aarhus. Brooks, Rodney A. and Lynn A. Stein 1994 Building brains for bodies. Autonomous Robots 1 (1): 7-25. Brugman, Claudia 1985 The use of body-part terms as locatives in Chalcatongo Mixtec. In: Report No. 4 of the Survey of Californian and Other Indian Languages, 235-290. Berkeley: University of California. Carlson-Radvansky, Laura A. and David E. Irwin 1993 Frames of reference in vision and language: Where is above? Cognition 46: 223-244. Carpenter, Patricia A., Marcel Adam Just, Timothy A. Keller, William F. Eddy and Keith R. Thulbom. 1999 Time course of fMRI activation in language and spatial networks during sentence comprehension. Neuroimage 10: 216-224. Chatterjee, Anjan 2001 Language and space: Some interactions. Trends in Cognitive Sciences 5: 55-61. Chrisley, Ronald and Tom Ziemke 2002 Embodiment. Encyclopedia of Cognitive Science, 1102-1108. Macmillan Publishers.

372

Tim Rohrer

ChurcWand, Paul 1981 Eliminative materialism and the propositional attitudes. Journal of Philosophy 78: 67-90. 1989 A Neurocomputational Perspective: The Nature of Mind and the Structure ofScience. Cambridge, MA: MIT Press. Collins, Steven H., Martijn Wisse and Andrew Ruina 2001 A 3-D passive-dynamic walking robot with two legs and knees. International Journal ofRobotics Research 20: 607-615. Coslett, H. Branch and Eunhui Lie 2000 Spatial Influences on Language Performance. J. Cognitive Neuroscience (Suppl.): 76. Coulson, Seana and Cynthia Van Petten 2002 Conceptual integration and metaphor: An event-related potential study. Memory and Cognition 30: 958-968. Cummins, Robert 1977 Programs in the explanation of behavior. Philosophy of Science 44: 269-287. Damasio, Antonio R. 1994 The Feeling of What Happens: Emotion, Reason and the Human Brain. Cambridge, MA: MIT Press. Damasio, Hanna, Thomas J. Grabowski, Daniel Tranel, Laura L. B. Ponto, Richard D. Hichwa and Antonio R. Damasio 2001 Neural correlates of naming actions and of naming spatial relations. NeuroImage 13: 1053-1064. Deacon, Terrence 1997 The Symbolic Species: The Co-evolution ofLanguage and the Brain. New York: W. W. Norton. Donald, Merlin 1991 Origin ofthe Modern Mind: Three Stages in the Evolution of Culture and Cognition. Cambridge, MA: Harvard University Press. Edelman, Gerald M. 1992 Bright Air, Brilliant Fire: On the Matter ofMind. New York: Basic Books. Eden, Guinevere F., Frank B. Wood and John F. Stein 2003 Clock drawing in developmental dyslexia. Journal of Learning Disability 36: 216-228. Elman, Jeffrey L., Elizabeth A. Bates, Mark H. Johnson, Annette Karmiloff-Smith, Dominico Parisi and Kim Plunkett 1996 Rethinking Innateness. A Connectionist Perspective on Development. Cambridge, MA: MIT Press.

The body in space: Dimensions ofembodiment

373

Emmorey, Karen, Hanna Damasio, Stephen McCullough, Thomas J. Grabowski, Laura L. B. Ponto, Richard D. Hichwa and Ursula Bellugi 2005 Neural systems underlying spatial language in American Sign Language. Neuroimage 17: 812-824. Fauconnier, Gilles and Mark Turner 1995 Conceptual integration and formal expression. Metaphor and Symbolic Activity 10: 183-204. 2002 The Way We Think. New York: Basic Books. Feldman, Jerome and Srini Narayanan 2004 Embodied meaning in a neural theory of language. Brain and Language 89: 385-392. Femandez-Duque, Diego and Mark L. Johnson 1999 Attention metaphors: How metaphors guide the cognitive psychology of attention. Cognitive Science 23: 83-116. Gallagher, Shaun this vo!. Phenomenological and experimental contributions to understanding embodied experience. Gallese, Vittorio, Luciano Fadiga, Leonardo Fogassi and Giacomo Rizzolatti 1996 Action recognition in the premotor cortex. Brain 119: 593-609. Geeraerts, Dirk 1985 Paradigm and Paradox: Explorations into a Paradigmatic Theory of Meaning and its Epistemological Background. Leuven, Belgium: Leuven University Press. Halligan, Peter, Gereon R. Fink, John C. Marshall and Giuseppe Vallar 2003 Spatial cognition: Evidence from visual neglect. Trends in Cognitive Sciences 7: 55-61. Hamad, Steven 1990 The symbol grounding problem. Physica D 42: 335-346. Hodges, Andrew 1983 Alan Turing: The Enigma. New York: Walker and Co. Howard, Harry 2001 Age/gender morphemes inherit the biases of their underlying dimensions. In: Esra Sandikcioglu and Rene Dirven (eds.), Language and Ideology, 165-196. AmsterdamlNew York: John Benjamins. Hutchins, Edwin 1995 Cognition in the Wild. Cambridge, MA: MIT Press. 1999 Blending and material anchors. Ms. San Diego, CA: Cognitive Science Department, University of California at San Diego. James, William 1900 Psychology (American Science Series, Briefer Course). New York: Henry Holt and Co.

374

Tim Rohrer

Jensen de Lopez, Kristine 2002 Language-specific patterns in Danish and Zapotec children's comprehension of spatial grams. In: Eve Clark (ed.), The Proceedings of the 31st Stanford Child Language Forum: Space in Language Location, Motion, Path and Manner, 50-59. Stanford University: Center for the Study of Language and Information. Jensen de Lopez, Kristine and Chris Sinha 1998 Corn stomach basket. Spatial language and cognitive development in Danish- and Zapotec-acquiring children. Paper presented at the Child Language Seminar, Sheffield University England, September 1998. Johnson, Mark L. 1987 The Body in the Mind: The Bodily Basis of Meaning, Imagination and Reason. Chicago: University of Chicago Press. Johnson, Mark L. and Tim Rohrer this vol. We are live creatures: Embodiment, American Pragmatism and the cognitive organism. Kita, Sotaro Interplay of gaze, hand, torso orientation and language in pointing. 2003 In: Sotaro Kita (ed.), Pointing: Where Language, Cognition and Culture Meet, 307-328. Mahwah, NJ: Lawrence Earlbaum. Kosslyn, Stephen M. 1994 Image and Brain: The Resolution of the Imagery Debate. Cambridge, MA: MIT Press. Kosslyn, Stephen M., William L. Thompson, Irene J. Kim and Nathaniel M. Alpert 1995 Topographic representations of mental images in primary visual cortex. Nature 378: 496-498. Kosslyn, Stephen M., Gregory J. DiGirolamo, William L. Thompson and Nathaniel M. Alpert Mental rotation of objects versus hands: Neural mechanisms revealed 1998 by positron emission tomography. Psychophysiology 35: 151-161. Kosslyn, Stephen M., William L. Thompson, Mary J. Wraga, and Nathaniel M. Alpert 2001 Imagining rotation by endogenous versus exogenous forces: Distinct neural mechanisms. NeuroReport 12: 2519-2525. Kosslyn, Stephen M., Giorgio Ganis and William L. Thompson 2002 Neural foundations of imagery. Nature Reviews Neuroscience 2: 635-642. Lakoff, George 1987 Women, Fire and Dangerous Things. Chicago: University of Chicago Press. Lakoff, George and Mark L. Johnson 1980 Metaphors We Live By. Chicago: University of Chicago Press.

The body in space: Dimensions ofembodiment 1999

375

Philosophy in the Flesh: The Embodied Mind and Its Challenge to Western Thought. New York: Basic Books. Lakoff, George and Mark Turner 1990 More than Cool Reason: A Field Guide to Poetic Metaphor. Chicago: University of Chicago Press. Langacker, Ronald 1990 Concept, Image and Symbol: The Cognitive Basis of Grammar. New York, NY: Mouton de Gruyter. Levinson, Stephen C. 2003 Space in Language and Cognition: Explorations in Cognitive Diversity. Cambridge: Cambridge University Press. MacLaury, Robert 1989 Zapotec body-part locatives: Prototypes and metaphoric extensions. International Journal ofAmerican Linguistics 55: 119-154. MacWhinney, Brian 1999 The emergence of language from embodiment. In: Brian MacWhinney (ed.), The Emergence ofLanguage, 213-256. Mahway, NI: Lawrence Earlbaum. 2003 The emergence of grammar from perspective-taking. Ms., Pittsburgh, PA: Camegie Mellon University. Majid, Asifa, Melissa Bowerman, Sotaro Kita, Daniel B. M. Haun and Stephen C. Levinson 2004 Can language restructure cognition? The case for space. Trends in Cognitive Sciences 8: 108-114. Matlock, Teenie, Michael Ramscar and Lera Boroditsky 2005 The experiential link between spatial and temporal language. Cognitive Science 29: 655-664. Miisseler, Iochen 1994 Position-dependent and position-independent attention shifts: Evidence against the spotlight and premotor assumption of visual focusing. Psychological Research 56: 251-260. Nagel, Thomas 1979 Mortal Questions. New York: Cambridge University Press. NUfiez, Rafael and Eve Sweetser 2006 In Aymara, Next Week Is Behind You: Convergent Evidence from Language and Gesture in the Crosslinguistic Comparison of Spatial Construals of Time. Cognitive Science 30: 1-49. Parsons, Lawrence M. 1987 a Imagined spatial transformations of one's hands and feet. Cognitive Psychology 19: 178-241. 1987 b Imagined spatial transformation of one's body. Journal of Experimental Psychology 116: 172-191.

376

Tim Rohrer 1994

Temporal and kinematic properties of motor behavior reflected in mentally simulated action. Journal of Experimental Psychology: Human Perception and Performance 20: 709-730. 2003 Superior parietal cortices and varieties of mental rotation. Trends in Cognitive Science 7: 515-551. Parsons, Lawrence M., Peter T. Fox, J. Hunter Downs, Thomas Glass, Traci B. Hirsch, Charles C. Martin, Paul A. Jerabek and Jack L. Lancaster 1995 Use of implicit motor imagery for visual shape discrimination as revealed by PET Nature 375: 54-58. Pederson, Eric, Eve Danzinger, David Wilkins, Stephen Levinson, Sotaro Kita and Gunter Senft 1998 Semantic typology and spatial conceptualization. Language 74: 557589. Pfeifer, Rolf and Christian Scheier 1999 Understanding Intelligence. Cambridge, MA: MIT Press. Posner, Michael and Marcus Raichle 1994 Images ofMind. New York: Scientific American. Pylyshyn, Zenon W. 1973 What the mind's eye tells the mind's brain: a critique of mental imagery. Psychological Bulletin 80: 1-24. Regier, Terry 1992 The acquisition of lexical semantics for spatial terms: A connectionist model of perceptual categorization. PhD. Dissertation, University of California at Berkeley. 1995 The Human Semantic Potential. Chicago: University of Chicago Press. Rizzolatti, Giacomo and Laila Craighero 2004 The mirror neuron system. Annual Review ofNeuroscience 27: 169192. Rohrer, Tim When metaphors bewitch, analogies illustrate and logic fails: Con1998 troversies over the use of metaphoric reasoning in philosophy and science. PhD. dissertation, University of Oregon. 2001 a Pragmatism, Ideology and Embodiment: William James and the philosophical foundations of cognitive linguistics. In: Rene Dirven, Bruce Hawkins and Esra Sandikcioglu (eds.), Language and Ideology: Cognitive Theoretic Approaches. Volume 1, 49-81. Amsterdam: John Benjamins. 2001 b Understanding through the Body: fMRI and of ERP studies of metaphoric and literal language. Paper presented at the 7th International Cognitive Linguistics Association Conference, July 2001.

The body in space: Dimensions ofembodiment

2005

377

Image schemata in the brain. In: Beate Hampe (ed), From Perception to Meaning: Image Schemas in Cognitive Linguistics, 165-196. Berlin: Mouton de Gruyter. Schwoebel, John, Robert Friedman, Nanci Duda and H. Branch Coslett 2001 Pain and the body schema evidence for peripheral effects on mental representations of movement. Brain 124: 2098-2104. Shepard, Roger N. and Jacqueline Metzler 1971 Mental rotation of three-dimensional objects. Science 171: 701-703. Shulman, Gordon L., Roger W. Remington and John P. McLean 1979 Moving Attention Through Visual Space. Journal of Experimental Psychology: Human Perception and Performance 5: 522-526. Sinha, Chris and Kristine Jensen de Lopez 2000 Culture and the embodiment of spatial cognition. Cognitive Linguistics 11: 17-41. Tsal, Yehoshua 1983 Movements of attention across the visual field. Journal of Experimental Psychology: Human Perception and Performance 9: 523530. Turing, Alan M. 1950 Computing machinery and intelligence. Mind 59: 433-460. Vingerhoets, Guy, Floris P. de Lange, Pieter Vandamaele, Karel Deblaere and Erik Achten 2002 Motor Imagery in Mental Rotation: An fMRI study. Neuroimage 17: 1623-1633. Wilson, Margaret 2002 Six views of embodied cognition. The Psychological Bulletin and Review 9: 625-636. Wraga, Maryjane, Sarah H. Creem, and Dennis R. Proffitt 1999 The influence of spatial reference frames on imagined object- and viewer rotations. Acta Psych 0 logica 102: 247-264. Zlatev, Jordan Situated Embodiment: Studies in the Emergence ofSpatial Meaning. 1997 . Stockholm: Gotab Press. 2003 Polysemy or generality? Mu. In: Hubert Cuyckens, Rene Dirven and John Taylor (eds.), Cognitive approaches to lexical semantics, 447494. Berlin: Mouton de Gruyter.

On the biosemiotics of embodiment and our human cyborg nature*

Claus Emmeche "my faculty of discussion is equally localized in my inkstand" Peirce 1

Abstract This chapter briefly introduces biosemiotics, a new perspective on living systems based upon standard contemporary biology reinterpreted through a qualitative organicist tradition in biology inspired by Jakob von Uexkiill and Charles S. Peirce. To be emphasized is the difference between a living organism as a general semiotic system with vegetative and self-reproductive capacities ("the body of biology") and an animal body also with sentience and phenomenal states. The "representationalism" invoked by critiques of cognitive science tends to focus only on simplistic notions of representations. However, these must be distinguished from a Peircean notion of representation as an embedded ongoing process. The implications of this distinction for theorizing about the physical, biological, animate, phenomenal and social body will be made. It will be argued that these very dimensions of embodiment must be accounted for coherently and seen within an evolutionary emergentist perspective. Such a perspective requires a "Non-standard Neo-Aristotelian Pluralist notion of Causation" (NNPC), and I will argue that biosemiotics offers an approach to animate embodiment compatible with NNPC. However, biosemiotics is not enough to account for the characteristics of human embodiment because of what shall be called our intrinsic cyborg nature as techno-culturally embedded beings within a space of meanings that are not only symbolic, but argumentative and socially empowered by different kinds of sociocultural systems.

* The author is indebted to Frederik Stjernfelt, Simo Keppe, Jesper Hoffmeyer, Charbel Niiio EI-Hani and Joao Queiroz for valuable discussions, and to Tom Ziemke and an anonymous reviewer for critical comments on earlier versions of this paper. 1. Peirce ([1931-1958] 1902, CP 7.366)

380

Claus Emmeche

Keywords: animal, biosemiotics, causality, cyborg, embodiment, emergence, organicism, representation, society.

1.

Introduction

The desire we have as persons for knowledge about the body is often redirected to science. If the body is a living organism, the science of the body should be biology, as one might envision this "science of life" as being able to account for the nature of human bodies and the embodiment of human existence. It was only some time after having started training as a biologist that it became clear to me that biology cannot really be a science of living bodies. It is a science of organisms and their parts and relations within systems of organisms and environments, and these two domains of inquiry, living bodies and organisms, are not co-extensive. Intuitively I had held the idea that biology because of its generality simply subsumed humans as a species, just as physics subsumed biological processes in the sense that any biological theory of these processes has to conform to physical laws. Embodiment had to be seen as a biological phenomenon, I thought. This stance was naIve for two reasons, an epistemic one and an ontological one. The epistemic one is that each science in an important sense actively constitutes is own paradigmatic objects of investigation, and thus body and embodiment in biology is different from body and embodiment in anthropology. In fact, embodiment is usually neither considered a technical term nor a theoretical issue within biology. The ontological reason why embodiment (how to be a body, have a body, use and express one's body2) is not simply a biological phenomenon is that the human body in every specific concrete context is a far more complex and multi-Ievel3 phenomenon than any single scientific perspective can account for. 2. Often expressions in everyday language exemplifying the non-identity between the notions of self, person and body (like "I moved my body"), appear as if presupposing a metaphysical mind/body dualism, leading to questions like "Do you use your body to express yourself, or is your body and your self the same expressive entity?". As Wittgenstein remarked, we are often bewitched by language. 3. As we shall see later, "multi-level" means including at least a physical, a general biological-organismic, a specific animate-zoological, and a human-sociocultural level. There is a one-way relation of inclusion between the levels (cf. the notion of a "specification hierarchy" in Salthe 1989). The biological level includes the

On the biosemiotics ofembodiment and our human cyborg nature

381

To understand the full implications of embodiment, one has to grasp at least the distinction between the concept of an organism and the concept of a body. This distinction matches the distinction between biology and sociology (roughly speaking, leaving out animals for a moment). Of course, being an organism is a precondition for having a living body which is neither a thing not a corpse. We could not, as humans, be embodied - that is, be material creatures co-evolving with an existential-phenomenal world of situated activity involving emotional experiences, feelings, cognitive processes, perception and action, a specific perspective, placed in historical and biographic time regimens - if we were not rooted in an organic world by our very being, that is, if we were not biological creatures. 4 Yet in significant ways we transcend our biological form of existence by producing, through culture, language and social institutions, specific dynamic modes or patterns in which our organic and animate existence is realized. Here we shall try to bring together some perspectives upon the body that may help increase theoretical sensitivity to our (minimally double) existence as belonging both to a first biological nature bound by animate and metabolic processes, and a second social and semioticS nature in which the significance of the organism's forms of movement can only be comprehended if

zoological - being an animal implies being an organism - but not the other way around. However, even though humans are animals, zoology as a science is not adequate and too unspecific to acount for all there is to say about the cultural life of Homo sapiens, a being socialized in a human society. 4. Some AI researchers, cognitive psychologists and philosophers might consider that bio-chauvinism if the intended implication is the impossibility of "intrinsic" machine intentionality. However, machines should not be reified as something outside a human sociocultural context. Machine intentionality might be possible, but the action of signs and interpretation (i.e., semiosis) in machines must be considered as tightly bound to their sociocultural context. Machines are extensions of human capacities and intrinsic parts of human sociocultural systems; they do not originate ex nihilo or fall readymade from heaven (Marx 1867; Ziemke and Sharkey 2001). 5. "Social" can be said to mean concerning human actions within a community (as studied by sociology, the science of social processes). "Semiotic" is here taken to mean concerning sign production, information transmission, communication and interpretation within some system (as studied by semiotics, the general science of sign action and interporetation, as founded by the philosopher and scientist C. S. Peirce). The word nature has several meanings but here I rely on my reader as a competent language user.

382

Claus Emmeche

seen as formed by the norms, social roles and institutions of a human society. However, we shall circumvent the all too common dualism of seeing these modes as exclusive, or seeing, as often expressed within the humanities, meaning and significance as an exclusively human social phenomenon, implying non-human nature to be either merely an abstraction or a boundary concept denoting forms of materiality devoid of any intrinsic meaning or significance. First, the modes of existence of embodiment are inclusive; briefly, the human body includes the biological organism. 6 Secondly, the organism as our "first" biological nature is not something nonsemiotic. On the contrary, the roots of processes involving communication, information, signs and interpretation (in one word: semiosis) - and more specifically, human forms of sign action and interpretation (anthroposemiosis) - must be found within an evolutionary framework. This should not, however, simply be standard neo-Darwinism (as this paradigm during the history of twentieth century biology unfortunately has become dominated by a physicalist metaphysics), but supplemented with a biosemiotic perspective. In Section 2 I shall briefly introduce biosemiotics as one useful resource to sketch a framework for an overall understanding of embodiment in its material, processual, externalist as well as internalist aspects. A brief comment on the current interest in the phenomena of embodiment may be adequate to discern the specific message ofbiosemiotics. The concern with embodiment has several sources: (1) the study of human cognition, especially criticisms of the classical functionalism of traditional cognitive science and AI, alternatives like "enactive cognitive science" (Varela, Thompson and Rosch 1991) or ideas of "situated and embodied robots" Brook 2002), which have led to a radical questioning of the need for representations in cognitive science (Riegler, Peschl and Stein 1999); (2) the study of human language, especially critiques of modeling language as a disembodied abstract module, and the alternative movement in cognitive semantics that sees basic metaphoric forms of language as rooted in the organization of human bodily action, and claims these unconscious metaphors are based on common bodily experiences (Lakoff and Johnson 1999); and (3), a widespread interest, within the human sciences and interdisciplinary cultural studies, in the transformation of the body, in which

6. See also note 3. This notion ofinclusivity (see below) was developed in relation to emergent levels of reality in Emmeche, Keppe and Stjernfelt (1997).

On the biosemiotics ofembodiment and our human cyborg nature

383

"the new body" is seen no longer as a "fixed, material entity subject to the empirical rules of biological science" (Csordas 1996). Instead, the "new body" is seen as a body with a history, behaving "in new ways at particular historical moments" (ibid.), characterized by indeterminacy, flux, selfcreating processes. This has lead to a deep questioning of a set of supposedly basic boundaries, namely the boundaries of corporeality itself, and the boundaries between physical and non-physical, animal and human, and between organic bodies and machines (cf. Hayles 1999). Let me state preliminarily and in simple terms what biosemiotics is about: It is an attempt to study life not only by chemical approaches seeing cells and organisms as assemblies of molecules, but from the perspective of semiotics (cf. footnote 5) seeing those same molecules (correctly described by chemistry and molecular biology) as sign vehicles for information and interpretation processes. One main concern of biosemiotics is thus to see the phenomena of significance, meaning and interpretation in the human sphere as rooted in and to some extent continuous with the same very general kind of phenomena in the non-human sphere. We can take the words of the Peirce scholar T. L. Short (1982: 298) to be emblematic of this evolutionary yet nonreductionist perspective ofbiosemiotics: Of course human speech is unique: it consists of the ability to replicate legisigns in ever new patterns, as well as in the ability to create new legisigns, not by the slow process of evolution, but within the lifetime of individuals, and by their own volition. The point, rather, is that the distinctive power of human speech is not a supernatural gift, but is a remarkable development of basic principles found elsewhere in nature."

From a biosemiotic point of view, more than one approach to embodiment is needed for the above intellectual movements (1) and (2) may lead too hastily to radical anti-representationalist stances, while (3), within the posthuman or postmodern discourse, they may tend simply to dissolve real boundaries between the categories of body/machine, human/animal, or nature/culture, instead of analyzing their dynamics, inclusivity relations, or how they are being contested. Biosemiotics is critical about these implications; it builds on a more adequate notion of representation (namely that of Peirce; see below) than classical cognitive science; and it allows for keeping fundamental distinctions while seeing these in a dynamic evolutionary perspective which involves a modest dialectics of continuity and emergence. Yet I will also claim that as a scientific perspective, biosemiotics has a limited area of validity, corresponding more or less to the subject

384

Claus Emmeche

matter of contemporary biology, and it cannot fully capture the specificities of human embodiment. The idea to be presented is in some respects a very simple one of emergent levels of embodiment: A human body - e.g., the body of a child, a soccer player, or a diplomat - includes (in the sense specified in footnote 3) an organism, but is also something more, transcending the mere set of organismic properties (like metabolism, growth, homeostasis, reproduction), just as, if we go down one step to a lower level of integration, an organism is a physical system, yet it transcends the basic physics of that system. "Transcending" here does not imply some transcendentalism or metaphysical dualism. The expression "Z transcends Y" has two aspects. (a) One aspect is epistemic, i.e., "Z's description cannot adequately be given in terms of a theory generally accounting for Y, even though this Zdescription in no way contradicts a description of the Y-aspects of Z". (b) The other aspect is ontological, i.e., "crucial properties and processes of Z are of a completely different category than the ones of Y, even though they may presuppose and depend on Y". 7 The organism is a material, physical processual entity with a form of movement so specific that physics alone (as a science) cannot account for that entity. The organism is a very special type of physical being, as it includes certain purposeful (functional) partwhole relations, based upon genuine sign systems of which the genetic code is the most well-known but not the only example. It is time to insert a one-paragraph crash-course in semiotics (cf. Peirce [1931-1958] 1902): A sign is anything that can stand for something (an object) to some interpreting system (e.g., a cell, an animal, a legal court), where "standing for" means "mediating a significant effect" (called the interpretant) upon that system. Thus, semiosis, or sign action, always involves an irreducibly triadic process between sign, object and interpretant. Just as in chemistry we see the world from the perspective of molecules, in semiotics (as a general logic of sign action) we see the world from the perspective of sign action, process, mediation, purposefulness, interpretation, generality. Those are not reducible to a dyadic mode of mechanical actionreaction, or merely efficient causality.8 Thus, organisms are certainly com7. Both emergence and variants of supervenience have been suggested as candidates for an ontological dependence relation, but we need not enter the debate on these technicalities. 8. The details of this argument are more complex in the sense that the distinction fmallefficient causation and organismic/physical processes are not co-extensive, and thus, the irreducibility of fmal (teleologic) causation to efficient causation

On the biosemiotics ofembodiment and our human cyborg nature

385

posed of molecules, but these should be seen as sign vehicles having functional roles in mediating sign action of, e.g., genotype and environment, upon the phenotype. Biological organization is emergent upon physical order, i.e., the organism includes its physical processes (for instance, one can break a leg) but from the point of view of physics their processual forms are radically new, unpredictable and realizing processes (e.g., self-reproduction) that are not strictly deducible from properties found in the non-biological realm. However, discussing embodiment, one cannot limit oneself to the gross primary levels of reality including the physical, the biological and the social level of embodiment, one has to do a more fine-grained analysis of various biological forms of organismic existence before we can make sense of a distinction between biological and human embodiment. To this end biosemiotics is valuable although it cannot alone account for human embodiment.

2.

Biosemiotics - a qualitative organicist account of embodiment

Biosemiotics is a growing field that studies the production, action and interpretation of very different types of signs (such as sounds, objects, smells, movements, but also signs on molecular scales normally not perceived by an organism) in the physical and biological realm, in an attempt to integrate the findings of biology and semiotics. One goal ofbiosemiotics is to form a new view of life and meaning as immanent features of the natural world, rather than as epiphenomena, however complex they might be. Early pioneers ofbiosemiotics include Charles S. Peirce (1839-1914), Jakob von Uexkiill (1864-1944), Charles Morris (1901-1979), Thure von Uexkiill (b. 1908), Heini Hediger (1908-1992), Thomas A. Sebeok (19202001), and Giorgio Prodi (1928-1987). Contemporary scholars who have contributed to the field or commented extensively include, among others, biologists Jesper Hoffmeyer, Kalevi Kull, Alexei Sharov, Anton Markos, Marcello Barbieri and Seren Brier, and semioticians Walther A. Koch, Floyd Merrell, John Deely, Frederik Stjemfelt, Winfried N6th and Lucia Santaella (for a historical introduction, see KuII1999).

(sensu Peirce) is not necessarily the same as the irreducibility of organismic processes to physical processes, as for Peirce, every lawful process has an element offmality (see Hulswitt 2002).

386

Claus Emmeche

One of the central characteristics of living systems is the highly organized nature of their physical and chemical processes. These processes are based, in part, on the informational and molecular properties of what came to be known in the 1960s as the genetic code. Some biologists, such as Emst Mayr (1982), have viewed these program-like properties of the genetic code as distinguishing life from anything else in the physical world, except computers. However, although the informational teleology (or goaldirectedness based upon a stored informational code) of a computer program is not an original form of teleology because the program is designed by humans to achieve specific goals, the teleology and informational characteristics of organisms are intrinsic, as they evolved naturally through self-organized evolutionary processes. Traditional biology, at least in popular and text book versions, and a large part of mainstream philosophy of biology have regarded such informational processes as in the end purely physical; unfortunately also adopting a restricted notion of the physical as having to do with only efficient causation. Thus these traditions have often aimed at a "naturalization" of teleology (of, e.g., genetic information) and intentionality (of cognitive information). The admirable purpose of this is making these phenomena continuous with the rest of nature, and to account for them in a manner acceptable to physical science. Contemporary science, however, is often prejudiced as the only legitimate basis of judgments on the nature of Nature. Taking the animal or human mind and its teleological and intentional processes to be embodied is often perceived, within biology and cognitive science, as implying a search for reductive explanations of mental phenomena in terms of, e.g., neural information processing, or dynamical systems approaches continuous with the kind of explanations given for non-linear complex systems within physics. This is the case even for some of the search for other notions of embodiment (e.g., "radical embodiment", Clark 1999). In many programmatic naturalization strategies, the "natural" seems to be reduced beforehand to restricted metaphysical versions of nature as only involving certain kinds of properties or causal processes, like, as said, classical efficient causation. 9 Drawing upon the insights of Peirce who founded semiotics as a logic and scientific study of dynamic sign action in human and non-human nature, biosemiotics attempts to use semiotic concepts to investigate ques-

9. Partly based on mathematical arguments Scott (2004) convincingly shows that a reductionist notion of causality is not even defensible for many non-linear complex systems in physics.

On the biosemiotics ofembodiment and our human cyborg nature

387

tions about the biological and evolutionary emergence of meaning, intentionality and a psychic IO world which are difficult if not impossible to answer within a mechanicist or physicalist framework. II Biosemiotics see the evolution of life and the evolution of semiotic systems as two aspects of the same process. The main tenet of biosemiotics is the belief that the scientific approach to the origin and evolution of life has given us highly valuable accounts of the external aspects of these processes, but has overlooked the "inner", qualitative and significant aspects of sign action, thus leading to a reduced picture of causality (cf. Santaella Braga 1999). Complex self-organized living systems are governed by formal and final causality (Van de Vijver, Salthe and Delpos 1998). They are mereologicallyI2 governed by formal causality in the sense of a non-temporal "downward causation" (Andersen et al. 2000) "from" a whole structure (such as the organism) "to" its individual molecules, constraining their action but also endowing them with functional meanings in relation to the whole metabolism. They are governed by final causality in the sense of their tendency to take habits (find new attractors in phase space) and to generate future interpretants of present sign actions. In this sense, biosemiotics draws upon insights of fields like systems theory, theoretical biology and the physics of complex self-organized systems. I3 Particular scientific fields like molecular biology, cognitive ethology, cognitive science, robotics and neurobiology deal with information processes at various levels and thus - in that minimal sense of biosemiosis as informational processes in living organisms - spontaneously contribute to knowledge about biosemiosis and living sign action, even though these

10. Allow me to use "psychic" instead of "mental", as the former word has less Cartesian connotations than the latter and is rooted in a realist, neo-Aristotelian and Peircean philosophical tradition. Intentionality is used here not according to Brentano's dualist metaphysics but in a way inspired by Peirce to designate both the representational "aboutness"-aspect of animate sign processes, and their qualitative, psychic, phenomenal aspects. 11. This applies equally for non-reductive physicalism within philosophy of mind. 12. Mereology, the study of parts and wholes, usually refers to a mathematical or at least formal theory thereof, such as that of Lesniewski or Goodman (cf. Varzi 2004), developed by the former in the hope of forming an alternative to set theory as a foundation for mathematics. 13. See Hoffmeyer (1996), Emmeche, Kull and Stjernfelt (2002), Hulswitt (2002), Emmeche (2004a), EI-Hani, Queiroz and Emmeche (2005). (More on causality, see also section 5 below).

388

Claus Emmeche

findings are not framed within a theory of biosemiotics. However, biosemiotics is not so much a specific disciplinary research programme as a general scientific perspective on living systems. It attempts to integrate and re-interpret empirical findings, and to build a new foundation for biology that acknowledges the genetic "information talk" of codes, messenger molecules, translation etc., as being not simply problematic metaphors but a symptom of real organic semiosis, emphasizing that a notion of "information processing" can be provided, that is theoretically richer than standard notions of information processing in contemporary cognitive science (cf. Lindblom and Ziemke, this volume). What is this alternative notion of information and representation? According to biosemiotics, the cognitive system is not a dual system of hardware (a physical brain) and software (an algorithmic symbolic system), and similarly the organism is not a dual system of "information and flesh", with a DNA code as governor of all bodily processes. Sign processes are active components of the overall physiology of the organism, and accordingly everything which may become "a difference that makes a difference" to the organism (to use Bateson's cybernetic definition of information; Bateson 1972) is acting as a sign. Just as the mind is not a computer program, DNA is not the central governor of the body, and genes are not preformed descriptions of phenotypic characters with a privileged causal status in every cell, as often assumed in standard genetics and molecular biology. It is possible to use Peircean semiotics to analyse sign processes not only in protein synthesis but in any significant process within and between cells. Though not identical, the notions of sign action sensu Peirce and of information sensu Bateson are intersecting and not conflicting, if interpreted correctly: Both notions are generally denoting processes of production, transmission and interpretation of something of significance for some interpreting system (like a body, an organism, a cell), and the important aspects of both notions are relational and semiotic. If information is a difference that makes a difference (implied is "to an organism" or "to some interpreting system"), then by Peircean terminology this can be restated more precisely in a semiotic definition of information as a triadic process we may formulate as follows: Information is a process in which a sign (representamen), simple or composite, makes a difference (interpretant) to some system (the interpreter) by making the interpretant (the effect of the sign as a difference) stand in a similar relation to something else (the object of the sign) as that to which the representamen stands or refers, thus mediating object and

On the biosemiotics ofembodiment and our human cyborg nature

389

interpretant, i.e., conveying to the interpretant a dynamic form that signifies the object. 14 This definition can be exemplified on the level of both molecular biology and cognitive ethology: Fine-grained analysis of genes and protein synthesis as semiotic processes can be made (EI-Hani, Queiroz and Emmeche 2005) as well as analyses of, for instance, the alarm system in vervet monkeys (Queiroz and Ribeiro 2002). Here one has to remember the very general character of the formal sign definition in Peirce, of which the proposed definition is a variant that emphasizes the processual character. That the meaning of the sign is its causal effects (i.e., the interpretant) implies that these effects have to be physically, biologically, or culturally embodied in an interpreting system (or in the dialogic process between such systems; see Petrilli 1999) to the extent that we are dealing with material systems of a physical, biological or cultural kind. In philosophical jargon, meaning is supervenient on the physics of the system, not reducible to physics, yet dependent upon a material basis. Furthermore, biosemiotics helps to resolve some remnants of Cartesian dualism that are still haunting philosophers and scientists. It provides an alternative to the implicit Cartesianist metaphysics in evolutionary biology that tend to reduce organisms to non-sentient mechanical beings, a reduction that makes it almost impossible to conceive of a satisfying evolutionary account of the qualitative aspects of consciousness (Emmeche 2004a). By subscribing to a process rather than substance metaphysics, and by describing the continuity between matter and mind, biosemiotics may also help to understand higher embodied forms of mind and the evolutionary roots of cultural phenomena. In a historical account of "embodiment", Keller (2003) comments upon the tendency of science writers and journalists to articulate both a quasireligious awe when facing the structure of the genetic code or the human genome, seeing it as "the word becoming flesh", but also repeating, by such expressions, a dualist framework that has been operative since the foundation of modem science: "The body as a machine, as a closed system, 14. Cf. one ofPeirce's own defmitions: "A Sign, or Representamen, is a First which stands in such a genuine triadic relation to a Second, called its Object, as to be capable of determining a Third, called its Interpretant, to assume the same triadic relation to its Object in which it stands itself to the same Object." ([19311958] 1902, CP 2.274), emphasis in the original. Peirce distinguished between several kinds of signs, objects, and interpretants, which may be used to develop further the informational variant of the sign defmition.

390

Claus Emmeche

to which spirit, mind and God, if they exist, were posited as external agents, dominated the western imagination until the twentieth century" (ibid., 255). Even though Keller locates this new "gospel of genetics" in a culture-historical trend towards more inclusive senses of embodiment (deriving from not only the process theology of Alfred North Whitehead, but also from feminist and ecological movements) and thus sees it as contributing to re-opening the closed system of the body as a machine, these very rhetorical figures of popular genetics tend to reinforce the dualisms between mind and body; information and flesh. Here, biosemiotics may function as one of the resources for critical study of mind/body dualism in Western science and religion. The biosemiotic account of embodiment is a form of organicicm, an important tradition in biological thought. The resolution of the debate between vitalism and mechanicism in the history of biology looks on the surface like a victory for mechanicism, but was really a compromise, a sort of mainstream organicism, exemplified by the writings of such well-mown biologists as J. Needham, P. Weiss, C. H. Waddington, J. Woodger, E. Mayr, R. C. Lewontin, R. Levins and S. J. Gould, (see also Gilbert and Sarkar 2000), and functioning more or less tacitly as a background philosophy of biology (Emmeche 2001). Organicism takes the complexity and physical uniqueness of the organism as a symptom of the distinctiveness of biology as a natural science sui generis. As a compromise, although often framed within a naturalist evolutionary perspective, it was anticipated by Kant's more critical (non-naturalist) notion of a living organism as an organized product of nature in which every part is reciprocally purpose (end) . and means (cf. Van de Vijver, Van Speybroeck and Vandevyvere 2003). However, within mainstream organicism this teleology is interpreted as a more or less "mechanical" teleonomy being the result of the forces of blind variation and natural selection, plus eventually some additional "order for free" or physical self-organization (Kauffman 2000). Mainstream organicism is thus non-vitalist, ontologically non-reductionist (methodologically allowing for reductionist research strategies though) and emergentist. What is studied as emergent properties are common material structures and processes within several levels of living systems (developmental systems, evolution, self-organizing properties etc.), all of which are treated in the usual way as objects with no intrinsic experiential properties. In contrast, qualitative organicism represents a more colored view on living beings; it emphasizes not only the ontological reality of biological higher level properties or entities (such as systems of self-reproducing

On the biosemiotics ofembodiment and our human cyborg nature

391

organisms being parts of the species' historical lineage) but also the existence of phenomenological or qualitative aspects of at least some higher level properties. When sensing light or colors, an organism is not merely performing a detection of some external signals which then are processed internally (described in terms of neurochemistry or "information processing"); something additional is part of the story, at least if we want the full story, namely the organism's own experience of the light, and this experience is seen as something very real. 15 As a scientific position qualitative organicism is concerned with qualities which are not only of the famous category of "primary" qualities (roughly corresponding to the scientifically measurable quanta) including shape, magnitude and number; but also concerned with the "secondary" qualities of color, taste, sound, feeling, etc. Neither qualitative nor mainstream organicism are fully coherent stances, theories or paradigms; many authors cannot be consistently categorized as belonging to either one or the other position exclusively; the important thing is to recognize that in fact two different conceptions of life are at stake. To the extent that biosemiotics is based in a Peircean logic of signs of varying levels of complexity, it is a form of qualitative organicism. We leave it to others to form a taxonomy of the variants of qualitative organicism. Here we will concentrate on its biosemiotic brand instead. Biosemiotics recognizes not only the external effects of signs upon their interpreters, forming new interpretants (remember that the interpretant, being an effect of the sign, need not be the whole interpreter organism), but also the internalist and phenomenal aspects of this process. The phenomenology of basic biological ("vegetative") processes of life is not yet developed, and we shall primarily focus on the animal level of biosemiosis. A living being that is not simply an organism but also an animal, which is emphatically "animated", that is, it realizes a specific form of movement which is not reducible to pure physical change or basic biologic processes (like metabolism). Movement, understood as living change and as semiotic, when realized by self-moving organisms (i.e., animals), crucially depends on mediating all the elements of quality, physical force and interpretation three general ontological aspects of every semiotic process that correspond to the triadic structure of categories in Peirce. Single cells, plants and animals are all, so to speak, semiotic machines (the machine metaphor is not

15. This is the case in Jakob von Uexkiill's biology (von Uexkiill1982) as well as in Peirce's synechism, i.e., his philosophy of mind/matter continuity (Santaella Braga 1999).

392

Claus Emmeche

implying any mechanicism here, cf. Noth 2003), nodes in complex webs of sign interpretation processes. Every process,I6 even in non-living nature, has an aspect of (i) "tone" (i.e., possibility, chance and phenomenal quality), (ii) "token" (existence, force, here-and-now), and (iii) "type" (generality, being governed by a general law). This is true for sign processes as well. The characteristic level of sign interpretation processes in plants are intra- and inter-cellular, with molecular signs representing the immediate environment and longterm morphogenetic changes (processes like metabolism, growth, differentiation and reproduction). A plant's body is the result of an organism's vegetative interpreting systems, and irritability is a crucial component of vegetative embodiment. I? Animate embodiment is different. Although it includes the vegetative form of embodiment (thus an animal's skin may indeed get irritated, its growth may be disturbed, etc.), it is a form of embodiment where self-movement is a crucial new development of the general biological self-control one finds in plants and single cells. In selfmovement, animal agency emerges as a temporary structure of bodily sign action processes. Animate agency is semiotic, meaning that it mediates the three dimensions of (i) a qualitative phenomenal feeling, i.e., the "inner" side of the sign considered not in relation to anything else (thus, a form of Firstness in Peirce); (ii) the here-and-nowness of signs or stimuli (the "haecceity" of a stimulus being an example of Secondness) that can have an environmental or bodily proprioceptive origin, but whose being is existent and thus simply consists in their immediate effects, action-reaction, their "actually acting and being acted on" (Peirce [1931-1958] 1902, CP. 6.318); and (iii) the cybernetically organized cycles of organism16. Peirce was not satisfied with a standard type/token distinction which only operates with Thirdness (type, a form of generality) and Secondness (token, existence), so he added "tone" (quality) to represent Firstness. It is not possible in brief space to explain his three basic categories, so I only can offer my reader these hints (see also Hausman 1993; Hulswitt 2002; Peirce [1931-1958] 1902). 17. Irritability is used here in the sense of a general semiotic capacity to feel and interpret stimuli so as to adapt to changes in the environment or in other parts of the organism. Compare the term's original denotation, viz. the capacity of certain parts of the body to contract when stimulated, as introduced by the English physician Francis Glisson (circa 1597-1677) who saw it as a property of all the body's fibres independent of consciousness and the nervous system (cf. Lawrence 1981). It has played an important role in debate between mechanicists and vitalists over the basic defInition of life.

On the biosemiotics ofembodiment and our human cyborg nature

393

environment interaction, the "functional cycle" of Jakob von Uexkiill's biology, describing how the environmental stimuli are integrated and interpreted in the Umwelt 18 of the organism - this integrative mediating process involves emotional and proprioceptive interpretation systems that mediate the past and present Umwelt into anticipatory schemes for further action and habit-formation (i.e. the mediative and future-directed category of Thirdness). Animal embodiment, accordingly, is the graceful integration, accomplished by a living animal body, of physical change, biologicmetabolic processes (neural as well as general physiological) and the emerging interpretative Umwelt relation (of a body within its specific perceived and acted upon environment) representing the animal's own detailed movements in an ever-changing flux of environmental and bodily stimuli.

3.

Triadic representations vs. representationalism

It is important to note that a biosemiotic notion of animal embodied cognition is not committed to the same kind of functionalist representationalism often criticized by "new AI" or cognitive semantics, i.e., the notion that cognition and language consist of rule-governed manipulation of a distinct set of symbolic representations inside an organism referring to an outside world (cf. Riegler and Stein 1999; Ziemke and Sharkey 2001). The representationalism of classical cognitive science, like the physical symbol system hypothesis of functionalism, has a dyadic notion of representation (tending to disregard the open-ended and processual character of interpretation) and an atomistic concept of symbol, in strong contrast to the triadic, general and evolutionary nature of sign processes and causality involved in a Peircean conception of representation. On the level of general semiotics, signs are forms of mediation (i.e., generals embodied as relational tokens) that are embedded within a network of local processes of ongoing generation of new interpretants; this applies to cellular "vegetal" signs as well as 18. The Umwelt is the species-specific subjective universe, or phenomenal world, of an organism; the part of the environment of a subject that it selects with its species-specific sense organs according to its organization and biological needs. Everything in the Umwelt is labeled with perceptual cues and effector cues of the subject. According to J. von Uexkiill, every subject is the construtor of its Umwelt; cf. J. von Uexkiill 1940 [1982], with a good introduction by his son Thure von Uexkiill (1982). See also the volume edited by Kull (2001).

394

Claus Emmeche

cognitive "animate" ones. Embodiment in this basic sense is a sine qua non for all sign action in nature and culture, and here, the term "embody" designates a general metaphysical aspect of semiosis: A sign must, so to speak, materially incarnate its meaning. 19 It is on this level of generality, we might call it semiotic embodiment, one must answer the question whether embodiment is necessary or sufficient for meaning or mind: From the point of view of semiotics, meaning and mind are all-pervasive phenomena; meaning always has to be embodied in signs, but need not be confined to organic bodies. 20 Although Peirce's own semiotics is embedded within his broader views on the nature of reality, evolution and thought, contemporary Peircean semiotics and biosemiotics need not subscribe to the total ofPeirce's fascinating blend of pragmaticist philosophy of science, evolutionary cosmology and metaphysics, although these inform our understanding of his notion of logic as tightly connected to semiotics. What is real is not merely what is existing, because the existing "here and now" of individual events (their Secondness) are always, on the one hand, embedded within the generalities of law, tendencies, habit formation and thought, that is, the general logic of things (the category of mediation, Thirdness), and on the other hand, influenced by genuine chance, random deviations from lawfulness, spontaneous generation of new possibilities (belonging to the category of Firstness). Thus, any existing sign, such as this word "hand", is only understandable as a sign of something to some interpreter, by being a token (Peirce called it a sinsign) of some general type of sign (a legisign), namely the kind of word just mentioned. As Santaella states, "The legisign depends on individual cases to be actualized. The symbolic legisign" - here, the word hand - "is embodied in individual cases. In this same act of embodi19. A physical precondition for signification, so to say. Sign action is mediation, and as such requires a material medium for this process (as well as energy, at least under normal thermodynamic conditions; it is well-known that information processing has physical limits and cost energy, cf. Zurek 1989) 20. "Thought is not necessarily connected with a brain. It appears in the work of bees, of crystals, and throughout the purely physical world; and one can no more deny that it is really there, than that the colors, the shapes, etc., of objects are really there. Not only is thought in the organic world, but it develops there. But as there cannot be a General without Instances embodying it, so there cannot be thought without Signs. We must here give 'Sign' a very wide sense, no doubt, but not too wide a sense to come within our defmition. (Peirce [19311958] 1902, CP 4.551).

On the biosemiotics ofembodiment and our human cyborg nature

395

ment the individual cases are conformed to the symbol's domain. The symbol functions as a rule for the formation of a certain sub class of sinsigns which are called replicas. The rules for the formation of the replicas also involve the interpretative rules of these replicas. Hence, the replica of a symbol is a special kind of index which acts to apply the general rule or habit of action or expectance associated to the symbol to a something particular" (Santaella 2003; cf. Short 1988). Also symbolic representations are embedded ongoing processes, and thus we see that the notion of symbol in semiotics is very different from the notion of symbol within classical cognitive science. Regarding the specific human form of embodiment, the symbol attains a specific role accounting for the social nature of human consciousness.

4.

The body within evolutionary emergent levels

As we have seen, the animate body as a self-moving organism transcends the simpler category of an organism realizing the vegetative functions of growth and reproduction. Already at this stage biosemiotics provides not only an alternative (non-physicalist) "naturalization" of semiotic processes in general, but also a framework for distinguishing between levels of embodiment corresponding to different degrees of complexity involved in sign interpretation processes. To clarify the structure of the present argument for a multiplicity of phenomena of embodiment, a simple schematic list of levels of embodiment may help. The full meaning of the scheme's concepts will be explained in the following. Here is the scheme (adapted from Emmeche 2002a): 1) 2)

3) 4)

The body of physics: Dissipative self-organizing structures. Thermoteleology. The body of biology: The organism as a vegetative, physiologichomeostatic self-organizing structure. Bio-functionality and irritability. The body of zoology: The animal as an autonomous, self-moving organism. Intentionality and consciousness. (a) The body of anthropology: The human body as a signifying animal, incarnating a socio-cultural specific life-world. Desire, histronics and conscience. (b) The body of sociology: A "cybody", i.e.,

396

Claus Emmeche

a societal body dependent upon technology, embedded in a civilization. Cosmopolitics, hybridicity, posthumanity. The scheme indicates ordering relations between some forms of embodiment. Their epistemic dimension is mentioned first, by organizing those forms according to different domains of science each constituting its own objects; their ontic dimension is implied by an underlying ontology of levels of organization in Nature. Four such levels are mentioned. The point is not the exact number of levels (these are contingent upon a historically relative state of science) but the fact that irreducible levels do exist. E.g., the animal as a phenomenon includes an organism, yet the animal form of embodiment transcends (in the sense explained above) the vegetative form of embodiment; that of a cell or a plant. No distinction is made between embeddedness and situatedness, but between their vegetative, animate and sociocultural forms: Organismic embodiment: The organism is situated in its ecological nice. Animate embodiment: The animal is additionally embedded in a complementary subjective Umwelt (see below). Anthropic embodiment: A sociocultural Umwelt constitutes a lifeworld enacting social roles and expectations (histrionics) as well as conscience. Thus, different notions of embodiment are partly constituted by different areas of science, yet from a point of view of "semiotic realism", science can indeed capture true aspects of "the joints of nature". Those joints are inclusive, in the sense that being a blackbird includes being a bird, which in turn includes being a vertebrate (and so forth), and more generally, being an animal includes being an organism, and this in turn includes being physical (cf. footnote 3). The modes of human embodiment are also inclusive: The point of calling non-living entities "bodies" and talking about "the body of physics" is that we only get a more complete comprehension of even non-living matter when we realize that it can evolve into higher forms of organization whose properties transcend mere physical properties. 21 Such properties are emergent, though still depending on physical processes. Physical embodiment. Classical as well as modem physics deal with three kinds of objects; first, general forces in nature, particles, general 21. This point is reminiscent of Marx' use of Hegel's philosophy. Cf. Marx' aphorism "The anatomy of man is a key to the anatomy of the ape" (from Introduction to Grundrisse) that should be understood within a frame of a radically reconstructive historical approach as opposed to naively constructive chronological historiography (see Stah11975: 59-62).

On the biosemiotics ofembodiment and our human cyborg nature

397

bodies (matter in bulk), and the principles ("laws") governing their action; second, more specifically the structural dynamics of self-organized bodies (galaxies, planets, solid matter clusters, etc.); third, physical aspects of machines (artefacts produced by human societies and thus only fully explainable also by use of social sciences, like history of technology). In the history of science one has often seen attempts to reduce all of physics to a formalism equivalent to some formal model of a machine, but there are strong arguments against the completeness of this program (Rosen 1991), i.e., mechanical aspects of the physical world are only in some respects analogous to a machine. Some of the general properties of bodies studied in physics have a teleonomic character (a kind of directedness or finality), in the scheme called "thermo-teleology", as this phenomenon of directedness is most known from the second law of thermodynamics (a directedness towards disorder), as well as the opposing self-organizing tendencies in far from equilibrium dissipative systems. "Final causation" indeed is a pervasive aspect of purely physical processes (Hulswitt 2002). Organismic embodiment. A biological notion of function 22 is not yet present in physics, while it is crucial for all biology. Biofunctionality is not possible unless the living system is self-organizing in a specific way, based upon a memory of how to make components of the system that meet the requirement of a functional metabolism of a high specificity. For Earthly creatures this principle is instantiated as a code-plurality between a "digital" code of DNA, a dynamic regulatory code of RNA (and other factors as well, partly digital partly analogic), and a dynamic mode of metabolism involving molecular recognition networks of proteins and other components. This establishes a basic form of living embodiment, the single cell (as a simple organism) in its ecological niche, which presupposes the workings of "the physical body" as a thermodynamic system in nonequilibrium, yet transcends that form by its systematic "memory,,23 of organism components and organism-environment relations. Biosemiotics posits that organismic embodiment24 is the first genuine form of embodi22. Or that for which a part works in order to sustain a living whole, cf. Emmeche (2002 b). 23. This memory is often described as "genetic information" but it includes other forms of memory as well, i.e., stable inheritance systems located within the organism (but outside DNA), or within the ecological habitat of the organism (Jabonka 2001). 24. A concept very similar to the same term in Ziemke 2003, who bases his term "organismic embodiment" on J. von Uexkiill and the notion of autopoiesis.

398

Claus Emmeche

ment in which a system becomes an autonomous agent "acting on its own behalf' (cf. Kauffman 2000), i.e., taking action to secure access to available resources necessary for continued living. It is often overlooked that the subject-object structure of this active agent is mediated not only energetically by a structured entropy difference between organism and environment, but also by signs of this difference; signs of food, signs of the niche, signs of where to be, what to eat, and how to trigger the right internal processes of production of organismic components the right time. It is often forgotten that the active responsitivity of the agent organism (based upon observable molecular signs) has, as an "inner" dimension, a quality of feeling,2s implied here by what is called irritability at the level of a single cell. Irritability is probably both real, logically in accordance with a basic matter-mind evolutionary continuity, rationally conceivable, though impossible for humans to sense or perceive "from within" or empathetically know "what it feels like", say, for an amoeba or an E.coli. Animate embodiment. As the reader might have guessed, the biosemiotic idea is that when we consider the realm of animal mind, the intentionality of an animal presupposes the simpler forms of feelings and irritability we stipulate in single cells (including the "primitive" free-living animals, such as protozoa, lacking, e.g., a nervous system), yet transcends these forms by the phenomenal qualities of the perceptual spaces that emerge in functional perception-action cycles as the animal's Umwelt. As already mentioned, proprioceptive semiosis is a crucial element of phenomenal as well as functional properties (cf. Sheets-Johnstone 1998). More generally However, Ziemke's "organismic embodiment of autopoietic, living systems" (2003: 1306) does not distinguish, as we do here, between the body of biology and the body of zoology and thus lacks a specific notion of animate embodiment which is characterized by not only (vegetative) autopoiesis but also (animate, proprioceptive, perception-motor-coupling based) movement. 25. The concept of "feeling" is not quite the same as "qualia" in contemporary philosophy of mind, but closer to its use in Peirce's phaneroscopy (e.g., [19311958] 1902, CP.1.306-311)."Inner" is not referring to a separation between brain states and bodily or environmental states (which is problematic, cf. Thompson and Varela 2001: 422) but to an ontological category of possibility and quality. As Thompson and Varela do not distinguish organismic from animate embodiment, the fIrst of their three dimensions of embodiment, "organismic regulation" (including "sentience" or "the feeling of being alive", ibid., 424) is a mixture of feeling on this basic organismic level and "primal" or "core" consciousness at the level of animate embodiment.

On the biosemiotics ofembodiment and our human cyborg nature

399

(and less controversially), the animal body is a highly complex and specific kind of a multicellular organism, a kind that builds upon the simpler systems of embodiment on the level of biology, such as physiological and embryogenetic regulation of the growth of specific organ systems, including the nervous system. These regulatory systems are semiotic in nature, and rely on several levels of coded communications within the body and their dynamic interpretations (for details, see Hoffmeyer 1996; Barbieri 2001; Markos 2002). The expression "the body of zoology" in the schematic list is used to emphasize both its distinctness as a level of embodiment, and that zoology instead of being simply part of an old-fashioned division of the sciences should be the study of animated movement, including its phenomenal ("mental") qualities. Anthropic embodiment. Two forms of human (anthropic) embodiment should be distinguished, (a) social embodiment universally characteristic of humans as beings forming distinct language-dependent sociocultural groups; (b) societal embodiment, forming "cyborg bodies" (Emmeche 2004b), or cybodies. The later is a special form of (a) in which a sociocultural body is embedded in a society with several social subsystems (e.g., work, education, politics, law, economy, love/reproduction! consumption, health care, art, science, media), i.e., a civilization, and where the body's animate mode of being is intrinsically connected to technology (e.g., medico-technology) dissolving sharp body/machine boundaries; see section 6 below. First let us concentrate on the anthropic body of a person as a societal being, that is, not simply social in the sense of being a social animal, but emphatically "human-social", being part of a society with division of labor, institutional subsystems, social roles, culture, etc.; as Aristotle said, a political animal. It is beyond the scope of this chapter to develop the specific senses in which humans (i) are embedded within a society, (ii) embody distinct social systems with all their peculiarities, (iii) have their minds constituted by institutional habits, which they (iv) internalize during socialization. The "habitus" sensu Bourdieu (Jenkins 1992) of a human being in a social field is the implicit way his or her process of socialization has enacted a semiotic system of embodied habits, some of which have even physically formed the very animate body of the person - juxtaposing people from different times and classes; an attritioned third world coal miner, a North American teenager, or a 17th Century French noblewoman, may serve as illustration. No doubt, other theoretical notions in sociology could express culturally specific forms of human embodiment, but Bourdieu's concept of habitus suffices here, as it represents a set of class-

400

Claus Emmeche

specific dispositions generating specific practices and patterns of perception; it is embodied in individuals, and at the same time, it is a collective and homogeneous phenomenon, mutually adjusted for and by a social group or class. 26 Thus, on this genuinely anthroposemiotic level, biosemiotics as an approach does not suffice to capture the complexities and specificities of human embodiment. Animals have various needs and appetites, including sexual ones, whereas human desire, though developmentally presupposing appetites, cannot be reduced to them. Furthermore, consciousness in animals like apes and monkeys can exhibit quite complex forms of social cognition (Bekoff, AlIen and Burghardt 2002), but without a linguistic system (for symbolic representation of abstract structures) and a societal consciousness (of different language games, social roles, institutions and normative systems) even animal social consciousness is not that form of societal consciousness we find among humans, together with selfconsciousness in its more complex forms. One of those forms is conscience, internalized culture-specific schemata of right and wrong relations between self and other. Histrionics, normally a name for dramatic art, is used in the schematic list to signify the intimate relation of human embodiment and socio-cultural situatedness: "The naked animal" is not really human unless it realizes itself as a histrion, an actress on a social scene, a

26. Using the concept of habitus should make the distintion between animate and anthropic forms of embodiment clear. However, one could ask (a) what is embodied in what; (b) what is not embodied, and (c) if "embodied" could simply be replaced by "physical", "situated" or "embedded". As for (a), sociocultural history is embodied in social institutions, and the norms and rules of an institution are again embodied in a person functioning within such an institution. Thus, a person in some role incarnates a part of sociocultural history. Sometimes but far from always, forms of anthropic embodiment imprint their marks on the animate body of a person. Question (b) is like asking "what is not cultural?" when dealing with sociocultural systems. The answer depends on the purpose of the analysis (are we interested in the pervasiveness of cultural processes in a human lifeworld, or, e.g., in delimiting borders between a civil society and a state as distinct spheres of influence?). The answer to (b) depend on the purpose, focus and conceptual tools of any specific analysis. Regarding (c), similar remarks applies (cf. the schematic list). The distinct levels of organismic, animate, and sociocultural embodiment imply distinct forms of embedment and situatedness.

On the biosemiotics ofembodiment and our human cyborg nature

401

person mediating - as if by a social "mask,,27 - a general status in the social field "in front of the mask" and a narratively structured unique selfconsciousness "behind the mask" (cf. Welker 2000).

5.

Causality and emergent levels of interpretation

The above levels of embodiment can be located within an evolutionary macro-history and further differentiated in detail. The attempts to characterize more precisely what notions of causality should be necessary to account for the whole series of levels would lead into technicalities we will eschew here (see, e.g., El-Rani and Emmeche 2000). Instead, let us briefly describe the intuitions and demands that such a notion of causality should conform to. The idea from emergence theory (cf. review by Pihlstrom 2002) is that once a new emergent level is formed, it implies both new properties and other principles than those found on the previous level, but also has an effect to constrain the possibilities of processes and actions on the level below. This "downward" influence (Andersen et al. 2000; Thompson and Varela 2001) may be seen as a formative, structuring cause, not to be mixed up with the efficient cause of classical physics. Thus, a biosemiotic perspective demands a "Non-standard Neo-Aristotelian Pluralist notion of Causation (NNPC), that allow for dealing with a complex system of several levels. Within a NNPC, the classical notion of a temporal dyadic cause-effect relationship is just one element (often to be used for within-level explanations) of a more complete set of causes also including material causes (as answers to question about composition), formal causes (corresponding to structural constraints of a higher level upon its components), and final causes (that may take a level-specific form, and thus may be interpreted either as the enactment of anticipatory purposeful behavior within a cybernetic system, or more generally (and Peircean) as a pervasive form of structural causation of which the physical laws of nature are just an example).28 For the notion of inclusive levels of embodiment this means, (I) for the physics-biology border, that for instance the metabolic regularities of organismic embodiment of a grass constrain the physics of protein 27. Cf. the Latin persona, a mask used by an actor. 28. There is an unsolved question here about the relation between a Peircean conception of causality operating with efficient and formal causation and an even more pluralist notion as the NNPC. See also Hulswitt 2002; Emmeche 2004a; Andersen et al. 2000.

402

Claus Emmeche

action within its leaves; (11) for the biology-zoology border, that the cognitive action-reaction cycles of an animal's perception and movement constrain the formation of interpretants within its nervous system; and (Ill) for the zoology-anthropology border, that even societal structures like a market economy in specific ways may have constraining effects upon the embodiment of not just human freedom and action, but also the very animal existence of human beings. Another important distinction is the one between, on the one hand, a general ontological principle of embodiment stating that general forms have to be incarnated or realized in particular instances and can hardly exist disembodied as platonic forms (the general type needs its individual tokens to exist), and on the other hand, system-specific embodiment, i.e., embodiment holding for particular phenomena of mind, consciousness, meaning and significance, as these phenomena are embodied in particular system types (therefore the general principle that the mind is materially embodied must be specified). We have already seen the first principle exemplified by what we called semiotic embodiment: A particular general sign, such as a symbol, needs particular instances to embody it. The second principle was illustrated by the explanation of the scheme of levels of embodiment. Seeing different forms of embodiment both in a semiotic perspective and as evolutionarily emergent means seeing them as involving emergent levels of interpretation and understanding. One could learn all physiological facts about, say, the effects of iron nails against the skin of a person; perhaps useful for learning surgery, yet without getting to the essence of some phenomenon. Also the biology of wound healing and the neurophysiology of pain experiences may not satisfy our inquiry. The question is, what is the phenomenon? Closer to revealing fragments of that mystery may be developmental psychology, but at this stage we also have to ask questions about the specific sociocultural context - are we talking about skin piercing as a fashion of youth culture, or crucifixion as a tool for torturing political prisoners, or an alleged messiah?

6.

The cyborg nature of human embodiment

From the reflections on system-specific forms of embodiment thus far it transpires that biosemiotics is not enough to account for the characteristics of human embodiment. Some of these, such as the capacity to handle language and language-dependent societal processes, have often been seen as

On the biosemiotics ofembodiment and our human cyborg nature

403

disembodied, almost Platonic forms. But what may seem to be disembodied societal structures, like a state's constitution, a moral principle, or the very spoken language, are always specific historical products that indeed have to be realized in specific social fields and systems to be real. Thus the general metaphysical principle of embodiment holds, also for the societal level. The human animal was from its beginning societal, language-based, technical and political, but not all these determinations were equally developed. Let us just briefly look at the technological dimension that only became fully realized in the industrial phases of civilization, when we discover what we can call the intrinsic cyborg nature of humans, as technoculturally embedded beings within a space of meanings that are not only symbolic, but argumentative and culturally empowered by different kinds of social systems. Often, in cyber-punk and science fiction literature, the cyborg is seen either as the idea of a "melding of human and machine" and a "new era of participant evolution", or as people modified mechanically to perform specific tasks or redesigned to work in an alien environment (Stableford 1999). By a strangely familiar kind of reification, the machine is seen as something alien to the human body, and the melding of body and machine is then perceived as threatening human dignity or the 'essence' of being a human subject. 29 The anti-essentialist threat of the possibility, afforded by biotechnology, of a complete "hybridization" of a human organism and a machine has been used critically within feminist thinking (Haraway [1991]; Hayles 1999; Flanagan and Booth 2002) which in turn has inspired phenomenological approaches to technology (Ihde 2002). We will only use these sources to emphasize that the machine is not any "alien" entity. Though usually made of inorganic materials, any machine is a product of human work and its societal network of skills, ideas, thought, institutions, etc., and thus, we reify or blackbox the machine when we see it as a non-human entity.30 Marx was probably the first historian to make this point which is also a point about embodiment. On the level of human embodiment, tools, technical artefacts, machines, all embody human co29. Thus, one tends to forget that "The body in a contemporary society is already a cyborg body (a partly scientific-technologically governed body) as we are dependent, from birth to death, on the blessings of present-day medical science" (Emmeche 2002a: 155; see also Clark 2003. 30. This does not mean that there are no crucial differences between organismic and robotic forms of embodiment, as the work of Tom Ziemke (2003, see also this volume) clearly shows.

404

Claus Emmeche

operative division of labor, and of course, in our society, knowledgeintensive machines embody societal forms specific for a "post-industrial," "late capitalist," "risk" or "knowledge" society. Today the use of computers or "thinking machines" has become widespread, and we should be aware of the ease of blackboxing or reifying the semiotic processes involved in their interpretation. In current Western society, it is an aspect of the cyborg nature of human existence that machines both embody and hide complex social relations, and somehow make us forget the distributedness of processes like meaning and cognition, as we are led to commit failures of misplaced concreteness when we localize thought either in a single machine or a single brain. Our cyborg nature implies as a concept that technical artefacts, as one of the conditions of existence of the human form of life, embody human thought processes. This applies also to simple artefacts as part of the anthroposemiotic network of tools, as Peirce noticed: A psychologist cuts out a lobe of my brain (nihil animale me alienum puto) and then, when I fmd I cannot express myself, he says, "You see your faculty of language was localized in that lobe. " No doubt it was; and so, if he had filched my inkstand, I should not have been able to continue my discussion until I had got another. Yea, the very thoughts would not come to me. So my faculty of discussion is equally localized in my inkstand. It is localization in a sense in which a thing may be in two places at once. On the theory that the distinction between psychical and physical phenomena is the distinction between fmal and efficient causation, it is plain enough that the inkstand and the brain-lobe have the same general relation to the functions of the mind. (Peirce [1931-1958] 1902, CP 7.366; see also Noth 2003; Skagestad 1999).

When we have arrived at a phase of civilization, in which the societal form of human embodiment31 within the health care system has made us all ex31. Levels of embodiment (cf. the schematic list above) are partly dependent upon the epistemic interest and analytical perspective; this applies also for the above mentioned (a) social and (b) societal forms of anthropic embodiment. The kind of "social embodiment" reviewed in Barsalou et al. (2003) corresponds to (a) although some of the research reviewed tends to assume that, e.g., facial and bodily responses (examplifying social embodiment) are cross-cultural universals. The theory of Barsalou (cf. Lindblom and Ziemke this volume), especially the notion of simulators, is interesting (in spite of its metaphorical load of the brain as a computer) but raises this question: To which extent are simulators (e.g., "the simulator for the social category face", ibid., 67) specific for social

On the biosemiotics ofembodiment and our human cyborg nature

405

istentially connected to that system (and thus, to the system of technoscience that health care relies on), our cyborg nature takes on a double form. In a basic dimension, we continue to be that kind of animal, who can only survive in its present form by being interconnected to a global technosocietal system. On a supervening dimension, historically specific for a modem risk society, this interconnected existence change our very animal bodily appearance by the possibility of, e.g., prolonging life, exchanging body parts because of ageing and desires to be forever young, tendentially transforming us into the cyborgs of science fiction - what some has called "the posthuman condition".32 Due to the inclusivity of the forms of embodiment, the term posthuman is misleading though, as it is only now, when we come to consider ourselves to be beyond human nature, that our nature is revealed to us as at once human and cyborgian: To paraphrase Marx, the anatomy of the cyborg is the key to the anatomy of humans. We are already cyborgs, hooked-up to our fellow beings by tools and technology, and in late modernity, also via the technoscientific health care system. Human cyborg embodiment can take many forms, not all equally desirable, and will continue to be a contested issue in the biopolitics of the present century.

References Andersen, Peter. B., Niels O. Finnemann, Peder V. Christiansen and Claus Emmeche (eds.) 2000 Downward Causation. Minds, Bodies and Matter. Aarhus: Aarhus University Press. Barbieri, Marcello 2001 The Organic Codes. Cambridge: Cambridge University Press.

embodiment in humans? Could the theory also apply to, say, social embodiment in a herd of wolfs? And if so, in which sense is it a theory specifically explaining social embodiment in humans, in contrast to more general animal-based cognitive capacities? 32. Most famous in the posthuman discourse is Fukuyama (2002); see also Hayles (1999).

406

Claus Emmeche

Barsalou, Lawrence W., Paula M. Niedenthal, Aron K. Barbey and Jennifer A. Ruppert Social embodiment. In: Brian H. Ross (ed.), The Psychology of 2003 Learning and Motivation 43: 43-92. San Diego, CA: Academic Press. Bekoff, Marc, Colin AlIen and Gordon M. Burghardt 2002 The Cognitive Animal. Empirical and Theoretical Perspectives on Animal Cognition. Cambridge, Mass.: MIT Press. Brooks, Rodney A. 2002 Flesh and Machines. New York: Vintage Books. Clark, Andy An embodied cognitive science? Trends in Cognitive Science 3 (9): 1999 345-351. 2003 Natural-Born Cyborgs. Oxford: Oxford University Press. Csordas, Thomas J. 1996 Body. In: Adam Kuper and Jessica Kuper (eds.), The Social Science Encyclopedia. 2nd ed., 55-56. London: Routledge. EI-Hani, Charbel Nifio and Claus Emmeche 2000 On some theoretical grounds for an organism-centered biology: Property emergence, supervenience and downward causation. Theory in Biosciences 119 (3/4): 234-275. EI-Hani, Charbel Nifio, Joao Queiroz and Claus Emmeche 2005 A semiotic analysis of the genetic information system. (unpublished manus., in prep.) Emmeche, Claus 2001 Does a robot have an Umwelt? Reflections on the qualitative biosemiotics of Jakob von UexkiilI. Semiotica 134 (1/4): 653-693. 2002 a Kroppens kaput som organisme. In: Gert Balling (ed.), Homo sapiens 2.0. Nar teknologien kryber ind under huden. 121-156, 238-239. Kebenhavn: Gads Forlag. 2002 b The chicken and the Orphean egg: On the function of meaning and the meaning of function. Sign Systems Studies 30 (1): 15-32. 2004 a Causal processes, semiosis and consciousness. In: Johanna Seibt (ed.), Process Theories: Crossdisciplinary Studies in Dynamic Categories, 313-336. Dordrecht: Kluwer. 2004 b Alife, organism and body: The semiotics of emergent levels. In: Mark Bedeau, Phil Husbands, Tim Hutton, Sanjev Kumar and Hideaki Suzuki (eds.), Workshop and Tutorial Proceedings. Ninth International Conference on the Simulation and Synthesis of Living Systems (Alife IX), 117-124. Boston Massachusetts.

On the biosemiotics ofembodiment and our human cyborg nature

407

Emmeche, Claus, Simo Keppe and Frederik Stjernfelt 1997 Explaining emergence - towards an ontology of levels. Journal for General Philosophy ofScience 28: 83-119. Emmeche, Claus, Kalevi Kull and Frederik Stjernfelt 2002 Reading Hoffmeyer, Rethinking Biology. (Tartu Semiotics Library 3). Tartu: Tartu University Press. Flanagan, Mary and Austin Booth (eds.) 2002 reload: rethinking woman + cyberculture. Cambridge, Mass.: MIT Press. Fukuyama, Francis 2002 Our Posthuman Future: Consequences of the Biotechnology Revolution. New York: Farrer, Strauss and Giroux. Gilbert, Scott F. and Sahotra Sarkar 2000 Embracing complexity: Organicism for the 21st Century. Developmental Dynamics 219: 1-9. Haraway, Donna 1985 A cyborg manifesto: science, technology and socialist-feminism in the late twentieth century. In: Donna Haraway (ed.), Simians, Cyborgs and Women: The Reinvention ofNature, 149-181. New York: Routledge. Reprint 1991. Hausman, Carl R. 1993 Charles S. Peirce's Evolutionary Philosophy. Cambridge: Cambridge University Press. Hayles, N. Katherine 1999 How We Became Posthuman. Chicago: The University of Chicago Press. Hoffmeyer, Jesper 1996 Signs of Meaning in the Universe. Bloomington: Indiana University Press. Hulswitt, Menno 2002 From Causation to Cause. A Peircean Perspective. Dordrecht: Kluwere Ihde, Don 2002 Bodies in Technology. Minneapolis: University of Minnesota Press. Jablonka, Eva 2001 The systems of inheritance. In: Susan Oyama, Paul E. Griffiths and Russell D. Gray (eds.), Cycles of Contingency, 99-116. Cambridge, MA: MIT Press. Jenkins, Richard 1992 Pierre Bourdieu. New York: Routledge. Kauffman, Stuart 2000 Investigations. Oxford: Oxford University Press.

408

Claus Emmeche

Keller, Catherine 2003 Embodiment. In: J. Wentzel Vrede van Huyssteen (ed.), Encyclopedia ofScience and Religion. Vol. I, 254-256. New York: Macmillan Reference. Kull, Kalevi (ed.) 2001 Jakob von Uexkull: A paradigm for biology and semiotics. Berlin and New York: Mouton de Gruyter (= Semiotica 127 (1/4): 1-828). Kull, Kalevi 1999 Biosemiotics in the twentieth century: A view from biology. Semiotica 127 (1/4): 385-414. Lakoff, George and Mark Johnson 1999 Philosophy in the Flesh. The Embodied Mind and its Challenges to Western Thought. New York: Basic Books. Lawrence, Christopher J. 1981 Irritability/sensibility. In: W.F. Bynum, E.J. Browne and Roy Porter (eds.), Macmillan Dictionary ofthe History ofScience, 214. London: Macmillan Press. Markos, Anton 2002 Readers ofthe Book ofLife. Oxford: Oxford University Press. Marx, Karl The Capital. Vol. I. London: Penguin Classics. Reprint 1990. 1867 Mayr, Emst 1982 The Growth ofBiological Thought. Harvard, Cambridge: Belknap. Noth, Winfried 2003 Semiotic machines. S.E.E.D. Journal (Semiotics, Evolution, Energy and Development) 3 (3): 81-99. Peirce, Charles Sanders 1902 Collected Papers of Charles Sanders Peirce. Vol. 1-6, Charles Hartshome and Paul Weiss (eds.); Vols. 7-8, Arthur W. Burks (ed.). Cambridge, Mass.: Harvard University Press. [References: CP, followed by vol. and paragraph number]. Published 1931-1958. Petrilli, Susan 1999 About and beyond Peirce. Semiotica 124 (3/4): 299-376. Pihlstrom, Sami 2002 The re-emergence of the emergence debate. Principia 6 (1): 133181. [Universidade Federal de Santa Catarina, Florian6polis, Brasil] Queiroz, Joao and Sidharta Riberio 2002 The biological substrate of icons, indexes and symbols in animal communication: a neurosemiotic analysis of Vervet monkey alanncalls. In: Michael Shapiro (ed.), The Peirce Seminar Papers: Essays in Semiotic Analysis, Vol. 5, 69-78 New York: Berghahn Books.

On the biosemiotics ofembodiment and our human cyborg nature

409

Riegler, Alexander, Markus Peschl and Astrid von Stein (eds.) 1999 Understanding Representations in Cognitive Science. Does representation need reality? Dordrecht: Kluwer. Rosen, Robert 1991 Life Itself. A Comprehensive Inquiry Into the Nature, Origin and Fabrication ofLife. New York: Columbia University Press. Salthe, Stanley N. 1989 Self-organization of/in Hierarchically Structured Systems. Systems Research 6 (3): 199-208. Santaella Braga, Lucia 1999 A new causality for understanding the living. Semiotica 127 (1/4): 497-519. Santaella, Lucia 2003 What is a symbol. S.E.E.D. Journal (Semiotics, Evolution, Energy and Development) 3 (3): 54-60. Scott, Alwyn 2004 Reductionism revisited. Journal of Consciousness Studies 11 (2): 51-68. Sheets-Johnstone, Maxine 1998 Consciousness: A natural history. Journal ofConsciousness Studies 5 (3): 260-294. Short, Thomas L. 1988 The growth of symbols. Cruzeiro Semiotico 8 (Janeiro): 81-87. (Porto, Portugal). 1982 Life among the legisigns. Transactions ofthe Charles Sanders Peirce Society 18 (4): 285-310. Skagestad, Peter 1999 Peirce's inkstand as an external embodiment of mind. Transactions ofthe Charles S. Peirce Society 35: 551-561. Stableford, Brian 1999 Cyborg. In: John Clute and Peter Nicholis (eds.), The Encyclopedia ofScience Fiction, 290-291. London: Orbit. Stahl, Gerry 1975 Marxian Hermeneutics and Heideggerian Social Theory: Interpreting and Transforming our World. Ph.D. thesis, Northwestern University, Evanston, Illinois. (also available at http://www.cs.colorado.edu /~gerry/publications/dissertations/philosophy) Thompson, Evan and Francisco J. Varela 2001 Radical embodiment: Neural dynamics and consciousness. Trends in Cognitive Sciences 5 (10): 418--425.

410

Claus Emmeche

Van de Vijver, Gertrudis, Stanley Salthe and Manuela Delpos (eds.) 1998 Evolutionary Systems: Biological and Epistemological Perspectives on Selection and Self-Organization. Dordrecht: Kluwer. Van de Vijver, Gertrudis, Linda Van Speybroeck and Windy Vandevyvere 2003 Reflecting on complexity of biological systems: Kant and beyond? Acta Biotheoretica 51 (2): 101-140. Varela, Francisco J., Evan Thompson and Eleanor Rosch 1991 The Embodied Mind: Cognitive Science and Human Experience. Cambridge, MA: MIT Press. Varzi, Achille 2004 "Mereology", The Stanford Encyclopedia of Philosophy (Fall 2004 Edition). In: Edward N. Zalta (ed.), URL = http://plato.stanford.edu/ archives/fa1l2004/entries/mereology/ von Uexkiill, Jakob 1940 Bedeutungslehre. (Bios 10. Johann Ambrosius Barth, Leipzig), [English translation: The theory of meaning, Semiotica 42 (1): 2582, 1982]. von Uexkiill' Thure 1982 Introduction: Meaning and science in Jakob von Uexkiill's concept of biology. Semiotica 42 (1): 1-24. Welker, Michael 2000 Is the autonomous person of European modernity a sustainable model of human personhood? In: Niels Henrik Gregersen, Willem B. Drees and UlfGorman (eds.), The Human Person in Science and Theology, 95-114. Edinburgh: T&T Clark. Ziemke, Tom 2003 What's that thing called embodiment? In: Richard Alterman and David Kirsh (eds.), Proceedings of the 25 th Annual Meeting of the Cognitive Science Society, 1305-1310. Mahwah, NJ: Lawrence Erlbaum. Ziemke, Tom and Noel E. Sharkey 2001 A stroll through the worlds of robots and animals: Applying Jakob von Uexkiill's theory of meaning to adaptive robots and artificial life. Semiotica 134 (1/4): 701-746. Zurek, Wojcieh H. 1989 Thermodynamic cost of computation, algorithmic complexity and information metric. Nature 341: 119-124.

Embodiment and self-organization of human categories: A case study for speech

Luc Steels and Bart de Boer

Abstract The paper considers explanations for the kinds of categories that have been found to be involved in human behavior. It insists that embodiment not only plays an important role in shaping these categories, but also that the collective dynamics generated by social interaction is of equal importance. A case study is developed for speech sounds. We show through computer simulations how a group of autonomous agents equipped with a (sufficiently) realistic perceptual and auditory apparatus can arrive at a shared repertoire of vowels and that these vowel systems exhibit the same universal trends that are found in human vowel systems. It is significant that this happens without innate a priori categories. Keywords: Embodiment, categorization, self-organization, speech.

1.

Introduction

The term "embodied" is being adopted by a growing number of researchers interested in cognition (Ziemke 2003), but often with different meanings. At least three senses of embodiment are being used: The first one emphasizes that cognitive processes, such as those required for vision or language, are to be implemented in terms of networks that have neural plausibility, both in terms of their computational microstructure and in terms of the general architecture of the brain. While there is of course a wide consensus that cognitive processes must somehow be mapped onto brain hardware, the "strong embodiment" hypothesis claims more, namely, that without this neural realism many cognitive phenomena cannot be understood. This line of reasoning finds its most enthusiastic defenders among phi-

412

Luc Steels and Bart de Boer

losophers like Searle and those who argue that the brain should not be viewed as an information processor at all (e.g. Edelman 1990). The second type of embodiment research first manifested itself in "behavior-based AI" (Steels and Brooks 1995). It argues that part of our interaction with the world is not explicitly controlled by any kind of information processing. Instead, the physical properties of the body, the physical characteristics of sensors and actuators, the structure of the task, and the physical properties of the environment all play an important role in shaping actual behavior. This line of research has proven to be very fruitful for designing and building autonomous robots that interact with real-world environments in real time (Pfeifer and Iida 2004). A neat example is the robot Stumpy that exhibits locomotion without any kind of explicit control or sensing (Iida, Dravid and Paul 2002). The third sense of embodiment refers to the grounding and expression of human concepts. It argues that many concepts, even abstract mathematical concepts, find their ultimate foundation in grounded interaction with the world (Lakoff 2000) and that this is still visible in the metaphors and expressions used to communicate using these concepts. For example, the temporal preposition "back" (as in "back in time") is historically derived from a body part (the back) which first became used to indicate a spatial relation or area (as in the back of the car) and then a temporal relation (see also Nlifiez 1999). It follows from this position that conceptualization and concept acquisition must be embedded in a developmental history of embodied interaction with the world. This paper explores the second view on embodiment, as manifested in "embodied AI", and tries to demonstrate what kind of explanatory power can be drawn from it. We take the domain of speech sounds as a case study because we believe that without studying concrete cognitive tasks and showing what embodiment can bring, the discussion may become too divorced from reality. Section 2 contrasts classical theories of speech that do not take embodiment into account with theories grounded in the function of speech that do take embodiment into account. Section 3 presents theoretical considerations about what constitutes an explanation in cognitive science as well as an overview of the phonological phenomena under consideration. Section 4 presents a case study in which the role in explaining universals of speech sounds of embodiment and situatedness are investi-

Embodiment and self-organization ofhuman categories

413

gated with a computer model. Section 5 presents a summary and the conclusion.

2.

Embodied versus disembodied speech research

The domain of speech is known to be extraordinarily difficult to understand and model because it not only involves complex fine-grained motor control of the vocal articulators for producing speech sounds, but also complex acoustic signal processing and categorization for recognizing speech sounds. Therefore, there is at least the potential for a strong role of embodiment. At the same time speech also seems to imply abstract categories (in the sense that some distinctions are relevant in a language and others are not) and abstract rules (such as "a voiced consonant becomes unvoiced at the end of a word", a rule which holds for Dutch but not for English). There is a long tradition in linguistics, shaped by the Prague structuralists, with Roman Jakobson (Jakobson and Halle 1956) as its most prominent member, and Chomsky and Halle's influential work on abstract phonology (Chomsky and Halle 1968), that completely ignores the embodied aspects of speech. Instead, these linguists move immediately to an abstract level in terms of distinctive features (such as +/- voiced, +/- labial, +/aspirated, etc.). These distinctive features are assumed to be both grounded in articulation, as well as in acoustic recognition and categorization. For example, voiced means that the vocal chords are vibrating while a sound is being produced, such as in the final consonant of "bad" (voiced) versus "bat" (voiceless). Yet it is never specified in detail how exactly this grounding is realized. Abstract phonologists simply use the abstract features to formulate phonological constraints (e.g. which combinations of consonants and vowels are allowed in a language). But it turns out that it is very hard to recognize these features in the acoustic data and that they are not sufficiently precise to realistically drive motor control. Moreover this branch of phonology often assumes that the features are genetically predetermined as part of the innate language acquisition device (LAD) and argues that systematicity is genetically encoded in terms of a set of principles with parameters to be set by cues from the environment (Dresher 1992). Abstract phonology is therefore a prime example of disembodied cognitive science and it contrasts with work by another group of phoneticians and phonologists who employ what can be considered an embodied ap-

414

Lue Steels and Bart de Boer

proach to speech (see e.g. Lindblom and Maddieson (1988); Browman and Goldstein (1992)). They start from the actual acoustic signals and from sophisticated articulatory models, taking constraints coming from the physics of the world and embodiment into account. For example, many coarticulation phenomena are no longer assumed to be governed by rules but simply follow from the dynamics of motor control. Grounding now plays a role in explaining the acquisition of speech sounds or why certain sounds or sound combinations are occurring (MacNeilage 1998). Perception is always coupled tightly to production (Plaut and Kello 1999). The research discussed in this paper fits within this embodied view of speech. Section 3 presents theoretical considerations about what constitutes an explanation in cognitive science

3.

Explanatory theories

Before delving into the concrete case study presented in this paper, it is worthwhile to make some general methodological remarks about the nature of theories in cognitive science. It is well known that categorization is a central topic in cognition; indeed we can say that a behavior becomes cognitive only when some form of categorization is implied. Traditionally, linguistics, psychology and anthropology have collected a large number of facts about human categorization. For example, Berlin and Kay (1969) collected data about the kinds of color categories found in human languages by doing naming and memory experiments, as first pioneered by Lenneberg and Roberts (1956). Subjects are shown a series of color samples (the Munsell chips) and are asked to show the best representative for a specific color like "red". Berlin and Kay found that there are some surprising universal trends in focal colors and their work has generated a wide body of literature that seeks to identify additional evidence which confirms (Kay and Regier 2003) or refutes (Davidoff 2001) these basic human color categories. Another domain that has been studied extensively is space. Here again research has shown that there are universal tendencies in spatial categories. For example, IN, ON, LEFT OF, etc. are very common categories in human spatial reasoning and language (Herskovits 1986) and so it has been argued that they are largely innate. But on the other hand, comparative studies of the expression of spatial categories has shown significant intercultural differences. Also, children only gradually acquire spatial cate-

Embodiment and self-organization ofhuman categories

415

gories under the influence of their native language (Bowerman and Choi 2001). Similarly the categories (and hence the articulatory gestures) used in sound systems of human languages show remarkable regularities. Languages do not use random subsets of the many possible speech sounds, but rather certain sounds occur more often than others. Although humans are able to produce and distinguish an amazing number of different speech sounds, the average number of distinctive speech sounds that is used in the languages of the world lies between 20 and 37 (Maddieson 1984). Some sounds, such as [a], [m] or [k] occur almost universally, while others occur only rarely. For example, English does not use the rather rare vowel [y] (pronounced as in French "rue") whereas French (and Dutch) do. Phoneme inventories tend to exhibit symmetries as well. If, for example, a repertoire contains [0] it is very likely to also contain [e]. Such symmetries are also found in consonant systems (Lindblom and Maddieson 1988). Languages furthermore constrain the set of possible sound combinations. For example, in English [mb] can occur at the end of a word as in "lamb" but not in the beginning, whereas in some African languages this is possible (as in Swahili "mbali" (far)). So sounds fall into classes and the classes form a combinatorial system. Again there appear to be universal tendencies in these sound combinations although they have been less adequately surveyed. Regularities like these (and many others can be demonstrated, including universal tendencies in grammar) demand an explanation. Linguists and psychologists in the disembodied tradition often propose that these universal tendencies are based on innate mechanisms (e.g. the distinctive phonological features and their markedness is assumed to be part of the innate language faculty), and so they are said to be universally shared by all human beings, the same way each of us has (normally) five fingers on each hand. But that means that these theories simply pass the buck to evolutionary biologists. However, biologists are often not so keen about ascribing complex human behavior to innate features, partly because it seems rather implausible that the micro circuitry of the brain is under detailed genetic control (Edelman 1990), and partly because we would still need an evolutionary scenario to explain for instance why [y] (as in French "rue") is not common and [a] (as in English "bar") is, or why a language with four color terms usually has terms for either green or yellow. So simply saying that human categories are (to a large extent) innate is not a sufficient explanation; we also need to show which neural structures can perform the categorization, how these neural structures could be ge-

416

Luc Steels and Bart de Boer

netically determined in development and how the relevant genes could have become universally present in human populations. All of these are non-trivial tasks that remain to be carried out. Another difficulty for the genetic position is that without exception, human categorization is not absolutely universal across all human populations and cultures. Rather it shows significant variation. Although this variation is supposedly captured in parameters set during development, it is unclear how such a parameterized system could evolve under realistic selection pressures based on language communication. If genetics is not very satisfactory as a source of explanations, we should perhaps investigate other ways in which certain categories like colors, spatial categories or speech sounds can become shared (universally or locally). Three possible determining factors have been discussed: (1)

(2)

(3)

Either human embodiment is so constraining that it is not necessary to make the categories innate because they could never be perceived or the motor movements could never be made. For example, the human vision system restricts perception to a certain range of the spectrum; the human vocal tract is limited to certain number of sounds, the human body suggests a division into front and back, etc. This enables and limits the set of possible solutions for a specific task. Human beings interact socially and this creates a collective dynamics that may also constrain individual choices, pushing them in the direction of a shared system. For example, in Britain cars drive on the left side of the road and in continental Europe they drive on the right side. This is not because British drivers have "left driving genes" so that they are born to drive on the left, or that their embodiment makes driving on the right impossible, but rather because a cultural choice was made and this choice is enforced by the subsequent behavior of all involved. Even if somebody would like to drive on the right in England, they would be quickly sanctioned or maybe even lose their life. It could be that the regularities in the environment are such that a statistical learning process picks them out easily and so there would be no need to encode the category in the genome. For example, the chromatic distribution of colors in the world might show enough statistical regularity that statistical clustering algorithms could detect them.

Embodiment and self-organization ofhuman categories

417

In the work being carried out by our research group, we have been exploring explanations for human categorization in which these three factors play a role. A good example is our research on color categories, in which we have attempted to explain the universal tendencies in color categories using embodiment, the collective dynamics generated by social interaction, and the statistical structures inherent in the world (Steels and Belpaeme 2005). In the domain of language in general and speech sounds in particular, the statistical regularity is not present in the world independently of the agents. So only factors (1) and (2) are relevant, particularly when a totally new system is being bootstrapped. In the case study that follows, originally described in (de Boer 1999, 2000), embodiment and the dynamics of language use in a population are investigated as factors explaining regularities in systems of human speech sounds. As vowels are the simplest speech sounds to model, this paper focuses on vowels, but there also exists work on syllables and syllable structures (Steels and Oudeyer 2000; Redford, Chen and Miikkulainen 2001). Our main point is that embodiment in itself is not sufficient to explain many universal tendencies or culturally accepted norms. Earlier embodiment approaches have shown that embodiment constrains the possibilities, e.g. that there are limitations of actuators and sensors which prevent or discourage humans from using certain sounds. In addition, these approaches demonstrate that typical speech categories are in some sense optimal (see e.g. Lindblom, MacNeilage and Studdert-Kennedy. 1984). Specifically the sounds in a language are claimed to have a particular distribution and structure that is optimized for the task of reliably producing and perceiving sounds, and hence the sounds are maximally contrastive. If that is the case, then embodiment and the constraints of the task are sufficient to explain why human languages show particular sounds and not others. Nonetheless, observations of real phonetic data show that suboptimal phonetic systems also exist in many languages. For example, the UPSID database of human sound systems lists two examples with suboptimal 3vowel systems, namely with the sounds [e] [a] [0] instead of the more common optimal [i] [a] [u]. They come from a North-American Indian language, Alabaman (Rand 1968) and a Peruvian language, Amuesha (Fast 1953). These 3-vowel systems are suboptimal because they do not provide the maximal contrast in the acoustic space. Therefore embodiment (and the constraints of the task) are in themselves not sufficient to explain the vowel

418

Lue Steels and Bart de Boer

systems one finds in human languages. One also has to take the collective dynamics of language use in a population into account. All users have to conform to the system of speech sounds that has historically been chosen in order to achieve communicative success. This system is partly arbitrary and may be suboptima1.

4.

Experiments in self-organization of speech sounds

To examine whether the structure of vowel systems is sufficiently determined by self-organization in a population under constraints of embodiment, we need to find a mechanism that incorporates these two factors, and then show that this indeed leads to emergent vowel systems that reflect the universal tendencies observed in human vowel systems. In our case, we use a population of artificial agents that can produce, perceive and remember speech sounds in a human-like way. Each agent is equipped with an articulatory synthesizer, a model of human perception for calculating the distances between different signals, and an associative memory for storing vowel prototypes. Also, each agent can interact with other agents (following a fixed pattern) by imitating them. These interactions are called imitation games, and are in the same line as the "language games" we have used in our work for the past decade (see Steels 2003). The agents can update their vowel repertoires depending on the outcome of the interactions in such a way that the expectation of future imitation success is maximized. The agents' goal in life is to imitate the other agents as well as possible with a repertoire of vowels that is as large as possible. However in doing this, the agents only use local information, and do not carry out any explicit optimization. The behavior of the agents is summarized in table 1 at the end of section 4.1.

4.1.

Embodiment constraints on perception and production

The repertoire of speech signals that humans can produce is limited by the physics and the physiology of the vocal tract (figure 1). The repertoire of sounds that they can distinguish is limited by the properties of the auditory apparatus. In this example the focus is on vowels, and therefore only the articulatory and perceptual properties of vowels are integrated. These properties are also better understood than those of consonants. This has to

Embodiment and self-organization ofhuman categories

419

do with the fact that vowels are static signals, while many consonants are dynamic, i.e. their articulation and their signal do not remain constant over time.

:fculators

Alveolum (teeth ridge)

\

Teeth

1 1

Lips

.~~-

\I,

..

.........

Tip

IBlade Tongue

Figure 1.

-GloWs

Structure of the human vocal tract. Speech sounds are produced by very fme-grained movements of the articulators

The acoustic signal of a vowel can be described relatively straightforwardly by the resonances of the vocal tract. These resonances cause prominent peaks in the frequency spectrum of a vowel signal. The frequencies at which these peaks occur are called the formant frequencies. This is illustrated for the vowel [a] (as in "father") by the black line in figure 2. For different vowels, these peaks occur in different places. Vowels can therefore be uniquely characterized by their formant frequencies. In practice, only the three or four resonances at the lowest frequencies are relevant. In the simulations discussed here, no acoustic signals are generated, although it would be straightforward to add the extra complexity. Vowels are represented by their first four formant frequencies. If an agent wants to generate a given vowel, it synthesizes these formants using an articulatory

420

Lue Steels and Bart de Boer

synthesizer. This synthesizer takes as inputs the three major vowel parameters: tongue position, tongue height and lip rounding (Ladefoged and Maddieson 1996, Ch. 9). The outputs of the synthesizer are the first four formant frequencies of the corresponding vowel. The inputs are modeled as real values in the range 0 to 1. For tongue position, 0 means most to the front and 1 means most to the back. For tongue height, 0 means lowest and 1 means highest. For lip rounding, 0 means least rounded and 1 means most rounded. Thus (0, 0, 0) corresponds to the vowel [a] and to the formant frequencies (708, 1517, 2427, 3678) Hertz. The parameters (1, 1, 1) correspond to the vowel [u] and to the formant frequencies (276, 740, 2177,3506) Hertz. Formant values were taken from (ValIee 1994). This synthesizer is able to generate all possible basic vowels.

2..,.---------------------, o• 700 H4380 Hz ..

~

-2 \ . / ·4·

A

;.~

-~

~580

\ '

Hz

V~C

-!---".....-....r----,,--.....,..--..,.,....----l

o

1000

2000 1, q

Figure 2.

3D

4000

0000

5000

n(;:y(~)

The fIrst four fonnants of the vowel [a] black line, and its perception in tenns of the effective second fonnant (gray line)

Humans also perceive vowels based on their formant frequencies. These can be used to calculate a perceptual distance between vowels. Unfortunately, the distance calculation is not as simple as calculating a Euc1idean distance between two formant vectors. As the bandwidth of the sound receptors at higher frequencies is greater than those at lower frequencies, blurring of spectral detail takes place at higher frequencies. This means that the formant peaks at higher frequencies can generally be compressed into one broader peak. The center frequency of this peak is called the ef-

Embodiment and selforganization ofhuman categories

421

fective second formant. The first peak in the spectrum is usually perceived as remaining in the same place, and is still referred to as the first formant. The perception in terms of the first and effective second formant of the signal [a] is shown as the gray line in figure 2. Note that this is an idealized view to illustrate the process. While there are several ways to calculate the effective second formant, the one adopted in the research described here has been developed by Schwartz et al. (1997b). It is a non-linear weighted average of the 2nd, 3rd and 4th formant frequencies. In order to calculate distances between vowel signals, the first and effective second formant are expressed in the Bark frequency scale. The Bark scale is a perceptually inspired frequency scale that can be considered to be logarithmic for the frequencies that are relevant to formants. Equal frequency differences in Bark are perceived as equal intervals, in contrast to the way frequencies in Hertz are perceived.

..

21 1

~

I I

16

12

14

F (Bark) 2

Figure 3.

10

8

8

16

14

12 F (Bark) 2

10

2

a\.

~4

!!!.

g

LL9""

t .0.

6

;:

1

8

LL9""

10000 games

~.

!!!.

~

:'

6!

~4

,.

~4

~ 41 LL....

2000 games

~

~.

;

.

500 games

20 games

.-:.....

~.

...-:1·

LL'" '.

.J. 8

8

16

14

12 10 F (Bark) 2

" ,

8

8 16

14

...

..

~

~

'"

....... :

12

10

8

F (Bark) 2

Emergence of a vowel system in a population of twenty agents.

Using the first and effective second formant frequencies in Bark, the distance between two vowels can be calculated as an ordinary weighted Euclidean distance. As differences in the effective second formant are perceptually less important than differences in the first formant, the effective second formant is multiplied by 0.3 when calculating distances. These distances are then used to determine which vowel in an agent's repertoire is recognized. This is done by calculating the distance between a perceived signal and all the vowels in an agent's repertoire. The vowel that is closest to the perceived signal is considered to be the one that the agent heard.

1

Initiator If ( V = ) Add random vowel to V Pick random vowel v from V uv uv + 1 Produce signal A1: A1 acv + noise

Imitator

Receive signal A1. If ( V = ) Find phoneme( vnew, A1 ) V V vnew Calculate vrec: ¬ v2 :

vrec V

( v2

V

D ( A1 , acv 2 ) < D ( A1 , acv rec ) )

Produce signal A2: A2 + noise 3

2

acvrec

Receive signal A2. Calculate vrec: ¬ v2 :

vrec V

( v2

V

D ( A2 , acv 2 ) < D ( A2 , acv rec ) )

If ( vrec = v ) Send non-verbal feedback: success. sv sv + 1 Else Send non-verbal feedback: failure.

5

Do other updates of V.

Receive non-verbal feedback. Update V according to feedback signal. Do other updates of V.

4 5

Embodiment and self-organization ofhuman categories

4.2.

423

Results

The agents all start out with an empty vowel repertoire. By playing imitation games with each other, the agents have to develop a vowel system that is as large as possible, that allows for successful communication and that should be realistic if self-organization and embodiment are really factors in explaining the structure of human vowel systems.

4.2.1. Emergence ofa vowel system The emergence of a vowel system in a population of twenty agents under 10% acoustic noise is shown in figure 3. In each of the frames of the figure, the acoustic aspects of the prototypes of the agents' vowels in the population are plotted in acoustic space. In this particular acoustic space (based on the first and effective second formant frequencies of a vowel signal), equal distances between points correspond to equal perceptual distances. Each vowel of each agent in the population is represented by a dot. Note that due to articulatory constraints, only a roughly triangular area of the acoustic space is available to the agents. This is indicated in the fourth frame of figure 3. From the figure it is clear that after the first 20 games the agents still only have very few vowels. The vowels that exist are more or less randomly dispersed through the acoustic space, although some of them already show a tendency to cluster. This is caused by the fact that all agents start out with an empty vowel repertoire. In order to get the imitation games started, random vowels are inserted. However, the imitating agents in the games try to make imitations that are as close as possible and add these to their vowel repertoires. This accounts for the clustering. After some 500 imitation games, shown in the second frame, the clustering has become more pronounced. The most important process at this moment is the compacting of the clusters due to the fact that the agents move their vowel prototypes closer to the signals they perceive. However, there is still sufficient room in the auditory space for extra vowels, so the random addition of new vowels also plays a role. After 2000 games, the available vowel space becomes filled more evenly with vowels and the shape of the vowel system becomes more realistic. After 10.000 imitation games, the available acoustic space has become more or less filled up with vowels and the vowel system has become realistically symmetric and dispersed. After this

424

Lue Steels and Bart de Boer

has happened, the vowel system remains stable. However, it is not static. The vowel prototypes of agents (and therefore the clusters) tend to move, and it is even possible that they merge or that new clusters are formed (if they do not interfere with other clusters). Type B (10/54)

Type A (30/54)

2

lil~

••••\

.~.

• -a/ul

•ift..f a"~. , IfJ ,.:.• ~ I'JI lal

.~~

6

¥

/',1 ••" ..

e?.

6

/fB/

i·

•

,i 101

•

8 1...-

Type D (7/54)

Type E (3/54)

Type F (1/54)

10

8

lif •••

li! .,

2

• ,. lul

~ ~

2

•• hI

/if •

lel •

..

.t.

.Ial

•

• • I'JI

------'

10

8

8 16

L--

lce/·

• 101

"""'--~_..........,

14

12

• lul

/tl

• Jul

4

8

2

la/ •

...0--

12

14

2

lal • • a 14

16

------'

F (Bark)

12

F2 (Bark)

Figure 4.

.- • I'J/

F (Bark)

14

• • I'JI

16

... 12 10 F (Bark)

• /01

8 1...-

/ 1 ,.:: u

.

u......

lal • :-.: 8 '---------.;.,....--"--.........., 16 14 12 10 8

----..J

lit ......

/01

1£/ .,... • • •

2

2

...

e•• : .

u...... 6

2

.::. : lu!

lel .~I·

4

UI

lal .-.,. ..\ ~,r _ """--_

8 '--16

.p

2

Type C (7/54)

10

8

F (Bark) 2

8 I...16 14

"""'-----"-_----.J

12

10

8

F2 (Bark)

Emerged vowel systems with six vowels.

4.2.2. Evaluation ofthe emerged vowel system Are the vowel systems that emerge realistic? If in the emerged systems the same types of vowel systems are found in the same proportions as in human languages, they are realistic. The classifications of human vowel systems that were used as a reference were the ones by Crothers (1978) and by Schwartz et al. (1997a). An example of a classification of emerged vowel systems containing six vowels is shown in figure 4. The data in this figure were obtained by run-

Embodiment and self-organization ofhuman categories

425

ning the simulation a hundred times for a given parameter setting (acoustic noise set to 12%). In each run 25 000 imitation games were played. Note that although the frames in figure 4. look similar to the figures of vowel systems shown previously in figure 3, there is a crucial difference. In previous figures, the agents shown in one frame were all members of the same population (and had therefore interacted with each other). In the present figure, all agents shown are members of different populations. The fact that there still is a large amount of similarity between the vowel systems of agents from different populations is a strong demonstration of how self-organization can make populations converge towards similar vowel systems. The emerging systems are realistic. Most of them conform to the universals that Crothers (1978) found for human vowel systems. When the percentages with which the different emerged systems occur are compared to the percentages with which human vowel systems occur, a good match is found as well. Schwartz et al. (1997a) have measured the occurrence of different vowel system types in the different languages in the UCLA Phonological Segment Inventory Database (UPSID, a database based on speech sound data of 451 languages, Maddieson 1984; Maddieson and Precoda 1990). They find 60 vowel systems with 6 vowels. Although their classification is not exactly the same as the classification shown in figure 4, there is good agreement. Of the systems they found in UPSID, 43% is of type A, 20% is of type B, 5% is of type C, 7% is of type D and 20% is of type E. No systems of type F were found, and two of the systems from UPSID (3%) cannot easily be fitted into the classification used here, but are probably of type A. Equally good agreement was found for systems of 4 and 5 vowels and reasonable agreement was found for systems of 7 and 8 vowels (de Boer 1999, ch. 6). It seems that the simulation is capable of not only predicting the most frequently found vowel systems in human language (as was already possible with systems that optimize acoustic distinctiveness), but also of predicting the less frequently occurring vowel systems and, approximately, their relative abundance.

5.

Conclusions

The results of the simulation show that vowel systems can emerge through self-organization in a population of agents whose production and perception is constrained by their embodiment. Although the agents start with an

426

Luc Steels and Bart de Boer

empty vowel repertoire and do not have any constraints on the kinds of vowel systems they can learn, the vowel systems that emerge in the population tend to be symmetric and dispersed, and optimized for acoustic distinctiveness. The model also predicts the types of vowel systems that occur less frequently and their abundance with a reasonable degree of accuracy. This illustrates that dynamics and embodiment play an important role in determining the structure of phonological and phonetic systems. The importance of embodiment both for constraining the space of possibilities and for pushing towards optimality in the task was known already (see e.g. Lindblom, MacNeilage and Studdert-Kennedy 1984) and the computer model presented here provides additional empirical support for this thesis. At the same time, the model clearly shows that embodiment can (and has to be) complemented with the collective dynamics flowing from social interaction. As the imitation success of agents is partly determined by how well the agents' vowel repertoires conform to the repertoires that are used by the other agents in the population, different (sub-optimal) configurations can emerge and be maintained. Some configurations can be considered stronger attractors than others in the dynamical system that is defined by the agents and their interactions. The same universal tendencies that are found in human vowel systems are found in the systems that emerge. This indicates that innate rules and constraints are not necessary in order to explain why these tendencies are present; neither is it necessary to assume that they have to be genetically encoded. More generally (and based on other works such as reported in Steels and Belpaeme 2005) we argue that the universal tendencies found in human categorization, as well as the cognitive systems that use these categorizations for action and decision-making, are a function of multiple forces, and in particular embodiment and collective dynamics. In domains where the statistical structure inherent in the real world is relevant (such as color categories), sensitivity to this structure will also play a role. In domains where there is genuine choice (like driving to the right or left) the regularity is generated by the agents and cannot be explained on the basis of preexisting statistical structure. All this does not exclude the possibility that certain categorizations could not have become genetically assimilated. However, in the case of cultural systems, like language, which undergo relatively rapid change, this seems less likely than is often assumed.

Embodiment and self-organization ofhuman categories

427

Acknowledgement

The Sony Computer Science Laboratory in Paris funds the work by Luc Steels, with additional funding from the OHLL CNRS project, the OMLL ESF project, and the EU FET ECAgents project. The work ofBart de Boer reported here took place at the University of Brussels AI lab and was funded by a GOA project.

References Berlin, B. and P. Kay 1969 Basic color terms: Their universality and evolution. University of California Press. Bowerman, Melissa and Soonja Choi 2001 Shaping meanings for language: Universal and language specific in the acquisition of spatial semantic categories. In: Melissa Bowerman and Stephen C. Levinson (eds.), Language Acquisition and Conceptual Development, 475-511. Cambridge: Cambridge University Press. Browman, Catherine P. and Louis Goldstein 1992 Articulatory phonology: An overview. Phonetica 49: 155-180. Chomsky, Noam and Morris Halle 1968 The Sound Patterns ofEnglish. Cambridge, Mass: MIT Press. Christophe, Anne, Jacques Mehler and Nuria Sebastian-Galles 2001 Perception of prosodic boundary correlates by newborn infants. Infancy 2: 285-394. Crothers, John 1978 Typology and universals of vowel systems. In: Joseph H. Greenberg, Charles A. Ferguson and E. A. Moravcsik (eds.), Universals of Human Language, Volume 2 Phonology, 93-152. Stanford: Stanford University Press. Davidoff, J. 2001 Language and perceptual categorisation. Trends in Cognitive Sciences 5 (9): 382-87. de Boer, Bart 1999 Self Organisation in Vowel Systems, PhD Thesis, AI-Lab, Vrije Universiteit Brussel. Self organization in vowel systems. Journal of Phonetics 28 (4): 2000 441-465. 2001 The Origins of Vowel Systems. Oxford: Oxford University Press.

428

Luc Steels and Bart de Boer

Dresher, Elan 1992 A learning model for a parametric theory in phonology. In: Robert Levine (ed.), Formal Grammar: Theory and Implementation, 290317. Oxford: Oxford University Press. Edelman, Gerald 1990 Neural Darwinism. New York: Basic Books. Fast, Peter W. 1953 Amuesha (Arawak) phonemes. International Journal of American Linguistics 19: 191-194. Herskovits, Annette 1986 Language and Spatial Cognition. Cambridge: Cambridge University Press. Iida, Fumiya, Raja Dravid and Chandana Paul 2002 Design and control of a pendulum driven hopping robot. Proceedings ofInternational Conference on Intelligent Robots and Systems 2002 (IROS 02) Lausanne, Switzerland, 2141-2146. Jakobson, Roman and Morris Halle 1956 Fundamentals ofLanguage. The Hague: Mouton and Co. Kay, P. and T. Regier 2003 Resolving the question of color naming universals. Proceedings of the National Academy of Sciences 100 (15): 9085-9089. Ladefoged, Peter, and Ian Maddieson 1996 The Sounds ofthe World's Languages. Oxford: Blackwell. Lakoff, George and Mark Johnson 1980 Metaphors We Live by. Chicago: University of Chicago Press. Lindblom, Bjom, Peter MacNeilage and Michael Studdert-Kennedy 1984 Self-organizing processes and the explanation of phonological universals. In: Brian Butterworth, Bernhard Comrie and Osten Dahl (eds.), Explanations for Language Universals, 181-203. Berlin: WaIter de Gruyter. Lindblom, Bjom and Ian Maddieson 1988 Phonetic universals in consonant systems. In: Larry Hyman and Charles N. Li (eds.), Language, Speech and Mind, 62-79. London: Routledge. Lenneberg, Eric H. and John M. Roberts 1956 The language of experience: A study in methodology. International Journal ofAmerican Linguistics memoir 13. MacNeilage, Peter 1998 The frame/content theory of evolution of speech production. Behavioral and Brain Sciences 21: 499-548. Maddieson, Ian 1984 Patterns ofSounds. Cambridge: Cambridge University Press.

Embodiment and self-organization ofhuman categories

429

Maddieson, Ian and Kristin Precoda 1990 Updating UPSID. UCLA Working Papers in Phonetics 74: 104-111. NUflez, Rafael 1999 Could the future taste purple? Reclaiming mind, body and cognition. Journal ofConsciousness Studies 6 (11-12): 41-60 Pfeifer, Rolf and Fumiya Iida 2004 Embodied artificial intelligence: Trends and challenges, In: Fumiya Iida, RolfPfeifer, Luc Steels and Yasuo Kuniyoshi (eds.), Embodied Artificial Intelligence, Lecture Notes in Computer Science 3139: 126. Berlin: Springer Verlag. Plaut, D. C. and C.T. Kello 1999 The emergence of phonology from the interplay of speech comprehension and production: A distributed connectionist approach. In: Brian MacWhinney (ed.), The emergence of language: 381-415. Mahwah, NJ: Erlbaum. Rand, Earl 1968 The structural phonology of Alabaman, a Muskogean language. International Journal ofAmerican Linguistics 34: 94-103. Redford, Melissa A., Chun C. Chen and Risto Miikkulainen 2001 Constrained emergence of universals and variation in syllable systems. Language and Speech 44: 27-56. Schwartz, Jean-Luc, Louis-Jean Boe, Nathalie Vallee and Christian Abry 1997 a Major trends in vowel system inventories. Journal of Phonetics 25: 233-253. 1997 b The dispersion-focalization theory of vowel systems. Journal of Phonetics 25: 255-286. Steels, Luc 2003 Evolving grounded communication for robots. Trends in Cognitive Science. 7 (7): 308-312. Steels, Luc and Tony Belpaeme 2005 Coordinating perceptually grounded categories through language. Behavioral and Brain Science. 28 (4): 469-529. Steels, Luc and Rodney Brooks 1995 The Artificial Life Route to Artificial Intelligence. Building Embodied Situated Agents. New Haven: Lawrence Erlbaum. Steels, Luc and Oudeyer, Pierre-Yves 2000 The cultural evolution of syntactic constraints in phonology. In: M. A. Bedau, J. S. McCaskill, N. H. Packard and S. Rasmussen (eds.), Proceedings of the VIfh artificial life conference (alife 7): 382-394. Cambridge (MA): MIT Press.

430

Luc Steels and Bart de Boer

Vallee, Nathalie 1994 Systemes vocaliques: de la typologie aux predictions. These. ICP, Grenoble. Ziemke, Tom 2003 What's that thing called embodiment? In: R. Alterman and D. Hirsch (eds.), Proceedings o/the 25th Annual Meeting o/the Cognitive Science Society: 1305-1310. Hillsdale, NI: Lawrence Erlbaum.

Communication as situated, embodied practice

Wolff-Michael Roth

Abstract Researchers generally are preoccupied with written language as the paradigm of communication. This paradigm makes communication and what is communicated appear as something that exists apart from invariably embodied and situated action. A large part of human communication involves participants in face-to-face conversation. In such situations, participants attend to semiotic resources other than words produced and attended to, including gestures, body positions, and structures in the environment. To avoid the dualism that characterizes much of the scholarship on communication, I provide a dialectical account of human activity, one component of which is communication, itself characterized by the dialectical relation of different modes (speech, gesture, structures in environment). The analyses of an episode from a school science classroom exemplify the central role of the body in communication, which by far exceeds what participants articulate in so many words. Its relationship to environmental features, however, is dialectical, making communication a (materially, socially) situated and embodied practice. Keywords: agency situated practice.

1.

I

structure dialectic, communication, embodiment, gestures,

Introduction

Since the late 1980s, it has become fashionable to talk about knowing and learning in terms of embodied cognition Of, alternatively, in terms of distributed or situated cognition. All of these terms imply that knowing exceeds that what can be found in the brain - viewed as computational wetware - and that has been the preoccupation of psychologists.! Like many others concerned with cognition and development, I had been involved in 1. Etymologically, to know and cognize have the same origins, generally held to have the same root as can.

432

Wolff-Michael Roth

building models of human cognition based on Piaget's work, which came to be associated with information processing. However, once I conducted research in the attempt to understand knowing and learning in real time and particularly the evolution of language, the then valid measures of reasoning ability and short-term memory were unable to predict performances in the real-world settings that I attempted to explain. Theoretical approaches that included or were based on social processes seemed to be much more promising, as the turn to social constructivism in disciplines such as social studies of science (e.g., Latour and Woolgar 1979), anthropology, and education (e.g., Lave and Wenger 1991) seemed to imply. In the course of the 1990s, I had taken hundreds of hours of videotape first in classrooms and then among scientists and technicians at work. It became increasingly clear to me that analyzing only the words they used was insufficient to understand what participants knew, what they communicated to one another, and what they knew to be going on. It was impossible to model communication unless one also accounted for the bodily origin of communication and nature of knowing, on the one hand, and for material cultural processes, on the other. There was a problem, however, which I understood only afterward: the individual (body) and the sociality of culture cannot be thought together in dualistic models, because each approach ends up reducing one to the other. It was at then that dialectical approaches to communication became salient in my work. In this chapter, I exemplify the kind of communicative action that I found among students and professionals of science. Drawing on an episode from a science classroom, I exemplify the (socially, materially) situated and embodied nature of communicative action. I begin by articulating a dialectical framework for activity and action that allows me to treat communicative action and situated cognition in a non-dualist way. I move on to provide a brief description of the ethnographic and analytic background, before providing an extended analysis of one typical episode.

2.

Integrating body, mind and culture

Body, mind, and culture can easily be integrated into the same unit within a dialectical approach (e.g., Leont'ev 1978). In the following, I first sketch a general theory resting on the dialectic of agency and structure, and then articulate communicative action as one aspect within this framework.

Communication as situated, embodied practice

2.1.

433

Agencylstructure dialectic

The embodied, situated, and cultural nature of cognition in general and communication of a conscious agent more specifically can be understood within an agencylstructure dialectic (Sewell 1992). Agency denotes the capacity to act. It is immediately clear that there is no agency without structure, material or cognitive. For living organisms, structure is everywhere. Bodies are structured in a material sense, exhibiting particular articulations, symmetries, and sizes, all of which mediate the kind of movements organisms are capable of. The world in which we live is structured, too, with its characteristic forces and articulations among things, but structured differently for different organisms (von Uexkiill 1972/1928). Scholars attend less to the structured ways in which we see, hear, feel, move, and do things, including thinking about them. These structured actions, including perception, have arisen from our bodily engagement with the world (Merleau-Ponty 1945; see also Gallagher this volume; Lindblom and Ziemke this volume). The very cognitive structures that allow us to perceive the world in structured ways are themselves a consequence of interactions of a structured body with a structured world (Bourdieu 1980). But there would not be the need for speaking of structures in the world unless there were organisms to which these structures are behaviorally relevant. To distinguish the two forms of structure, the notions of objectively given (sociomaterial) resources and (mental) schema have been proposed (Sewell 1992). At any moment in activity, the given structures, i.e., sociomaterial resources and schema, enable and constrain actions. In any consideration of human activity, the two forms of structures have to be considered dialectically. Any object of activity simultaneously exists as a vision or hypothesis of the forthcoming product (which implies consciousness), a set of material conditions including materials and tools, and knowledge of sociomaterial conditions of the productive activity (Saari and Miettinen 2001). Therefore, neither sociomaterial resources nor schema individually or together determine action; rather, actions emerge in response to the contradictions embodied in the unit of analysis, which comprises the identity of (the non-identical) sociomaterial resources and their appearance to conscious mind in action. Any analysis of activity simultaneously has to consider three levels, which integrate the bodily and cultural nature of engagement with the world: activity, action, and operation (Leont'ev 1978). Activities, such as schooling, farming, or researching are always collective endeavors but

434

Wolff-Michael Roth

concretely intended and realized in varying form, and are directed toward some generalized (conscious) object or motive. Concretely realization occurs by means of sequencing of particular conscious actions, each directed toward a specific conscious goal formulated by the acting subject. While researching, a professor might decide to measure a certain variable or to do an ANOVA. The relation of activity and action is dialectical: the activity determines the sense of an action - an ANOVA done by a pollster is likely to have a different sense than the professor's - but actions concretely realize the activity. Properly sequenced smaller units, operations, constitute actions; operations are unconsciously produced responses to conditions, such as when the professor keys the statistical model into the computer but is not consciously aware of hitting particular keys but focuses on the equations appearing on the screen. The relation of action and operation is, again, dialectical: the action is a generalized referent for the proper sequencing of operations, but operations concretely realize actions. Operations have been shaped in and are the outcome of previous interactions of the body with the sociomaterial world, they constitute a concrete realization of cultural possibilities; they are a form of unconscious collective consciousness, that is, an embodiment of culture. This approach corresponds to that taken by Gallagher (this volume), where actions are goal-directed and conscious but realized by unconscious operations brought about by body schema. Communication is but one form of actions that develop the current activity, without necessarily being about the activity.2 This approach therefore contrasts the claims of those scholars (e.g., Zlatev, this volume) who fail to recognize that (a) language-in-use (parole) is most frequently produced without prior planning (i.e., conscious supervision) and frequently does not follow the conventions grammarians have adopted and (b) communicative action and thought are frequently produced in real time rather than reproduced through speech.

2. Space limitations do not allow me to elaborate the relation between resources and schema on the one hand, and activity, action, and operation on the other hand. Suffice it to say that resources exist externally to the agent, at the interface of activity and action; and schemas exist internally to the agent, at the interface of action and operation.

Communication as situated, embodied practice

2.2.

435

Communicative action

Communicative action is more than sending information (words) from a sender (speaker) to a receiver (listener), which would require equal tuning of the two participants to code and encode concrete material sound parcels. Because of the dialectical relation material resources and schema, the identical tuning of speaker and listener is never given in human communication. A dialectical approach to human activity leads us to a different conception of communication. Each action produces a change in the current setting, thereby providing changes to the semiotic resources available for the conduct of activity in general and face-to-face communication in particular. Speech act theory allowed us to understand that utterances, too, have effects in the setting, they constitute resources that can be part of any future action, current or subsequent speaker. Artifacts (Streeck 1996), diagrams (Goodwin 2000), and other structured aspects in the setting, including various forms of gestures (e.g., Bavelas, Chovil, Coates and Roe 1995), pointing and body orientation (Hindmarsh and Heath 2000), and pitch (Goodwin, Goodwin and Yeager-Dror 2002), constitute resources produced and used by speakers and listeners alike. To understand communication in everyday settings, all of these structures need to be accounted for simultaneously (Roth 2004). These different structures that are apparent in communicative action (words, gestures, things pointed to) are not simply additive but dialectically related. Thus, for example, speech and iconic gestures constitute two ways in which thought is not merely made public but actually finds its expression as productive activity (McNeill and Duncan 2000). Speech and gestures are dialectically related because they constitute different expressive forms for the same intentional stance toward the world. This stance is not some idea somehow behind or underlying and therefore expressing (translating) speech. Rather, it is my bodily me that gestures and speaks, and thereby signifies thought and intention for others and myself (MerleauPonty 1945). Making sounds, speaking individual words, gesturing, nodding the head, turning the body and so forth are operations that together constitute a communicative act (Mikhailov 1980). The production and recognition of these operations always requires a concrete body.3 But each communica-

3. Gallagher (this volume) and Lindblom and Ziemke (this volume) elaborate on the link between production and recognition of body-produced movements

436

Wolff-Michael Roth

tive act does not have a sense in itself; rather, sense emerges from the relation between the speech act and the overall, collective activity. Communicative action is therefore at the same time a concretely embodied (intrapsychological) and sociocultural (inter-psychological) phenomenon; its meaning emerges from the simultaneous grounding of action in cultural activity and bodily operations. Conscious action therefore lies at the interface between the unconscious operations that realize them and the cultural processes that give them their sense.

3.

Ethnographic context

The data analyzed here derive from an in-situ naturalistic study of communication in a seventh-grade science classroom where students learned about the physics of simple machines largely by designing machines. To make the learning environment as authentic as possible, the specifications for the design tasks were articulated in a call for proposals. As part of their submission, the students included paper-and-pencil sketches (e.g., Figure 1) and prototypes. They subsequently presented their prototypes to the student collective, which instantiated a form of peer review. This curriculum therefore provided many opportunities for students to engage in conversations with others, both during the design of the prototype and the review sessions. Two cameras and two tape recorders were used to record smallgroup and whole-class conversations. To begin with, all recordings were roughly but word for word transcribed to capture all utterances as completely as possible under the prevailing conditions (e.g., with students hammering and using power tools). In a second pass, the VHS videotapes were digitized (Macintosh iMovie); offprint images capturing characteristic moments were imported into the transcripts. Each lesson subsequently was broken into episodes, which were exported to QuickTime versions. These digitized clips were then analyzed using Peak™ DV 3.21, which allows the graphical representation of waveform (for determining time, relative loudness, and duration) and its Fast Fourier Transform (for determining pitch changes). Individual utterances were timed with an accuracy of ±O.OOl seconds but rounded to the

(signs) and the role of mirror neurons to bring about this integration between perception and action.

Communication as situated, embodied practice

437

nearest 10 milliseconds and coordinated with the frames that allowed timing to 33 milliseconds (30 frames/s rate of the VHS video).

4.

Language and body in design activity

Linguistic analyses generally are concerned with language independent of communicative praxis. This leads directly leads to the grounding of language, that is, its connection to things and situations, which is continuously realized in practice but has been a problem for traditional cognitive science (Harnad 1990). Furthermore, most linguistic studies are hardly relevant to speech, for the language of writing is a fundamentally different cognitive phenomenon than language in practical face-to-face communication (Ong 1982). Here, I am concerned with language as it appears in face-to-face communication alongside other semiotic resources humans produce in interaction and for interactive purposes - body orientation, body movements, and hand gestures. Following the presentation and brief gloss of a 23-second episode, analyses are provided of the concrete role played by human bodies in communication.

4.1.

A moment in design activity

The following episode was recorded at the moment when the three girls were deliberating how to go about building the tower that would hold an elevator, the first element in their design of a Rube Goldberg machine (Figure 1).4 Leanne (far left in offprints) had previously explained that they needed to build a two-sided tower; Amanda (far right) and Bella (center) went along so that it appeared that there was agreement on this design. Following Leanne's suggestion, the three had decided to cut the wood needed to build that part of their design. At one moment, Leanne talked about cutting two pieces of wood for the two sides of the structure. This

4. Rube Goldberg was a cartoonist who drew complicated machines that accomplished the most mundane everyday actions. Constructing a Rube Goldberg machine is a popular design activity of middle school students around the world. More information is available at http://www.rube-goldberg.com/.

438

Wolff-Michael Roth

was when the following episode began. 5 (I first provide the raw transcript and then a verbal description of the events.)

Figure 1.

The students had decided to have a tower and a chute as the frrst two elements in their Rube Goldberg machine, a complicated device that would accomplish the simple task of feeding a cat. The episode concerns the construction of the tower part.

5. To get a better sense for the changes in body positions and hand gestures involved, readers should visually scan the images before reading the transcript. The following transcription conventions have been used in line with traditional conversation analysis: (0.41) : time in seconds; [ttwo sides] : square brackets enclose speech overlapped by gesture or other body movement; [t((r~ lH moves up)) : description of gesture (rH, ill = right and left hand, respectively), body movement; * : the asterisk aligns speech and video offprints; = : equal sign shows latching, that is, two utterances are not separated by the normal pause; .,?! : punctuation is used to indicate speech features, such as rising intonation heard as a question, or falling intonation to indicate the end of an idea unit (sentence); - : the n-dash indicates stop in utterance without voice inflection indicating end of idea unit.

Communication as situated, embodied practice

01 Bella: 02

Okay, which two sides? * (0.47)

03 Leanne:

Okay, you know [\how we are doing that thing where the el]\bevator thing comesh bup* [l(body rotates to the left, rH places saw on table)) [z((rH moves upward, IH slightly upward)) b((begins turning upper body,

04 05 Leanne: 06

07

08 09 10 Bella: 11 Bella

12 13 Leanne:

14 Bella: 15 Leanne:

(0.43)h b((head turns toward board to right, rH points to board)) [4these * are the sides] [4((rH single stroke along the board)) [sthat like keep it] * [6standing]. [s((body turns to left, both hands come up palms facing one another to pinnacle oftrajectory)) [6((two-handed gesture halfway down, slightly up, then completely down)) (0.59) hO.60] h((two-handed up down gesture, at half height)) [8 The two sides?] * [8((two-handed up gesture)) [9(0.88)] =[9((nods three times, at the end, turns to her right toward the piece ofboard)) [\oUh, okay.] [1O((turns to her right toward the piece ofboard))

-

439

440

WolfJ-Michael Roth

16 Leanne:

[111 think it'll work. But if it doesn't, I _"-:~M!I" know what we can[lI((PicAJ up saw, begins sawing))

17 (0.42) fix how we can fix it. 18 19 Amanda: Don't we need four?* 20 (0.56) 21 22 23 24 25

Leanne: No,justAmanda: Just two. Leanne: Two.

(0.44) Leanne:

26 Bella: 27 Leanne:

[12Right?]*

[12((turns upper body, head to gaze at Bella)) =[13((nods twice))] 11 [14((turns head to gaze at saw and wood))

At the beginning of this episode, Bella, apparently confused about just how the part was to look, then asked, "Okay, which two sides?" (turn 01). A brief pause followed Bella's question, before Leanne began to answer, "you know how we are doing that thing where the elevator thing comes up" (turn 03). While uttering this description of the current state of affairs, she turned her body toward Bella, laid down the saw she had been operating, and moved her right arm and hand upward (turn 03). In the ensuing pause in the verbal articulation, Leanne turned to her right, her hand moved forward initially pointing to the board on the table, and then moving her hand about 5 centimeters along one of the edges. It was precisely at the place where she had previously shown with the saw where and how they needed to cut the word (Figure 2). Contemporaneous with the gesture, she uttered, "these are the sides" (turn 06). She oriented toward Bella again and, in a continuous motion, produced two two-handed up-down hand movements, palms facing, suggesting two pieces facing one another, while uttering, "that like keep it standing" (turn 07). After a longish pause, Bella mimetically reproduced Leanne's doublehanded gesture, before repeating the gesture all the while uttering, with rising pitch, "The two sides?" At the verbal level, there followed a longer, 0.88-second pause (turn 10). But Leanne began the first of three head nods just as Bella had completed her utterance (turn 13). That is, as a turn, her head nod latched onto the ending of Bella's turn. Bella responded with a sign of understanding and idea completion (falling pitch), "Uh okay,"

Communication as situated, embodied practice

441

while Leanne turned to the piece of board from which she had earlier indicated the intention to cut two narrow pieces. She explained that she thought her idea would work, and ifnot, she knew an alternative (turns 1618). She had begun to saw and would not stop for the remainder of the episode.

Figure 2.

The iconic gesture produced here with the saw moving along the board where a cut was to be made was reproduced later in an episode analyzed in this chapter.

At this point, Amanda, who had sat on the table next to her two peers, entered the conversation for the first time in this episode. Up to this point, she had faced away and now, turning her head only slightly to her right, asked, "Don't we need four?" (turn 19). Continuing to saw and interrupted by Amanda's own articulation "two," Leanne responded, "No, just two" (turns 21,23). After a brief pause and without stopping to saw, Leanne turned her upper body (slightly) and head to gaze squarely at Bella in the process uttering with rising inflection, "Right?" (turn 25). She had barely completed when Bella began to nod twice, upon which Leanne turned her head back to look toward the board that she was working on. In terms of the overall project, the episode was important in that what had appeared to be a common design vision of the tower and the elevator it subtended did not exist but was brought into alignment. Although the three girls previously had talked about and seemed to agree on the design, and although Leanne had explicated the two-sided version of the tower once before, it was apparent that both of her two peers wondered in the course of this episode why she wanted to cut two pieces of board rather than some other number, including the four Amanda articulated. This alignment, however, was not produced merely by talk. Rather, pointing (deictic) and iconic gestures, head nods, body turns were important aspects of the communicative exchange. Without participants' attention to the body of the respective other, they would find it difficult to produce the alignment achieved here. The latter part of the episode exemplifies this: Amanda, not

442

Wolff-Michael Roth

having had perceptual access to the gestures and body orientations of the other two girls, did not understand why there ought to be two sides to the tower despite the fact that Leanne just explained it. Thus, Amanda began the second part of the episode with fundamentally the same question that Leanne had just answered in response to Bella's query. In this episode, therefore, alignment of at least two of the three participants with respect to a future state of their activity was achieved. The episode had begun with a query about the number of sides for the tower, although this issue seeme,d to have been settled during a previous interaction. Twice during the episode, Bella aclmowledged the two-sided version of the tower, once by means of an iconic gesture and confirming utterance (turns 10, 14) and then by means of a head nod. As Amanda acquiesced (turn 22), we have no indication about her personal vision of what their collective project was to be. In the following, I analyze different aspects of this episode, which bring to the foreground different dimensions of the relationship of body and language in action.

4.2.

Orienting the body and gesturing mediate turn taking

Communication is an interactive phenomenon requiring the analysis of pairs of turns. A body of studies in conversation analysis provided an understanding of mostly audiotaped conversations (ten Have 1999). Whoever speaks, holds the floor; others normally do not speak at the same time. However, participants do not own pauses in speech. If the current speaker has the intention to continue, he or she has to provide other semiotic resources that indicate this intentionality. However, videotaped interactions reveal that in face-to-face situations, there are many other semiotic resources that participants bring to in play that mediate turn taking. In this situation, those who want a turn at talk may provide bodily and gestural resources that indicate this intentionality (Goodwin 2000). Mutual awareness of the interaction participant's body orientation, perceptual access, and (hand, head) gestures mediates the rules of interaction in ways that traditional conversation analysis could not reveal. Videotaped materials, as used here, reveal the central role of the body in face-to-face interaction. The role of body orientation and perceptual access was quite apparent in this episode, which had started with Bella's question about the nature of the tower sides. Leanne responded, followed by a renewed query and affirmation that there would be two sides. During the interaction, both had

Communication as situated, embodied practice

443

produced gestures in the context of changing body orientations. Amanda had been sitting on the table facing away from the two. Thus, she had no access to all those parts of the communicative interchange between her peers other than the verbal utterances. In particular, she had not been in the position to perceive either Leanne or Bella's two-handed production of an image that pertained both to the role of the two pieces of board that they were in the process of cutting and the design drawing. Her question "Don't we need four?" essentially repeated Bella's question about the relationship between raw materials, design drawing, and their vision of the emerging artifact. The words alone had been insufficient to bring Amanda in alignment with what was to be the collective vision for the unfolding design artifact. Pauses are interactive resources that play a particular role in the situated evaluation of communicative action (Bourdieu 1980). As a pause in talk is lengthening, it increasingly becomes a resource for any participant to begin speaking. Following a question, however, a pause becomes an increasing liability for the addressee, as not answering can be heard as an intentional exclusion of the querying person from interaction. This happened repeatedly in this group, where Leanne did not answer queries that Amanda directed to her, whereas she answered all questions that Bella posed. Thus, the lengthening pause following Amanda's question (turn 19) opened up the possibility that this situation, too, would be one in which Amanda did not receive a response. Perceptual access to the interlocutor's body changes the rules identified in conversation analytic studies. Body movements required by the interaction may take time, but the very fact that they are enacted articulates communicative intention. So actions other than speech also articulate the intent to contribute, and therefore that the current turn lies with the acting person. For example, after Leanne first articulated the sides of the tower ("that thing where the elevator thing comes up"), there was a pause at the verbal level. However, at that time she turned her body and moved arm and hand nearly 90 degrees around to her right. The motion ended in the pointing gesture and led into the gestural and verbal articulation of the target as "the sides." Later (turn 13), she nodded three times in response to Bella's query for confirmation of the two-walled nature of the future artifact. The head nodding was therefore a different form of turn taking than what has been available to traditional conversation analysis. In the head nod, the body articulated an affirmation, which in a different spatial arrangement might

444

Wolff-Michael Roth

have come forth through an activation of the vocal cords. The relationship between pause and orientation was also evident in the following situation. In the final part of the episode (turns 25-27), Leanne had answered Amanda's question about the number of sides of the tower. There followed a pause. Leanne then uttered with rising pitch, "Right?" (turn 25) simultaneously turning her head to Bella (still continuing to saw). Leanne thereby held Bella accountable for not having confirmed the answer, not only in asking for a confirmation but also, by turning her gaze in the direction of Bella, designating her as the addressee of the question. Bella did not respond verbally, but, immediately following the end of Leanne's head movement, nodded twice. Without a word, Leanne returned her head to gaze again at the saw, board, and sawing action. Bella had answered with a double nod, which would not have made sense had she not been aware of the fact that Leanne could see her. Leanne's subsequent turn was an acknowledgment that the request for confirmation had been satisfied, and therefore that she had seen the nods. Bella, too, could know that Leanne was satisfied with her non-verbal response, for she could have expected a conversational repair if this had not been the case. A converse case existed earlier in the episode when Bella acknowledged the confirmation received to her query about the two sides (turn 14). Bella was moving her head downward as part of a grooming gesture and Leanne was turning around. Bella therefore was forced to utter "Ub okay" rather than enacting acknowledgment in another way (head nod, eye movement, literal "thumb up"). First, because Bella turned her gaze away from Leanne, she was not able to see whether her peer was in the position of seeing her gesture. Second, because Leanne was moving, she could not have seen the gesture either. It is likely that the verbal utterance was forming at the same moment that the downward head movement was. In yet another instance (turn 10), Bella had produced a two-handed gesture while looking down at her hands; she reproduced the gesture this time uttering questioningly, "The two sides?" while simultaneously raising her gaze to Leanne's face. This provided a resource for Leanne to respond by nodding but did not require a verbal equivalent, which, here, was not provided. These analyses exemplify that the mutual bodily orientation of the participants mediated how communication is articulated, that is, verbally or gesturally (e.g., hand, arm, head). Participants are not only aware of their own positions and orientations but also of the positions and orientations of their interlocutors. Speech is the preferred mode if there is evidence that the listener does not have access to bodily gestures and that both speaker

Communication as situated, embodied practice

445

and addressee are aware of this. The speaker may produce gestures when the listener has access, even though the speaker herself does not face the listener. These rules also mediate the pauses between verbal utterances. If participants have full perceptual access to one another, a lot of the communicative work is generally done by producing non-verbal semiotic resources, such as indicating intent to take the turn, articulating ideas, orienting the communicative partner toward, and so forth.

4.3.

Pointing

Pointing is classified on the spontaneous-idiosyncratic-movement end of a gesture continuum that reaches to fully developed linguistic systems (e.g., American Sign Language) on the other end (Kendon 1988). Acts of pointing occur in situations that involve at least two participants, one of whom is, in the act of pointing, producing resources for the joint establishment of a particular space as a common focus (Haviland 2000). This common focus constitutes a resource for the organization of subsequent situated action and cognition. Pointing is jointly contextualized by semiotic resources that include, at a minimum, a body that visibly performs the act, a dialectically related speech act, the material properties of the in pointing targeted entity, the orientation of relevant participants toward one another and toward the target space, and the larger activity within which the act of pointing (itself only part of the communicative act) is embedded (Goodwin 2003). In this episode, there were repeated instances of verbal deixis ("that," "it," and "these"). One of these instances, "these," immediately followed a deictic gesture (turn 05) and overlapped with an iconic gesture whereby the hand moved along the piece of wood (turn 06). Such acts of pointing constitute prime examples where the double nature of the object in activity becomes apparent: from the speaker's perspective, it appears twice, once as material and once in perception. The body oriented itself toward an entity other than itself (piece of wood), which it was perceptually isolating from a more inchoate ground. The body, here, repeatedly pointed to this entity, verbally by means of the utterance "these" and gesturally by the stretched arm and edge of the hand coming to rest on the piece of wood, the current object of attention. The deictic term "these" locates the entity pointed to near the speaker (Hanks 1992), in the present case, within reach. The deictic term "these" not only instructed Bella to attend to something beyond the speech act, the

446

Wolff-Michael Roth

targeted material structure, but also articulated that what she pointed to as a multiplicity (i.e. "these" rather than "this"). Furthermore, by using "these" rather than an articulation of a point in space (here, there), Leanne formulated the target as material things rather than spaces or sounds. There existed, however, a contradiction: there was only one piece of wood, and Leanne' s subsequent iconic gesture produced the image of one rather than two cuts. But her pointing gesture was not only to the piece of wood but also to the previous interaction where she had gesturally enacted twice the iconic gesture of two cuts. Because these previous gestures were part of the girls' interactional history, they constituted production resources "once-removed," which make connections over larger chunks of the ongoing activity (Erickson 1982). This pointing (turn 05) therefore indexed the earlier sawing gesture with the edge of the right hand at the same place where she had earlier moved the saw (Figure 2). In a continuous motion, the pointing configuration then moved along a line about five centimeters from the edge where she had earlier indicated that they needed to cut (Figure 2). Although most classification systems categorize deictic and iconic gestures differently, there are situations where the same movement enacts both types, for example, when the pointing body part traces out some shape. Thus, Leanne's initial deictic gesture toward the piece of wood (turn 05) also became an iconic gesture (turn 06), because its trace took the shape of the projected linear cut. Here, movement along a line that separated the object pointed to from what was to be taken as diffuse ground enhanced salience. The single iconic gesture along the board constituted a reproduction of an earlier gesture showing where she was to cut the board (Figure 2). In fact, at that earlier moment, Leanne produced the gesture twice, holding the saw where the cuts were to be done. That is, the distance of the saw from the edge doubled for the second movement in the first pair of gestures; in the second pair, she produced for practical purposes identical gestures at either end of the board, again indicating that they needed two pieces of board for the tower part. Despite the iconic relation between the earlier and present gestures, the contradiction between the gestural (one) and verbal articulations (more than one) was only resolved in the subsequent iconic gestures (turn 07) that articulated a vision of two upright boards, as it was further materialized in a subsequent episode (Figure 3). From the listener's perspective, the associated task is to perceptually isolate the entity pointed out by the speaker's body (orientation, deictic gesture). This is not necessarily an easy task, for even in the sparsest set-

Communication as situated, embodied practice

447

tings, it may not be evident to the listener what the salient feature is for the speaker. Different speakers may use the same word for denoting different material entities or may use different words for denoting the same material entities (e.g., Roth 1995). Other, semiotic resources are required as constraining factors to assist the listener in perceptually articulating the target structure. Such constraining factors are inculcated culturally marked object or field, or may be found in the history of the activity and associated talk. In the present episode, the perceptual field articulated in Leanne' s pointing was not culturally marked, but marked by having been the focus of previous interactions. By turning to the piece of wood, pointing it out as a resource, and then returning to the initial body orientation facing Bella, Leanne actually brought the pieces of wood into the space between them, where she gesturally enacted a vision of the future artifact. In the present case, Leanne had already used repeatedly an iconic gesture to point out where and how many times they needed to cut for getting the required pieces in the building of the tower part. The nature of the entity pointed to was further constrained by the talk that immediately followed, where Leanne's two-handed palm-facing gesture articulated a vision of boards facing one another (turn 07, Figure 3).

Figure 3.

The two-handed gesture, which had fIrst appeared in a conversation about what the tower would look like is here reproduced but here in a holding movement, fIrst by Leanne (a), then by both students (b).

In the present situation, Leanne was not just pointing somewhere, but she rotated her entire body. The sequence of the first four offprints in the transcript (turns 01-07) shows that she had first rotated away from an orientation to the board (turn 01) in order to face Bella (turn 03), then rotated back to face the board while pointing to it (turn 06), and then rotated again to face Bella (turn 07). She was aware that Bella shifted her gaze from facing Leanne (turn 03) toward the board (turn 06), the endpoint of the gesture,

448

Wolff-Michael Roth

and then back toward the space between them, where she could see both Leanne into the face and her two-handed gesture. This intentional pointing therefore was instantiated not merely the hand but in fact by the entire body (person) under the auspices of her knowing that the listener Bella was following her changing orientations.

4.4.

Iconic gesturing

Iconic gestures belong to the same class of gestures as pointing. They include those hand-arm movements that bear a perceptual relation with concrete or projected entities and events that the speaker is oriented to (e.g., McNeill 1992). In contrast to a deictic gesture, however, an iconic gesture is itself the focal entity. From a speaker's perspective, the iconic gesture articulates an immanent or unfolding sense that stands in a dialectical relation both with concurrent speech and the material entity in the setting so articulated (Roth 2004). Iconic gestures are important semiotic resources in interaction (e.g., Ochs, Jacoby and Gonzales 1994). Research in school science classrooms showed that for students of all ages, iconic gestures are important aspects of communication, especially while the learners are becoming familiar with the phenomena at hand and do not yet have a stable (viable) language to talk about them (Roth 2003). Iconic gestures also were important interactional resources continuously produced and reproduced in this episode. The first iconic gesture occurred while Leanne reminded Bella that they were currently working on the first part ("thing") of their Rube Goldberg machine where the elevator ("thing") moves (turn 03). The rising right hand palm facing down evoked the image of something moving upward, like the elevator in the design (Figure 1). More so, while they had talked about how to realize the tower, both Leanne and Bella had gesturally followed the vertical outline of the tower, indicated by the orientation of the hands. These gestures therefore bore iconic relations to other material forms, apparent as tower and elevator in the design drawing. At the verbal level, the upward movement of the hand was iconic to the articulated "comes up." It is a concurrent gestural and verbal concretization of the updown or verticality schema (Johnson 1987, Roth and Lawless 2002b). In this case, the timing between the verbal and gestural modality was precise (simultaneous), consistent with the contention that gesture and speech are simultaneous formulations of an immanent or emerging idea by one and the

Communication as situated, embodied practice

449

same body (McNeill 1992). However, the left hand only moved upward slightly, about one-quarter the way of the right hand, and given Bella's gaze direction, probably invisible to her. In the same part of the episode, one can also note that the idea had begun to form earlier. While uttering that they were "doing that thing," Leanne still had the saw in her hand, nearing the end of the body rotation. She placed the saw and then, in a continuous motion, moved her right hand up just as the utterance evoked an upward moving elevator. That is, the placement of the saw was in preparation of the subsequent gesture. But even before uttering the locative "up," her body and hand already began to rotate sideways toward enacting the deictic gesture that pointed to the wood to be used in the tower construction, thereby preparing the subsequent gesture. After the intervening (already analyzed) pointing episode, Leanne' s whole body turned back toward Bella and, in a continuous motion, both hands moved upward while she uttered, "that like keep it" (turn 07). The hands then descended halfway to the table, but, as if hesitating moved three-quarter the width of her hand upward, before completely descending to the table level; in the process, she completed the word "standing." The two-handed gesture produced the image of two upright boards facing one another, which, according to the verbal part, "keep it [elevator] standing." The two-handed gesture not only produced an iconic expression of the design vision, but also produced (a) an image of the two boards that provided the scaffold for the elevator in their design drawing and (b) an image of the boards that she had just pointed to in their raw-material state. Through the two-handed gesture, the design drawing (Figure 1) was coming alive while simultaneously implicating the raw materials to be used: it was a bodily realization of the vision. Later on, very similar hand movements included pieces of wood now standing for themselves as the "sides" (Figure 3a). At that point, the design was even further concretized but not yet an independent artifact: Leanne's body was still implicated by holding the pieces. Only subsequently, after adding more materials and using fastening resources would the emerging design become self-supporting and exist independent of the body. In this situation, we observe an opposite movement to my previous research on gesture in learning science (Roth and Lawless 2002a): here, ideas and fleeting images are initially concretized in body position and hand gestures whereas the inverse was observed in learning by experimenting.

450

Wolff-Michael Roth

At this point, Leanne had apparently ended her response to Bella's query. There was a long pause in the talk. During the second half of this pause, however, Bella reproduced a mimetic image of the two-handed gesture Leanne had just completed - and, similarly to her peer, used the same hand movements to hold up the two pieces of wood (Figure 3b). She then reproduced the same two-handed gesture uttering with increasing pitch, "The two sides?" (turn 11). Here, although the hands mimetically reproduced Leanne's gesture, thereby confirming that she had perceived it, the verbal utterance indicated uncertainty. Why should there be uncertainty? What must have appeared in doubt was the relation to the diagram, that is, their ongoing activity and its motive to construct a Rube Goldberg machine, with an elevator as its first part. That is, from the receiver's part, Leanne's production was intelligible as such (Bella reproduced it), but she had not yet found the relation Leanne' s gestures bore to the envisioned object of activity, partially concretized in the diagram. Facing Bella, who looked straight at her, Leanne nodded three times; she did not produce a verbal affirmation. As shown previously, Bella was in a position where her sign of affirmation had to be produced verbally. Leanne then turned around to continue where she had left off. Leanne had articulated her version of the design in response to Bella's query about the sides involved in the elevator tower. Thus, although they had talked about this before, that is, although Leanne had articulated this vision in a previous episode, there was apparently a lack of alignment or common ground between the two. Bella's query articulated this uncertainty about the nature of tower in what was to be their common design. The iconic and mimetic character of Leanne's two-handed gesture thus constituted a solution to the problem of establishing common ground with respect to an initially ephemeral vision of the object of activity, which was increasingly concretized in the unfolding of the design artifact. Bella enacted an independent concrete version of what is taken to be the common object at hand, always under revision in future action.

5.

Body, mind and culture

In this chapter, I articulate and exemplify the embodied and situated nature of human communication drawing of a non-dualist, dialectic framework that integrates mind and culture, body and mind, speech and gesture, materiality and conscious nature of human experience, and so on. The analyses

Communication as situated, embodied practice

451

show that the separate, interlocking displays of conversational participants form a unit that is greater than the sum of its parts, a particular kind of participation framework. Bella and Leanne treated each other's bodies as resources in the ongoing production of intentional action; Amanda, though she was close and overheard the two, did not have access to the same resources, as she faced away. For conversation participants, the body is a complex resource that enables different kinds of articulations that stand in a dialectically relation to, and therefore elaborate one another. The body therefore is a different type of entity than the piece of wood Leanne pointed to, the saw she used, or the paper-and-pencil diagram they had collectively produced, although it is of the same material kind. 6 Participants in communicative interaction must not only attend to multiple perceptual arrays (body orientation, gesture, speech, entities in the setting), but also arrays that are significantly different in their structure. In the featured episode, there were repeated instances of thoughts in the process of forming concurrently with the bodily productions (speech, gesture, orientation). Thus, Bella had already produced a two-handed gesture that enacted the image of the two-walled tower before she began to utter the question, "The two sides?" while simultaneously reproducing the twohanded gesture. In the first instance, one sees the question forthcoming, not yet formed in its entirety; it only became the full interactional resource when both gesture and speech aligned in the articulation. In fact, the first hand movement had a shorter trajectory, as if Bella had stopped it to reproduce it in full simultaneous with the utterance. This process was not planned, as gesticulations are generally produced unconsciously. The continuous unfolding of a thought was also noticeable in Leanne's performance of her response to Bella's initial question. It had started with a reference to the diagram, continued by articulating the pieces of board that they were currently working on, and then articulated the design vision that brought the two earlier pieces together into one unit. The analyses showed how her entire body performed this response. The analyses also showed that the body, through forms of both spontaneous and idiosyncratic gesture, is tied to its environment so that communication is not only an embodied (production of speech, gesture) but also a situated (distributed) phenomenon. In the deictic gesture, the identity of the

6. Gallagher (this volume) elaborates on the differences between the perceptions of material objects and human bodies (that of others and one's own) and the implications this has for intersubjectivity.

452

Wolff-Michael Roth

perceptual (intra-psychological) and material (extra-psychological) aspects of the object becomes apparent and articulated. It is in a reflexive movement that this identity is destroyed, that is, that subject becomes separated from object and object from subject; the reflexive movement, however, only returns the idea or thought but not experience of a body (MerleauPonty 1945). The body is therefore not only a different type of entity, but also it cannot be the object proper in the phenomenological or activitytheoretic formulation. The proper analytic starting point is the dialectical unity of activity - bodies, thoughts, experiences, and even emotions and motivations emerge as the effects of the subject I object dialectic that constitutes the activity (Roth 2006). The analyses also provide a hint at the fact that body, mind, and culture are inextricably related and cannot be reduced to one another. Thus, each utterance, gesture, body orientation, and so on produced by one participant not only provided a resource for future action to herself but also to the other; each communicative act was explicitly produced for the other. That is, the idea of a semiotic resource inherently requires the other participant to be aware of the semiotic potential of these resources. At the moment that the body articulates itself by producing semiotic resources, these already have to exist for the recipient other. An individual bodily me that articulates itself has to draw on resources that are already intelligible by the other and therefore part of the cultural possibilities. This, however, should not come as a surprise, if we recall that operations (and therefore schemas) are unconscious collective consciousness. All actions are therefore synthesized from operations, the concrete realizations of cultural (generalized) possibilities, and therefore are simultaneously the person's and the collective other's. The present analyses of one brief event testify to the complexity of communication, which cannot be reduced to language. The analyses articulated communication as an embodied and situated phenomenon and showed how ideas, questions, and responses emerged in real time. The episode analyzed here was not different than those preceding or following it during the six-hour project, or, for that matter, any other design activity by these students or their peers. If 23 seconds of situated activity are already so complex, how much more complex will be entire forms of activities that stretch over days, months, years, and generations? Because communication, ideas, questions, and responses are distributed across different bodily modes of expressions and across the salient aspects of the setting, the structure of thought cannot be linguistic structure. Thought cannot be

Communication as situated, embodied practice

453

reduced to language; nor can any other form of expression be reduced to the body. Rather, thought exists in the dialectical unity of all forms of expreSSIon. The present analyses make salient the special role of the body, seat and source of agency, in design activity, constituted in the unfolding ephemeral ideas into concrete artifacts that we had also observed among environmental activists (Lee and Roth 2001). The body was the first site where ideas took shape, where they were articulated; it was then augmented by the introduction of materials and subsequently emerging stages of the artifact. Still later, the body removed itself as additional actions allowed the artifact, the concrete rendering and development of the initial ideas, to take on a life of its own. Communication was but one form of action that moved the unfolding activity along its trajectory toward completion. This role of gestures complements the findings of our earlier research (e.g. Roth 2002), which showed the mediation gesturing plays in the development of language following manipulations of objects. That is, hand gestures in particular again played an important role in the transition between two representational poles, here, ephemeral design ideas sketchily articulated in speech, on the one hand, and concrete artifacts, on the other. There are direct practical consequences of the way of thinking about and analyzing human communication. For example, in lecture-type situations, students generally copy whatever the lecturer notes on the inscription device present; they may also note some of the utterances that do not get publicly recorded. However, students generally do not record all the other semiotic resources that the lecturers provide in their performances. It is therefore not surprising that students experience difficulties to reproduce the sense of what has been communicated from their notes alone. It also should not surprise that so many students experience difficulties in understanding the lecture content at some later point, although during the lecture it all had made sense. The perspective developed here suggests that these students are in fact missing most of the resources that were available to them in the lecture but have neither been recorded in their notebooks nor were recoverable from the existing entries.

Acknowledgments This study was made possible in part by a grant from the Social Sciences and Humanities Research Council of Canada. I thank Sylvie Boutonne,

454

Wolff-Michael Roth

Michelle K. McGinn, and Carolyn Woszczyna for the help they provided during data collection.

References Bavelas, Jan B., Nicole Chovil, Linda Coates and Lori Roe 1995 Gestures specialized for dialogue. Personality and Social Psychology Bulletin 21: 394-405. Bourdieu, Pierre 1980 Le sens pratique. Paris: Les Editions de Minuit. Erickson, Frederic 1982 Money tree, lasagna bush, salt and pepper: Social construction of topical cohesion in a conversation among Italian Americans. In: Deborah Tannen (ed.), Analyzing Discourse: Text and Talk, 43-70. Washington, DC: Georgetown University Press. Goodwin, Charles 2000 Action and embodiment within situated human interaction. Journal of Pragmatics 32: 1489-1522. 2003 Pointing as situated practice. In: Sotaro Kita (ed.), Pointing: Where Language, Culture and Cognition Meet, 217-241. Mahwah, NJ: Lawrence Erlbaum Associates. Goodwin, Charles, Marjorie H. Goodwin and Malcah Yaeger-Dror 2002 Multi-modality in girls' game disputes. Journal of Pragmatics 34: 1621-1649. Hanks, William F. 1992 The indexical ground of deictic reference. In: Alessandro Duranti and Charles Goodwin (eds.), Rethinking Context: Language as an Interactive Phenomenon, 43-76. Cambridge: Cambridge University Press. Hamad, Stevan 1990 The symbol grounding problem. Physica D 42: 335-346. Haviland, John B. 2000 Pointing, gesture spaces, and mental maps. In: David McNeill (ed.), Language and Gesture, 13-46. Cambridge: Cambridge University Press. Hindmarsh, Jon and Christian Heath 2000 Embodied reference: A study of deixis in workplace interaction. Journal ofPragmatics 32: 1855-1878. Johnson, Mark 1987 The Body in the Mind: The Bodily Basis ofImagination, Reason, and Meaning. Chicago: University of Chicago Press.

Communication as situated, embodied practice

455

Kendon, Adam 1988 Goffman's approach to face-to-face interaction. In: Paul Drew and Anthony Wooton (eds.), Erving Goffman: Exploring the Interaction Order, 14-40. Cambridge: Polity Press. Latour, Bruno and Steve Woolgar 1979 Laboratory Life: The Social Construction ofScientific Facts. Beverly Hills, CA: Sage. Lave, Jean 1988 Cognition in Practice: Mind, Mathematics and Culture in Everyday Life. Cambridge: Cambridge University Press. Lave, Jean and Etienne Wenger 1991 Situated Learning: Legitimate Peripheral Participation. Cambridge: Cambridge University Press. Lee, Stuart and Wolff-Michael Roth 2001 How ditch and drain become a healthy creek: Representations, translations and agency during the re/design of a watershed. Social Studies ofScience 31: 315-356. Leont'ev, Alexei N. 1978 Activity, Consciousness and Personality. Englewood Cliffs, NJ: Prentice Hall. McNeill' David 1992 Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press. McNeill, David and Susan D. Duncan 2000 Growth points in thinking for speaking. In: David McNeill (ed.), Language and Gesture, 141-161. Cambridge: Cambridge University Press. Merleau-Ponty, Maurice 1945 Phenomenologie de la perception. Paris: Gallimard. Mikhailov, Feliks 1980 The Riddle ofSelf. Moscow: Progress. Ochs, Elenor, Sally Jacoby and Patrick Gonzales 1994 Interpretive journeys: How physicists talk and travel through graphic space. Configurations 2: 151-171. Ong, WaIter J. 1982 Orality and Literacy: The Technologizing of the Word. New York: Routledge. Roth, Wolff-Michael 1995 Affordances of computers in teacher-student interactions: The case of Interactive Physics™. Journal of Research in Science Teaching 32: 329-347.

456

Wolff-Michael Roth

2002

From action to discourse: the bridging function of gestures. Journal ofCognitive Systems Research 3: 535-554. 2003 Gesture-speech phenomena, learning and development. Educational Psychologist 38: 249-263. 2004 Perceptual gestalts in workplace communication. Journal of Pragmatics. 2006 Identity as dialectic: Making and Re/making self in urban schooling. In: Joe L. Kincheloe, kecia hayes, Karel Rose and Phil M. Anderson (eds.), The Praeger Handbook of Urban Education: 143-153. Westport, eT: Greenwood. Roth, Wolff-Michael and Daniel Lawless 2002 a Signs, deixis, and the emergence of scientific explanations. Semiotica 138: 95-130. 2002 b When up is down and down is up: Body orientation, proximity and gestures as resources for listeners. Language in Society 31: 1-28. Sewell, William H. 1992 A theory of structure: duality, agency and transformation. American Journal ofSociology 98: 1-29. Streeck, Jiirgen 1996 How to do things with things: Objets trouves and symbolization. Human Studies 19: 365-384. ten Have, Paul 1999 Doing Conversation Analysis: A Practical Guide. London: Sage. Uexkiill, Jakob von 1972 Theoretische Biologie. Frankfurt/M: Suhrkamp. (Originally published in 1928)

Index

action 1,3,8,20,21,26,34,36,41, 42,45,46,58,72,109,132,136, 138, 139, 147, 152, 167, 169, 179-183,185,186,188-190, 199,200,204,205,211,214, 219,241-251,253-259,271275,277,287,307,313,320, 323,324,326,328,351,369, 381,382,384-388,392,394, 395,397,398,402,426,431436,442-445,450-453 active perception 197, 198, 204206,214 affordances 7,8,55,68,69,72-74, 76,98,99,186,251,257 agencYlstructure dialectic 431, 433 animal 43, 44, 48, 49, 65, 66, 69, 71,72,76,94-96,98,101,102, 134-138,140,141,168,177, 189,214,218,245,307,321, 327,365,379-381,383,384, 386,391-393,395,396,398400,402,403,405 biosemiotics 7, 9, 85, 101, 102, 105, 109,379,380,382,383,385391,394,395,400,402 bodily mimesis 219,228,231,232, 297,301,312,318,320,322324,326,329 body 2-4,6-9,17,18,22,24,25, 30,35,37,38,40,41,43,47,48, 55-59,62-64,67,69,70,73,7577,85-91,93,96,101,108-114, 116-120,130-134,137,138, 142-145, 147-149, 151-153, 169,179,180,183,189,205, 206,217,219,225,228,231,

241-243,252,253,257-259, 271-288,297,298,301,308, 312,313,318,320,325,329, 339-348,350,351,356,364370,379-384,388,389,392, 395-400,403,405,412,414, 416,431-435,437-447,449-453 body image 8, 217, 228, 231, 271283,285,288,308,313,351 body schema 8, 205, 217, 271-284, 286,288,308,313,351,434 categorization 167, 169, 175, 176, 178,190,206-208,212,214, 215,231,371,376,411,413415,417,426 causality 65, 314, 315, 380, 384, 386,387,393,401 cognitive linguistics 1, 6-8, 17, 85, 113,119,120,198,220,259, 298-300,304,306,308,315, 318,325,339,346,357,358 cognitive neuroscience 49, 144, 183,185,252,339,341,346, 351,361,363,369 cognitive science 1, 3-5, 7, 9, 10, 17-21,47,48,81,86,92,98, 119, 129-13 1, 133, 141-151, 154,168,197,200,242,250, 252,271,273,297,310,311, 313,315,316,339-341,343, 344,347-349,356,358,359, 362-364,369,370,379,382, 383,386,387,393,395,412414,437 communication 9, 43, 44, 46, 47, 115, 130, 131, 138, 152, 153, 208,218,223,228,252,255,

458

Index

284,307,319,320,324,381, 382,416,423,431-433,435437,444,448,450-453 complex systems 197, 199, 200, 231,386 conceptual spaces 8,167,168,171, 173,175,179,182,185,187, 190, 191, 197, 199 consciousness 2, 6, 7, 9, 22, 85-89, 91,97,98,100,102,107-109, 113,119,135,136,201,202, 217,219,222,228,232,275277,279-281,297-300,302304,307,309,312,313,315318,322-325,363,389,392, 395,398,400,402,433,434,452 conventions 297, 303-305, 310, 324,326,328,434,438 cyborg 379,380,399,402-405 deafferentation 271, 273, 277-279 dynamic categorization 197, 204,

206-208,213 ecology 7,55,56,64,75,85,100,

101

416-418,423,425,426,431, 434,453 emergence 23,86,139,219,231, 258,380,383,384,387,401,423 evolution 3, 80, 85, 114, 116, 134, 209,215,231,258,284,301, 318-322,326-328,383,387, 390,394,403,432 force dynamics 8,167,169,172,

179-183,186-191 frames of reference 339, 343, 345,

346,352,354,362-365,367-369 gesture/-s 6, 9, 129, 130, 134, 153,

202,227,228,231,241,255, 258,282-284,301,307,319321,323,324,326-329,362, 365,366,415,431,435,437-453 Gibson, James 7, 55, 56, 61, 64-69, 71-78,79,97-100,102,108, 122, 156, 182, 183, 186, 189, 193,198,204,205,207,234,261 history of cognitive science 129-

131, 133, 138

embodied cognition 1, 3-8, 10, 18,

48, 129, 131, 144, 145, 148-151, 154,206,241-243,249,256, 258,297-300,311,312,348, 356,360,364,393,431 emb0dnnent 1-4,6-9,17,18,21, 48,55,56,76,85-89,91,109, 118-120,129-133,137,143146, 148-153, 185, 187, 197, 206,207,230,241-243,249, 252,255,258-260,271,272, 280,284,286,288,297-301, 308,309,311-314,318,319, 322,324-329,339,340,342, 344,348-354,356-360,363, 365,366,368-370,379-386, 389,390,392,394-404,411414,416-418,423,425,426,

image schema/-s 17,21,32-37,42,

44,45,85,86,90,92,109,111, 114, 120, 167, 187, 189, 191, 198,309,314,324,325 information 7, 48, 55, 62, 68, 69, 72,76,77,105,115,142,146, 154,168,178,181-183,185, 189,227,246,249,255,315, 316,340,382,383,386-388, 390,391,394,397,412,432,435 interactive technology 129, 133, 151, 153, 154 internal meaning space 8, 197, 198, 204,214-217,219,222,223, 225,227,229,231,232

Index intersubjectivity 8, 133, 241, 252,

271,285-288,301,307,351,451

459

nature-culture dualism 55-57, 69,

76, 77 neonate imitation 271, 273, 279,

Japanese mimetics 197, 199, 223,

224,226-228,230

280,284,285 neurobiology 17,21,26,29,48,88,

91,308,319,341,387 language 1,2,4,7-10,17,19,20,

22,23,31,36,37,41,42,44,45, 47,48,59,61,63,66,69,87,93, 96,99,101,104,106,107,110, 115-118, 120, 130, 131, 133, 134,139,140,148,152,173, 174,197-200,202,204,216, 218-220,223,226-228,230232,241,243-245,253,255260,283,297-301,303-314, 318-323,325-329,339,345347,349,350,353-355,359, 361,362,364,365,367,380382,393,400,402,404,411, 413-418,425,426,431,432, 437,442,448,452,453 Lifeworld 85,87,88,99-101,107, 109,110,120 memory 58,59,85,94,114-119,

142,184,203,212-214,229, 231,283,350,397,414,418,432 mental rotation 9, 250, 339, 341343,345-348,356,359,362, 364,365,368 metaphor 4, 5, 7, 9, 17, 20, 37-42, 48,55,61-63,98,111,129,130, 133,142,144,202,218,250, 301,308,310,312,339,357, 358,362,363,366,368,391 mimetic schemas 114, 180, 199, 203,228,231,297,301,322, 324-329 mirror neurons 8, 133, 147, 148, 150,189,217,241,251-255, 271,287,328,436 mutualism 55,75-77

organicism 380, 390, 391 perception 8, 23, 27, 32, 33, 35, 36,

44,60,61,73-75,78,88,95,96, 98,99,106,108,117,132,138, 147,149,167,169,170,172, 182, 183, 186, 190, 193, 195, 200,201,203-205,207,208, 214,217-219,223,226,231, 241-245,249,250,253,257259,271-276,278,281-284, 286,287,311,355,381,400, 402,416,418,420,421,425, 433,436,445 phenomenology 3,7,65,85-91,97, 120,150,200,206,218,271, 272,285,288,297,309,351,391 picture 5,44,67,76,85,96,98,99, 116,130,200,205,231,250, 306,311,329,348,367,387 pragmatism 17-21,32,37,39-42, 47,48,306 proprioception 55, 74, 225, 278, 279,284,286,320 radical embodiment 3, 129, 147,

149-151,386 representation 8, 9, 20, 25, 30, 43,

46,69,86,92,96,104,114,115, 144,147,149,152,168,170, 173,177-180,182-184,186, 190,197,198,200,201,203, 213,219,220,223,226,228, 230,231,244,246,252,259, 260,297-300,306,309-312, 317,322-325,350,364,367,

460

Index

369,379,380,383,388,393, 400,436 Representationalism 17, 19-21, 3033,35,36,42-48 self-organization 390, 411, 418,

423,425 semantics 8, 17, 167, 176, 187, 188,

191,192,223,304,349,357, 382,393 semiotic function 85, 93-95 semiotics 7, 85, 89-92, 94, 99, 105, 132,351,381,383-386,388, 393,394 sign 85,86, 92-109, 114-116, 118, 120,200,223,226,304,319, 320,381-384,386-389,391395,402,438,440,450 simulation theories 5, 12, 147, 241, 243,245,250,258-260 situated practice 431-433, 450-453 social interaction 7, 9, 43, 47, 48, 129-133, 137, 140, 144, 145,

147,148,151-153,252,253, 307,309,313,314,411,417,426 society 141, 381, 382, 397, 399, 400,403-405 space 8,9,23,28,29,33,38,66, 67, 86, 87, 93, 110, 113, 119, 167-178,183,185,186,188, 190,197-199,214-217,219, 224,225,231,232,281,299, 317,319,339-341,346,360, 367,369,414,417,423,426, 445-448 speech 9,47,59,140,174,181, 223,226,231,255,305-307,319, 326,411-418,425,431,434-438, 442,443,445,448,450,451,453 synaesthesia 8, 197, 199, 216-223, 225,230,231 unilateral neglect 271, 273, 277-

279