Musical Imagery

MUSICAL IMAGERY

STUDIES ON NEW MUSIC RESEARCH Series Editor:

Marc Leman, Institute for Psychoacoustics and Electronic Music, University of Ghent, Belgium.

MUSICAL IMAGERY

Edited by

Rolf Inge God¢y and Harald ]¢rgensen

@

Taylor & Francis Taylor & Francis Group

NEW YORK AND LONDON

Library of Congress Cataloging-in-Publication Data Applied for ...

Published by Taylor & Francis 270 Madison Ave, New York NY 10016 2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN

Transferred to Digital Printing 2009

Serie-cover design: Ivar Hameling.

© 2001 Taylor & Francis All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, by photocopying, recording or otherwise, without the prior written permission of the publishers.

ISBN 90 265 1831 5 (hardback)

Publisher's Note The publisher has gone to great lengths to ensure the quality of this reprint but points out that some imperfections in the original may be apparent.

Contents Contributors

VB

Editors Preface

VBI

Rolf Inge God¢y and Harald J¢rgensen

Part I Theoretical Perspectives Overview

.

Rolf Inge God¢y and Harald J¢rgensen

1

Perspectives and Challenges of Musical Imagery . . . . . . . . . . . .

5

Albrecht Schneider and Rolf Inge God¢y

2

Neuropsychological Mechanisms Underlying Auditory Image Formation in Music

27

Petr Janata

3

Musical Imagery and Working Memory . . . . . . . . . . . . . . . . . ..

43

Virpi Kalakoski

4

Modeling Musical Imagery in a Framework of Perceptually Constrained Spatio-Temporal Representations

57

Marc Leman

5

Mental Images of Musical Scales: A Cross-cultural ERP Study

77

Christiane Neuhaus

6

Complex Inharmonic Sounds, Perceptual Ambiguity, and Musical Imagery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..

95

Albrecht Schneider

7

Musical Imagery between Sensory Processing and Ideomotor Simulation

117

Mark Reybrouck

8

Musical Imagery as Related to Schemata of Emotional Expression in Music and on the Prosodic Level of Speech .... 137 Dalia Cohen and Edna lnbar

9

Imaging Soundscapes: Identifying Cognitive Associations between Auditory and Visual Dimensions . . . . . . . . . . . . . . . . .. 161 Kostas Giannakis and Matt Smith

Part II Performance and Composition Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 181 Rolf lnge God¢y and Harald J¢rgensen

10 Expressive Timing in the Mind's Ear Bruno H. Repp

185

MUSICAL IMAGERY

VI

11 Control of Timbre by Musicians - A Preliminary Report ..... 201 Wolfgang Auhagen and Viktor Schoner

12 Images of Form: An Example from Norwegian Hardingfiddle Music

219

Tellef Kvifte

13 Imagined Action, Excitation, and Resonance

237

Rolf Inge God¢y

14 The Keyboard as Basis for Imagery of Pitch Relations

251

James M. Baker

15 Composers and Imagery: Myths and Realities

271

Rosemary Mountain

16 The Musical Imagery of India . . . . . . . . . . . . . . . . . . . . . . . . . .. 289 Lewis Rowell

Name Index Subject Index

303 311

Contributors Wolfgang Auhagen and Viktor Schoner, Musikwissenschaftliches Seminar der Humboldt-Universitat zu Berlin, Am Kupfergraben 5, D-I0099 Berlin, Germany, Wolfgang.Auhagen @rz.hu-berlin.de

James M. Baker, Brown University, Box 1924, Providence, RI 02912, USA, [email protected]

Dalia Cohen and Edna Inbar, Department of Musicology, The Hebrew University of Jerusalem, 13 Rashba Street, 92264 Jerusalem, Israel, [email protected]

Kostas Giannakis and Matt Smith, School of Computing Science, Middlesex University, Bounds Green Rd, London NIl 2NQ, UK, [email protected] Rolf Inge God0Y, Department of Music and Theatre, University of Oslo, P.O. Box 1017 Blindem, 0315 Oslo, Norway, [email protected]

Petr Janata, Psychological and Brain Sciences, 6207 Moore Hall, Dartmouth College, Hanover, NH 03755, USA, [email protected]

Harald J0rgensen, Norwegian State Academy of Music, P.O. Box 5190 Majorstua, 0302 Oslo, Norway, [email protected]

Virpi Kalakoski, Department of Psychology, P.O. Box 13,00014 University of Helsinki, Finland, [email protected]

Tellef Kvifte, Department of Music and Theatre, University of Oslo, P.O. Box 1017 Blindem, 0315 Oslo, Norway, [email protected]

Marc Leman, Institute for Psychoacoustics and Electronic Music, University of Ghent, [email protected]

Blandijnberg

2,

B-9000

Ghent,

Belgium,

Rosemary S. Mountain, Department of Music, Concordia University, 7141 Sherbrooke Street West, Room RF 322, Montreal, QC H4B 1R6, Canada, mountain @vax2.concordia.ca Christiane Neuhaus, Institute of Musicology, University of Hamburg, Neue Rabenstr. 13, D-20354Hamburg, Germany, [email protected]

Vlli

MUSICAL IMAGERY

Bruno Repp, HaskinsLaboratories,270 Crown Street, New Haven, CT 06511-6695, USA,[email protected] Mark Reybrouck, Katholieke Universiteit Leuven, Eikendreef 21, B-8490 Varsenare, Belgium, [email protected] Lewis Rowell, School of Music, Indiana University, 1201 East Third Street Bloomington, IN 47405-7006, USA, rowell @indiana.edu Albrecht Schneider, Institute of Musicology, University of Hamburg, Neue Rabenstr. 13, D-20354 Hamburg, Germany, [email protected]

Preface This is a book about images of musical sound in our minds. Our belief is that musical imagery is at the very core of music as a phenomenon, because, after all, what would music be if we did not have images of sound in our minds? Yet, a review of outstanding writings on music in our culture from antiquity to the present shows that there is little material dealing directly with our mental images of musical sound. The study of imagery in other domains, and in particular visual imagery, has made important advances in the past couple of decades, raising the important debate about 'mental imagery' in the 1980s, and developing ever more innovative approaches to research. The past three decades have, however, also witnessed a growing interest in research into music cognition. This research has covered a broad range of issues of perception and has also contributed to the formation of explanatory models of musical behaviour. Accordingly, we felt that it was high time musical imagery was put on the agenda of an international conference. When the opportunity arose to choose a topic for the VI. International Conference on Systematic and Comparative Musicology, to be held at the Section for Musicology of the University of Oslo in June 1999, we suggested musical imagery as the topic. In the call for papers prior to the conference we tentatively defined musical imagery as 'our mental capacity for imagining musical sound in the absence of a directly audible sound source, meaning that we can recall and re-experience or even invent new musical sound through our 'inner ear'.' In the reading of abstract proposals submitted for the conference, as well as in the process of selecting and editing the chapters for this book (mostly written by authors present at the Oslo conference), we realised that we would have to reconsider our definition of musical imagery. For one thing, the distinction between perception and imagery is at best one of principle. Trying to distinguish between actually listening to music (be that in a live performance situation or with a phonograph source), and imagining music without such a source, is a matter of distinguishing two different situations. However, it is also clear that in an ordinary listening situation, the memory of what has been heard as well as the expectations of what is to come, play an integral role in the process of any 'primary' perception of musical sound. It could thus be argued that contextual images as well as more general schemata at work in music perception, are in fact elements of musical imagery 'present' in an ordinary listening or performance situation (e.g. the improviser has an image of what he/she just played and what to play next).

x

MUSICAL IMAGERY

Another point is that studies of musical imagery to a large extent are indirect in the sense that we do not have an 'observer' situated in the mind, capable of giving accounts of what is going on when we are experiencing images of musical sound. Although the use of non-invasive methods (cf. chapters 2 and 5 in this book) may indeed give us useful observations of what is going on in the brain when we imagine musical sound, this does not tell us much about the qualities involved in musical imagery. One consequence of this indirect access to mental images of musical sound is that we have to deduce, assume or simply guess a number of things from other sources, in particular from what can be observed in the more 'primary' perception of music. Another consequence of this is that we have to rely on introspective accounts of our mental images of musical sound. We should also add to this that musical sound 'in itself' may be considered 'impure' in the sense that musical imagery seems in many situations to be accompanied by, or even inseparable from, images of source, of sound-generation, of the environment, as well as various images of 'meaning', such as emotional content or highly extramusical associations (e.g. the sound of a drum roll could for many people be inseparable from the image of someone ferociously beating the drum membrane with a couple of drum sticks). Given the complex and multifaceted nature of our topic, we believed that there had to be a plurality of approaches in this book: no one single domain of research can claim to possess the most appropriate approach to musical imagery. We believe that people from such diverse domains as neurology, cognitive psychology, philosophy, music theory, ethnomusicology, music education, composition and performance may all make valuable contributions to the study of musical imagery. We emphasise this, because the reader of this book will encounter a pluralism of scientific paradigms and terminology, and it is our hope that the various contributions will be appreciated on their own terms. We have organized the chapters of this book into two parts. The first part deals with theoretical perspectives and gives a presentation of what we see as some of the main issues in musical imagery today, including a historical overview and an overview of the neurophysiological bases of musical imagery, as well as some other theoretical issues of musical imagery. The second part is focused on issues related to performance and composition and presents some more practical applications of musical imagery. Both main parts will be preceded by an overview section where the main issues of the chapters will be presented and briefly discussed in the perspective of the domain of musical imagery as a whole. We believe that the subject of musical imagery concerns everyone working

xi

PREFACE

with music. Our belief that musical imagery is at the very core of musical experience, perhaps even being the very content of musical thought, makes us hope that this book will be read by not only music psychologists, but by performers, composers, arrangers, music theorists, musicologists, music educators, and of course, by any person interested in music. This book is the result of the efforts of our contributors, and we thank them all for sharing their work with us and the readers of this book. We sincerely thank the reading committee, consisting of Marc Leman, Bruno Repp and Albrecht Schneider, for their effort. Without their meticulous reading, commenting, and extensive knowledge, this book would hardly have been possible. Our thanks go to the publisher, Swets and Zeitlinger, for receiving and endorsing our project and for patient and helpful support throughout the production process, to the Norwegian State Academy of Music, to the Department of Musicology, University of Oslo, and to The International Society for Systematic and Comparative Musicology for supportin"g this project. Last, but not least, we thank Gisela Attinger for doing the production work of the manuscripts. Her efforts, her reliable attention to detail and her ingenuity have made a major contribution to the quality of the book. May 2001 Rolf Inge God0Y

Harald 10rgensen

I Theoretical Perspectives Rolf luge

and Harald

The first part of this book deals with some fundamental issues in musical imagery. Although there seems to be a broad consensus about the term 'musical imagery' as denoting images of musical sound in our minds, there are obviously many opinions on the nature of such images and on their relationship to perception and memory, and there are many explanations of how they work. Also, there are many, and rather different, approaches to investigating these images of musical sound in our minds. Such a plurality of notions of what musical imagery is supposed to be, as well as of paradigms for exploration may perhaps seem confusing. There is of course a considerable distance not only in method, but also in fundamental attitudes, between for instance 'arm chair' style introspective approaches and various methods of brain activity measurements. As we know, such differences in approaches and fundamental attitudes are often encountered in other domains of the cognitive sciences as well. On the positive side, we can see these differences in approaches and attitudes as an expression of the composite and multifaceted nature of our topic. With this understanding, the plurality of notions and approaches becomes an asset rather than a liability, and we should read this book as a frame-by-frame exposure, from various angles, of this rich and essential topic that we think musical imagery is. In line with this idea of a pluralism of approaches, the contribution of Albrecht Schneider and Rolf Inge (chapter 1) gives an overview of some important notions of imagery and musical imagery in the past, as well as a brief assessment of some challenges confronting us now and in the immediate future. The authors believe

2

THEORETICAL PERSPECTIVES

that some ideas manifest in phenomenological and Gestalt theoretical works of the nineteenth and twentieth century still have relevance. As an example, we can look at the issue of context in both perception and imagery. This was elegantly depicted of Brentano and later of Husserl in his tripartite model of retentions, primal impressions and protentions, meaning that there are always images of past experience and future expectancies exerting influence on what we perceive and imagine at any given moment. This, and similar 'introspective' insights are actually also epistemological questions, questions which reappear in experimental approaches to musical imagery. Interestingly, expectancies and/or violations of expectancies are important ingredients in investigating the neurophysiological bases of musical imagery, as we can see in Petr Janata's contribution (chapter 2). Another central issue is the relationship between imagery and more 'primary' perception, and Janata gives us a model for structuring our understanding of this as well as other central issues of the neurophysiological basis for musical imagery. Various methods for collecting data on brain activity during tasks of musical imagery, as well as various proposed models for the workings of musical imagery are presented in this chapter. In addition, the author presents some of his own observational findings. Although there will always be a neurophysiological basis for musical imagery, the focus of Virpi Kalakoski (chapter 3) is on various experiments and models of human memory and faculties of imagery. The topic of memory is indissociable from that of imagery, and a number of 'classical' theories of memory are presented here, as well as some of the authors' own experimental findings. As a conclusion, the author suggests that musical imagery is multimodular as well as multimodal, ideas that we find in several other contributions in this book. We hope that these first three chapters will give the reader an overview of some basic epistemological-philosophical, neurophysiological, and cognitive issues of musical imagery, and by that preparing the ground for the following chapters. In the next three chapters we move on to consider in more detail what kinds of constraints and/or schemata might be at work in musical imagery. Is it possible to imagine the unimaginable, or is that a contradiction of terms? Or rather: Is everything that we can possibly imagine based upon bits and pieces of what we have already experienced in our lives, so that imagery is mostly a matter of making new combinations? And furthermore: Does whatever we imagine follow schemata that we all have learned during the long process of acculturation? And: Are there effects of context, both short-term and longterm, at work in imagery so that whatever we may imagine at a given moment, in a 'now', is actually conditioned by what we imagined a little while ago as well as by what we are expecting to imagine in the immediate future? Posing these and similar questions shifts our attention towards the very content of our images of musical sound. Given what is known about constraints and schemata at work in other cognitive domains, it is not unreasonable to guess that such constraints and schemata are also at work in musical imagery. In the contribution of Marc Leman (chapter 4), the point of departure is what can be termed an ecological view of musical sound, perception and cognition. Here, the basis is the continuous, sub-symbolic acoustic substrate and the neurophysiological workings of audition, including the selforganizing behaviour of neurons. The paper presents a modeling of a 'low-level' basis for musical imagery, with higher level brain processes linked to this basis.

ROLF INGE

AND HARALD J0RGENSEN

3

From a neurophysiological point of view, constraints on musical imagery should be detectable in the sense that there should be some kind of observable trace in cases where there is a 'violation of expectancies' , i.e. where musical sound does not conform to schemata in our minds. This is the topic in the contribution of Christiane Neuhaus (chapter 5). She sets out to measure and interpret patterns of brain activity across different cultural groups. What may seem 'right' to one group, i.e. in accordance with learned schemata, may seem 'wrong' to another group, i.e. as a violation of learned schemata. The research of Neuhaus shows that neurophysiological responses to expected versus non-expected patterns in musical scales are different in groups of people from different cultures, and this makes it plausible to suggest that such culture specific schemata are also present in musical imagery. Another aspect is how our learned, internal schemata work when we are confronted with acoustically highly complex and even ambiguous sounds in terms of pitch, as is the case with many inharmonic sounds. This is discussed in the contribution of Albrecht Schneider (chapter 6), and exemplified with inharmonic sounds from carillons. These sounds are analysed in view of acoustic properties such as spectral content and possibilities or impossibilities of having an unambiguous pitch. It seems that beyond a certain point of complexity, listening has to rely on some kind of simplification of the sound material. That is, we have to 'overrule' the acoustic material and make an 'idealised' or 'stylised' image of the musical sound by filtering out features which would otherwise lead to ambiguous images. This concerns the general question of the ecological versus the more schematic nature of our images of musical sound, something that can also be understood as a dichotomy between the abstract and the concrete in music (to borrow an expression of Pierre Schaeffer), ultimately related to a distinction between the 'pure' and the 'impure' in musical thought. Speaking of the 'pure' versus the 'impure', the last three chapters of Part I deal with various schemata in other modalities than the auditive. There are various properties in the acoustic signal which over a certain stretch of time can lead to the formation of schemata by principles of self-organisation (which can be seen as a kind of learning). On the other hand, there are associations formed which link musical sound to events in other modalities, such as to vision, various motor-related sensations or to more general emotional images. The questions in our context are to what extent, as well as how, the images, schemata and/or constraints in other modalities can engender, enhance and influence images of musical sound, and conversely, to what extent images of musical sound may trigger images in other modalities. For instance, is it possible to imagine highly emotionally charged music, e.g., Schonberg's Erwartung, without also arousing emotions, and conversely, can various emotions evoke certain images of musical sound? Does imagining certain kinds of music also engender images of certain kinds of movement, e.g., does imagining a juicy tango also evoke sensations of dance movements, or does the image of a frenetic drum sequence evoke images of equally frenetic mallet, hands, and body movements? Can images of colours be associated with certain images of musical sound? The list of such questions can be very long indeed, but from what has emerged from crossmodality research in recent years, there seems to be an increasing amount of research supporting the idea that the simultaneous presence in our minds of elements from different modalities can enhance images in many cases (as well as of course inhibit or

4

THEORETICAL PERSPECTIVES

weaken images in some cases). One very good reason for posing such questions is to try to develop systematic methods for generating and enhancing images of musical sound, in other words, to have a better understanding of what triggers images of musical sound in our minds, or what is the 'engine' of musical imagery. The contribution of Mark Reybrouck (chapter 7) argues that there are many indications of motor components involved in perception and cognition, and in imagery as well, and that traditional notions of 'passive' sensory input coupled with abstract symbol cognition is now rejected by several neuroscientists. There is a close functional resemblance between perception and action on the one hand, and imagery of the same perceptions and actions on the other hand. A similarly close relationship between emotional schemata in the imagery of music and speech is suggested in the contribution of Dalia Cohen and Edna Inbar (chapter 8). We may assume that for most people in our culture, images of emotions are integral to much of the music we experience. An exploration of emotional images and schemata could then be useful in further explorations of the nature of musical imagery, in particular as possible agents for the recall and enhancement of musical images. At the end of this first part of the book, the contribution of Kostas Giannakis and Matt Smith (chapter 9) explores the link between colour and images of musical sound. The idea is that there could be a correlation here, not only for people who claim to have had synaesthetic experiences. A correlation between colour and images of musical sound can be exploited in evoking images of musical sound in the mind, as well as be useful in machine representations of musical sound. In summary, the first part of this book is a spreading out of the topic of musical imagery, progressively exposing several aspects. Is it possible then to have a coherent, or even unitary perspective of what musical imagery is supposed to be? Probably yes, if we by musical imagery understand images of musical sound in our mind. Yet, if we think that this is too loose and all-embracing, the problem is not musical imagery, but rather music as a phenomenon. If we accept that music is infinitely complex and diversified, there is really no reason to think that our images of musical sound should be less complex and diversified.

1 Perspectives and Challenges of Musical Imagery Albrecht Schneider and Rolf Inge

Introduction

The study of musical imagery as well as imagery in other domains is not a recent invention. Although the so called 'cognitive revolution' of the last quarter of the twentieth century has given the study of imagery the status of scientific investigation, we believe it is important to be aware of some historical elements here. There is sometimes an embarrassing lack of historical knowledge in contemporary cognitive science, sometimes giving us a feeling of witnessing a 'reinvention of the wheel'. But in a more positive sense, we believe several of the questions posed today have actually been given much attention as well as creative answers in the past. Because of this, we shall in this chapter give a brief, and necessarily selective, survey of some main points in the history of musical imagery and imagery in general. To complete this overview, we shall also at the end of this chapter try to give a summary of what we see as the main challenges for the study of musical imagery now and in the immediate future. As will become clear when reading the various contributions in this book, musical imagery is a complex and multifaceted topic, allowing for a multitude of approaches and scientific paradigms. Also, the term imagery has been, and is still, used to denote a manifold of phenomena as well as conceptual objects. It could be useful then to start by giving a survey of the most prevalent meanings of the term, a term which plays a central role in philosophical treatises on epistemology, and is of similar importance in works on cognition and 'inner perception' (often approached as apperception in the

6

PERSPECTIVES AND CHALLENGES OF MUSICAL IMAGERY

philosophical and psychological literature from Descartes to Wundt). Further, mental imagery plays an important role in clinical psychology and psychiatry (see contributions in, e.g., Shorr et aI., 1980; Klinger, 1981). More recently, the term imagery has been widely used also in the field of neuroscience where a number of experimental techniques have been developed to study the anatomy and actual function of the living brain, such as computerized tomography, magnetic resonance imaging, positron emission tomography (PET), and magnetoencephalography (MEG, see Clarke, 1994). These techniques have also been employed in the study of audition and music perception (see NaaUinen & Winkler, 1999; Marin & Perry, 1999).

Philosophical approaches The following notes cover, roughly speaking, philosophical commentaries on mental imagery from Aristotle and Descartes to Husserl and the phenomenological movement of the twentieth century. Given that epistemology includes aspects of perception and cognition, on the one hand, and that psychology cannot do without epistemology, on the other, a clear distinction between 'philosophical' and 'psychological' concepts of imagery is hardly possible. It should be clear, in this connection, that any discussion of concepts of imagery (see, e.g., Segal, 1971; Block, 1981; MacDaniel & Pressley, 1987) at some point will enter a much larger and still more complex field which generally is labelled philosophy of mind (for an overview, see Carrier [1995]). There are quite many works in philosophy and related fields (nowadays often subsumed under the term cognitive science) which deal with the nature of mental and psychic phenomena, mind-body relations as well as with the brain-mind problem (e.g., Carrier & Mittelstrass, 1991; Dennett, 1991; Scheerer, 1993). Even though imagery may comprise a limited area of mental phenomena, to understand these seems to be especially intricate since mental images are said to be notoriously subjective, on the one hand, and difficult to investigate by experimental procedures, on the other. It cannot be denied, however, that imagery is essential in artistic production as well as apprehension of works of art. The English term imagery which denotes, in particular, mental images as being the products of imagination, is obviously related to the Latin terms imaginatio and imago. The word imago in Latin has a variety of meanings which range from picture, portrait, mask, guise to scheme, vision, phantom, and also to echo, metaphor, allegory (Georges, 1869, pp. 2293-2295.). Basically, the different meanings can be grouped as follows: imago:

1. 2. 3. 4. 5. 6. 7. 8.

picture, portrait (imago ficta; if painted: imago picta); death-mask, portrait of ancestors; image, effigy, (copy, duplicate, exact likeness); shadow, scheme, vision, phantom, echo (also of a true voice); metaphor, allegory, figure of speech; phantom, illusion, delusion, mirage, apparition; view, sight, phenomenon, appearance; idea, conception;

ALBRECHT SCHNEIDER AND ROLF INGE

7

This rather large semantic field of imago allows us perhaps to extract some basic features. First, imago relates to objects existing in the real world which are pictured in a realistic manner. Second, subjects conceive of objects that may exist or did exist once in the real world so that these objects are somehow mirrored as images. Third, subjects may create mental images of things that never did exist in the real world, and are thus the products of imagination (e.g., witches riding on broom-sticks, or more musically interesting, a set of planets which rotate in space whereby they produce a complex sound which is a chord of perfect harmony). Fourth, subjects are capable of producing some kinds of immaterial 'copies' of objects of the material world to be stored in one's mind. This in fact is also the central meaning of imaginatio which in turn is the scholastic translation of the Greek term phantasia as was used by Aristotle (Arist. met., 980b, 1010b, 1024b). According to Aristotle, perception has to be distinguished from imagination. Humans have a capacity to produce immaterial pictures of objects which they have perceived previously. This capacity can also be used to recall objects which are stored in memory, as well as to create visions of real or imaginary objects. For Descartes, the vis imaginandi (German: Einbildungskraft; English: power of imagination) is a source of a multitude of imaginations which include the creation of new objects not learned by previous experience (Descartes, 1642, med. sec.). For example, such new objects may arise in dreams which are not 'triggered' by actual sensations because we have our eyes closed, yet we can 'see' such objects. These imaginations, however, are not very reliable with respect to truth, and are prone to error and illusion. From the point of epistemology, they are thus inferior to cogitationes which are the only reason that we are able to obtain reliable knowledge: Ego cogito, ergo sum, sive existo. Or, as Descartes expressed himself in the French text of his Discours de la methode (written earlier than the Meditationes): je pense, donc je suis (Descartes, 1637, ch. 4). In the Discours de la methode, Descartes had elaborated on the problem of true knowledge, and had come to the conclusion that only such things we comprehend very clearly, and very distinctly, are true. This expression, which in the French original reads fort clairement et fort distinctement, and in the Latin translation valde delucide et distincte, is of prime importance since in effect it relates to the difference between perception and apperception. 'Very clear' (fort clairement), in this respect, are objects that we perceive by eye or ear, and of which we have a precise and unambiguous impression. 'Very distinct' (fort distinctement), however, are conceptions of things that bear to their very nature. If, for example, one imagines a triangle, we see three lines, and also recognize a geometric figure with certain features, which is formed by these. If we now extend, as Descartes in fact did, this little example to a polygon of a thousand lines, it is not possible to have an 'inner picture' of such a configuration any more, however it is possible to understand the principles of such a construction at once (Descartes, 1642, med. sexta). What he wants to distinguish then, is mental imagery as related to perception from cognition as a form of conscious knowledge. We should add that Descartes' thoughts on 'very clear' and 'very distinct' played a significant role in a key work on the theory of perception, namely, Franz von Brentano's Psychologie vom empirischen Standpunkt (Brentano, 1874, 1924, 1928). We will return to this work below, but if we

8

PERSPECTIVES AND CHALLENGES OF MUSICAL IMAGERY

now only sum up briefly some of Descartes' terminology and considerations in the Meditationes we find: cogitatio idea imaginatio imaginari intelligere

= = =

thinking (and also consciousness) idea, conception imagination (and also power of imagination) to imagine (to conceive of, and to understand things by means of 'images', and in a sensuous mode) to comprehend (to understand the nature of things)

It is not possible here to go much into David Hume's Enquiry concerning human understanding (Hume, 1758/1951), and to discuss the relationship of impressions to thoughts and ideas he explains there. Basically, 'ideas' are but weak aftereffects of impressions based on sensory data. The mind then has the task to order and to combine data acquired by the senses, and by experience in general. Hume, 'the hero of the atomistic theory' (James, 1981, p. 691), advances an associationist view of the mind which makes use of the principle of 'similarity'. Specific 'ideas' are thereby associated because of objective similarity of content (see also Wilbanks, 1968). An elaborate concept of mental synthesis is found in the philosophy of Immanuel Kant. The basic question Kant put forward in his Critique of Pure Reason (Kritik der Reinen Vernunft, KdRV B 19) is: how can synthetic propositions (Urteile, 'judicii') a priori be derived? And in particular: how is pure mathematics possible? how is pure natural science possible? In answer to these questions he developed a systematic approach of epistemology called Transcendental-Philosophie, whereby transcendental means transcending common experience, yet not the limits of possible human knowledge (the latter condition, in Kant's epistemology [KdRV B 352ff], is labelled transcendent, whereas reasoning that takes place within the limits of possible experience, is labelled immanent). The foundation of the Critique of pure reason is the transcendental aesthetics where Kant elaborates such concepts as sensation (Empfindung), perception (Anschauung), phenomenon (Erscheinung), sensuousness (Sinnlichkeit), conception (Vorstellung). His reasoning is that, in dealing with the world around us, we strive for knowledge which requires both notions and perceptions. Perception of things are 'given' (gegeben), in a first stage, by sensory data. These are however, processed mentally by making reference to a system of categories such as space and time, as well as certain basic concepts (Raum, Zeit, Stammbegriffe), to result in perception. Perception is regarded as a conception accompanied by sensations (mit Empfindung begleitete Vorstellung, KdRV B 147). It might be added that both space and time are considered by Kant (KdRV B, 38ff, 116ft) as categories and as 'forms of perception' (Formen der Anschauung). Objects which are perceived can also be constituted by rational thought, what then results in notions (Begriffe). Since knowledge in many cases implies abstraction and generalization, it requires such notions, and a framework of categories (reine Verstandesbegriffe) which are needed as the foundation stones for pure reasoning. Objects which 'affect' (affizieren) us, cause sensations which are empirical (KdRV B 34). In our mind, the object of empirical sensations is registered as a

ALBRECHT SCHNEIDER AND ROLF INGE

9

phaenomenon (Erscheinung). Phaenomena in general call for further mental treatment to become perceptions in a strict sense. Opposed to phaenomena, Kant (KdRV B 307) defines noumena as objects of a type of perception which is not dependent on sensory 'input' (nichtsinnliche Anschauung). One might think of, for example, moral norms such as honesty or dignity which can be conceived of independent from actual sensations and/or perceptions. Noumena would actually be objects of a pure intellectual mode of perception, or rather, apperception. Kant (KdRV B 310-311) admits that his notion of noumenon is a problematic issue that he had introduced, in the first place, to delimit the range of the notion of phaenomenon, and of things covered by this concept. The basic problem now is: how can coherent knowledge be derived by one person? Obviously, each of us has a single, unique, and in itself coherent experience which in sum makes up one's life as well as one's personality. The capacity to perform an ongoing synthesis of the manifold of sensations, perceptions, as well as of intellectual and moral understanding into one configuration of theoretical and practical knowledge, is defined by Kant as pure apperception (KdRV B 131 ft). This is a spontaneous activity of the mind and the primeval force of our self-consciousness which first of all gives rise to the conception of 'cogito ergo sum'. Pure apperception (as distinguished from empirical apperception) is thus the condition a priori which enables identity of the self as well as enables us to acquire coherent knowledge. It is prior to actual experience and the conditio sine qua non to 'assemble' many perceptions and ideas which relate to each other, so as to make up our consciousness and our personal experience. In dealing with the necessary conditions for synthesis of the manifold of things which are perceived (synthesis speciosa), and the manifold of thoughts (synthesis intellectualis), Kant introduces another concept labelled Einbildungskraft (called facultas imaginandi by Leibniz), which is defined as the capacity to perceive an object without its actual presence. In particular, Kant employs this concept to mediate between perception as based on sensory data, and apperception which is purely intellectual. He distinguishes between two types of Einbildungskraft, one which he labels 'productive', and one which he labels 'reproductive'. Synthesis guided by reason (Verstand) and its categories is achieved by the productive Einbildungkraft which is assigned to the spontaneous activity of our mind, whereas reproductive Einbildungskraft is a process driven merely by association, and thus by rules established by empirical psychology. The concept of Einbildungskraft plays a major role also in Kant's theory of aesthetics. In the Critique of Judgement (Kritik der Urteilskraft [KdU], 2nd ed. 1793), both the productive and the reproductive types are considered with respect to artistic expression. While the reproductive type of Einbildungskraft again is discussed with reference to association and memory, the productive type is regarded as a major factor which accounts for aesthetic creation as well as aesthetic experience (KdU 68ff, 192ft). In particular, productive Einbildungskraft leads to artistic 'ideas' which tend to transcend everyday experience (e.g., surrealistic paintings, arrangements of sound that bring about auditory illusions). Aesthetic ideas (transformed into works of art), cause us, in Kant's opinion, to think a lot even though it may be impossible to explain such ideas by speCific concepts or notions (KdU 193). In this respect, because of a wealth of intrinsic meaning, works of art may withstand complete description and

10

PERSPECTIVES AND CHALLENGES OF MUSICAL IMAGERY

explanation by means of language. Productive Einbildungskrajt, however, is also used by the viewer or listener who perceives works of art. Because of its formal relation to reason, Einbildungskrafi accounts for validity of aesthetic judgements even though such judgements are also a matter of taste, and of subjective pleasure (Wohlgefallen). As Kant elaborates throughout this work, the main reason for such pleasure is that works of art exhibit expediency without a formal purpose (Zweckmiissigkeit ohne Zweck). The high degree of coherence and expediency which we register when looking at DUrer's Melancholia, or listening to Bach's Art ofthe Fugue, can be attributed to our productive Einbildungskrajt which is spontaneous and free on the one hand, yet also rational in certain ways, on the other hand. It has recently been argued again that without imagination, hearing music as music would be impossible because of the metaphorical nature of many, if not most compositions (cf. Scruton, 1997, ch. 3; see below).

Psychological approaches We shall now turn to Brentano's Psychologie vom empirischen Standpunkt as well as to other writings of this scholar who investigated philosophical foundations of psychology. The very center of Brentano's approach is the analysis of human consciousness and the principle of intentionality. It is characteristic of all acts of our consciousness (Bewusstseinsakte) that these are directed towards objects which cover much more than things found in the world around us. The main reason for this is that we are capable of what Brentano calls inner perception, distinguished from outer perception which is based on sensation (Brentano, 1974, pp. 40ff and 104ff; as to the principles of Brentano's psychology of 'acts' which can be distinguished from various approaches to psychology of 'content' , see Boring [1950, chs. 17, 18, and 19].). Objects of inner perception may for instance be judgements and decisions. However, also objects usually understood as fictional, such as the flying horse Pegasus, or angels singing with a 'crystal voice', can become the actual 'content' of our consciousness, and will in this respect be regarded as 'real'. This point is of great relevance when we consider, for example, artistic invention. We might point to the surrealistic movement, and to Salvador Dali in particular who propagated a concept labelled 'critical paranoia' which is based on both rational analysis of phenomena and seemingly irrational images which are however totally coherent and meaningful (see Gomez de la Serna, 1977). In music, works of the composer Bernd Alois Zimmermann (19181970), such as Monologe (for two pianos) and the Musique pour les soupers du Roi Ubu, come to mind, works which are the offspring of what Zimmermann called pluralistic composition. 1 This approach incorporates techniques of montage and collage whereby spontaneous ideas and the momentary content of consciousness contribute to the emergence of a new arrangement together with various resources stored in memory. Pluralistic composition means that many small musical elements (themes, motifs from well-known works plus tunes from jazz standards etc.) which come to the composer's mind are used as building blocks for a type of work which also makes use of different textures, changes in dynamics, etc. Since composition implies meaningful arrangements of musical elements (which may be heterogeneous with respect to

ALBRECHT SCHNEIDER AND ROLF INGE GOD0y

11

historical and cultural criteria, as in this case), composers quite often start their work with sketches which reflect basic ideas and other 'early' conceptualizations. In his psychology, Brentano grouped all intentional activity of the mind into three classes, namely (1) conceiving (Vorstellen), (2) judgements (Urteilen), (3) emotions and feelings (Gemiitstiitigkeit). The German term Vorstellen could be translated also by imagining as well as by imaging, since in English to form a conception means to imagine (to form a mental image of something not present) and/or to image (to call up a mental picture of something not present; see also Casey [1976]). Psychic phenomena are related to imagining, which in turn typically will be combined with judgements. Imaging, to be sure, does not in the first place refer to the objects which are imagined, but to the psychic act of imaging. For example, listening to music leads to imaging, the object of which being the listening process itself as a psychic phenomenon, whereas Brentano defines the sound that we listen to as a physical phenomenon, and thus the object of outer perception (Brentano, 1924, pp. 170ft). If we imagine a certain sound or chord as part of inner perception and are aware of this so that we have an image of us as imaging, this process would according to Brentano take place in but one psychic act of our mind. It would include, however, two different objects, one being a physical phenomenon (the sound), the other a psychic phenomenon (the act of listening). In accordance with Aristotle, Brentano regards the physical phenomenon of sound which we listen to as the primary object of listening, and the psychic phenomenon of listening as the secondary object which is 'perceived' (The notion of perception in a phenomenological perspective deserves a more detailed discussion; cf. Chisholm [1956]). This type of inner perception also relates closely to apperception, a term which was used by Descartes, Leibniz, Kant and others (see above). To understand fully something which enters our mind either by way of sensations and outer perception, or by imaging, requires mental activity such as vigorous attention, concentration, memory, and forms of judgement, such as comparison or distinction. Only this will result in valde clare ac distincte percipere, as was stated already by Descartes. Brentano argues that there are two different modes relevant to how things may come to our minds. The first he describes as explicit and distinct, the second as implicit and indistinct (Brentano, 1928, pp. 33ft). If, for example, someone listens to a chord and is able to distinguish the different notes which make up this chord, he or she will be able to identify the chord as being of the 'Tristan'-type (cf. Vogel, 1993, pp. 478-481), and will also be aware of listening to this specific chord. This awareness would qualify as explicit and distinct, whereas someone who perceives the chord as one entity without further analysis would only reach the level of implicit and indistinct knowledge. It is this type of explicit and distinct perception accompanied by conscious awareness of the act of perceiving which yields what phenomenologists from Brentano to Husserl have accepted as 'evidence'. The epistemological concept of Evidenz, and the term itself, especially in Brentano's philosophical (but still empirical) psychology, is fundamentally connected with inner perception. Husserl, who was a student of Brentano and Stumpf, argued that such Evidenz could stem also from outer perception, and in this respect has stressed the importance of apperception which takes place regardless of whether we deal with sensory input or with things we only conceive of

12

PERSPECTIVES AND CHALLENGES OF MUSICAL IMAGERY

(Husserl, 1901, pp. 222ft). In particular, Husserl points to the fact that since apperception involves mental acts, it will result in a conception of what has been perceived. This conception is of course not just a copy of things in the world around us but the phenomenon (Erscheinung) we conceive of. Phenomena can be understood as the intentional correlates of acts of perceiving. Several phenomena, for example musical tones, may appear to be similar in such a way that they allow abstraction of invariant features, and to conclude from here as to the essence (Wesen, 'Eidos') of all musical tones (Husserl, 1976, pp. 410ft). It is from this point of view that Husserl also developed his theory of categorical perception (Husserl, 1901, pp. 40ft). Categorical perception means transcending perception based on actual sensation, and is defined by perceiving what Husserl labels ideal objects, objects distinguished from real objects ('ideale' versus 'reale' Gegenstiinde). Ideal objects are constituted by way of mental acts such as comparison, judgement, and, in general, abstraction, whereas real objects are sensed in direct access. Perception of real objects is more simple since it is achieved in but one step. By contrast, categorical perception may involve several acts to constitute ideal objects. Thereby, Husserl's approach to categorical perception is quite different from that of contemporary psychology, linguistics and musicology where 'categorical perception' often is understood as perceiving stimuli which subjects order and assign to a limited number of learned 'categories' (cf. Schneider, 1997a). In this perspective, 'categorical' means 'classificatory' in the first place. Husserl (1976) has pointed out that this type of classification is basic to human modes of rational thought which, however, comprise also more abstract procedures.

Listening to music: the constitution of ideal objects in the 'time domain' Listening to a work of music (or music which is improvised according to some preconceived scheme) is a task which involves constitution of ideal objects, namely grasping the compositional structure as well as principles of musical form such as symmetry and contrast, repetition and variation. Constitution of ideal objects is achieved, most of all, by way of abstracting formal properties of such an object from actual sensations as well as by making reference to knowledge acquired earlier. Regarding works of music, this knowledge will relate to the syntactic, pragmatic and semantic levels, respectively, and is needed to grasp the 'meaning' inherent in both formal structures as well as in particular textures or even in the composition as a whole (cf. Stoffer, 1996). Since music is an art which is centered in the 'time domain' , the ongoing process of constitution in actual listening has to be worked out along the time axis by many consecutive acts of consciousness. Brentano had devoted much labour investigating continua, and how continua such as space and time could be perceived. He discusses the concept of Proteriisthese which has to be regarded as eine Reihe von kontinuierlich sich folgenden Wahrnehmungen, that is, perceiving is an act which is repeated again and again along the time axis, and results in a sequence of 'frames' of perception from some point in the past to the present. 2 To perceive objects such as melodies which unfold in time, it is necessary, however, that the parts of the object which have been perceived already still be present while the next parts follow. Edmund Husserl

13

ALBRECHT SCHNEIDER AND ROLF INGE GOD0Y

(who was a student of Brentano and for a short period, also of Stumpf) discusses such problems in his theory of inner time consciousness (Husserl, 1928), where he elaborates on such principles as protention (anticipation and expectancy, Erwartung or Protention in German) and retention (that is, Erinnerung vergangener 'letztpunkte', recollection of such 'points' of the present which have just passed, yet have not ceased to exist in memory), as well as on the idea of the 'present' (letzt, Gegenwart). Husserl (1928, p. 385) explicitly relates to Brentano in that a sequence of acts of perceiving forms an Aktkontinuum which with respect to temporal objects includes retention, actual perception, and protention. Since the temporal object, e.g., a melody or even a work of music, extends over a certain stretch of time, retention, perception, and expectancy cover certain parts of this Zeitstrecke: past (retention in memory) .....I... (

actual perception ('now')

future (expectancy) ...

Husserl's considerations are of interest especially regarding the perception of configurations organized in time such as melodies. 3 His views have been influential and are found in writings on perception and music of, among others, the social scientist Alfred Schlitz (1976), and the musicologist Thomas Clifton (1976, 1983) who also drew on ideas of the French phenomenologist Maurice Merleau-Ponty (MerleauPonty, 1945). Clifton (who sadly passed away much too young) applied Husserl's concept of protention, retention etc., to the analysis of compositions, for instance, to Webern's Bagetelle No. J for String Quartet, Ope 9. Jean-Paul Sartre (1940) who also adopted elements of Husserl's philosophy, argued that the constitution of a musical work - as an example, he pointed to Beethoven's 7th symphony - by way of listening can only result in an analogue of the musical object created in an actual performance. Through the process of listening, the work as a whole is in the end imagined rather than perceived. This is the reason, Sartre concludes, that we have difficulties to return to 'reality' of everyday life after attending a concert. In a systematic treatise on the ontology of musical works (which are neither identical with their score nor with the manifold of their realizations by way of performance), the Polish philosopher Roman Ingarden (1893-1970), also a student of Husserl, has elaborated on the conceptualization of musical works (Ingarden, 1962). He argues that in one respect, the constitution of the work of music as an organized whole by an experienced listener is achieved because the parts of the work are structured in time according to hierarchical levels. Thus, when listening to music, we register both the constituents of individual compositions as well as the hierarchies inherent in a sequence of different parts which in total make up the temporal Gestalt experienced as a work of music. This experience also means that different parts are perceived to be of different 'weight' with respect to the overall configuration, and also of different quality in terms of aesthetic criteria. Consequently, the time of musical experience is not homogeneous, since the 'transfer rate' of musical information and meaning is uneven due to changes in complexity and intensity of a work which is performed, or played from some recording, to a listener. Detailed considerations of these issues are

14

PERSPECTIVES AND CHALLENGES OF MUSICAL IMAGERY

found in the Musikpsychologie of Ernst Kurth (Kurth, 1947) who -like Ingarden - was influenced by the concept of time consciousness and subjective experience of time put forward by Henri Bergson. Perhaps one of the most extensive applications of phenomenological ideas in music theory can be found in the work of Pierre Schaeffer. One aim of his monumental Traite des objets musicaux (Schaeffer, 1966) was to give a foundation for characterizing sound features in a general way, that is, sound not restricted to that of traditional musical instruments or the human voice. On the way to establish a multidimensional matrix for characterizing any sound object based exclusively on the subjective listening experience, Schaeffer goes through a number of exclusions of what the sound object is not (Schaeffer, 1966, pp. 95-98; here quoted from a slightly different summary in Chion [1983, pp. 34-35]): The sound object is not the sounding body. The sound object is not the physical signal. The sound object is not a fragment of a recording. The sound object is not a symbol notated in a score. The sound object is not a state of the soul. This is a progressive ontological differentiation running from the source of the sound to the intentional constitution of the sound object in the 'listening consciousness' (Schaeffer, 1966, p. 147). In fact, the 'ultimate' reality for Schaeffer is clearly the intentional constitution of the sound object, and later on (with addition of certain criteria), of the musical object, as an intentional unit in our minds, hence as a rich and vivid instance of musical imagery. His entire typo-morphological matrix ('Tableau recapitulatif du solfege des objets musicaux' in Schaeffer [1966, pp. 584-587]) can in fact be seen as a mental technique for guiding our scrutiny of internal images of musical objects by establishing progressively finer differentiations of the various feature dimensions in the musical object through shifts in intentional focus. The question then is if musical experience ruled by individual intentionalities is similar for a number of subjects. In general, experienced listeners can be expected to perceive formal structures in works of music in similar ways. However, subjects have different thoughts and different imagery of what they hear, something which may be understood as 'unasserted thought' (cf. Scruton, 1997, ch. 3). Also, musical structure and processes are often conceived of, and described, in a rather metaphorical way. For example, the changes in instrumentation, tempo and dynamics which are found in many symphonic works have been interpreted in terms of forces, energy, and maUer, the interplay of which brings about movement, as well as shape, in music (see, e.g., Kurth, 1947). But musical imagery is by no means restricted to subjective 'associations' or 'connotations' (cf. Meyer, 1956, ch. VIII). Rather, seemingly metaphorical conceptions of music may reflect features of the temporal organization of a work (or improvisation) fairly well. The reason for metaphorical conceptualizations, as well as imagery, has been explained thus: In hearing music (which is based on, yet not restricted to, organized sound), we develop a kind of double intentionality: 'one and the same experience takes sound as its object, and also something that is not and cannot be sound - the life and movement that is music. We hear this life and movement in the sound, and situate it in an imagined space...' (Scruton, 1997, p. 96). Music in the

ALBRECHT SCHNEIDER AND ROLF INGE GOD0y

15

imagined space thereby appears to move up and down, melodies are felt to be 'rising and falling', etc. Since the impression of 'rising' and 'falling' of course is connected with physical parameters such as 'high' and 'low' frequencies, the space of musical imagery would basically comprise the same dimensions as does the phenomenal space of normal experience (See Schneider [1992] for a discussion of 'tonal space' and phenomenal attributes of sound).

Stumpf and Riemann We now turn to Carl Stumpf (1848-1936), another pupil of Brentano, and one of the founders of systematic musicology. Stumpf has addressed fundamental issues in perception and cognition in several of his books as well as in other publications. Besides the two volumes of the Tonpsychologie (1883, 1890), there are two philosophical treatises which are of relevance, namely Erscheinungen und psychische Functionen (1907) and Empfindungen und Vorstellung (1918); also, there are chapters dealing with perception in Stumpf's book Die Sprachlaute (1926) and especially in Vol I of his Erkenntnislehre (published posthumously in 1939). In particular in his Tonpsychologie he considers various aspects of auditory perception as related to mental acts (labelled by Stumpf, and also by Klilpe, psychische Funktionen, psychical functions; cf. Stumpf [1907, 4: Akte, Zustiinde, Erlebnisse]), such as judgement, comparison, recognition, conceiving, imagination, emotion, desiring, and intentions. In his treatise Empfindung und Vorstellung (1918), Stumpf gives a systematic account of how sensation and imagination differ in intensity yet also in quality.4 Stumpf further argues that imaginations can be modified any time at will, whereas sensations are more resistant to subjective interpretation. Also of interest are Stumpf's remarks on how actual sensations (of, e.g., musical tones) might be complemented and reinforced by recollections of the same objects. Of the many problems discussed in detail in the Tonpsychologie, one might mention Stumpf's approach to scaling by way of subjective estimation of stimulus differences cognitively turned into distances on one or several dimensions. This of course implies that subjects conceive of the perceptual difference as a spatial distance. One interesting case Stumpf investigated is how we try to estimate the relative distance of two tone complexes (clusters) with respect to a single dimension of pitch (Tonpsychologie II, 1890, pp. 406ft). Since such clusters cannot be easily analyzed into their components (they appear to be 'complex wholes'), subjects are forced to make judgements based on a phenomenal difference which then is translated into a distance. The distance itself, however, is not perceived, yet is a spatial model employed in cognition. Cognitive analysis of complex sounds, as well as musical structures, typically involves images because musically trained subjects can be expected to mentally project the sounds they hear, and the notes they apprehend in terms of a musical syntax, on a spatial model. If we listen, for example, to the chord c-e-g-bD-d' made up of complex tones (played by, e.g., five saxophones), we might conceive of the resulting sound structure in terms of a two- or three-dimensional spectral representation. Also, the notes could be projected onto a two- or three-dimensional structure known to music theorists as 'tone net' or 'tone lattice' (see Riemann, 1914/15; Fokker, 1945; Vogel,

16

PERSPECTIVES AND CHALLENGES OF MUSICAL IMAGERY

1993). The spectral representation would be helpful in understanding acoustical properties of the sound, for example, the coincidence of partials which belong to different notes. Projecting notes on a tone net which is made up of constituent musical intervals (fifths, major thirds, 'natural' seventh) can be useful if we want to understand the harmonic structure of chords as well as of textures comprising several notes played simultaneously (see below). Stumpf's theory of consonance is based on the principle of Verschmelzung which is not to be understood simply as a sensation of fusion or coalescence of a set of tones (or chords). Rather, Verschmelzung is a special case of Gestalt perception of several sounds (e.g., musical notes) played simultaneously (see Gurwitsch, 1975, pp. 66ff; Schneider, 1997b). As we hear such a mixture of several complex harmonic sounds which (in just intonation) blend perfectly, we employ acts of both analysis and integration so that we may 'switch' between apperception of several or even all the constituents of the harmonic complex, on the one hand, and perception of a highly integrated whole, on the other. Perceiving configurations such as chords thus means that subjects may concentrate, at one instant in time, on the constituents in their harmonic relations (e.g., the notes as well as the intervals they form), and on the resulting sonorous object in the very next moment. Stumpf investigated this process of focusing attention on either the constituents or the resultant whole experimentally by listening again and again to complex chords played with mixture stops on a pipe organ tuned to just intonation. As to the specific experience of Verschmelzung, Stumpf argued that this quality is valid also if one only imagines two tones, c and g, to be played simultaneously (Tonpsychologie II, pp. 138ff). In this respect, imagery would not differ from listening to real notes. There are other phenomena, though, where sensation and imagery are not the same anymore. For example, Stumpf points to two notes, c and which if played on an instrument simultaneously will cause roughness and/or beats. He says that he could imagine those two notes either with no beats at all, or with beats, whereby the number and intensity of such beats could be freely modified. On the other hand, Stumpf admits that hearing in certain cases exceeds the capabilities of imagery. From experiments he conducted on himself (Tonpsychologie I, p. 179), he found that even though one is able to perceive a very high note (at the very limit of human hearing), it is impossible to clearly imagine such a stimulus if it was not sensed just prior to imagining. Notwithstanding his keen interest in cognition, Stumpf took a nativist stand in many issues of auditory perception. He admitted that consonance is in the first place a matter of sensation, whereas harmony is much more dependent on apperception and relational thinking (cf. Stumpf, 1898, 1911). He therefore distinguished between consonance and concordance, the latter being a cognitive principle: if we listen to music composed in tonal harmony which is played with poor intonation, the actual sound structure can hamper the sensation of consonance. Poor intonation, however, cannot prevent us from conceiving the musical structure in terms of tonal harmony, as far as musical syntax is concerned, and in terms of just intonation, if we imagine how the music should have been played correctly. With respect to this distinction, Stumpf was thinking in particular of tonal music played on a piano tuned in equal temperament where we have only twelve keys per octave to realize many more notes. Stumpf recognized that given such circumstances,

17

ALBRECHT SCHNEIDER AND ROLF INGE

all chords, and in particular all chords and sonorities comprising more than three simultaneous notes, will be ambiguous when played on a piano or other keyboard tuned in equal temperament since they can have different functions within a harmonic texture, and thus, different 'meanings'. A fact which further complicates the issue is that also notation is often equivocal in the sense that c-sharp and d-flat, g-sharp and a-flat etc. are taken so as to represent the 'same' note. With respect to conceptualizations of tonal music, Stumpf, and also Hugo Riemann (in Riemann, 1914-16), did argue that expert listeners will not take a for an ab yet will conceive of notes with respect to the harmonic context notwithstanding actual intonation which might be poor or even explicitly wrong. The problem addressed by Stumpf and Riemann is in fact that of the widespread use of equal temperament. Tonal relations which can be conceived of, and be found in compositions, to be complex and diversified, will be levelled if works rich in harmony are played on a piano or organ. To illustrate the case, consider a simple cadence comprising the chords C-d-F-D-G. To realize this cadence in just intonation would necessitate that we have the following notes (and pitches) at hand:

I

I

I

I

f-c-g-d-a

(horizontal rows

= perfect fifths 3/2)

(vertical columns = major thirds 5/4)

It is easy to see that the 'a' of the F-major chord is not identical with the 'a' of the D-major chord, as these are in fact two different pitches with respect to intonation (which differ by a syntonic comma of 22 cents). Also the 'd' in the d-minor chord is not identical with the 'd' of the D-major chord. Since on a piano or other keyboards, however, there is only one 'a' as well as only one 'd' per octave available, these have to be regarded as 'compromise pitches' in equal temperament which are used to realize various chords which differ with respect to interval structure and tonal relations. It is the expert listener who has to find out which 'a' and which 'd' is needed to realize certain chords. Therefore, he or she has to conceive of the harmonic structure which, according to Stumpf (1911, 1926) and also Riemann (1914-16), by expert listeners is always done on the basis of pure intervals such as 2/1, 3/2,4/3,5/4,6/5 etc. Thus, even though music as actually played on a piano or other conventional keyboard deviates in intonation from the tonal relations the composer had in mind when writing chords, chord progressions and modulations, the expert listener is capable of the appropriate conceptualization of the harmonic structure. The tertium comparationis which enables relating the actual performance of a given work to the rule system of tonal harmony, would be to take the harmonic texture 'as if' it was written to be played injust intonation. Thereby, it could also be perceived in an unambiguous way. The expert listener, according to Riemann, is someone capable of analysing complex sequences of chords in terms of correct Tonvorstellungen (Riemann gives some examples in the Jdeen). In practice, this means that the listener would have to abstract the correct harmonic scheme of a given piece from the notation and the intonation which both can be quite ambiguous. Regarding conceptualization, Riemann tentatively discussed a number of principles such as what he calls 'economy of listening'. That is, in actual listening, simple tonal relations based on

18

PERSPECTIVES AND CHALLENGES OF MUSICAL IMAGERY

small integer ratios should in general be preferred against more complex ones. 5 Riemann's ldeen certainly were a serious attempt at formulating a theory of harmonic imagery (this being a central part of musical imagery). Unfortunately, Riemann died soon after he had published his ldeen which he regarded as a preliminary work. So far, there have been very few attempts at putting Riemann's assertions to test. He had claimed that, by systematically studying the tonal relations inherent in certain works of music, one might be able to take these as 'tracks' (or traces) of the composer's imagination which thereby would become accessible. Riemann himself believed to have understood, most of all, works of the late Ludwig van Beethoven (piano sonatas and string quartets) of which he had made extensive analyses. As interesting as his theory of Tonvorstellungen is from the point of view of cognitive musicology, to validate this concept will necessitate closer examination as well as experimental research. Imagery, which Riemann had addressed regarding harmony, apparently plays a role also in timbre perception. Stumpf found that, in order to identify different timbres, as well as to distinguish instruments and voices by their respective 'tone colours' , listeners try to single out characteristics like brightness, sharpness, density, fullness etc. from complex sound entities. Thereby, perceptual analysis of sound qualities results in distinguishing timbres on the basis of phenomenal attributes, attributes which with respect to complex sounds can be considered dimensions of what Stumpf (and also his student, Wolfgang Kohler) describes as Tonfarbe (tone colour; see Stumpf, 1926, especially ch. 15). Conceptualizations of timbre again involve spatial characteristics which are helpful in the classification of sounds. Of course, certain qualities we perceive in complex sounds must have correlates in the time function and spectrum of the physical stimulus as well. However, different timbres seem to be perceived, analyzed, and compared to each other with reference to such dimensions which stem from, and are bound to, the phenomenal appearance of sounds. To subjects perceiving sounds produced by various instruments, some appear to be 'fat' or 'voluminous', others are regarded as 'thin', 'sharp', 'dense' or 'hollow'. Subjects tend to imagine such specific sound qualities along with certain instruments. It should be noted that Stumpf's findings and thoughts on timbre have been acknowledged as outstanding in a more recent publication on sound colour (Slawson, 1985).

Aspects of research on musical imagery Besides Stumpf's observations on Verschmelzung and Tonfarbe which were based on several experiments he had carried out alone or with co-workers and students, some interesting research on musical imagery was started by other psychologists, and from somewhat different angles. As early as 1885, Hermann Ebbinghaus had published a book on memory which deals with aspects of learning as well as memory for, and reproduction of, learned verbal items. The findings reported by Ebbinghaus (1885) were the result of extensive experimental work which includes statistical analyses (see Boring, 1950, pp. 386ft). With the publication of Ebbinghaus' study, memory became a topic which attracted many psychologists. To be sure, memory with respect to tones and other musical objects is extensively covered also by Stumpf (1883, 1890), but one

ALBRECHT SCHNEIDER AND ROLF INGE

19

of the most remarkable works from this period is the essay 'On "Gestalt Qualities'" by Christian von Ehrenfels from 1890 (Ehrenfels, 1988). In this essay, von Ehrenfels describes a holistic approach to musical imagery, meaning that imagery is enhanced when as many as possible of the senses and contextual elements are mobilized, e.g. in order to imagine an orchestral work, we should imagine a scene with the entire orchestra in front of us, the conductors movements, the concert hall, the lightening in the concert hall, the ambiance, etc. Another area of research which became influential within the phenomenological movement, on the one hand, and Gestalt psychology, on the other, was that of eidetic imagery (for an introduction to the field as it was developing around 1920, see Jaensch [1927]). In particular, this research centred on subjects ability to recollect objects (which had been perceived earlier) and to reproduce them as visual or acoustical images as precisely as possible. Eidetic images were regarded as intermediate between perception - which, according to a broad and well-known definition is the experience of objects and events which are here now (Newman, 1948, p. 216) - and imagery, which deals with mental images being the product of 'free' or 'pure' imagination (see section Philosophical approaches (pp. 6ft) above). Regarding eidetic memory in music, there are some famous cases reported in the literature: Mozart is said to have been able to write down Allegri's Miserere in full after hearing it only once or twice. 6 Also, he is said to have had stored hundreds, if not more, compositions in memory (Knepler, 1991, ch. 2). Further, there are some sources - unfortunately dubious as far as philology is concerned - relating to Mozart's imagery as being a central part of his creative activity (see Duchesneau, 1986, pp. 103ft). Another case of interest is Beethoven who - after his sense of hearing became impaired - did write some of his greatest works, something which could of course not have been achieved without an unusual strength of imagery. Quite many composers have said that they had 'visions' of a complete work before they did put anything to paper (see Duchesneau, 1986, pp. 103-109). We may also point to reports from neurologists who have observed patients suffering from serious brain dysfunctions, yet still seem to be able to recollect a large number of melodies, or even complete works of music. Perhaps one of the most unusual cases is that reported by Sacks (1985, ch. 22). In German psychology of the 1920's, there were attempts to classify eidetic images with respect to memory. Shortly after a (visual) stimulus has been presented, an after image (Nachbild) will remain accessible to the subject who, after some time (in general, several minutes), will also be able to form an Anschauungsbild from recollection which correctly gives the features of the original stimulus. Finally, after more time has elapsed, it is possible for many subjects to recover what they had perceived earlier as an image (Vorstellungsbild). Apparently, these images are formed by means of 'retrieval' of information stored in long-term memory. To check whether this tripartite classification would hold also for musical imagery, Rudolf Kochmann (1923) carried out experiments with school children, because it had been found by some researchers that eidetic memory of the Anschauungsbilder type develops to a maximum during childhood, and seems to degenerate later on. Kochmann played sequences of up to ten notes, sequences which did not correspond to any popular melodies, to individuals (boys of age 10 to 17 who attended different types of schools, and also differed with respect to musical training and abilities) who

20

PERSPECTIVES AND CHALLENGES OF MUSICAL IMAGERY

were asked to recollect and sing these sequences after a break in which the subjects were set to do other musical tasks. The reproductive task (singing the tone sequence which had to be recollected) started five minutes after the sequence had been presented 'first, and was repeated after another five minutes as well as after twenty minutes. The results of the experiments showed that subjects varied substantially with respect to the extent and the precision of recollection (as well as reproduction) of quasi-melodic tone sequences. Kochmann concluded on the basis of his observations that there is no clear boundary between recollection (Anschauungsbild) and imagery (Vorstellungsbild) of musical objects such as tone sequences. Not so many experimental studies on musical imagery were published subsequent to Kochmann's.7 One investigation published by Mainwaring (1933) explores kinaesthetic imagery, as for example in the case of piano players who can be expected to recollect musical phrases by imagining the action of their hands and fingers playing a certain melody or piece. (As to imagery and motor behaviour, see and also Reybrouck [this volume].) More recently, experiments were carried out which checked the influence of imaged pictures and sounds on the detection of visual and auditory signals (Segal & Fusella, 1970). It was found, not surprisingly, that mental imagery can hamper actual perception of both visual and acoustical stimuli if perception has to be achieved during the same time as the subjects are occupied with imagery. Also, it was found that whereas intra-modal (visual or auditory) imagery did affect perception, cross-modal imagery did not. This indicates, besides other evidence (see also Kosslyn, 1980, 1994), that imagery basically employs the same neural 'channels' as well as cognitive mechanisms as are needed for perception. Among the experiments related to imagery for melodies and melody-like phrases, there were some which by their design are not too far away from Kochmann's investigations. In the study of Weber & Brown (1986), subjects were asked to designate the contour of simple melodies (all in 4/4 meter) which they heard as sung from tape. This was done by writing horizontal lines of equal length, one for each quarter note, whereby the position of the lines relative to each other represent relative pitches of the melody (as well as steps of a diatonic scale). The melodies were either sung with the proper words ('song condition'), or only with the syllable 'ba' ('melody condition') on each note. Subjects in one trial were allowed to sing the melodies or songs aloud while drawing the contour, and in a second trial had to designate the contour while auditorily imagining the musical phrase. The parameters measured were processing time and errors made by subjects. No significant effect as to the overt versus the imagined condition was observed. However, songs were processed faster than were melodies alone. The results suggest that, with respect to this specific task, overt singing does not give better results in recognition of simple melodic phrases than does imagined singing. Other studies considered imagery of (complex) tones and chords (Hubbard & Stoeckig 1988) as well as imagery for timbres (Crowder 1989, Pitt & Crowder 1992). Hubbard and Stoeckig confirmed some of the observations already made by Stumpf (1883,1890,1907,1911,1918), namely that images in certain cases can substitute perceptions, and that the time needed to form images of, for example, chords or other musical objects in general, increases with complexity. Crowder had subjects judge whether or not two tones of a pair were equal in pitch. The timbre of the two tones

ALBRECHT SCHNEIDER AND ROLF INGE GOD0Y

21

could be either the same or different. From the reaction time analysis Crowder (1989, p. 474 and p. 477) concluded that people are faster to judge that two tones have the same pitch if they also have the same timbre than otherwise. This result remained virtually the same when the first of the two tones was an internally generated image of a timbre rather than a true tone. In experimental work related to that just mentioned, Pitt and Crowder (1992) found that for imagery of timbre, spectral characteristics are most important for subjects. This is of interest since in many investigations on timbre perception, the transient and onset portion of sounds (plucked, bowed, blown etc.), and thus temporal and dynamic features, turned out to be a cue for identification of timbres (see, e.g. Iverson & Krumhansl [1993] and references there), which however may sometimes have been overestimated (see findings in Reuter [1995]). Musical imagery is not only closely connected to perception and memory, it relates also to research in synaesthesia and to semiotics of music. As to the latter, it is interesting to see, for example, what musically trained subjects conceive of works they either only hear, or read as a score, and which 'connotations' arise from either hearing or reading a work which evidently calls for formation of images (for a case study, see Schneider 1995).

Challenges for musical imagery research One conclusion to our brief overview here is that musical imagery is a composite or 'impure' phenomenon in the sense that it comprises many things at the saIne time. In line with this, and in looking forward, it could be useful to make a short assessment of what we see as some important challenges for the study of musical imagery. Obviously, there is a need for more knowledge about the neurological bases of imagery. An overview of this is presented in the next chapter (Janata, this volume), but from what is already known today, there seems to be a 'functional equivalence' between perception and imagery, meaning that much of the same neurological substrates involved in 'primary' perception are also involved in 'pure' imagery. The distinction between perception and imagery seems then even on the neurological level not so clear-cut, something which may help us understand better the complex interactions of more 'primary' perception and more 'pure' imagery, a topic which we saw in the previous sections of this chapter has been recurrent in the history of musical imagery. Related to this is the issue of the neurological bases of contextual images, i.e. the workings of memory images of the recent past and expectancy images of what is to come next, what in the above mentioned terminology of Husserl was called respectively 'retentions' and 'protentions'. Also, advances in knowledge of the neurological bases of cross-modality (Stein & Meredith, 1993) could hopefully help us to better understand what triggers images of musical sound in our minds. Various studies referred to above seem to suggest that there are in many cases strong links between visual imagery, motor imagery, and musical imagery. There are many questions concerning the relationship between imagery and various schemata in perception and cognition (see for instance chapters 4, 5, and 8 in this book). Does musical imagery follow learned schemata, and are various schemata for musical sound (such as categories for pitch relationships, for harmonic, melodic

22

PERSPECTIVES AND CHALLENGES OF MUSICAL IMAGERY

and formal elements, rhythmic patterns, etc.) really instances of more long-term or 'slower' kinds of musical imagery? Related to this are questions about the ecological content of musical imagery, meaning the 'concrete' and particular qualia of the images as opposed to the more 'abstract' structural features, e.g the distinction between a detailed, salient image of a particular vocal performance of a well known tune and a more indistinct and 'generalized' image of that tune. This distinction between 'concrete' and 'abstract', adopted from Pierre Schaeffer (Schaeffer, 1966), could also be seen as a distinction between particular and general, or between low-level and highlevel features in images, something which has been studied in visual imagery (Rouw, Kosslyn & Hamel, 1997). This is again related to the ecological constraints at work in musical imagery, meaning the question of whether all the material in musical imagery is derived from experienced sound and is in accordance with principles of sonorous behaviour in the 'real world', e.g. that when we imagine a tune, we also imagine some kind of 'carrier' or performance of this tune, be that our own sub-vocalizations, our own imagined fingers moving along a keyboard, imagining someone else singing or playing, etc. It could be tempting to use the term 'dynamics of musical imagery' to denote not only the possible shifts between different qualia of images as just mentioned (e.g. shifting between different timbres in the imagery of a well known tune), but also to denote a number of other apparently not well explored aspects of musical imagery. For one thing, there will probably be highly variable degrees of salience or acuity in our images of musical sound, e.g. sometimes we may have very clear and intense images of sound, at other times images may seem pale and distant. This does possibly have to do with priming and 'recency' effects, but it could be very useful (in particular for practical applications of musical imagery) to have a better understanding of such shifts in acuity. Related to this is the apparent possibility of variable resolution in musical imagery, meaning that we are probably all capable of zooming in on detail, re-playing some fragment again and again or in slow motion, zooming out, playing 'fast forward', etc. Even the vague, macroscopic, retrospective, cumulative images of long works of music, e.g. the sense of recollecting an entire concert 'in a now' , could be included in this dynamics of musical imagery. And of course, musical imagery has some very practical applications, and may be considered as integral to musical craftsmanship. For this reason, 'ear-training' (or solfige) has traditionally been a part of the curriculum of most schools of music. There is much to be said about the efficiency of the pedagogical methods used in ear-training, and it seems quite clear that this subject could profit from advances in the understanding of musical imagery, in particular when we consider the demands placed on composers and arrangers to make reasonable predictions of how their compositions or arrangements are going to sound. The same goes of course for conductors and for other performers as well, and will here be closely linked with mental practice of the motor components of performance. Notably, this is not only relevant for the professional musician, but also for music education at all levels. We can for instance think of string instrument education for children where imagining sound is a crucial element and has in fact been implemented in some methods of teaching. Such practical applications of musical imagery, i.e. the capability of generating salient images of musical sound more or less at will, seems to be one of the least

ALBRECHT SCHNEIDER AND ROLF INGE GOD0y

23

studied, yet in our opinion most crucial aspects of musical imagery: What is it that triggers images of musical sound in our minds, or what is the 'engine' or the driving force? There are some studies which suggest that the triggering of musical imagery is closely linked with motor imagery (Mikumo, 1994, 1998), meaning that imagining sound-producing actions will also trigger mental images of the resultant sounds. This could be understood as related to the idea of 'motor theory' in perception and cognition, a theory which has been controversial but which now seems to gain support from the application of brain observation techniques, producing data which suggests that motor areas of the brain are indeed involved in musical imagery tasks. From these last remarks, we think it is fitting then to conclude this introductory chapter by situating musical imagery at the intersection of musicianship and several scientific disciplines. This means that musical imagery will be the meeting place for subjective images and more universally observable phenomena, something which in turn will mean that the study of musical imagery will have to draw both on personal introspections and various inter-subjective methods of research.

Notes I.

2.

3.

4. 5.

6.

7.

Monologe (fUr zwei Klaviere), Mainz: Schott 1964; Musique pour les soupers du Roi Ubu. Ballet noir en sept parties et une entree, Kassel and Basel: B:1renreiter 1966. Recorded versions of both works will be found in the anthology Zeitgeniissisc!le Musik in der Bundesrepublik Deutschland, Vol 5, edited by the Deutscher Musikrat, Deutsche Harmonia Mundi (EMI) DMR 1013-15 (1983). Most of the relevant material (including papers and sketches previously unpublished) is contained in Brentano (1976). Some of Brentano's investigations pertaining to perception, including musical issues, are found in Brentano (1979). As to Brentano's epistemology, and especially with respect to his concepts of time and time perception, see Bergmann (1967, pp. 320ft). See Gurwitsch (1975, pp. 60ft). Gurwitsch was a student of Husserl and the Gestalt psychologist Wolfgang Ktlhler who discussed Husserl's (and Stumpf's) works with respect to Gestalt theory. In his book, he offers a concise introduction to the phenomenological concepts of perception and cognition. Another systematic account is of course offered by William James in his Principles of psychology (James, 1890/1981, chapters XVII [sensation] and XVIII [imagination]). This implies that, for example, a Pythagorean major third 81/64 actually played would be 'simplified' perceptually to the just major third 5/4. With respect to Riemanns Ideen as well as to Stumpf's Konkordanz, there are a number of unsolved factual and methodological problems (see Schneider, 1986, pp. 182ft). It has been argued by Sloboda (1985, p. 192) that because of a rather simple structure of the Miserere, Mozart's 'memorization ... does not involve inexplicable processes which set him apart from ordinary musicians.' There are several studies, however, which have to do with recall of (familiar or new) melodies from memory, and which basically explore related problems; see e.g. Davies (1978, ch. 5), Sloboda (1985, pp. 183ft) and Crowder (1983). In this respect as well as in others, it is not always easy to separate perception from imagery and other aspects of conceptualization and memory.

References Bergmann, G. (1967). Realism: A critique of Brentano and Meinong. Madison: University of Wisconsin Press. Block, N. (Ed.) (1981). Imagery. Cambridge, Mass. and London: The MIT Press.

24

PERSPECTIVES AND CHALLENGES OF MUSICAL IMAGERY

Boring, E. (1950). A History ofexperimental psychology (2nd ed.). Englewood Cliffs: Prentice Hall. Brentano, F. v. (1874, 1924, 1928). Psychologie vom empirischen Standpunkt, Vol. 1-3. Leipzig: F. Meiner. Brentano, F. v. (1976). Philosophische Untersuchungenzu Raum, Zeit und Kontinuum (Edited by St. Kfirner and R. Chisholm). Hamburg: Meiner. Brentano, F. v. (1979). Untersuchungen zur Sinnespsychologie (2nd ed., edited by R.M. Chisholm and R. Fabian). Hamburg: Meiner. Carrier, M. (1995). Philosophy of mind. In J. Mittelstrass (Ed.), Enzyklopadie Philosophie und Wissenschaftstheorie (Vol. 3, pp. 220-226). Stuttgart and Weimar: Metzeler. Carrier, M. and Mittelstrass, J. (1991). Mind. brain. behavior. The mind-body problem and the philosophy ofpsychology. Berlin and New York: de Gruyter. Casey, E.S. (1976). Imagining. A phenomenological study. Bloomington: Indiana University Press. Chion, M. (1983). Guide des objets sonores. Paris: Editions BuchetlChastel. Chisholm, R. (1956). Perceiving. Ithaca, N.Y.: Cornell University Press. Clarke, J.M. (1994). Neuroanatomy: brain structure and function. In D.W. Zaidel (Ed.), Neuropsychology (pp. 31-51). San Diego, London: Academic Press. Clifton, T. (1976). Music as constituted object. In FJ. Smith (Ed.), In Search of musical method (73-98). London and N. Y.: Gordon & Breach. Clifton, T., (1983). Music as heard. A Study in applied phenomenology. New Haven and London: Yale University Press. Crowder, R. G. (1989). Imagery for musical timbre. Journal of Experimental Psychology: Human perception and performance, 15, 472-478. Crowder, R. (1993). Auditory memory. In S. McAdams & E. Bigand (Eds.), Thinking in sound. The cognitive psychology ofhuman audition (pp. 113-145). Oxford: Clarendon Press. Davies, J.B. (1978). The Psychology of music. London: Hutchinson. Dennett, D.C. (1991). Consciousness explained. Boston: Little, Brown & Co. Descartes, R. (1637). Discours de la methode pour bien conduire sa raison et chercher la verite dans les sciences. Paris: M. Soly. Descartes, R. (1642). Meditationes de prima philosophia (ed. alt.). Amsterdam: L. Elzevir. Duchesneau, L. (1986). The Voice of the muse: A study of the role of in!Jpiration in musical composition. Frankfurt: P. Lang. Ehrenfels, C. v. (1988). On 'Gestalt Qualities'. In B. Smith (Ed.), Foundations of Gestalt Theory (pp. 82117). MUnchenlWien: Philosophia Verlag. Fokker, A.D. (1945). Rekenkundige Bespiegeling der muziek (A mathematical approach to music). Gorinchem: J. Noorduijn en Zoon. Ebbinghaus, H. (1885). Ober das Gedachtnis. Untersuchungen zur experimentellen Psychologie. Leipzig: J.A. Barth. Georges, K. E. (1869). Ausfiihrliches Lateinisch-deutcsches Handworterbuch, Vol. 1 (6th edition). Leipzig: Hahn'sche Verlagsbuchhandling. G6mez de la Serna, R. (1977). Dali. Madrid: Espasa & Calpe. Gurwitsch, A. (1975). Das Bewusstsein!Jfeld (edited by W. Frfihlich). Berlin and N.Y.: de Gruyter. Hubbard, T., L., & Stoeckig, K. (1988). Musical imagery: Generation of tones and chords. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14, 656-667. Hume, D. (1758/1951). Enquiry concerning human understanding (2nd ed, reprint 1951). Oxford: Clarendon Press. Husserl, E. (1901). Logische Untersuchungen. Elemente einer phanomenologischenAujklarung der Erkenntn;s, Vols. 1-3. Halle: Niemeyer. Husserl, E. (1928). Husserls Vorlesungen zur des inneren Zeitbewusstseins (edited by M. Heidegger). Jahrbuchfiir Philosophie und phanomenologische Forschung, 9,367-496. Husserl, E. (1976). Erfahrung und Urteil. Untersuchungen zur Genealogie der Logik (5th ed.). Hamburg: Meiner. Ingarden, R. (1962). Untersuchungen zur Ontologie der Kunst. TUbingen: Niemeyer Ivarson, P. and Krumhansl, C.L. (1993). Isolating the dynamic attributes of musical timbre. Journal ofthe Acoustic Society ofAmerica, 94,2595-2603. Jaensch, E. (1927). Die Eidetik und die typologische Forschungsmethode (2nd ed.). Leipzig: Quelle & Meyer. (English translation by?? (1930): Eidetic imagery and typological methods of investigation.

ALBRECHT SCHNEIDER AND ROLF INGE GOD0y

25

London: K. Paul, Trench, Trubner & Co.; Ney York: Harcourt, Brace & Co.) James, W. (1890/1981). The Principles ofpsychology, Vols. 1-3 (edited by F. Bowers and I.K. Skrupskelis). Cambridge, MAlLondon: Harvard University Press. Kant, I. (1787). Kritik der Reinen Vernunft (2nd ed.). Riga: Hartknoch. Kant, I. (1792). Kritik der Urteilskraft. Ktinigsberg and Riga: Hartknoch Klinger, E. (Ed.) (1981). Imagery [2]: Concepts, results, and applications. London, New York: Plenum Press. Knepler, G. (1991). Wolfgang Amade Mozart. Annaherungen. Berlin: Henschel-Verlag. Kochmann, R. (1923). Uber musikalische Ged:tchtnisbilder. Zeitschriftfiir angewandte Psychologie, 22, 329-351. Kosslyn, S.M. (1980). Image and mind. Cambridge, MA: Harvard University Press. Kosslyn, S.M. (1994). Image and Brain: The Resolution of the Image Debate. Cambridge, Mass. and London: The MIT Press. Kurth, E. (1947). Musikpsychologie (2nd ed.). Bern: Krompholz. Mainwaring, J. (1933). Kinaesthetic factors in the recall of musical experience. British Journal of Psychology, 23, 284-307. Marin, O.S.M. & Perry, D.W. (1999). Neurological Aspects of music perception and performance. In D. Deutsch (Ed.), The Psychology ofmusic (2nd ed., pp. 653-724). San Diego etc.: Academic Press. McDaniel, M.A. & Pressley, M. (Eds.) (1987). Imagery and related mnemonics processes. Theories, individual differences, and applications. New York, Berlin etc.: Springer. Meyer. L.B. (1956). Emotion and meaning in music. Chicago: University of Chicago Press. Mikumo, M. (1994). Motor encoding strategy for pitches of melodies. Music Perception, 12,175-197. Mikumo, M. (1998). Encoding strategies for pitch information. Japanese Psychological Monographs, No. 27. N:t:tt:inen, R., & Winkler, I. (1999). The Concept of auditory stimulus representation in cognitive neuroscience. Psychological Bulletin, 125, 826-859. Newman, E.B. (1948). Perception. In E.G. Boring, H.S. Langfeld & H.P. Weld (Eds.), Foundations of psychology (pp. 215-249). New York: Wiley (London: Chapman & Hall). Pitt, M.A. & Crowder, R. (1992). The role of spectral and dynamic cues for musical timbre. Journal of Experimental Psychology: human perception and performance, 18,728-738. Reuter, C. (1995). Der Einschwingvorgang nichtperkussiver Musikinstrumente. Frankfurt, etc.: Lang Riemann, H. (1914-16). Ideen zu einer 'Lehre von den Tonvorstellungen' . Jahrbuch Peters, 21/22 (1914/15), 1-26; 23 (1916),1-15. Rouw, R., Kosslyn. S. M., & Hamel. R. (1997). Detecting high-level and low-level properties in visual images and visual percepts. Cognition, 63, 209-226. Sacks, O. (1985). The Man who mistook his wife for a hat. New York: Summit Books. Sartre, J.P. (1940). L'lmaginaire. Psychologie phenomenologiquede l'imagination. Paris: Gallimard Schaeffer, P. (1966). Traite des objets musicaux. Paris: Editions du Seuil. Scheerer, E. (1993). Mentale Repr:tsentation in interdisziplin:trer Perspektive. Zeitschrift fiir Psychologie, 201,136-166.

Schneider, A. (1992). On Concepts of 'tonal space' and the dimensions of sound. In R. Spintge & R. Droh (Eds.), MusicMedicine (pp. 102-127). St. Louis: MMB Music. Schneider, A. (1995). Musik sehen - Musik htlren. Ober Konkurrenz und Komplementarit:it von Auge und Ohr. Hamburger Jahrbuch der Musikwissenschaft, 13, 123-150. Schneider, A. (1997a). On categorical perception of pitch and the recognition of intonation variants. In Pylkk:tnen, P. Pylkkti, A. Hautam:iki (Eds.), Brain. mind and physics (pp. 250-261). Amsterdam and Tokyo: IDS Press and Ohmsha. Schneider, A. (1997b). 'Verschmelzung', tonal fusion, and consonance: Carl Stumpf revisited. In M. Leman (Ed.), Music, Gestalt, and Computing (pp. 117-143). Berlin, New York etc.: Springer 1997. SchUtz, A. (1976). Fragments on the phenomenology of music (edited by F. Kersten). In FJ. Smith (Ed.), In Search ofmusical method (pp. 5-71). London: Gordon & Breach,. Scruton, R. (1997). The Aesthetics ofmusic. Oxford: Clarendon Press. Segal, SJ. (Ed.) (1971). Imagery: Current cognitive approaches. New York, London: Academic Press. Segal, SJ. & Fusella, V. (1970). Influence of imagined pictures and sounds on the detection of visual and auditory signals. Journal of Experimental Psychology, 83, 458-464.

26

PERSPECTIVES AND CHALLENGES OF MUSICAL IMAGERY

ShOff, lE. et al. (Eds.) (1980). Imagery ( I]: Its many dimensions and applications. New York, London: Plenum Press. Slawson, W. (1985). Sound Color. Berkeley and Los Angeles: University of California Press. Sloboda, J. (1985). The musical mind. The cognitive psychology ofmusic. Oxford: Clarendon Press. Stein, B. E. & Meredith, M. A. (1993). The Merging ofthe Senses. Cambridge, Mass.: The MIT Press. Stoffer, T. (1996). Mentale Repr:tsentation musikalischer Strukturen. Zeitschrift fiir Semiotik, 18, 213-234. Stumpf, C. (1883, 1890). Tonpsychologie, Vois. 1 & 2. Leipzig: Hirzel. Stumpf, C. (1898). Konsonanzund Dissonanz. Leipzig: J. Barth. Stumpf, C. (1907). Erscheinungen und psychische Funktionen. Abhandlungen der Kt)niglich Preussischen Akademie der Wissenschaften, Jahrgang 1906, Phil.-hist. Klasse Nr. 4. Berlin: Akademie der Wissenschaften. Stumpf, C. (1911). Konsonanzund Konkordanz. Zeitschriftfiir Psychologie, 58, 321-355. Stumpf, C. (1918). Empfindung und Vorstellung. Abhandlungen der Kt)niglich Preussischen Akademie der Wissenschaften, Jahrgang 1918, Phil.-hist. Klasse Nr. 1. Berlin: Akademie der Wissenschaften. Stumpf, C. (1926). Die Sprachlaute. Berlin: Springer Stumpf, C. (1939). Erkenntnislehre, Vois. 1 & 2. Leipzig: J. Barth. Vogel, M. (1993). On the relations oftone. Bonn: Verlag fUr Systematische Musikwiss. Weber, RJ. & Brown, S. (1986). Musical imagery. Music Perception, 3, 411-426. Wilbanks, J. (1968). Hume's theory ofimagination. The Hague: Nijhoff.

2 Neurophysiological Mechanisms Underlying Auditory Image Formation in Music Petf Janata

Introduction The formation of contextually dependent expectancies is an important feature of music cognition. Both explicit and implicit knowledge about the structure of a piece of music serve to establish highly specific expectations about the pitch, timbre, and other features of ensuing musical information. Musical expectancies represent a specific type of musical imagery. On the one hand, musical imagery might be thought of as a mental process that occurs over an extended period as a person imagines hearing or performing a piece of music. This type of imagery differs from expectancy formation in that it may transpire in the absence of sensory input. Active expectancy formation, on the other hand, generally requires that specific images for subsequent sensory input are based on preceding sensory input and established knowledge of what type of sensory input to expect. A neuroscientific way of framing the general question is, 'What are the brain areas and mechanisms that support the formation of such images and the interaction of these images with incoming sensory information?' Electrophysiological measures of brain activity provide a description of how the human brain implements these types of processes, and a variety of different experimental designs can be used to address various components of these processes. Over the past 30 years, studies of auditory evoked potentials have provided support for models of how voluntarily maintained

28

NEUROPHYSIOLOGICAL MECHANISMS

images (expectancies) of single tones interact with sensory input consisting of multiple auditory stimuli. Stimuli based on musical considerations 1) extend the types of designs that can be used to study mechanisms of auditory image formation, 2) provide important tests of the existing models, and 3) provide a framework, rooted in the neuroethological tradition (PflUger & Menzel, 1999), for understanding the neural underpinnings of human musical behavior.

Forms of musical imagery Perhaps the first step in studying musical imagery is to place musical imagery in the broader context of auditory imagery and the general domain of mental imagery, if for no other reason than to borrow from definitions of mental imagery derived primarily from considerations of the form and formation of visual images. Finke (1989) defines mental imagery as, 'the mental invention or recreation of an experience that in at least some respects resembles the experience of actually perceiving an object or an event, either in conjunction with, or in the absence of, direct sensory stimulation.' In order to pinpoint and characterize specific neural mechanisms underlying musical imagery, it is necessary to define what a musical image is and what the processes of forming such an image or series of images are. Mirroring Finke's definition, I consider two contexts in which musical imagery occurs. In the first context, musical imagery is purely a mental act: an endogenous phenomenon in which the content of the images is internally generated from long-term memory stores of musical knowledge and is uninfluenced by any concurrent sensory input. In the second context, formation of musical images depends on an interaction of memory-dependent processes (expectancies) with representations of incoming auditory input. The relationship between perception and mental imagery has been elaborated and tested extensively with visual material by Kosslyn (1980, 1994). In Kosslyn's view (1994, p. 287), 'images are formed by the same processes that allow one to anticipate what one would see if a particular object or scene were present.' Thus, postulating two contexts for musical imagery is in keeping with other theories of mental imagery. Figure 1 on the facing page shows a theoretical framework for thinking about how imagery processes within these two different contexts might be instantiated in a set of brain structures. Because a complete theory of musical imagery should include also imagery for musical performance and the interaction of sensory and motor information, the diagram in Figure 1 is restricted, in the interest of relative simplicity, to represent processes that may be involved in 'sensory' imagery rather than 'motor' imagery. Following a brief description of the framework, I summarize the physiological methods and experiments in support of it. The arrows represent the flow of information across different general brain areas listed at the right through time. Those brain areas involved more immediately with sensory processing are listed at the bottom, while those involved in abstract reasoning and memory storage/retrieval are listed toward the top. The first type of auditory imagery unfolds over longer time periods (seconds or minutes) and is generally unconstrained by sensory input. I call it 'non-expectant' because we neither expect to hear anything as we are imagining, nor are we forming expectations of what we will

PETRJANATA

29

Forms of imagery in relation to brain structures (a sensory perspective) "Non-Expectant"

Abstract-H Eidetic_v

"Expectant"

"'/ /

Prefrontal cortex Association cortex Sensory cortex Subcortical structures

Time

Figure 1.

Schematic view of different types of auditory imagery and how these might be instantiated in the human brain (see text for details).

hear in the immediate future. This is the type of imagery we engage in when we imagine a melody in our mind. Similarly, we might mentally improvise melodies that we have never heard before but are able to compose based on knowledge, either explicit or implicit, of tonal sequences, etc. Thus, this type of imagery relies on long-term memories we have of specific musical material, or on a more abstract knowledge of musical structure, e.g. the tonal relationships in western tonal music. Non-expectant imagery may be differentiated further into two modes of imagery that I call 'abstract' and 'eidetic'. In the abstract mode, the sequence of pitches in a melody might be imagined without the sense of 'hearing' the melody being played by any particular instrument. In the eidetic mode, a strong component of the image is the impression that an instrument or group of instruments is performing the imagined piece of music, and the image has a strong sensory quality to it. To the extent that this 'non-expectant' imagery depends on retrieving information from long term memory stores, it may rely heavily on areas of prefrontal cortex which have been implicated in general memory functions (Goldman-Rakic, 1996). One prediction of the 'abstract/eidetic' distinction is that the eidetic qualities of images engage brain regions more immediately involved in processing sensory information. Increased eidetic qualities of the images are represented by increases in the depths of the arcs of the solid arrows in Figure 1. For example, a strong impression of hearing an orchestra in our minds might be indicative of auditory cortex involvement in the imagery process. These relationship between the vividness of the mental image and the brain areas that are activated by the image remains to be tested. A modicum of support for the notion that more 'true-to-life' images activate those brain areas more immediately involved in the formation of sensory representations comes from neuroimaging studies of schizophrenic patients experiencing auditory hallucinations which show that auditory cortical areas, including primary auditory cortex are activated during hallucinations (Dierks et aI., 1999; Griffiths, Jackson, Spillane, Friston, & Frackowiak, 1997).

30

NEUROPHYSIOLOGICAL MECHANISMS

'Expectant' imagery refers to the process of forming mental images when listening, attentively, to music or sounds in general. In addition to relying on long-term memory for musical structure or a specific piece of music, the mental images are additionally constrained by the interactions of contemporaneous sensory information with the memorized information. In other words, as we listen to the notes of an ascending major scale we can form a very specific image/expectation of the next note in the scale. The specificity arises in part from our knowledge of the intervalic relationships between successive notes in a major scale, as well as the exact frequencies of the notes being used to create this particular instance of the scale. If the playing of the notes in the scale were to stop, we could continue forming images of the remaining notes. Similarly, in listening to a chamber ensemble playing a familiar piece of music, our expectancies are formed from familiarity with the piece of music as well as the sensory information we are receiving about the tone of these particular instruments, or the expressive tendencies of the particular ensemble. In Figure 1, the merging of the straight arrows represents the interaction of 'top-down' expectancies with 'bottom-up' sensory input, and a subsequent influence of this interaction on the expectancy/image forming processes in a continuous, iterative, process. The extensive literature on the formation of auditory representations (reviewed in & Winkler, 1999) and the interaction of these representations with top-down influences such as selective attention (NaaUinen, 1992) implicate areas such as the secondary auditory cortex as a neural substrate for these interactions.

Methods for probing human brain activity Inferences about brain function are made by measuring changes in one or several dependent variable(s) as a subject performs specific cognitive tasks. The dependent variables can range from behavioral measures such as reaction time and accuracy to physiological measures of regional cerebral blood flow or field potentials generated by neurons. I will focus on the physiological measures, first providing a brief description of the signals being measured along with the measurement and data analysis techniques and then summarizing results of studies that are of particular relevance to musical imagery. Studies of the brain's physiological responses typically strive to 1) identify brain areas that are responsible for performing specific tasks or computations, and 2) describe the neural mechanisms by which stimuli are represented and cognitive tasks are performed. Functional imaging techniques such as positron emission tomography (PET) and functional magnetic resonance imaging (fMRI) are well-suited to address the first goal. Both of these methods monitor blood flow changes in the brain. As large populations of neurons in those brain areas that perform a cognitive task become active and increase their metabolism, the blood supply to those areas increases in order to meet the increased demand for oxygen (Villringer, 1999). Note that neuronal activity is modulated more quickly than is the associated blood flow, resulting in a temporal resolution limit (peak response is 2-4 seconds from onset of event) of PET and fMRI. While PET and fMRI provide only an indirect measure of neural activity, they do so with much better spatial resolution than do direct, non-invasive measurements

PETRJANATA

31

of neural activity. Thus, these methods are invaluable tools for localizing cognitive functions, and their application to issues of auditory imagery is described below. The temporal properties of neural responses in cognitive tasks are best captured by direct measures of the neural activity. The electrical fields generated by large populations of neurons comprise the electroencephalogram (EEG), and the magnetic counterpart to the electrical fields forms the magnetoencephalogram (MEG). Given the distance ofEEG recording electrodes positioned at the scalp from the cortical surface, large populations of similarly oriented neurons must be active synchronously in order for them to create a sufficiently strong electrical field that can be detected at the scalp surface (Nunez, 1981). The superposition of many electrical fields from many neuronal populations, along with the low-pass filtering characteristics of the skull, makes the problem of unambiguously localizing the neural sources based on EEG data a difficult one. The nature of the experimental situation generally dictates how the EEG is analyzed. When the experiment consists of short, discrete, and clearly defined stimuli, such as single tones embedded in longer sequences of tones, the stimuli are presented many (30-1000) times while the EEG is recorded. The EEG responses to each presentation are then averaged in order to extract the mean waveform. This waveform is interpreted as the unique response to the particular type of stimulus, and is generally referred to as an event-related potential (ERP) waveform. The ERP waveform is analyzed in terms of the amplitudes, latencies, and areas of the peaks and troughs. The nomenclature reflects the polarity and typical latency of the deflection in the ERP waveform. For example, the auditory NI00 is a negative peak (as measured from an electrode at the vertex of the head relative to a mastoid, ear, noise or non-cephalic reference electrode) which occurs approximately 100 ms following the onset of the stimulus. Features of the ERP waveform, such as the NI00 or P300, are commonly called 'components' to indicate their dependence on the perceptual/cognitive factors that modulate their presence, size, and latency. Individual ERP components, particularly those occurring in the hundreds of milliseconds do not necessarily reflect unitary cognitive phenomena. For example, the amplitude of the Nl00 is modulated by both physical features of the stimulus as well as the attentional state of the subject. The NI00 represents a conglomerate of brain processes associated with processing an auditory stimulus (NaaUinen & Picton, 1987), and it may overlap with other ERP 1992). components such as the mismatch negativity (MMN) In tasks employing continuous stimulus situations in which individual discrete events cannot be identified, such as listening to a recorded piece of music, the EEG is typically analyzed in the frequency domain. Here, the assumption is that the performance of any given cognitive task will be associated with a sustained and stable pattern of neural activity involving some number of brain areas. The field potential arising from the neural activity pattern is then described by its frequency spectrum. Typically, successive 2s EEG epochs are converted into the frequency domain and the average power spectrum is computed. The magnitude of the power spectrum indicates the strength with which oscillations at a particular frequency are represented in the EEG during a cognitive state. Brain activity associated with any given cognitive task can be isolated by subtracting the average power spectrum during the task from the average power spectrum during rest, and the synchronization of different brain regions

32

NEUROPHYSIOLOGICAL MECHANISMS

can be assessed through the frequency-specific coherence between pairs of electrodes (Rappelsberger & Petsche, 1988; Srinivasan, Nunez, & Silberstein, 1998). Overall, the spectral analysis of the EEG tends to provide information about global brain dynamics associated with a particular cognitive task, whereas ERPs are used to elucidate the sequence of processing steps associated with discrete stimulus events.

Physiological measures of mental images 'Non-expectant imagery' What is the manifestation of different types of auditory/musical imagery in the brain, and what is the evidence supporting the functional architecture described above? To the extent that 'non-expectant' imagery establishes a set of stationary processes in the brain, i.e. a stable pattern of activity within specific neural circuits over the duration that subjects perform an imagery task, it should be possible to capture signatures of these processes in EEG recordings. Petsche and colleagues have found wide-spread coordination, as manifested in the coherence of the EEG, of brain areas as subjects imagine hearing or composing musical material (Petsche, Richter, von Stein, Etlinger, & Filz, 1993) or mentally play an instrument (Petsche, von Stein, & Filz, 1996). Imagining a piece of music leads to an increase in the number of observed coherence changes compared to listening to the same piece of music. The exact patterns of changes differ appreciably among subjects, however. For example, in one subject imagery is associated with theta and alpha band decreases and beta band increases, whereas in another subject there are coherence increases across all frequency bands. So far, the best evidence for those brain areas involved in auditory imagery in the absence of acoustic input come from PET studies by Zatorre and colleagues (Halpern & Zatorre, 1999; Zatorre, Evans, & Meyer, 1994; Zatorre, Halpern, Perry, Meyer, & Evans, 1996). In their tasks, mentally scanning through a melody results in superior temporal gyrus (STG) and right frontal lobe activations. The auditory cortex lies along the STG. The frontal lobes are widely implicated in memory retrieval processes (Goldman-Rakic 1996). The observation that the auditory cortex is activated during these tasks is extremely important because it indicates that those structures responsible for the processing of auditory stimuli are also activated under certain conditions of musical/auditory imagery when no sensory stimuli are present. Indirectly, these results suggest that discrete, mentally generated, auditory images might be compared against incoming sensory information in the auditory cortex. Formation of specific musical expectancies Although spectral analysis of the EEG is typically applied to sustained tasks rather than to event-related tasks, it has been applied to analyzing the build up and resolution of harmonic expectancies (Janata & Petsche, 1993), implicating right frontal and temporal areas in the processing of cadences and their resolutions. ERPs analyzed as time-domain averages have been used by several researchers to probe musical expectancy. Although the results of the studies differ slightly, the general finding is that unexpected notes and chords elicit larger positive potentials from 300 to 600 ms following the onset of the stimulus than do highly expected notes and chords (Besson

PETRJANATA

8 6

33

Electrode C4

"

P200

4 ,--..

>::t

2

'-'

0

bJ)

Cd

0 -2

-4 -6 -8

- - Tonic ........... Minor - - Dissonant

NI00

/

1000

2000

3000

4000

5000

Time (ms) Figure 2.

Averaged event related potentials (ERPs) recorded during a hannonic priming task in which subjects heard a I, IV, V cadence (at 0 ms, 1000 ms, 2000 ms, respectively), imagined the best possible resolution (3000-4000 ms) and heard one of three possible resolutions at 4000 ms, whereupon they had to decide if it was the resolution they had imagined. The waveform components labeled P3a and P3b typically vary in amplitude as a function of expectancy. Adapted from Janata, 1995.

& Fai"ta, 1995; Besson & Macar, 1987; Hantz, Kreilick, Kananen, & Swartz, 1997; Janata, 1995; see also Janata & Petsche, 1993 for a frequency domain analysis of event-related EEG data; Patel, Gibson, Ratner, Besson, & Holcomb, 1998). The general names in the ERP literature for the large, late positive waves are 'P300' and late positive complex (LPC), and their amplitude is inversely proportional to the subjective probability of the eliciting event (for a review, see Donchin & Coles, 1988; Verleger, 1988). More recently, researchers have focussed on contributions of frontal brain areas to the processing of harmonic expectancies, showing negative shifts in the waveforms for harmonically deviant chords, compared to contextually consonant chords (Koelsch, Gunter, Friederici, & Schrager, 2000; Patel et aI., 1998). Figure 2 shows ERP waveforms in response to chords in a priming (I, IV, V) cadence and three resolutions of the cadence. In this experiment (Janata, 1995), chords in the priming cadences were presented in numerous inversions and several different keys, using a sampled grand piano sound. Each chord was presented for 1 s. For Is between the offset of the final chord and the onset of the resolution (3000-4000s), subjects imagined the best possible resolution. At 4000 ms, one of three possible resolutions was heard: the expected resolution to the I (thick solid line), a harmonically plausible resolution to the tonic of the relative minor (dashed line), or a harmonically implausible resolution to a triad based on the tritone (thin solid line). The large negative (Nl00) and positive peaks (P200) characterize the auditory evoked potential. The

34

NEUROPHYSIOLOGICAL MECHANISMS

response to the harmonically incongruous ending elicited the largest amplitudes in the two P300 components. The brain circuitry that gives rise to these late potentials is not well understood. Intracerebral recordings indicate multiple sites that exhibit P300-like activations (Halgren, Marinkovic, & Chauvel, 1998). Thus, the details of how activity in auditory cortical regions is coordinated with activity in other brain areas necessarily remain murky. Interestingly, Janata (1995) observed a large waveform in the period when subjects were asked to imagine the best possible resolution, suggesting that this was a measurable brain response to the act of imagining the resolution to the tonic. Unfortunately, the evoked potential may have been caused by the offset of the previous chord, rather than voluntary image formation. Further studies have been performed to investigate evoked potentials elicited by imagined events, and some preliminary results are presented below. Measures of expectancies in the absence of sensory input In ERP studies of auditory expectancy, brain potentials resulting from the expectancy forming process and potentials arising in response to sensory stimulation are combined. Although it is possible to measure the outcome of the interaction of the expectancy (mental image) with the sensory information, determining the electrophysiological signature of each information stream poses a greater challenge. One way of studying the expectation is to simply omit an expected stimulus, thereby removing all sensory components from the ERP response (Besson & Fai"ta, 1995; Besson, Fai"ta, Czternasty, & Kutas, 1997; Ruchkin, Sutton, & Tueting, 1975; Simson, Vaughan, & Ritter, 1976; Sutton, Tueting, Zubin, & John, 1967; Weinberg, Walter, & Crow, 1970). Such omissions generate a large P300, typical of unexpected stimuli. In some cases, earlier components, reminiscent of the auditory evoked potential are also present (Besson et aI., 1997). The relationship between potentials generated in response to unexpected stimulus omissions and voluntarily generated images has not been explored in more detail, however. These earlier studies raise several questions about the neural processes involved in forming auditory expectancies and the interactions of these expectancies with sensory input. Specifically, can the mental processes involved in mental image formation be dissociated further from the process of expectation? In other words, is it possible to measure emitted potentials associated with the formation of a mental image in the complete absence of an expectation that a stimulus will occur? If the answer is yes, is there any evidence that such emitted potentials arise from the auditory cortex where expectancies and sensory input are believed to interact? To begin investigating these questions, I performed a study in which musically trained subjects were asked to first listen to and then imagine completions of simple eight-note melodic phrases (Janata, in press). Examples of the melodic fragments and the various experimental conditions are schematized in Figure 3 on the facing page. On each 'imagery' trial (Figure 3B), subjects first heard all eight notes of the melody, and made a key-press synchronously with the last note ('All Heard' condition). Next, they heard the first five notes of the same melody and continued imagining the remaining three, pressing a key at the time they thought the last note would have occurred ('3 Imagined' condition). They then heard the first three notes of the melody, imag-

35

PETRJANATA

A

.J = 120

-

Melody 1

Melody 2

B

Imagery trials

J

J-. J-. J-. J-. J-. J-. J-. J-.

All Heard (AH) 3 Imagined (31)

J-. J-. J-. X X J-. J-. J-. J-. J-.

5 Imagined (51) No Imagery (NI)

heard note

X Imagined note

X

X

X 7 k e y press

X

X

X

I msec

Figure 3.

o

}000

2000

3000

4000

Melodies and experimental conditions used in the auditory imagery experiments. A) The two simple melodies in musical notation. B) Schematic diagram of imagery trials. In the 'All Heard' condition subjects heard the entire melodic fragment. In the '3 Imagined condition,' the initial five notes were heard and the remaining three imagined. In the '5 Imagined' condition, subjects heard the initial three notes and imagined the remaining five. On each trial, these three conditions appeared in immediate succession. In separate blocks of 'No Imagery' trials, subjects heard the initial five notes but did not imagine the remaining three.

ined the remaining five, once again making a key-press synchronously with the last imagined note ('5 Imagined' condition). In a separate control block of 'no-imagery' trials, subjects heard the first five notes of the melody but did not continue imagining the remaining notes and made no key presses ('No Imagery' condition). They were explicitly instructed to not imagine a continuation of the melody and to try to hear the five notes as a complete phrase. In order to achieve an appropriate signal to noise ratio in the ERP waveform, subjects performed 100 imagery trials and 100 no-imagery trials. Subjects' brain electrical activity was recorded throughout each trial using a geodesic array of 129 electrodes (Electrical Geodesics Inc., Eugene, OR) distributed across the scalp. Such dense sampling of the electrical potential at the scalp surface allows one to construct an accurate topographical map of the voltage at each sampled time point. Figure 4A (see page 36) illustrates a typical auditory N100 topography in which

36

NEUROPHYSIOLOGICAL MECHANISMS

NlOO

6th Heard 0.9

0.9

1.2

0.3

0.3

0.4

·0.3

-0.3

-0.4

·0.9

- 0.9

- 1.2

-1.5

- 1.5 376 - 480 ms

Ist of 3 Imagined

2nd Imagined 2.0

1.0

0.9

1.2

0.6

0.3

0.4

0.2

-0.3

·0.4

-0.2

-0.9

- 1.2

-0.6

- 1.5

- 2.0

96-192ms

-i

704 - 808 ms

P300

1.5

"NlOO"

2.0

1.5

96-192ms

"N 100"

7th Heard

1.5

- 1.0

376 - 480 ms

Ist of 5 Imagined 1.5

P300

704 - 808 ms

2nd Imagined 2.0

1.0

0.9

1.2

0.6

0.3

0.4

0.2

-0.3

- 0.4

-0.2

-0.9

- 1.2

-0.6

- 1.5

- 2.0

96·192ms

- 1.0 704 - 808 ms

376 - 480 ms

2.0

1.5

1.0

0.9

1.2

0.6

0.3

0.4

0.2 ';;' -02 •

·0.4

-0.6

- 1.2 - 2.0 96-192ms

Figure 4.

376 - 480 ms

...J

L-

>:s. Cl.

e

<

- 1.0

704· 808 ms

Summary topographical maps of the activation elicited by different tasks (each of the four rows) in the auditory imagery experiment. Each circle represents a view down onto the top of the head. The nose would be at the top of the circle. Plotted are the average voltage values in time windows that encompass the N I00 (left column). the P300 (center column). and the N loolP2oo for the next note. Maps with negative values at centro-frontal sites and positive values around the perimeter (A. D. G) are typical of the auditory N 100 response. The large parietal positivities in the imagery conditions (E. H) are characteristic of a P300 response. Each map is the activation averaged across seven subjects and 60-90 trials/subject.

PETRJANATA

37

there is a large negative focus across centro-frontal electrode sites on top of the head, and a ring of positive voltage at electrodes at more inferior positions of the scalp around the perimeter. Within the same time window, imagining the first of a sequence of notes elicited a topographical pattern in the brain electrical activity that resembled the topographical pattern elicited by the corresponding heard note in the original phrase (Figure 4D, G). No such pattern was elicited in the condition in which subjects were asked to abstain from imagining the continuation of the phrase (Figure 4J). In the imagery conditions, another stable topographical pattern was assumed from 375480 ms after the time at which the first note was to be imagined (Figure 4E, H). The positive peak above centro-parietal electrodes is characteristic of the P300 component mentioned earlier. Because subjects were expecting the sounds to cease, the presence of the P300 does not indicate an expectancy violation response to an unexpected cessation of input. This interpretation is further supported by the absence of a P300 response in the no-imagery condition (Figure 4K). The presence of the P300 is associated specifically with the imagery task, though it is difficult to assign a further functional role at this time. Given the relatively recent advent of dense-EEG methods, statistical techniques for quantitative assessment of the similarity or dissimilarity of different topographies have not been well established. Nonetheless, one way to compare topographical maps is to calculate the correlation between them. The similarity of the topographical states across the different conditions was assessed by correlating the topographies in the 3 Imagined condition with corresponding topographies in the other conditions. The temporal evolution of correlations among voltage topographies elicited in the four experimental conditions is depicted in Figure 5 on the next page. Average topographies were computed for successive 100 ms epochs. A 100 ms window size reduces the amount of data as much as possible while preserving the most prominent and stable topographical distributions. In order to compare the 3 Imagined and 5 Imagined conditions, the topographies from the two conditions had to be aligned with respect to the onsets of the first imagined events. Figure 5A shows the correlations of topographies among the conditions during the first two notes of the melodies. During these epochs, the acoustical parameters were identical across the conditions. Although the degree of correlation between the 3 Imagined condition and the other conditions varied from time-window to time-window, as expected, there were no differences among any of the conditions. Figure 5B shows the correlations when the tasks and acoustic stimulation diverged. The two imagery conditions were most highly correlated, whereas the topographies in the 3 Imagined and no-imagery conditions were uncorrelated. Note that the acoustic input in the latter pair of conditions was identical. The correlation between the 3 Imagined and All Heard conditions assumed intermediate values, particularly during the first portion of the first imagined note epoch. As a point of reference, the topographies shown in the leftmost column of Figure 4 correspond to the second 100 ms window in Figure 5B. While the instruction to imagine the continuation of the melody resulted in a clear emitted potential in response to the first imagined note, and the topographical pattern was similar to the N 100 elicited by the corresponding heard note, the same pattern was not observed for the subsequent imagined notes. Rather, the topographical activation pattern shifted to a frontal-positive peak around the time that the second note was to

38

NEUROPHYSIOLOGICAL MECHANISMS

1st heard

A

2nd heard

5 0.8

;g

0.6

]

(,J

0.4

§

0.2

-g

0

o

3 imagined vs 5 imagined 3 imagined vs all heard 3 imagined vs no imagined

U-O.2 - 0.4 ' - - - J . -_ _...L-_ _. L . - _ - - - - I ._ _--..J.. 5

-..I-_ _

6

100 msec window#

1st imagined

B

5 0.8

;g

_ _.J...__ _.L..___

9

___J____'

10

2nd imagined

••

• p