Interdisciplinary Workshop on Movement Grammars: Brains, Robots and Dance

Description and aim

Below you will find a summary of the description and aim of the workshop. For the full proposal, please scroll down.

Description and aim summary

Humans convey meaning not only through speech or written language, but also through bodily movement. Body language and non-verbal communication are rich sources of meaning and can account for the majority of information transmitted during interpersonal interactions. Humans also have the natural ability to receive such information, to interpret and apply it effortlessly in everyday interactions. Yet, we know very little about how this is achieved. 

The primary question we deal with is on how movement contains and expresses meaning, and what we are sensitised to when we look for meaning in movement. A systematization of how different characteristics of movement contribute to meaning would be extremely useful for neuroscience and the development of robotics and new social/interactive technologies. Our hypothesis is that such systematization can be approached as form of grammar. This hypothesis emerges from the neuroscientific literature (Rizzolatti and Arbib 1998), but is also present in robotics (Nishimoto & Tani 2009) and in dance science. This workshop brings together experts in neuroscience, cognitive science, robotics, linguistics, and dance in order to advance an interdisciplinary framework for dealing with meaning in movement through the development of a grammatical system.

The workshop will include practical dance sessions in which scientists and technologists will be given insight, through watching and producing movement in the studio, on the principles that govern meaning in movement, including the dynamic aspects of phrasing, specific spatial and embodied elements, and the harmonics of the different components. This experiential knowledge in the dance context will be the basis for the development of a grammar system for movement, aimed to be a valid tool both in neuroscience as in the development of new technologies (its range of application is everyday movement not just dance). Thus we will, as we go along, identify different ways the resulting grammar may contribute to and be validated in experimental work in these fields.

The aims of the workshop are:

1.       To bring together scientists, technologists and dance experts to formulate an interdisciplinary approach to understanding (extracting/interpreting) meaning in movement, based on the development of a movement grammar structure.

2.       To give scientists practical insights on movement understanding through producing and watching dance and movement in the studio in a systematic way – a rare opportunity in the academic context.

3.       To identify potential experimental set-ups in neuroscience and novel technologies that would validate the approach and exploit the proposed grammar.

Full Proposal

1. Scientific case

The idea behind this workshop is that there are fundamental similarities between the way we process grammatical information, as in language, and the way we perceive movement. This could mean many things, and the first goal of the workshop is to refine this hypothesis, taking into account a number of versions that appear in different fields: neuroscience, robotics and dance.

In neuroscience Rizzolatti and Arbib (1998) have shown that some of the neural structures in charge of action recognition form the basis for communication. "A plausible hypothesis is that the transition from the australopithecines to the first forms of ‘Homo’ coincided with the transition from a mirror system, enlarged, but used only for action recognition, to a human-like mirror system used for intentional communication". They argue language capacities have evolved from the ability to understand movement in others. Another evidence in support of the hypothesis is that the centres for language do not evolve from early forms of voice communication in animals, which are connected more with emotional centres than with semantic abilities. (Rizzolatti & Arbib 1998).

Other researchers accept that the faculty of language has a sensory-motor component (Hauser et al 2002). There is a large body of psychological and neuroimaging experiments that have interpreted their findings in favour of a functional equivalence between action generation, action simulation, action verbalization, and perception of action (cf. Grezes & Decety 2001). Iacoboni and colleagues (1999) have argued that motor imitation may underlie aspects of language acquisition, and Binkofski et al (2000) have argued that Broca’s region subserves imagery of motion. Broca’s area retains function that is not directly related to language processing, but that neurons here have response characteristics that may give rise to imitation of complex motor behaviours including language.

The idea that networks used for action recognition could be exploited in compositional semantics is also present in robotics. Jun Tani has investigated hierarchical structures for actions and motor imagery (Nishimoto & Tani 2009), showing Multiple Timescales Recurrent Neural Network can realize predictions of sensory streams and abstract compositional information (Yamashita & Tani, 2008). Robots are guided by hand through sequences of movements, such as grasping a ball and lifting it. The neural network is trained to make predictions of proprioception and vision senses as the robot goes through a series of predetermined movements. After training, the neural network can, not only replicate the movements, but also “learn to extract compositional semantic rules with generalization in the higher cognitive level” (Park and Tani 2015). MTRNN can also “acquire the capabilities of recognizing and generating sentences by self-organizing a hierarchical linguistic structure” (Hinoshita et al 2010).

These experiments show that continuous recurrent neural networks can support “the compositionality that enables combinatorial manipulations of images, thoughts and actions” (Nishimoto & Tani 2009), grounding compositionality naturally on sensory-motor interaction. This touches on the classic AI problem of symbol grounding, which has seen many attempts to ground language in action and perception (see Cangelosi 2010). An influential work is the Perceptual Symbol Systems hypothesis (Barsalou, 1999), that claims  perceptual experience captures bottom-up patterns of activation in sensorimotor areas, through association in the brain processing sensory multi-modal information. 

Dance is deeply concerned with movement production and compositional meaning, and what sort of experiences watching another person move can produce in the spectator. The idea that dance is related to language has been present in dance for a long time. Plutarch (46–120 AD) called dancing “mute poetry”, and poetry “speaking dancing” (cf. Hagendoorn 2010). Researchers in dance science have claimed that dance draws on the same cognitive infrastructure as the capacity for language (Hagendoorn 2010). Earlier, Collinwood had claimed that every kind of language is  “a specialised form of bodily gesture, in this sense it may be said that the dance is the mother of all languages” (Collingwood, 1958, p.244). Orgs, Caspersen and Haggard (2015) distinguish between “the processing of syntactic information of postures, movements and movement sequences on the one hand, and processing of semantics of movement intentions on the other hand”. Successful message-passing between performer and spectator (Orgs, Caspersen and Haggard 2015) provides cues about what they are thinking, feeling, sensing or intuiting (the four mental factors in Laban’s theory). For Schubert & Stevens “The idea of ‘engagement’ allows that a dance might be termed ‘successful’ on the basis of its ability to create a clear embodiment in movement of a choreographer’s intentions and its development of a clear choreographic structure or syntax, rather than on the basis of the aesthetic or genre preferences of the viewer”.

Taking into account the different version, a working hypothesis in interdisciplinary terms will be formulated and refined during the workshop, and a novel framework developed. The preliminary linking hypothesis is that humans perceive movement as meaningful, engaging neural structures shared by other forms of communication and compositional meaning, and these result in principles (laws, regularities, structures) concerning movement that prescribe how meaning in movement is grasped by a watcher, which could be investigated and formalised into some form of movement grammar.

2 Constructing the basic pillars of a dance-based movement grammar

During the workshop we will take the first steps towards the development of a movement grammar. By investigating the way we observe meaning in movement systematically in the dance context, we aim to establish general principles that can be applied to everyday movement as well. These principles do not only refer to sensitivity to shapes and gestures on a gross scale but more importantly they involve a recognition of a high degree of sensitivity to the way that shape and gesture and other macro-scale movement emerges from subtle changes in the tonal disposition of the mover. They also refer to the way that meaning is recognised according to the complex composition of multiple aspects.

The question is what are we watching for, or what we are sensitised to, when we look for meaning in movement. Dance exploits this natural ability to read meaning in /make sense of movement, and composes “non-prosaic” versions. Three studio sessions, intended to raise theoretical and practical issues, will take place in the mornings and will be led by Steve Batts, artistic director of the Echo Echo Dance Theatre Company. Participants will be introduced to core themes related to spatial and temporal aspects of human movement in a movement research environment (see appendix for further detail).

As a preliminary structure, we conceive three points that a movement grammar should address: time, space and complexity. First, the very notion of grammar suggest that movement may be divided or segmented in some way into meaningful units. What constitutes a primitive in a movement grammar is therefore an essential question. In the first dance session we will experience in practice how phrases of movement are structured, how they contain directive and adaptive elements and the dynamics of phrasing as framed by equilibrium and disequilibrium processes. The second day we distinguish different spatial components that contribute to meaning in movement, and that could in principle be analysed independently. These include the structure of human movement through vertical space and the structure of proximity.

The third day we deal with the human ability to process, interrogate and make sense of  complex layers of information involving multiple spatial and temporal aspects.  We will focus on the ability that people have to both propose and “read” complex movement information which contains dissonance and consonance. It will include reference to addressing levels, spatial disposition, the centrally important role of the torso and trunk and the integration of information over time.

The three sessions will allow participants to reflect and experiment with the guidelines given – they are conceived as labs rather than as a class. This experience in the studio will be further elaborated in the Lorentz centre where we will work towards constructing an interdisciplinary movement grammar. We will divide in small trans-disciplinary teams that will work out the different challenges during the week. In summary, we will investigate how to formalise temporal and spatial elements of the proposed grammar, and how to structure the grammar so that the relationship between the different elements can be analysed systematically.

3. Identifying Opportunities for experimental research

A constant concern throughout the workshop will be relevance of the developed grammar as a useful tool for neuroscience and robotics. We aim for a usable degree of formalisation, and for ideas on how the grammar could be validated in neuroscientific experiments and applied in artificial systems. We will achieve this through identifying opportunities for experimental research in both areas.

Neuroscience deals with the scientific understanding of the biological basis of movement understanding – as well as movement production, and the intimate relationship between production and recognition. Having a grammar means we can make predictions regarding the processes of watching movement and the neural structures that may be active, predictions that could be validated or falsified. We will begin to formulate a set of experiments that would be to the test the proposed framework.

For intelligent systems technologies and robotics the capacity to understand to some extent and respond adaptively to human movement is a necessary frontier for the development of future and emerging technologies. Robots designed to interact, assist or collaborate with humans will require at least some understanding of the significance of people’s movements (Dautenhahn 2007). We will identify how a movement grammar would impinge on research in the automated recognition of expressive movement, as well as production of movement in robotics generally.

References

·         Arbib, M.A. (2005) ‘From monkey-like action recognition to human language: an evolutionary framework for neurolinguistics’, Behavioral and Brain Sciences, Vol. 28, pp.105–167.

·         Arbib, M.A. (2016) Toward the Language-Ready Brain: Biological Evolution and Primate Comparisons, Contribution to a special issue on "Language Evolution" (Tecumseh Fitch, Ed.), Psychonomic Bulletin and Review, doi.10.3758/s13423-016-1098-2

·         Argyle, M. (2013). Bodily communication. Routledge.

·         Batts, S. (2015) Poetic Movement. Talk delivered at the  Festival Symposium on Somatics and Composition. Echo Echo Festival of Dance and Movement 2015

·         Binkofski F, Amunts K, Stephan K, Posse S, Schormann T, et al. 2000. Broca’s region subserves imagery of motion: a combined cytoarchitectonic and fMRI study. Hum. Brain Mapp. 11:273–85

·         Cangelosi A. (2010). Grounding language in action and perception: From cognitive agents to humanoid robots. Physics of Life Reviews, 7(2), 139-151

·         Cross, E. S., Acquah, D., & Ramsey, R. (2014). A review and critical analysis of how cognitive neuroscientific investigations using dance can contribute to sport psychology. International Review of Sport and Exercise Psychology, 7(1), 42-71.

·         Dautenhahn, K. (2007). Socially intelligent robots: dimensions of human–robot interaction. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 362(1480), 679-704.

·         Eddy, M. (2009). A brief history of somatic practices and dance: Historical development of the field of somatic education and its relationship to dance. Journal of Dance & Somatic Practices, 1(1), 5-27.

·         Ferreira, M. P., De Mello, D. C., & Duarte, J. P. (2011). The grammar of movement: A step towards a corporeal architecture. Nexus Network Journal,13(1), 131-149.

·         Grezes, J., & Decety, J. (2001). Functional anatomy of execution, mental simulation, observation, and verb generation of actions: a meta‐analysis. Human brain mapping, 12(1), 1-19.

·         Hagendoorn, I. (2010). Dance, language and the brain. International Journal of Arts and Technology, 3(2-3), 221-234.

·         Hauser, M. D., Chomsky, N., & Fitch, W. T. (2002). The faculty of language: What is it, who has it, and how did it evolve?. science, 298(5598), 1569-1579.

·         Hinoshita, W., Arie, H., Tani, J., Ogata, T., & Okuno, H. G. (2010). Recognition and generation of sentences through self-organizing linguistic hierarchy using MTRNN. In Trends in Applied Intelligent Systems (pp. 42-51). Springer Berlin Heidelberg.

·         Iacoboni M, Woods R, Brass M, Bekkering H, Mazziotta J, Rizzolatte G. 1999. Cortical mechanisms of human imitations. Science 286:2526–28

·         Miles, L. K., Nind, L. K., & Macrae, C. N. (2010). Moving through time.Psychological Science, 21(2), 222-223.

·         Nishimoto, R., & Tani, J. (2009). Development of hierarchical structures for actions and motor imagery: a constructivist view from synthetic neuro-robotics study. Psychological Research PRPF, 73(4), 545-558.

·         Orgs, Caspersen and Haggard (2015) A neurocognitive model of aesthetic appreciation in dance. Book Chapter to be published in: SHARED REPRESENTATIONS: SENSORIMOTOR FOUNDATIONS OF SOCIAL LIFE Sukhvinder S. Obhi & Emily S. Cross, Editors, Cambridge University Press 2016.

·         Park, G., & Tani, J. (2015). Development of compositional and contextual communicable congruence in robots by using dynamic neural network models. Neural Networks, 72, 109-122.

·          Pierce, A., & Pierce, R. (2002). Expressive movement: Posture and action in daily life, sports, and the performing arts. Da Capo Press.

·         Rizzolatti, G. and Arbib, M.A. (1998) ‘Language within our grasp’, Trends in Neurosciences, Vol. 21, pp.188–194.

·         Vincs, K., Schubert, E., & Stevens, C. (2009, January). Measuring responses to dance: is there a 'grammar' of dance?. In WDA 2008: Proceedings of the 2008 World Dance Alliance Global Summit (pp. 1-8). QUT Creative Industries and Ausdance.

·         Yamashita, Y., & Tani, J. (2008). Emergence of functional hierarchy in a multiple timescale neural network model: a humanoid robot experiment. PLoS Comput Biol, 4(11), e1000220.