Scientific Achievements

Language and action have been found to share a common neural basis and in particular a common ‘syntax’, an analogous hierarchical and compositional organization. While grammatical formalisms and associated discriminative or generative computational models exist for language structure, such formalisms and models for the structure of action are still elusive. However, structuring action has important implications on action learning and generalization, in both human cognition research and computation. We have developed a minimalist grammar of action which is corroborated by neurobiological evidence. This is a formal specification of the structure of action, which employs the  Chomskyan generative transformation grammar paradigm and is corroborated by neurobiological evidence. Though there are a variety of grammars for describing the structure of language, we chose the Chomskyan approach and its latest evolution into the Minimalist Program. The main reason was the fact that this framework is the culmination of an attempt to describe and explain language syntax in terms of principles and parameters that are not tied to the idiosyncrasies of the human language system, but instead may have counterparts in other biological systems. Thus, this perspective allows one to look for universals not only within the structures of different human languages, but also across natural language to non-symbolic sensorimotor spaces, such as human action. 

Our generative action grammar comprises a set of terminals, features, non-terminals, and production rules in the sensorimotor domain.  The need for filling in the values of the action features, drives the merging of action constituents into binary structures organised hierarchically into temporal sequences of actions of increasing complexity. These driving features are the tool of an action, the affected object and its goal. A minimal set of production rules (in which the tool and affected object complements of an action have a primary role) apply recursively in generating the denoted action tree(s), while a parser builds such trees bottom up dealing with both tail and true recursion (i.e. discontinuous action structures and long dependencies). Recursion, merge and move, are shown to be mechanisms that manifest themselves not only in human language, but in human action too.

The need of structured knowledge for intelligent systems across domains and applications has led to the development of a number of knowledge bases and ontologies in the form of relational databases and semantic association networks. Neuroscience findings and theories point to a multisensory and distributed semantic memory in the human brain; however, from a cognitive perspective, our common sense knowledge bases and ontologies remain still static stores of mostly lexical concepts, with ad-hoc created semantic associations. From an engineering perspective, large-scale generalisation and common sense reasoning in intelligent systems and robotics still stumbles on the semantic gap between high level symbolic representations and low level visuomotor experiences. We have developed the PRAXICON, a dynamic, recursive and referential semantic network with biological basis that aims to address the representation and integration challenges of embodied and enactive cognition. 

Concepts in the PRAXICON may have a concrete (physical) or abstract reference pertaining to entities, movements and features with no domain constraints. They have multiple representations (symbolic, perceptual, motoric). All of them are important, but some of them are more important than others for generalisation and reasoning. Associations between concepts in the PRAXICON are simple or recursive in nature; they pertain to a finite set of pragmatic relation types that provide constraints on the production of new associations, without restricting such production though. The PRAXICON draws a clear line between language and semantic memory; it considers language an additional modality that contributes (significantly and uniquely) to the acquisition and use of generalised knowledge through dynamic interaction with perception and action, rather than an efficient representation means of such knowledge. We argue that the PRAXICON is necessary for intelligent systems, so that they move beyond one shot learning to large scale generalisation and common sense reasoning.