THE MINIMALIST GRAMMAR OF ACTION

Language and action have been found to share a common neural basis and in particular a common ‘syntax’, an analogous hierarchical and compositional organization. While grammatical formalisms and associated discriminative or generative computational models exist for language structure, such formalisms and models for the structure of action are still elusive. However, structuring action has important implications on action learning and generalization, in both human cognition research and computation. We have developed a minimalist grammar of action which is corroborated by neurobiological evidence. This is a formal specification of the structure of action, which employs the Chomskyan generative transformation grammar paradigm and is corroborated by neurobiological evidence. Though there are a variety of grammars for describing the structure of language, we chose the Chomskyan approach and its latest evolution into the Minimalist Program. The main reason was the fact that this framework is the culmination of an attempt to describe and explain language syntax in terms of principles and parameters that are not tied to the idiosyncrasies of the human language system, but instead may have counterparts in other biological systems. Thus, this perspective allows one to look for universals not only within the structures of different human languages, but also across natural language to non-symbolic sensorimotor spaces, such as human action.
Our generative action grammar comprises a set of terminals, features, non-terminals, and production rules in the sensorimotor domain. The need for filling in the values of the action features, drives the merging of action constituents into binary structures organised hierarchically into temporal sequences of actions of increasing complexity. These driving features are the tool of an action, the affected object and its goal. A minimal set of production rules (in which the tool and affected object complements of an action have a primary role) apply recursively in generating the denoted action tree(s), while a parser builds such trees bottom up dealing with both tail and true recursion (i.e. discontinuous action structures and long dependencies). Recursion, merge and move, are shown to be mechanisms that manifest themselves not only in human language, but in human action too.
SEVEN YEARS OF POETICON RESEARCH
A core part of CSRI research and development has been formulated and enabled through the POETICON project series. POETICON is an interdisciplinary European funded project in the field of Cognitive Systems and Robotics. It started in January 2008, continued as POETICON++ in January 2012 and is currently running its seventh year of focused research. POETICON started with an ambitious basic research objective: to explore and model the “poetics of everyday life”, i.e. the synthesis of sensorymotor representations and natural language in everyday human interaction. This is related to an old problem in Artificial Intelligence on how meaning emerges, which was approached from an embodied and enactive cognition perspective. Basic tools for language, vision and action parsing, were developed and the modeling of their integration dynamics was explored. Experimental research fed the technology development, while a humanoid platform was used for a proof of concept demonstration of what could be achieved in Cognitive Systems through real integration of individual cognitive modules.
POETICON++ builds on these preliminary results arguing that robots need natural language for controlled generalisation of learned behaviours and for creativity. Its main objective is the development of an innovative computational mechanism for robust generalisation of motor programs and visual experiences for robots that will utilise the hierarchical and generative nature of language for ‘indexing’ (labelling) sensorymotor experiences at different levels of abstraction. The mechanism will integrate natural language and visual action and object recognition tools with advanced manipulation and mobility skills, affordance-based self-exploration abilities and a bio-inspired action-language learning module for: (a) behaviour generation through verbal instruction, and (b) visual scene understanding by a humanoid.
The POETICON project series is coordinated by the CSRI Director and brings together an international team of PIs and their teams, including Prof. Yiannis Aloimonos (University of Maryland, USA), Prof. Giulio Sandini, Prof. Luciano Fadiga and Prof. Giorgio Metta (Italian Institute of Technology, Italy), Prof. Angelo Cangelosi (University of Plymouth, U.K.) and Prof. Jose Santos Victor (Istituto Superior Tecnico, Portugal).