Learning, Development, and Emergence of Compositionality in Natural Language Processing

There are two paradigms in language processing, as characterised by symbolic compositional and statistical distributional modelling, which may be regarded as based upon the principles of compositionality (or symbolic recursion) and of contextuality (or the distributional hypothesis), respectively. Starting with philosophy of language as in Frege and Wittgenstein, we elucidate the nature of language and language processing from interdisciplinary perspectives across different fields of science. At the same time, we shed new light on conceptual issues in language processing on the basis of recent advances in Transformer-based models such as BERT and GPT-3. We link linguistic cognition with mathematical cognition through these discussions, explicating symbol grounding/emergence problems shared by both of them. We also discuss whether animal cognition can develop recursive compositional information processing.