Distributed Representation Revisited

Figure-132-The-distributed-representation-of-language-meaning-in-neural-networks

If the conventional symbolic model mandates a creation of theory that is sought to address the issues pertaining to the problem, this mandatory theory construction is bypassed in case of distributed representational systems, since the latter is characterized by a large number of interactions occurring in a nonlinear fashion. No such attempts at theoretical construction are to be made in distributed representational systems for fear of high end abstraction, thereby sucking off the nutrient that is the hallmark of the model. Distributed representation is likely to encounter onerous issues if the size of the network inflates, but the issue is addressed through what is commonly known as redundancy technique, whereby, a simultaneous encoding of information generated by numerous interactions take place, thus ameliorating the adequacy of presenting the information to the network. In the words of Paul Cilliers, this is an important point, for,

the network used for the model of a complex system will have to have the same level of complexity as the system itself….However, if the system is truly complex, a network of equal complexity may be the simplest adequate model of such a system, which means that it would be just as difficult to analyze as the system itself.

Following, he also presents a caveat,

This has serious methodological implications for the scientists working with complex systems. A model which reduces the complexity may be easier to implement, and may even provide a number of economical descriptions of the system, but the price paid for this should be considered carefully.

One of the outstanding qualities of distributed representational systems is their adaptability. Adaptability, in the sense of reusing the network to be applicable to other problems to offer solutions. Exactly, what this connotes is, the learning process the network has undergone for a problem ‘A’, could be shared for problem ‘B’, since many of the input neurons are bounded by information learned through ‘A’ that could be applicable to ‘B’. In other words, the weights are the dictators for solving or resolving issues, no matter, when and for which problem the learning took place. There is a slight hitch here, and that being this quality of generalizing solutions could suffer, if the level of abstraction starts to shoot up. This itself could be arrested, if in the initial stages, the right kind of framework is decided upon, thus obscuring the hitch to almost non-affective and non-existence impacting factor. The very notion of weights is considered here by Sterelny as a problematic, and he takes it to attack distributed representation in general and connectionsim as a whole in particular. In an analogically witty paragraph, Sterelny says,

There is no distinction drawable, even in principle, between functional and non- functional connections. A positive linkage between two nodes in a distributed network might mean a constitutive link (eg. Catlike, in a network for tiger); a nomic one (carnivore, in the same network), or a merely associative one (in my case, a particular football team that play in black and orange.

It should be noted that this criticism on weights is derived, since for Sterelny, relationship between distributed representations and the micro-features that compose them is deeply problematic. If such is the criticism, then no doubt, Sterelny still seems to be ensconced within the conventional semantic/symbolic model. And since, all weights can take part in information processing, there is some sort of a democratic liberty that is accorded to the weights within a distributed representation, and hence any talk of constitutive, nomic, or even for that matter associative is mere humbug. Even if there is a disagreement prevailing that a large pattern of weights are not convincing enough for an explanation, as they tend to complicate matters, the distributed representational systems work consistently enough as compared to an alternative system that offers explanation through reasoning, and thereby, it is quite foolhardy to jettison the distributed representation by the sheer force of criticism. If the neural network can be adapted to produce the correct answer for a number of training cases that is large compared with the size of the network, it can be trusted to respond correctly to the previously unseen cases provided they are drawn from the same population using the same distribution as the training cases, thus undermining the commonly held idea that explanations are the necessary feature of the trustworthy systems (Baum and Haussler). Another objection that distributed representation faces is that, if representations are distributed, then the probability of two representations of the same thing as different from one another cannot be ruled out. So, one of them is the true representation, while the other is only an approximation of the representation.(1) This is a criticism of merit and is attributed to Fodor, in his influential book titled Psychosemantics.(2) For, if there is only one representation, Fodor would not shy from saying that this is the yucky solution, folks project believe in. But, since connectionism believes in the plausibility of indeterminate representations, the question of flexibility scores well and high over the conventional semantic/symbolic models, and is it not common sense to encounter flexibility in daily lives? The other response to this objection comes from post-structuralist theories (Baudrillard is quite important here. See the first footnote below). The objection of true representation, and which is a copy of the true representation meets its pharmacy in post-structuralism, where meaning is constituted by synchronic as well as diachronic contextualities, and thereby supplementing the distributed representation with a no-need-for concept and context, as they are inherent in the idea of such a representation itself. Sterelny, still seems to ride on his obstinacy, and in a vitriolic tone poses his demand to know as to why distributed representation should be regarded as states of the system at all. Moreover, he says,

It is not clear that a distributed representation is a representation for the connectionist system at all…given that the influence of node on node is local, given that there is no processor that looks at groups of nodes as a whole, it seems that seeing a distributed representation in a network is just an outsider’s perspective on the system.

This is moving around in circles, if nothing more. Or maybe, he was anticipating what G. F. Marcus would write and echo to some extent in his book The Algebraic Mind. In the words of Marcus,

…I agree with Stemberger(3) that connectionism can make a valuable contribution to cognitive science. The only place, we differ is that, first, he thinks that the contribution will be made by providing a way of eliminating symbols, whereas I think that connectionism will make its greatest contribution by accepting the importance of symbols, seeking ways of supplementing symbolic theories and seeking ways of explaining how symbols could be implemented in the brain. Second, Stemberger feels that symbols may play no role in cognition; I think that they do.

Whatever Sterelny claims, after most of the claims and counter-claims that have been taken into account, the only conclusion for the time being is that distributive representation has been undermined, his adamant position to be notwithstanding.

(1) This notion finds its parallel in Baudrillard’s Simulation. And subsequently, the notion would be invoked in studying the parallel nature. Of special interest is the order of simulacra in the period of post-modernity, where the simulacrum precedes the original, and the distinction between reality and representation vanishes. There is only the simulacrum and the originality becomes a totally meaningless concept.

(2) This book is known for putting folk psychology firmly on the theoretical ground by rejecting any external, holist and existential threat to its position.

(3) Joseph Paul Stemberger is a professor in the Department of Linguistics at The University of British Columbia in Vancouver, British Columbia, Canada, with primary interests in phonology, morphology, and their interactions. My theoretical orientations are towards Optimality Theory, employing our own version of the theory, and towards connectionist models.

 

Complexity Theory and Philosophy: A Peace Accord

neuron_spark

Complexity has impacted fields diverse from the one it originated in, i.e. science. It has touched the sociological domains, and organizational sciences, but sadly, it has not had much of a say in mainstream academic philosophy. In sociology, John Urry (2003) examines the ideas of chaos and complexity in carrying out analyses of global processes. He does this, because he believes that systems are balanced between order and chaos, and that, there is no teleological move towards any state of equilibrium, as the events that pilot the system are not only unpredictable, but also irreversible at the same time. Such events rupture the space-time regularity with their dimension of unpredictability that was thought of as characterizing hitherto known sociological discursive practices. A highly significant contribution that comes along with such an analyses is the distinguishing between what Urry aptly calls “global networks” and “global fluids”. Global fluids are a topographical space used to describe the de-territorialized movement of people, information, objects, finances in an undirected, nonlinear mode, and in a way are characteristic of emergentism and hybridization. The topographies of global networks and global fluids interact in complex manner to give rise to emergent properties that define systems as always on the edge of chaos, pregnant with unpredictability.

emergentism

Cognitive science and evolutionary theory have been inspirational for a lot of philosophical investigations and have also benefited largely from complexity theory. If such is the case, the perplexing thing is complexity theory’s impact in philosophy, which has not had major inroads to make. Why could this be so? Let us ponder this over.

Analytical philosophy has always been concerned with analysis, and logical constructs that are to be stringently followed. These rules and regulations take the domain of philosophical investigations falling under the rubric of analytical tradition away from holism, uncertainty, unpredictability and subjectivity that are characteristics of complexity. The reason why this could be case is attributable to complexity theory as developed on the base of mathematics and computational theories, which, somehow is not the domain of academic philosophy dealing with social sciences and cultural studies in present days, but is confined to discussions and debates amongst philosophers of science (biology is an important branch here), mathematics and technology. Moreover, the debates and deliberations have concerned themselves with the unpredictable and uncertain implications as derived from the vestiges of chaos theory and not complexity theory per se. This is symptomatic of the fact that a lot of confusion rests upon viewing these two path-breaking theories as synonymous, which, incidentally is a mistake, as the former happens at best to be a mere subset of the latter. An ironical fate encountered philosophy, since it dealt with complex notions of language, without actually admitting to the jargon, and technical parlance of complexity theory. If philosophy lets complexity make a meaningful intercourse into its discursive practices, then it could be beneficial to the alliance. And the branch of philosophy that is making use of this intervention and alliance at present is post-modern philosophy ++++

The works of Freud and Saussure as furthered by Lacan and Derrida, not only accorded fecundity for a critique of modernity, but, also opened up avenues for a meaningful interaction with complexity. French theory at large was quite antagonistic to modernist claims of reducing the diverse world to essential features for better comprehensibility, and this essentially lent for its affinity to complexity. Even if Derrida never explicitly used the complexity parlance in his corpus, there appears to be a strong sympathy towards the phenomenon via his take on post-structuralism. On the other hand, Lyotard, in setting his arguments for post-modern conditions of knowledge was ecstatic about paralogy as a defining feature, which is no different from the way complexity, connectionism and distributed systems would harbor.

cc40a61d21aaf144f8e7cf31c50cd31b

Even Deleuze and Guattari are closer to the complex approach through their notions of rhizomes, which are non-reductive, non-hierarchical, and multiplicities oriented connections in data representations and interpretations, and are characterized by horizontal connectivities, as contrasted with arborescent models that find their characterizations in vertical and linear determinations. The ideas are further developed by De Landa (2006), where the attempt is to define a new ontology that could be utilized by social scientists. Components that make up the assemblages are characterized along two axes viz, material, explicating on the variable roles components might undergo, and territorializng/deterritorializing, explicating on processes components might be involved with.

mehretu

Relations of exteriority define components, implying that components are self-subsistent, or that there is never a loss of identity for them, during the process of being unplugged from one assemblage to be plugged into another. This relationship between the assemblages and components is nonlinearly and complexly defined, since assemblages are affected by lower level ones, but could also potentially act on to these components affecting adaptations in them. This is so similar to the way distributed systems are principally modeled. Then why has philosophy at large not shown much impact from complexity despite the French theoretical affinities with the latter?

Chaos theory is partly to blame here, for it has twisted the way a structure of a complex system is understood. The systems have a non-linear operational tendencies, and this has obfuscated the notion of meaning as lying squarely on relativism. The robustness of these systems, when looked at in an illuminating manner from the French theoretical perspective could be advantageous to get rid of ideas about complex systems as based on a knife’s edge, despite being nonlinearly determinable. If the structure of the system were a problematic, then defining limits and boundaries was no easy job. What is the boundary between the system and the environment? Is it rigorously drawn and followed, or is it a mere theoretical choice and construct? These are valid question, which philosophy found it difficult to come to terms with. These questions gained intensity with the introduction of self-organizational systems and/or autopoietic ones. Classical and modern philosophies either had to dismiss these ideas as chimerical, or it had to close off its own analyzing methods in dealing with these issues, and both of these approaches had a detrimental effect of isolating the discipline of philosophy from the cultural domains in which such notions were making positive interventions and inroads. It could safely be said that French theory, in a way tried its rescue mission, and picked up momentum in success. The major contribution from continental philosophy post-60s was framing solutions. Framing, as a schema of interpretation helped comprehending and responding to events and enabled systems and contexts to constitute one another, thus positing a resolution on the boundaries and limits issues that had plagued hitherto known philosophical doctrines.

The notion of difference, so central to modernism was a problematic that needed to be resolved. Such was never a problem within French theory, but was a tonic to be consumed along side complexity, to address socio-economic and political issues. Deleuze (1994), for example, in his metaphysical treatise, sought a critique of representation, and a systematic inversion of the traditional metaphysical notions of identity and difference. Identities were not metaphysically or logically prior to differences, and identities in whatever categories, are pronounced by their derivation from differences. In other words, forms, categories, apperception, and resemblances fail to attain their differences in themselves. And, as Deleuze (2003: 32) says,

If philosophy has a positive and direct relation to things, it is only insofar as philosophy claims to grasp the thing itself, according to what it is, in its difference from everything it is not, in other words, in its internal difference.

But Deleuzean thesis on metaphysics does make a political intervention, like when he says,

The more our daily life appears standardized, stereotyped, and subject to an accelerated reproduction of objects of consumption, the more art must be injected into it in order to extract from it that little difference which plays simultaneously between other levels of repetition, and even in order to make the two extremes resonate — namely, the habitual series of consumption and the instinctual series of destruction and death. (Deleuze 1994: 293).(1)

Tackling the complexity within the social realm head-on does not lie in extrapolating convenient generalities, and thereafter trying to fathom how finely they fit together, but, rather in apprehending the relational schema of the network, within which, individuals emerge as subjects, objects and systems that are capable of grasping the real things.(2) 

One major criticism leveled against complexity is that it is sympathetic to relativism, just like most of the French theoretical thought is. Whether, this accusation has any substance to it could be measured by the likes of circular meaningless debates like the Sokal hoax. The hoax was platitudinous to say the least, and vague at best. And why would this be so? Sokal in his article, “Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity”, incorporated the vocabulary of his specialized discipline to unearth the waywardness of usage by the French theorists. This, for Sokal was fashionable nonsense, or an act of making noise. He takes the French theorists to task for a liberal use of terms like chaos, complexity, quantum, relativity, gender, difference, topology, and deconstruction, without any proper insight. Who would be vague in the Sokal affair? The physicist, or the bunch of French theorists? Such an issue could be tackled on an intelligibility concern. Intelligibility is a result of differentiation and not a guarantee of truth-giving process (Cilliers 2005: 262).

Clearly communicated does not give any indisputable identity to a concept. The only way, (such a meaning can) be meaningful is through limitations being set on such communications, an ethical choice once again. These limitations enable knowledge to come into existence, and this must be accepted de facto. In a parallel metaphoric with complexity, these limitations or constraints are sine qua non for autopoiesis to make an entry. Cilliers (2005: 264) is quite on target, when he lays down the general schema for complexity, if it is, aligned with notions of chaos, randomness and noise, the accusations of relativism and vagueness will start to hold water. It is aligned with notions of structure as the result of contingent constraints, we can make claims about complex systems, which are clear and comprehensible, despite the fact that the claims themselves are historically contingent.

Undoubtedly, complexity rides on modesty. But, the accusations against this position only succeed to level complexity as weak, a gross mistake in itself. Let us take Derrida here, as read by Sweetman (1999). Sweetman cites Derrida as an ideal post-modernist, and thereafter launches an attack on his works as confusing aesthetics with metaphysics, as mistakenly siding with assertions over arguments in philosophy, as holding Derrida for moral and epistemological relativism and, self-contradictory with a tinge of intellectual arrogance. Such accusations, though addressed by Derrida and his scholars at various times, nevertheless find parallels in complexity, where, the split is between proponents of mathematical certainty in dealing with complexity on the one hand, and proponents of metaphorical proclivities in dealing with the phenomenon on the other. So, how would relativism make an entry here? Being a relativist is as good as swimming in paradoxical intellectual currents, and such a position is embraced due to a lack of foundational basis for knowledge, if nothing more. The counter-argument against the relativistic stance of complexity could be framed in a simplistic manner, by citing the case of limited knowledge as not relativistic knowledge. If these forms of knowledge were equated in any manner, it would only help close doors on investigations.

A look at Luhmann’s use of autopoiesis in social theory is obligated here. This is necessitated by the fact of autopoiesis getting directly imported from biological sciences, to which, even Varela had objections, though intellectually changing tracks. Luhmann considers the leaving out of self-referentiality as a problematic in the work of Chileans (Maturana + Varela), since for Luhmann systems are characterized by general patterns which can just be described as making a distinction and crossing the boundary of the distinction [which] enables us to ask questions about society as a self-observing systems[s] (Hayles, K., Luhmann, N., Rasch, W., Knodt, E. & Wolfe, C., 1995 Autumn). Such a reaction from Luhmann is in his response to a cautious undertaking of any import directly from biological and psychological sciences to describe society and social theory. Reality is always distorted through the lens of perception and, this blinds humans from seeing things-in-themselves (the Kantian noumenon). One could visualize this within the analytical tradition of language as a problematic, involving oppositional thinking within the binary structure of linguistic terms themselves. What is required is an evolutionary explanation of how systems survive to the extent that they can learn to handle the inside/outside difference within the system, and within the context of their own operations, since they can never operate outside the system (Hayles, K., Luhmann, N., Rasch, W., Knodt, E. & Wolfe, C., 1995 Autumn). For the social theory to be effective, what requires deconstruction is the deconstruction of the grand tautological claim of autopoiesis, or the unity of the system as produced by the system itself. Luhmann tells us that a methodology that undertakes such a task must do this empirically by identifying the operations which produce and reproduce the unity of the system (Luhmann 1992). This is a crucial point, since the classical/traditional questions as regards the problem of reference as conditioning meaning and truth, are the distinctions between the subject and the object. Luhmann thinks of these questions as quasi-questions, and admonishes a replacement by self-reference/external-reference for any meaningful transformation to take effect. In his communications theory(3), he states flatly that as a system, it depends upon “introducing the difference between system and environment into the system” as the internal split within the system itself that allows it to make the distinction to begin its operative procedures to begin with (Luhmann 1992: 1420). The self-reference/external-reference distinction is a contingent process, and is open to temporal forms of difference. How to define the operation that differentiates the system and organizes the difference between system and environment while maintaining reciprocity between dependence and independence is a question that demands a resolution. The breakthrough for autopoietic systems is provided by the notion of structural coupling, since a renunciation of the idea of overarching causality on the one hand, and the retention of the idea of highly selective connections between systems and environments is effected here. Structural coupling maintains this reciprocity between dependence and independence. Moreover, autopoietic systems are defined by the way they are, by their mode of being in the world, and by the way they overcome or encounter entropy in the world. In other words, a self-perpetuating system performing operational closure continuously are autopoietic systems that organize dynamic stability.

wpid-dznthe-autopoiesis-of-architecture-by-patrik-schumacher-6

Even if the concepts of complexity have not traveled far and wide into the discipline of philosophy, the trends are on the positive side. Developments in cognitive sciences and consciousness studies have a far reaching implications on philosophy of mind, as does in research in science that helps redefine the very notion of life. These researches are carried out within the spectrum of complexity theory, and therefore, there is a lot of scope for optimism. Complexity theory is still in the embryonic stage, for it is a theory of the widest possible extent for our understanding the world that we inhabit. Though, there are roadblocks along the way, it should in no way mean that it is the end of the road for complexity, but only a beginning in a new and novel manner.

Complexity theory as imbibed within adaptive systems has a major role in evolutionary doctrines. To add to this, the phenomenon of French Theory has incited creative and innovative ways of looking at philosophy, where residues of dualism and reductionism still rest, and resist any challenges whatsoever. One of the ways through which complexity and philosophy could come closer is, when the latter starts withdrawing its investigations into the how- ness of something, and starts to seriously incorporate the why-ness of it. The how- ness still seems to be arrested within the walls of reductionism, mechanicism, modernism, and the pillars of Newtonian science. So, an ontological reduction of all phenomenon under the governance of deterministic laws is the indelible mark, even if epistemologically, a certain guideline of objectivity seems apparent. What really is missed out on in this process is the creativity, as world in particular and universe in general is describable as a mechanism following clockwork. Such a view held sway for most the modern era, but with the advent of scientific revolutions in the 20th century, things began to look awry. Relativity theory, quantum mechanics, chaos, complexity, and recently string/M-theory were powerful enough in their insights to clean off the hitherto promising and predictable scientific ventures. One view at quantum mechanics/uncertainty and chaos/non-linear dynamics was potent to dislodge predictability from science. This was followed in succession by systems theory and cybernetics, which were instrumental in highlighting the scientific basis for holism and emergence, and showing equally well that knowledge was intrinsically subjective. Not just that, autopoiesis clarified the picture of regularity and organization as not given, but, rather dependent on a dynamically emergent tangle of conflicting forces and random fluctuations, a process very rightly referred to by Prigogine and Stengers (1984) as “order out of chaos”. In very insightful language, Heylighen, Cilliers and Gershenson (2007) pin their hopes on these different approaches, which are now starting to become integrated under the heading of “complexity science”. It’s central paradigm is the multi-agent system: a collection of autonomous components whose local interactions give rise to a global order. Agents are intrinsically subjective and uncertain about the consequences of their actions, yet they generally manage to self-organize into an emergent, adaptive system. Thus uncertainty and subjectivity should no longer be viewed negatively, as the loss of the absolute order of mechanicism, but positively, as factors of creativity, adaptation and evolution….Although a number of (mostly post-modern) philosophers have expressed similar sentiments, the complexity paradigm still needs to be assimilated by academic philosophy.

Such a need is a requisite for complexity to become more aware about how modeling techniques could be made more robust, and for philosophy to understand and resolve some hitherto unaddressed, but perennial problems.

———————————————————–

1  The political implications of such a thesis is rare, but forceful. To add to the quote above, there are other quotes as well, that deliberate on socio-political themes. Like,

“We claim that there are two ways to appeal to ‘necessary destructions’: that of the poet, who speaks in the name of a creative power, capable of overturning all orders and representations in order to affirm Difference in the state of permanent revolution which characterizes eternal return; and that of the politician, who is above all concerned to deny that which ‘differs,’ so as to conserve or prolong an established historical order.” (Deleuze 1994: 53).

and,

“Real revolutions have the atmosphere of fétes. Contradiction is not the weapon of the proletariat but, rather, the manner in which the bourgeoisie defends and preserves itself, the shadow behind which it maintains its claim to decide what the problems are.” (Deleuze 1994: 268).

2 It should however be noted, that only immanent philosophies of the sort Deleuze propagates, the processes of individuation could be accounted for. Moreover, once such an aim is attained, regularities in the world are denied any eternal and universal validation.

3 He defines communication as “a kind of autopoetic network of operations which continually organizes what we seek, the coincidence of self-reference (utterance) and external reference (information)” (1992: 1424). He details this out saying,

“Communication comes about by splitting reality through a highly artificial distinction between utterance and information, both taken as contingent events within an ongoing process that recursively uses the results of previous steps and anticipates further ones”. (1992: 1424).

Bibliography

Ciliers, P. (2005) Complexity, Deconstruction and Relativism. In Theory, Culture & Society, Vol. 22 (5). pp. 255 – 267.

De Landa, M. (2006) New Philosophy of Society: Assemblage Theory and Social Complexity. London: Continuum.

Deleuze, G. (1994) Difference and Repetition. Translated by Patton, P. New York: Columbia University Press.

—————- (2003) Desert Islands and Other Texts (1953-1974). Translated by Taormina, M. Los Angeles: Semiotext(e).

Hayles, K., Luhmann, N., Rasch, W., Knodt, E. & Wolfe, C. (1995 Autumn) Theory of a Different Order: A Conversation with Katherine Hayles and Niklas Luhmann. In Cultural Critique, No. 31, The Politics of Systems and Environments, Part II. Minneapolis, MN: University of Minnesota Press.

Heylighen, F., Cilliers, P., and Gershenson, C. (2007) The Philosophy of Complexity. In Bogg, J. & Geyer, R. (eds), Complexity, Science and Society. Oxford: Radcliffe Publishing.

Luhmann, N (1992) Operational Closure and Structural Coupling: The Differentiation of the Legal System. Cardoza Law Review Vol. 13.

Lyotard, J-F. (1984) The Postmodern Condition: A Report on Knowledge. Translated by Bennington, G. & Massumi, B. Minneapolis, MN: University of Minnesota Press.
Prigogine, I. and Stengers, I. (1984) Order out of Chaos. New York: Bantam Books.

Sweetman, B. (1999) Postmodernism, Derrida and Différance: A Critique. In International Philosophical Quarterly XXXIX (1)/153. pp. 5 – 18.

Urry, J. (2003) Global Complexity. Cambridge: Polity Press.