Distributed Representation Revisited

Figure-132-The-distributed-representation-of-language-meaning-in-neural-networks

If the conventional symbolic model mandates a creation of theory that is sought to address the issues pertaining to the problem, this mandatory theory construction is bypassed in case of distributed representational systems, since the latter is characterized by a large number of interactions occurring in a nonlinear fashion. No such attempts at theoretical construction are to be made in distributed representational systems for fear of high end abstraction, thereby sucking off the nutrient that is the hallmark of the model. Distributed representation is likely to encounter onerous issues if the size of the network inflates, but the issue is addressed through what is commonly known as redundancy technique, whereby, a simultaneous encoding of information generated by numerous interactions take place, thus ameliorating the adequacy of presenting the information to the network. In the words of Paul Cilliers, this is an important point, for,

the network used for the model of a complex system will have to have the same level of complexity as the system itself….However, if the system is truly complex, a network of equal complexity may be the simplest adequate model of such a system, which means that it would be just as difficult to analyze as the system itself.

Following, he also presents a caveat,

This has serious methodological implications for the scientists working with complex systems. A model which reduces the complexity may be easier to implement, and may even provide a number of economical descriptions of the system, but the price paid for this should be considered carefully.

One of the outstanding qualities of distributed representational systems is their adaptability. Adaptability, in the sense of reusing the network to be applicable to other problems to offer solutions. Exactly, what this connotes is, the learning process the network has undergone for a problem ‘A’, could be shared for problem ‘B’, since many of the input neurons are bounded by information learned through ‘A’ that could be applicable to ‘B’. In other words, the weights are the dictators for solving or resolving issues, no matter, when and for which problem the learning took place. There is a slight hitch here, and that being this quality of generalizing solutions could suffer, if the level of abstraction starts to shoot up. This itself could be arrested, if in the initial stages, the right kind of framework is decided upon, thus obscuring the hitch to almost non-affective and non-existence impacting factor. The very notion of weights is considered here by Sterelny as a problematic, and he takes it to attack distributed representation in general and connectionsim as a whole in particular. In an analogically witty paragraph, Sterelny says,

There is no distinction drawable, even in principle, between functional and non- functional connections. A positive linkage between two nodes in a distributed network might mean a constitutive link (eg. Catlike, in a network for tiger); a nomic one (carnivore, in the same network), or a merely associative one (in my case, a particular football team that play in black and orange.

It should be noted that this criticism on weights is derived, since for Sterelny, relationship between distributed representations and the micro-features that compose them is deeply problematic. If such is the criticism, then no doubt, Sterelny still seems to be ensconced within the conventional semantic/symbolic model. And since, all weights can take part in information processing, there is some sort of a democratic liberty that is accorded to the weights within a distributed representation, and hence any talk of constitutive, nomic, or even for that matter associative is mere humbug. Even if there is a disagreement prevailing that a large pattern of weights are not convincing enough for an explanation, as they tend to complicate matters, the distributed representational systems work consistently enough as compared to an alternative system that offers explanation through reasoning, and thereby, it is quite foolhardy to jettison the distributed representation by the sheer force of criticism. If the neural network can be adapted to produce the correct answer for a number of training cases that is large compared with the size of the network, it can be trusted to respond correctly to the previously unseen cases provided they are drawn from the same population using the same distribution as the training cases, thus undermining the commonly held idea that explanations are the necessary feature of the trustworthy systems (Baum and Haussler). Another objection that distributed representation faces is that, if representations are distributed, then the probability of two representations of the same thing as different from one another cannot be ruled out. So, one of them is the true representation, while the other is only an approximation of the representation.(1) This is a criticism of merit and is attributed to Fodor, in his influential book titled Psychosemantics.(2) For, if there is only one representation, Fodor would not shy from saying that this is the yucky solution, folks project believe in. But, since connectionism believes in the plausibility of indeterminate representations, the question of flexibility scores well and high over the conventional semantic/symbolic models, and is it not common sense to encounter flexibility in daily lives? The other response to this objection comes from post-structuralist theories (Baudrillard is quite important here. See the first footnote below). The objection of true representation, and which is a copy of the true representation meets its pharmacy in post-structuralism, where meaning is constituted by synchronic as well as diachronic contextualities, and thereby supplementing the distributed representation with a no-need-for concept and context, as they are inherent in the idea of such a representation itself. Sterelny, still seems to ride on his obstinacy, and in a vitriolic tone poses his demand to know as to why distributed representation should be regarded as states of the system at all. Moreover, he says,

It is not clear that a distributed representation is a representation for the connectionist system at all…given that the influence of node on node is local, given that there is no processor that looks at groups of nodes as a whole, it seems that seeing a distributed representation in a network is just an outsider’s perspective on the system.

This is moving around in circles, if nothing more. Or maybe, he was anticipating what G. F. Marcus would write and echo to some extent in his book The Algebraic Mind. In the words of Marcus,

…I agree with Stemberger(3) that connectionism can make a valuable contribution to cognitive science. The only place, we differ is that, first, he thinks that the contribution will be made by providing a way of eliminating symbols, whereas I think that connectionism will make its greatest contribution by accepting the importance of symbols, seeking ways of supplementing symbolic theories and seeking ways of explaining how symbols could be implemented in the brain. Second, Stemberger feels that symbols may play no role in cognition; I think that they do.

Whatever Sterelny claims, after most of the claims and counter-claims that have been taken into account, the only conclusion for the time being is that distributive representation has been undermined, his adamant position to be notwithstanding.

(1) This notion finds its parallel in Baudrillard’s Simulation. And subsequently, the notion would be invoked in studying the parallel nature. Of special interest is the order of simulacra in the period of post-modernity, where the simulacrum precedes the original, and the distinction between reality and representation vanishes. There is only the simulacrum and the originality becomes a totally meaningless concept.

(2) This book is known for putting folk psychology firmly on the theoretical ground by rejecting any external, holist and existential threat to its position.

(3) Joseph Paul Stemberger is a professor in the Department of Linguistics at The University of British Columbia in Vancouver, British Columbia, Canada, with primary interests in phonology, morphology, and their interactions. My theoretical orientations are towards Optimality Theory, employing our own version of the theory, and towards connectionist models.

 

Simulations of Representations: Rational Calculus versus Empirical Weights

While modeling a complex system, it should never be taken for granted that these models somehow simplify the systems, for that would only strip the models of the capability to account for encoding, decoding, and retaining information that are sine qua non for the environment they plan to model, and the environment that these models find themselves embedded in. Now, that the traditional problems of representation are fraught with loopholes, there needs to be a way to jump out of this quandary, if modeling complex systems are not to be impacted by the traces of these very traditional notions of representation. The employment of post-structuralist theories are sure indicative of getting rid of the symptoms, since they score over the analytical tradition, where, representation is only an analogue of the thing represented, whereas, simulation with its affinity to French theory is conducive to a distributed and a holistic analogy. Any argument against representation is not to be taken as meaning anti-scientific, since it is merely an argument against a particular scientific methodology and/or strategy that assumes complexity to be reducible, and therefore implementable or representable in a machine. The argument takes force only as an appreciation for the nature of complexity, something that could perhaps be repeated in a machine, should the machine itself be complex enough to cope with the distributed character of complexity. Representation is a state that stands-in for some other state, and hence is nothing short of “essentially” about meaning. The language, thought that is incorporated in understanding the world we are embedded in is efficacious only if representation relates to the world, and therefore “relationship” is another pillar of representation. Unless a relationship relates the two, one gets only an abstracted version of the so-called identities in themselves with no explanatory discourse. In the world of complexity, such identity based abstractions lose their essence, for modeling takes over the onus of explanations, and therefore, it is without doubt, the establishment of these relations that bring together states of representations as taking high priority. Representation holds a central value in both formal systems and in neural networks or connectionism, where the former is characterized by a rational calculus, and the latter by patterns that operate over the network lending it a more empirical weight.

figure8

Let logic programming be the starting point for deliberations here. The idea behind this is using mathematical logic to successfully apply to computer programming. When logic is used as such, it is used as a declarative representational language; declarative because, logic of computation is expressed without accounting for the flow of control. In other words, within this language, the question is centered around what-ness, rather than how-ness. Declarative representation has a counterpart in procedural representation, where the onus is on procedures, functions, routines and methods. Procedural representation is more algorithmic in nature, as it depends upon following steps to carry out computation. In other words, the question is centered around how-ness. But logic programming as it is commonly understood cannot do without both of them becoming a part of programming language at the same time. Since both of them are required, propositional logic that deals primarily with declarative representational languages would not suffice all alone, and hence, what is required is a logic that would touch upon predicates as well. This is made possible by first-order predicate logic that distinguishes itself from propositional logic by its use of quantifiers(1). The predicate logic thus finds its applications suited for deductive apparatus of formal systems, where axioms and rules of inferences are instrumental in deriving theorems that guide these systems. This setup is too formal in character and thus calls for a connectionist approach, since the latter is simply not keen to have predicate logic operate over deductive apparatus of a formal system at its party.

If brain and language (natural language and not computer languages, which are more rule-based and hence strict) as complex systems could be shown to have circumvented representationism via modeling techniques, the classical issues inherent in representation would be gotten rid of in the sense of a problematic. Functionalism as the prevalent theory in philosophy of mind that parallels computational model is the target here. In the words of Putnam,

I may have been the first philosopher to advance the thesis that the computer is the right model for mind. I gave my form of this doctrine the name ‘functionalism’, and under this name, it has become the dominant view – some say the orthodoxy – in contemporary philosophy of mind.

The computer metaphor with mind is clearly visible here, with the former having an hardware apparatus that is operated upon by the software programs, while the latter shares the same relation with brain (hardware) and mind (software). So far, so good, but there is a hitch. Like the computer side of metaphor, which can have a software loaded on to different hardwares, provided there is enough computational capability possessed by the hardware, the mind-brain relationship should meet the same criteria as well. If one goes by what Sterelny has hinted for functionalism as a certain physical state of the machine realizing a certain functional state, then a couple of descriptions, mutually exclusive of one another result, viz, a description on the physical level, and a description on the mental level. The consequences of such descriptions are bizarre to the extent that mind as a software can also find its implementation on any other hardware, provided the conditions for hardware’s capability to run the software are met successfully. One could hardly argue against these consequences that follow logically enough from the premisses, but a couple of blocks are not to be ignored at the same time, viz, the adequacy of the physical systems to implement the functional states, and what defines the relationships between these two mutually exclusive descriptions under the context of the same physical system. Sterelny comes up with a couple of criteria for adequate physical systems, designed, and teleological. Rather than provide any support for what he means by the systems as designed, he comes up with evolutionary tendencies, thus vouching for an external designer. The second one gets disturbing, if there is no description made, and this is precisely what Sterelny never offers. His citation of a bucket of water not having a telos in the sense of brain having one, only makes matters slide into metaphysics. Even otherwise, functionalism as a nature of mental states is metaphysical and ontological in import. This claim gets all the more highlighted, if one believes following Brentano that intentionality is the mark of the mental, then any theory of intentionality can be converted into a theory of of the ontological nature of psychological states. Getting back to the second description of Sterelny, functional states attain meaning, if they stand for something else, hence functionalism gets representational. And as Paul Cilliers says it cogently, grammatical structure of the language represents semantical content, and the neurological states of the brain represent certain mental states, thus proving without doubt, the responsibility on representation on establishing a link between the states of the system and conceptual meaning. This is again echoed in Sterelny,

There can be no informational sensitivity without representation. There can be no flexible and adaptive response to the world without representation. To learn about the world, and to use what we learn to act in new ways, we must be able to represent the world, our goals and options. Furthermore we must make appropriate inferences from these representations.

As representation is essentially about meaning, two levels are to be related with one another for any meaning to be possible. In the formal systems, or the rule-based approach, these relations are provided by creating a nexus between “symbol” and what it “symbolizes”. This fundamental linkage is offered by Fodor in his 1975 book, The Language of Thought. The main thesis of the book is about cognition and cognitive processes as remotely plausible, when computationally expressed in terms of representational systems. The language in possession of its own syntactic and semantic structures, and also independent of any medium, exhibits a causal effect on mental representations. Such a language is termed by him “mentalese”, which is implemented in the neural structure (a case in point for internal representation(2)), and following permutations allows for complex thoughts getting built up through simpler versions. The underlying hypothesis states that such a language applies to thoughts having propositional content, implying thoughts as having syntaxes. In order for complex thoughts to be generated, simple concepts are attached with the most basic linguistic token that combine following rules of logic (combinatorial rules). The language thus enriched is not only productive, with regard to length of the sentence getting longer (potentially so) without altering the meaning (concatenation), but also structured, in that rules of grammar that allow us to make inferences about linguistic elements previously unrelated. Once this task is accomplished, the representational theory of thought steps in to explicate on the essence of tokens and how they behave and relate. The representational theory of thought validates mental representations, that stand in uniquely for a subject of representation having a specific content to itself, to allow for causally generated complex thought. Sterelny echoes this when he says,

Internal representation helps us visualize our movements in the world and our embeddedness in the world. Internal representation takes it for granted that organisms inherently have such an attribute to have any cognition whatsoever. The plus point as in the work of Fodor is the absence of any other theory that successfully negotiates or challenges the very inherent-ness of internal representation.

For this model, and based on it, require an agent to represent the world as it is and as it might be, and to draw appropriate inferences from that representation. Fodor argues that the agent must have a language-like symbol system, for she can represent indefinitely many and indefinitely complex actual and possible states of her environment. She could not have this capacity without an appropriate means of representation, a language of thought. Mentalese thus is too rationalist in its approach, and hence in opposition to neural networks or connectionism. As there can be no possible cognitive processes without mental representations, the theory has many takers(3). One line of thought that supports this approach is the plausibility of psychological models that represent cognitive processes as representational thereby inviting computational thought to compute.

(1) Quantifier is an operator that binds a variable over a domain of discourse. The domain of discourse in turn specifies the range of these relevant variables.

(2) Internal representation helps us visualize our movements in the world and our embeddedness in the world. Internal representation takes it for granted that organisms inherently have such an attribute to have any cognition whatsoever. The plus point as in the work of Fodor is the absence of any other theory that successfully negotiates or challenges the very inherent-ness of internal representation.

(3) Tim Crane is a notable figure here. Crane explains Fodor’s Mentalese Hypothesis as desiring one thing and something else. Crane returns to the question of why we should believe the vehicle of mental representation is a language. Crane states that while he agrees with Fodor, his method of reaching it is very different. Crane goes on to say that reason: our ability as humans to decide a rational decision from the information giving is his argument for this question. Association of ideas lead to other ideas which only have a connection for the thinker. Fodor agrees that free association goes on but he says that is in a systemic, rational way that can be shown to work with the Language of Thought theory. Fodor states you must look at in a computational manner and that this allows it to be seen in a different light than normally and that free association follows a certain manner that can be broken down and explained with Language of Thought. Language of Thought.

scientificamericanmind0516-22-i5

 

Connectionism versus Representational Theory of Mind

mind-1272x400

Although there are some promises shown by the representational theory of mind (functionalism) with its insistence of rationalistic tendencies, there are objections that aim to derail this theory. Since the language of thought (representational theory of mind) believes in the existence of folk psychology, opponents of folk psychology dismiss this approach. As was discussed in the first chapter, folk psychology is not a very successful guide to explain the workings of the mind. Since representational theory explains how mental states are responsible for causing behaviors, it believes in folk psychology, and therefore the most acrimonious criticisms for this approach come from the eliminative materialist camp. For the eliminative materialist, there is a one to one mapping between psychological states and neurophysiological states in the brain thus expositing the idea that mental behavior is better explained as compared to when neurophysiology attempts to do the same, since the vistas for doing so are quantifiably and qualifiably more. Behaviorism with its insistence on the absence of linkages between mental states and effects of behavior would be another objection. Importantly, even Searle refuses to be a part of representational theory, with his biological naturalism (as discussed in the second chapter) which is majorly non-representational and investing faith in the causal efficacy of mental states. Another objection is the homunculi regression, according to which there is an infinite regression of explanation about how sentences get their meanings. For Searle, even if this is true, it is only partly true, since it is only at the bottom-level homunculi where manipulation of symbols take place, after which there is aporetic situation. Daniel Dennett on the other hand talks about a “no-need-for-interpretation” at this bottom level, since at this level simplicity crops up. Searle is a monist, but divides intentional states into low-level brain activity and high-level mental activity. So it is these lower-level, nonrepresentational neurophysiological processes that have causal powers in intention and behavior, rather than some higher-level mental representation. Yet another form of challenge comes from within the camp of representational theory of mind itself, which is suggestive of scientific-cognitive research as showing the amount of intelligent action as generated by complex interactions involving neural, bodily and environmental factors. This threat to representational theory(1) is prosaically worded by Wheeler and Clark , when they say,

These are hard times for the notion of internal representation. Increasingly, theorists are questioning the explanatory value of the appeal to internal representation in the search for a scientific understanding of the mind and of intelligent action. What is in dispute is not, of course, the status of certain intelligent agents as representers of their worlds…What is in doubt is not our status as representers, but the presence within us of identifiable and scientifically well-individuated vehicles of representational content. Recent work in neuroscience(2), robotics, philosophy, and development psychology suggests, by way of contrast, that a great deal of (what we intuitively regard as) intelligent action maybe grounded not in the regimented activity of inner content-bearing vehicles, but in complex interactions involving neural, bodily, and environmental factors and forces.

There is a growing sense of skepticism against representational theory, and from within the camp as well, though not all that hostile as the above quote specified. Speculations are rife that this may be due to embracing the continental tradition of phenomenology and Gibsonian psychology, which could partly indicate a move away from the strictures of rule-based approach. This critique from within the camp of internal representation would come in very handy in approaching connectionism and in prioritizing its utility, as a substitute to representation. What is of crucial importance here is the notion of continuous reciprocal causation, which involves multiple simultaneous interactions alongside complex dynamical feedback loops, thus facilitating a causal contribution of each component in the system as determining and determined by the causal contributions of large number of other components on the one hand, and the potentiality of these contributions to change in a radical manner temporally. When the complexity of the causal interactions shoots up, it automatically signals the difficulty level of representation’s explanatory prowess that simultaneously rises. But the real threat to the representational theory comes from connectionism, which despite in agreement with some of the premisses of representational theory, deviate greatly when it comes to creating machines that could be said to think. With neural networks and weights attached to them, a learning algorithm makes it possible to undergo modifications within the network over time. Although, Fodor defends his language of thought, or the representation theory in general against connectionism by claiming that the neural network is just a realization of the computational theory of mind that necessarily employs symbol manipulation. He does this through his use of cognitive architecture. Campers in connectionism however, deny connectionism as a mere implementation of representational theory, in addition to claiming that the laws of nature do not have systematicity as resting on representation, and most importantly deny the thesis of Fodor that essentially cognition is a function that uses representational input and output in favor of eliminative connectionism. Much of the current debate between the two camps revolves around connectionst’s denial of connectionism as a mere implementation of representational theory based on cognitive architecture, which is truism in the case of classicist model. The response from connectionist side is to build a representational system, that agrees with mental representations as constituting the direct objects of propositional attitudes and in possession of combinatorial syntax and semantics, with the domain of mental processes as causally sensitive to the syntactic/formal structure of representations as defined by these combinatorial syntax, thus relying upon a non-concatenative realization of syntactic/structural complexity of representations, in turn yielding a non-classical system.

redneuronal-1024x768

—————————

(1) As an aside, on the threat of representational theory, I see a lot of parallel here between the threat to internal representation and Object Oriented Ontology (L. Bryant flavor), where objects are treated as black boxes, and hence objects are withdrawn, implying that no one, including us would claim any direct access to the inner world, thereby shutting the inner world from any kind of representation. Although paradoxically sounding, it is because of withdrawal that knowledge of objects becomes thoroughly relational. When we know an object, we do not know it per se, but through a relation, thus knowledge shifts from the register of representation to that of performance, meeting its fellow travelers in Deleuze and Guattari, who defend a performative ontology of the world ala Andrew Pickering, by claiming that more than what a language represents, what a language does is interesting.

(2) Not a part of the quotation, but a brief on the recent work in neurons, which are prone to throwing up surprises. Neurons are getting complicated, but the basic functional concept is still synapses as transmitting electrical signals to the dendrites and the cell body (input), whereas the axons carry signals away (output). What is surprising is the finding by the scientists at the Northwestern University, that axons can act as input agents too. There is another way of saying this: axons talk with one another. Before sending signals in reverse, axons carry out their own neural computations without the aide from the cell body or dendrites. Now this is in contrast to a typical neuronal communication where an axon of one neuron is in contact with another neuron’s cell body or dendrite, and not its axon. The computations in axons are slower to a degree of 103 as compared with the computations in dendrites, potentially creating a means for neurons to compute fast things in dendrites, and slow ones in axons. Nelson Spruston, senior author of the paper (“Slow Integration Leads to Persistent Action Potential Firing in Distal Axons of Coupled Interneurons”) and professor of neurobiology and physiology in the Weinberg College of Arts and Sciences, says,

“We have discovered a number of things fundamental to how neurons work that are contrary to the information you find in neuroscience textbooks. Signals can travel from the end of the axon toward the cell body, when it typically is the other way around. We were amazed to see this.”

He and his colleagues first discovered individual nerve cells can fire off signals even in the absence of electrical stimulations in the cell body or dendrites. It’s not always stimulus in, immediate action potential out. (Action potentials are the fundamental electrical signaling elements used by neurons; they are very brief changes in the membrane voltage of the neuron). Similar to our working memory when we memorize a telephone number for later use, the nerve cell can store and integrate stimuli over a long period of time, from tens of seconds to minutes. (That’s a very long time for neurons). Then, when the neuron reaches a threshold, it fires off a long series of signals, or action potentials, even in the absence of stimuli. The researchers call this persistent firing, and it all seems to be happening in the axon. Spruston further says,

“The axons are talking to each other, but it’s a complete mystery as to how it works. The next big question is: how widespread is this behavior? Is this an oddity or does in happen in lots of neurons? We don’t think it’s rare, so it’s important for us to understand under what conditions it occurs and how this happens.”