I also don't see that it is an open question as to whether language is associated with cognition or communication -- I think it is uncontroversial that it is associated with both. Perhaps you mean something more specific than "associated with".

What I mean is that others would completely disagree with you. It's a controversial topic, more than an open question: most people seem to have an answer, but that answer differs, with each perspective having a very strong view.

It seems to me that a lot of people link language directly to communication, while others (perhaps more) make the argument you're making. That's all.

I've always found that argument (regarding imperfections in human communication) to be very confusing, for two reasons:

1. It may just be inherent that, given the complexity of our message and use in the real world (rather than, say, transferring a file by email with machines that are inherently accurate for every bit of information-- literally), such imperfections are just natural. Nothing "human" about it, just part of having a complex code.

2. It is, I think, a distraction from the real issue: it confuses interpretation and information, in that no code is fully unambiguous regarding interpretation, and no code is fully explicit. For example the data stream for an audio file of the human voice has exactly the same ambiguities and underspecifications as that human voice plus more when the channel is noisy and certain sounds are less clear.

So we must distinguish between the code itself (the combination of sounds->words being transmitted from one individual to another) and their purpose in communication (where these imperfections really arise).

There are of course some legitimate cases of ambiguity in the code itself. But this is just an effect of a natural evolution of the code by trial and error, where some messages happen to look alike. They are also rarely truly ambiguous in context and with intonation. Furthermore, this ambiguity is an effect of linearization of the signal more than anything.

The reason it (that is, ambiguity) doesn't come up for animals is that their codes aren't that complicated (at least as far as we understand them), but remember that they are certainly underspecified quite often ("danger" -- of what?).

I'm not at all sure why one would conclude that animal communication is unambiguous or less ambiguous than human language.  If language is evolved to facilitate replication of a train of thought--an understanding of the intentions of other animals--then it may well be that the more limited modes of expression available to non-human animals may be open to far many more interpretations.  The human "call system"--cries, laughter, screams, etc.--is certainly open to lots of different interpretations.  For example, crying can indicate sorrow, but also relief.  Human language gives us the advantage that it can be far more precise about thought content in different contexts than a limited set of calls, postures, and expressions can.

Your argument seems impressive --at least for a fellow like me who becomes dizzy when I see formulae in a text. As I am a perfect nullity in information theory, I could not make heads and tales of your last posting. But I am also a member of another forum concentrating on Relevance Theory, where I wrote asking for helping me decipher (and, if possible, respond to) your text.

Here is what I got:

if I understand it correctly, your interlocutor might have a point in saying that, since (if only because of channel noise) there is no such thing in reality as an absolutely unambiguous signal, the optimality of a code is a gradual (sc. statistical) notion in code theory; from their point of view, this would make an "ideal" code a straw man.

Where this could be countered, I think, is the introduction of "the real world context" C as a factor in disambiguation. As shown by Sperber & Wilson, the context is not given, but "chosen". And this choice is not a matter of decoding (even in a more sophisticated sense of decoding); moreover, in metaphor, irony or ad hoc concepts, this choice may even make us override linguistic code rather than disambiguate it.

In fact, there has been some work in information science, esp. with respect to information retrieval systems, on the problem of "relevance" and the need to include context and users' knowledge and preferences in the definition of relevance. So possibly an overly simple code model could even be challenged on its home ground? (Jan Straßheim)

I can't speak for Guijarro, but I can try to express how I understood his point.  It might help to consider Michael Reddy's conduit metaphor of language, which he developed in the 1970s.  Basically, the idea is that language is thought of as a "pipe" through which information flows.  It is packaged or encoded at one end and decoded at the other.  This metaphor has dominated thinking about language for a very long time, and it is very powerful.  Formal languages tend to conform to it.  So to extract information from a linguistic signal, all one has to do is simply decode the content contained in it.  Information theory is essentially about signal processing, not natural human language processing.  The conduit metaphor is misleading for human language, but not formal symbolic systems.

An alternative metaphor--one that Charles Fillmore once expressed to me--is that language is "word-guided mental telepathy".  That is, it is thought replication that uses linguistic expressions as keys to unlock associative clumps of information.  So, if you look at his FrameNet approach to semantics, the "frames" represent clumps of conceptual associations that one can then map words onto.  So the semantic structure of a sentence is not actually structured like the linguistic signal, but the linguistic signal evokes it through association with frames.  Frames then structurally represent the information that the speaker is trying to communicate.  So I suppose that that is one way of looking at what you call "accessibility heuristics".  However, there could be non-linguistic information that also contributes to the slot-filling activity of assigning roles to entities.  Formal languages are always literal and compositional in usage, whereas natural language can have layers of non-literal and conventional significance.  The natural linguistic signal is always going to be a defective component of a discourse context.

Hmm, I still don't see how these points argue against the code view. Formal languages may be literal and compositional, but information-theoretically optimal codes often aren't. Arithmetic coding, for example, approaches information-theoretic limits and is not compositional.

IMO, the problem is that you still think of the meaning of an expression as somehow fully encoded in the signal.  The point I was trying to make is that it isn't.  The signal is semantically defective, but it contains information that enables the receiver to assemble the meaning, given assumptions made by the sender.  That is, linguistic meanings are essentially inferred from components of the signal, not encoded in it.  My criticism of the information-theoretic approach is that it fully buys into the "conduit metaphor" view of language.  That metaphor seems to hold up at the sentence level, but it ignores the fact that sentences only convey meaning in an assumed context.  However, once you start looking at the level of discourse processing, it breaks down rather quickly.

I think Fillmore's metaphor is fully compatible with an information-theoretic approach. Listeners don't just have a model for relating semantic representations to strings; they also have a models for relating real world situations to semantics, models of likely real-world situations, models of other talkers, etc.. Probability theory provides a natural and mathematically well-grounded language for expressing and testing potential models, and information theory is so profoundly tied up in probability theory that probabilistically sensible behavior is bound to be also information-theoretically sensible.

I think you are basically agreeing with me that the meaning of linguistic expressions requires context in order for a listener to discover it, but the information-theoretic approach is basically about signal processing, not meaning comprehension.  To understand an expression is essentially to integrate it with one's experiences--what you might call a "world model".  I'm not saying that the problem is computationally impossible, but that it involves a lot more than mere signal processing.  Probabilistic approaches work very well on a gross level for disambiguating word senses primarily because mutual information is an extremely powerful concept.  I think that they have proven their worth in applications such as text mining large amounts of data.

Maybe probability theory will end up being inadequate, but opponents of the code view are going to need to make much more specific criticisms to convince me.

How does probability theory actually help you generate linguistic structure?  There are two sides to language--production and comprehension.  The best that probability theory can do for you is provide you with a cloud of more or less related words.  How do you assemble those words into structured phrases that can be understood in a given context?   Where does probability help you to decide the quantifier scope?  You can extract meaning from clouds of words in a document, but constructing the document requires knowledge about how to structure the information for a discourse context.  Probabilistic approaches help with certain types of linguistic processing, but they are a dead end when it comes to real text understanding.  In fact, I don't think any approach that relies just on signal processing is scalable.  However, the conduit metaphor is well-ensconced in our thinking about language, so signal processing approaches sound very promising at first blush.

Incidentally, the compositionality of natural language is actually an argument against the view that natural language achieves an information-theoretic optimum. Briefly, if language is compositional, then the length of the signal for a message is equal to the sum of the lengths of each component of the message:



It turns out that this summation over lengths corresponds to an assumption that the message components are statistically independent from each other (more discussion and derivation at the above link to my blog). In natural language, of course, what we usually consider to be components (words and constructions) are not statistically independent -- a sentence that mention scrambled eggs is more likely to mention orange juice or coffee. Non-compositional language phenomena provide an opportunity to provide much shorter signals than would be possible in an exclusively compositional approach (think of the difference in length between "United States of America" and "USA"). So, the question then is whether talkers choose non-compositional forms in a way that moves language closer to the information-theoretic optimum, and there's some evidence that they do (e.g. Frank and Jaeger, 2008.).

If your assumption is that all of the information necessary to decode the signal is in the signal "pipeline", then you are right about that length metric.  I do not believe that that assumption is correct.  The expression "scrambled eggs" represents a very complex web of associations.  The trick is to get the right set of associations in the given discourse context.  If you base your notion of "discourse context" on just the literal meanings of the word cloud, you are going to miss critical information that relates to an analogical mapping.  That is, you aren't going to be able to handle a fundamental aspect of human language--metaphor.

It seems to me that the idea of "information" in the signal is at least trivially true: something is transmitted and we can quantify that in terms of information. I believe jkpate's point is that we can theoretically optimize over that information at various levels including context. There are some questions of what the best way is, but I don't see a problem with the basic point.

I'm not really attacking information theory per se.  From my perspective, the problem is that "information" in "information theory" is not the same thing as "meaning" in linguistic semantics.  The criticism that Michael Reddy originally made regarding the conduit metaphor was that much of what a sentence means is not actually contained in the signal.  It is not just a matter of decoding a signal.  Rather, it is "constructed" or inferred from that information (in an information-theoretic sense of "information").  Reddy talks about a "toolmakers paradigm" as an alternative metaphor.  His seminal work actually had a very powerful influence on the development of the rather eclectic school of Cognitive Linguistics, which grounds meaning in embodied cognition.