Lecture 7 - Plunkett

---

Simple Associative Chains - Sequences which involve state transition.

a --> b --> c

a --> c --> b

Thus individual states describe actions to be performed.

Repetition - Need to re-enter states

a --> a --> b --> c

does not capture state identities, so...

ab --> c

Complex Sequences - Unilimited sequence production.

abca,b, or c

Such a fully recurrent network is good but because probabilistic Lashley (1951) notes that there is a lack of control. (Problem of the serial order of behavior.)

Recurrent Networks

Consider the following network. If we let A = 1 1, and B = 0 0, then...

This network will produce the sequence AAAB. How? The recurrent connections keep track of the previous state of the network. Allowing the network to keep track of the number of A's it previously outputed. Still, such a network only implements one sequence. How do we get it to implement a plan?

Turns out with the addition of some plan units you have now created a network which can implement virutally any combination of sequences. How? The input of the plan units trigger a diffenent sequence depending on the input.

Simple Recurrent Networks

Elman (1990) was the first to train a network, by giving the hidden units recurrent connections. In this manner he gave the network a kind of memory.

In this manner, he trained the network to predict a simple grammar. Turned out, as one might expect, the network was pretty bad, particularly as more and more words were presented.
     However, turns out that the hidden units were actually quite good a predicting nouns or verbs, and even types of nouns and verbs (like animal or human, transitive or intransitive.) Thus, from limited representational space the network had abstracted quite abstract and very interesting properties. How?
     Well, you might think it had used the meanings of the words, but it can't have, because it has no representation of meaning. What it did do, however, was to extract the distributional probabilities of the sentences (much like classical linguistics), and actually find and predict the parts of speech were coming?