Lecture 11 - Edward Rolls

---

Memory and the Dynamics of Autoassociators

Autoassociators are recurrent collateral networks in which the output feeds backwards onto itself. As a result, it can learn and represent the pattern of inputs presented it, even from a very small sub-fragment of the original memory presented to it. In this manner autoassociators of this sort are said to be content addressable.
     The learning rule is just a simple Hebb rule which changes the weight of a synapse based on the activity of the pre-and post synaaptic neurons. Interestingly, the arcitecture of this network is such that the connection weights will be symetric across the diagonal. Also, because of the architecture, there will be runaway feedback, unless their is some sort of threshold function. Thus, such a network MUST have a non-linear activation function.
   So, to make a "few" points...

• with complete connectivity the weights will be symetric
• "one shot" learning
• get feedback without threshold function
• content addressable
• shows completion
• generalisation
• graceful degradation or fault tolerance
• prototype extraction (depending on how teach it. If exemplears are very close to the prototype then yes, but if exemplars are very different then no.)
• speed (yes, these networks require several times around to get accurate recall, 20msX10, using normal networks, but if use "integrate anf fire" neurons, ie, the membrane is acting as a continous analog capacitor, than such networks are much faster. Although interestingly, the synaptic time constant is the only thing that effect the speed of such networks.)
• Local learning rule
• Context effects (since memorizes everything to do with a pattern, context will also effect the recall of the network.
• Mixture states (the network can't learn pattern which overlap, mixtures, but expansion encoding will take care of this problem. Indeed, within the hypicampus, it appears the dentate granules do this.
• Memory for sequences cna only be accomplished by introducin delay in the network.
• Utility - these networks are good for episodic memory and as a kind of short term memory.

Hopfield investigated such networks extensively. Found that the final energy set of the system will either be consistent with each other if the connections were the same (and fall into a stable state), OR frustrated if the conections where opposite. From this, Hopfield not only showed that these networks will be stable if they are symetric, but he also calulated the number of possible stable states per network. Like the pattern associator, the capacity is determined by the number of synapses per neuron (from 12,000 - 50,000 in the rat), divided by the "sparseness" factor. Again, the sparser the representation, the better.

Coding in AutoAssociator Networks

Generalisation, completion, and graceful degradation only come with a sparse or fully distributed networks. Local and fully distributed networks can only store the number of patterns equal to the number of inputs. Sparse can be much larger. In addition, as one moves from local to fully distributed the amount of information in each pattern increases. Thus, fully distributed networks can hold the most, BUT at the greatest cost to efficency.

Biological Plausibility and Things to Explain

• Low connectivity (sparse coding is best)
• How to force learning of a new attractor. (CA3 Mossy fibres synapse very close to the cell body, thus stronger than the reccurent feedback)
• How to trigger retrival (perforant path fo CA3 neurons)
• Operate in a clamped condition (need not opperate as a stable attractor to perform information retrival)
• may need to expand overlapping patterns using preprocessor so as to avoid the overlapp
• dynamics: must operate in 1-2 time constants of the neurons
• multistage pattern associator can be analyzed as an auto associator
• graded versus binary firing rates (information storage is independent of the representation, more patterns if bianary, BUT more information if graded.)