This chapter discusses several issues that are pertinent for the PSOM algorithm (which is described more fully in Chap. 4). Much of its motivation derives from the field of neural networks. After a brief historic overview of this rapidly expanding field we attempt to order some of the prominent network types in a taxonomy of important characteristics. We then proceed to discuss learning from the perspective of an approximation problem and identify several problems that are crucial for rapid learning. Finally we focus on the so-called “Self-Organizing Maps”, which emphasize the use of topology information for learning. Their discussion paves the way for Chap. 4 in which the PSOM algorithm will be presented.

In (1969) Minsky and Papert showed that certain classes of problems, e.g. the “exclusive-or” problem, cannot be learned with the simple perceptron. They doubted that learning rules could be found for computationally more powerful multi-layered networks and recommended to focus on the symbolic oriented learning paradigm, today called artificial intelligence (“AI”). The research funding for artificial neural networks was cut, and it took twenty years until the field became viable again. An important stimulus for the field was the multiple discovery of the error back-propagation algorithm. Its has been independently invented in several places, enabling iterative learning for multi-layer perceptrons (Werbos 1974, Rumelhart, Hinton, and Williams 1986, Parker 1985). The MLP turned out to be a universal approximator, which means that using a sufficient number of hidden units, any function can be approximated arbitrarily well. In general two hidden layers are required - for continuous functions one layer is sufficient (Cybenko 1989, Hornik et al. 1989). This property is of high theoretical value, but does not guarantee efficiency of any kind.

Other important developments where made: e.g. v.d. Malsburg and Willshaw (1977, 1973) modeled the ordered formation of connections between neuron layers in the brain. A strongly related, more formal algorithm was formulated by Kohonen for the development of a topographically ordered map from a general space of input stimuli to a layer of abstract neurons. We return to Kohonen's work later in Sec. 3.7. Hopfield (1982, 1984) contributed a famous model of the content-addressable

The Radial Basis Function Networks (“RBF”) became popular in the connectionist community by Moody and Darken (1988). The RFB belong to the class of local approximation schemes (see p. 33). Similarities and differences to other approaches are discussed in the next sections.