Statistical mechanics of learning from examples

摘要: Learning from examples in feedforward neural networks is studied within a statistical-mechanical framework. Training assumed to be stochastic, leading Gibbs distribution of characterized by temperature parameter T. realizable rules as well unrealizable considered. In the latter case, target rule cannot perfectly realized network given architecture. Two useful approximate theories learning are studied: high-temperature limit and annealed approximation. Exact treatment quenched disorder generated random sampling leads use replica theory. Of primary interest generalization curve, namely, average error ${\mathrm{\ensuremath{\epsilon}}}_{\mathit{g}}$ versus number P used for training. The theory implies that, reduction that remains finite large-N limit, should generally scale \ensuremath{\alpha}N, where N independently adjustable weights network. We show smooth networks, i.e., those with continuously varying transfer functions, curve asymptotically obeys an inverse power law. contrast, nonsmooth other behaviors can appear, depending on nature nonlinearities realizability rule. particular, discontinuous transition state poor perfect occur rules.We illustrate both gradual continuous detailed analytical numerical study several single-layer perceptron models. Comparing exact learning, we find provide very good approximations performance. Assuming this hold multilayer well, propose classification possible asymptotic forms curves general For above fail predict correctly shapes curves. Another indication important role not necessarily monotonically increasing function temperature. Also, possess genuine spin-glass phases indicative degenerate minima separated high barriers.