Adaptive Learning of Polynomial Networks: Genetic Programming, Backpropagation and Bayesian Methods

Front Cover
Springer Science & Business Media, Aug 18, 2006 - Computers - 316 pages
0 Reviews
This book provides theoretical and practical knowledge for develop ment of algorithms that infer linear and nonlinear models. It offers a methodology for inductive learning of polynomial neural network mod els from data. The design of such tools contributes to better statistical data modelling when addressing tasks from various areas like system identification, chaotic time-series prediction, financial forecasting and data mining. The main claim is that the model identification process involves several equally important steps: finding the model structure, estimating the model weight parameters, and tuning these weights with respect to the adopted assumptions about the underlying data distrib ution. When the learning process is organized according to these steps, performed together one after the other or separately, one may expect to discover models that generalize well (that is, predict well). The book off'ers statisticians a shift in focus from the standard f- ear models toward highly nonlinear models that can be found by con temporary learning approaches. Speciafists in statistical learning will read about alternative probabilistic search algorithms that discover the model architecture, and neural network training techniques that identify accurate polynomial weights. They wfil be pleased to find out that the discovered models can be easily interpreted, and these models assume statistical diagnosis by standard statistical means. Covering the three fields of: evolutionary computation, neural net works and Bayesian inference, orients the book to a large audience of researchers and practitioners.
 

What people are saying - Write a review

We haven't found any reviews in the usual places.

Contents

INTRODUCTION
1
11 Inductive Learning
3
111 Learning and Regression
4
112 Polynomial Models
5
12 Why Polynomial Networks?
7
121 Advantages of Polynomial Networks
8
122 Multilayer Polynomial Networks
9
13 Evolutionary Search
16
623 Batch Backpropagation
157
624 Incremental Backpropagation
158
625 Control of the Learning Step
159
626 Regularized Delta Rule
162
63 SecondOrder Backpropagation
163
631 SecondOrder Error Derivatives
164
632 Newtons Method
169
633 PseudoNewton Training
170

131 STROGANOFF and its Variants
17
14 Neural Network Training
21
15 Bayesian Inference
22
16 Statistical Model Validation
23
INDUCTIVE GENETIC PROGRAMMING
25
21 Polynomial Neural Networks PNN
26
211 PNN Approaches
27
212 Treestructured PNN
29
22 IGP Search Mechanisms
35
221 Sampling and Control Issues
36
23 Genetic Learning Operators
38
232 Crossover Operator
40
233 Sizebiasing of the Genetic Operators
41
234 TreetoTree Distance
42
24 Random Tree Generation
46
25 Basic IGP Framework
48
26 IGP Convergence Characteristics
50
262 Markov Model of IGP
51
27 Chapter Summary
54
TREELIKE PNN REPRESENTATIONS
55
31 Discrete Volterra Series
56
32 Mapping Capabilities of PNN
57
33 Errors of Approximation
59
332 Empirical Risk
60
34 Linear Polynomial Networks
62
342 Kernel PNN Models
66
35 Nonlinear Polynomial Networks
68
352 Orthogonal PNN Models
69
353 Trigonometric PNN Models
71
354 Rational PNN Models
75
355 Dynamic PNN Models
77
36 Chapter Summary
80
FITNESS FUNCTIONS AND LANDSCAPES
81
41 Fitness Functions
83
411 Static Fitness Functions
84
412 Dynamic Fitness Functions
91
413 Fitness Magnitude
94
42 Fitness Landscape Structure
95
43 Fitness Landscape Measures
96
432 Probabilistic Measures
102
433 Information Measures
104
434 Quantitative Measures
107
44 Chapter Summary
109
SEARCH NAVIGATION
111
51 The Reproduction Operator
112
511 Selection Strategies
113
512 Replacement Strategies
117
513 Implementing Reproduction
118
52 Advanced Search Control
119
522 Memetic Search
120
523 Search by Genetic Annealing
122
524 Stochastic Genetic Hillclimbing
124
525 Coevolutionary Search
125
526 Distributed Search
128
531 Fitness Evolvability
129
532 Convergence Measures
130
533 Diversity Measures
133
534 Measures of SelfOrganization
139
54 Chapter Summary
146
BACKPROPAGATION TECHNIQUES
147
61 Multilayer Feedforward PNN
148
62 FirstOrder Backpropagation
149
621 Gradient Descent Search
150
622 FirstOrder Error Derivatives
151
635 LevenbergMarquardt Method
171
64 Rational Backpropagation
172
65 Network Pruning
176
652 SecondOrder Network Pruning
177
66 Chapter Summary
179
TEMPORAL BACKPROPAGATION
181
71 Recurrent PNN as StateSpace Models
182
72 Backpropagation Through Time
184
721 Derivation of BPTT Algorithms
185
722 RealTime BPTT Algorithm
189
723 Epochwise BPTT Algorithm
190
73 RealTime Recurrent Learning
191
74 Improved Dynamic Training
198
742 Truncating in Time
199
744 Common Temporal Training Problem
200
76 Recursive Backpropagation
204
77 Recurrent Network Optimization
206
771 Regularization
207
78 Chapter Summary
208
BAYESIAN INFERENCE TECHNIQUES
209
81 Bayesian Error Function
211
82 Bayesian Neural Network Inference
212
821 Deriving Hyperparameters
215
822 Local vs Global Regularization
217
823 Evidence Procedure for PNN Models
218
824 Predictive Data Distribution
221
83 Bayesian Network Pruning
222
84 Sparse Bayesian Learning
224
85 Recursive Bayesian Learning
229
852 Sequential Dynamic Hessian Estimation
230
853 Sequential Hyperparameter Estimation
232
86 Monte Carlo Training
234
861 Markov Chain Monte Carlo
235
862 Importance Resampling
237
87 Chapter Summary
239
STATISTICAL MODEL DIAGNOSTICS
241
91 Deviations of PNN Models
242
92 Residual Bootstrap Sampling
243
93 The BiasVariance Dilemma
244
932 Measuring Bias and Variance
245
94 Confidence Intervals
248
942 Bootstrapping Confidence Intervals
252
95 Prediction Intervals
254
951 Analytical Prediction Intervals
255
952 Empirical Learning of Prediction Bars
256
96 Bayesian Intervals
262
961 Analytical Bayesian Intervals
263
962 Empirical Bayesian Intervals
265
97 Model Validation Tests
267
98 Chapter Summary
271
TIME SERIES MODELLING
273
101 Time Series Modelling
274
102 Data Preprocessing
276
103 PNN vs Linear ARMA Models
277
104 PNN vs Genetically Programmed Functions
279
105 PNN vs Statistical Learning Networks
281
106 PNN vs Neural Network Models
283
107 PNN vs Kernel Models
285
108 Recurrent PNN vs Recurrent Neural Networks
288
109 Chapter Summary
290
CONCLUSIONS
291
References
295
Index
312
Copyright

Other editions - View all

Common terms and phrases

Bibliographic information