## Adaptive Learning of Polynomial Networks: Genetic Programming, Backpropagation and Bayesian MethodsThis book provides theoretical and practical knowledge for develop ment of algorithms that infer linear and nonlinear models. It offers a methodology for inductive learning of polynomial neural network mod els from data. The design of such tools contributes to better statistical data modelling when addressing tasks from various areas like system identification, chaotic time-series prediction, financial forecasting and data mining. The main claim is that the model identification process involves several equally important steps: finding the model structure, estimating the model weight parameters, and tuning these weights with respect to the adopted assumptions about the underlying data distrib ution. When the learning process is organized according to these steps, performed together one after the other or separately, one may expect to discover models that generalize well (that is, predict well). The book off'ers statisticians a shift in focus from the standard f- ear models toward highly nonlinear models that can be found by con temporary learning approaches. Speciafists in statistical learning will read about alternative probabilistic search algorithms that discover the model architecture, and neural network training techniques that identify accurate polynomial weights. They wfil be pleased to find out that the discovered models can be easily interpreted, and these models assume statistical diagnosis by standard statistical means. Covering the three fields of: evolutionary computation, neural net works and Bayesian inference, orients the book to a large audience of researchers and practitioners. |

### What people are saying - Write a review

We haven't found any reviews in the usual places.

### Contents

1 | |

3 | |

4 | |

5 | |

12 Why Polynomial Networks? | 7 |

121 Advantages of Polynomial Networks | 8 |

122 Multilayer Polynomial Networks | 9 |

13 Evolutionary Search | 16 |

623 Batch Backpropagation | 157 |

624 Incremental Backpropagation | 158 |

625 Control of the Learning Step | 159 |

626 Regularized Delta Rule | 162 |

63 SecondOrder Backpropagation | 163 |

631 SecondOrder Error Derivatives | 164 |

632 Newtons Method | 169 |

633 PseudoNewton Training | 170 |

131 STROGANOFF and its Variants | 17 |

14 Neural Network Training | 21 |

15 Bayesian Inference | 22 |

16 Statistical Model Validation | 23 |

INDUCTIVE GENETIC PROGRAMMING | 25 |

21 Polynomial Neural Networks PNN | 26 |

211 PNN Approaches | 27 |

212 Treestructured PNN | 29 |

22 IGP Search Mechanisms | 35 |

221 Sampling and Control Issues | 36 |

23 Genetic Learning Operators | 38 |

232 Crossover Operator | 40 |

233 Sizebiasing of the Genetic Operators | 41 |

234 TreetoTree Distance | 42 |

24 Random Tree Generation | 46 |

25 Basic IGP Framework | 48 |

26 IGP Convergence Characteristics | 50 |

262 Markov Model of IGP | 51 |

27 Chapter Summary | 54 |

TREELIKE PNN REPRESENTATIONS | 55 |

31 Discrete Volterra Series | 56 |

32 Mapping Capabilities of PNN | 57 |

33 Errors of Approximation | 59 |

332 Empirical Risk | 60 |

34 Linear Polynomial Networks | 62 |

342 Kernel PNN Models | 66 |

35 Nonlinear Polynomial Networks | 68 |

352 Orthogonal PNN Models | 69 |

353 Trigonometric PNN Models | 71 |

354 Rational PNN Models | 75 |

355 Dynamic PNN Models | 77 |

36 Chapter Summary | 80 |

FITNESS FUNCTIONS AND LANDSCAPES | 81 |

41 Fitness Functions | 83 |

411 Static Fitness Functions | 84 |

412 Dynamic Fitness Functions | 91 |

413 Fitness Magnitude | 94 |

42 Fitness Landscape Structure | 95 |

43 Fitness Landscape Measures | 96 |

432 Probabilistic Measures | 102 |

433 Information Measures | 104 |

434 Quantitative Measures | 107 |

44 Chapter Summary | 109 |

SEARCH NAVIGATION | 111 |

51 The Reproduction Operator | 112 |

511 Selection Strategies | 113 |

512 Replacement Strategies | 117 |

513 Implementing Reproduction | 118 |

52 Advanced Search Control | 119 |

522 Memetic Search | 120 |

523 Search by Genetic Annealing | 122 |

524 Stochastic Genetic Hillclimbing | 124 |

525 Coevolutionary Search | 125 |

526 Distributed Search | 128 |

531 Fitness Evolvability | 129 |

532 Convergence Measures | 130 |

533 Diversity Measures | 133 |

534 Measures of SelfOrganization | 139 |

54 Chapter Summary | 146 |

BACKPROPAGATION TECHNIQUES | 147 |

61 Multilayer Feedforward PNN | 148 |

62 FirstOrder Backpropagation | 149 |

621 Gradient Descent Search | 150 |

622 FirstOrder Error Derivatives | 151 |

635 LevenbergMarquardt Method | 171 |

64 Rational Backpropagation | 172 |

65 Network Pruning | 176 |

652 SecondOrder Network Pruning | 177 |

66 Chapter Summary | 179 |

TEMPORAL BACKPROPAGATION | 181 |

71 Recurrent PNN as StateSpace Models | 182 |

72 Backpropagation Through Time | 184 |

721 Derivation of BPTT Algorithms | 185 |

722 RealTime BPTT Algorithm | 189 |

723 Epochwise BPTT Algorithm | 190 |

73 RealTime Recurrent Learning | 191 |

74 Improved Dynamic Training | 198 |

742 Truncating in Time | 199 |

744 Common Temporal Training Problem | 200 |

76 Recursive Backpropagation | 204 |

77 Recurrent Network Optimization | 206 |

771 Regularization | 207 |

78 Chapter Summary | 208 |

BAYESIAN INFERENCE TECHNIQUES | 209 |

81 Bayesian Error Function | 211 |

82 Bayesian Neural Network Inference | 212 |

821 Deriving Hyperparameters | 215 |

822 Local vs Global Regularization | 217 |

823 Evidence Procedure for PNN Models | 218 |

824 Predictive Data Distribution | 221 |

83 Bayesian Network Pruning | 222 |

84 Sparse Bayesian Learning | 224 |

85 Recursive Bayesian Learning | 229 |

852 Sequential Dynamic Hessian Estimation | 230 |

853 Sequential Hyperparameter Estimation | 232 |

86 Monte Carlo Training | 234 |

861 Markov Chain Monte Carlo | 235 |

862 Importance Resampling | 237 |

87 Chapter Summary | 239 |

STATISTICAL MODEL DIAGNOSTICS | 241 |

91 Deviations of PNN Models | 242 |

92 Residual Bootstrap Sampling | 243 |

93 The BiasVariance Dilemma | 244 |

932 Measuring Bias and Variance | 245 |

94 Confidence Intervals | 248 |

942 Bootstrapping Confidence Intervals | 252 |

95 Prediction Intervals | 254 |

951 Analytical Prediction Intervals | 255 |

952 Empirical Learning of Prediction Bars | 256 |

96 Bayesian Intervals | 262 |

961 Analytical Bayesian Intervals | 263 |

962 Empirical Bayesian Intervals | 265 |

97 Model Validation Tests | 267 |

98 Chapter Summary | 271 |

TIME SERIES MODELLING | 273 |

101 Time Series Modelling | 274 |

102 Data Preprocessing | 276 |

103 PNN vs Linear ARMA Models | 277 |

104 PNN vs Genetically Programmed Functions | 279 |

105 PNN vs Statistical Learning Networks | 281 |

106 PNN vs Neural Network Models | 283 |

107 PNN vs Kernel Models | 285 |

108 Recurrent PNN vs Recurrent Neural Networks | 288 |

109 Chapter Summary | 290 |

CONCLUSIONS | 291 |

295 | |

312 | |

### Other editions - View all

Adaptive Learning of Polynomial Networks: Genetic Programming ... Nikolay Nikolaev,Hitoshi Iba No preview available - 2006 |

Adaptive Learning of Polynomial Networks: Genetic Programming ... Nikolay Nikolaev,Hitoshi Iba No preview available - 2011 |

### Common terms and phrases

activation polynomials approach approximation backprop error backpropagation Bayesian inference BPTT coefficients complexity computed confidence intervals convergence crossover delta method delta rule distribution dynamic equation error derivatives error function error surface estimated evaluated evolutionary search Figure first-order fitness function fitness landscape formula functional nodes genetic programs genotype global GMDH gradient descent Hessian matrix hidden nodes hyperparameters IGP system implemented incremental individuals inductive learning initial input variables Ivakhnenko kernel large number layer learning rate least squares linear Mackey-Glass measures method multilayer mutation operators network weights neural network node outputs noise nonlinear offspring optimal output node overfitting parameters performance PNN models polynomial networks prediction intervals probabilistic random recurrent PNN regularization root RTRL sampling search space second-order selection statistical step Algorithmic sequence structure subtree techniques temporal tion training algorithm training data tree tree traversal tree-like PNN tree-structured tree-to-tree distance update variance weight posterior weight vector