Nonlinear System Identification: From Classical Approaches to Neural Networks and Fuzzy Models

Front Cover
Springer Science & Business Media, 2001 - Computers - 785 pages
The goal of this book is to provide engineers and scientIsts in academia and industry with a thorough understanding of the underlying principles of nonlinear system identification. The reader will be able to apply the discussed models and methods to real problems with the necessary confidence and the awareness of potential difficulties that may arise in practice. This book is self-contained in the sense that it requires merely basic knowledge of matrix algebra, signals and systems, and statistics. Therefore, it also serves as an introduction to linear system identification and gives a practical overview on the major optimization methods used in engineering. The emphasis of this book is on an intuitive understanding of the subject and the practical application of the discussed techniques. It is not written in a theorem/proof style; rather the mathematics is kept to a minimum and the pursued ideas are illustrated by numerous figures, examples, and real-world applications. Fifteen years ago, nonlinear system identification was a field of several ad-hoc approaches, each applicable only to a very restricted class of systems. With the advent of neural networks, fuzzy models, and modern structure opti mization techniques a much wider class of systems can be handled. Although one major characteristic of nonlinear systems is that almost every nonlinear system is unique, tools have been developed that allow the use of the same ap proach for a broad variety of systems.
 

What people are saying - Write a review

User Review - Flag as inappropriate

nonlinear system identification

User Review - Flag as inappropriate

very nice book for the engineers who want to learn system identification.

Contents

1 Introduction
xvii
113 Simulation
1
114 Optimization
2
117 Fault Detection
3
12 Tasks in Nonlinear System Identification
4
121 Choice of the Model Inputs
6
122 Choice of the Excitation Signals
7
123 Choice of the Model Architecture
8
1332 Different Objectives for Structure and Parameter Optimization
370
1333 Smoothness Optimization
372
1334 Splitting Ratio Optimization
374
1335 Merging of Local Models
376
1336 Flat and Hierarchical Model Structures
378
1337 Principal Component Analysis for Preprocessing
381
1338 Models with Multiple Outputs
383
134 Summary
387

124 Choice of the Dynamics Representation
9
127 Choice of the Model Parameters
10
128 Model Validation
11
13 White Box Black Box and Gray Box Models
13
14 Outline of the Book and Some Reading Suggestions
14
15 Terminology
16
Optimization Techniques
19
2 Introduction to Optimization
21
21 Overview of Optimization Techniques
23
23 Loss Functions for Supervised Methods
26
231 Maximum Likelihood Method
28
232 Maximum APosteriori and Bayes Method
30
24 Loss Functions for Unsupervised Methods
32
3 Linear Optimization
33
31 Least Squares LS
34
311 Covariance Matrix of the Parameter Estimate
42
312 Errorbars
43
313 Orthogonal Regressors
46
314 Regularization Ridge Regression
47
315 Noise Assumptions
52
316 Weighted Least Squares WLS
53
317 Least Squares with Equality Constraints
55
318 Smoothing Kernels
56
32 Recursive Least Squares RLS
58
321 Reducing the Computational Complexity
61
322 Tracking TimeVariant Processes
62
323 Relationship between the RLS and the Kalman Filter
63
33 Linear Optimization with Inequality Constraints
64
34 Subset Selection
65
341 Methods for Subset Selection
66
342 Orthogonal Least Squares OLS for Forward Selection
70
343 Ridge Regression or Subset Selection?
73
35 Summary
75
4 Nonlinear Local Optimization
77
41 Batch and Sample Adaptation
79
42 Initial Parameters
81
43 Direct Search Algorithms
84
432 HookeJeeves Method
86
44 General GradientBased Algorithms
88
441 Line Search
89
442 Finite Difference Techniques
90
443 Steepest Descent
91
444 Newtons Method
94
445 QuasiNewton Methods
96
446 Conjugate Gradient Methods
98
45 Nonlinear Least Squares Problems
100
451 GaussNewton Method
102
452 LevenbergMarquardt Method
103
46 Constrained Nonlinear Optimization
105
47 Summary
108
5 Nonlinear Global Optimization
111
51 Simulated Annealing SA
114
52 Evolutionary Algorithms EA
118
521 Evolution Strategies ES
121
522 Genetic Algorithms GA
124
523 Genetic Programming GP
130
53 Branch and Bound BB
131
54 Tabu Search TS
133
6 Unsupervised Learning Techniques
135
61 Principal Component Analysis PCA
137
62 Clustering Techniques
140
621 KMeans Algorithm
141
622 Fuzzy CMeans FCM Algorithm
144
623 GustafsonKessel Algorithm
146
624 Kohonens SelfOrganizing Map SOM
147
625 Neural Gas Network
150
626 Adaptive Resonance Theory ART Network
151
627 Incorporating Information about the Output
152
63 Summary
153
7 Model Complexity Optimization
155
721 Bias Error
158
722 Variance Error
159
723 Tradeoff
162
73 Evaluating the Test Error and Alternatives
165
731 Training Validation and Test Data
166
732 Cross Validation
167
733 Information Criteria
169
734 MultiObjective Optimization
170
735 Statistical Tests
172
736 CorrelationBased Methods
174
Implicit Structure Optimization
177
752 Regularization by NonSmoothness Penalties
178
753 Regularization by Early Stopping
180
754 Regularization by Constraints
182
755 Regularization by Staggered Optimization
184
756 Regularization by Local Optimization
185
76 Structured Models for Complexity Reduction
187
761 Curse of Dimensionality
188
762 Hybrid Structures
190
763 ProjectionBased Structures
193
764 Additive Structures
194
765 Hierarchical Structures
195
766 Input Space Decomposition with Tree Structures
196
77 Summary
198
8 Summary of Part I
201
Static Models
205
9 Introduction to Static Models
207
921 Global and Local Basis Functions
209
922 Linear and Nonlinear Parameters
210
93 Extended Basis Function Formulation
213
94 Static Test Process
214
10 Linear Polynomial and LookUp Table Models
217
102 Polynomial Models
219
103 LookUp Table Models
222
1031 OneDimensional LookUp Tables
223
1032 TwoDimensional LookUp Tables
225
1033 Optimization of the Heights
227
1034 Optimization of the Grid
229
1035 Optimization of the Complete LookUp Table
230
1037 Properties of LookUp Table Models
233
104 Summary
235
11 Neural Networks
237
111 Construction Mechanisms
240
1112 Radial Construction
242
1113 Tensor Product Construction
243
112 Multilayer Perceptron MLP Network
244
1121 MLP Neuron
245
1122 Network Structure
247
1123 Backpropagation
250
1124 MLP Training
251
1125 Simulation Examples
254
1126 MLP Properties
258
1127 Multiple Hidden Layers
259
1128 Projection Pursuit Regression PPR
260
113 Radial Basis Function RBF Networks
262
1132 Network Structure
265
1133 RBF Training
267
1134 Simulation Examples
275
1135 RBF Properties
277
1136 Regularization Theory
279
1137 Normalized Radial Basis Function NRBF Networks
281
114 Other Neural Networks
284
1142 Cerebellar Model Articulation Controller CMAC
286
1143 Delaunay Networks
290
1144 JustinTime Models
291
115 Summary
294
12 Fuzzy and NeuroFuzzy Models
297
1212 Logic Operators
300
1213 Rule Fulfillment
301
122 Types of Fuzzy Systems
302
1222 Singleton Fuzzy Systems
305
1223 TakagiSugeno Fuzzy Systems
307
123 NeuroFuzzy NF Networks
308
1231 Fuzzy Basis Functions
309
1232 Equivalence between RBF Networks and Fuzzy Models
310
1233 What to Optimize?
311
1234 Interpretation of NeuroFuzzy Networks
314
1235 Incorporating and Preserving Prior Knowledge
318
1236 Simulation Examples
319
124 NeuroFuzzy Learning Schemes
321
1242 Nonlinear Global Optimization
323
1244 Fuzzy Rule Extraction by a Genetic Algorithm FUREGA
325
1245 Adaptive Spline Modeling of Observation Data ASMOD
335
125 Summary
337
Fundamentals
339
131 Basic Ideas
340
1311 Illustration of Local Linear NeuroFuzzy Models
341
1312 Interpretation of the Local Linear Model Offsets
344
1313 Interpretation as TakagiSugeno Fuzzy System
345
1314 Interpretation as Extended NRBF Network
347
132 Parameter Optimization of the Rule Consequents
349
1322 Local Estimation
350
1323 Global Versus Local Estimation
354
1324 Data Weighting
359
133 Structure Optimization of the Rule Premises
360
1331 Local Linear Model Tree LOLIMOT Algorithm
363
Advanced Aspects
389
1411 Identification of Processes with Direction Dependent Behavior
393
142 More Complex Local Models
395
1422 Local Quadratic Models for Input Optimization
398
1423 Different Types of Local Models
400
143 Structure Optimization of the Rule Consequents
402
144 Interpolation and Extrapolation Behavior
406
1442 Extrapolation Behavior
409
145 Global and Local Linearization
414
146 Online Learning
418
1461 Online Adaptation of the Rule Consequents
419
1462 Online Construction of the Rule Premise Structure
426
147 Errorbars Design of Excitation Signals and Active Learning
428
1471 Errorbars
429
1472 Detecting Extrapolation
432
1473 Design of Excitation Signals
433
1474 Active Learning
434
148 From Local Linear NeuroFuzzy Models to Hinging Hyperplanes
435
1481 Hinging Hyperplanes
436
1482 Smooth Hinging Hyperplanes
437
1483 Hinging Hyperplane Trees HHT
439
1484 Local Linear NeuroFuzzy Models Versus Hinging Hyperplane Trees
441
149 Summary and Conclusions
442
15 Summary of Part II
449
Dynamic Models
453
16 Linear Dynamic System Identification
455
162 Excitation Signals
457
163 General Model Structure
460
1631 Terminology and Classification
463
1632 Optimal Predictor
469
1633 Some Remarks on the Optimal Predictor
472
1634 Prediction Error Methods
474
164 Time Series Models
476
1641 Autoregressive AR
477
1642 Moving Average MA
478
1643 Autoregressive Moving Average ARMA
479
165 Models with Output Feedback
480
1652 Autoregressive Moving Average with Exogenous Input ARMAX
490
1653 Autoregressive Autoregressive with Exogenous Input ARARX
494
1654 Output Error OE
497
1655 BoxJenkins BJ
501
1656 State Space Models
503
1657 Simulation Example
504
166 Models without Output Feedback
507
1661 Finite Impulse Response FIR
508
1662 Orthonormal Basis Functions OBF
510
1663 Simulation Example
518
167 Some Advanced Aspects
522
1672 Consistency
524
1674 Relationship between Noise Model and Filtering
526
1675 Offsets
527
168 Recursive Algorithms
529
1681 Recursive Least Squares RLS Method
530
1683 Recursive Extended Least Squares RELS Method
531
1684 Recursive Prediction Error Methods RPEM
532
169 Determination of Dynamic Orders
534
1610 Multivariable Systems
535
16101 PCanonical Model
537
16102 Matrix Polynomial Model
538
16103 Subspace Methods
539
16111 Direct Methods
540
16112 Indirect Methods
542
16113 Identification for Control
543
1612 Summary
544
17 Nonlinear Dynamic System Identification
545
172 External Dynamics
547
1721 Illustration of the External Dynamics Approach
548
1722 SeriesParallel and Parallel Models
553
1723 Nonlinear Dynamic InputOutput Model Classes
555
1724 Restrictions of Nonlinear Dynamic InputOutput Models
560
173 Internal Dynamics
561
174 Parameter Scheduling Approach
562
1751 BackpropagationThroughTime BPTT Algorithm
563
1752 Real Time Recurrent Learning
565
176 Multivariable Systems
566
177 Excitation Signals
567
178 Determination of Dynamic Orders
572
179 Summary
574
18 Classical Polynomial Approaches
577
182 KolmogorovGabor Polynomial Models
579
183 VolterraSeries Models
580
184 Parametric VolterraSeries Models
581
186 Hammerstein Models
582
187 Wiener Models
583
19 Dynamic Neural and Fuzzy Models
585
1911 MLP Networks
586
192 Interpolation and Extrapolation Behavior
587
193 Training
589
1931 MLP Networks
590
194 Integration of a Linear Model
591
195 Simulation Examples
592
1951 MLP Networks
593
1952 RBF Networks
595
1953 Singleton Fuzzy and NRBF Models
597
196 Summary
598
20 Dynamic Local Linear NeuroFuzzy Models
599
201 OneStep Prediction Error Versus Simulation Error
602
202 Determination of the Rule Premises
604
203 Linearization
606
2032 Dynamics of the Linearized Model
608
2033 Different Rule Consequent Structures
610
204 Model Stability
611
2041 Influence of Rule Premise Inputs on Stability
612
2042 Lyapunov Stability and Linear Matrix Inequalities LMIs
614
2043 Ensuring Stable Extrapolation
615
205 Dynamic LOLIMOT Simulation Studies
616
2052 Hammerstein Process
618
2053 Wiener Process
622
2054 NDE Process
623
206 Advanced Local Linear Methods and Models
624
2061 Local Linear Instrumental Variables IV Method
626
2062 Local Linear Output Error OE Models
628
2063 Local Linear ARMAX Models
629
208 Structure Optimization of the Rule Consequents
634
209 Summary and Conclusions
638
21 Neural Networks with Internal Dynamics
643
212 Partially Recurrent Networks
644
213 State Recurrent Networks
645
214 Locally Recurrent Globally Feedforward Networks
646
215 Internal Versus External Dynamics
648
Applications
651
22 Applications of Static Models
653
2212 Smoothing of a Driving Cycle
655
2213 Improvements and Extensions
656
2214 Differentiation
657
2221 The Role of LookUp Tables in Automotive Electronics
658
2222 Modeling of Exhaust Gases
661
2223 Optimization of Exhaust Gases
664
Dynamic Models
670
223 Summary
672
23 Applications of Dynamic Models
675
2312 Experimental Results
677
232 Diesel Engine Turbocharger
681
2321 Process Description
682
2322 Experimental Results
683
233 Thermal Plant
689
2331 Process Description
690
2332 Transport Process
691
2333 Tubular Heat Exchanger
696
2334 CrossFlow Heat Exchanger
700
234 Summary
705
24 Applications of Advanced Methods
707
242 Online Adaptation
711
2421 Variable Forgetting Factor
712
2422 Control and Adaptation Models
713
2423 Parameter Transfer
715
2424 Systems with Multiple Inputs
716
2425 Experimental Results
717
243 Fault Detection
721
2432 Experimental Results
724
244 Fault Diagnosis
727
2442 Experimental Results
729
245 Reconfiguration
730
A Vectors and Matrices
733
A2 Gradient Hessian and Jacobian
735
B Statistics
737
B2 Probability Density Function pdf
739
B3 Stochastic Processes and Ergodicity
741
B4 Expectation
743
B5 Variance
746
B6 Correlation and Covariance
747
B7 Properties of Estimators
751
References
755
Index
777
Copyright

Other editions - View all

Common terms and phrases

Popular passages

Page 772 - Locally recurrent globally feedforward networks: a critical review of architectures", IEEE Transactions on Neural Networks, vol. 5, no. 2, 229239, March 1994.
Page 774 - Performance Evaluation of a Sequential Minimal Radial Basis Function (RBF) Neural Network Learning Algorithm", IEEE Transactions on Neural Networks, Vol.