Markov Models & Optimization

Front Cover
CRC Press, Aug 1, 1993 - Mathematics - 308 pages
This book presents a radically new approach to problems of evaluating and optimizing the performance of continuous-time stochastic systems. This approach is based on the use of a family of Markov processes called Piecewise-Deterministic Processes (PDPs) as a general class of stochastic system models. A PDP is a Markov process that follows deterministic trajectories between random jumps, the latter occurring either spontaneously, in a Poisson-like fashion, or when the process hits the boundary of its state space. This formulation includes an enormous variety of applied problems in engineering, operations research, management science and economics as special cases; examples include queueing systems, stochastic scheduling, inventory control, resource allocation problems, optimal planning of production or exploitation of renewable or non-renewable resources, insurance analysis, fault detection in process systems, and tracking of maneuvering targets, among many others.

The first part of the book shows how these applications lead to the PDP as a system model, and the main properties of PDPs are derived. There is particular emphasis on the so-called extended generator of the process, which gives a general method for calculating expectations and distributions of system performance functions. The second half of the book is devoted to control theory for PDPs, with a view to controlling PDP models for optimal performance: characterizations are obtained of optimal strategies both for continuously-acting controllers and for control by intervention (impulse control). Throughout the book, modern methods of stochastic analysis are used, but all the necessary theory is developed from scratch and presented in a self-contained way. The book will be useful to engineers and scientists in the application areas as well as to mathematicians interested in applications of stochastic analysis.
 

What people are saying - Write a review

We haven't found any reviews in the usual places.

Contents

Analysis probability and stochastic processes
1
Analysis
2
Probability theory
8
Stochastic processes
16
Markov processes
23
Notes and references
33
Piecewisedeterministic Markov processes
35
Markov models and supplementary variables
36
E BarndorffNielsen W S Kendall and M N M van Lieshout 1998
81
The differential formula and transformations of PDPs
82
Expectations
92
Subset Selection in Regression Second Edition Alan Miller 2002
96
Applications
101
Stationary distributions
111
Notes and references
132
Control theory
134

Generalized Linear Models 2nd edition P McCullagh and J A Nelder 1989
38
Ordinary differential equations and vector fields
52
Simulation
55
Definition of the PDP
57
The strong Markov property
62
Measurement Error in Nonlinear Models
63
The extended generator of the PDP
66
Further Markov properties of the PDP
74
Notes and references
79
Feedback control of PDPs
135
Na´ve dynamic programming
139
Relaxed controls
147
Control via discretetime dynamic programming
151
Control by intervention
186
Jump processes and their martingales
256
Bibliography
280
Index of notation
287
Copyright

Other editions - View all

Common terms and phrases

Popular passages

Page iii - Department of Electrical and Electronic Engineering Imperial College of Science, Technology and Medicine Exhibition Road, London, SW7 2BT, United Kingdom Abstract In this paper we develop a notion of "objective trust" for Software Agents, that is trust of, or between, Agents based on actual experiences between those Agents.