The Maximum Entropy Method

Nailong Wu

The Maximum Entropy Method With 53 Figures

Springer

Table of Contents

1.

Introduction 1.1 What is the Maximum Entropy Method 1.2 Definition of Entropy 1.3 Rationale of the Maximum Entropy Method 1.4 Present and Future Research

2.

Maximum Entropy Method M E M l and Its Application in Spectral Analysis 2.1 Definition and Expressions of Entropy HI 2.1.1 Approach 1 2.1.2 Approach 2 2.1.3 Discussion 2.2 Formulation and Solution 2.2.1 Formulation 2.2.2 Solution 2.2.3 Discussion 2.3 Equivalents and Signal Model 2.3.1 ACF Extension Subject to the Nonnegativity Constraint 2.3.2 Principle of MCE 2.3.3 AR Process (Signal Model) 2.3.4 Bayesian Method 2.3.5 Wiener Filter and Approximation Theoretic Approach 2.4 Algorithms and Numerical Example (Given ACF) 2.4.1 Levinson's Recursion for 1-D Noiseless Data 2.4.2 Lim-Malik Algorithm for 2-D Noiseless Data 2.4.3 Wernecke-D'Addario Algorithm for 2-D Noisy Data 2.4.4 Numerical Example 2.5 Algorithms and Numerical Example (Given Time Series) . . . . 2.5.1 Burg Algorithm 2.5.2 Marple Algorithm 2.5.3 Other Fast Algorithms 2.5.4 Numerical Example

1 1 4 6 9 15 15 16 20 23 25 25 25 32 34 34 38 44 48 50 51 53 57 62 67 67 68 72 89 91

X

Table of Contents 2.6 Order 2.6.1 2.6.2 2.6.3 2.6.4

3.

Selection FPE Criterion AIC Criterion Other Criteria Summary

Maximum Entropy Method MEM2 and Its Application in Image Restoration 3.1 Definition and Expressions of Entropy HI 3.1.1 MLM 3.1.2 Direct Definition Method 3.1.3 Discussion 3.2 Formulation and Implicit Solution 3.2.1 Formulation 3.2.2 Implicit Solution 3.2.3 Iterative Algorithm 3.2.4 Discussion 3.3 Explicit Solution 3.3.1 Explicit Solution 3.3.2 Discussion 3.3.3 Examples 3.4 Equivalents and Signal Model 3.4.1 ACF Extension Subject to the Nonnegativity Constraint 3.4.2 Principle of MCE 3.4.3 Exponential Process (Signal Model) 3.4.4 Bayesian Method 3.4.5 MLM 3.5 R-X Procedure 3.5.1 Statements of the MEM2 Problem 3.5.2 R-X Procedure 3.5.3 Example 3.6 Algorithms and Numerical Examples (I) 3.6.1 Frieden Algorithm 3.6.2 Gull-Daniell Algorithm 3.6.3 Revised GD Algorithm 3.6.4 Simplified Newton-Raphson Algorithm 3.6.5 Numerical Example 3.7 Algorithms and Numerical Examples (II) 3.7.1 Skilling-Bryan Algorithm 3.7.2 Differential Equation Approach 3.8 Algorithms and Numerical Examples (III) 3.8.1 MEM/MemSys5 Package 3.8.2 MEM Task in IRAF 3.8.3 Restoration with Variable Resolution

92 92 99 105 108 109 110 110 112 113 115 115 115 120 122 124 124 126 130 133 133 138 139 140 141 142 142 143 147 149 150 152 154 157 161 162 162 169 176 177 181 184

Table of Contents

3.8.4 Numerical Examples 3.8.5 Other Algorithms 4.

5.

Analysis and Comparison of the Maximum Entropy Method 4.1 Generalized MEM 4.1.1 Formulation of GMEM 4.1.2 "Entropy" Expressions in GMEM 4.1.3 Properties of GMEM 4.2 Expressions of Entropy 4.3 Solution's Properties 4.3.1 Existence 4.3.2 Uniqueness 4.3.3 Consistency 4.3.4 Statistical Properties 4.4 Resolution Enhancement and Data Extension (Experimental Results) 4.4.1 Examples 4.4.2 Resolvability in 1-D Spectral Estimation 4.4.3 Resolvability in 2-D Spectral Estimation 4.4.4 Superresolution and Spectral Line Splitting 4.5 Resolution Enhancement and Data Extension (Theoretical Analysis) 4.5.1 Data Extension in MEM1 and MEM2 4.5.2 Resolution Enhancement of MEM1 and MEM2 4.5.3 MEM1 and MEM2 Spectra at Low SNR 4.5.4 Line Splitting of MEM1 4.6 Peak Location and Relative Power Estimation (Experimental Results) 4.6.1 Peak Location (Given ACF) 4.6.2 Peak Location (Given Time Series) 4.6.3 Relative Power Estimation (Given ACF) 4.6.4 Summary and Comments 4.7 Peak Location and Relative Power Estimation (Theoretical Analysis) 4.7.1 Interference Between Peaks Causes Peak Shifting 4.7.2 Explanation of the Peak Shifting in MEM1 Spectra . . . 4.7.3 Relative Power Estimation for MEM1 4.7.4 Summary for Sects. 4.4-4.7 4.8 Comments on the Three Schools of Thought on MEM Applications of the Maximum Entropy Method in Mathematics and Physics 5.1 Solution of Moment Problems 5.1.1 General Theory

XI

187 189 191 191 192 194 196 198 198 198 201 204 209 210 212 216 219 220 224 224 226 227 229 231 231 233 235 237 238 239 239 243 247 247 251 252 252

XII

Table of Contents

5.2

5.3

5.4 5.5 5.6

5.7

5.1.2 Numerical Methods 5.1.3 Noisy Moment Problems 5.1.4 Numerical Examples Solution of Integral Equations 5.2.1 Conversion of Integral Equations to Moment Problems 5.2.2 Solution of Moment Problems by MEM 5.2.3 Numerical Examples 5.2.4 Discussion Solution of Partial Differential Equations 5.3.1 Theory 5.3.2 Numerical Example 5.3.3 Discussion Predictive Statistical Mechanics 5.4.1 Formulation and Solution 5.4.2 Useful Formulae Distributions of Particles Among Energy Levels 5.5.1 Boltzmann Distribution 5.5.2 Fermi-Dirac and Bose-Einstein Distributions Classical Statistical Ensembles 5.6.1 Micro canonical Ensemble 5.6.2 Canonical Ensemble 5.6.3 Grand Canonical Ensemble Quantum Statistical Ensembles 5.7.1 Microcanonical Ensemble 5.7.2 Canonical Ensemble 5.7.3 Grand Canonical Ensemble

255 256 258 263 263 264 267 273 273 273 275 280 282 282 285 287 287 290 293 295 296 297 299 301 302 303

Appendices A. Cepstral Analysis A.I Cepstral Analysis System A.2 I/O Relationship A.3 Properties of the Complex Cepstrum A.4 I/O Relationship for Minimum-Phase Input B. Image Restoration B.I Image Formation B.2 Image Restoration B.3 Relationship Between Image Restoration and Spectral Estimation

305 305 305 306 307 309 311 311 313

References

317

Index

323

314