| 1 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
|---|
| 2 | <html xmlns="http://www.w3.org/1999/xhtml"> |
|---|
| 3 | <head> |
|---|
| 4 | <meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/> |
|---|
| 5 | <title>mixpp: Theory of ARX model estimation</title> |
|---|
| 6 | <link href="tabs.css" rel="stylesheet" type="text/css"/> |
|---|
| 7 | <link href="doxygen.css" rel="stylesheet" type="text/css"/> |
|---|
| 8 | </head> |
|---|
| 9 | <body> |
|---|
| 10 | <!-- Generated by Doxygen 1.6.1 --> |
|---|
| 11 | <script type="text/javascript"> |
|---|
| 12 | <!-- |
|---|
| 13 | function changeDisplayState (e){ |
|---|
| 14 | var num=this.id.replace(/[^[0-9]/g,''); |
|---|
| 15 | var button=this.firstChild; |
|---|
| 16 | var sectionDiv=document.getElementById('dynsection'+num); |
|---|
| 17 | if (sectionDiv.style.display=='none'||sectionDiv.style.display==''){ |
|---|
| 18 | sectionDiv.style.display='block'; |
|---|
| 19 | button.src='open.gif'; |
|---|
| 20 | }else{ |
|---|
| 21 | sectionDiv.style.display='none'; |
|---|
| 22 | button.src='closed.gif'; |
|---|
| 23 | } |
|---|
| 24 | } |
|---|
| 25 | function initDynSections(){ |
|---|
| 26 | var divs=document.getElementsByTagName('div'); |
|---|
| 27 | var sectionCounter=1; |
|---|
| 28 | for(var i=0;i<divs.length-1;i++){ |
|---|
| 29 | if(divs[i].className=='dynheader'&&divs[i+1].className=='dynsection'){ |
|---|
| 30 | var header=divs[i]; |
|---|
| 31 | var section=divs[i+1]; |
|---|
| 32 | var button=header.firstChild; |
|---|
| 33 | if (button!='IMG'){ |
|---|
| 34 | divs[i].insertBefore(document.createTextNode(' '),divs[i].firstChild); |
|---|
| 35 | button=document.createElement('img'); |
|---|
| 36 | divs[i].insertBefore(button,divs[i].firstChild); |
|---|
| 37 | } |
|---|
| 38 | header.style.cursor='pointer'; |
|---|
| 39 | header.onclick=changeDisplayState; |
|---|
| 40 | header.id='dynheader'+sectionCounter; |
|---|
| 41 | button.src='closed.gif'; |
|---|
| 42 | section.id='dynsection'+sectionCounter; |
|---|
| 43 | section.style.display='none'; |
|---|
| 44 | section.style.marginLeft='14px'; |
|---|
| 45 | sectionCounter++; |
|---|
| 46 | } |
|---|
| 47 | } |
|---|
| 48 | } |
|---|
| 49 | window.onload = initDynSections; |
|---|
| 50 | --> |
|---|
| 51 | </script> |
|---|
| 52 | <div class="navigation" id="top"> |
|---|
| 53 | <div class="tabs"> |
|---|
| 54 | <ul> |
|---|
| 55 | <li><a href="main.html"><span>Main Page</span></a></li> |
|---|
| 56 | <li class="current"><a href="pages.html"><span>Related Pages</span></a></li> |
|---|
| 57 | <li><a href="annotated.html"><span>Classes</span></a></li> |
|---|
| 58 | <li><a href="files.html"><span>Files</span></a></li> |
|---|
| 59 | </ul> |
|---|
| 60 | </div> |
|---|
| 61 | <div class="navpath"><a class="el" href="tutorial.html">Tutorial in Bayesian estimation</a> |
|---|
| 62 | </div> |
|---|
| 63 | </div> |
|---|
| 64 | <div class="contents"> |
|---|
| 65 | |
|---|
| 66 | |
|---|
| 67 | <h1><a class="anchor" id="tut_arx">Theory of ARX model estimation </a></h1><p></p> |
|---|
| 68 | <p>The <code>ARX</code> (AutoregRessive with eXogeneous input) model is defined as follows: </p> |
|---|
| 69 | <p class="formulaDsp"> |
|---|
| 70 | <img class="formulaDsp" alt="\[ y_t = \theta' \psi_t + \rho e_t \]" src="form_115.png"/> |
|---|
| 71 | </p> |
|---|
| 72 | <p> where <img class="formulaInl" alt="$y_t$" src="form_3.png"/> is the system output, <img class="formulaInl" alt="$[\theta,\rho]$" src="form_116.png"/> is vector of unknown parameters, <img class="formulaInl" alt="$\psi_t$" src="form_117.png"/> is an vector of data-dependent regressors, and noise <img class="formulaInl" alt="$e_t$" src="form_24.png"/> is assumed to be Normal distributed <img class="formulaInl" alt="$\mathcal{N}(0,1)$" src="form_118.png"/>.</p> |
|---|
| 73 | <p>Special cases include: </p> |
|---|
| 74 | <ul> |
|---|
| 75 | <li>estimation of unknown mean and variance of a Gaussian density from independent samples.</li> |
|---|
| 76 | </ul> |
|---|
| 77 | <h2><a class="anchor" id="off"> |
|---|
| 78 | Off-line estimation:</a></h2> |
|---|
| 79 | <p>This particular model belongs to the exponential family, hence it has conjugate distribution (i.e. both prior and posterior) of the Gauss-inverse-Wishart form. See [ref]</p> |
|---|
| 80 | <p>Estimation of this family can be achieved by accumulation of sufficient statistics. The sufficient statistics Gauss-inverse-Wishart density is composed of: </p> |
|---|
| 81 | <dl> |
|---|
| 82 | <dt>Information matrix </dt> |
|---|
| 83 | <dd>which is a sum of outer products <p class="formulaDsp"> |
|---|
| 84 | <img class="formulaDsp" alt="\[ V_t = \sum_{i=0}^{n} \left[\begin{array}{c}y_{t}\\ \psi_{t}\end{array}\right] \begin{array}{c} [y_{t}',\,\psi_{t}']\\ \\\end{array} \]" src="form_119.png"/> |
|---|
| 85 | </p> |
|---|
| 86 | </dd> |
|---|
| 87 | <dt>"Degree of freedom" </dt> |
|---|
| 88 | <dd>which is an accumulator of number of data records <p class="formulaDsp"> |
|---|
| 89 | <img class="formulaDsp" alt="\[ \nu_t = \sum_{i=0}^{n} 1 \]" src="form_120.png"/> |
|---|
| 90 | </p> |
|---|
| 91 | </dd> |
|---|
| 92 | </dl> |
|---|
| 93 | <h2><a class="anchor" id="on"> |
|---|
| 94 | On-line estimation</a></h2> |
|---|
| 95 | <p>For online estimation with stationary parameters can be easily achieved by collecting the sufficient statistics described above recursively.</p> |
|---|
| 96 | <p>Extension to non-stationaly parameters, <img class="formulaInl" alt="$ \theta_t , r_t $" src="form_121.png"/> can be achieved by operation called forgetting. This is an approximation of Bayesian filtering see [Kulhavy]. The resulting algorithm is defined by manipulation of sufficient statistics: </p> |
|---|
| 97 | <dl> |
|---|
| 98 | <dt>Information matrix </dt> |
|---|
| 99 | <dd>which is a sum of outer products <p class="formulaDsp"> |
|---|
| 100 | <img class="formulaDsp" alt="\[ V_t = \phi V_{t-1} + \left[\begin{array}{c}y_{t}\\ \psi_{t}\end{array}\right] \begin{array}{c} [y_{t}',\,\psi_{t}']\\ \\\end{array} +(1-\phi) V_0 \]" src="form_122.png"/> |
|---|
| 101 | </p> |
|---|
| 102 | </dd> |
|---|
| 103 | <dt>"Degree of freedom" </dt> |
|---|
| 104 | <dd>which is an accumulator of number of data records <p class="formulaDsp"> |
|---|
| 105 | <img class="formulaDsp" alt="\[ \nu_t = \phi \nu_{t-1} + 1 + (1-\phi) \nu_0 \]" src="form_123.png"/> |
|---|
| 106 | </p> |
|---|
| 107 | </dd> |
|---|
| 108 | </dl> |
|---|
| 109 | <p>where <img class="formulaInl" alt="$ \phi $" src="form_124.png"/> is the forgetting factor, typically <img class="formulaInl" alt="$ \phi \in [0,1]$" src="form_125.png"/> roughly corresponding to the effective length of the exponential window by relation:</p> |
|---|
| 110 | <p class="formulaDsp"> |
|---|
| 111 | <img class="formulaDsp" alt="\[ \mathrm{win_length} = \frac{1}{1-\phi}\]" src="form_126.png"/> |
|---|
| 112 | </p> |
|---|
| 113 | <p> Hence, <img class="formulaInl" alt="$ \phi=0.9 $" src="form_127.png"/> corresponds to estimation on exponential window of effective length 10 samples.</p> |
|---|
| 114 | <p>Statistics <img class="formulaInl" alt="$ V_0 , \nu_0 $" src="form_128.png"/> are called alternative statistics, their role is to stabilize estimation. It is easy to show that for zero data, the statistics <img class="formulaInl" alt="$ V_t , \nu_t $" src="form_129.png"/> converge to the alternative statistics.</p> |
|---|
| 115 | <h2><a class="anchor" id="str"> |
|---|
| 116 | Structure estimation</a></h2> |
|---|
| 117 | <p>For this model, structure estimation is a form of model selection procedure. Specifically, we compare hypotheses that the data were generated by the full model with hypotheses that some regressors in vector <img class="formulaInl" alt="$\psi$" src="form_33.png"/> are redundant. The number of possible hypotheses is then the number of all possible combinations of all regressors.</p> |
|---|
| 118 | <p>However, due to property known as nesting in exponential family, these hypotheses can be tested using only the posterior statistics. (This property does no hold for forgetting <img class="formulaInl" alt="$ \phi<1 $" src="form_130.png"/>). Hence, for low dimensional problems, this can be done by a tree search (method <a class="el" href="classbdm_1_1ARX.html#a16b02ae03316751664c22d59d90c1e34" title="Brute force structure estimation.">bdm::ARX::structure_est()</a>). Or more sophisticated algorithm [ref Ludvik]</p> |
|---|
| 119 | <h2><a class="anchor" id="soft"> |
|---|
| 120 | Software Image</a></h2> |
|---|
| 121 | <p>Estimation of the ARX model is implemented in class <a class="el" href="classbdm_1_1ARX.html" title="Linear Autoregressive model with Gaussian noise.">bdm::ARX</a>. </p> |
|---|
| 122 | <ul> |
|---|
| 123 | <li>models from exponential family share some properties, these are encoded in class <a class="el" href="classbdm_1_1BMEF.html" title="Estimator for Exponential family.">bdm::BMEF</a> which is the parent of ARX </li> |
|---|
| 124 | <li>one of the parameters of <a class="el" href="classbdm_1_1BMEF.html" title="Estimator for Exponential family.">bdm::BMEF</a> is the forgetting factor which is stored in attribute <code>frg</code>, </li> |
|---|
| 125 | <li>posterior density is stored inside the estimator in the form of <a class="el" href="classbdm_1_1egiw.html" title="Gauss-inverse-Wishart density stored in LD form.">bdm::egiw</a> </li> |
|---|
| 126 | <li>references to statistics of the internal <code>egiw</code> class, i.e. attributes <code>V</code> and <code>nu</code> are established for convenience.</li> |
|---|
| 127 | </ul> |
|---|
| 128 | <h2><a class="anchor" id="try"> |
|---|
| 129 | How to try</a></h2> |
|---|
| 130 | <p>The best way to experiment with this object is to run matlab script <code>arx_test.m</code> located in directory <code></code>./library/tutorial. See <a class="el" href="arx_ui.html">Running experiment <code>estimator</code> with ARX data fields</a> for detailed description.</p> |
|---|
| 131 | <ul> |
|---|
| 132 | <li>In default setup, the parameters converge to the true values as expected. </li> |
|---|
| 133 | <li>Try changing the forgetting factor, field <code>estimator.frg</code>, to values <1. You should see increased lower and upper bounds on the estimates. </li> |
|---|
| 134 | <li>Try different set of parameters, filed <code>system.theta</code>, you should note that poles close to zero are harder to identify. </li> |
|---|
| 135 | </ul> |
|---|
| 136 | </div> |
|---|
| 137 | <hr size="1"/><address style="text-align: right;"><small>Generated on Sun Aug 30 22:10:50 2009 for mixpp by |
|---|
| 138 | <a href="http://www.doxygen.org/index.html"> |
|---|
| 139 | <img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.6.1 </small></address> |
|---|
| 140 | </body> |
|---|
| 141 | </html> |
|---|
