MVR^*

While the CCA and SVD models were based on maximising correlations and covariance, the regression models aim to minimise the root mean square (RMS) error. The regression models are based on the least squares solutions to an inconsistent system:

$${\bf Y} = {\bf X \Psi} - {\bf\zeta},$$ (8.36)

of n equations with q unknowns, and where ${\bf\zeta}$ represents a noise term. If the noise term is insignificant then the linear expression is satisfied by the normal equations:

$${\bf X}^T {\bf X \Psi} = {\bf X}^T {\bf Y},$$ (8.37)

The matrix product ${\bf X}^T {\bf X}$ is invertible if the columns of ${\bf X}$ are independent, and we can express ${\bf\Psi}$ in terms of ${\bf X}$ and ${\bf Y}$ only [, pp.156]:

$${\bf\Psi} = ({\bf X}^T {\bf X})^{-1} {\bf X}^T {\bf Y}.$$ (8.38)

Equation 8.36 may, however, involve significant noise levels and a ``true'' estimate of ${\bf\Psi}$ is then:

$${\bf\Psi} = ({\bf X}^T {\bf\Omega}^{-1} {\bf X})^{-1} {\bf X}^T {\bf\Omega}^{-1} {\bf Y},$$ (8.39)

where ${\bf\Omega} = E({\bf\zeta \zeta}^T)$ is the error-covariance matrix. One problem is that we only have an estimate of ${\bf\Omega}$ if ${\bf\zeta}$ is known ( $\hat{{\bf\Omega}} = {\bf\zeta \zeta}^T$). ${\bf\Omega}$ may also be non-invertible. We can get around these problems by excluding the noise term from the analysis, and only attempt to predict the signal in ${\bf Y}$ that is related to ${\bf X}$, which we refer to as $\hat{{\bf Y}}$.

$$\hat{{\bf Y}} = {\bf\Psi X}.$$ (8.40)

By applying PCA to the data and truncating to the kth leading EOF, we also remove noise in ${\bf X}$ and ensure that $({\bf X}^T {\bf X})^{-1}$ is invertible by writing the matrix in terms of its PCA products,

$$[{\bf X}^T {\bf X}]^{-1} = [({\bf E}_{(k)} W_{(k)} {\bf A}^T_... ...}]^{-1} = [{\bf A}_{(k)} {\bf W}_{(k)}^2 {\bf A}^T_{(k)}]^{-1}.$$

Hence equation  8.39 can be expressed as:

$${\bf\Psi} = ({\bf A}_{(k)} {\bf W}^2_{(k)} {\bf A}^T_{(k)})^{-1} {\bf X}^T {\bf Y},$$ (8.41)

MVR results are in general similar to the BP-CCA results. See Chapter 7 for more discussion on linear and multi-variate regression techniques.