Maximum likelihood and SIMEX

Pan Miroslav · **Joined:** Sat, 1 Nov 2014 09:07:24 UTC **Posts:** 47

I got some troubles proving following theorems:

1) In linear regression model with DEPENDENT normally distributed errors (with covariance matrix $\Sigma$ ) the maximum likelihood estimation of $\beta$ gives
$\hat{\beta}_{ML} = (X' \Sigma X)^{-1}X'\Sigma^{-1}y$

2) Let $\hat{\beta}_1 = \frac{\sum_{i = 1}^n(x_i - \overline{x})y_i}{\sum_{i = 1}^n(x_i - \overline{x})}$ be estimated by least squares method, let $\sigma_{x \delta} = cov(x_i^A, \delta)$ , $\sigma_x^2 = \sum_{i = 1}^n(x_i^A - \overline{x}^A)^2/n$ , $\sigma_{\delta}^2 = Var(\delta_i)$ .

Show that $E(\hat{\beta}_1) = \beta_1\frac{\sigma_x^2 + \sigma_{x \delta}}{\sigma_x^2 + \sigma_{\delta}^2 + 2\sigma_{x \delta}}$

My progress - in the second one, simply putting all the things I know to the result and hoping that it will give expectation of the $\hat{\beta}_1$ simply did not work. For the first one I don't know what to try, because I'm not even sure how the likelihood function looks like when we got dependent errors.

outermeasure · **Posted:** Thu, 17 Nov 2016 03:56:56 UTC

Pan Miroslav wrote:

I got some troubles proving following theorems:

1) In linear regression model with DEPENDENT normally distributed errors (with covariance matrix $\Sigma$ ) the maximum likelihood estimation of $\beta$ gives
$\hat{\beta}_{ML} = (X' \Sigma X)^{-1}X'\Sigma^{-1}y$

2) Let $\hat{\beta}_1 = \frac{\sum_{i = 1}^n(x_i - \overline{x})y_i}{\sum_{i = 1}^n(x_i - \overline{x})}$ be estimated by least squares method, let $\sigma_{x \delta} = cov(x_i^A, \delta)$ , $\sigma_x^2 = \sum_{i = 1}^n(x_i^A - \overline{x}^A)^2/n$ , $\sigma_{\delta}^2 = Var(\delta_i)$ .

Show that $E(\hat{\beta}_1) = \beta_1\frac{\sigma_x^2 + \sigma_{x \delta}}{\sigma_x^2 + \sigma_{\delta}^2 + 2\sigma_{x \delta}}$

My progress - in the second one, simply putting all the things I know to the result and hoping that it will give expectation of the $\hat{\beta}_1$ simply did not work. For the first one I don't know what to try, because I'm not even sure how the likelihood function looks like when we got dependent errors.

(1) Recall the probability density function of multivariate normal distribution $N(\mu,\Sigma)$
$f(\mathbf{y})=\dfrac{1}{\sqrt{\det(2\pi\Sigma)}}\exp\left(-\dfrac{1}{2}(\mathbf{y}-\mu)^T\Sigma^{-1}(\mathbf{y}-\mu)\right)$
(2) What are x_i^A

and

? What is $\delta$ ?
Anyway, split X=(X_1,X_2)

where

contains all the other independent variables that we ignore and write $\beta=(\beta_1,\beta_2)$ . Expand out $Y=X'\beta+\epsilon$ in the formula for ordinary least-square and take (conditional) expectation (so the $\epsilon$ disappears). The result is the so-called omitted variable (bias) formula
$\mathbb{E}(b_1\mid X)=\beta_1+(X'_1X_1)^{-1}X'_1X_2\beta_2$
which has an intuitive description --- the bias $(X'_1X_1)^{-1}X'_1X_2\beta_2$ is precisely the weighted proportion of the omitted variables X_2

that are "explained" by the variables X_1

we included.

Pan Miroslav · **Joined:** Sat, 1 Nov 2014 09:07:24 UTC **Posts:** 47

outermeasure wrote:

Pan Miroslav wrote:

I got some troubles proving following theorems:

1) In linear regression model with DEPENDENT normally distributed errors (with covariance matrix $\Sigma$ ) the maximum likelihood estimation of $\beta$ gives
$\hat{\beta}_{ML} = (X' \Sigma X)^{-1}X'\Sigma^{-1}y$

2) Let $\hat{\beta}_1 = \frac{\sum_{i = 1}^n(x_i - \overline{x})y_i}{\sum_{i = 1}^n(x_i - \overline{x})}$ be estimated by least squares method, let $\sigma_{x \delta} = cov(x_i^A, \delta)$ , $\sigma_x^2 = \sum_{i = 1}^n(x_i^A - \overline{x}^A)^2/n$ , $\sigma_{\delta}^2 = Var(\delta_i)$ .

Show that $E(\hat{\beta}_1) = \beta_1\frac{\sigma_x^2 + \sigma_{x \delta}}{\sigma_x^2 + \sigma_{\delta}^2 + 2\sigma_{x \delta}}$

My progress - in the second one, simply putting all the things I know to the result and hoping that it will give expectation of the $\hat{\beta}_1$ simply did not work. For the first one I don't know what to try, because I'm not even sure how the likelihood function looks like when we got dependent errors.

(1) Recall the probability density function of multivariate normal distribution $N(\mu,\Sigma)$
$f(\mathbf{y})=\dfrac{1}{\sqrt{\det(2\pi\Sigma)}}\exp\left(-\dfrac{1}{2}(\mathbf{y}-\mu)^T\Sigma^{-1}(\mathbf{y}-\mu)\right)$
(2) What are x_i^A

and

? What is $\delta$ ?
Anyway, split X=(X_1,X_2)

where

contains all the other independent variables that we ignore and write $\beta=(\beta_1,\beta_2)$ . Expand out $Y=X'\beta+\epsilon$ in the formula for ordinary least-square and take (conditional) expectation (so the $\epsilon$ disappears). The result is the so-called omitted variable (bias) formula
$\mathbb{E}(b_1\mid X)=\beta_1+(X'_1X_1)^{-1}X'_1X_2\beta_2$
which has an intuitive description --- the bias $(X'_1X_1)^{-1}X'_1X_2\beta_2$ is precisely the weighted proportion of the omitted variables X_2

that are "explained" by the variables X_1

we included.

So, for the first one it's now basically about using log(f(y))

so it will be simpler and then derivate and solve first order conditions for maximum, right?

For the second one I'll write down some more information
Let (x_i^O, y_i^O)

be observed values for $i = 1, 2, \ldots, n$ and let (x_i^A, y_i^A)

be the close actual values, close means $y_i^O = y_i^A + \epsilon_i, x_i^O = x_i^A + \delta_i$ , where $\epsilon$ and $\delta$ are independent with $E(\epsilon_i) = E(\delta_i) = 0$ and $Var(\epsilon_i) = \sigma_{\epsilon}^2, Var(\delta_i) = \sigma_{\delta}^2$ . We want to model $y_i^A = \beta_0 + \beta_1 x_i^A$ , but we observed values (x_i^O, y_i^O)

so our model is $y_i^O = \beta_0 + \beta_1 x_i^O + (\epsilon_i - \beta_1 \delta_i)$

outermeasure · **Posted:** Fri, 18 Nov 2016 05:49:54 UTC

Pan Miroslav wrote:

outermeasure wrote:

Pan Miroslav wrote:

I got some troubles proving following theorems:

1) In linear regression model with DEPENDENT normally distributed errors (with covariance matrix $\Sigma$ ) the maximum likelihood estimation of $\beta$ gives
$\hat{\beta}_{ML} = (X' \Sigma X)^{-1}X'\Sigma^{-1}y$

2) Let $\hat{\beta}_1 = \frac{\sum_{i = 1}^n(x_i - \overline{x})y_i}{\sum_{i = 1}^n(x_i - \overline{x})}$ be estimated by least squares method, let $\sigma_{x \delta} = cov(x_i^A, \delta)$ , $\sigma_x^2 = \sum_{i = 1}^n(x_i^A - \overline{x}^A)^2/n$ , $\sigma_{\delta}^2 = Var(\delta_i)$ .

Show that $E(\hat{\beta}_1) = \beta_1\frac{\sigma_x^2 + \sigma_{x \delta}}{\sigma_x^2 + \sigma_{\delta}^2 + 2\sigma_{x \delta}}$

My progress - in the second one, simply putting all the things I know to the result and hoping that it will give expectation of the $\hat{\beta}_1$ simply did not work. For the first one I don't know what to try, because I'm not even sure how the likelihood function looks like when we got dependent errors.

(1) Recall the probability density function of multivariate normal distribution $N(\mu,\Sigma)$
$f(\mathbf{y})=\dfrac{1}{\sqrt{\det(2\pi\Sigma)}}\exp\left(-\dfrac{1}{2}(\mathbf{y}-\mu)^T\Sigma^{-1}(\mathbf{y}-\mu)\right)$
(2) What are x_i^A

and

? What is $\delta$ ?
Anyway, split X=(X_1,X_2)

where

contains all the other independent variables that we ignore and write $\beta=(\beta_1,\beta_2)$ . Expand out $Y=X'\beta+\epsilon$ in the formula for ordinary least-square and take (conditional) expectation (so the $\epsilon$ disappears). The result is the so-called omitted variable (bias) formula
$\mathbb{E}(b_1\mid X)=\beta_1+(X'_1X_1)^{-1}X'_1X_2\beta_2$
which has an intuitive description --- the bias $(X'_1X_1)^{-1}X'_1X_2\beta_2$ is precisely the weighted proportion of the omitted variables X_2

that are "explained" by the variables X_1

we included.

So, for the first one it's now basically about using log(f(y))

so it will be simpler and then derivate and solve first order conditions for maximum, right?

For the second one I'll write down some more information
Let (x_i^O, y_i^O)

be observed values for $i = 1, 2, \ldots, n$ and let (x_i^A, y_i^A)

be the close actual values, close means $y_i^O = y_i^A + \epsilon_i, x_i^O = x_i^A + \delta_i$ , where $\epsilon$ and $\delta$ are independent with $E(\epsilon_i) = E(\delta_i) = 0$ and $Var(\epsilon_i) = \sigma_{\epsilon}^2, Var(\delta_i) = \sigma_{\delta}^2$ . We want to model $y_i^A = \beta_0 + \beta_1 x_i^A$ , but we observed values (x_i^O, y_i^O)

so our model is $y_i^O = \beta_0 + \beta_1 x_i^O + (\epsilon_i - \beta_1 \delta_i)$

(1) After you take the product of likelihoods of each observation to get the likelihood of all observed values, yes.
(2) I think you better check the denominator of $\hat{\beta}_1$ , other than that you have X_1=(1,x^A)

, $X_2=(\delta)$ , $X=(1,x^A,\delta)$ and $\beta=(\beta_0,\beta_1,\beta_2)=(\beta_0,\beta_1,-\beta_1)$ so plugging into the omitted variable formula gives $\mathbb{E}\hat{\beta}_1$ .

S.O.S. Mathematics CyberBoard

Maximum likelihood and SIMEX

Who is online