Introduction to Linear Regression Analysis. Douglas C. Montgomery

maximum-likelihood estimation requires more stringent statistical assumptions than the least-squares estimators. The least-squares estimators require only second-moment assumptions (assumptions about the expected value, the variances, and the covariances among the random errors). The maximum-likelihood estimators require a full distributional assumption, in this case that the random errors follow a normal distribution with the same second moments as required for the least- squares estimates. For more information on maximum-likelihood estimation in regression models, see Graybill [1961, 1976], Myers [1990], Searle [1971], and Seber [1977].

2.13 CASE WHERE THE REGRESSOR x IS RANDOM

      The linear regression model that we have presented in this chapter assumes that the values of the regressor variable x are known constants. This assumption makes the confidence coefficients and type I (or type II) errors refer to repeated sampling on y at the same x levels. There are many situations in which assuming that the x’s are fixed constants is inappropriate. For example, consider the soft drink delivery time data from Chapter 1 (Figure 1.1). Since the outlets visited by the delivery person are selected at random, it is unrealistic to believe that we can control the delivery volume x. It is more reasonable to assume that both y and x are random variables.

      Fortunately, under certain circumstances, all of our earlier results on parameter estimation, testing, and prediction are valid. We now discuss these situations.

2.13.1 x and y Jointly Distributed

      Suppose that x and y are jointly distributed random variables but the form of this joint distribution is unknown. It can be shown that all of our previous regression results hold if the following conditions are satisfied:

      1 The conditional distribution of y given x is normal with conditional mean β0 + β1x and conditional variance σ2.

      2 The x’s are independent random variables whose probability distribution does not involve β0, β1, and σ2.

      While all of the regression procedures are unchanged when these conditions hold, the confidence coefficients and statistical errors have a different interpretation. When the regressor is a random variable, these quantities apply to repeated sampling of (xi, yi) values and not to repeated sampling of yi at fixed levels of xi.

      2.13.2 x and y Jointly Normally Distributed: Correlation Model

      Now suppose that y and x are jointly distributed according to the bivariate normal distribution. That is,

      (2.60)

      where μ₁ and the mean and variance of y, μ₂ and the mean and variance of x, and

      is the correlation coefficient between y and x. The term σ₁₂ is the covariance of y and x.

      The conditional distribution of y for a given value of x is

      (2.61)

      where

      (2.62a)

      (2.62b)

and

      (2.62c)

      That is, the conditional distribution of y given x is normal with conditional mean

      (2.63)

      and conditional variance . Note that the mean of the conditional distribution of y given x is a straight-line regression model. Furthermore, there is a relationship between the correlation coefficient ρ and the slope β₁. From Eq. (2.62b) we see that if ρ = 0, then β₁ = 0, which implies that there is no linear regression of y on x. That is, knowledge of x does not assist us in predicting y.

      The method of maximum likelihood may be used to estimate the parameters β₀ and β₁. It may be shown that the maximum-likelihood estimators of these parameters are

(2.64a)

      and

      (2.64b)

The estimators of the intercept and slope in Eq. (2.64) are identical to those given by the method of least squares in the case where x was assumed to be a controllable variable. In general, the regression model with y and x jointly normally distributed may be analyzed by the methods presented previously for the model where x is a controllable variable. This follows because the random variable y given x is independently and normally distributed with mean β₀ + β₁x and constant variance . As noted in Section 2.12.1, these results will also hold for any joint distribution of y and x such that the conditional distribution of y given x is normal.

      It is possible to draw inferences about the correlation coefficient ρ in this model. The estimator of ρ is the sample correlation coefficient

      (2.65)

      Note that

(2.66)

so that the slope is just the sample correlation coefficient r multiplied by a scale factor that is the square root of the spread of the y’s divided by the spread of the x’s. Thus, and r are closely related, although they provide somewhat different information. The sample correlation coefficient r is a measure of the linear association between y and x, while measures the change in the mean of y for a unit change in x. In the case of a controllable variable x, r has no meaning because the magnitude of r depends on the choice of spacing for x. We may also write, from