Introduction to Linear Regression Analysis. Douglas C. Montgomery

Cases Stocked, x 10.15 25 2.96 6 3.00 8 6.88 17 0.28 2 5.06 13 9.14 23 11.86 30 11.69 28 6.04 14 7.57 19 1.74 4 9.38 24 0.16 1 1.84 5

Figure 2.14 Scatter diagram of shelf-stocking data.

Figure 2.15 The confidence and prediction bands for the shelf-stocing data.

      Figure 2.15 also shows the 95% confidence interval or E(y|x₀) computed from Eq. (2.54) and the 95% prediction interval on a single future observation y₀ at x = x₀ computed from Eq. (2.55). Notice that the length of the confidence interval at x₀ = 0 is zero.

      SAS handles the no-intercept case. For this situation, the model statement follows:

      model time = cases/noint

2.12 ESTIMATION BY MAXIMUM LIKELIHOOD

      The method of least squares can be used to estimate the parameters in a linear regression model regardless of the form of the distribution of the errors ε. Least squares produces best linear unbiased estimators of β₀ and β₁. Other statistical procedures, such as hypothesis testing and CI construction, assume that the errors are normally distributed. If the form of the distribution of the errors is known, an alternative method of parameter estimation, the method of maximum likelihood, can be used.

      Consider the data (yi, xi), i = 1, 2, …, n. If we assume that the errors in the regression model are NID(0, σ²), then the observations yi in this sample are normally and independently distributed random variables with mean β₀ + β₁xi and variance σ². The likelihood function is found from the joint distribution of the observations. If we consider this joint distribution with the observations given and the parameters β₀, β₁, and σ² unknown constants, we have the likelihood function. For the simple linear regression model with normal errors, the likelihood function is

      (2.56)

      The maximum-likelihood estimators are the parameter values, say , , and , that maximize L, or equivalently, ln L. Thus,

      (2.57)

      and the maximum-likelihood estimators , , and must satisfy

(2.58a)

      (2.58b)

      and

      (2.58c)

The solution to Eq. (2.58) gives the maximum-likelihood estimators:

      (2.59a)

(2.59b)

      (2.59c)

      Notice that the maximum-likelihood estimators of the intercept and slope, and , are identical to the least-squares estimators of these parameters. Also, is a biased estimator of σ². The biased estimator is related to the unbiased estimator [Eq. (2.19)] by . The bias is small if n is moderately large. Generally the unbiased estimator is used.

      In general, maximum-likelihood estimators have better statistical properties than least-squares estimators. The maximum-likelihood estimators are unbiased (including , which is asymptotically unbiased, or unbiased as n becomes large) and have minimum variance when compared to all other unbiased estimators. They are also consistent estimators (consistency is a large-sample property indicating that the estimators differ from the true parameter value by a very small amount as n becomes large), and they are a set of sufficient statistics (this implies that the estimators contain all of the “information” in the original sample of size