Correction
20 Nov 2012: Su L, Zhao Y, Yan T, Li F (2012) Correction: Local Polynomial Estimation of Heteroscedasticity in a Multivariate Linear Regression Model and Its Applications in Economics. PLOS ONE 7(11): 10.1371/annotation/01ebf0b5-ccd3-494d-b577-170981f7bc36. https://doi.org/10.1371/annotation/01ebf0b5-ccd3-494d-b577-170981f7bc36 View correction
Figures
Abstract
Multivariate local polynomial fitting is applied to the multivariate linear heteroscedastic regression model. Firstly, the local polynomial fitting is applied to estimate heteroscedastic function, then the coefficients of regression model are obtained by using generalized least squares method. One noteworthy feature of our approach is that we avoid the testing for heteroscedasticity by improving the traditional two-stage method. Due to non-parametric technique of local polynomial estimation, it is unnecessary to know the form of heteroscedastic function. Therefore, we can improve the estimation precision, when the heteroscedastic function is unknown. Furthermore, we verify that the regression coefficients is asymptotic normal based on numerical simulations and normal Q-Q plots of residuals. Finally, the simulation results and the local polynomial estimation of real data indicate that our approach is surely effective in finite-sample situations.
Citation: Su L, Zhao Y, Yan T, Li F (2012) Local Polynomial Estimation of Heteroscedasticity in a Multivariate Linear Regression Model and Its Applications in Economics. PLoS ONE 7(9): e43719. https://doi.org/10.1371/journal.pone.0043719
Editor: Timothy Ravasi, King Abdullah University of Science and Technology, Saudi Arabia
Received: May 2, 2012; Accepted: July 24, 2012; Published: September 17, 2012
Copyright: © Su et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This work was supported by Chongqing CSTC foundations of China (CSTC2010BB2310, CSTC2011jjA40033), Chongqing CMEC foundations of China (KJ080614, KJ100810, KJ100818). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The heteroscedasticity in classical linear regression model is defined by the variances of random items and which are not the same for different explanatory variables and observations [1], [2]. Especially the heteroscedasticity is usually inevitable when we study the cross-sectional data [3]–[6]. When there is a heteroscedasticity in a linear regression model, estimations of parameters we obtained by ordinary least squares estimation (OLS) are still linear and unbiased [7]–[10]. However, the efficiency is bad [11], [12]. This could lead to a uncorrect statistical diagnosis for the parameters’ significance test. Similarly, it unnecessarily enlarges the confidence interval when we estimate the parameter interval. Besides, the accuracy of predictive value may lower when we estimate with the regression model that we obtained. In order to solve the problem above, we can use generalized least squares estimation (GLS) when the covariance matrix of the random items is known. If it is unknown, we usually use two-stage least squares estimate. In other words, we first estimate variances of the residual error, and then the generalized least squares estimator is used to obtain the coefficients of the model by using the estimate of variances of the random items. However, the traditional estimation method is that we suppose the residual error variances as a certain parametric model. So estimations we obtained are always inaccurate. For a heteroscedasticity model which has only one explanatory variable, we have discussed in detail and given a rigorous proof in [13]. In this paper, we try to apply multivariate local polynomial fitting to random item variances as the first step, and then generalized least squares estimation is used to estimate the coefficients of the model. On the one hand, because of local polynomial fitting’s various nice statistical properties, the estimations obtained with this technology also have the same good statistical properties [14]. On the other hand, we exploit a heteroscedastic regression model rather than the artificial structure of heteroscedasticity. Then, we can directly get the heteroscedastic function based on the nonparametric technique, which shows the relationship between variance function of random items and explanatory variables from regression results [15]. Thus, it is unnecessary to test heteroscedasticity of the model. Particularly, the estimated value by multivariate local polynomial fitting are more accurate than that by the traditional method and univariate local polynomial fitting.
The rest of this paper is organized as follows. The multivariate local polynomial estimation adopted is explained in detail in Section Methods [16]–[18]: in its the first Subsection we study multivariate estimation with local polynomial fitting. The second subsection contains selections of parameters. Estimations of model coefficients and heteroscedastic function are presented in the third subsection Parametric estimations with multivariate local polynomial regression. In Sections Results and Discussion, we firstly give two models, do the simulations, then collect some real data, and use the local polynomial estimating the coefficients, respectively. Finally, Section Conclusions conclude.
Methods
Multivariate local polynomial fitting is an attractive method both from theoretical and practical point of view. It possesses a smaller mean squared error than that of classical kernel Nadaraya-Watson estimator which leads to an undesirable form of the bias and the Gasser-Muller estimator which has to pay a price in variance when dealing with a random design model. It also possesses other advantages [19], [20]. In this Section, we briefly outline the idea of the extension of multivariate local polynomial fitting to multivariate linear regression.
Multivariate Estimator with Local Polynomial Fitting
We treat the m-dimension estimation problem where the measured data at the position
is given by
(1)where
is the regression function to be estimate,
,
(
denoting the identity matrix of order
), and
and
are independent. We always denote the conditional variance of
given
by
and the marginal density of
, that is, the design density, by
.
While the specific form of may remain unspecific, if we assume that the
th derivative of
at the point
exsits, then in order to estimate the value at this point, we can rely on a generic local expansion of the function about this point. Specifically, for
in a neighborhood of
, a
-term Taylor expansion gives,
(2)where
Given the series , this polynomial is fitted locally by a weighted least squares regression problem: minimize.
(3)where
is a bandwidth matrix controlling the size of the local neighborhood and
is the kernel function which penalizes both geometric and radiometric distances, where
is defined as [21]
Denote by the solution to the least squares problem (3)). It is clear from the Taylor expansion in (2) that
is an estimator for
.
It is more convenient to work with matrix notation. For the weighted least squared problem, a matrix form can be depicted by(4)where
(7)We then have the least squared solution with multivariate local polynomial fitting [22].
(8)where
denotes pseudo-inverse, or when
is inverse, the estimation can be written by
(9)Then, we can get the estimation
,
(10)where
is a column vector (the same size of a in (5)) with the first element equal to 1, and the rest equal to zero, that is,
Computing the will suffer from large computational cost. We can use the recursive least squared method to reduce the computation complexity, and it is very powerful especially in the local polynomial fitting problems. There are several important issues about the bandwidth, the order of multivariate local polynomial function and the kernel function which have to be discussed. The three problems will be presented in following subsection.
Parameters Selections
For the multivariate local polynomial estimator, there are three important problems which have significant influence to the estimation accuracy and computational complexity [23].
First of all, there is the choice of the bandwidth matrix, which plays a rather crucial role. The bandwidth matrix is taken to be a diagonal matrix. For simplification, the bandwidth matrix is designed into
. Therefore, the most important thing is to find the bandwidth
[24], [25]. In theory, there exists a optimal bandwidth
in the meaning of mean integrated square error(MISE), fulfilling the equality
(11)
However, the theoretical bandwidth in formula (23) can not be directly calculated. Here, we propose a search method to select the bandwidth: Compare values of the objective function as the bandwidth
from small to large, and then find out the optimal bandwidth which minimize the objective function.
Suppose that where
is the minimum,
is coefficient of expansion. We search a bandwidth
to minimize the objective function in the interval
, where the objective function refers to the prediction mean square error (MSE), denoted by
.
Firstly, we assume , then increase
by efficient of expansion
and calculate value of objective function for each
. Stop down when
, and choose a bandwidth
which minimizes
as the approximate optimal bandwidth.
can be taken place by a estimation
In this paper, we choose(12)where
Compared with other methods, this method is more convenient.
In order to closer to the ideal optimal bandwidth, we search once again by narrowing the interval on the basis of the above searching process. Supposing is the bandwidth which make
optimal in the above searching process. Now, divide the small interval
into
equal intervals. Supposing
(13)among these
bandwidths, the approximate optimal bandwidth is the one that makes
minimize.Obviously, this search method can quickly select the right bandwidth.
Another issue in multivariate local polynomial fitting is the choice of the order of the polynomial. For a given bandwidth , a large value of
would expectedly reduce the modeling bias, but would cause a large variance and a considerable computational cost. Since the bandwidth is used to control the modeling complexity, and due to the sparsity of local data in multi-dimensional space, a higher-order polynomial is rarely used. So we apply the local quadratic regression to fit the model (that is to say,
).
The third issue is the selection of the kernel function. In this paper, we choose the spherical kernel as kernel function
(14)where
,
, and
represents
function. This is the optimal kernel function, see [14], [19], [20].
Parametric Estimations with Multivariate Local Polynomial Regression
Let the dependent variable and the explanatory variable
fulfill the following regression model £°.
(15)where
are the observations and
are independent variables. Denote
(16)and
(17)Therefore, the equation (15) can be abbreviated as
(18)
Suppose that
where
are all not equal, that is, there is a heteroscedasticity in model (18). Therefore, the WLS of
is.
(19)or it can be written as
(20)where
If covariance matrix
of
is known, formula (20) can be estimated. Equation (20) is considered as the weighted least squares estimate of
and it possesses nice qualities. However, in many practical situations, the form of
is unknown. Therefore, the so-called two-stage method of estimation is used to solve the heteroscedasticity problem. It can be depicted as follows: first, apply multivariate local polynomial fitting to get the estimate
of
, and then we can obtain the estimate
of
by using equation (20). The estimator follows that
(21)Because of
, we construct the following regression model in order to estimate
,
(22)where
is the difference between
and its expectation. Suppose that
is the OLS of model (18). Although the ordinary least squares estimate
is ineffective, it is still consistent. Therefore, the corresponding residuals hold that
(23)Consequently, we can approximately get
(24)
It can be taken as a regression model, in which the variance function is regression function and the squared residuals are dependent variables. In order to estimate this model, parameter estimation method would usually be taken in some articles. In other words, they suppose
, where the form of
is known and
are the parameters to be estimated. Note that what we usually discuss about more and more detail are
,
and so on [26]. However, the discussion of these models requires that the analyst have a better understanding for the background in practical problems. As an example, variance of corporate profits is often in direct proportion with family income. Since the variance function must be non-negative, a non-parametric method is proposed to fit
. This method can be depicted as follows. Then, the
-order local polynomial estimate of the variances function
is according to formula (2). Using the least squares method for the data around the local window, we can estimate the local intercept via minimizing
(25)
Furthermore, the solution vector can be written as(26)where the weighted matrix
the the design matrix
(27)and
. Consequently, the estimated variance function is
. Finally, we can get two-stage estimate
of
by substituting estimate
of
into equation (21).
Results and Discussion
Simulations and Analysis
In this section, we give the following model to discuss the qualities of under the limited sample. Considering the practical background which is applied to economics, we assume the variance function of the following two forms.
Model 1.
Denote the linear model by(28)where
and
are independent variables. Here, we assume the variance function of the error term
.
Step 1: Firstly, obtain the estimation of two coefficients in model (28) with the ordinary least squares estimation. Second, calculate the squares of the residuals . Third, do the local polynomial regression on the model (24) Figure 1 is the result after 10000 replicates, where
,
, and the kernel function is given by.
(29)
In the kernel function, the ranges of and the same with
are
, and the optimal bandwidth matrix is
The criteria of selecting
is according to MISE, which can be given by
That is .
Minimizing the asymptotic MISE with respect to the bandwidth parameter results in a bandwidth, called the asymptotically optimal bandwidth or simply optimal bandwidth. So we can obtain the optimal bandwidth matrix
. Figure 2 shows the plot which generates from the variance function.
Figure 3 indicates the variance of the error term function and the fitting functions drawn in the same situation. Besides, we compare some fitted value with the true value in Table 1. It is worth pointing out that our fitting is very close to the real value from Figures 1, 2, 3 and Table 1.
Step 2: We substitute which obtained from step 1 into model (20), then we will get
. Figures 4, 5, 6 depict the histograms and asymptotic distributions of
,
, and
, respectively, which we do 10000 replicates with GLS and choose
. It is easy to see from Figures 4, 5, 6 that the estimated distributions of parameters subject to normal asymptotically, which the proofs are in [27]–[30]. Further, we give normal Q-Q plots of residuals of estimated parameters in Figures 7, 8, 9. As illustrated in Figures 10, 11, 12, the histograms and asymptotic distributions of
,
, and
can be get by using OLS, respectively. Similarly, the normal Q-Q plots of residuals are depicted in Figures 13, 14, 15. However, the results are obviously unsatisfying in Figures 10, 11, 12, 13, 14, 15. Besides, the fitted value and true value of GLS and ones of OLS are listed in Table 2. Therefore, we can conclude that our fitting is very perfect. Here, we define the relative error as
, where
is a true value, and
is a fitted value. So the GLS estimates of parameters are much better than OLS according to Table 2.
Model 2.
In formula (28), we assume the error term function is
.and the optimal bandwidth matrix is
, where
. The process of parameter’s estimation and calculation is the same with Model 1. Also we choose
, the ranges of
and the same with
are
. Besides, the kernel function is still defined as equation (29). We do a total of 10000 replicates. Figure 16 shows the estimated variance function
, where the selection of optimal bandwidth matrix is similar with that in step 1. Figure 17 shows the plot which generates from the variance function
Figure 18 indicates the plot of drawing fitted plot and original plot together. According to Figures 16, 17, 18, we can easily reach the conclusion that our fitting is nice and close to the true value. The histograms and asymptotic distributions of ,
and
are depicted in Figures 19, 20 and 21. Besides, Figures 22, 23 and 24 contain the normal Q-Q plots of residuals, respectively. It is worth pointing out that the asymptotic distributions of parameters are symmetric. As shown in Figures 25, 26 and 27, the histograms and asymptotic distributions of
,
, and
of OLS are not symmetrical, but irregular. There are larger fluctuations in normal Q-Q plots of residuals in Figures 28, 29 and 30 than those in Figures 22, 23 and 24. So it is not what we want to reach. Besides, we list fitted value and true value of
,
, and
of GLS and ones of OLS in Table 3. In order to obtain a visual effect, we calculate the relative errors and do a comparison. It is easy to see that the GLS estimates of parameters are much better than OLS from Table 3.
Application
As an example of pure cross-sectional data with potential for heteroscedasticity, consider the data given in Table 4, which gives data on per capita consumption expenditure (PCCE), agricultural business income (ABI) and other income (OI) for 31 different regions in China in 2009. The cross-sectional data presented in this table are quite heterogenous in a regression of PCCE on ABI and OI, so heteroscedasticity is likely.
We hope to understand the relationship of PCCE, ABI and OI, then the regression function is as follows:
Firstly, the ordinary least squares estimators can be obtained, say, ,
,
. Then, inspect that if there is a heteroscedasticity. According to formula (23), the residual squares can be considered as variances. So we need only to fit residual squares, seeing Figure 31, where residual squares are variance squares. Obviously, there are biases among the residual squares fitted. Therefore, it can be concluded that there is a heteroscedasticity. Furthermore, according to the two-stage estimate, the generalized least squares estimation for
,
, and
can be estimated after 10000 replicates provided with the bandwidth matrix
, the order
, and
. The estimations are
,
, and
. Finally, the regression equation of GLS can easily be written as follow:
(30)
The scatter of original and fitting data is also given, see Figure 32.
Conclusions
In this paper £ we presented a new method for estimation of multivariate linear heteroscedastic regression model based on multivariate local polynomial estimation with non-parametric technique. The proposed scheme firstly adopted the local polynomial fitting to estimate heteroscedastic function, then the coefficients of regression model are obtained based on generalized least squares method. Our approach avoided the test of heteroscedasticity for the multivariate linear model. Due to non-parametric technique of local polynomial estimation, if the heteroscedastic function is unknown, the precision of estimation was improved. Furthermore, the asymptotic normality of parameters was verified by the results of numerical simulations normal Q-Q plots. Finally, the simulation results and local polynomial estimation of real data really indicated that our approach was effective in finite-sample situations, which did not need to assume the form of heteroscedastic function. The presented algorithm could be easily used to heteroscedastic regression model in some practical problems.
Author Contributions
Conceived and designed the experiments: LYS YYZ. Performed the experiments: LYS YYZ TSY. Analyzed the data: LYS FLL. Contributed reagents/materials/analysis tools: LYS YYZ. Wrote the paper: LYS YYZ TSY. Collected the data: TSY FLL. Applied the real data: LYS TSY FLL.
References
- 1. Galtchouk L, Pergamenshchikov S (2009) Adaptive asymptotically efficient estimation in heteroscedastic nonparametric regression. J Korean Stat Soc 38: 305–322.
- 2. Galtchouk L, Pergamenshchikov S (2009) Sharp non-asymptotic oracle inequalities for nonparametric heteroscedastic regression models. J Nonparametr Stat 21: 1–18.
- 3. Amemiya T (1977) A note on a heteroscedastic model. J Econom 6: 365–370.
- 4. Chen P, Suter D (2007) A bilinear approach to the parameter estimation of a general heteroscedastic linear system, with application to conic fitting. J Math Imaging Vis 28: 191–208.
- 5. Ma JH, Liu LX (2008) Multivariate nonlinear analysis and prediction of Shanghai stock market. Discrete Dyn Nat Soc 2008: 526734.
- 6. Liu H, Zhang ZQ, Zhao Q (2009) The volatility of the index of shanghai stock market research based on ARCH and its extended forms. Discrete Dyn Nat Soc 2009: 743685.
- 7. Cai L, Hayes AF (2008) A new test of linear hypotheses in OLS regression under heteroscedasticity of unknown form. J Educ Behav Stat 33: 21–40.
- 8. Hayes AF, Cai L (2007) Using heteroskedasticity-consistent standard error estimators in OLS regression: An introduction and software implementation. Behav Res Methods 39: 709–722.
- 9. Lima VMC, Souza TC, Cribari-Neto F, Fernandes GB (2009) Heteroskedasticity-robust inference in linear regressions. Commun Stat Simul Comput 39: 194–206.
- 10. Almeida AM, Castel-Branco MM, Falcão AC (2002) Linear regression for calibration lines revisited: weighting schemes for bioanalytical methods. J Chromatogr B 774: 215–222.
- 11.
Härdle W, Werwatz A, Müller M, Sperlich S (2004) Nonparametric and semiparametric models. New York: Springer. 85–88.
- 12. Fan JQ (1993) Local linear regression smoothers and their minimax efficiency. Ann Stat 21: 196–216.
- 13. Su LY, Zhao YY, Yan TT (2012) Two-stage method based on local polynomial fitting for a linear heteroscedastic regression model and its application in economics. Discrete Dyn Nat Soc 2012: 696927.
- 14. Fan JQ (1992) Design-adaptive nonparametric regression. J Am Stat Assoc 87: 998–1004.
- 15. Godfrey LG (2006) Tests for regression models with heteroskedasticity of unknown form. Comput Stat Data Anal 50: 2715–2733.
- 16. Su LY (2010) Prediction of multivariate chaotic time series with local polynomial fitting. Comput Math Appl 59: 737–744.
- 17. Su LY (2011) Multivariate local polynomial estimation with application to Shenzhen component index. Discrete Dyn Nat Soc 2011: 930958.
- 18. Su LY, Ma YJ, Li JJ (2012) Application of local polynomial estimation in suppressing strong chaotic noise. Chinese Physics B. 21: 020508.
- 19.
Fan JQ, Yao QW (2003) Nonlinear time series: nonparametric and parametric methods. New York: Springer. 313–317.
- 20. Fan JQ, Zhang WY (1999) Statistical estimation in varying coefficient models. Ann Stat 27: 1491–1518.
- 21. Farsiu S, Robinson MD, Elad M, Milanfar P (2004) Fast and robust multiframe super-resolution. IEEE Trans Image Process 13: 1327–1344.
- 22. Ruppert D, Wand MP (1994) Multivariate locally weighted least squares regression. Ann Stat 22: 1346–1370.
- 23. Su LY, Li FL (2010) Deconvolution of defocused image with multivariate local polynomial regression and iterative Wiener filtering in DWT domain. Math Probl Eng 2010: 605241.
- 24.
Fan JQ, Gijbels I (1996) Local polynomial modelling and its applications. London: Chapman and Hall. 263–276.
- 25. Fan JQ, Gijbels I (1995) Data-driven bandwidth selection in local polynomial fitting: variable bandwidth and spatial adaptation. J R Stat Soc Series B Stat Methodol 57: 371–394.
- 26. Rutemiller HC, Bowers DA (1968) Estimation in a heteroscedastic regression model. J Am Stat Assoc 63: 552–557.
- 27. White H (1980) A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 48: 817–838.
- 28. Cribari-Neto F (2004) Asymptotic inference under heteroskedasticity of unknown form. Comput Stat Data Anal 45: 215–233.
- 29. Cribari-Neto F, Souza TC, Vasconcellos KLP (2007) Inference under heteroskedasticity and leveraged data. Commun Stat Theory Methods 36: 1877–1888.
- 30.
Greene WH (1993) Econometric analysis. New York: Macmillan. 362–366.