Why the Median of Residuals in \(L_1\) Regression Model With Intercept is Zero
"Everyone" knows that if you fit a regression model with a constant term using ordinary least squares the average of the residuals is zero. Recall that a regression model with intercept means it is assumed that each observation \(y_i\) can be predicted by a linear combination of \(k\) predictors \(x_{ij}\) and an intercept term: $$ y_i= \alpha + \beta_1x_{i1} + \dots + \beta_kx_{ik} + \epsilon_i $$ The numbers \(\alpha, \beta_1, \dots, \beta_k;\) are known as parameters (in Machine Learning, they are often referred to weights as weights instead). The \(\epsilon_i\) terms represent random and unpredicatble error. It is usually assumed that these errors are independent and identically distributed, and we shall do so for the rest of this article. Usually the parameters are unknown and must be estimated from the observed data. There multiple means by which parameter estimates, denoted by \(\hat{\alpha}, \hat{\beta_1}, \dots, \hat{\beta_k};\) can be computed. Using th...