An outlier is an observation that is located far away relative to the distribution of the remaining observations. In the regression context, the term outlier is usually used in the context of the target variable, where observations far away in the feature space are called leverage or influence points.
Outliers are often identified subjectively, but common heuristics include points beyond 1.5 interquartile ranges from the first and third quartiles, or those a certain number of standard deviations beyond the mean. Outliers can be problematic when they have undue influence on an algorithm’s fit, such as pulling a regression line one way or the other compared to if that observation was not present. They can also indicate issues pertaining to the quality or data generation mechanism, and it is necessary to understand the context of outliers before deciding how to address them.
