Okay, I am more than a quarter century out from my statistics course and am too lazy to look it up to provide a more specific explanation for DS. Anyhow, he was complaining that the standard deviation formula was unnecessarily complicated since you have to take the square root of the sum of the squares of the differences divided by sample size. He argued that a simpler formula using absolute values would lead to the same result. I pointed out that the formula was derived to apply to all kinds of data sets even though his simpler formula applies to the data sets at issue. It may be due to caffeine deprivation but I can't recall off the top of my head specific examples. Anyone?

It's called
median absolute deviation (MAD)
http://en.wikipedia.org/wiki/Median_absolute_deviation

Standard deviation
http://en.wikipedia.org/wiki/Standard_deviation
and its square, Variance
http://en.wikipedia.org/wiki/Variance
are much nicer and more natural in terms of their properties, see e.g.
http://en.wikipedia.org/wiki/Variance#Basic_properties
http://en.wikipedia.org/wiki/Standard_deviation#Identities_and_mathematical_properties

For more fun, see here
http://en.wikipedia.org/wiki/Metric_(mathematics)
http://en.wikipedia.org/wiki/Norm_(mathematics)

In general, the mean absolute deviation and standard deviation are different quantities. For normally distributed data, their expected ratio is sqrt(2/pi).

The standard deviation is the best estimate of dispersion for normally distributed data, but if the data has heavier tails than the normal, the mean absolute deviation may be preferable since it does not rely on the existence of the second moment.

It may be interesting to compare the probability densities of the

Normal distribution: p(x) = c*exp(-0.5*x^2)
Laplace distribution: p(x) = c'*exp(-|x|)

The Normal distribution has an x^2 term in the exponential, vs. an |x| term for the Laplace. That is why the mean and standard deviation are used to estimate the parameters of the normal distribution, whereas the median and median absolute deviation are used to fit the Laplace distribution. The median and mean absolute deviation are more "robust" measures of location and scale.

A good place to ask statistics questions is Cross Validated.

Bostonian beat me to it-- Cross Validated is a great source for this sort of question.


Here's one that addresses it quite specifically, though--

http://www.mathsisfun.com/data/standard-deviation.html

Actually, DS derived the mean absolute deviation and I recognized that it was a measure used in Stats but I couldn't even remember the name.

Thanks for the wikipedia links. I am not a big fan of wikipedia (or any encyclopedia) in general since it's not always completely accurate or up to date, but it is a good and easy starting point. In this case, wikipedia even pointed out that some people agree with him regarding mean absolute deviation versus standard deviation.

Thanks - I remembered that standard deviation was a better estimator for normal distributions but couldn't articulate why/when other measures would be preferable.

It's helpful to pinpoint that the expected ratio is about 0.8 although sometimes (as in his data sets) the ratio can be 1.

I'll bookmark that site for future reference.

Thanks - that simple comparison with different data sets at the bottom of the page makes it very clear. For the life of me (the caffeine-free me), it didn't occur to me to use extremely simple and tiny data sets.