Saturday, January 17th, 2009 07:05 pm
So I may have a Phd in "maths", but it's pure maths, and I am woefully ignorant of statistics, not having studied it beyond a first year level (I've tutored second year stats. That was a challenge :D)

Thus my mind is actually quite easily blown by fairly basic statistics facts presented in an engaging way. See for example I'm not normal, which makes an obvious point I'd never really thought about before: The normal distribution became central to statistics back before computers, and once people get taught it, unless they go on to study more stats it's their only hammer so everything starts to look like a nail. But lots of large samples are NOT normally distributed, and given computing power we generally don't need to smoosh them into a nice distribution at all but can actually look at what the real data is doing.

My mind was also blown by Full House: The Spread of Excellence from Plato to Darwin which is pretty much just a basic description of evolution being a bounded random walk (Namely, the mean complexity goes up over time while still leaving 99.9% of life at the same basic level, with anything but the most simple species being as likely to get more simple as less)
Saturday, January 17th, 2009 12:02 pm (UTC)
Awesome post - thank you so much for sharing those links!

The only maths I've done at a tertiary level is statistics (unless you count finance, which I guess is applied maths too), which I found unexpectedly fascinating and fulfilling. (Interesting to see the random walk concept applied in that context; it's something I apply every day to value financial instruments.)

Have you read Freakonomics? It's a pretty light read and I think you might find it enjoyable, both from a statistical point of view and to pick holes in. :)

Monday, January 19th, 2009 12:44 am (UTC)
I keep hearing that Freakanomics is infuriating (both in terms of science and economic bias) from various people so am not sure I'm up to it :)

I remember the random walk from thermodynamics in physics, it's funny how these things come up.
Monday, January 19th, 2009 01:13 am (UTC)
I keep hearing that Freakanomics is infuriating (both in terms of science and economic bias) from various people so am not sure I'm up to it :)

Absolutely agree with this; it's one of those peculiar books where I read it and then figured that even if it was All Wrong (and I'm certain much of it was), then the thinking I got out of it was valuable. I think I might be a little peculiar.

If you do decide you're feeling critically robust enough to go there at some point, I'd take the writers' conclusions with a pinch of salt, but the themes of challenging common wisdom and looking for alternative theories to support statistical findings (not to mention confusing correlation and causation - a trap the authors themselves fall into at times) are worthwhile.
Monday, January 19th, 2009 03:29 am (UTC)
It's on my "When I have nothing better to read and a copy falls into my lap" list :)
Saturday, January 17th, 2009 12:15 pm (UTC)
I'm pretty sure that if the 2nd year "calculus and probability" unit that I did had been split out into two separate units, I would've done great on the calc part and failed the stats part. As it is, it was one of the few maths units I passed with a mediocre mark. This may be correlated with the fact that I didn't turn up to the probability lectures, but since I'm not a statistician, I couldn't possibly figure out how to measure that correlation ;-)

When I was working in finance, I was exposed to a few of the ideas mentioned in the articles you linked to, and also discovered the whole controversy over the philosophy of statistics. Are you a Bayesian or a frequentist?
Monday, January 19th, 2009 03:35 am (UTC)
I always got ok marks in stats but I put it down to a general talent with equations, I never felt like I understood what I was doing :)

Are you a Bayesian or a frequentist?"
*googles*
*reads*
Um..
o.O
x_x

I dunno :)
(My brain is a bit dead today after just mopping the whole house after the cat was sick everywhere. I may come to a conclusion later after some pondering)

Saturday, January 17th, 2009 03:07 pm (UTC)
You would enjoy The Lady Tasting Tea, I think. (*eyes the subtitle dubiously, because that's not the way I'd characterize the book at all, at all*) Salsburg lays out a history of statistics, emphasizing the philosophical issues, ever-changing technical limits, and the questions clients wanted answered as shaping forces for the field. He also makes it clear exactly how philosophically shaky the current underpinnings of the field are.

One of the things I especially like is that Salsburg roots his narrative in a social world, so you get stuff about how all the early computing power was provided by "computers" -- women who were employed to compute calculations all day, much in form like a secretarial pool -- and how the statisticians running the firm thought women were especially well-suited to that work, since they were so "docile and patient" and could be trusted to not let bursts of creativity disrupt them from the tedium. (It gets better, later, fortunately. In fact, women in stats is in way better shape than women in pure maths.) In terms of social world, you also get to see some of the academic fights, like how math departments missed the boat in not embracing stats as a legitimate, substantive academic field, and then how statisticians did exactly the same thing with computer science. ;-)

And now that I've got the book review out of the way... *puts on stodgy stats-geek hat*

Most stats education is pretty bad -- the vast bulk of it is for people who don't want to know anything more than how to calculate a number to put into a paper or report, and so what most people get is a simple recipe book. What is generally left out -- or rather, mentioned as a set of assumptions at the beginning of every recipe, and which is thereafter ignored by the students who just want their number -- is when and how these techniques break, what they can and can't do, how to decide if the question being answered is the question you wanted answered, let alone decide if that question is even answerable.

It's a discipline with tricksy philosophical issues threaded deeply throughout it, and most of that gets completely ignored by most people who handle or calculate statistics. Which makes me sad, because it's a pretty mind-blowingly awesome field.
Monday, January 19th, 2009 03:37 am (UTC)
That sounds fantastic. *adds to books to check out list*

It's a discipline with tricksy philosophical issues threaded deeply throughout it, and most of that gets completely ignored by most people who handle or calculate statistics. Which makes me sad, because it's a pretty mind-blowingly awesome field.

So my statsy friends keep telling me :)
Saturday, January 17th, 2009 07:15 pm (UTC)
"anything but the most simple species being as likely to get more simple as less"

I suppose you get a similar effect (long, low-population "tail" of higher-complexity organisms) regardless of the exact probabilities of species evolving to more / less complex forms. I wouldn't have thought those probs were necessarily close, or consistent from organism to organism? Cool stuff anyway.
Monday, January 19th, 2009 03:40 am (UTC)
Yeah, that's probably a complete oversimplification (I'm very prone to those). On the whole there's no trend one way or the other, is all.
Saturday, January 17th, 2009 12:02 pm (UTC)
Awesome post - thank you so much for sharing those links!

The only maths I've done at a tertiary level is statistics (unless you count finance, which I guess is applied maths too), which I found unexpectedly fascinating and fulfilling. (Interesting to see the random walk concept applied in that context; it's something I apply every day to value financial instruments.)

Have you read Freakonomics? It's a pretty light read and I think you might find it enjoyable, both from a statistical point of view and to pick holes in. :)

Monday, January 19th, 2009 12:44 am (UTC)
I keep hearing that Freakanomics is infuriating (both in terms of science and economic bias) from various people so am not sure I'm up to it :)

I remember the random walk from thermodynamics in physics, it's funny how these things come up.
Monday, January 19th, 2009 01:13 am (UTC)
I keep hearing that Freakanomics is infuriating (both in terms of science and economic bias) from various people so am not sure I'm up to it :)

Absolutely agree with this; it's one of those peculiar books where I read it and then figured that even if it was All Wrong (and I'm certain much of it was), then the thinking I got out of it was valuable. I think I might be a little peculiar.

If you do decide you're feeling critically robust enough to go there at some point, I'd take the writers' conclusions with a pinch of salt, but the themes of challenging common wisdom and looking for alternative theories to support statistical findings (not to mention confusing correlation and causation - a trap the authors themselves fall into at times) are worthwhile.
Monday, January 19th, 2009 03:29 am (UTC)
It's on my "When I have nothing better to read and a copy falls into my lap" list :)
Saturday, January 17th, 2009 12:15 pm (UTC)
I'm pretty sure that if the 2nd year "calculus and probability" unit that I did had been split out into two separate units, I would've done great on the calc part and failed the stats part. As it is, it was one of the few maths units I passed with a mediocre mark. This may be correlated with the fact that I didn't turn up to the probability lectures, but since I'm not a statistician, I couldn't possibly figure out how to measure that correlation ;-)

When I was working in finance, I was exposed to a few of the ideas mentioned in the articles you linked to, and also discovered the whole controversy over the philosophy of statistics. Are you a Bayesian or a frequentist?
Monday, January 19th, 2009 03:35 am (UTC)
I always got ok marks in stats but I put it down to a general talent with equations, I never felt like I understood what I was doing :)

Are you a Bayesian or a frequentist?"
*googles*
*reads*
Um..
o.O
x_x

I dunno :)
(My brain is a bit dead today after just mopping the whole house after the cat was sick everywhere. I may come to a conclusion later after some pondering)

Saturday, January 17th, 2009 03:07 pm (UTC)
You would enjoy The Lady Tasting Tea, I think. (*eyes the subtitle dubiously, because that's not the way I'd characterize the book at all, at all*) Salsburg lays out a history of statistics, emphasizing the philosophical issues, ever-changing technical limits, and the questions clients wanted answered as shaping forces for the field. He also makes it clear exactly how philosophically shaky the current underpinnings of the field are.

One of the things I especially like is that Salsburg roots his narrative in a social world, so you get stuff about how all the early computing power was provided by "computers" -- women who were employed to compute calculations all day, much in form like a secretarial pool -- and how the statisticians running the firm thought women were especially well-suited to that work, since they were so "docile and patient" and could be trusted to not let bursts of creativity disrupt them from the tedium. (It gets better, later, fortunately. In fact, women in stats is in way better shape than women in pure maths.) In terms of social world, you also get to see some of the academic fights, like how math departments missed the boat in not embracing stats as a legitimate, substantive academic field, and then how statisticians did exactly the same thing with computer science. ;-)

And now that I've got the book review out of the way... *puts on stodgy stats-geek hat*

Most stats education is pretty bad -- the vast bulk of it is for people who don't want to know anything more than how to calculate a number to put into a paper or report, and so what most people get is a simple recipe book. What is generally left out -- or rather, mentioned as a set of assumptions at the beginning of every recipe, and which is thereafter ignored by the students who just want their number -- is when and how these techniques break, what they can and can't do, how to decide if the question being answered is the question you wanted answered, let alone decide if that question is even answerable.

It's a discipline with tricksy philosophical issues threaded deeply throughout it, and most of that gets completely ignored by most people who handle or calculate statistics. Which makes me sad, because it's a pretty mind-blowingly awesome field.
Monday, January 19th, 2009 03:37 am (UTC)
That sounds fantastic. *adds to books to check out list*

It's a discipline with tricksy philosophical issues threaded deeply throughout it, and most of that gets completely ignored by most people who handle or calculate statistics. Which makes me sad, because it's a pretty mind-blowingly awesome field.

So my statsy friends keep telling me :)
Saturday, January 17th, 2009 07:15 pm (UTC)
"anything but the most simple species being as likely to get more simple as less"

I suppose you get a similar effect (long, low-population "tail" of higher-complexity organisms) regardless of the exact probabilities of species evolving to more / less complex forms. I wouldn't have thought those probs were necessarily close, or consistent from organism to organism? Cool stuff anyway.
Monday, January 19th, 2009 03:40 am (UTC)
Yeah, that's probably a complete oversimplification (I'm very prone to those). On the whole there's no trend one way or the other, is all.