Revolutions

A free book on probability and statistics with R

G Jay Kerns has published a 400+ page introductory text on Probability and Statistics. All of the examples and illustrations are done using R (as Jay puts it, "The people at the party are Probability and Statistics; the handshake is R") so if you want to brush up on your probability and learn R at the same time, this might be a good resource. It would also be great for teaching: Jay wrote the book based on an undergraduate course he gave at Youngstown State University. There's also a plug-in for R Commander to access some of the methods via dialogs.

Jay's book is free, in both senses of the word. You can download the PDF for free from Lulu, or purchase a printed copy for just over $30. Jay has also published all of the LaTeX sources if you want to build the book yourself. And if you're already using R, you can read the book with just three commands:

install.packages("IPSUR")
library(IPSUR)
read(IPSUR)

I haven't read the entire book, but glancing through it, it looks like a comprehensive overview of the basics of Statistics: distributions, hypothesis testing, estimation, linear regression, and even touches on resampling and nonparametric methods.

I did want to point out one minor error on page xiii, though. It's "We're not in Kansas any more, Toto", not "We're not in Kansas any more, Dorothy". They'd take away my Rainbow Card if I didn't mention that. 

G Jay Kearns: Introduction to Probability and Statistics using R

New StackOverflow site for Statistics

Since launching StackOverflow.com as a Q&A forum for programmers, the StackExchange network is now launching about three new sites a week on topics as specialized as cooking, jobs, and many many more. There's a well-defined, community-oriented process for setting up a new site: first it must be proposed, and then enough potential users must commit to the site and define its mission with sample questions. Finally, with enough committed users it enters a beta phase before being formally launched.

A new site devoted to questions about Statistics has completed the proposal and definition phases, is about to enter the beta phase. A typical question, voted as an example of the type of question appropriate to the site, is:

Can you give an example of where I might prefer to use a z-test vs a t-test?

The more people that participate in the beta and support a thriving ecosystem of questions and answers, the more likely it is that the Statistics StackExchange will be a reality. You can't ask or answer questions at the site just yet, but you can commit to be a participant in the site and you'll receive an invitation to the beta test. If you're interested, follow the link below and click the "commit" button.

Update: The private beta has now started, so if you hadn't already committed, you'll have to wait until the public beta starts at noon Pacific time on Monday, July 26.

StackExchange Area 51: Statistics 

The case of the missing zeroes

Political polling is a big industry these days, especially here in the US, and both mainstream news outlets and many of the bigger political blogs commission their own polls to measure (for example) the popularity of a sitting or candidate politician or policy. In the last week though, a very public spat has erupted between the left-leaning political blog Daily Kos and the polling firm Research 2000, commissioned by Daily Kos to produce a series of polls for the site. An investigation by statistician-readers of Daily Kos suggested some unusual features of the polling data provided by Research 2000, mainly that the results from tracking polls appear to be underdispersed compared to what you'd expect given the sample sizes. Other results suggest data drawn from unexpected distributions, as illustrated for example by this histogram of week-to-week changes in a measure of Obama's favorability over a period of 60 weeks:

This and other anomalies in the data have led Daily Kos to sue Research 2000, claiming that they "handed Daily Kos fiction and claimed it was fact". (On the other hand, Nate Silver of fivethirtyeight.com, while having his own reservations about Research 2000, suggests a less nefarious explanation for modifying data by hand.)

In any case, this story is an illustrative example of how statistical concepts (randomness, distributions, samples, data) have increasing prominence in political discourse today -- largely as a result of this increased use of polling -- and how the stakes have been raised by the data they generate.

Daily Kos: Research 2000: Problems in plain sight

Conditional probability: an easier way

Conditional probabilities are bane of many students of Statistics, but statements of conditional probability come up surprisingly often in real life. For example, as Steven Strogatz writes in the New York Times, when doctors are asked to estimate the probability that a woman has breast cancer given a positive mammogram test result, most get the answer wildly wrong despite being given the population frequency of breast cancer and the conditional probability of false positives from a mammogram test. Here's one doctor's experience trying to come up with a number:

“[He] was visibly nervous while trying to figure out what he would tell the woman.  After mulling the numbers over, he finally estimated the woman’s probability of having breast cancer, given that she has a positive mammogram, to be 90 percent.  Nervously, he added, ‘Oh, what nonsense.  I can’t do this.  You should test my daughter; she is studying medicine.’  He knew that his estimate was wrong, but he did not know how to reason better.  Despite the fact that he had spent 10 minutes wringing his mind for an answer, he could not figure out how to draw a sound inference from the probabilities.” [The correct answer is 9 percent.]

Most students (and doctors!) are taught to use Bayes' Theorem to calculate marginal probabilities from conditional probabilities, but as Strogatz point out this isn't exactly an intuitive calculation, with the dividing of probabilities by probabilities and all. He suggests a more intuitive (but slightly less accurate) method is to think instead about frequencies within concrete groups and sub-groups. For the mammogram test, the calculation becomes:

Eight out of every 1,000 women have breast cancer.  Of these 8 women with breast cancer, 7 will have a positive mammogram.  Of the remaining 992 women who don’t have breast cancer, some 70 will still have a positive mammogram.  Imagine a sample of women who have positive mammograms in screening. How many of these women actually have breast cancer?

Since a total of 7 + 70 = 77 women have positive mammograms, and only 7 of them truly have breast cancer, the probability of having breast cancer given a positive mammogram is 7 out of 77, which is 1 in 11, or about 9 percent.

This method is frowned upon by textbooks, because it's not as accurate (in the example above, rounding to whole numbers of women in the groups), and because it implicitly assumes that the frequency of the event (here, breast cancer) is determined solely by the probability, with no accounting for variation. But it is an intuitive method for understanding conditional probability, that seems more likely (ha!) to come up with an reasonably accurate answer for many people.

Read the rest of Strogatz's article for other examples of intuitive conditional probability calculations, including a great example from the OJ Simpson trial.

New York Times Opinionator: Chances Are

R and the Next Big Thing

I've been travelling for the past few days (for the R/Finance 2010 conference in Chicago), so I'd missed much of the reaction to AnnMaria De Mars' article last week where she claimed that "R is an epic fail". Understandably, that inflammatory statement provoked many reactions from the R community on Twitter and in the blogosphere. (I suspect the fact that she was attending a SAS conference when writing the post only added fuel to the fire.) Yihui Xie was the first to bring attention to the article, and Drew Conway followed up with a detailed and well-reasoned response, and Tal Galili provides a great round-up of other responses along with his own commentary.

Now that I'm back at my desk and have had a chance to reflect on the post and its responses, it seems like the disconnect is less around what R does, but rather who uses it. According to De Mars, "The vast majority of people are NOT programmers. They are used to looking at things and clicking on things." And for the most part, I think that's true: most of the people who use R are statisticians, and there are more non-statisticians than statisticians in the world. And R does have a steep learning curve today.

But I think the point that's being missed here is that different communities have a different concept of what statistical analysis is. It's almost a generational difference: if your statistical education was based around SAS or SPSS, it seems that your view of statistics is a very procedural one: for this kind of problem, fit that kind of model, and look at these results. That's a worldview that's easily accommodated by the procedural nature of SAS programming, or the regimented nature of point-and-click GUIs such as one finds in SPSS.

But there's a newer -- and, for the most part, younger -- cohort of statisticians out there now, who have a different view of statistics. For them, statistics is less of a set of rigid procedures and more of a fluid process. A process where trying out different data transformations and models is encouraged. Where innovative visualizations of data are created. Where cross-discipline fertilization is commonplace, and models from (say) biostatistics are successfully applied to marketing data. I meet the members of this cohort mainly within industries where innovation is particularly valued: places like Web 2.0 and social-media companies; hedge funds; drug discovery; and marketing optimization at a greater rate than I do in "traditional" venues for statisticians. And most of them have been trained in R.

Can these two communities ever be bridged? I think it's inevitable. The need for data analysts is only going to grow further, but given that the way we think about data analysis and statistics has changed, so the tools we use in the future are going to need to accommodate that change. R's flexibility (the programming language) and innovation (the exponential growth in add-on packages) driven by the open-source community is what makes it attractive to those organizations where innovation is paramount. (And given competitive forces, innovation in data analysis will soon be paramount for most.)  But it's true: there will always be more non-programmers than programmers, and even they are going to need a software platform that supports a freer, less procedural style of data analysis. "Statistics 2.0", if you will.

So why can't R be that platform? What if we could bring the innovation and flexibility of R, and make it available to non-programmers? I think it's possible: in fact, making R accessible to more users (and therefore customers, natch) has been a focus area for REvolution since its inception. As it happens, I'm looking forward to sharing some exciting developments in an extensible and flexible user-interface for large-scale data analysis that we've been prototyping for R. R is a lot more than a programming language, and it will certainly become much more in the future. In fact, I'd second Joe Dunn in saying:

"I will not be surprised if in ten years R is the standard for statistical data analysis, much as Linux has supplanted commercial UNIX and gone on to explore territory that its predecessor never touched (look at Ubuntu). R may not be the next big thing, but R is certainly a big thing that is forthcoming."

AnnMaria’s blog: The Next Big Thing

Arizona court rules statistical sampling is legal

A court in Arizona has ruled that statistical sampling is legal for determining damages awarded to individual claimants when there are thousands of similar cases to be assessed simultaneously. In a case where 30,000 claims were filed Maricopa County, AZ by hospitals for improper reimbursement, the trial judge appointed a former judge as a special master in the case to resolve all the claims within 24 months. The special master in turn hired a statistician who drew a sample of 1,150 claims that would be used as the basis for all claims.

The county appealed this process, claiming due process rights of the hospitals were violated. The appellate court ruled that sampling was valid, but sent the case back to the trial court to validate that an appropriate sampling process was used. that the viability of the sampling methods or even whether the methods chosen were properly implemented and complies with all the necessary legal principles.

The case is Scottsdale Memorial et al. v Maricopa County CA-CV 07-0150.

http://www.azcentral.com/business/abg/articles/2010/04/08/20100408abg-fischer0408.html

A free book on Geostatistical Mapping with R

Tomislav Hengl of the University of Amsterdam has published new book, A Practical Guide to Geostatistical Mapping. It's jam-packed with 291 pages on mapping and analyzing spatial data using free software including R, SAGA, GRASS, ILWIS and Google Earth, and freely-available map data. The book itself is also available for free, as an Open Access Publication. You can order the book in printed form for US$12.78, or download it for free as a PDF.

Surprisingly (given the title), this book isn't just about visual displays of spatial data. In fact, the first two chapters offer a nice overview of statistical analysis of spatial data (although with a greater focus on continuous-field models than point-process models). If you want a concise overview of regression-kriging, this is a great resource.

Chapter 3 addresses the various software tools you'll use to analyze the data and create the maps. Some care has been taken in considering how the software elements should be integrated, and Hengl recommends a "R on top" model, where R scripts drive the other tools. 

This is a clever move: making use of the scripting capabilities of R means you can avoid much of the tedious manual back-and-forth activities that are usually associated with working with several software tools. Hengl offers some other reasons for working with R, too (p. 90):

• It is of high quality — It is a non-proprietary product of international collaboration between top statisticians. 
• It helps you think critically — It stimulates critical thinking about problem-solving rather than a push the button mentality. 
• It is an open source software — Source code is published, so you can see the exact algorithms being used; expert statisticians can make sure the code is correct.
• It allows automation — Repetitive procedures can easily be automated by user-written scripts or functions.
• It helps you document your work — By scripting in R, anybody is able to reproduce your work (processing metadata). You can record steps taken using history mechanism even without scripting, e.g. by using the savehistory() command.
• It can handle and generate maps — R now also provides rich facilities for interpolation and statistical analysis of spatial data, including export to GIS packages and Google Earth. 

Chapter 4 covers the various auxiliary data sources available, listing sources global environmental and socio-economic data, and sources of maps and satellite imagery like GADM, Google Earth and MODIS. 

The remaining chapters are devoted to worked examples of spatial data analysis and mapping. By working through the examples, you can recreate charts like these (click to enlarge):

One minor complaint: most of the images in the book are in black-and-white (most likely to facilitate the printing process). But at least you have the R scripts and data for all exercises (these, plus updated maps, are available from the book's website), so at least you can re-run the examples in R to recreate them in color.

Tomislav Hengl: A Practical Guide to Geostatistical Mapping (via @fernando_mayer)

How ideological is Google?

Adam Bonica, a grad student in political science at NYU, recently published a ranking of the political slant of various professions, based on the amount and recipient (Republican or Democratic) of political donations by lawyers, lobbyists, physicians and many other occupations. This paper (PDF) gives the complete analysis, but the chart below (created using the ggplot2 graphics package in R) sums up the results nicely (click to enlarge): 

(I liked this quote from Paul Kedrosky about this chart: "How come gas station attendants are so damn partisan?") Now, Adam has taken the analysis to the next level, by looking at employees of individual companies, instead of professions as a whole. 

(Unfortunately, it's not entirely clear if the data point lies to the left or at the middle of the company label; this table helps sort out the exact rankings.) It may not be a surprise, for example, that Google's employees tend to give to Democratic-leaning candidates, but does that influence Google's policies as a whole? Adam's article at the link below delves into this question in more detail.

Ideological Cartography: The University of Google: Was the decision to exit China ideological or business as usual?

Scientists misusing Statistics

In ScienceNews this month, there's controversial article exposing the fact that results claimed to be "statistically significant" in scientific articles aren't always what they're cracked up to be. The article -- titled "Odds Are, It's Wrong" is interesting, but I take a bit of an issue with the sub-headline, "Science fails to face the shortcomings of Statistics". As it happens, the examples in the article are mostly cases of scientists behaving badly and abusing statistical techniques and results:

• Authors abusing P-vales to conflate statistical significance with practical significance. A for example, a drug may uncritically be described as "significantly" reducing the risk of some outcome, but the the actual scale of the statistically significant difference is so small that is has no real clinical implication.
• Not accounting for multiple comparisons biases. By definition, a test "significant at the 95% level" has 5% chance of having occurred by random chance alone. Do enough tests, and you'll find some indeed indicate significant differences -- but there will be some fluke events in that batch. There are so many studies, experiments and tests being done today  (oftentimes, all in the same paper)that the "false discovery rate" maybe higher than we think -- especially given that most nonsignificant results go unreported.

Statisticians, in general, are aware of these problems and have offered solutions: there's a vast field of literature on multiple comparisons tests, reporting bias, and alternatives (such as Bayesian methods) to P-value tests. But more often than not, these "arcane" issues (which are actually part of any statistical training) go ignored in scientific journals. You don't need to be a cynic to understand the motives of the authors for doing so -- hey, a publication is a publication, right? -- but the cooperation of the peer reviewers and editorial boards is disturbing.

ScienceNews: Odds Are, It's Wrong

Clustering the world's diets

Cluster Analysis is a useful technique for classifying the members of a group (people, events, measurements, etc) into "similar" groups. How "similar" is defined depends on the application, but generally involves looking at a number of attributes of the group. For example, we could cluster people by looking at their skin color, hair type, facial features, perhaps even genetic markers and find that we end up with clusters that are somehow associated with ethnicity. 

Here's a fascinating application of cluster analysis: given data on what the citizens of each country eat (on aggregate), can we cluster the countries of the world into groups with similar diets? That's what Diego Valle did, using the pam (partitioning around medioids) function in R. He presents the six clusters he identifies as a color-coded world map (click to enlarge):

Australia gets grouped with North America and much of Europe and Russia as countries whose citizens enjoy a high-calorie diet with all kinds of foods (except not many beans). Countries in yellow have a cereal-rich diet. The diets of the south-east Asian cluster are heavy on fish and rice, but not dairy foods. See Diego's blog for the description of the other clusters, and the R code which created the analysis. The code reads the data directly from a Google Spreadsheet in the cloud, so you can easily run it yourself. It also produces an interesting chart comparing the American diet to that of the rest of the world.

Diego Valle's Food & Fishing Blog: Cluster Analysis of What The World Eats