Revolutions

How ideological is Google?

Adam Bonica, a grad student in political science at NYU, recently published a ranking of the political slant of various professions, based on the amount and recipient (Republican or Democratic) of political donations by lawyers, lobbyists, physicians and many other occupations. This paper (PDF) gives the complete analysis, but the chart below (created using the ggplot2 graphics package in R) sums up the results nicely (click to enlarge): 

(I liked this quote from Paul Kedrosky about this chart: "How come gas station attendants are so damn partisan?") Now, Adam has taken the analysis to the next level, by looking at employees of individual companies, instead of professions as a whole. 

(Unfortunately, it's not entirely clear if the data point lies to the left or at the middle of the company label; this table helps sort out the exact rankings.) It may not be a surprise, for example, that Google's employees tend to give to Democratic-leaning candidates, but does that influence Google's policies as a whole? Adam's article at the link below delves into this question in more detail.

Ideological Cartography: The University of Google: Was the decision to exit China ideological or business as usual?

Scientists misusing Statistics

In ScienceNews this month, there's controversial article exposing the fact that results claimed to be "statistically significant" in scientific articles aren't always what they're cracked up to be. The article -- titled "Odds Are, It's Wrong" is interesting, but I take a bit of an issue with the sub-headline, "Science fails to face the shortcomings of Statistics". As it happens, the examples in the article are mostly cases of scientists behaving badly and abusing statistical techniques and results:

• Authors abusing P-vales to conflate statistical significance with practical significance. A for example, a drug may uncritically be described as "significantly" reducing the risk of some outcome, but the the actual scale of the statistically significant difference is so small that is has no real clinical implication.
• Not accounting for multiple comparisons biases. By definition, a test "significant at the 95% level" has 5% chance of having occurred by random chance alone. Do enough tests, and you'll find some indeed indicate significant differences -- but there will be some fluke events in that batch. There are so many studies, experiments and tests being done today  (oftentimes, all in the same paper)that the "false discovery rate" maybe higher than we think -- especially given that most nonsignificant results go unreported.

Statisticians, in general, are aware of these problems and have offered solutions: there's a vast field of literature on multiple comparisons tests, reporting bias, and alternatives (such as Bayesian methods) to P-value tests. But more often than not, these "arcane" issues (which are actually part of any statistical training) go ignored in scientific journals. You don't need to be a cynic to understand the motives of the authors for doing so -- hey, a publication is a publication, right? -- but the cooperation of the peer reviewers and editorial boards is disturbing.

ScienceNews: Odds Are, It's Wrong

Smoothing time series with R

Smoothing is a statistical technique that helps you to spot trends in noisy data, and especially to compare trends between two or more fluctuating time series. It's a useful visualization tool that I'm pleased to see cropping up more and more in statistical graphics on the Web -- it's now a staple in econometric charts and is heavily used in polling analysis. For example, here's smoothing used to combine data from various polls over time on Obama's job approval (from pollster.com).

The S language was, to the best of my knowledge, the first software that made statistical smoothing a core part of the graphics system: first with the lowess function and later with other more powerful alternatives. These days in R (S's successor), loess (local polynomrial regression fitting) is the usual go-to alternative for smoothing. With just a couple of lines of code, you can take a noisy time series in R and overlay a smooth trend line to guide the eye. Nathan Yau at FlowingData shows us how to take data like this:

and, with just a few lines of R code and some touching-up in Illustrator, create a chart like this:

FlowingData: How to: make a scatterplot with a smooth fitted line

Predicting Pizza

What's the secret to the best pizza in New York? That's what statistical consultant and R user Jared Lander sought to find out, by analyzing the rankings of NY pizza joints at MenuPages.com, and building a regression model for ratings based on variables like localion, price, number of reviews, and pizza-oven type (gas, coal or wood)? Here's a scatterplot matrix of the data set:

Jared published his conclusions in a paper (PDF), "New York Pizza: How to Find The Best". He used a logit analysis in R to model the five-star rank from the various variables. His conclusions? First of all, there's a big discrepancy between critics' "Top 10" pizza rankings and those of the general public (at least as measured at MenuPages.com), with only one of MenuPage's Top 10 listed in the typical critic's list. Secondly, while an Uptown location and a coal oven both popular draws (as measured by number of the reviews) none of the variables have a significant influence in rating:

Our findings were able to discern the factors that go into a pizzeria’s popularity but did not discover much differentiation in quality. Popularity and quality are not always equivalent. It is likely that we may have just proved the old adage about pizza: “Even when it’s bad, it’s still good.”

Slice: The 'Moneyball' of Pizza? Using Statistics to Find NYC's Best Pies and Slices

Future of Open Source Survey - Results

The results of the 2010 Future of Open Source survey were presented at last week's Open Source Business Conference in San Francisco, and here are they are in slide format:

While I was at the presentation I captured a few additional tidbits from the presentation that weren't in the slides. The continued growth of open-source generally was a prevalent theme. For example, did you know that more than 19,000 open-source projects were initiated in 2009? The growth of open-source software in the commercial sector was also noted, and the promotion of open-source projects by commercial open-source companies was cited as a factor. (REvolution Computing promoting the R Project was one example given.) The other theme I spotted was innovation: while lowering costs is still the #1 ranked feature of open-source, access to new methods and the rapid pace of innovation in FOSS compared to proprietary software is now being listed as a critical reason to switch.

R blogger Tal Galili has some other insights on the survey results, from an R user's perspective. I recommend checking them out.

Statistical learning with MARS

Steve Miller at the InformationManagement blog has been looking at predictive analytics tools for business intelligence applications, and naturally turns to the statistical modeling and prediction capabilities of R. Says Steve:

The R Project for Statistical Computing continues to dazzle in the open source world, with exciting new leadership at Revolution Computing promising to align commercial R with business needs.

The technique which is the subject of his most recent post is MARS: Multivariate Adaptive Regression Splines, available for R in the "earth" package:

“Mars is an adaptive procedure for regression, and is well suited for high-dimensional (i.e., a large number  of inputs). It can be viewed as a generalization of stepwise linear regression or a modification of the CART procedure to improve the latter's performance in the regression setting.” 

Read on in Steve's post for a more detailed review of MARS and its applications.

Information Management blogs: Predictive Models, Mars to Earth – Part 1

Video: Hadley Wickham gives a short course on graphics with R

Hadley Wickham (the creator of the popular ggplot2 graphics package for R) has posted video of a 2-hour short course on Visualisation in R at his blip.tv channel. The video is split into four thirty-minute segments:

• Basic Graphics
• Displaying Large Data
• Data manipulation and transformations
• Polishing your plots for publication

The course is peppered with self-guided exercises, for which data, code and the supporting slides are all available. So if you want to be able to create beautiful, informative visualizations of data sets large and small, like this:

then it's well worth two hours of your time to watch these videos. (And don't forget Hadley's book, ggplot2: Elegant Graphics for Data Analysis, for a complete reference.)

Hadley Wickham: data vis mini course

R 2.11.0 scheduled for April 22

Announced this morning on the r-announce mailing list is the impending release of R 2.11.0, scheduled for April 22. As usual, the release soon goes into a beta-test phase, with updated sources to be available on the 22nd and binaries to follow a few days thereafter.

Charting SVN commits with R

Want to get a quick sense of who are the most active committers to your SVN project? Using just a few lines of R code and the SVN log file, reader and new R user Rhys Kidd created this chart to review commits to the Freespace 2 Source Code Project: 

Rhys posts the 6 lines of R code to create the plot in this forum post, and I recreate it here (with some slight reformatting):

system("svn log > fs2_open.svnlog")
x <- readLines("fs2_open.svnlog")
rx <- x[grep("^r[0-9]{1,5} \\|",x)]
who <- gsub(" ","",sapply(strsplit(rx,"\\|"),"[",2))
ctab <- table(who)
dotplot(ctab[order(ctab)],
  scales=list(x=list()),
  xlab="",
  main="Number of fs2_open SVN commits Jan 2007 to Mar 2010")

Note the use of the system command to dynamically create the SVN log -- each time the R code is run, an up-to-date analysis of the commit rates is made. Regular expressions extract the commits and user names from the log file, which are then tabulated for display with the dotplot command. That's some pretty tight code for (in RK's words) a "new-ish R user" -- although he says he did get inspirations from Dirk Eddelbuettel's recent talk at the University of Chicago. (Update: Dirk points out that Ben Bolker originated the idea at UseR 2007. It's a great example of how reading from connections -- here, a log file -- makes for compact, powerful code.)

Incidentally, I was amused by the first response on the forum: "Independent variable...on vertical axis...*twitch*". Actually, a dot chart is a great choice for data like this: the sorting gives a convenient read of the most and least prolific committers; it's easy for the eye to make numerical comparison between rows, and the use of dots instead of the "natural" bars makes for a clean, easy-to-read chart.

The FreeSpace Source Code Project forum: Charting the development of the fs2_open codebase

Because it's Friday: Kittens, beware Tufte

Edward Tufte has been a tireless promoter of good infographics, and he's even taken some controversial steps to rid the world of chartjunk. But now he's gone too far:

 

Then again, this chart from the Wall Street Journal could lead anyone to felinicide:

 

What's wrong with a simple bar chart, WSJ?

Mark Goetz: My New Wallpaper (via @sarahd23 and @statalgo)