Revolutions

Because it's Friday: Gangnam Style

Apparently I'm late to the table with with Psy's Gangnam Style, but now that I've seen it I can't get it this infectious Korea-pop out of my head:

 

Makes me want to visit Seoul again (and Andrew Sullivan has more on the local nods). I'm sure I'll be listening to this more over the long weekend (while the blog takes a break on Monday). Hope you enjoy yours!

RStan: Fast, multilevel Bayesian modeling in R

For the last decade or so, the go-to software for Bayesian statisticians has been BUGS (and later the open-source incarnation, OpenBugs, or JAGS). BUGS is used for multi-level modeling: using a specialized notation, you can define random variables of various distributions, set Bayesian priors for their parameters, and create the network of relationships that describe how the random variables influence each other. To apply these models to data requires some heavy-duty number crunching: hundreds or thousands of simulations of every random variable in the model (of which there can be dozens), all in parallel. This has generally been done using a technique called Markov-Chain Monte Carlo (MCMC) with the Gibbs Sampler. But as Andrew Gelman found, applying this algorithm to complex models was showing the strain:  

"The models we wanted to fit turned out to be a challenge for current general- purpose software to fit. A direct encoding in BUGS or JAGS can grind these tools to a halt. Matt Schofield found his multilevel time-series regression of climate on tree-ring measurements wasn’t converging after hundreds of thousands of iterations."

To solve this problem, Gelman and collaborators from Columbia University announced last night that they have created STAN: new, high-performance open-source software for Bayesian inference on multi-level models. Stan has all the generality and ease of use of BUGS, and can solve the multilevel generalized linear models described in Part II of the book Data Analysis Using Regression and Multilevel/Hierarchical Models. Rather than the traditional Gibbs sampler, Stan uses a variant of Hamiltonian Monte Carlo (HMC) to speed up calculations. Stan supports truncated and/or censored data, and so can be used to fit survival and reliability models with non-standard (or even user-defined) probability distributions. By automatically converting models to compiled C++ code, Stan can solve such complex models with non-standard distributions quickly. (Check out these comparisons of Stan's performance with BUGS and JAGS.)

While Stan has a command-line interface, it's most easily used via the R package interface, RStan. R users will feel right at home with the Stan syntax used to describe multi-level models. For example, the code below describes a model with bivariate covariance matrix in which the variances are known, but the covariance is not:

data {
       int N;
       vector[2] y[N];
       double var1;  double var2;
     }
     transformed data {
       double max_cov;  double min_cov;
       max_cov <- sqrt(var1 * var2);
       min_cov <- -max_cov;
     }
     parameters {
       vector[2] mu;
       real cov;
     }
transformed parameters {
  matrix[2,2] sigma;
  sigma[1,1] <- var1;
  sigma[2,1] <- cov;
  sigma[1,2] <- cov;
  sigma[2,2] <- var2;
} model {
      for (n in 1:N)
        y[n] ~ multi_normal(mu,sigma);
}

For more information about Stan, check out the Stan home page and especially the comprehensive (178-page) manual. R users should check out the page Running Stan from R for information about the RStan package. The discussion group for Stan users is already quite active.

Three ways of visualizing the growth of Walmart

It's a wonderful thing when people make interesting data sets available to the public. When Thomas Jones wrote a paper in Econometrics about the growth of US retail giant Walmart, he made the data he collected about every Walmart store opening in history (location and date) available to the public. Since then, several people have used different techniques to visualize the data. 

Biostatistician Corey Chivers used the R language to create this animation of the growth of the mega-retailer over time:

 

Corey used an R script to create this animation above, which you could easily adapt to other geolocated data to show accumulation over time.

FlowingData's Nathan Yau used Actionscript to create a similar animation (including both Walmart and Sam's Club openings; Sam's Club is the big-box warehouse variant of Walmart) a while back.

 

Finally, this animated GIF of Walmart store openings appeared at the blog All That is Interesting. It was created using Excel.

Many thanks go to Thomas Jones for making the source data available for everyone to use.

R does CSI: Using R to nail break-in suspects

You've probably heard (or seen in TV shows) how the unique pattern of rifling in a gunbarrel generates forensic evidence: microscopic scoring on the bullets left at the scene of the crime can be linked to the shooter by matching the marks to the firearm. What you might not know is that the same technique can be applied to other crimes: for example, scoring on a screwdriver blade can leave identifying marks on a broken-into window frame, which can then be linked to a screwdriver-possessing burglary suspect.

With screwdrivers, the process is a little more complex: there's never an exact match between scoring on the screwdriver and the marks left behind, and the angle at which the screwdriver is used affects the marks as well. But at the JSM 2012 conference, Iowa State University R user Amy Hoeksema presented a method of measuring the "fingerprint" of marks on the screwdriver tip (as shown below), and then using a statistical model implemented in R to decide if it's a statistically significant match to marks found at the scene of the crime. 

  

You can find the details of Amy's method in her JSM poster, linked below (PDF). Using this method, she can not only determine whether the suspect's tool was used, but even at what angle it was used, within 5 degrees! Amy told me that this method hasn't yet been used to solve any crimes to date, but I'm sure CSI departments would welcome a new tool like this in their crime-fighting arsenal.

Amy Hoeksema: Statistical Comparison of Tool Marks in Forensics

Revolution Newsletter: August 2012

The most recent edition of the Revolution Newsletter is out. The news section is below, and you can read the full August edition (with highlights from this blog and community events) online. You can subscribe to the Revolution Newsletter to get it monthly via email.

Top Innovator for Data Science Technology: Revolution Analytics is proud to be recognized as a Top Innovator for Data Science Technology by DataWeek. Join us at the DataWeek 2012 Conference and Festival in San Francisco Sep 24-27 for the R Lab and other data science events. Use code DataWeek25 for a 25% registration discount.

Revolution Analytics expands to Asia. We're excited to announce the latest outpost of the Revolution Analytics team is now open in Singapore, with participation from the Infocomm Development Authority of Singapore. 

Williams Sonoma uses Revolution R Enterprise. By teaming up with UpStream Software and their online marketing analytics platform, the kitchenware and home furnishings retailer makes use of advanced statistical models and millions of records per day to optimize marketing operations. 

Public R Training Courses. A few seats remaining for Visualization in R with ggplot2 (in Redwood City, CA on August 28). Or attend one of our online live-instructions courses from your desktop: Big Data Analytics with RevoScaleR (Sep 6-7), Intro to R for Data Mining (Sep 13 & 20), and Marketing Analytics with R (Sep 25-28).

Big Data Innovation in Boston. Come see how Revolution R Enterprise is used for Big Data Science at the Big Data Innovation conference in Boston, September 13-14. 

Call for Beta Testers. The next release of Revolution R Enterprise will include Big Data decision trees, enhanced XDF file format for R with file compression, and the ability to analyze data from the Hadoop Distributed File System (HDFS).  Please contact [email protected] if you are interested in participating.

Revolution Analytics: Newsletter Archive

Arctic sea-ice at lowest levels since observations began

RealClimate.org used the R language and data from the National Snow and Ice Data Center to create this chart showing the extent of Arctic sea-ice in each year since satellite observations began in 1978, and the current extent of ice coverage (in red). Even though there are several weeks of annual melting yet to come, the area of ice covering the north pole is at already at a record low:

The record low is the horizontal red line. You can track how far this year's melt will exceed that record by running this R script, which will download the latest data from NSIDC to create an updated chart. (R users may also be interested in the use of readJPEG and rasterImage to create the background for the chart.)

Putting those time series into geographic context, NASA has also released the visualization below, which shows the average minimum extent of sea ice compared with a current satellite view. The yellow line is the average ice pack of all previous years, i.e. the low point of each of the white lines above.

BadAstronomer has more details about how the image above was made.

RealClimate.org: An update on the Arctic sea-ice

Two R community milestones

Two significant R community milestones were achieved over the weekend.

Firstly, the number of community-contributed R packages on CRAN is now above 4000. (As of this writing, it's 4004.) Figure 10 of The Popularity of Data Analysis Software charts the exponential growth of R packages: at the end of last year the figure stood at 3500, and the number has doubled in just 3 years. And this isn't even counting packages at non-CRAN repositories: there are an additional 553 packages available on BioConductor, and there are untold numbers of R packages in development at github, R-forge, and at private repositories.

The second milestone is that this very blog has now had more than 2 million visits since its inception. I started the Revolutions blog back in December 2008, back before there were any other blogs solely or primarily devoted to the R language. (Today, there are 376 R blogs.) The most-visited post of all time, other than the home page, is the one about random numbers in R. Thanks to everyone for visiting Revolutions, and for all the great feedback and suggestions. Keep 'em coming.

Because it's Friday: Bad dogs on parade

Anyone who has dogs knows that they can get into a mighty amount of mischief. But I'd never have guessed all the different types of mischief that dogs can get into, until I came across the Dog Shaming website. Here you'll find pictures of guilty-looking dogs (and a few cats) with notes describing their nefarious deeds, like "I locked myself in the car. They had to call a locksmith", and "I hid meat in the couch". I'm glad our dogs aren't the only ones with shameful behaviour! Here's an entry from our 7 year old rat terrier / Jack Russell mix, Easy:

There's more at dog-shaming.com. Enjoy the schadenfreude over the weekend!

Does playing baseball shorten your lifespan? (Answer: No.)

A National Institute for Occupational Safety and Health study, published in March, found that professional American football (NFL) players lived longer, on average, than similar "mere mortals" in the general population. Football is a dangerous sport, so that might seem surprising at first, until you consider the fact that NFL players are elite sportsmen: only the strongest, fastest and most healthy members of the population get a chance to play. On top of that, they have access to excellent healthcare, even after they retire.

But it's important to note what this study does NOT say: it does not claim that playing professional football will make you live longer. (In fact, throwing a couple of dozen men selected at random into a scrimmage with the Dallas Cowboys is most likely to shorten their lifespan, given the inevitable injuries that would entail.) All it says that the population of men selected to play in the NFL tend to live longer than similar counterparts in the general population.

Sports Journalist Bill Barnwell attempted to answer the question by using a different method: why not compare NFL players to baseball players, who are also elite athletes. But as R user and biostatistician Gregory Matthews reports, Barnwell's "Mere Mortals" article makes some profound statistical errors in making the claim that MLB players live shorter lives than NFL players.

The basic error is that the populations of MLB players and NFL players are not directly comparable, mainly because baseball players tend to be older than football players. Matthews created the age distribution chart below to illustrate the difference (blue is baseball, red is football):

If you merely count the number of baseball players that have died and compare that to the number of football players have died, you'll find more baseball player deaths:

	Baseball	Football
Qualifying Players	1,494	3,088
Alive	1,256	2,694
Deceased	238	394
Mortality Rate	15.9 percent	12.8 percent

Even though that difference is statistically significant (using a Fisher Test) this is hardly surprising: because the average baseballer in the sample was older than the average footballer, you'd expect to find more deaths in the interim. But if you include player age in a logistic regression model, as Gregory Matthews used R to do, the effect of the sport (baseball vs football) disappears. It's only the relative age differences between the sports that causes the discrepancy in the mortality rates above.

There are two lessons to learn from this tale:

• When you read reports in the newspaper about how 'X causes Y', always ask yourself whether the direction of causality is correct. While researches and journals are (for the most part) careful to merely report that there's an association between X and Y, journalists often upgrade that to direct causality. Sometimes it's just the case that people who are afflicted by Y tend to be people who are likely to have consumed/performed/been exposed to X.
• Unless a formal random trial has been conducted, whenever two (or more) groups A and B are compared, it's amost never appropriate to directly compare averages or other statistics with the two groups. The analysis must control for any factors (age, sex, location, lifestyle, genetics, ...) that might influence the outcome, using regression or other appropriate statistical techniques. And even then, you can never be certain that all possible variables have been controlled for: it can strengthen the confidence of the association, but never prove causality outright.

Thanks go to Gregory Matthews for illustrating these important lessons in his critique of Bill Barnwell's "Mere Mortals" article, linked below.

Stats in the Wild: Mere Mortals: Retract This Article

Revolution Analytics receives Top Innovator award for Data Science Technology

A big thank-you to all the R users out there who voted for Revolution R Enterprise in DataWeek Awards. We're so pleased to be recognized by the voters and the DataWeek judging panel with the Top Innovator Award for Data Science Technology. We're looking forward to the awards ceremony next week at DataWeek SF (in San Francisco, September 24-27). If you'd like to come along and join us and 1000 of the other most innovative data-centric companies who are attending, speaking, and participating in this first annual event, you can register here with the code DataWeek25 for a 25% discount. 

DataWeek is the first week-long conference and festival to showcase the innovations behind the Data Revolution. The founders of Techweek in Chicago and the Data 2.0 Summit in San Francisco are organizing this year’s DataWeek 2012 Conference & Festival, September 22-27 in San Francisco, CA. Join over 200 speakers from Google, Twitter, IBM, Cloudera, CitrusLeaf, Dun & Bradstreet, and hundreds of other data-centric companies for a 4-day conference inside a week-long series of data events, workshops, hackathons, and hiring mixers, and meetups!

You can find more information about DataWeek and Revolution Analytics' Data Science Technology Top Innovator Award at the link below.

Revolution Analytics Press Releases: Revolution Analytics Wins Top Data Science Technology Innovator Award by DataWeek