Revolutions

R is Hot: Part 5

This the final installment of a five-part article series. You can download the complete article from the Revolution Analytics website.

Building a Business 

The value of R to business is borne out by the experiences of John Lucker and his team of advanced analytics professionals at Deloitte Consulting LLP. John is a Deloitte Consulting Principal and leads the firm’s Advanced Analytics and Modeling (AAM) practice, one of the leading analytics groups in the professional services industry.

When the group was launched fourteen years ago, its main focus was solving vexing business problems for clients in the insurance industry. One of the challenges facing the industry was the lack of robust analytic processes for supporting critical underwriting decisions. This challenge was particularly acute in the rapidly growing commercial insurance industry. “The commercial insurance underwriting process was rigorous but also quite subjective and based on intuition,” says John. 

Convincing experienced underwriting management to change their time-honored traditions was not an easy task.  That’s where R proved exceptionally helpful in recent years. “R enables us to communicate our analytic results in appealing and innovative ways to non-technical audiences through rapid development lifecycles,” says John. “Sometimes people know they have a problem, but they don’t know how to fix it. And sometimes they don’t even know they have a problem. R helps us show our clients how they can improve their processes and effectiveness by enabling our consultants to conduct analyses efficiently.”

The group’s success with clients in the insurance industry became a blueprint for expanding into new markets. Today, the Deloitte Consulting Advanced Analytics and Modeling practice also serves clients in healthcare, banking and financial services, retail, consumer products, telecomm, automotive, media, hospitality, public/state/federal sector and other major industries. R played an important role in growing the practice by allowing it to offer robust analytics addressing the specific needs of clients in a variety of markets.

“I find the diversity of R solutions and add-ons very appealing,” says John. “R has served as a catalyst in the marketplace. It forces everyone to raise their game, and it incents software developers to enhance their offerings. Everybody becomes more competitive and users of analytic tools benefit.”

Changing, transforming and evolving

With thousands of contributors and two million users worldwide, R is a truly global phenomenon. Unlike traditional commercial software for data analysis, R is both flexible and extensible. Supported by an active community of users and developers, R is constantly changing to meet the changing needs of our rapidly shifting global economy. 

The popularity of R is no fluke or fad. R has become the common language of data analysis because it was designed – from the ground up – as a practical system for handling the real-world challenges of complex data sets. R-based programs are applied routinely to solve problems in real-time trading, finance, risk assessment, forecasting, biotechnology, drug development, social networking and more.

But the wide acceptance of R as the lingua franca of statistics is based on its unique ability to change, to transform and to evolve. When new techniques in statistical analysis are discovered, they tend to emerge as R packages first – years before those innovations are incorporated in traditional enterprise software products.

Thanks to its open-source roots, R has spread virally across the map. It has become both ubiquitous and indispensable. The R community supports development, innovation and continuous improvement. New players are welcome and encouraged. The R eco-system has become a fertile breeding ground for novel ideas and original ways of thinking about numbers.

No one can foretell the future of quantitative analytics, but it’s safe to wager that a good deal of it will be written in R.

Read all parts of this series

Keeping up with election results, with R

Yesterday's US election is pretty much over now: most of the results are in, the pundits have offered their political analysis, and there's even been a bit of mathematical analysis of the results, too. But last night as the results were flowing in, R user Brock Tibert just wanted to track the results of the Massachusetts governor's race. The Boston Globe was providing regular updates to the precinct level counts, but it's hard to visualize the horse-race from the raw numbers. So with a little hackery and a ingenuity, Brock wrote some R code to scrape the results table from the Boston Globe webpage and visualize the results as a barplot. Here's an excerpt of his code (from his gist page): 

# grab the tables from the results page
tables <- readHTMLTable(URL)
 
# take only the largest one
n.rows <- unlist(lapply(tables, function(t) dim(t)[1]))
results.temp <- tables[[which.max(n.rows)]]
 
# lets clean up the data a little....
names(results.temp) <- c("city", "pctreport", "baker", "cahill", "patrick", "stein")
 
for(i in c(3:6)) {
	# convert to number, but need to remove the comma that gets pulled from the site
	results.temp[, i] <- as.numeric(as.character(gsub(",", "", results.temp[, i])))
}
 
results.temp$city <- as.character(results.temp$city)
  
# create a city/town detail dataset
results.detail <- results.temp[1:nrow(results.temp)-1, ]  # remove temp row
 
# create a "dataset" that has the totals added up for you
totals <- results.temp[nrow(results.temp), ]
totals
 
# a basic plot
plot.data <- as.vector(t(totals[,3:6]))
names(plot.data) <- c("Baker", "Cahill", "Patrick", "Stein")
race.plot <- barplot(plot.data,
		main="2010 Gubernatorial Results using R",
		xlab="Candidate")
text(c(1:4), y=20000, labels=plot.data)

Brock used the XML package to extract the table from the webpage, the stringr package to process the text results, and (after arranging the data), the barplot function to visualize the results.

And here's the resulting bar plot (which I ran today, so it now represents the final tallies):

Brock said in this tweet: "Not elegant, but works.". Perhaps true, but that tweet making the code available was sent at 9PM East Coast time, not long after the polls had closed and as the returns were coming in. It may not be elegant, but it's a really impressive way to visualize the results in real time.

Another lottery coincidence

Last week, the Freakonomics blog in the NYT reported that the Israeli lottery had drawn the same six numbers twice in a month. The seventh "bonus ball" was different, but still: quite a coincidence, right? Cue the quote from an expert to explain just how remarkable this is:

Yitzhak Melechson, a statistics professor at the University of Tel Aviv, said that “the incident of six numbers repeating themselves within a month is an event of once in 10,000 years.”

Hmm, not so much. As Xi'an pointed out, there's actually about an 8% 8 in 1000 chance of this having occurred in the two years the Israeli lottery has been run under these rules. And then if you consider all the lotteries around the world, as we did when the Bulgarian lottery had back-to-back identical drawings just over a year ago, then this isn't looking so surprising after all. It's likely that Yitzhak Melechson was misquoted, but still, as Gelman says, it's surprising the guys at Freakonomics missed this one.

Update: Corrected estimate of the probability of coincidence. Apologies for the error, which was mine, not Xi'an's.

Freakonomics blog: Very Long Odds in the Israeli Lottery

SAS vs Open Source, ctd

Following up on the story from last week, where SAS CEO Jim Goodnight said he "hadn't noticed" competition from open-source alternatives, open-source BI vendor Pentaho's "Chief Geek" James Dixon responds:

What this means is that SAS has moved from the Igorance phase to the Ridicule phase of battling open source, they only have Fighting and Losing to go.

There are some interesting parallels with Microsoft's reaction to the insurgency of Linux 10 years ago, as James analyzes in the full post linked below.

James Dixon’s Blog: SAS under pressure from Pentaho

Google TechTalk on integrating R

As noted on the Google Open Source Blog last week, R package authors Dirk Eddelbuettel and Romain Francois recently gave a presentation on R at the Googleplex, on various topic related to "bridging" R into other systems. Their 90-minute talk is available for replay on YouTube (as part of the Google TechTalks series), and you can download the slides from Dirk's blog. For the Google audience, I'm sure it prompted many ideas on how to further marry R into analytical systems there.

The first part of the talk is all about how to speed up R by recoding tight loops in C++ and calling them from R via the .Call function. Without help it could be tricky to whip up a C++ function to call from R, but Dirk and Romain have made the process hassle-free by providing the Rcpp package, which provides a high-level mapping between R and C++ object types. Next, they give examples of the inline package by Oleg Skylar et al, which makes it easy to embed C++ code in R functions, and automates the process of compiling and linking the C++ code so you don't have to worry about it. They also discussed their Rinside package, which makes it possible to call R directly from a C++ program (and of course share objects and data via Rcpp). Finally, they gave examples of integrating R with protocol buffers (a widely used mechanism used at Google for serializing data for exchange between different systems) with the RProtoBuf package.

The second part of the talk focused on the various ways possible to build data structures in R: S3 classes, S4 classes and now the new reference classes available in R 2.12. These reference classes (which they call R5 classes) are objects with components which are not copied when the object is copied (they are passed by reference, not by value). There's a detailed overview of reference classes in the "methods" package of the R documentation, under ReferenceClasses. The audience of the talk included John Chambers who designed R (and S's) class systems. He asked the first question during the discussion period, so be sure to listen through to the end.

Open Source at Google: Integrating R with C++: Rcpp, RInside, and RProtobuf