Revolutions

How UpStream uses R for Attribution Analysis

Major retailers like Williams Sonoma use UpStream Software for marketing analytics, including revenue attribution, targeting, and optimization. In the video below Tess Nesbitt (senior statistician at UpStream) describes how she uses Revolution R Enterprise and Hadoop to figure out the impact on various marketing channels (for example direct mail, email offers, and catalogs) on consumer retail sales.

 

(The slides for Tess's presentation, The Impact of Big Data on Marketing Analytics, are available for download.) Because Tess needs to massive amounts of consumer behaviour data to tease out the effects of the different marketing channels, she switched from SAS/WPS and now uses the big-data capabilities of Revolution R Enterprise to fit a survival model to more than 36 million records in less than 4 minutes. (You can find some of the details of the model and the R code used to fit it in Tess's recent presentation to the Bay Area R User Group, Statistical Marketing Analytics with Big Data (PPT).) By reducing the time it takes to fit the model from an overnight wait to the time it takes to make a cup of coffee, Tess has more opportunity to refine and improve the model, as she describes in a recent Channel Reseller News article:

"Revolution R Enterprise 6.2 enables UpStream to build highly efficient statistical models on extremely large data. Because their parallelized algorithms are so efficient, it enables us to take multiple passes at the data, build iterative models, and it provides everything we need to glean as much information and build the best models we can for our customers."

These more powerful models in turn mean that retailers can finally understand what marketing activities are most likely to lead consumers to make purchases, as Mohan Namboodiri (VP Customer Analytics, Williams Sonoma) describes in this GigaOM Structure 2013 panel discussion.

To learn more about how UpStream uses Revolution R Enterprise, take a look at this Revolution Analytics case study.

Webinar May 1: What's new in Revolution R Enterprise 6.2

Revolution R Enterprise 6.2 is now available, and includes many new features that enhance the performance, scalability and enterprise readiness of R. On May 1, product manager Thomas Dinsmore will give an overview of the new features in a 30-minute webinar. You can register for the webinar (and the post-webinar slides and replay) at the link below.

Revolution Analytics webinars: What's new in Revolution R Enterprise 6.2

Because it's Friday: Ten Tech Tips from David Pogue

You probably thought you knew all of these tech tips, but I bet there'll be at least one that you'll find useful. In my case, as someone that always trie to use the keyboard instead of the mouse, I never new you could use shift-space to go back up a page in a browser. Handy!

 

Here's one I'd add to the list: you can use alt-TAB to switch to a different application (or cmd-TAB on a Mac). I'm always surprised how many people don't know that one.

Enjoy your weekend!

TED Talks (YouTube): David Pogue: 10 top time-saving tech tips (via Jose Camoes Silva)

Revolution Newsletter: April 2013

The most recent edition of the Revolution Newsletter is out. The news section is below, and you can read the full April edition (with highlights from this blog and community events) online. You can subscribe to the Revolution Newsletter to get it monthly via email.

Now Available: Revolution R Enterprise 6.2. Released today, this update further enhances the performance, scalability and enterprise readiness of Revolution R Enterprise. Key new features include fast data import from Teradata data warehouses, big-data stepwise linear regression support, upgrades to the RevoDeployR web services framework, and the open-source R 2.15.3 language engine. Upgrade instructions will be sent to existing subscribers on Monday, April 29th, 2013. Learn more about the new features in Revolution R Enterprise 6.2 in this blog post, and in this press release.

Webinar: What's New in Revolution R Enterprise 6.2. Product Manager Thomas Dinsmore describes the new features of Revolution R Enterprise 6.2 in a 30-minute webinar at 11:00AM ET on May 1st. Register for the webinar here to attend the live session and for access to the recorded replay.

New R Training Courses. The Revolution Analytics training department has just launched new R training courses, including Data Visualization in R, Marketing Analytics with R and Using R with Hadoop. Check out the R Course Catalog (PDF) for the complete list.

Applications of R in Finance at R/Finance 2013. Revolution Analytics is a proud sponsor of the premier gathering of quantitative analysts, portfolio managers, risk analysts and researchers using R for finance.Registration is open for the R/Finance 2013 conference, which will take place May 17-18 in Chicago.

Revolution Analytics named Top 100 Big Data Vendor by CRN. The CRN 2013 Big Data 100 list includes leading vendors in Big Data business analytics, data management and data infrastructure and includes leading companies like SAP, SAS, Tableau, Tibco and Revolution Analytics.

Revolution Analytics in the News: Get tips on hiring data scientists in this article from the Wall Street Journal(sub. req.), learn how Revolution R Enterprise is used for risk management in Toronto's Globe and Mail, or get the latest company news in this analyst update from James Taylor.

Revolution Analytics: Newsletter Archive 

 

Lahman: A New R Package for Baseball Stats

by Joseph Rickert

Baseball fans have been serious about statistics since Carl Pearson was a young man (although I doubt that Carl followed the game). It is not clear, though, exactly when baseball statisticians moved from doing descriptive stats into predictive analytics. In his book Super Crunchers, Ian Ayers credits Bill James of Baseball Abstract fame for getting this particular ball into play. And of course, Michael Lewis' Moneyball brought the power of predictive statistics to reshape the game of baseball itself to the public’s attention in a very dramatic way.

Moreover, baseball statistics are likely to become even more dramatic in the future. For a look at how companies like Sportvision are building the infrastructure to overlay a TV image of a batter with a heat map of the sweet spots of his strike zone and the like, take a look at the presentation Graham Goldbeck made to the Bay Area User Group last October.

So, today, it is widely recognized that a deep understanding of the venerable American pastime requires a fairly high level of statistical play. Those of us blogging at Revolution Analytics have encouraged testing assumptions, looking for patterns and making predictions with several baseball related posts including:

• Learning R through baseball: sab-R-metrics
• Comparing baseball pitcher styles with lattice graphics
• Baseball Games: getting longer?
• Mariano Rivera’s baseball prowess, illustrated with R
• Where Ichiro hits

But now, Professor Michael Friendly of Canada’s York University has taken the analysis of the game to a new, more approachable level by wrapping up Sean Lahman’s Database into the R package Lahman. Version 2.0-1 is available on R-Forge and should be up on CRAN by he end of the week. Michael recently wrote me that "the original motivation was to provide a comprehensive, R version of data on baseball statistics for an annual project I run in a graduate course on multivariate data analysis". (This project is actually a pretty cool thing itself. Students hone their data analysis and conference presentation skills by preparing papers on topics related to baseball statistics for presentation to the prestigious but fictional "Hotelling Society".)

The file Lahman Data Sets (Download Lahman Data Sets) lists the 25 data sets that are available in the Lahman package. Note that some of these are of a pretty good size: Batting contains 96,600 rows and 24 columns while Fielding is a 164,898 by 18 table. The file 400 Hitters Plot (Download 400 Hitters Plot) contains Michaels code for the following plot which is big league (geekier) version of the graph that appeared in the New York Times two years ago.

Looking at the way this curve breaks I find myself vacillating between a sense of awe at the accomplishments of Ty Cobb and Rogers Hornsby and wondering how much money could be made by explaining the dip and rise of the curve. In any event, I'd like to think that, the Lahman package will forever link Spring, baseball and lazy afternoons of doing some stats with R.

Michael also indicated that he would welcome contributions to the Lahman package project. So, if you have some examples that you would like to share, or you would like to write some code please sign up on R-Forge. 

Announcing Revolution R Enterprise 6.2

We are pleased to announce that Revolution R Enterprise Release 6.2 is available to new subscribers today. This new software release from Revolution Analytics includes a number of key new features:

• Support for open source R 2.15.3, the latest stable release of R.  Since Release 2.14.2, the R Project has added 89 new features, 11 performance enhancements and 139 bug fixes.  Moreover, there are 137 CRAN packages whose latest version depends on R 2.15.3.

• The new High-Speed Teradata Connector, which is based on Teradata Parallel Transporter. According to benchmark testing performed by the Teradata Center of Excellence, the High-Speed Connector is at least six times faster than the previously available ODBC connector, as shown in the chart below.

• Stepwise Regression for Linear Models. Stepwise Regression is a method for automated selection of variables in a predictive model. For an excellent article about Stepwise Regression in action, read this post. 

• The Parallel Random Number Generator provides an R interface to the parallel random number generators supplied with the Intel MKL libraries, enabling high quality parallel random numbers to be used in distributed computations performed by RevoScaleR’s rxExec function. This capability is a necessary foundation for randomized decision trees (trademarked by Breiman and Cutler as "Random Forests"), and is also useful for simulation and Monte Carlo analysis.

• Revolution R Enterprise users sometimes want to "roll up" data from one level to another — for example, to summarize transaction facts at the customer level. The rxCube and rxSummary functions now provide options to write by-group counts or summary statistics directly to a high performance .xdf file for further analysis. Users also now have more control over the summary statistics that are reported.

• Release 6.2 also includes some performance enhancements, including a fast fixed format text data source capability, as well as optimized memory utilization for sorts and merges.

• For customers who use our RevoDeployR capability for enterprise deployment of analytics there are several enhancements, including priority scheduling for asynchronous jobs, the ability to execute external scripts, plus enhanced versioning and lifecycle management for repository-managed files and scripts.

Revolution R Enterprise is available to new customers today. If you are an existing subscriber you can expect to receive an email on Monday, April 29 with detailed instructions on how to download and install the new software. (If you don't receive your update notification, please contact the Support team.) If you'd like to learn more about the new features, you can find an overview in this press release, or you can join the webinar What's New in Revolution R Enterprise 6.2 on Wednesday, May 1. 

Learn how to analyze data with R with Coursera's "Data Analysis" videos

If you didn't manage to catch Coursera's Data Analysis course, don't despair. Instructor Jeff Leek has made the course videos available on YouTube, which you can review at your leisure to learn how to plan, carry out, and communicate analyses of real data sets with R. (The course assumes you already have familiarity with R, so if you're new to the R language start with the videos from the Computing for Data Analysis course.) Begin with the Course Introduction for "Data Analysis":

 

The lecture videos are organized by week (the first is at the bottom of the list, so work from bottom to top):

• Week 1
• Week 2
• Week 3
• Week 4
• Week 5
• Weeks 6-7

By the way, "Data Analysis" was an extremely popular course: just over 100,000 students enrolled in the course, of whom more than half watched videos. 20,000 students completed quizzes, and 5,500 completed the data analysis assignments. Jeff Leek offers further reflections on the course in this Simply Statistics podcast.

Coursera hasn't yet scheduled another run of the course, but keep an eye out for the next one if you want to get the certificate. In the meantime, check out the videos linked below.

Jeff Leek (YouTube): Coursera Data Analysis

Revolution Analytics named to inaugural CRN Big Data 100

Revolution Analytics has been named among the "CRN Big Data 100", a collection of 100 companies selected by the CRN editorial team that have demonstrated an ability to innovate in bringing to market products and technologies that help businesses manage big data. Revolution Analytics was included in the Big Data Business Analytics category, along with SAP, SAS, Tableau, Tibco, and other innovative analytics companies. Other notable companes in the Big Data Management and Big Data Infrastructure and Services categories of the Big Data 100 include Cloudera, EMC, Hortonworks, Oracle and Teradata. You can find more information about the CRN Big Data 100 in the press release linked below.

Revolution Analytics News: Revolution Analytics named to inaugural CRN Big Data 100

Because it's Friday: Self-sufficient dogs

Given all of the tragic news this week, perhaps a little unicorn chaser is in order. So without further ado, here are some adorable videos of dogs keeping themselves entertained.

Jerry's owner built him an automatic ball throwing machine. Just check out the look of anticipation on his face as the ball is loaded:

 

On the other hand, Sid doesn't even need mechanical assistance. All he needs is some stairs:

 

That's all for this week. Enjoy your weekend, and stay safe — especially our friends and colleagues in Boston and Texas.

Open Source software's opportunity to reform government

The results from the 2013 Future of Open Source Survey are in — thanks to everyone who contributed by completing the survey. You can read an overview of the results here, or see the detailed breakdowns in the slides at the end of this post.

For me, one of the most interesting nuggets from the survey is that a plurarity of respondents believed that of all sectors, the public sector will be impacted the most by open source software:

There is huge opportunity to improve the ways that we govern ourselves through the use of open source software, principles and communities. Federal, state and local governments have access to rich and large streams of data, but thus far government hasn’t been as quick as the business sector in building powerful data-based applications and services. But governments have one advantage that the business sector does not: givernment data is, by definition, public. Here are three ways that government can capitalize on rich sources of public data:

Democratize data. By making public data available on the internet (where practical and legal), governments can draw on the expertise of the data scientist community to unlock its value. At the Federal level, the Open Government Initiative is a good start, although many of the data sets made available at data.gov are over-aggregated and too small for truly innovative Big Data applications. Some local governments are making good progress here as well. New York City’s open data policy and new data transparency law has fostered a culture of civic “hacking” on city data, and created handy tools for locals like this iPhone app to find local WiFi-enabled venues suitable for getting out of the house when working “from home”.

Crowdsource data science talent with open tools. The data science community has created powerful open-source tools and resources for the analysis of Big Data. The open-source R language is designed to analyze data common to government applications (like economic, environmental and social data) and is used today by government departments such as NIST, FDA, NOAA, and the CIA. Crowdsourcing platforms and contests are a great way of getting data scientists engaged with public data, as NASA found in this Kaggle contest to find the location of dark matter from radio astronomy data.

Deploy data scientists within departments. Not all government can be made public, of course. There may be privacy issues (think IRS data), or national security issues (CIA, NSA) to consider. So it’s up to data science teams in civil service to work on sensitive data in a secure environment. It’s very encouraging to see departments like the Consumer Protection Bureau openly embrace big-data platforms and open-source tools for their teams. The growing drive towards evidence-based policy making, and the use of statistical experiments to improve government effectiveness, is another very positive sign.

It's great to see that the respondents to the 2013 Future of Open Source Survey recognize this opportunity to revolutionize government. It's up to us, as citizens, to press governments to capitalize on the opportunity, so make your voice heard.

Michael J Skok: 2013 Future of Open Source - 7th Annual Survey results