Revolutions

R: Free, Popular, Powerful, Flexible and Supported

Francis Smart offers five excellent reasons to use R, in a well-researched post ideal for sharing with anyone thinking about making the switch to R. (You might also share this YouTube video for a quick 90-second introduction to R.)

The post also includes a novel analysis of interest in R, as tracked by Google Trends. Given its single-letter name, comparitive metrics for R can be tricky to come by, but Google Trends lets you focus on interest in R the programming language (and not, say, the R Kelly album), and compare it to SAS the statistical software (not the airline) and other similar products. Google Trends also includes forecasts, and shows that not only does interest in R eclipse all others in Google searches, it's set to continue to grow:

(That's a static screenshot -- click to see an updated Google Trends analysis.)

Econometrics by Simulation: Why use R? Five reasons

Because it's Friday: Mexico City or Sim City?

This image looks like it comes from a video game...

... but in fact it's a real photo of a housing complex on the eastern outskirts of Mexico City, taken from a helicopter by photographer Oscar Ruiz. It's one of the most remarkable photographs I've seen in a while.

That's all for this week. Enjoy our other Friday posts, and we'll see you back here on Monday.

R Training for SAS users in Singapore and online

Revolution Analytics has just introduced a 10-module series of R courses in Singapore. If you'd like to learn how to do data analysis in R, already know data analysis in another language like SAS and want to transition to R, or just want to enhance your R skills in a specific area, one of these hands-on courses may be of interest. The available modules (which you mix-and-match) are:

• Module 1: Introduction to R
• Module 2: Taking on Big Data using RevoScaleR
• Module 3: Using R with Hadoop
• Module 4: Implementing Web Services using RevoDeployR
• Module 5: Introduction to Rattle and Alteryx for Data Mining 
• Module 6: Visualization in R with ggplot2
• Module 7: R for SAS Users
• Module 8: Introduction to Statistics and Analytics
• Module 9: Advanced Statistical Topics, Data Mining, and Machine Learning
• Module 10: System Administration

The next scheduled course on April 15 is R for SAS users, a 2-day R course in Singapore designed for existing SAS users. This quick-start for the R language builds on your existing knowlege of SAS to teach data handling, data visualization, statistical analysis and reporting with R. 

If you're not in the SIngapore area, you can find other R courses in North America, India, and Europe. And from anywhere in the world you can take Bob Muenchen's course R for SAS, SPSS and Stata Users online, April 21-23.

R User Group Activity for Q1 2014

by Joseph Rickert

Worldwide R user group activity for the first Quarter of 2014 appears to be way up compared to previous years as the following plot shows.

The plot was built by counting the meetings on Revolution Analytics R Community Calendar. R users continue to value the live, in person events and face-to-face meetings with their peers. Moreover, if you peruse the details of the meetings listed on the calendar you will see that there have been some fantastic presentation so far this year. And although, as with any live event, if you missed it you will never know how good it really was, some of the R user groups have left traces of what happened by posting presentation slides on public areas of their websites.

I am sorry that I missed Harold Baize’s presentation to the Berkeley R language Beginner Study Group on Using R in a Microsoft Office world. I think the following slide from Harold’s presentation indicates a pragmatic approach coupled with a wry sense of humor.

The presentation goes to discuss reading and writing to Excel, Tableau, the R2DOCX package, markdown and more. To get the presentation, download the file RinMSOffice from the Berkeley Site.

Jeroen Janssens’ presentation on Command Line Data Science to the New York Open Statistical Programming Meetup is a very nice introduction to command line essentials for either Linux or Mac OS. If you are not comfortable with cat, grep or awk this might be a place to start.

  

In a similar spirit of helping new users get up to speed with R, the New Hampshire R Users Group (NH UserRs) has posted a number of tutorials by David Hocking.  Introduction to Linear Regression and ANOVA in R is a useful 10 minute first look. Newcomers might also find it valuable to follow up with the David Lillis presentation, Data Analysis Tips in R, that has been posted on the Wellingtone R User Group (WRUG) site. This presentation contains a short discussion on calculating Tetrachoric and polychoric correlations in R that quite a few people might find valuable.

DublinR has graciously made available all of the materials from a Bayesian Data Analysis workshop that Mick Cooney delivered earlier this year. A zipped file containing R scripts, JAG files and data can be downloaded by clicking on “bdasingle2014” at this link.

Finally, I am glad that I was present at Megan Price’s takk: How a Small Non-Profit Human Rights Group uses R, to the Bay Area UseR Group. This was an inspirational talk where you really had to be there. Nevertheless, Megan’s slides do portray the big picture of how statistical analysis can make a difference in human rights investigations. The following photo of a cache of documents containing records of those who disappeared during Guatemala's civil war indicates the magnitude of the statistical sampling problem that Megan and her colleagues at HRDAG faced.

 

This coming Monday, March 31, 2014, is the last day for applying to Revolution Analytics R User Group Sponsorship Program for Matrix or Array level sponsorship. So if you are the organizer of an established R User group and you think a cash grant and a box of R related “goodies” could help you grow, please apply before midnight PST 3/31/14.

Give your R charts that Wes Anderson style

I'm a big fan of Wes Anderson's movies. I love the quirky characters and stories, the distinctive cinematography, and the unique visual style. Now you can bring some of that style to your own R charts, by making use of these Wes Anderson inspired palettes. Just choose your favourite Wes Anderson film or short: 

Install the wesanderson pallettes package, created by Karthik Ram:

require(devtools)
devtools::install_github("wesanderson","karthik")
require(wesanderson)

And convert your plain old ggplot2 charts like this:

Into this:

Here's the command for you cut-and-pasters:

ggplot(diamonds, aes(clarity, fill=cut)) + geom_bar() + 
 scale_fill_manual(values = wes.palette(5, "Cavalcanti")) 

The colors don't quite match up to the originals, but you can find other Wes Anderson-inspired palettes and adjust to your heart's content using the source code available from Karthik Ram's github page linked below. 

Karthik Ram: Wes Anderson Palettes

A Thumbnail History of Ensemble Methods

By Mike Bowles

Ensemble methods are the backbone of machine learning techniques. However, it can be a daunting subject for someone approaching it for the first time, so we asked Mike Bowles, machine learning expert and serial entrepreneur to provide some context.

Ensemble Methods are among the most powerful and easiest to use of predictive analytics algorithms and R programming language has an outstanding collection that includes the best performers – Random Forest, Gradient Boosting and Bagging as well as big data versions that are available through Revolution Analytics. 

The phrase “Ensemble Methods” generally refers to building a large number of somewhat independent predictive models and then combining them by voting or averaging to yield very high performance.  Ensemble methods have been called crowd sourcing for machines.  Bagging, Boosting and Random Forest all have the objective of improving performance beyond what’s achievable with a binary decision tree, but the algorithms take different approaches to improving performance. 

Bagging and Random Forests were developed to overcome variance and stability issues with binary decision trees.  The term “Bagging” was coined by the late Professor Leo Breiman of Berkeley.  Professor Breiman was instrumental in the development of decision trees for statistical learning and recognized that training and averaging a multitude of trees on different random subsets of data would reduce variance and improve stability.  The term comes from a shortening of “Bootstrap Aggregating” and the relation to bootstrap sampling is obvious. 

Tin Kam Ho of Bell Labs developed Random Decision Forests as an example of a random subspace method.  The idea with Random Decision Forests was to train binary decision trees on random subsets of attributes (random subsets of columns of the training data).  Breiman and Cutler’s Random Forests method combined random subsampling of rows (Bagging) with random subsampling of columns.  The randomForest package in R was written by Professor Breiman and Adele Cutler. 

Boosting methods grew out of work on computational learning theory.  The first algorithm of this type was called AdaBoost by its authors Freund and Shapire.  In the introduction to their paper they use the example of friends going to the race track regularly and betting on the horses.  One of the friends decides to devise a method of betting a fraction of his money with each of his friends and adjusting the fractions based on results so that his performance over time approaches the performance of his most winning friend.  The goal with Boosting is maximum predictive performance. 

AdaBoost stood for a long time as the best example of a black box algorithm.  A practitioner could apply it without much parameter tweaking and it would yield superior performer while almost never overfitting.  It was a little mysterious.  In some of Professor Breiman’s papers on Random Forests, he compares performance with AdaBoost.  Professor Jerome Friedman and his Stanford colleagues Professors Hastie and Tibshirani authored a paper in 2000 that attempted to understand why AdaBoost was so successful.  The paper caused a storm of controversy.  The comments on the paper were longer than the paper itself.  Most of the comments centered around whether boosting was just another way of reducing variance or was doing something different by focusing on error reduction.  Professor Friedman offered several arguments and examples to demonstrate that boosting is more than just another variance reduction technique, but commenters did not reach a consensus.

The understanding that Professor Friedman and his colleagues developed from analyzing AdaBoost led him to formulate the boosting method more directly.  That led to a number of several valuable extensions and improvements beyond AdaBoost – the ability to handle regression and multiclass problems, other performance measures besides squared error etc.  These features (and new ones being developed) are all included in the excellent R package gbm by Greg Ridgeway.

Today, ensemble methods form the backbone of many data science applications. Random Forests has become particularly popular with modelers competing in Kaggle competitions and according to google trends Random Forests has surpassed AdaBoost in popularity.

In a future post we will explore several of these algorithms in R.

References

Breiman  Leo, Bagging Predictors, Technical Report No. 421, Sept 1994, Dept of Statistics University of California, Berkeley.

Ho, T., Random Decision Forests, Proceedings of the Third International Conference on Document Analysis and Recognition, pp. 278-282, 1995

Breiman, Leo Random Forest – Random Features, Technical Report No. 567, Sept 1999, Dept of Statistics University of California, Berkeley.

Freund, Yoav and Schapire , Robert E. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55. 1997.

Friedman, Jerome, Hastie, Trevor, Tibshirani, Robert Additive Logistic Regression: A Statistical View of Boosting, Ann Stat, Vol 28, Number 2, (2000), 337-655

 

Free eBook on Big Data and Data Science

The fine folks behind the Big Data Journal have just published a new e-book Big Data: Harnessing the Power of Big Data Through Education and Data-Driven Decision Making. (Note: Adobe Flash is required to view the e-book.) In the eBook, you'll find the following technical papers on the topics of Big Data, Data Science, and R:

• Data Science and its Relationship to Big Data and Data-Driven Decision Making, by Foster Provost and Tom Fawcett. 
• Predictive Modeling With Big Data: Is Bigger Really Better?, by Enric Junqué de Fortuny, David Martens, and Foster Provost. 
• Educating the Next Generation of Data Scientists, a roundtable discussion including Edd Dumbill, Elizabeth D. Liddy, Jeffrey Stanton, Kate Mueller, and Shelly Farnham
• Delivering Value from Big Data with Revolution R Enterprise and Hadoop, by Thomas Dinsmore and Bill Jacobs.

There's also a video introduction to the papers from yours truly. View the e-book — sponsored by Revolution Analytics — here: Big Data: Harnessing the Power of Big Data Through Education and Data-Driven Decision Making

Because it's Friday: 1000 years of Europe's changing borders

Europe's borders are changing again. This is big news, but you might be surprised to find out just how malleable the borders in Europe have been over the past 1000 years, as this timelapse map shows:

 

When I was in primary school, I used to think that borders were as fixed and unchanging as the lines in the atlas in the classroom. I still remember the day I completed Jordan Mechner's classic adventure game The Last Express. The closing credits, which show Europe's changing borders from 1914-1994, was an eye-opener for me.

 

Those changes have continued since.

That's all for this week — have a great weekend, and we'll see you on Monday.

Learn about handling character data in R with this free e-book

Most people know R as a statistics/analytics language for analysis of quantitative data, and don't think of it as a tool for processing raw text. But R actually has some quite powerful facilities for processing character data. And as Gaston Sanchez learned, text manipulation is an important part of a modern data scientist's repertoire:

Many years ago I decided to apply for a job in a company that developed data mining applications for big retailers. I was invited for an on-site visit and I went through the typical series of interviews with the members of the analytics team. Everything was going smoothly and I was enjoying all the conversations. Then it came turn to meet the computer scientist. After briefly describing his role in the team he started asking me a bunch of technical questions and tests. Although I was able to answer those questions related with statistics and multivariate analysis, I had a really hard time trying to answer a series of questions related with string manipulations.

I will remember my interview with that guy as one of the most embarrassing moments of my life. That day, the first thing I did when I went back home was to open my laptop, launch R, and start reproducing the tests I failed to solve. It didn't take me that much to get the right answers. Unfortunately, it was too late and the harm was already done. Needless to say I wasn't offered the job. That shocking experience showed me that I was not prepared for manipulating character strings. I felt so bad that I promised myself to learn the basics of strings manipulation and text processing. "Handling and Processing Strings in R" is one of the derived results of that old promise.

Gaston's Creative-Commons licensed 112-page e-book, Handling and Processing Strings in R, is an excellent and comprehensive review of R's string handling capabilities. It cover's R's basic string-handling capabilities (reading, converting, manipulating and formatting), and also devotes a chapter to the higher-level functions of Hadley Wickham's stringr package. The two chapters on regular expressions are a must-read for anyone who hasn't yet come to grips with the power of regexes for handing string-based data. There are a few practical examples at the end of the e-book (frequency counting, word clouds) but the book sticks mainly with the fundamentals, and doesn't stray into semantic analysis. Highly recommended for anyone working with strings or character data in R.

Gaston Sanchez: Handling and Processing Strings in R (via Sharon Machlis)

Data Sets for Data Science

by Joseph Rickert

Recently, I had the opportunity to be a member of a job panel for Mathematics, Economics and Statistics students at my alma mater, CSUEB (California State University East Bay). In the context of preparing for a career in data science a student at the event asked: “Where can I find good data sets?”. This triggered a number of thoughts: the first being that it was time to update the list of data sets that I maintain and blog about from time to time. So, thanks to that reminder I have added a few new links to the page, including a new section called Data Science Practice that links to some of the data sets used as examples in Doing Data Science by Rachel Schutt and Cathy O’Neil.  Additionally, I have provided a direct link to the BigData Tag on infochimps and pointed out that multiple song data sets are available.

However, to do justice to student’s question it is necessary to give some thought to exactly what a “good” practice data set might look like. Here are three characteristics that I think a practice data set should have to be good:

1. It should be big enough to pose some computational challenges without being so big that it requires a cluster or some specialized hardware just to get started.
2. It should require some cleaning or pre-processing (making decisions about mission data for example) but not appear to be hopelessly corrupt; dirty but not to dirty.
3. It should be rich enough that once you have gone through the trouble of accessing and cleaning it there are enough variables or features to suggest multiple questions to analyze, or make it possible to try out different machine learning algorithms.

Here are three data sets that meet these criteria in ascending order of degree of difficulty:

The first suggestion is the MovieLens data set which contains a million ratings applied to over 10,000 movies by more than 71,000 users. The download comes in two sizes, the full set, and a 100K subset. Both versions require working with multiple files.

Near the top of anybody’s list of practice data sets, and second on my little list because of degree of difficulty is the airlines data set from the 2009 ASA challenge. This data set which contains the arrival and departure information for all domestic flights from 1987 to 2008 has become the “iris” data set for Big Data. With over 123M rows it is too big to it into your laptop’s memory and with 29 variables of different types it is rich enough to suggest several analyses. Moreover, although the version of the data set maintained on the ASA website is fixed and therefore perfect for benchmarking, the Research and Innovative Technology Administration Bureau of Transportation Statistics continues to add to the data on a monthly basis. Go to RITA to get all of the data collected since the ASA competition ended.

  

Last on my short list is the Million Song data set. This contains features and meta data for one million songs which were originally provided by the music intelligence company Echo Nest. The data is in the specialized HDF5 format which makes it somewhat of a challenge to access. The data set maintainers do provide wrapper functions to facilitate downloading the data and avoiding some of the complexities of the HDF5 format. However, there are no R wrappers! The last I checked, the maintainers had a paragraph about there being a problem with their code along with an invitation for R experts to contact them (This would clearly be for extra points.) For more details about the contents of the data set look here.

As a final note, it is much easier use R to analyze the Public Data Sets available through Amazon Web Services now that you can run Revolution R Enterprise in the Amazon Cloud. We hope to have more to say about exactly how to go about doing this in a future post. However, everything you need to get started is in place including a 14 day free trial (Amazon charges apply) for Revolution R Enterprise. All you need is your own Amazon account.

Please let me know if you have additional links to useful, publically available data sets that I have missed. We very much appreciate the contributions blog readers have made to the list of data sets.