Revolutions

Behind the NCAA Visualizer: Python, R and JavaScript

Rodrigo Zamith's NCAA Tournament Visualizer is a great example of an interactive data visualization. If you want to create something similar, Rodrigo has shared detailed behind-the-scenes information on how it was created. He used a mix of tools:

• Python was used to scrape team statistics fromt the NCAA website
• R was used to prepare the data for analysis, and to generate the visualizations (using the ggplot2 package)
• JavaScript and HTML was used to create the interactive interface

Many thanks go to Rodriguo for making all of the code behind the application available on GitHub as open source.

Rodrigo Zamith: Going Under the Hood of the NCAA Tournament Visualization

Make building R packages easier with devtools

If you're writing any significant amount of R code, you might want to start think about bundling it up into packages. An R package combines functions, data, documentation and unit tests, and is a convenient and reliable system to manage and version collections of R content that could otherwise become unwieldy. And if you want to share your code with colleagues or even the entire world via github or CRAN, an R package is the best way to do it.

The canonical documentation for building an R package is the Writing R Extensions Guide that comes with the R distribution. Following all of these instructions by hand can be a little complicated, but fortunately Hadley Wickham and his colleagues at RStudio have created the devtools package to make the process simpler.

The devtools package makes it much easier to create a package following the structure defined in the Extensions Guide. It also makes it easier to test your package, using the unit tests you've been creating. (You have been writing unit tests, right? Including unit tests in a package is an excellent idea: not only does it make your package more reliable in the long run, it also simplifies the development process in surprising ways. If you're not writing unit tests, definitely give it a go.)

Another good practice is to manage your package code in a source-code control system. (Packages have their own version numbers, which you can use to keep track of updates to your code.) If you're using github to manage your code, you can also install your package directly from github with the install_github command (this is also a handy way to share your package with others). And if you want to share your package to the entire world, devtools can also test if your package is ready to submit to CRAN (though be sure to read the CRAN submission guidelines first).

You can find more information about devtools at the package homepage linked below.

RStudio Projects: devtools

Everything you ever wanted to know about R functions

It's hard to overstate the importance of functions in the R language. Pretty much everything you do in R involves calling a function. From creating objects, to doing calculations, to creating plots and fitting statistical models, everything is done by calling a function. In fact, the first R command you probably ever ran:

x <- 2 + 2

involved two function calls: though it may not look like it, <- and + are both actually functions.

If you've been using R for any length of time, there's a good chance you've started writing your own functions, either as a way to automate tasks or as a way of extending the R language itself to suit your own needs. This is a great way to get the most out of R. When writing functions, it's useful to understand the details of how functions work, how variables are referenced, accessed and passed between functions, and how you can modify functions. Hadley Wickham has created an excellent overview of functions in R (part of his in-progress Making Reproducible Code book)

• the components of a function (the formal arguments, the body, and its environment)
• lexical scoping, and the rules for how objects and other functions are found within a function
• how arguments to a function are assigned (and how missing and named arguments are handled)
• how to write "special" functions, to create your own version of operators like [ and <- 

Also, if you're new to programming concepts like "scope", "closures" and "environments" you might want to check out Darren Wilkinson's blog post "Lexical scope and function closures in R" which provides a basic guide to such concepts.

github / Hadley Wickham: Functions

Examples combining C++ and R: The Rcpp Gallery

We've written before about how you can use the Rcpp package to speed up R, by converting performance-critical snippets of R code to C++. (Rcpp makes it easy -- you don't even have to be a C++ expert.) If you haven't tried it out yet, you might want to browse the Rcpp Gallery: an collection of simple but illustrative examples showing many ways that Rcpp can help. A few choice entries:

• A one-line C++ equivalent to Mathematica's "Clip" function runs 5x faster than optimized R code
• Calculation of matrix eigenvalues in R using the Eigen C++ library
• Accessing the date-time attributes of R xts time series objects from C++ code
• Using a simulation method to estimate pi (note how similar the R and C++ versions are)

You can find many more examples in the gallery at the link below. If you're interested in making a similar gallery yourself, Dirk Eddelbuettel shares some behind-the-scenes making-of details from the gallery's creator, RStudio's JJ Allaire.

(Updated Jan 16.)

Rcpp: Rcpp Gallery

Hadley's guide to high-performance R with Rcpp

Hadley Wickham has written a comprehensive tutorial for the Rcpp package, which makes it easy to create C++ code embedded in R programs. Hadley explains why you might want to do this in the introduction:

Sometimes R code just isn't fast enough - you've used profiling to find the bottleneck, but there's simply no way to make the code any faster. This chapter is the answer to that problem. You'll learn how to rewrite key functions in C++ to get much better performance, while not taking too much longer to write. The key to this magic is Rcpp, a fantastic tool written by Dirk Eddelbuettel and Romain Francois (with key contribution by Doug Bates, John Chambers and JJ Allaire), that makes it dead simple to connect C++ to R. It is possible to write high performance code in C or Fortran. This might produce faster code than C++ (but probably not), but it will take you much much longer to write. Without Rcpp, you must sacrifice many helpful wrappers and master the complex C internals of R yourself. Rcpp is currently the best balance between speed and convenience, and any other approach will be much more painful.

No knowledge of C++ is required, so if you have some R code — especially tight loops — that you need to speed up, you should definitely check it out the whole tutorial.

This guide comes from a book on writing reproducible code that Hadley's working on. You can browse the other sections of the book on GitHub — even though the book isn't yet complete there's a wealth of useful information there for any serious R programmer.

Hadley Wickham (github): Rcpp

Browse the in-development R sources at GitHub

As an open-source project, the R source code has always been available to download from the R-project website. You can find source code for the latest released version here, and for the changing-daily new version in progress (R-devel) here.

But if you don't have the R sources handy, and just want to check on the contents of a file or two, a read-only, browsable mirror of the R-devel sources is now available at GitHub. This mirror is updated hourly from the SubVersion repository that the R core team uses to manage the R source code, and so also includes the change history and commit logs for each file. (You'll see a lot of notes marked [ripley] and [hornik] as they and other R-core members update the code.)

Note that this is the source code for the development version of R, and so probably doesn't match the version you're using. But it's a great reference to see what's going on in the R sources right now. Also, it's only includes the files used for development; some human readable files which are built from other source files aren't included. (One example is the HTML version of the NEWS file, but you can always find the daily NEWS file for R-devel here.)

github: Read-only mirror of R source code (via Hadley Wickham)

Benchmarking random-number generation from C++

If you're writing C++ code and want to generate random numbers, you might not be aware that R provides an API to call the R RNG functionality directly. The Rcpp package's "syntactic sugar" feature makes this process easier, by automating the process of translating a subset of ordinary R code into compiled C++ code. That means you can write code that looks like R in C++:

draws = rnorm(N_, 0.0, 1.0) ;

and have it compiled to directly into C++ code, using R APIs where appropriate. You can even write R for loops and have them converted to compiled C++ loops. That prompted R user Jonathan Olmsted to benchmark four ways of generating a million random numbers (results from fastest to slowest):

1. Compiled vectorized call to rnorm (about 1.6 seconds)
2. Compiled loop calling Rf_rnorm (an older Rcpp API to the RNG) (about 50% slower)
3. Vectorized call to rnorm in R (not using C++, also about 50% slower)
4. Compiled loop calling rnorm (about over 250% slower)

Interestingly, 3 vs 4 is a case where calling the vectorized function in R directly is faster than writing a loop in C++, but the new vectorized API in Rcpp is the fastest option by far. Jonathan's code (linked below) is also a great example of using the benchmark function (from the rbenchmark package) to compare performance of snippets of R code.

Jonathan P. Olmsted Blog: RNG Performance with Rcpp

How R finds objects (or, what that :: operator is for)

Most of the time when we're programming in R, we don't think about how R gets from an object name (say, "stdev") to what it represents (a function to calculate standard deviation, perhaps). If you're writing functions, you've probably know about R's lexical scoping. And if you use a lot of packages, you probably know about the search list, the sequence of packages that R looks in for an object if it can't be found in the local environment. But if you've ever wondered what an "environment" actually is, or the specifics of the rules that R actually uses to find objects by name, don't miss Suraj Gupta's excellent article, "How R Searches and Finds Stuff". 

This is deep stuff, but it's also a fascinating look into R's inner workings. R seems like a simple language, so you might be surprised at the complexity behind the name-lookup process, which Suraj describes as "following the purple line road" in the diagram below.

As Suraj poins out, given that "R is an interpreted, weakly typed language that supports attaching multiple packages at-will", a design like this is necessary to set rules for what should happen in the cases of functions calling themselves, or multiple packages that provide definitions for the same function name. He also offers some excellent advice for package developers, for example on how to set the Depends and Import attributes of your package to avoid conflict with packages the user may have installed, and how to manage package namespaces with exported and hidden symbols. Oh, and if you've ever wondered what the mysterious :: and ::: operators (as in stats::sd) are for:

If you know exactly which package contains the object desired then you can reference it directly using the :: operator. Simply place the package name before the operator and the name of the object after the operator to retrieve it. 

If the object is not exported or you are unsure, then you can use the ::: operator (notice the extra colon).

Read the whole, fascinating article at the link below.

O Beautiful Code: How R Searches and Finds Stuff

Machine Learning for Hackers

"Machine Learning for Hackers" is a new book from O'Reilly Media by Drew Conway and John Myles White. A "hacker", here, is "someone who likes to solve problems and experiment with new technologies", and "Machine Learning" is usually thought of as a black-box, algorithmic approach to producing predictions or classifications from data.

This book, however, takes a pleasingly statistical approach to real-life prediction and classification problems. Rather than merely providing a "cookbook" approach to say, building a "who to follow" recommendation system for Twitter, it takes the time to explain the methodology behing the algorithms and give the reader a better basis for understanding why these methods work (and, equally importantly, how they can go wrong).

An analysis of author Drew Conway's Twitter network, classified by topic area favored by each Twitter user.

The book assumes familiarity with command-line scripting, programming, and algorithms in general. It does, however, give a gentle introduction to the R programming language, which is used to implement all of the examples. (The R scripts and associated data are also available for download.) In fact, this section also serves double-duty as an introduction to some of the basics of statistical thinking (moments, distributions, visualization, etc.), which is a very work addition in a "machine learning" book. It's also rich with many data visualizations (mostly created with the ggplot2 package), which not only helps explain the algorithms but is a useful demonstration in its own right of the value of data visualization in the data modeling process.

Machine Learning for Hackers is available for purchase now in hardcopy or digital format from the link below. I recommend it to any programmer who needs to generate predictions or classifications from data -- using R and learning more about the statistical techniques behind the methods will help you to create better data hacking applications in the long run.

O'Reilly Media: Machine Learning for Hackers, Case Studies and Algorithms to Get You Started

RInside updated

The RInside package provides C++ classes that make it easier to embed R in C++ code on many platforms. According to a post from package co-maintainer Dirk Eddelbuettel, the package has just been updated to version 0.2.6 with improved support on Windows, and compatibility with R's standard random number generator. You can download the latest version from CRAN or from the RInside homepage.

Thinking inside the box: RInside 0.2.6