Revolutions

How to: Debug in R

Revolution Analytics is proud to sponsor the New York R User Group. The last meeting was on the theme of debugging in R, and some videos of the talks are now available at the Video Rchive. Jay Emerson have a talk on Basic debugging in R and Harlan Harris dived deeper on advanced debugging techniques. Also presenting were Peter Flom (on demystifying error messages) and Revolution's Joseph Rickert, who demonstrated the interactive debugging features of the Revolution R Enterprise. Due to a Flip-cam battery failure, neither of those talks were recorded. But Joseph did recreate a 3-minute version of his demo of debugging in R for the Revolution website.

New York R User Group: Demystifying error messages and debugging in R

Getting a Web application to talk to R

Let's face it: you can do some pretty awesome things with R -- statistical models, beautiful charts, you name it -- but if the only way to do those things is from the R command line you're limiting the audience of people who might make use of all this awesomeness to a limited subset: R programmers.

What if you could get the results of R programs, charts and data, into a Web application? R's an open system, so it's definitely possible. To get started, Neil Saunders shows us how to build a simple Web application that displays data and a chart from an R script, by serving JSON or CSV from Rails to RApache and sending a result back to Rails. As he points out, it's not a complete solution -- "a real application would require more model or controller methods, views, R functions, error checks and prettier views, perhaps with a dash of AJAX thrown in", but it's a great starting point.

What You’re Doing Is Rather Desperate: Getting your web application and R(Apache) to talk to each other

Saving the world with R

Tuesday's meeting of the Bay Area R UseR Group at the LinkedIn offices was a great event. The headline speaker was Joe Adler, author of the excellent R reference manual, R in a Nutshell. Joe's presentation was an in-depth look at the relative speed of various options in R for looking up values from a key in a key-value pair sequence. One of the simplest ways of doing this is to assign a names to a vector with the names function, and then you can look up the value associated with name "Australia" (if, say, you'd named your vector population with country names) with the statement population["Australia"] or population[["Australia"]]. With simulations, Joe revealed that the latter version is a bit faster than the former, but both have lookup times proportional to the length of the vector (i.e. lookups get slower for longer vectors). An faster (but more complicated) option is to use environments, where lookups can be done via a hash table in constant time. Joe provides all the details in this blog post. He did offer some good advice: even though the simpler constructs are slower, in general it's best to program for clarity, and only optimize for speed when really necessary.

A surprise addition to the program -- and a very pleasant surprise, at that -- was a lightning talk from Megan Price at Benetech. Benetech is a non-profit organization contracted by the likes of Amnesty International and Human Rights Watch to answer thorny geopolitical questions through the use of data and science. For example: "Were acts of genocide committed against the Mayan people in Guatemala?" (the answer, sadly, is yes.) Megan opened her talk by saying that she "uses R to save the world" -- and I think the was only half-joking. As Megan explained in a fascinating presentation, using statistical techniques to address these questions can "transform emotional political debates into debates about methodology and science". Specifically, she uses Multiple Systems Estimation techniques (from the rcapture package) to count things that are otherwise difficult to quantify. The process is related to capture-recapture methods used, for example, to count the number of animals in the wild. Megan also offered some great advice for creating scientific reports with R: her process of building reports dynamically with xtable and Sweave means that when something changes at the last minute -- and "something always changes at the last minute!", says Megan -- it's a simple process to recreate the report to incorporate the changes, without having to reformat and cut-and-paste everything together again.

Bay Area UseR Group: April 13 2010

Speeding up R code: A case study

On his Psychology and Statistics blog, Jeromy Anglim tells how he was analyzing some data from a skill acquisition experiment. Needing to run a custom R function across 1.3 million data points, Jeromy estimated it would take several hours for the computation to complete. So, Jeromy set out to optimise the code.

First, he used the Rprof function, which inspects your R functions as they run, and counts the amount of time spent in each sub-function. This is a useful tool to identify the parts of your functions that are ripe for optimisation, and in this case (with some help from the system.time function to time a specific section of the code) he learned that most of the time wasn't taken performing actual calculations: most time was actually spent selecting the subset of the data to analyze!

And thus a solution was born: rather than repeatedly selecting from the large data frame in an iterative loop, he instead split the data frame into its constituent parts once, and then looped over the parts. This reduced the analysis time from hours down to just a couple of minutes. As the end of his case study, Jeromy shares some valuable lessons learned about optimising R functions:

• R is very fast most of the time.
• A single slow command can be the cause of a slow analysis.
• system.time is a very useful function.
• Optimisation can proceed from theory or from experimentation. 
• Optimisation proceeds from diagnosing the cause of the problem to exploring solutions.
• Optimisation is about orders of magnitude. Focus on saving hours before tackling saving seconds.

Jeromy Anglim's Blog: Psychology & Statistics: A Case Study in Optimising Code in R

How to call C++ from R with ease

At last night's meeting of the ACM Student Chapter at the University of Chicago, DIrk Eddelbuettel gave an invited guest lecture, "Programming with Data: Using and Extending R". I wasn't there myself, but Dirk has already posted his slides, and they're a treat. After a backgrounder on R itself (BTW, I'm flattered he referenced my Introduction to R talk here), Dirk dives into some case studies on various ways to call C, C++ and Fortran code from R. Most of the time is spent on the Rcpp interface (by Dirk and Romain François), which greatly simplifies the process of writing C++ code to be called from R. You can even "inline" C++ code directly in your R program, freeing you from the need to write complete C++ source files, and compile and link the code yourself. Dirk presents some impressive performance gains from using this technique. 

[Update 9:42am: Attribution of Rcpp]

Dirk Eddelbuettel / Thinking Inside the Box: U of C ACM talk

Creating your own R packages: a tutorial

Most R users have downloaded and used packages from CRAN to extend R's capabilities. But relatively few have created packages themselves. The official reference documentation, Writing R Extensions, can be daunting, but creating a package is actually easier than you might think. Even if you don't plan to submit a package to CRAN, organizing your own work into packages confers a number of benefits:

• Packages offer a well-defined structure for organizing R source code, data, and documentation which makes for excellent best practice to compartmentalize and organize your work.
• The file-based structure of packages makes it easy to manage your work within source-code control systems, for security against losses and access to historical versions.
• The packaging system has tools for software validation which check that documentation exists and is technically in sync with the code, can spot common errors, and checks that examples actually run.

If you do plan to submit to CRAN, creating a package is a pre-requisite. But even if you're just sharing code with others, the package format is an ideal vehicle for distributing statistical methodology to others. For the recipient of your package, there are the following benefits:

• The validity of the package contents can be assured by running the automated tests.
• Packages can be dynamically loaded and unloaded at run-time and hence only occupy memory when actually used.
• Installations and updates are fully automated and can be executed from inside or outside R.

Freidrich Leisch, R-core member and manager of CRAN, has written an excellent tutorial (PDF) which takes you step-by-step through the process of creating your first package. It runs through a worked example of creating a package for linear regression (and incidentally also serves as a concise tutorial for writing method functions for a user-defined S3 class). It shows how to create the R source files, how to write help-files, and how to create and run tests on your package, and how to structure the package itself.

If you've been thinking about writing a package, or just need to organize your R code better, this is an excellent place to start.

LMU University Library: Freidrich Leisch (2008): Creating R Packages: A Tutorial

 

Google's coding standards for R

For any software application in production and produced by a development team, coding standards are an essential tool for promoting maintainability and quality. Google has recently published their coding standards for R, with a compendium of common-sense tips for keeping an R project in shape. The standards Google R programmers adhere to include:

• File Names: end in .R 
• Identifiers: variable.name, FunctionName, kConstantName  
• Line Length: maximum 80 characters  
• Indentation: two spaces, no tabs  
• Curly Braces: first on same line, last on own line  
• Assignment: use <-, not =  
• Semicolons: don't use them  
• Commenting Guidelines: all comments begin with # followed by a space; inline comments need two spaces before the #

Follow the link below to see the other standards and all of the details. (Thanks to Daniel Levine of TechCrunch for the tip.)

Google Code: Google's R Style Guide

Finding the Source

The thing that makes R different from commercial packages like SAS or Excel is that it's open-source. This means that if you ever wonder exactly how R calculates a certain value, or you want to adapt an existing R routine for your own purposes, you can always find the source code R uses and take a look yourself.

Much of R is written in the R programming language itself: if you want to see how a function is R is defined, all it usually takes is to type the name of the function at the command line.  For example, to see how the sd (standard deviation) function in R works, just type its name:

> sd
function (x, na.rm = FALSE) 
{
    if (is.matrix(x)) 
        apply(x, 2, sd, na.rm = na.rm)
    else if (is.vector(x)) 
        sqrt(var(x, na.rm = na.rm))
    else if (is.data.frame(x)) 
        sapply(x, sd, na.rm = na.rm)
    else sqrt(var(as.vector(x), na.rm = na.rm))
}

This works because in R, functions are just like any variable like a matrix or vector, in the sense that they have a value. (In the usual parlance, functions are "first-class objects" in R.) The value displayed is the definition of the function, in the R language.

There are some wrinkles in finding the source code for some functions: some are in other packages or namespaces, and some are implemented in the core R sources in C or Fortran.  You can find detailed instructions on finding the source code in these cases in a 2006 R-News article by Uwe Ligges: Accessing the Sources (see pages 44-45).  But in general, the source is always there for you to look at if you need it.  

Using R as a scripting language with Rscript

In a recent posting to r-devel, Prof. Brian Ripley of R-Core announced an interesting development (pun intended) for future versions of R.  Several of the core utility scripts, such as INSTALL and REMOVE, have been converted from sh scripts or Perl into native R code.

You might not think of R as a utility scripting language, but it has many useful features that make it an excellent candidate for such a task, not least powerful string-processing tools and functions for manipulating file names. In fact, if you're familiar with programming in R, but (like me) can never remember the exact syntax for a loop in bash, you might be tempted to create your own command-line tools using R as the scripting language.

In Unix-like systems (including MacOS X), this is easy to do.  Create an ordinary text file where the first line is #!/usr/bin/Rscript (replacing /usr/bin with your actual R executable installation directory).  Then on the following lines include ordinary R commands.  Set the executable bit of the file (with chmod +x), put it in your PATH, and you can then use it as an ordinary command-line tool.

Rscript is a noninteractive variant of the standard R command, designed for just this use.  Here's an example (from the Rscript documentation) that allows you to install packages from the shell prompt, without having to invoke interactive R:

#! /usr/bin/Rscript --vanilla --default-packages=utils
args <- commandArgs(TRUE)
res <- try(install.packages(args))
if(inherits(res, "try-error")) q(status=1) else q()

If you have some tasks that you need to automate from the command-line, Rscript might be a useful tool to make use of your R programming skills, without having to dive for the bash documentation. And as a bonus for scripts you intend to distribute to other R users, your script will be platform-independent, easy to maintain, and may even run faster than alternative scripting languages on platforms like Windows.

The R Inferno

Patrick Burns has published "The R Inferno", a manual for R developers with lots of useful tricks and tips for reducing memory usage, improving performance, and avoiding errors in computational analysis.  Says Patrick: "If you are using R and you think you're in hell, this is a map for you."  (Excerpts from Dante make this quote more literal than it may first appear!)

Download it here: The R Inferno.