Revolutions

Seven reasons to use R

Biological anthropologist James Holland Jones is asked a simple question by his professor colleagues at Stanford: Why use R? Of course, it's free, but are there other reasons to choose R for research and teaching over the commercial alternatives. In his essay, Jones not only offers a defense of free but also offers seven other excellent reasons to be using R:

• R is used by the majority of academic statisticians; 
• R is platform-independent;
• R has unrivaled help resources; 
• R makes the best graphics; 
• The command-line interface is better for teaching;  
• R is a full programming language, not just a statistics application; and 
• The online distribution system beats anything out there. 

See the link below for detailed explanations of these points. But his final point needs no further explanation: "Let's face it: all the cool kids use it."

Monkey's Uncle: Why Use R?

Because it's Friday bonus: Bonnie Tyler, explained

Flowing Data points us to this flowchart guide for better understanding the seminal 80's classic Total Eclipse of the Heart:

Personally though, I think nothing reveals the inner meanings of the lyrics better than this music video:

Flowing Data: Total Eclipse of the Heart (Flowchart)

Because it's Friday: Correcting the academic record

Physicist Rick Trebino just wanted to point out a mistake in a published journal article by publishing a comment to correct the archival record. He published the process in a hilarious article, How to Publish a Scientific Comment in 1 2 3 Easy Steps (PDF). Thats 123 steps mind you, not 1-2-3 steps. The whole thing sounds like something Douglas Adams would have written, had his milieu been academic publishing and not space adventures.

On a serious note, Trebino brings up some interesting points about the conflict of interest of journal editors have with regard to poor-quality content that makes it to the printed volume. It reminds me of one of the BioConductor talks I went to last month. I don't remember the exact details (or the speaker, sorry -- it was a great talk, though -- update: it was Keith Baggerly), but it concerned a published article where the results seemed odd to the speaker. The speaker tried in vain to get hold of the source data to reproduce the analysis, but the article authors didn't cooperate. So in an amazing feat of data forensics, the speaker managed to recreate the data by matching public sources to measurements from the printed graphs, and figured out that there were gross data errors in the article: labels transposed, data duplicated, that kind of thing. The conclusions were completely bunk, but the journal refused to print a correction, despite the fact that it meant actual patients were being trialled on inappropriate drugs. It's a long time since I've published in a refereed journal, but are things really this bad? And is open access publishing the solution?The analysis below from scienceblogs.com takes on the issue in more detail.

Adventures in Ethics and Science: The saga of the journal comment

plyr: "group by" for R data

JD Long has just posted a great intro to Hadley Wickham's plyr package. One of the best ways to show off your R mojo is to perform iterative tasks without explicit use of the "for" loop, and for this purpose R has a plethora of functions to apply an operation to parts of a data object. For example, you can use the function apply to calculate the mean of each column of a matrix. There's also sapply, lapply, mapply ... the list goes on.

The plyr package is an elegant rationalization of the various interfaces to these types of functions. JD Long provides a nice analogy: it's kind of like using the GROUP BY statement in SQL, where you can specify some variables to slice the data (zipcode and sex, for example), and operations to calculate on each slice (like, calculate the mean of each age). Check out the link below for some examples of plyr in action. 

Cerebral Mastication: A fast intro to plyr for R

Late Summer Reading

I just placed my order for a couple of books I've been meaning to read, and with a couple of weekend trips away coming up I might actually get the time to review them. 

The first is Hadley Wickham's ggplot2: elegant graphics for data analysis. I've been meaning to get more into ggplot2 graphics for some time, and the printed version of this book (plus Hadley's upcoming talk at the Bay Area R User Group meeting) is just the impetus I need.

The second is Beautiful Data, a book I've heard a lot about but haven't yet seen. But with chapters from Nathan Yau (responsible for the wonderful Flowing Data), Andrew Gelman (whose statistical modeling blog is a daily read), and the aforementioned Hadley Wickham, plus lots of R examples, I'm really looking forward to it.

Now, if I could just pack my comfy reading chair...

Creating your own R packages: a tutorial

Most R users have downloaded and used packages from CRAN to extend R's capabilities. But relatively few have created packages themselves. The official reference documentation, Writing R Extensions, can be daunting, but creating a package is actually easier than you might think. Even if you don't plan to submit a package to CRAN, organizing your own work into packages confers a number of benefits:

• Packages offer a well-defined structure for organizing R source code, data, and documentation which makes for excellent best practice to compartmentalize and organize your work.
• The file-based structure of packages makes it easy to manage your work within source-code control systems, for security against losses and access to historical versions.
• The packaging system has tools for software validation which check that documentation exists and is technically in sync with the code, can spot common errors, and checks that examples actually run.

If you do plan to submit to CRAN, creating a package is a pre-requisite. But even if you're just sharing code with others, the package format is an ideal vehicle for distributing statistical methodology to others. For the recipient of your package, there are the following benefits:

• The validity of the package contents can be assured by running the automated tests.
• Packages can be dynamically loaded and unloaded at run-time and hence only occupy memory when actually used.
• Installations and updates are fully automated and can be executed from inside or outside R.

Freidrich Leisch, R-core member and manager of CRAN, has written an excellent tutorial (PDF) which takes you step-by-step through the process of creating your first package. It runs through a worked example of creating a package for linear regression (and incidentally also serves as a concise tutorial for writing method functions for a user-defined S3 class). It shows how to create the R source files, how to write help-files, and how to create and run tests on your package, and how to structure the package itself.

If you've been thinking about writing a package, or just need to organize your R code better, this is an excellent place to start.

[Update Sep 25 2013: The tutorial is now part of the contributed documentation for R, and the link below has been updated.]

R-project contributed documentation: Freidrich Leisch (2008): Creating R Packages: A Tutorial

For More R Tutorials, Check Out Revolution Analytics R Training. 

How pie charts fail

Welcome, LifeHacker readers! If you want to run the R code referenced below, download REvolution R, and check out our Beginner Tips section for links to tutorials and guides to help you get started making your own charts with R.

The Wikipedia page for Pie Charts now has a nice section illustrating how bar charts do a better job of visualizing differences. This comparative chart (created using R, see the code here) makes the point clearly:

On a related note, someone called Omegatron is on a humanitarian mission to rid Wikipedia of uninformative pie charts. You can see his accomplishments (1,2) at Junk Charts.

Junk Charts: Community outreach: pie-making

 

Parallel programming with foreach and snow

One of the design goals of the foreach construct for parallel programming was to create a syntax that would allow R users to specify a loop whose iterations could be run in parallel without having to worry about the mechanics of how it was done. In fact, foreach supports several different backend architectures for distributing code in parallel: we've already discussed multicore (for single multiprocessor or multicore workstations), NetworkSpaces (for fault-tolerant heterogeneous clusters -- currently only available with REvolution R Enterprise), and sequential programming (for testing and for single-processor machines). All the user needs to do is to "register" their preferred backend, and the foreach loop will use the selected method to achieve the parallelism.

Now there's a fourth parallel backend: snow. Snow, or "simple network of workstations" is an R package from Luke Tierney that allows you to connect multiple workstations into a parallel cluster. If you're already using snow, it's trivial to use foreach for parallel programming. Simply install the doSNOW package (just released to CRAN by the REvolution Computing development team), and then call:

> registerDoSNOW(clustobj)

where clustobj is your snow object defining your cluster. From then on, foreach will use that cluster to parallelize its iterations. 

Why XML is incompatible with big data

Mike Driscoll lays out his misadventures using XML for managing large amounts of data: it's too verbose and slows down translation operations, and despite the goals of the XML standard, tags are opaque and cumbersome for humans to deal with. He concludes that there must be a better way: the simple, delimited text files we've been using since the fifties. He offers an analogy with LaTeX and MathML:

Spoken languages are strengthened by usage, not by imperial fiat, and data formats are no different. Far better to evolve and adapt the standards we already have (as JSON and SQLite’s file format do), than to fabricate new ones from whole cloth.

XML offers some nice advantages for interoperability when managing "human-sized" documents, but when it comes to truly large datasets I have to agree these benefits are outweighed by the overheads detailed in Mike's article.

Dataspora Blog: How XML Threatens Big Data

CRAN R 2.9.2 released

The source code for R 2.9.2 is now available and propagating through the CRAN mirrors. Binary releases will be available soon (update: looks like the Windows binaries are available now). As we mentioned last week, this is mainly a bug-fix release. The full list of changes is available in the NEWS file, but the more significant differences are:

• Improved flexibility for interpolation rules in approx and approxfun
• agrep can handle more complex regular expressions (on some 32-bit systems) 
• More efficient PDF graphics generation when using the Symbol font 
• P-value significance stars now given in ANOVA tables for Poisson and Binomial GLMs
• Improved accuracy for extreme tails of the pbeta function (in log-scale mode)  
• Several bug fixes to eliminate memory corruption and crashes in unusual cases