Revolutions

June 2010 edition of R Journal available

The latest edition of the R Journal is now available. This issue includes in-depth articles on the packages IsoGene, glmmBUGS, cshapes, tmvtnorm, neuralnet, glmperm and exactci/exact2x2, plus an example of reproducible research in R. There's also a review of the book A Beginner's Guide to R. The complete edition is available for download in PDF format at the link below.

The R Journal: Volume 2/1, June 2010

Tips for managing memory in R

R is an in-memory application, so every new object you create takes up RAM. (Yes, there are ways around that, but that's a topic for another article.) If you're working on a small machine (say, a 32-bit Windows system with 1Gb of RAM or less) you might need to be careful with the object you create. This StackOverflow question offers some useful tips for managing objects in RAM, including code for the function lsos to list objects sorted by size:

 
A great way to identify objects (such as X, above) ripe for deletion with rm when you don't need them anymore.

StackOverflow: Tricks to manage the available memory in an R session?

How to peg 7 cores with doSMP

Statistics PhD student Nathan VanHoudnos has an 8-core laptop, and by his own admission, takes "an almost unhealthy pleasure in pushing [his] computer to its limits". It seems like he's found an outlet for this passion with the new doSMP library included with Revolution R, that allows him to use all his processors for some gnarly simulations in R:

 

Actually, he's written his code to use 7 processors instead of 8: it's good practice to leave one CPU unpegged if you want to continue to use your computer interactively during the computations. See Nathan's R code at the link below.

Nathan VanHoudnos: Pegging your multicore CPU in Revolution R, Good and Bad

Because it's Friday: Insect sex

Birds do it, bees do it. But the bees and their insect brethren definitely do it in a more interesting way. Don't believe me? Check out Isabella Rosselini's description of bee sex and the other videos in her educational "Green Porno" series. It's fascinating stuff. For some light summer reading, I also recommend Olivia Judson's "Dr. Tatiana's Sex Advice To All Creation", a book of "Dear Abby"-style questions and answers about sex problems from the weirder corners of the animal kingdom. The folks at McSweeny's have taken this one step further from Dear Abby to the raunchier style of Savage Love with Dan Savage's Wild Kingdom. One question begins: "I'm a 2-hour-old asexual amoeba, and I have kind of a weird problem... like most single-cell folks, I went through a phase in my teens where I was binary-fissionizing myself so often I thought my psuedopods would fall off." Follow the link below for the answer.

McSweeny's: Dan Savage's Wild Kingdom

Pollution from the BP oil spill

There's been a lot of talk about the slicks and plumes of oil from the Deepwater Horizon disaster, but how does the presence of that oil translate into measurable pollution in the air, water, and sediment? The EPA is now releasing pollutants and making the data available for analysis. Because the data are online, it's a simple process to read it into R and do visualization and analysis. The blog R-Chart shows us how to map the sampling locations around the Louisiana coastline, for example:

This is a simple analysis, but demonstrates the principle of working with this public data in R. It would be interesting to see some more in-depth analyses of the data.

R-Chart: Plotting BP Oil Spill Testing Data using R

Why Learn R? It's the language of Statistics

In the Introduction to his book “R for SAS and SPSS Users” (Springer 2009) Robert Muenchen offers ten reasons for learning R if you already know SAS or SPSS. All ten reasons say something important about R. However, his fourth reason: “R’s language is more powerful than SAS or SPSS. R developers write most of their analytic methods using the R language; SAS and SPSS developers do not use their own languages to develop their procedures” is fundamental. To me, this expresses something about R that should speak to anyone who does statistical modeling no matter what tools she or he may be using. What is so compelling about R’s powerful language? I think that there is a direct analogy here with natural language. Every language enables thoughts peculiar to the culture in which it developed. If you speak more than one language, how many times have you labored to say something in another language that just comes so naturally in your mother tongue? Whether by design, or historical accident, some languages are just better than others for saying certain things. I propose that in the same way that English is the language of business , and that French may still be the language of diplomacy, R is the language of Statistics. I don’t just mean that R “is spoken” by many or even most statisticians. R’s superiority for statistics is deeper than that. R is a language with syntax and structure that have been explicitly designed to formulate expressions about statistical objects. At this time, it may be le premier langue for statistical thinking that enables the formulation of ideas, and notions about statistical models and data that are difficult to express succinctly in other languages including mathematical notation. For example, suppose you want to discuss multiple regression. A mathematical exposition might begin with the equation:

(1)    Y = X(beta) + epsilon  

A statistician will naturally interpret this as an expression of the regression model, but (1) is primarily a statement about the relationship of random variables, abstract mathematical entities not statistical models. In contrast, the R expression

(2)    model <- lm(y ~ x)

is a statement about the linear model that relates the data structures x and y. For a person who “speaks” some R, (2) “means” the model object coefficients, residuals etc. that result from fitting a linear model to the data structures x and y. Moreover, (2) actually makes “model” an object, packed with information that describes the regression and can be thought about as a whole and  “talked about” with other R expressions. There is certainly some overlap, but expressions (1) and (2) are really about different concepts.

As another example of the expressive power of the R language, consider how difficult it is to formulate multi-level hierarchical models in standard mathematical notation. With the aid of the notation “j[i]” which is used to encode group membership (j[10] = 2 means the tenth element in the data indexed by i belongs to group 2) Gelman and Hill (Cambridge 2007: a must read for anyone new at multi-level modeling) write the simple varying intercept model with one additional predictor as yi =( alpha)j[i] + (beta)xi +(epsilon)i. This nonstandard notation gets messy quickly as complexity increases, and as Gelman and Hill point out it doesn’t enable the unique specification of the model. (They discuss five ways to write the same model.) By way of contrast, their R code using the lmer function (now lme4):

model <- lmer(y ~ x + (1 | group) )

is succinct, preserves the syntax for linear models, and generalizes reasonably well with complexity. Just like there are some things that come naturally in your mother tongue but are awkward to express in another language, R enables concise statements about statistical models that are difficult to express otherwise.

R’s syntax is not the only feature that contributes to its expressive power. The interplay of R’s objects (nouns) with methods (verbs) facilitates formulating expressions that make sense under many different circumstances. For example, one can use the same R function “summary” to  form statements that talk about data: summary(x),  as well as statements that talk about models: summary(model). This ability of functions to have methods for different kinds of objects is a hallmark of R that greatly eases the process of learning the language.  It is easy to say statistical things in R, and just a little bit of language skill goes a long way in turning statistical thoughts into action packed, working statistical sentences.

How to: Debug in R

Revolution Analytics is proud to sponsor the New York R User Group. The last meeting was on the theme of debugging in R, and some videos of the talks are now available at the Video Rchive. Jay Emerson have a talk on Basic debugging in R and Harlan Harris dived deeper on advanced debugging techniques. Also presenting were Peter Flom (on demystifying error messages) and Revolution's Joseph Rickert, who demonstrated the interactive debugging features of the Revolution R Enterprise. Due to a Flip-cam battery failure, neither of those talks were recorded. But Joseph did recreate a 3-minute version of his demo of debugging in R for the Revolution website.

New York R User Group: Demystifying error messages and debugging in R

Analyzing competitive nordic skiing with R

Here's another great example of R being used to analyze sports data. Statistician and skier Joran Elias has started a project to analyze and visualize international cross country ski racing results, and he publishes his analysis at the blog Statistical Skier. All of the analyses are done using R (and for data, SQLite via the RSQLite package). As much as I love to ski, I'm not so familiar with the intricacies of ski racing as a sport, but it looks like there's a rich source of data ripe for analysis, and Joran is pulling out some interesting results. For example, here's the recent history of one recently-retired five-time Olympian Sabina Valbusa:

 

I can only assume a low "FIS Points" is better: Joran notes "best season overall might have been 2002-2003". A great use of ggplot2 and smoothing to visualize these data. Check out the blog for some other analyses -- especially if you're a ski junkie.

Statistical Skier: Data based analysis and commentary on nordic skiing

MMDS 2010

The 2010 Workshop on Algorithms for Modern Massive Data Sets (MMDS 2010) finished up this past Friday (June 18th) at Stanford. This was an exceptionally well organized conference: four days of mind-stretching talks on algorithm development and the challenges of working with massive data sets approached from almost every conceivable angle. The approximately 100 attendees were a diverse group of computer scientists, developers and quantitative marketing people from industry and academia. The mix of academics and industry types appeared to be pretty well balanced with a number of speakers listing themselves as having both university and company affiliations. This mashup of people, backgrounds and ideas reflected what I think was the overall theme and rationale for the workshop: moving deep theory ideas to practical solutions.

The level of the talks ranged from Peter Norvig’s (head of Google research) opening tutorial on the practical problems of running Google’s data center that is so massive that shark attacks, blasphemy outbreaks and other 6+ sigma events are real worries -- to Konstantin Mischaikow’s (Rutgers university) presentation on a “Combinatorial Framework for Nonlinear Dynamics” for which a good background in algebraic topology would have been helpful. Most of talks were fairly technical, but many of the speakers came across as being very skilled at communicating to non-specialists. (For a short time, I even had the illusion that I understood some algebraic topology!)

Maybe it was the perfect weather, or the beautiful Stanford campus, or the fact that the coffee was always hot, but it seemed to me that people were just in the mood for talking openly about their work.  Too much and too varied a body of material was presented for me, an outsider to this field, to summarize here, but the following is my take on some of the major themes -- the recurring melodies if you will -- that played as background music through a good many of the presentations:

• Many modern data sets of practical interest, those containing social networking information for example, are best described by graphs not flat tables. The need to look at networked data has triggered a shift in perspective were researchers are looking at almost all massive data sets as graphs, and are investigating aspects of network structure that might be useful in developing new algorithms. There are deep theoretical connections between linear algebra and graph theory; however, it is not clear to what extent these connections can be exploited to create practical, efficient algorithms. John Gilbert (UC Santa Barbara), referred to linear algebra as the “middle ware” of scientific computing. So, finding analogous computational tools for graphical data sets or augmenting linear algebraic methods with new techniques could indeed have profound, practical implications.

• The analysis of massive data is tied to the underlying hardware in at least two ways. First there is the matter of capacity. There seemed to be some agreement that data sets are small to medium if they can fit on a laptop, medium to large if they can fit on a single server, and massive if they are so big that they need to be spread among multiple servers. Hence, both the tools and techniques to analyze data will depend on the underlying hardware architecture. So for example, memory bound languages such as MatLab and R, viable choices for small to large data sets, are not be feasible for massive data sets without extensions to blow through the memory constraint problem. Then, there is the matter of parallelism. As several speakers remarked, parallel computing is no longer an option. These days, even low end laptops are designed around multiple cores. It is very likely that the next generation of middleware algorithms will take maximum advantage of parallel computing.

• Growth in the number of records to be processed is not the only contributor to the size and complexity of modern massive data sets. The drive to bring all of the relevant data to bear on a problem is leading statisticians to seek new methods of analyzing heterogeneous data. Susan Holmes (Stanford) presented work that integrated data describing networks of gene interactions, microarray data and image data into a single framework for analysis. Flexible tools for working with multiple data types and perhaps new data types altogether will be necessary to advance this program. For example, David Bader (George Tech College of Computing) described work on a new data type for dealing with streaming graphs with vertices that change over time, while several speakers presented work to develop structures and algorithms to deal with streaming data.

• Computer scientists and statisticians working with massive data sets are pursuing dual strategies for dealing with the challenge of size. On the one hand there is the continuing search for algorithms that scale in less than exponential time. On the other, there is the strategy of reducing the size of the problem either through sampling or dimensionality reduction techniques. A favorite trick of algorithm developers is to approximate a large matrix M, with a smaller matrix M’ that is “close” to M according to some relevant norm. However, as Petros Drineas (Rensselaer Polytechnic Institute) pointed out in his tutorial on randomized algorithms, clever algebra won’t always work if the resulting structures do not have straightforward interpretations. In describing work he did with SNPs, Petros pointed out that the geneticists he was working with were not happy with linear combinations of the SNPs that contained the maximum amount of relevant information; they wanted to know the individual SNPs that were most important

The papers presented at the workshop should be posted at the link above in the near future. I highly recommend them not only as a snapshot of the state of the science, but also as a way for non-specialists to learn something about the field.

New blog from Rmetrics Foundation

The Rmetrics Foundation (the sharp minds behind the Rmetrics suite of packages for financial analysis in R) have just launched a new blog where you can keep up with the latest Rmetrics news. Amongst the recent news: a ne eBook about data management of Indian financial market data, and a new interface between Rmetrics and AMPL. You can also follow Rmetrics on Twitter for updates.

Rmetrics Foundattion: Blog