Revolutions

Tricks to manage memory in an R session

Unless you're using an out-of-memory solution to manage large data objects (such as the RevoScaleR package in Revolution R Enterprise), then R always allocates memory for every object in your working session. If you're working with many objects (or even just a few large objects) then you'll need to take care to manage R's memory usage to avoid the dreaded "cannot allocate memory" error. This question on StackOverflow offers several handy tips, including an enhanced version of the objects function to identify the biggest memory hogs for deletion.

StackOverflow: Tricks to manage the available memory in an R session? (via)

Because it's Friday: Exploring Zipcodes

I live in San Francisco, so my Zip code (postal code) is 94107. But have you ever wondered where the 5 digits in your zip code come from? It turns out it's based on a hierarchical system: the initial 9 represents the western region of the US; the 5 represents the San Francisco area within that, and so on. You can see the system in action in the elegant zipdecode data visualization by Ben Fry (also known as the author of the Processing data visualization language):

Once the visualization has loaded, type your zip code. The key though is to do it slowly, digit by digit, so you can watch as it zipdecoder zooms into the levels of the hierarchy.

Ben Fry: zipdecode (with thanks to reader KW for the suggestion)

For data visualization, circles don't cut it

During President Obama's State of the Union speech on Tuesday night, the live feed at the White House website featured PowerPoint-style infographics and bullet points to accompany the President's statements. One of those charts was a comparison of the GDP of the United States and its nearest rivals:

Notice anything odd? The 2010 GDP of the US isn't quite three times that of China, but you could easily fit four of those China circles in the US one, if not more. In fact, the area of those circles are proportional to the square of GDP, as Dan Meyer illustrates with amusing snark. Scaling a variable to one dimension (here, diameter) of a two-dimensional shape is a rookie mistake in data visualization, an issue Edward Tufte popularized in his book The Visual Display of Quantitative Information. And given that Tufte was recruited by the White House to improve communication about data that chart is, well, a bit embarrassing.

But even when the area of is appropriately scaled, circles are still a poor tool for communicating information about numbers. Pie charts are circles, and it's well known that pie charts suck when it comes to comparing proportions. And even simple circles are hard to compare, as amply demonstrated in this demonstration from the blog Contrast:

"If you could have the profits of company A, or those of B,C,D,and E combined, which would you take?"

The answer is that the area of A is greater than that of B, C, D and E together, but that's hardly apparent to my eye. The whole post is well worth a read -- it's chock filled with examples of good and ba data visualizations -- and a fine testament to the principles of Tufte.

Constrast: Infographics and Data Visualisations (via)

Slides and replay for "Portfolio design, optimization and stability analysis"

Thanks to everyone who attended yesterday's webinar, "Portfolio design, optimization and stability analysis", presented by Diethelm Würtz of the Rmetrics Association and sponsored by Revolution Analytics, Sybase, Finance Online and NeuralTechSoft. (And thanks in particular for your patience for the last start -- in a perfect demonstration of Murphy's law a microphone failed moments before the scheduled start.)

If you couldn't make the live webinar, you missed a great overview of the capabilities of the Rmetrics suite of R packages, including some great details on new methods like portfolio ridge profile lines and using Google Motion Charts to explore portfolio sensitivities, as shown below:

But you can still catch up at the link below, where you can find the slides (PDF) and video replay (WMV format) available for download (reg. req.). 

Revolution Analytics Webinars: Portfolio design, optimization and stability analysis

 

New R User Group in Minneapolis/St. Paul

The Twin Cities R User Group has been around for a little while, but has just launched a new site at meetup.com. Their next meeting will be on February 16, where Erik Iverson will be giving a talk on using R to generate dynamic statistical reports using R's literate programming tool, Sweave. If you're in the Minneapolis-St.Paul area, this would be a great way to meet other local R users and learn about reporting in R as well. You can find them and 50 other local R user group in the Local R User Group Directory.

MeetUp.com: Twin Cities R User Group

A new sponsorship program for local R user groups

Over the past year, Revolution Analytics has sponsored a number of local R user groups, and we've been thrilled to see the enthusiasm with which R users are coming together. This year, we decided to expand and formalize our sponsorship program, so that any local R user group, whether just starting up or already established, can benefit from additional support from Revolution. We're inviting the leaders of R user groups from around the world to apply for an annual cash grant to provide refreshments, support speakers, organize events ... whatever the group decides.

If you haven't yet joined a local R user group, take a look at the local R user group directory to find one in your area. And if there's not one in your area, why not start one up yourself? (In addition to sponsorship for new groups, we've provided some tips to help get you started.) To get a sense of what goes on at an R user group meeting, we asked the leaders of three of the largest US-based R user groups in the to comment in a press release issued today:

From the Chicago R Users Group:

“R user groups are a great learning channel for analytics professionals and we intend to use Revolution Analytics’ support to continue having user groups that are enjoyable and educational,” said James Long of the Chicago R Users Group. “Over the last year alone, the Chicago user group has been able to share ideas on topics like integrating R with C++, nonlinear optimization, using R to devise spread trading strategies and running R in parallel on Amazon Web Services. There is a gamut of topics we would love to cover with Revolution’s support, and we look forward to other great presentations in 2011."

From the Los Angeles R Users Group:

“The Los Angeles R user group started in 2009 when we saw a clear need for greater networking opportunities between data analysis professionals,” said Dr. Szilard Pafka, co-founder of the Los Angeles R User Group. “Today, we have over 350 online members and a very active community of 50 professionals who convene monthly. We are thrilled to be among the first participants in Revolution Analytics’ program and anticipate that their support will enable us to attract more data analysts and scientists who want to promote increased collaboration and innovation around R.” 

From the New York R User Group:

“Revolution’s user groups sponsorship can open up a lot of collaboration options, which is a huge benefit to us, as well as other groups,” said Drew Conway of the New York R User Group. “We consistently uncover new ways to solve problems with R and we’re very grateful to see a commercial company care about working with data scientists who are ’at the wheel’ of advanced analytics.”

By the way, the next New York user group meeting on February 15 will feature Amanda Cox from the New York Times’ Graphics Department to present how R is used in the news cycle to crunch data and prepare graphics before they go to print or online, which is an excellent example of some of the awesome applications of R that get presented and discussed at R user group meetings. What could your R group be meeting about next? Follow the link below for more details on the Revolution Analytics R User Group sponsorships available.

Revolution Analytics: R User Group Sponsorship Program

 

A twitter feed for new R packages

Want to keep up-to-date on the latest R packages released to CRAN? Dirk Eddelbuettel's CRANberries service now tweets the release of new R packages to @CRANberriesFeed, so all you need to do is follow that user on Twitter.

R hackers may also be interested to see how this Twitter feed was implemented -- in R, of course. Dirk has shared the details of how the feed is automated by a 3-line R function (and some supporting packages) on his blog.

Thinking inside the box: CRANberries is now tweeting

Because it's Friday: Entropy, schmentropy

Here's a neat little science experiment: put three drops of dye into a cylinder of liquid, mix up the colours by turning the handle, and then reverse the process:

 

Pretty mind-boggling, eh? Now, I'm sure this isn't a trick in the sense that there's no duplicity here, but I think it *is* an illusion in that the three colours of dye never actually mix because they're inserted at different depths (from the wall) in the cylinder. I'm guessing that the dyes are smeared out in the laminar flow of concentric cylinders of liquid, but the camera is looking through three "smears" of dye that don't aren't actually mixed. If the dyes *do* mix, I can only assume someone repealed the second law of thermodynamics without telling me. But then again, fluid dynamics was never my strong suit: to this day I still believe that the Bernoulli effect is what lets airplanes fly. Any experts out there care to explain what's going on?

Learning R through baseball: sab-R-metrics

The words "statistics" and "baseball" are often found near each other, but there's a lot more to statistics than dividing the number of hits by the number of swings to get a batting average. And there's a lot more to sabermetrics -- the statistical analysis of baseball -- than averages, too. Many baseball fans are also stats geeks (and vice versa) and have done deep statistical analysis of baseball data, oftentimes with R. 

Dave Allen of the Baseball Analysts website regularly uses R to visualize PitchFX data, as does his stablemate Jeremy Greenhouse (for example, in this analysis of optimal swing rates). ESPN's the Sports Guy inspired a detailed analysis in R by Ryan Elmore. Ricky Zanker of the Hardball Times has published a guide on reading baseball data into R. Mike Driscoll created an interactive R application to visualize data from PitchFX. Even the widely-read New York Times election analyst Nate Silver got his statistical start with sabermetrics.

If you're a baseball fan and you'd like to do some sabermetrics of your own, but haven't made the leap to learn R yet, Millsy is here to help. He's a graduate student in Sport Management at the University of Michigan, and has created a series of tutorials to help you learn R by analyzing baseball data. The tutorials take the new R user and baseball fan step-by-step through your first R commands, reading in data, manipulating objects, and even creating charts and doing simple analysis. Good practical advice is scattered throughout, such as advice "to use color to help portray the information you are trying to communicate, rather than just to make things bright", with this awesome example of what not to do:

There are five parts to the series so far, and I'm looking forward to seeing more in this great series.

The Prince of Slides: sab-R-metrics

Some blogs for the Stats geek

For blogs specific to R nothing beats the r-bloggers.com aggregator, but if you're looking for a general list of blogs of interest to the Stats geek (or data scientist, if you prefer), check out this list of 40 Fascinating Blogs for the Ultimate Statistics Geek. While has some notable omissions (Andrew Gelman's Statistical Modeling blog and Drew Conway's Zero Intelligence Agents come immediately to mind), it's still a great list of blogs with interesting statistical content. And I'm not just saying that because this here statistics blog is on the list. (Well, maybe just a little bit.) 

Update: Hmm, Gelman's blog is on the list, but is oddly listed as being "from Dartmouth".

B School: 40 Fascinating Blogs for the Ultimate Statistics Geek