Revolutions

The Environmental Performance Index, visualized with R

The Environmental Performance Index (EPI) ranks countries on performance indicators for environmental public health and ecosystem vitality. Yale University hosts the EPI website, which was used to present the 2012 EPI Rankings to world leaders at the 2012 World Economic Forum at Davos. The Country Profiles section of the website allowed members to browse the performance characteristics of their countries, and see how they compared with other countries with similar environmental performance. (Switzerland was the 2012 overall leader.)

In the Visualizing and Interacting with Data Over the Web session at JSM 2012 this morning, Yale professor Jay Emerson revealed details of how this interactive website was created. The profile chart you see above is generated dynamically, using R grid graphics and the FastRWeb web framework for R.  The interactive "EPI Performance Score versus Trend Score" graphic was created using the gridSVG package. The country map and numeric tables were embedded using the googleVis package. And the "Countries with similar levels of performance" were generated using one of R's many clustering algorithms.

The EPI website provides a great example of how R can be used to power an attractive, informative and useful interactive website.  And because all of these charts and analyses are generated dynamically from live data (rather than pre-rendered in advance), this same website will be trivial to update when the 2013 EPI performance scores are announced.

If you'd like more details of how these charts were created, check out Jay's slides on grid graphics from the Introduction to Data Visualization and Analysis course at JSM. (There's also a very useful section on parallel programming with the foreach and doSNOW packages.) Jay also provides some useful notes on setting up fastRWeb and Rserve on his blog.

A prediction for the Olympic men's 100m sprint

R user Markus Gesmann used the gold-winning times from the Olympic Men's 100m sprint since 1990 as the basis of the following prediction for the London Games:

My simple log-linear model forecasts a winning time of 9.68 seconds, which is 1/100 of a second faster than Usain Bolt's winning time in Beijing in 2008, but still 1/10 of a second slower than his 2009 World Record (9.58s) in Berlin.

Markus used the readHTMLtable function to scrape the data from the web into R, and the lm function to fit a regression model and make the prediction (the red line above). (The R code is provided at Markus's blog.) He admits the model is overly simplistic: it doesn't use data from non-Olympic competitions, and a model like this implies that 100m sprint times will shrink indefinitely. But you can see how well this prediction pans out this Saturday, in the men's 100m final. (Update: The winning time was actually 9.63 seconds.)

mages' blog: London Olympics and a prediction for the 100m final

Because it's Friday: New 2001: A Space Odyssey trailer

Movie trailers today are kind of a strange beast. They seem to be designed only to optimize just one thing: getting bums on seats for the opening day of the film. You'd think the smarter long-term strategy would be to set appropriate expectations about the film, and thereby enhance enjoyment of those who do go, to create positive word-of-mouth and generate future patrons. But sadly, that's not the usual case. The result: mystifying spoiler-laden movie trailers that routinely portray a film that's very different to what you eventually see on-screen.

Kubrik's 2001: A Space Odyssey is an epic, cerebral and rather philosophical film about space exploration and what an interaction with a higher intelligence might be like. But that's not the impression you'd get from a trailer cut for the film today, like this one:

 

This mocked-up trailer comes from Film School Rejects, and you can read more about it at the link below.

i09.com: What a Trailer for 2001: A Space Odyssey Would Look Like Today (via CB at Federated Media)

Revolution Analytics at JSM 2012

Revolution Analytics is proud to once again be a gold sponsor and Wi-Fi sponsor of the JSM 2012 conference in San Diego, the largest gathering of statisticians, biostatisticians, analysts, data miners and data scientists in the world. The conference begins on Sunday, and you'll find the Revolution Analytics team in the exhibit hall. Drop by to take a look at the new Revolution R Enterprise 6 and pick up R-branded goodies. There will also be refreshments available in the booth 10am-2pm on Monday and Tuesday, and we're pleased to be able to provide free WiFi in the exhibition hall during exhibition hours.

Members of the Revolution team are also participating in the formal conference program. I'll be part of a panel discussion on big data and predictive analytics. Our Chief Scientist Lee Elefsen will deliver a presentation on how to effectively scale analytics systems from a single to multiple core approach to deal with exponential increases to the size of typical data sets. And VP of Product Development Sue Ranney will deliver a presentation on the increased incidence of multiple birth rates in the United States over the last 30 years, applying multivariate analysis to individual-level data obtained from public-use data sets. Full details with links to the JSM program are below:

Who: Lee Edefsen, chief scientist, Revolution Analytics
What: “Parallel External Memory Algorithms Applied to Generalized Linear Models”
When: Monday, July 30, from 10:35 a.m. - 10:50 p.m. PT
Where: San Diego Convention Center, Room 30B

Who: Sue Ranney, vice president of product development, Revolution Analytics
What: “The Rise in Multiple Births in the U.S.: An Analysis of a Hundred-Million Birth Records with R”
When: Wednesday, August 1, from 11:50 a.m. - 12:05 p.m. PT
Where: San Diego Convention Center, Room 32B

Who: Dean Abbott, President, Abbott Analytics (Moderator)
  David Smith, vice president of marketing and community, Revolution Analytics
  Wayne Thompson, analytics product manager, SAS Institute
  Daniel Fystra, founder and president, Frontline Systems
  Ralf Klinkenberg, co-founder and chief business development officer, Rapid-I
  Dan Steinberg, founder and president, Salford Systems
  Thomas Hill, vice president of analytic solutions, Statsoft
  Jon K. Peck, senior engineer, IBM-SPSS           
What: “Predictive Analytics and Data Mining: The Software Makers” (panel discussion)
When: Wednesday, August 1, from 2:00-3:50 p.m. PT
Where: Hilton San Diego Bayfront, room Sapphire D

There are many other R-related talks in the program from R users as well. A couple I'm particularly looking forward to are the panel on Evolution of Statistical Graphics and Visual Analytics, Google's talk on Scaling R to Internet Scale Data, Saptarshi Guha's talk on RHIPE, and the talk on Enhancing Web Pages with R in the Browser.

What other R-related talks are you going to at JSM? Let us know in the comments.

Big vectors coming to R

R has been available as a 64-bit application since it's earliest days. But the internal representation of R's fundamental data type — the vector — has long been subject to a 32-bit limitation: the maximum number of elements is capped at 2^31 (or just over 2.1 billion) elements. Now, at 8 bytes per element that's 16Gb of data, so that wasn't a limitation until machines with massive amounts of RAM came along. And even then compound objects like data frames and lists can contain multiple vectors (and so exceed the 16Gb limit), so not many people noticed the issue.

But now that R-compatible servers with hundeds of gigabytes of RAM are available, the vector-length limitation does rear its head occasionally. The main culprit is actually computations with large-scale matrix algebra: in R, a matrix is internally represented as a single vector, so the largest possible square matrix has a dimension of 46,340. This can be a problem in financial analysis, for example, where calculating a covariance matrix of 50,000 or more time series is no longer unusual.

But it looks like that limitation is set to be lifted in a future version of R, as comments in the NEWS file for the development version of R suggest:

There are the beginnings of support for vectors longer than 2^31 - 1 elements on 64-bit platforms. This applies to raw, logical, integer, double, complex and character vectors, as well as lists. (Elements of character vectors remain limited to 2^31 - 1 bytes.)

All aspects are currently experimental.

This is a very exciting development: with full big-vector support, linear algebra in R will extend to a whole new order of magnitude, provided you have enough RAM available to hold the data. For those of us with more modest platforms, the bigmemory and RevoScaleR packages continue to provide tools for analyzing "Big Data" in the R environment.

The Simply Statistics blog has more on the forthcoming big vectors in R.

Another R mention in the NYT

The R language gets a brief mention in an article in yesterday's New York Times on automated bond trading:

The traders here are mostly educated in math or physics, often outside the United States, and their desks are piled high with textbooks like the “R Graphs Cookbook,” for working with obscure computer programming languages.

R an obscure programming language? Perhaps the finance desk should drop by the graphics desk sometime, where R is routinely used for the Times' interactive online and print graphics features. (A visualization of the last century of drought in the US is a particularly elegant recent example.) A language which currently ranks #12 amongst all languages in the Transparent Language Popularity Index can hardly be called obscure. R was also the subject of a feature article in, er, the New York Times.

New York Times: Bond Trading Loses Some Swagger Amid Upheaval

Revolution Newsletter: July 2012

The most recent edition of the Revolution Newsletter is out. The news section is below, and you can read the full July edition (with highlights from this blog and community events) online. You can subscribe to the Revolution Newsletter to get it monthly via email.

Quick Start Program for Hadoop: Revolution Analytics makes it easy for data analysts and IT teams to explore the powerful combination of Revolution R Enterprise and Hadoop. Our team of experts will help you plan, develop and deploy a prototype proof-of-concept analytic application, or take advantage of our R and Hadoop analytics technology to improve an existing application. Get started with the Revolution Analytics Quick Start Program.

Find a list of all of R's capabilities for analytics, data visualization, programming and applications at our new list of R Language Features.

The Joint Statistical Meetings (JSM) conference in San Diego starts on July 28, and Revolution Analytics is a proud sponsor. Join us for presentations on big-data GLM algorithms, census data analysis, and predictive analytics software, or find us in the Exhibition Hall for a demonstration of Revolution R Enterprise 6.

The Big-Data Generalized Linear Models in Revolution R Enterprise aren't just powerful and flexible: they're fast, too. See these benchmarks and a video demonstration.

Read: CIO magazine says Revolution R is a Big Data open source technology to watch.

The Revolution Analytics Webinar Series is taking a break for the Summer, but you can still watch free replays of recent presentations on operational risk management, spatial statistics and all the other webinars in the series. Got an idea for a future webinar? Let us know.

Revolution Analytics: Newsletter Archive

Civic Data Challenge closes July 29

There's only a few days left to enter the Civic Data Challenge: entries are due before midnight EST on July 29 to qualify for the $100,000 in prizes. The competition, open to US residents, challenges particpants to applications and visualizations from civic health data. Prizes will be awarded by a panel of prestigious judges. Looks like a great opportunity to break out R and the big-data capabilities of Revolution R Enterprise to win cash and provide insights to the community in the process! You can find information on how to participate at the Civic Data Challenge website.

Faster R in Hadoop: rmr 1.3 now available

The RHadoop project continues the Big Data integration of R and Hadoop, with a new update to its rmr package. Version 1.3 of rmr improves the performance of map-reduce jobs for Hadoop written in R. New features include:

• An optional vectorized API for efficient R programming when dealing with small records.
• Fast C implementations for serialization and deserialization from and to typedbytes.
• Other readers and writers work much better in vectorized mode, namely csv and text
• Additional steps to support structured data better (use more data frames and fewer lists in the API)
• More forgiving behavior for package loading and bug fixes

Also, the documentation has gotten a major overhaul in this version, with pages of combined text, code and graphics generated automatically using the knitr package. (RHadoop lead developer Antonio Piccolboni provides some background on how knitr is used in these documentation guidelines.)

If you haven't take a look at rmr before, this tutorial by Jeffrey Breen is a great place to get started. Otherwise, check out the wiki pages on the RHadoop github site, linked below.

github: RevolutionAnalytics / RHadoop 

Because it's Friday: Hypno SF

The video below isn't just a tribute to the beauty of San Francisco coupled with a pretty decent indie-rock song. It also has some incredible photographic work:

 

The "bouncing zoom" effect is a new one to me -- my favourite is the zoom along Market St from the Ferry Building to the Big Gay Flag in the Castro (at around 0:50), and Lombard Street to Treasure Island is also pretty cool. But I'm totally gobsmacked by the rotating shots towards the end of the video, especially the one of Sutro Tower:

To keep the tower the same size and alignment in all the shots, the photographer would have had to trace a circular path around the tower at a constant elevation. But it's so hilly and canyonous (is that a word?) around there, I've got absolutely no idea how they pulled it off. Incredible stuff.

You can see more (although sadly not a making-of video) at the video's official site at Hypno SF.