Revolutions

Mariano Rivera’s baseball prowess, illustrated with R

Kevin Quealy, graphics editor at the New York Times, has published another fascinating behind-the-scenes look at how the Times creates data visualizations for print and online. In his latest post, he looks at how a visualization of the Yankee's Mariano Rivera performance compared to other Major League Baseball pitchers was created. (Detail below, click for the full image.) 

The infographic began its life as a hand-drawn sketch, that begat a line-chart created using R (based on data scraped from the Web). The R chart was was then cleaned up and annotated using Adobe Illustrator for publication. One interesting detail of the process: the source R graph is deliberately created using garish colours (purples, greens, etc.) to make the color-selection process easier in Illustrator.

Check out the ChartsNThings archive for other great studies of the process of data journalism. Many of the case studies involve the use of R code, such as these visualizations of visitors to the White House, Santorum's primary support, Santorum/Romney exit poll data, NFL players mentioned on ESPN, the defense budget and the richest 1%.

ChartsNThings: Sketches: How Mariano Rivera Compares to Baseball’s Best Closers

EU court's SAS ruling conflicts with Oracle v Google

In a blow to SAS's efforts to litigate competitor and low-cost SAS clone WPS out of existence, the European Union High Court has ruled that programming languages can't be copyrighted. SAS Institute (Cary, NC) had claimed that the WPS software — which allows users to process SAS data files and SAS "data step" scripts without SAS software — breached SAS's copyright in its re-implementation of SAS functionality. But given that WPS did not study SAS source code, and merely reimplemented its interfaces and behaviour based on observation, the court ruled that this was not a violation. In a press release following the decision, the court stated:

 ... neither the functionality of a computer program nor the programming language and the format of data files used in a computer program in order to exploit certain of its functions constitute a form of expression. Accordingly, they do not enjoy copyright protection.  

(Emphasis in press release.) SAS has issued no comment on the ruling to date.

The case has implications beyond SAS and WPS. In Europe, at least, it implies that developers of both proprietary and open-source software have the right to duplicate the functionality and exposed interfaces (in this case, the structure of SAS procedure calls) of a proprietary language provided they don't copy the source code of the implementation itself.

The EU ruling conflicts with that of the jury in the Google v Oracle trial, where it was decided that Google's re-implementation of Java APIs in the Android OS did infringe Oracle's copyright. Implementing a SAS procedure and a Java API are similar programming efforts, and the EFF has argued convincingly that preventing programmers from re-implementing APIs harms innovation. Nonetheless, the US-based jury did not decide if Google's Android implementation was a fair use infringement of copyright; if that question were to be resolved in Google's failure the outcome would have a similar practical effect as the EU decision.

JDSupra: European Union Court Rules that Software Functions Cannot Be Copyrighted 

 

 

 

Statistician Hans Rosling on the Time 100 list

Hans Rosling, the statistician behind the Joy of Stats documentary, the creator of Gapminder (which in turn inspired Google MotionCharts), and boundlessly enthusiastic presenter of statistical concepts, is named in this year's Time list of the 100 Most Influential People in the World. Great to see a statistican recognized in this way (have there been any other statisticians on the Time 100 list?). You can read the profile of Hans Rosling on the Time website.

Bigger data, better intelligence for government

Tomorrow (at 2PM Eastern on Thursday), the White House White House Office of Science and Technology Policy will host a 90-minute forum on Challenges and Opportunities in Big Data. You can watch the event live at the previous link, and see federal government science heads from OSTP, NSF, NIH, DoE, DoD, DARPA and USGS outline how their agencies are engaged in Big Data research. The event will also include a discussion by a panel of thought leaders from academia and industry, moderated by Steve Lohr of the New York Times.

Naturally, the event has prompted some speculation about the government's "big data agenda", but this in fact a direction the administration has been pushing ever since the Obama appointed Aneesh Chopra as the nation's first CTO in 2009. (Chopra passed the mantle to current CTO Todd Park just this this month.) One of the first initatives was the creation of data.gov, a vehicle to make government data available in a timely and accessible format for a community of citizen data scientists to make sense of it all. For example, independent analysis of data from data.gov (using Big Data statistical analysis techniques) revealed a flaw in the 2000 Census results that went unnoticed by the Census Bureau.

And there's the rub: while much of the talk around Big Data has been around storage and search for massive data sets, the real value comes from analysis and inference on the data themselves. I'd expect to hear more from the administration on this topic in today's forum. The administration is already pushing to use more open-source software in government, the open-source R language (along with the big-data enhanced distribution from Revolution Analytics) is already widely used at NIST, FDA, NOAA, CIA, and many other government agencies. Given that the data storage and data analysis infrastructure is already well in place, I expect we'll hear a vision for new kinds of applications, citizen services and data-driven strategies from the participating federal agencies in today's forum.

You can join the forum in the live-streaming webcast at the link below. 

NSF: OSTP and NSF Present: Challenges and Opportunities in Big Data (2PM EST, Thu Mar 29 2012)

Updated Mar 28 to correct the time of the forum.

Kony 2012: How to weaken your argument with charts

If the goal of the Invisible Children campaign, which has received millions of dollars of contributions since the Kony 2012 video went viral, is to convince us that the money is being put to humanitarian efforts, they could do a lot better than this chart: 

Putting 37% of expenses into programs in Africa is a decent result -- many charities have much higher overheads. But with its weirdly varying-width segments the chart above isn't telling a clear story. The "Awareness Programs" segment (26% of expenses) is, by my eye, less than half the area of "Central Africa Programs". "Management and General", at just over 16%, looks about the same area as the taller, yet narrower, "Awareness Products" segment (9.56%). I'm sure it's just the work of an overzealous graphical designer, but when you're trying to tell a story with data, shenanigans like this just makes it seem like you have something to hide.

By the way, here's a simpler version of the chart (created with the R language) that makes it easier to compare the relative contributions of the expense categories.

kony <- c(Africa=37.14, "Awareness Programs"=25.98, Management=16.24, 
  "Awareness Products"=9.56, "Media Creation"=7.87,Fundraising=3.22)
barplot(kony,ylab="Percent of expenses")

Invisible Children: Critiques

Bad Science at Strata 2012

Ben Goldacre, the physician and  biostatistician behind the always-excellent Bad Science column in the Guardian, gave a barnburner of a talk at Strata 2012 yesterday, "The Information Architecture of Medicine is Broken". For anyone not aware of the problems caused by publication bias in clinical trials (for example, ineffective drugs with a wide variety of side-effects coming to market), his talk is a must-watch. There's also a great explanation of meta-analysis (using the logo of the Cochrane Collaboration as a backdrop).

Goldacre is attempting to shed light on the issue of negative results in clinical trials going unreported, by matching the register of planned clinical trials with those that are actually published. At around 12:00 in the video below, learn how you can help with this important project. You'll also spot an R chart or two during the presentation.

 

Strata 2012: The Information Architecture of Medicine is Broken

R charts used for analysis at Politico

Zack Abrahamson, the "data whiz" at political analysis site Politico, is apparently an R user. Politico's Feb 10 2012 chart of the day clearly uses the ggplot2 graphics package and (quoting Politico)

looks into the disenchanted slice of the GOP that’s not engaged with its party’s primary. And that slice doesn’t like Mitt Romney. People say turnout's down. When it's down, we actually see Romney stronger. [Minnesota is an exception.] This is a general election problem because the people who don't show up today won't show up in November. It’s the CPAC base. Here’s a chart comparing the change in turnout from 2008 to 2012 against the change in Romney support.

Those counties that had higher turnout in the 2012 primaries than in 2008 were associated with a comparatively lower support for Republican presidential candidate Mitt Romney. If I were to nitpick, the legend and color coding is a bit redundant here but, still: it's nice to techniques like small multiples, scatterplots and trensdlines used for political analysis.

(Note: the content of the link below changes daily. I couldn't find a permanent link for the Feb 10 Morning Report where the chart and quote above appeared.)

Politico: Morning Score (via @alexlundry)

R Chart featured in Facebook IPO

Page 7 of Facebook's 213-page S-1 filing for their record-breaking IPO includes the following chart, under the headline: "Our Mission: To make the world more open and connected".

This chart was created using the R language and Hadoop by Facebook intern Paul Butler. (Thanks to the blog IOER Tools for first noticing the inclusion of the chart.)

And speaking of R charts appearing in widely-read places, the latest issue of the prestigious science journal Nature includes several charts created in R in an article on the evolution of the nematode C. elegans. Hadley Wickham, author of the ggplot2 package, noted that his package was used to create one of the charts.

NYT uses R to map the 1%

Last Saturday, the New York Times published a feature article on the wealthiest 1% of Americans. The on-line version of the article included interactive features like this interactive map showing where your household ranks in the country and in local regions. The print edition, however, included some different (and necessarily static) representations of US wealth data, such as this map of where the wealthiest 1% live:

(Image credit: New York Times, January 14 2002.)

Kevin Quealy, graphics editor at the New York Times, used R to help create this chart. The process began with an initial proof of concept, which used R's maptools package and required five lines of R code. 

The original plan was to use ArcMap to generate the chart, but ArcView lacks the flexibility of styling the output that R has. Later iterations added color and algorithmically moved the labels around for legibility, and some final tweaking outside of R led to the chart in the print edition.

Interestingly, the chart doesn't appear in the on-line edition of the NYT. I knew that some features like interactive charts, were online only, but I wasn't aware that the reverse was also true, and that special content was made only for the print edition. Quealy points out that this is a necessity:

This is a good example, I think, of using each medium to its best potential, meeting the design constraints of each. More and more, this means making totally separate versions of things – admittedly, it frequently takes twice the time and energy – but the mediums are just so different that works well in one just doesn’t work well in another. 

For the full story of how this lovely chart was created, check out Kevin Quealy's tubmlr linked below.

chartsnthings: Before, During and After: The Richest 1 Percent

Mapping the Iowa caucus results: how it's done with R

If you've been following the presidential primary process here in the US, you've probably seen many maps of the results of the Iowa caucuses by now (such as this infamous one from Fox News). But you might be interested to learn how such maps can be made using the R language.

The Offensive Politics blog explains the process in detail, from accessing the data (helpfully provided as a Google Fusion table), to using R packages (including maptools, ggplot2, RColorBrewer, and gpclib) to transform and chart the table, and creating heat maps like this one, showing the percentage point advantage Romney had over Santorum in each Iowa county:

You can find more such charts, along with the detailed instructions and R code, at the link below.

Offensive Politics: Mapping the Iowa GOP 2012 Caucus Results