Revolutions

The role of Statistics in the Higgs Boson discovery

News is starting to leak that the Large Hadron Collider may have accomplished its primary mission of confirming the existence of the hypothesised and heretofore elusive subatomic particle, the Higgs Boson. And sure, billions of Euros worth of state-of-the-art high-energy machinery and an army of experimental and theoretical physicists probably had something to do with the discovery. But did you know Statistics played a part as well? Check out this explainer video from PhD comics, below (an R chart even appears at the 00:27 mark): 

The basic method the LHC uses to detecting the Higgs Boson is to generate decay products from subatomic collisions, and to generate charts like the one below:

 

Depending on whether or not the Higgs Boson exists, this chart will look different. But the difference between what this chart looks like given that the Higgs Boson exists, or given that it doesn't, is very very small:

As the physicist interviewed in the video says, to resolve this difference "what you need is a HUGE amount of data". And we're talking REALLY huge: experiments like this are run 40 million times a second every day. (That's more than 40 terabytes of new data every day: now, that's Big Data!) Every day since the LHC was turned on, more evidence for the Higgs model has been accumulating, and it seems that now enough has accumulated for the researchers to be confident the Higgs Boson does indeed exist. Look for the formal confirmation in the next couple of days. 

NYT charts the Facebook IPO with R

In conjunction with Facebook's record-setting IPO last Thursday, the New York Times created an infographic to put the size of the offer in context with other recent IPOs. A detail of the graphic as it appeared in the print edition appears below:

ChartsNThings gives a fascinating peek into the weeklong process that went into creating this chart, where about a dozen "sketches" of charts were tried and considered until the final chart was selected when Thursday's deadline arrived. It's a great look at the developmental process that goes into creating the quality infographics that are the hallmark of the NYT team led by graphics editor Amanda Cox. It's also a testament to the value of working in the R language (used to create all of the prototype charts), which is expressly designed to encourage experimentation and creativity in data analysis and visualization. Plus, the highly expressive nature of the language allows such prototypes to be created quickly, allowing the NYT team to iterate through many alternatives as the deadline loomed.

The post also highlights an interesting angle related to the differing requirements of the web version and print version of NYT infographics: 

If you’ve seen the web version, though, you know it doesn’t look like this. [Amanda thinks print graphics can be smarter than web graphics.] For one, the browser window doesn’t give us this kind of space. But the medium itself plays a part too. Online, if you’re not engaged in 10 seconds, you’re not going to stay on the page, so they needed to keep it fun.

The prototype below was considered for the print version, but was instead ultimately adapted for the web:

The fun aspect here is that as you switch between the different view on the web, it's not just a slideshow; each IPO "bubble" moves into it's new position as the view changes. Like a motion chart, it's a great way to track comparisons of individual data points between data views. (Try it out here.)

There are lots more details about the process of creating these charts at the post linked below; the whole thing is a great read.

ChartsNThings: Amanda Cox and countrymen chart the Facebook I.P.O.

The grade level of Congress speeches, analyzed with R

As widely reported by CNN, the Huffington Post, Talking Points Memo, the sophistication of speeches by US politicians has declined in recent years, dropping from an 11th-grade level in 2005 to a 10th-grade level today. The reports are based on an analysis by the Sunlight Foundation, based on textual analysis of congressional speeches given since 1996 provided by the Capitol Words API. You can see where your favourite legislator ranks in this table. Picking out a couple of famous names, Sen. Rand Paul (R-KY) ranks near the bottom with an 8th-grade speaking level; near the top of the ranks is retiring senator Olympia-Snowe (R-ME) who speaks at a Grade 14 level. 

A regression analysis detailed in the article teases out some of the contributing factors. While the speech levels of members of both parties have declined over the past couple of years, the chart below shows that for Republicans, the more ideological extreme the congressperson, the less sophisticated the speech.

It's clear that the data visualizations were created using R's ggplot2 package; presumably the statistical analyses were peformed with the R language as well. For the infographic circulated to the media, the ggplot2 charts have been cleaned up using an editing tool like Illustrator (easy if you export the chart to PDF from R).

Read the details of the analysis in the full report from the Sunlight Foundation linked below.

Sunlight Foundation: The changing complexity of congressional speech

Mariano Rivera’s baseball prowess, illustrated with R

Kevin Quealy, graphics editor at the New York Times, has published another fascinating behind-the-scenes look at how the Times creates data visualizations for print and online. In his latest post, he looks at how a visualization of the Yankee's Mariano Rivera performance compared to other Major League Baseball pitchers was created. (Detail below, click for the full image.) 

The infographic began its life as a hand-drawn sketch, that begat a line-chart created using R (based on data scraped from the Web). The R chart was was then cleaned up and annotated using Adobe Illustrator for publication. One interesting detail of the process: the source R graph is deliberately created using garish colours (purples, greens, etc.) to make the color-selection process easier in Illustrator.

Check out the ChartsNThings archive for other great studies of the process of data journalism. Many of the case studies involve the use of R code, such as these visualizations of visitors to the White House, Santorum's primary support, Santorum/Romney exit poll data, NFL players mentioned on ESPN, the defense budget and the richest 1%.

ChartsNThings: Sketches: How Mariano Rivera Compares to Baseball’s Best Closers

EU court's SAS ruling conflicts with Oracle v Google

In a blow to SAS's efforts to litigate competitor and low-cost SAS clone WPS out of existence, the European Union High Court has ruled that programming languages can't be copyrighted. SAS Institute (Cary, NC) had claimed that the WPS software — which allows users to process SAS data files and SAS "data step" scripts without SAS software — breached SAS's copyright in its re-implementation of SAS functionality. But given that WPS did not study SAS source code, and merely reimplemented its interfaces and behaviour based on observation, the court ruled that this was not a violation. In a press release following the decision, the court stated:

 ... neither the functionality of a computer program nor the programming language and the format of data files used in a computer program in order to exploit certain of its functions constitute a form of expression. Accordingly, they do not enjoy copyright protection.  

(Emphasis in press release.) SAS has issued no comment on the ruling to date.

The case has implications beyond SAS and WPS. In Europe, at least, it implies that developers of both proprietary and open-source software have the right to duplicate the functionality and exposed interfaces (in this case, the structure of SAS procedure calls) of a proprietary language provided they don't copy the source code of the implementation itself.

The EU ruling conflicts with that of the jury in the Google v Oracle trial, where it was decided that Google's re-implementation of Java APIs in the Android OS did infringe Oracle's copyright. Implementing a SAS procedure and a Java API are similar programming efforts, and the EFF has argued convincingly that preventing programmers from re-implementing APIs harms innovation. Nonetheless, the US-based jury did not decide if Google's Android implementation was a fair use infringement of copyright; if that question were to be resolved in Google's failure the outcome would have a similar practical effect as the EU decision.

JDSupra: European Union Court Rules that Software Functions Cannot Be Copyrighted 

 

 

 

Statistician Hans Rosling on the Time 100 list

Hans Rosling, the statistician behind the Joy of Stats documentary, the creator of Gapminder (which in turn inspired Google MotionCharts), and boundlessly enthusiastic presenter of statistical concepts, is named in this year's Time list of the 100 Most Influential People in the World. Great to see a statistican recognized in this way (have there been any other statisticians on the Time 100 list?). You can read the profile of Hans Rosling on the Time website.

Bigger data, better intelligence for government

Tomorrow (at 2PM Eastern on Thursday), the White House White House Office of Science and Technology Policy will host a 90-minute forum on Challenges and Opportunities in Big Data. You can watch the event live at the previous link, and see federal government science heads from OSTP, NSF, NIH, DoE, DoD, DARPA and USGS outline how their agencies are engaged in Big Data research. The event will also include a discussion by a panel of thought leaders from academia and industry, moderated by Steve Lohr of the New York Times.

Naturally, the event has prompted some speculation about the government's "big data agenda", but this in fact a direction the administration has been pushing ever since the Obama appointed Aneesh Chopra as the nation's first CTO in 2009. (Chopra passed the mantle to current CTO Todd Park just this this month.) One of the first initatives was the creation of data.gov, a vehicle to make government data available in a timely and accessible format for a community of citizen data scientists to make sense of it all. For example, independent analysis of data from data.gov (using Big Data statistical analysis techniques) revealed a flaw in the 2000 Census results that went unnoticed by the Census Bureau.

And there's the rub: while much of the talk around Big Data has been around storage and search for massive data sets, the real value comes from analysis and inference on the data themselves. I'd expect to hear more from the administration on this topic in today's forum. The administration is already pushing to use more open-source software in government, the open-source R language (along with the big-data enhanced distribution from Revolution Analytics) is already widely used at NIST, FDA, NOAA, CIA, and many other government agencies. Given that the data storage and data analysis infrastructure is already well in place, I expect we'll hear a vision for new kinds of applications, citizen services and data-driven strategies from the participating federal agencies in today's forum.

You can join the forum in the live-streaming webcast at the link below. 

NSF: OSTP and NSF Present: Challenges and Opportunities in Big Data (2PM EST, Thu Mar 29 2012)

Updated Mar 28 to correct the time of the forum.

Kony 2012: How to weaken your argument with charts

If the goal of the Invisible Children campaign, which has received millions of dollars of contributions since the Kony 2012 video went viral, is to convince us that the money is being put to humanitarian efforts, they could do a lot better than this chart: 

Putting 37% of expenses into programs in Africa is a decent result -- many charities have much higher overheads. But with its weirdly varying-width segments the chart above isn't telling a clear story. The "Awareness Programs" segment (26% of expenses) is, by my eye, less than half the area of "Central Africa Programs". "Management and General", at just over 16%, looks about the same area as the taller, yet narrower, "Awareness Products" segment (9.56%). I'm sure it's just the work of an overzealous graphical designer, but when you're trying to tell a story with data, shenanigans like this just makes it seem like you have something to hide.

By the way, here's a simpler version of the chart (created with the R language) that makes it easier to compare the relative contributions of the expense categories.

kony <- c(Africa=37.14, "Awareness Programs"=25.98, Management=16.24, 
  "Awareness Products"=9.56, "Media Creation"=7.87,Fundraising=3.22)
barplot(kony,ylab="Percent of expenses")

Invisible Children: Critiques

Bad Science at Strata 2012

Ben Goldacre, the physician and  biostatistician behind the always-excellent Bad Science column in the Guardian, gave a barnburner of a talk at Strata 2012 yesterday, "The Information Architecture of Medicine is Broken". For anyone not aware of the problems caused by publication bias in clinical trials (for example, ineffective drugs with a wide variety of side-effects coming to market), his talk is a must-watch. There's also a great explanation of meta-analysis (using the logo of the Cochrane Collaboration as a backdrop).

Goldacre is attempting to shed light on the issue of negative results in clinical trials going unreported, by matching the register of planned clinical trials with those that are actually published. At around 12:00 in the video below, learn how you can help with this important project. You'll also spot an R chart or two during the presentation.

 

Strata 2012: The Information Architecture of Medicine is Broken

R charts used for analysis at Politico

Zack Abrahamson, the "data whiz" at political analysis site Politico, is apparently an R user. Politico's Feb 10 2012 chart of the day clearly uses the ggplot2 graphics package and (quoting Politico)

looks into the disenchanted slice of the GOP that’s not engaged with its party’s primary. And that slice doesn’t like Mitt Romney. People say turnout's down. When it's down, we actually see Romney stronger. [Minnesota is an exception.] This is a general election problem because the people who don't show up today won't show up in November. It’s the CPAC base. Here’s a chart comparing the change in turnout from 2008 to 2012 against the change in Romney support.

Those counties that had higher turnout in the 2012 primaries than in 2008 were associated with a comparatively lower support for Republican presidential candidate Mitt Romney. If I were to nitpick, the legend and color coding is a bit redundant here but, still: it's nice to techniques like small multiples, scatterplots and trensdlines used for political analysis.

(Note: the content of the link below changes daily. I couldn't find a permanent link for the Feb 10 Morning Report where the chart and quote above appeared.)

Politico: Morning Score (via @alexlundry)