Revolutions

Because it's Friday: Analytics of an Engagement

The always-interesting R blogger and political scientist Drew Conway -- and self-styled luckiest man on earth -- just announced some very happy news: he just got engaged to his future wife! Congratulations, Drew! (Drew proposed the traditional way, on a lovely European vacation. But if he'd just held off a couple of days he could have proposed via towel-clad adonis, instead.)

Always the analyst, Drew did manage to capture some metrics on the impact of the engagement: the rate of posts to his fiancée's facebook wall after she changed her relationship status to "Engaged" (click to enlarge):

Drew used the Facebook API to capture the data, R to create a kernel smooth of the post rate segmented by male and female friends, and the ggplot2 package to chart the results. Looks like the women are quicker to respond, but the men keep up the posts longer. (Which reminds me: I still have some Christmas cards to write around here somewhere...)  

And the chart makes for a lovely design for a wedding invitation, too! 

Zero Intelligence Agents: Anatomy of a Life-Milestone Announcement on Facebook

Charting the World Cup

Now that Spain has won the World Cup, it's interesting to go back and look at some metrics from the matches and see if we can tease out what characteristics made for a winning Cup team this time around. Fortunately, the Guardian's Data Blog has made a wealth of World Cup statistics available, with data on every player of every team (position, shots at goal, passes, tackles made, and saves), plus aggregate statistics for each team (goals, % shots on target, fouls, and much more). The data are ripe for analysis in R, especially given that you can download the data directly from the cloud as an R object with the following commands:

players <- read.csv("http://spreadsheets.google.com/pub?key=tOM2qREmPUbv76waumrEEYg&single=true&gid=1&range=A1%3AH596&output=csv")

teams <- read.csv("http://spreadsheets.google.com/pub?key=tOM2qREmPUbv76waumrEEYg&single=true&gid=0&range=A1%3AAG15&output=csv")

The method I've described before for accessing a Google Spreadsheet from R didn't quite apply here, as those instructions assume you own the document (and have access to the Publish menu). But some experimentation and tweaking of the spreadsheet URL made it work: the key parameters seem to be the "&gid=" (sheet number) and "%range=" (cell ranges, use %3A to encode the colon) and "&output=csv" to download in CSV format. It would be nice if Google published the specs to form URLs like these, but as far as I know they don't. 

Anyway, a couple of bloggers have used these data to great effect to express the results of the World Cup visually using R graphics. For example, the R Charts blog used ggplot2 to look at the number of fouls committed by each team during the tournament:

 

(Personally, I would have sorted the rows by descending number of fouls, rather than alphabetically.) Interesting to see that Cup champions Spain are in the middle of the pack on fouls, whereas runners-up Netherlands lead this table (boosted heavily by their performance in the final).

Blogger Jason Priem also took a look at the data, this time with a scatterplot of goals per game by fouls per game, related to how far each team advanced in the competition:

 

(Download Jason's code for this chart here.) Again it's interesting to see the positions of the two finalists here, with Netherlands on the extreme frontier for both fouls and goals, while spain is moderate on goals per game and near the lowest on fouls per games.

It's a rich dataset and I'm sure other Revolutions readers could come up with some equally interesting visualizations. If you do, tell us about it in the comments.

Guardian Data Blog: World Cup 2010 statistics: every match and every player in data

Visualizing the census

Now that the 2010 survey is over, you might be wondering what we can learn from the data when the aggregated results are published. For a good guide to the kinds of questions you'll be able to answer, take a look at StatJump, where you can see tables and charts of the results of the 2000 census: population data (e.g. the largest cities), social demographic data (where are the most high school graduates?), economic data (which cities have the longest average commute?), and housing data (where I learned I work in the city with the most houses valued over $1M -- although the housing crash may change that when the 2010 data are out). Nicest of all, many of the variables are presented as choropleths (created with R, of course), so you can see maps of the US showing how the variables are distributed geographically, like this:

Percent of People Born in U.S. by County:

Check out the StatJump website at the link below to check out dozens of other charts and tables from the 2000 Census.

StatJump: Home Page

The impact of the drug war in Mexico

For the last couple of years, Mexico has been in the midst of an escalating drug war, with violent crime on the upswing in many areas. But tracking the impact quantitatively is difficult: in Mexico, about 85% of crimes go unreported, and corruption leads to inaccurate reporting in some districts. Diego Valle has taken on the task of visualizing and analyzing the available data with R, and has come up with some startling results. For example, by comparing national with local statistics, Diego has identified a massive underreporting of 1153 murders in the state of Chihuahua:

Diego even employs Benford's Law to uncover evidence of data falsification. There's lots more fascinating analysis in Diego's report, including this choropleth of homicide rate across Mexico:

On a personal note, we'd planned to return to a favorite vacation spot near Zihuatanejo this year. After hearing from locals that the once-tranquil village had been overrun, we cancelled our planned trip. Zihua is near the centre of that hotspot on the south-central coast.

Read Diego's full post for lots more great analysis. He's made all of his R code available, too.

Diego Valle's Blog: Statistical Analysis and Visualization of the Drug War in Mexico

The distribution of online data usage

AT&T has recently announced it will no longer offer unlimited data plans for new iPhone users in the US, and now some carriers in the UK have followed suit. In each case, the providers claim that only a very small number of users actually use enough data to warrant an unlimited plan, and most users use relatively little and will benefit from the cheaper, capped plans. But what does "most users" mean? Journalist Charles Arthur of the Guardian, in an article about the impact of the data caps, illustrated the distribution of data usage amongst mobile users with this chart, created in R:

Reader Barry Rowlingson thought this chart looked odd (is that really a 97% percentile?), but given that the chart lacks axis labels, it's hard to make sense of anyway. So he set to recreate the chart: given the mean (200Mb) and one quantile (the 97% quantile is 500Mb), he figured out the unstated standard deviation of the Normal distribution (159Mb) and recreated the graph with axis labels:

ss=seq(0,700,len=100)
plot(ss,dnorm(ss,mean=200,sd=159),col="red",type="l")
abline(v=500,col="black")

 
I added the black vertical line to indicate the 97% percentile at 500Mb. Anyway, it's nice that R is being used to illustrate statistical concepts like this, just a shame that the chart wasn't quite right. Thanks to Barry for providing this alternative.

The Guardian: Why file-sharing has killed 'unlimited' mobile data contracts

Which font uses the most ink?

If you're being particularly cost-conscious about your use of printer ink or toner, you may be wondering which font you should choose to minimize ink use. Here's an infographic with the answer:

This is an interesting infographic in its own right, but what makes it cool is that these are not photoshopped images of Bic biros. Matt Robinson created this chart by writing the word "Sample" on a wall using a brand new biro for each font; the amount of ink remaining indicates the density of the type. In other words, the data is the chart. Brilliant! Click the link below to see how he did it.

Matt Robinson: Measuring Type 

A free book on Geostatistical Mapping with R

Tomislav Hengl of the University of Amsterdam has published new book, A Practical Guide to Geostatistical Mapping. It's jam-packed with 291 pages on mapping and analyzing spatial data using free software including R, SAGA, GRASS, ILWIS and Google Earth, and freely-available map data. The book itself is also available for free, as an Open Access Publication. You can order the book in printed form for US$12.78, or download it for free as a PDF.

Surprisingly (given the title), this book isn't just about visual displays of spatial data. In fact, the first two chapters offer a nice overview of statistical analysis of spatial data (although with a greater focus on continuous-field models than point-process models). If you want a concise overview of regression-kriging, this is a great resource.

Chapter 3 addresses the various software tools you'll use to analyze the data and create the maps. Some care has been taken in considering how the software elements should be integrated, and Hengl recommends a "R on top" model, where R scripts drive the other tools. 

This is a clever move: making use of the scripting capabilities of R means you can avoid much of the tedious manual back-and-forth activities that are usually associated with working with several software tools. Hengl offers some other reasons for working with R, too (p. 90):

• It is of high quality — It is a non-proprietary product of international collaboration between top statisticians. 
• It helps you think critically — It stimulates critical thinking about problem-solving rather than a push the button mentality. 
• It is an open source software — Source code is published, so you can see the exact algorithms being used; expert statisticians can make sure the code is correct.
• It allows automation — Repetitive procedures can easily be automated by user-written scripts or functions.
• It helps you document your work — By scripting in R, anybody is able to reproduce your work (processing metadata). You can record steps taken using history mechanism even without scripting, e.g. by using the savehistory() command.
• It can handle and generate maps — R now also provides rich facilities for interpolation and statistical analysis of spatial data, including export to GIS packages and Google Earth. 

Chapter 4 covers the various auxiliary data sources available, listing sources global environmental and socio-economic data, and sources of maps and satellite imagery like GADM, Google Earth and MODIS. 

The remaining chapters are devoted to worked examples of spatial data analysis and mapping. By working through the examples, you can recreate charts like these (click to enlarge):

One minor complaint: most of the images in the book are in black-and-white (most likely to facilitate the printing process). But at least you have the R scripts and data for all exercises (these, plus updated maps, are available from the book's website), so at least you can re-run the examples in R to recreate them in color.

Tomislav Hengl: A Practical Guide to Geostatistical Mapping (via @fernando_mayer)

How ideological is Google?

Adam Bonica, a grad student in political science at NYU, recently published a ranking of the political slant of various professions, based on the amount and recipient (Republican or Democratic) of political donations by lawyers, lobbyists, physicians and many other occupations. This paper (PDF) gives the complete analysis, but the chart below (created using the ggplot2 graphics package in R) sums up the results nicely (click to enlarge): 

(I liked this quote from Paul Kedrosky about this chart: "How come gas station attendants are so damn partisan?") Now, Adam has taken the analysis to the next level, by looking at employees of individual companies, instead of professions as a whole. 

(Unfortunately, it's not entirely clear if the data point lies to the left or at the middle of the company label; this table helps sort out the exact rankings.) It may not be a surprise, for example, that Google's employees tend to give to Democratic-leaning candidates, but does that influence Google's policies as a whole? Adam's article at the link below delves into this question in more detail.

Ideological Cartography: The University of Google: Was the decision to exit China ideological or business as usual?

Smoothing time series with R

Smoothing is a statistical technique that helps you to spot trends in noisy data, and especially to compare trends between two or more fluctuating time series. It's a useful visualization tool that I'm pleased to see cropping up more and more in statistical graphics on the Web -- it's now a staple in econometric charts and is heavily used in polling analysis. For example, here's smoothing used to combine data from various polls over time on Obama's job approval (from pollster.com).

The S language was, to the best of my knowledge, the first software that made statistical smoothing a core part of the graphics system: first with the lowess function and later with other more powerful alternatives. These days in R (S's successor), loess (local polynomrial regression fitting) is the usual go-to alternative for smoothing. With just a couple of lines of code, you can take a noisy time series in R and overlay a smooth trend line to guide the eye. Nathan Yau at FlowingData shows us how to take data like this:

and, with just a few lines of R code and some touching-up in Illustrator, create a chart like this:

FlowingData: How to: make a scatterplot with a smooth fitted line

Video: Hadley Wickham gives a short course on graphics with R

Hadley Wickham (the creator of the popular ggplot2 graphics package for R) has posted video of a 2-hour short course on Visualisation in R at his blip.tv channel. The video is split into four thirty-minute segments:

• Basic Graphics
• Displaying Large Data
• Data manipulation and transformations
• Polishing your plots for publication

The course is peppered with self-guided exercises, for which data, code and the supporting slides are all available. So if you want to be able to create beautiful, informative visualizations of data sets large and small, like this:

then it's well worth two hours of your time to watch these videos. (And don't forget Hadley's book, ggplot2: Elegant Graphics for Data Analysis, for a complete reference.)

Hadley Wickham: data vis mini course