Revolutions

R Chart featured in Facebook IPO

Page 7 of Facebook's 213-page S-1 filing for their record-breaking IPO includes the following chart, under the headline: "Our Mission: To make the world more open and connected".

This chart was created using the R language and Hadoop by Facebook intern Paul Butler. (Thanks to the blog IOER Tools for first noticing the inclusion of the chart.)

And speaking of R charts appearing in widely-read places, the latest issue of the prestigious science journal Nature includes several charts created in R in an article on the evolution of the nematode C. elegans. Hadley Wickham, author of the ggplot2 package, noted that his package was used to create one of the charts.

NYT uses R to map the 1%

Last Saturday, the New York Times published a feature article on the wealthiest 1% of Americans. The on-line version of the article included interactive features like this interactive map showing where your household ranks in the country and in local regions. The print edition, however, included some different (and necessarily static) representations of US wealth data, such as this map of where the wealthiest 1% live:

(Image credit: New York Times, January 14 2002.)

Kevin Quealy, graphics editor at the New York Times, used R to help create this chart. The process began with an initial proof of concept, which used R's maptools package and required five lines of R code. 

The original plan was to use ArcMap to generate the chart, but ArcView lacks the flexibility of styling the output that R has. Later iterations added color and algorithmically moved the labels around for legibility, and some final tweaking outside of R led to the chart in the print edition.

Interestingly, the chart doesn't appear in the on-line edition of the NYT. I knew that some features like interactive charts, were online only, but I wasn't aware that the reverse was also true, and that special content was made only for the print edition. Quealy points out that this is a necessity:

This is a good example, I think, of using each medium to its best potential, meeting the design constraints of each. More and more, this means making totally separate versions of things – admittedly, it frequently takes twice the time and energy – but the mediums are just so different that works well in one just doesn’t work well in another. 

For the full story of how this lovely chart was created, check out Kevin Quealy's tubmlr linked below.

chartsnthings: Before, During and After: The Richest 1 Percent

Mapping the Iowa caucus results: how it's done with R

If you've been following the presidential primary process here in the US, you've probably seen many maps of the results of the Iowa caucuses by now (such as this infamous one from Fox News). But you might be interested to learn how such maps can be made using the R language.

The Offensive Politics blog explains the process in detail, from accessing the data (helpfully provided as a Google Fusion table), to using R packages (including maptools, ggplot2, RColorBrewer, and gpclib) to transform and chart the table, and creating heat maps like this one, showing the percentage point advantage Romney had over Santorum in each Iowa county:

You can find more such charts, along with the detailed instructions and R code, at the link below.

Offensive Politics: Mapping the Iowa GOP 2012 Caucus Results

Propagation of the news of OBL's death via Twitter

SocialFlow's blog has a great case study today on how news from a single tweet -- in this case, speculation made an hour before the President's announcement that Osama bin Laden had been killed -- can propagate through social networks.

At 10:24 p.m. EST on Sunday May 1, Keith Urbahn tweeted: "So I'm told by a reputable person they have killed Osama Bin Laden. Hot damn." This information was rapidly repeated (in Twitter parlance, retweeted) by Keith's followers, and their followers, and so on. The rate of dissemination peaked rapidly as shown in the chart below (which I'm pretty sure is a chart created in R):

But it's the way in which the information propagated through the Twitter social network that's fascinating. When NYT reporter Brian Stelter retweeted, the information spread rapidly:

Read the full post at the SocialFlow blog for the full story of how information like this propagates through Twitter.

SocialFlow: Breaking Bin Laden: visualizing the power of a single tweet

How to load your iPhone location data into R

Earlier this week, data scientists Pete Warden and Alasdair Allen reported that iPhones and cell-enabled iPads keep an internal log of the devices location, which is accessible from the backup that iTunes creates when you sync the device. (Update Apr 27: Apple responds that the locations are those of nearby cell towers and wi-fi hotspots, not the device itself.)  Naturally, there's been some controversy over the privacy implications of the data keing kept, but from a data scientist's perspective this represents a rich and interesting data source for analysis. Personally, I'm kind of interested to get access to where I've been over the past year: wherever I go, my iPhone goes.

Pete Warden has provided a tool that lets you easily access and map the data. Here's why the data for my iPhone looks like when zoomed in on California:

If you have more than one iDevice the tool only shows the data from the one you most recently synced, and more importantly it doesn't (directly) give you access to the location data if you want to do more with it. Drew Conway comes to the rescue with an R package (amusingly called stalkR) that lets you import the data from a named device into an R object for visualization or other analysis. You'll need to make sure the dependent R packages are installed, and download and install the stalkR_0.01.tar.gz file in your R working directory:

install.packages("RSQLite")
install.packages("XML")
install.packages("ggplot2")
install.packages(c("maps","mapproj"))
install.packages("stalkR_0.01.tar.gz", repos=NULL, type="source")
 
library(stalkR)
iphone.locs<-get.mylocations("dsmith", "David Smiths iPhone")
viz.locations(iphone.locs, "usa")

The last two commands are the key ones. get.mylocations creates a data frame with timestamped latitute and longitude, along with a horizontal accuracy measure. There's also an Altitude variable, but at least in my data it was always zero (probably because the tracking data comes from cell tower triangulation, not GPS). The vis.locations function uses the syntax of the map function to map the locations, but the nice thing about having the data in R is that you can use it in whatever way you like.

Drew Conway's github: stalkR

 

Visualizing Tax Day

Today is Tax Day in the United States, when all tax returns are due to the IRS. But where do all those tax dollars go? The winners of the Google data visualization competition to visualize the 2009 Federal budget were announced today, and the winning entry has a similar feel to the official White House 2010 Federal Tax Recepit application. Personally, I preferred the runner-up entry which represented taxes not as dollars sent to the federal government, but as hours, minutes and seconds you spend working to fund the various federal services and programs.

The team at Revolution Analytics also took a look at the federal budget and represented the expenditures in this infographic (click to enlarge):

You can download a high-res version of this infographic here: Do You Know Where Your Taxes Go? (155.2K)

Visualizing tax brackets

With Tax Day fast approaching here in the US, there's been a lot of discussion about tax policy and in particular the tax rates paid by the highest income earners. Like in many countries, here the income tax system is bracketed:

Tax Bracket	Single	Married Filing Jointly
10% Bracket	$0 – $8,375	$0 – $16,750
15% Bracket	$8,375 – $34,000	$16,750 – $68,000
25% Bracket	$34,000 – $82,400	$68,000 – $137,300
28% Bracket	$82,400 – $171,850	$137,300 – $209,250
33% Bracket	$171,850 – $373,650	$209,250 – $373,650
35% Bracket	$373,650+	$373,650+

But there's a lot of confusion about how those brackets work. (One reason is apparently that official information is hard to come by: 5 minutes of Google searching ought to be enough to find an IRS page simply describing those brackets like in the table above, but I, for one, failed at the task.) While a single person earning $34,500 (in adjusted, not total income, BTW) is in the 25% bracket, this doesn't mean they are liable for 34,500*25% = $8850 in taxes. In fact, only $500 (the amount over the bracket threshhold) is subject to 25% taxes; $8,375 is subject to 10% taxes, and $25,625 (the amount in bracket 2) is subject to 15% taxes. Of course, all of this is subject to the myriad complications and loopholes of US tax law (and I"m not a tax lawyer), but the basic principle is sound. So if you hear someone complaining that they're heading into a new tax bracket, or hoping to find deductions to keep them in a lower one, that's all well and good, but the effect of moving from one bracket to the next might not be as great as they think.

So what is the effect of the tax brackets? The effective tax rate for our $34,500 earner is 13.9% ($4806), not 25% (as calculated in the R language):

> sum(c(8375, 25625, 500)*c(0.1, 0.15, 0.25))/34500
[1] 0.1393116

If we do that same calculation over a range of possible incomes (or use the data from IRS tables[*]) and calculate the effective tax rate for both singles and straight[**] married couples, you can create a chart that gives a great picture of the actual impact of the tax brackets. A reader of the Daily Dish did just that (possibly in R?):

The squares/circles represent the boundaries of the tax brackets; as you can see, moving from one bracket to the next has little impact on the effective tax rate (and therefore the tax paid). (Also of note: a married couple just in the 35% bracket is actually paying 25% in taxes.) That's because the system is designed to be monotone: if your taxes jumped suddenly moving from one bracket to the next and your net income dropped, there'd be a big disincentive to crossing the tax bracket boundaries. As it stands, earning more always leads to more net income.

The Daily Dish: Tax Brackets 101, Ctd

[*] Those IRS tables calculate taxes on $34,500 at $4813, not $4806 - I assume the discrepancy is because the income is apparently rounded to the nearest $50 and the tax to the nearest dollar. Why use a smooth curve when you can use a step-function, eh IRS?

[**] As a gay man married in California but not in the eyes of the federal IRS, our taxes were a total nightmare this year. We had to do a "synthetic" joint return for federal taxes which was not submitted (but still had to pay to prepare) because the numbers were required for the joint state return. Again, I'm not a tax lawyer, but I sure needed one this year. I wish it were as easy as described above.

Japan hit by earthquake, tsunami

A truly horrifying event -- if you know of anyone in the affected areas, please give them your help and support. For those in the US, you can text REDCROSS to 90999 to donate $10, or contact your local Red Cross post. Google has also set up a Person Finder to help reunite missing persons with loved ones.

The New York Times has maps of the reach of the ensuing tsunami; I'm thankful the rest of the Pacific rim wasn't affected by the devastation it wrought on Japan. The Big Picture has a heart-wrenching series of photographs of the impact.

Our thoughts and hopes go out to everyone affected by this force of Nature.

The Egyptian Revolution, in tweets (updated)

(Updated Feb 18) Note: These data actually relate to a gathering of Twitter users in Iran, not Egypt. Apologies for the error, and thanks to the commenters for the correction.

Twitter played a significant role in the recent uprising in Egypt, with protesters communicating via tweets marked with the #25bahman hastag (February 14 in the arabic calendar) to plan and rally for the demonstration. Michael Bommarito downloaded all such tweets and plotted their frequency over time using R's ggplot2 library:

Not surprisingly, the activity peaked on February 14. The complete set of tweets would be an interesting focus for further analysis; they can be downloaded from Bommarito's post linked below.

Update: Drew Conway has used the data Bommarito provided to chart the top tweeters over time during the revolution.

Update Feb 18: As commenters have pointed out, the #25bahman hashtag

Michael Bommarito: Dataset: 5 Days of #25bahman

Airport security: science vs backlash

The United States has recently introduced millimeter wave and backscatter x-ray scanners to the security screening process in many airports, prompting a backlash in some quarters. Much of the opposition is centered around the invasion of privacy: the scanners generate an image of the traveller's naked body. There are also health concerns, at least for the backscatter x-ray variants of the scanners. I had initially dismissed these concerns as part of the usual overblown fears of anything related to radiation, but some scientists have recently raised concerns that while the radiation dose is low, unlike a medical x-ray or cosmic radiation, the entire dose is concentrated on the skin, intensifying the local dose by orders of magnitude. Personally, I agree with the that more study is warranted.

But are the scanners worth it, overall? Whether they are successful in deterring terrorists is one question that's been endlessly debated. But what about the effect of these security procedures on the economy? Nate Silver of the New York Times likens these security procedures as a "tax upon air travel":

Teleconferences are often a poor substitute for person-to-person interaction, and when people are reluctant to travel, some business deals don’t get done that otherwise would have. Recreational travelers, meanwhile, may skip out on vacations that otherwise would have brought them pleasure and stress-relief (while improving revenues for tourism-dependent economies). The tenuous profits of the airline industry are also affected, of course. Revenue losses from the new bag-checking procedures may have measured in the billions.

It certainly seems plausible that there is a quantifiable drain on the economy. But could we measure it? Nate suggests we could, via a statistical experiment.

If, for instance, Chicago O’Hare installs new machines and Chicago Midway does not, and passenger traffic at O’Hare drops by 9 percent while traffic at Midway holds steady, that would provide considerably more tangible evidence of how travelers are reacting to the new protocols than polls ever could.

It's a great idea. It's just a shame that political considerations would never allow it to be implemented.

New York Times: The Hidden Costs of Extra Airport Security