Revolutions

The difference between Statistics and Machine Learning

I get my daily R fortune by following Rfortunes on Twitter. This one came up the other day:

To paraphrase provocatively, 'machine learning is statistics minus any checking of models and assumptions'. Brian D. Ripley.

In a similar vein, back in December Brendan O'Connor remarked upon Rob Tibshirani's comparison of machine learning and statistics, reproduced here:

Glossary
Machine learning	Statistics
network, graphs	model
weights	parameters
learning	fitting
generalization	test set performance
supervised learning	regression/classification
unsupervised learning	density estimation, clustering
large grant = $1,000,000	large grant = $50,000
nice place to have a meeting: Snowbird, Utah, French Alps	nice place to have a meeting: Las Vegas in August

It's certainly a pithy comparison. Brendan O'Connor concurs that the differences between the two are more superficial than substantive, and his thoughts on the cultural differences between the two disciplines are very interesting. Amongst other things, his comparison of two similar courses in Stanford (one from the Computer Science department, one from Statistics) leads him to conclude:

ML sounds like it’s young, vibrant, interesting to learn, and growing; Stats does not.

So, do statisticians "merely" have an image problem in this field, or is there something more substantive at play? Perhaps protests like this are in our future...

CMU machine learning students "protest" at the G20 summit in Pittsburg, September 25 2009. Photo by Arthur Gretton on Flickr.

AI and Social Science: Statistics vs. Machine Learning, fight! (via @Cmastication)

Analysis of airline performance

At the data expo held at this year's ASA meeting, the competition was to present as a poster an graphical analysis of 120 million airline flight records: all arrivals and departures of all but the smallest airlines in the USA from 1987 to 2008.

The winning poster from a team at the SAS Institute used SAS to produce a nice heat-meap of delays on a daily basis for the entire 22-year period. It reveals that winter is a bad time to fly (when the most cancellations occur) and that 9/11 changed everything (where a clear discontinuity appears as fewer flights led to fewer cancellations):

The runner-up entry from a team at Iowa State using R introduced other information into the analysis, including weather, crosswinds at landing, and fuel efficiency. They also managed to tease some fascinating facets out of the data, for example that three of the flights in the data set are actually registered as hot-air balloons, and (by tracking aircraft registrations and matching landings to subsequent takeoffs in a different city) that there have been more than 1 million "ghost flights" of empty planes since 1995. I thought this chart detailing the trends (and exceptions) in flights volumes at four major airports was well done:

An interesting third entry from Yale University's Michael Kane and Jay Emerson focused more on data-processing than graphical analysis. They used the bigmemory package to attach the entire data-set as a file-backed object, which can then be manipulated as an ordinary matrix in R. (Jay Emerson is a frequent REvolution collaborator, and the author of the bigmemory package with Michael Kane.) They then used the foreach package to process the data in parallel to investigate flight delays (flights leaving around 8PM have the greatest chance of a significant delay) and the effect of aircraft age on delays (a small but statistically significant effect, based on an analysis of 84 million records). 

Update 12:16 - corrected authorship of bigmemory package (thanks, DE)

ASA Data Expo 2009: Airline on-time performance (with thanks to reader ER for the suggestion)

How to use a Google Spreadsheet as data in R

One of the great strengths of R is that it promotes reproducible research: as an open-source system, you can easily send a script file to a colleague with the confidence that they'll be able to get the same results using R on their own system. Provided they have the same data, that is. If that's your goal, you can always send along a data file, but that can add some complications. You have to take care with the filenames in your script, and dealing with data that changes regularly can be annoying.

Google Docs offers a solution. As long as you have a Google account, you can store your data as a Google Spreadsheet, and then create a special URL for that spreadsheet that can be used as a CSV file source by R. The process is a bit complex, but it only needs to be done once, and then your data are freely available to anyone who wants to access it via R. 

First, change the permissions for your spreadsheet on the main Google Docs page (the one that lists all of your Google Documents: spreadsheets, documents, presentations, etc) and configure your spreadsheet so that it can be viewed by everyone. (I'm not sure if this is strictly necessary for the export, but it does give your collaborators the ability to view the data directly as a spreadsheet.)

• Open your spreadsheet in Google Docs.
• Click the "Share" botton in the toolbar, and choose "Anyone with the link" for Visibility options

Now anyone can view your spreadsheet in Google Docs using the link given in the "Share" dialog. But we want a link for the CSV export version of the spreadsheet, not the spreadsheet itself. Here's how to get that.

• Open your spreadsheet in Google Docs.
• Click the drop-down menu (down arrow) nexy to the Share button in the upper-right corner of the spreadsheet) and choose "Publish as a Web Page".
• For "Sheets to Publish" choose "All Sheets", and check the box "Automatically republish when changes are made" if you want to dynamically update the data for R when you edit it.
• Click "Start Publishing". This will activate the options in the box "Get a link to the published data", below.
• Change the export type from "Web Page" to "CSV (comma-separated values)".
• Change "All sheets" to "Sheet1" (or select the sheet you want to export)
• Change "All Cells" to the specific range you want to export, beginning with the header row. Use Excel-style notation, like "A1:C6" for the first 3 columns and the first 6 rows.
• Click "Republish now".

The Publish box should look something like this.

You can now use the URL in the bottommost box with the getURL function (you'll need to install the RCurl package first -- required because of the HTTPS connection), and then read the data with read.csv in R:

> require(RCurl)
> myCsv <- getURL("https://docs.google.com/spreadsheet/pub?hl=en_US&hl=en_US&key=0AkuuKBh0jM2TdGppUFFxcEdoUklCQlJhM2kweGpoUUE&single=true&gid=0&output=csv")
> read.csv(textConnection(myCsv))
  x   y  z
1 1 0.3 10
2 2 0.5 14
3 3 1.1 12
4 4 0.1  1
5 5 1.9  0

Better yet, if I ever change the data in the Google spreadsheet, the command above will always retrieve the updated data (provided I chose the "Automatically republish" option, above).

I believe you can do something similar with the RGoogleDocs package (currently in beta and not yet on CRAN), but this process requires no additional packages for you or the recipient of your data. 

Update Sep 15 2011: Google spreadsheets now publish using secure https:// URLs, which read.csv doesn't support directly. Updated the code above to use getURL from the RCurl package, which fixes the problem. Also updated the sharing details, which have changed in Google Docs since this article was first published.

Is the express line really faster?

You're at the supermarket. Which line should you choose for fastest service?

(The numbers are the number of items in each cart.) Dan Meyer wondered about this, and rather than merely speculating or diving into queue theory, went out and collected data. (A very Mythbusters attitude, of which I approve.) He spent ninety minutes watching the checkout lines at his local supermarket, counting the number of items in each shopper's cart and the amount of time it took them to be checked out (from loading up their first item to completing the financial transaction), and the method of payment.

The conclusion? In the example above, you're likely better off in the shorter line with one loaded cart, rather than the "express" lane with several carts with a few items. The reason is that it takes about 3 seconds to scan each item, but on average about 35 seconds to process each shopper. In the example above we have 11 items and 4 shoppers in the express line for a wait time of 176 seconds, versus 1 shopper and 19 items for a wait time of 96 seconds. Here's the calculation, as done in R:

> shopping <- read.csv(

+ "http://spreadsheets.google.com/pub?key=tE9pXlYLwTAeiDWxL8h_viA&single=true&gid=0&range=A1%3AE37&output=csv",

+ as.is=TRUE)

> shopping$seconds <- as.numeric(as.difftime(shopping$Total.Time))

> p1 <- coef(lm(seconds ~ Number.of.Items, shopping,subset=-8))

> p1

    (Intercept) Number.of.Items 

      35.309942        3.191313 

> p1 %*% c(4,11) # express lane

         [,1]

[1,] 176.3442

> p1 %*% c(1,19) # short lane

        [,1]

[1,] 95.9449

Note that there's one outlier (row 8) in the data file which I've deleted (and Dan apparently did so, too). The "Intercept" in our regression is the average processing time, and "Number.of.items" is the time to process each item. I used matrix math to calculate the wait times in our example, because I'm that awesome (or lazy, depending on your perspective).

The situation is complicated a bit by the method of payment: cash is by far the fastest (about 18 seconds) compared to credit card or check payments (41 and 54 seconds respectively). If you're curious, here's how I got those numbers in R:

> fit <- lm(seconds ~ Number.of.Items + Payment - 1, shopping,subset=-8)

> coef(summary(fit))

                   Estimate Std. Error  t value     Pr(>|t|)

Number.of.Items    2.955684  0.2997510 9.860464 6.340141e-11

Paymentcard       41.198885  6.7016800 6.147546 9.237407e-07

Paymentcard/cash 128.310215 22.1575721 5.790807 2.505366e-06

Paymentcash       17.974740  7.4723845 2.405489 2.252260e-02

Paymentcheck      53.997121 16.9930735 3.177596 3.430851e-03

Even if all the express lane patrons were paying with cash, the shorter line is still better even when the single patron pays with a credit card (104 seconds versus 97 seconds waiting).

dy/dan: What I Would Do With This: Groceries

CRAN R 2.10.0 scheduled for October 26

The next update to the R project, CRAN R 2.10.0, was announced today by core team member Uwe Ligges Peter Dalgaard. It will be released in source form on October 26 (following a beta test period), with binaries to follow soon thereafter.

Significant planned changes visible to users include:

• Improvements to the web-based help system and the underlying processes for generating help
• More mathematical operations on time-based quantities
• A continuity correction may now be applied to the Kendall and Spearman methods of cor.test
• User-contributed methods are now supported by glm
• Support for sparse matrices in statistical and tabulation functions
• Improved accuracy for some complex arithmetic
• Improved performance when using regular expressions

The complete list of changes (again, subject to change) can be found in the NEWS file.

(Updated 12:27 to fix attribution of announcement. Oops! Thanks DE.)

r-announce mailing list: R 2.10.0 is scheduled for October 26

A workflow transformed with R

Twitter only allows messages of 140 characters, but sometimes a picture (attached to a tweet) is worth a thousand words. Jose Silva tweets: "A big change in my workflow - gone R native. Cheaper too." This diagram (created by Jose) was attached:

Scraping NYT articles with R

Unstructured data (free-text documents, reports, articles and such) are a rich source of information, but for analysis in R the first step is often to convert the unstructured data into a structured, tabular format that lends itself to traditional statistical analysis and visualization techniques. We saw the other day how you can use Google Insights to convert volumes of search terms into a CSV of scaled counts. Now, you can accomplish a similar task by counting keywords in articles published in the New York Times.

Stanford professor Claudia Engel shows us how. It's a bit trickier than using Google Insights, because you need to first get a key for the NYT Developer API (it's free). With that key, you can adapt the code provided on the page linked below to count the number of articles matching a selected keyword (or words). The data is then available for analysis in R, where you can create charts like this one:

Claudia Engel: Scraping New York Times Articles with R

Bulgarian Lottery Coincidence, revisited

The Wall Street Journal's Numbers Guy has taken an in-depth look (in both his blog and print column) at the Bulgarian Lottery coincidence we looked at last week. I was interviewed by the author Carl Bialik yesterday, and we talked lotteries, probabilities, coincidences and R. Sadly, no mention of R made it into the final article. It's still a good read, though.

The Numbers Guy: Lottery Math 101

Bellkor wins Netflix Prize

It's official! The team Bellkor Pragmatic Chaos is the winner of the $1million Netlix prize. The competition to create the best predictive model for Netflix movie ratings ended in a nailbiting finish, and for a while there it seemed Bellkor had been pipped at the post by rival team The Ensemble, who submitted an apparently superior model just 10 minutes after theirs (and just before the close of the competition). As it turns out though, when evaluated on the unseen validation data both models had the exact same performance, making Bellkor the winner by dint of the earlier submission. Full details at Wired in the link below.

The competition was such a success that Neflix is running a sequel next year. Should be a good competition!

Wired: BellKor’s Pragmatic Chaos Wins $1 Million Netflix Prize by Mere Minutes

ggplot2 and the grammar of graphics

I really enjoyed Hadley Wickham's talk for the Bay Area UseR group last week. I've really been getting into ggplot2 lately, but it was Hadley's example of plotting housing sales data for the San Francisco area really made it click for me. When you first start using ggplot2, the syntax can seem a little, well, arcane: rather than using separate commands to build up your plot as with traditional R graphics, you string elements together with the '+' operator. The example Hadley showed, plotting time-series data, laying on a line, and then panelling the data by city, really made sense of the "grammar of graphics" concept for me.  

It looks like the folks at Biogeeks have had a similar epiphany recently:

Say you want to make 12 dose-response plots of various compounds tested in various cell lines. With basic R this would require writing a for-loop and fidling around a lot with axis and plot labelling and the par()-function to make them fit on one page. With basic R you would have be extremely careful to make the code general and reusable for next time when you have different compounds and different cell lines. Enter ggplot2 and the grammar of graphics. ggplot2 is a package for implementing the grammar of graphics, which allows you to write extremely succinct and natural languages like code that produces stunning visualizations.

Here's 6 lines of code in ggplot2, and the graph it creates:

p = qplot(Concentration, Percent.of.control, 
    data=screening_data, 
    geom=c("point", "smooth"), colour=Response.type) +
  scale_x_log10() +
  facet_grid(Compound ~ Cell.line) +
  coord_cartesian(ylim=c(-10, 110))
print(p)

So how does the grammar of graphics help, here? I liked the way the Biogeeks summed it up: "If you compare the code and the plot you will realize that the code contains about the words that you would use if you were told to briefly describe the plot using English." Indeed!

Biogeeks: Power plotting with ggplot2