Revolutions

Because it's Friday: The chances of finding Mr/Mrs Right

Speaking of family issues, this video by Emanuele Colombo is adorable:

This being a statistics blog, I have to take a small issue with the calculation, though. Let's look at the math:

7,000,000,000 people
× 0.04 (proportion one person might meet) 
× 0.5 (proportion sexually compatible -- hey, count yourself lucky!)
× 0.08 (proportion age compatible)
× 0.12 (proportion language compatible)
= 1,340,000 potential partners.

As any statistics student will tell you, you can't just multiply all those factors together willy-nilly. For example, the chance of meeting any given person is not independent from the chance they'll be language-compatible — most people you meet will be in your own city/country, of whom more than 12% are likely to speak the same language.

But it's still adorable. (And I'm glad I too have a soulmate who can put up with these geeky obsessions. Thanks, honey!)

Have a great weekend, all. 

Trying for a baby? Here's how long it might take.

Wanting to start a family the natural way? For a healthy 45-year-old woman, you may be in for a five-year wait.

That's the conclusion of Richie Cotton, a UK-based data scientist, who discovered when he and his girlfriend wanted to start a family that statistics on how long it takes to get pregnant are hard to come by. The National Health Service in the UK does provide statistics on the "monthly fecundity rate" (MFR) — the chance of becoming pregnant in a month. This varies by age: for a 25-year-old woman it's about 25%, dropping to 10% at age 35. But how can you translate the MFR to how long it will wait to conceive (given regular unprotected sex, of course)?

To answer the question, Richie used the R language and the negative binomial distribution to calculate the probability of conceiving after 1 month, 2 months, 3 months etc. for women of various ages. (You can see the R code in Richie's blog post.) The results are summarized in the chart below:

So, if you're a 25-year old woman (see the red line above), it's more likely than not you'll have conceived by the third month of trying (about a 57% chance, in fact), and there's about a 95% chance of conceiving in the first year. At age 40 (blue), the chance of conception in the first year drops to around 50%. It's a much more challenging scenario for older women: at age 45, even after five years of trying it's still less than even odds of conception. This is what makes alternative means such as IVF a compelling option compared to the natural process.

For the complete details of the analysis, follow the link to Richie Cotton's blog below.

4D Pie Charts: How long does it take to get pregnant?

Big Data Generalized Linear Models with Revolution R Enterprise

R''s glm function for generalized linear modeling is very powerful and flexible: it supports all of the standard model types (binomial/logistic, Gamma, Poisson, etc.) and in fact you can fit any distribution in the exponential family (with the family argument). But if you want to use it on a data set with millions or rows, and especially with more than a couple of dozen variables (or even just a few categorical variables with many levels), this is a big computational task that quickly grows in time as the data gets larger, or even exhaust the available memory.

The rxGlm function included in the RevoScaleR package in Revolution R Enterprise 6 has the same capabilities as R's glm, but is designed to work with big data, and to speed up the computation using the power of multiple processors and nodes in a distributed grid. In the analysis of census data in the video below, fitting a Tweedie model on 5M observations and 265 variables takes around 25 seconds on a laptop. A similar analysis, using 14 million observations on a 5-node Windows HPC Server cluster takes just 20 seconds.

 

This demonstration was part of last week's webinar on Revolution R Enterprise 6. If you're not familiar with Revolution R Enterprise, the first 10 minutes is an overview of the differences from open-source R, and the remaining 20 minutes describes the new features in version 6. Follow the link below to check out the replay.

Revolution Analytics webinars: 100% R and More: Plus What's New in Revolution R Enterprise 6.0

Orbitz and the Macs: Signals, not segmentation

By now you've probably heard about the fact that Orbitz users accessing the site via Macs are seeing more expensive hotel options when they search. But it seems worth clearing up a couple of fallacies.

First, it's not as if the same hotel room is being offered at a higher prices to Mac users. (So no, using Windows to access Orbitz won't get you a better deal.) Orbitz offers the same search results to Windows and Mac users, but Mac users may see more expensive hotels higher earlier in the search results.

And the reason why this is happening isn't segmentation, as some have claimed.  From what I remember of the Orbitz presentation at the R user conference a couple of years ago, all sorts of signals go into the R model to try and predict which hotels a given Orbitz user might be interested in: WebTrends data, search history, purchase history, planned travel dates ... and the user's browser and operating system. It would seem that the browser (perhaps Safari / iOS) and/or the operating system is a strong signal that the user is likely to purchase a higher-priced hotel room. The Orbitz system (as I understand it) is designed to rank search results showing the results the specific user is most likely to be interested in at the top of the list. If Mac users, on average, spend more on hotel rooms than other Orbitz visitors, then it's not surprising that more expensive results would appear at the top of the list. 

But it's not as if Mac users being singled out for special treatment. Segmentation is a marketing operation (and a conscious decision), whereas the use of signals like these is part of a predictive analytics operation (driven by data, not people). It's important to understand the distinction.

You can read more about how Orbitz uses R and Hadoop to drive its search at the link below.

Revolutions: How Orbitz uses Hadoop and R to optimize hotel search

Figuring an exchange rate for sports scores

While the US's Major League Soccer is using advanced analytics to analyze ball movement and improve team composition, they might want to think about a smaller, but possibly more impactful, goal for analytics. Like, how to explain to an American audience what a 1-2 game means to a basketball or baseball fan not familiar with scoring in the beautiful game.

An R user has tackled this problem for the game of rugby. By analyzing the distributions of scores of typical rugby and soccer games, he was able to determine that a 0-60 loss in Rugby (as befell the Irish team playing the All Blacks last weekend) is equivalent to a 0-7 loss in soccer. Now all someone needs to do is to create an exchange rate table for all common sports, so that the next MLS sports report makes more sense to more Americans. Sportscasters could say something like, "And today, the Seattle Sounders lost to Montreal 1 goal to 4, which feels like a 40-110 drubbing in the NBA." Or whatever the actual exchange rate is, I haven't actually done the analysis. But someone should! Just an idea.

DiffusePrior: How to Convert Rugby into Football/Soccer Scores

 

New R User Groups in Ankara, Toronto

Two new local R user groups to report this week.

In Turkey, the Ankara R Users Group has just started up. No meetings are scheduled yet, so be sure to suggest a meeting time/location when you sign up.

The Toronto-based R Matlab Users group focuses on financial services applications. Created by Bryan Downing (who also produces the QuantLabs blog), the group will hold virtual meetings for R and Matlab users in the financial services industry worldwide. There will also be social meetups in the Toronto financial district, the first of which will be held July 9.

Both groups are now listed in the Local R User Group directory, which is always available from the link in the right column of the Revolutions blog.

 

Because it's Friday: Tool does Fibonacci

It's not clear whether it's deliberate or just one of those coincidences inherent in Numerology, but the Tool song Lateralus appears to be based on the Fibonacci sequence:

 

Deliberate or not, the Hubble shots are pretty. It's not a bad song either, if you're in an early-millennium kind of mood or just like music with an unusual time signature. Either way, have a great weekend!

R 2.15.1 includes performance improvements inspired by dataframe package

The latest update to open-source R, R 2.15.1, was released this morning. (You can grab sources now, and binary versions will hit the CRAN mirrors over the next couple of days.) In addition to several new features and bug fixes (including the new globalVariables function, which will be a boon to package developers), this update also includes some significant performance improvements inspired by the dataframe package.

At the useR! 2012 conference last week, Google's Tim Hesterberg introduced the dataframe package (available now on CRAN), which has been in use for the last three years amongst Google's 500+ R users. (You can download Tim's PDF slides here.) The package makes no functional changes to R; instead, it improves the implementation of data frames to reduce the number of temporary copies made of data. Tim reported that using the dataframe package with R 2.15.0 improved performance by 21% for creation and column subscripting, and by 14% for row subscripting.

Tim mentioned during the talk that r-core member Luke Tierney was in the process of incorporating performance improvements from the dataframe package into base R, and indeed several such improvements are noted in the NEWS file. All the improvements are devoted to reducing the number of times R makes temporary internal copies of data, which improves both speed and memory usage of R. And because these are low-level improvements at R's core, these improvements will affect many R functions, not just those related to data frames.

If you've built R 2.15.1 already, have you noticed performance improvements? Let us know in the comments.

R-announce mailing list: R 2.15.1 is released

'Moneyball' for government policy

The UK government proposes to use data and evidence as the basis for deciding which policy initiatives to implement, in this Cabinet Office paper. The paper, co-authored by Ben Goldacre (who we've mentioned here before), proposes using randomized trials to evaluate the efficacy of government programs. It's like applying the 'Moneyball' principles to government policy: rather than using anecdotes and ideology to decide how to spend government money, why not try policies out on a small scale first and see what works? 

The US government has recently begun applying such trials to evaluate programs at the federal and state level, as well.

 

 

 

 

FDA: R OK for drug trials

In a poster (PDF) presented at the UseR 2012 conference, FDA biostatistician Jae Brodsky reiterated the FDA policy regarding software used to prepare submissions for drug approvals with clinical trials: Sponsors may use R in their submissions.

The FDA does not endorse or require any particular software to be used for clinical trial submissions, and there are no regulations that restrict the use of open source software (including R) at the FDA. Nonetheless, any software (R included) used to prepare data analysis from clinical trials must comply with the various FDA regulations and guidances. The R Foundation helpfully provides a guidance document for the use of R in regulated clinical trial environments, which provides details of the specific FDA regulations and how R complies with them.

Despite some mistaken conceptions in the pharmaceutical industry, SAS is not required to be used for clinical trials. This origin of this fallacy is probably related to the fact that data must be submitted in the XPT "transport format" (which was originally created by SAS). This data format is now an open standard: XPT files can be read into R with the standard read.xport function, and exported from R with the write.xport function in the SASxport package. (And if you have legacy data in other SAS formats, there's a handy SAS macro to export XPT files.)

R is widely used at the FDA. (Full disclosure: the FDA is a Revolution Analytics customer.) The poster describes how FDA offices "use R on a daily basis", and that FDA scientists have written R packages for use by other scientists (within and outside the FDA) involved in the submissions process. FDA-contributed packages include popPK, MAVTgsa and QT, which was written specifically for the FDA's QT Interdisciplinary Review Team.

Jae Brodsky also provides some personal views on the use of R at the FDA. She notes:

• All results from data should be reproducible independent of software used.
• In submissions, FDA reviewers would like to know: which R functions were accessed; how they were accessed; and that the results are accurate, reliable and consistent.
• The FDA has used R for its own reports. (Interesting side-note: as a government agency, its reports must be Section 508 compliant, one part of which demands that reports be accessible to readers with limited color vision. Jae mentioned to me that she uses R's Rcolorbrewer package to generate 508-compliant pallettes for charts in FDA reports, as the poster excerpt below demonstrates.)

In Jae's experience, "R use at the FDA is completely acceptable and has not caused any problems." So as more and more companies adopt R for statistical analysis, we can expect to see more FDA clinical trial submissions using R for statistical analysis, reporting, and data visualization.