Revolutions

Five things Biologists should know about Statistics

In a thoughtful blog post, Bioinformatician Ewan Birney (Head of Nucleotide Data at the European Bioinformatics Institute) talks about the importance of Statistics to biologists:

Biology is really about stats. Indeed, the foundation of much of frequentist statistics - RA Fisher and colleagues - were totally motivated by biological problems.

He also cites the "Five statistical things I wished I had been taught 20 years ago". In order, they are:

1. Non-parametric statistics
2. R
3. The problem of multiple testing
4. The relationship between P-value, effect size, and sample size, and
5. Linear models and PCA

Read Ewan's full post at the link below for his reasons why every biologist should learn about these five statistical things.

Bioinformatician at Large: Five statistical things I wished I had been taught 20 years ago (via @paulblaser)

 

R in the Bioinformatics Knowledgeblog

The Knowledge Blog progect is a new, light-weight way of publishing scientific, academic and technical knowledge on the web, across several scientific disciplines. One such discipline is bioinformatics, and the Bioinformatics Knowledgeblog contains useful scientific reference material for bioinformatics, including several resources for R users. There you'll find an R Guide for Complete Beginners, a tutorial on analyzing microarray data with Bioconductor, a guide to making volcano plots for microarray data, and much more. Check out all the content at the link below.

The BioInformatics Knowledgeblog: About (via Simon Cockell)

Norman Nie to speak on Big Data at EnZee Universe tomorrow

The Revolution Analytics team is at EnZee Universe in Boston this week (Revolution is a sponsor of the Netezza user conference), where we're demonstrating the integration between Revolution R and the IBM Netezza data warehouse appliance. Revolution's CEO Norman Nie will be part of the panel discussion tomorrow, "Exploiting Value from Ever-Growing Data Assets":

Tackling the Big Data Challenge with R
Enzee Universe Conference 2011, 2:15 PM - 3:05 PM, June 21, The Westin Boston Waterfront

Big data is presenting enterprises with a great challenge and a tremendous opportunity. Organizations today are generating exponentially more data than ever before, and the need to store, process and analyze data has emerged as a critical function for making business decisions and providing greater insights into organizational operations and trends.

Norman Nie, CEO of Revolution Analytics, will join a panel at the Enzee Universe conference to discuss best practices for gaining value from large data sets with the R statistical language. R is the most powerful statistical computing language on the planet and has emerged as a tool of choice for data scientists analyzing massive data sets.

Interested individuals can register for the Enzee Universe conference here.

For more information about the conference, follow the link below.

IBM Netezza: EnZee Universe 2011 User Conference

 

Because it's Friday: Keep Calm, and Carry on Charting

While I was out on vacation last week, my itinerary called for a one-day stopover in London. Somehow managed to miss most of the tourist attractions while I was actually living in the UK, so now I try and remedy that on visits. This time, we stopped by the Churchill War Rooms, the underground bunker used by the Prime Minister, his cabinet and military chiefs to direct the defense of the UK during World War II. 

It's a facinating museum and well worth a visit for anyone with an interest in history, but as a data geek the Map Room in particular caught my eye. There you can see how an army of secretaries kept meticulous data on the impact of Hitler's attacks on London:

And then visualized the data for the war planners with time series, histograms, bar charts, and many other hand-crafted analyses:

The chart below, "German U-boat situation as assessed at 16th July 1945" was especially interesting. The whole wall was an analysis of the extent of the remaining Axis war vessels (perhaps based on intelligence like this):

Finally, it was interesting to see that BI Dashboards aren't as recent of an invention as we may think (here, showing up-to-date troop strengths in the Pacific theatre):

It's hard to believe that these analyses were created in the most difficult conditions, with intermittent power and bombs dropping all around, with narrow lines of communications ... and of course without any of the technological tools we take for granted today. No computers, no printers, no R ... not even calculators! And it's nice to know that Statistics and Data Visualization -- what today we'd call Data Science -- had a strategic part in the war effort. Keep Calm, and Carry on Charting!

 Photos my me (David M Smith). Click to enlarge, and feel free to reuse under the Creative Commons Attribution-ShareAlike 3.0 Unported License.

Big-Data PCA: 50 years of stock data

In this post, Revolution engineer Sherry LaMonica shows us how to use the RevoScaleR big-data package in Revolution R Enterprise to do principal components analysis on 50 years of stock market data -- ed.

Principal components analysis, or PCA, seeks to find a set of orthogonal axes such that the first axis, or first principal component, accounts for as much variability as possible and subsequent axes are chosen to maximize variance while maintaining orthogonality with previous axes. Principal components are typically computed either by a singular value decomposition of the data matrix or an eigenvalue decomposition of a covariance or correlation matrix; the latter permits us to use the RevoScaleR function rxCovCor with the standard R function princomp.

Stock market data for open, high, low, close, and adjusted close from 1962 to 2010 is available from InfoChimps. As you might expect, these data are highly correlated, and principal components analysis can be used for data reduction. We read the original data (a set of 26 comma-separated text files, where each file is represented by a letter in the alphabet) into an .xdf file, NYSE_daily_prices.xdf:

nyseDataDir <- "C:/Users/Sherry/Downloads/NYSE"
dataSourceName <- file.path(nyseDataDir, "NYSE_daily_prices")
dataFileName <- "NYSE_daily_prices.xdf"
append <- "none"
for (i in LETTERS)
{
       importFile <- paste(dataSourceName, "_", i, ".csv", sep="")
       rxTextToXdf(importFile, dataFileName, stringsAsFactors=TRUE,
                append=append)
       append <- "rows"
}

The full data set includes 9.2 million observations of daily open-high-low-close data for some 2800 stocks:

> rxGetInfoXdf(dataFileName)
File name: NYSE_daily_prices.xdf
Number of observations: 9211031
Number of variables: 9
Number of blocks: 34 

We will use the rxCor function to calculate the Pearson's correlation matrix for the variable specified, and pass this to the princomp function:

stockCor <- rxCor(~ stock_price_open + stock_price_high +
stock_price_low + stock_price_close +
stock_price_adj_close, data="NYSE_daily_prices.xdf")
stockPca <- princomp(covmat=stockCor)
summary(stockPca)
loadings(stockPca)
plot(stockPca) 

This yields the following output:

> summary(stockPca)
Importance of components:
Comp.1 Comp.2 Comp.3 Comp.4
Standard deviation 2.0756631 0.8063270 0.197632281 0.0454173922
Proportion of Variance 0.8616755 0.1300327 0.007811704 0.0004125479
Cumulative Proportion 0.8616755 0.9917081 0.999519853 0.9999324005
Comp.5
Standard deviation 1.838470e-02
Proportion of Variance 6.759946e-05
Cumulative Proportion 1.000000e+00
 
> loadings(stockPca)
Loadings:
Comp.1 Comp.2 Comp.3 Comp.4 Comp.5
stock_price_open -0.470 -0.166 0.867
stock_price_high -0.477 -0.151 -0.276 0.410 -0.711
stock_price_low -0.477 -0.153 -0.282 0.417 0.704
stock_price_close -0.477 -0.149 -0.305 -0.811
stock_price_adj_close -0.309 0.951
Comp.1 Comp.2 Comp.3 Comp.4 Comp.5
SS loadings 1.0 1.0 1.0 1.0 1.0
Proportion Var 0.2 0.2 0.2 0.2 0.2
Cumulative Var 0.2 0.4 0.6 0.8 1.0

The default plot method for objects of class princomp is a screeplot, which is a barplot of the variances of the principal components. We can obtain the plot as usual by calling plot with our principal components object:

> plot(stockPca)

Between them, the first two principal components explain 99% of the variance; we can therefore replace the five original variables by these two principal components with no appreciable loss of information.

Where Ichiro Hits

Google research scientist Peter Hauck used Weka and k-means cluster analysis to describe where Mariners right-fielder Ichiro favours hitting the baseball. He then used R to visualize the 6 clusters the k-means analysis identified:

I sometimes find K-means clusting tough to explain as a statistical technique, but this makes for a great example: if you're a fielder facing Ichiro, it might be a good idea to keep an eye on those six spots when he hits. See the full ananlysis on the Infochimps blog at the link below.

Infochimps blog: Clustering Baseball Data with Weka

5000 R questions on stackoverflow.com

The R tag on stackoverflow.com hit a milestone yesterday: 5000 questions about the R language. (The 5000th question was about the fortunes package, incidentally -- thanks to Andrie de Vries for pointing this out on Twitter.) Stackoverflow.com continues to be an excellent resource for R users, and the number of questions (and answers!) continues to grow: 5020 at this writing, and with 1200 users actively following the tag. For comparison, Matlab has about 4400 questions; SAS has 327, and SPSS has 52.

There are also 735 questions tagged with r on the "stats" stackexchange, for general questions about statistics.

The Big Analytics Revolution starts with R

Thanks to everyone who attended our webinar The 'Big Analytics' Revolution Starts with R yesterday. If you missed the live session, you can download the presentation slides (PDF) and the 30-minute replay video (WMV) from the Revolution Analytics website.

The presentation focuses on the isse of Big Data, and how businesses can use advanced analytics methods implemented in the R language to get economic value from the data they've collected.  David Champagne (our CTO) also provides some best practices for setting up an "Analytics Stack" of interoperable techologies for data storage, ETL, analytics, and presentation, along with some specific examples of the Revolution R integrations with Hadoop and Netezza data wareshouses. Slide 25 has some links to useful resources on the topic as well.

Revolution Analytics Webinars: The 'Big Analytics' Revolution Starts with R

Hot Job in IT: Data Science

CIO Magazine today has an article on the "6 Hottest New Jobs in IT" in which features Data Science and R at #2: 

"There's now an intellectual consensus in business that the only way to run an enterprise is to use analytics with data scientists to find opportunities," says Norman Nie, CEO of Revolution Analytics, which produces the first commercial application to bring the R data analysis programming language into the business world. Because of the immense opportunity for strategic insight buried in all that data, says Nie, "corporations now have an unlimited demand for people with background in quantitative analysis."

In addition to Data Science, the other five hot jobs are: Business architect, Social media architect, Mobile technology expert, Enterprise mobile developer, and Cloud architect. Follow the link below for the complete story.

CIO Magazine: The 6 Hottest New Jobs in IT

New McKinsey Report on Big Data

Research firm McKinsey has a new, free report on the value of Big Data to business. In "Big data: The next frontier for innovation, competition and productivity", McKinsey defines Big Data as "datasets whose size is beyond the ability of typical database software tools to capture, store, manage and analyze". The report then looks at the economic value of such data: not just value already obtained by businesses, but also the value yet to be realized by sectors of industry. Some interesting facts from the report:

• Data is being collected at a rapidly increasing rate because data storage costs are dropping dramatically. Storage sufficient to hold all the world's music can now be purchased for just US$600.
• There is more than $300B potential annual value to US health care, by implemeting analysis of Big Data stored by the healthcare industry.
• Retailers could increase operating margins by 60% from analysis of their Big Data.

I'll be drawing on some of the report's conclusions in our webinar today, The Big Analytics Revolution Starts with R. You can find more facts and analysis in the 143-page McKinsey report, available at the link below.

McKinsey Global Institute: Big data: The next frontier for innovation, competition and productivity