Revolutions

Because it's Friday: Gardening with Gamma Rays

Did you know that the mint flavour in your chewing gum, and the ruby-red grapefruit on your breakfast plate, both come from cultivars created through a deliberate irradiation? I'd never heard of this before, but apparently in the 50's and 60's many countries set up "gamma gardens" to experiment with exposing food plants to powerful gamma radiation, with the hope of spontaneously creating new cultivars with beneficial mutations. The disease resistant properties of commercially-grown mint, and the colour of the "Rio Star" grapefruit grown widely in Texas, were "created" using this method. The Institute of Radiation Breeding in Japan conntinues such research today; here's an aerial view of one of their "gamma gardens":

 
Check out the article "Atomic Gardens" from the blog Pruned for a fascinating look into the history of gamma gardens, including this explanation of the design of the gardens themselves:

 

The circular spatial form of the gamma gardens, which in aerial view uncannily resembles the radiation danger symbol, was simply based upon the need to arrange the plants in concentric circles around the radiation source which stood like a totem in the center of the field. It was basically a slug of radioactive material within a pole; when workers needed to enter the field it was lowered below ground into a lead lined chamber. There were a series of fences and alarms to keep people from entering the field when the source was above ground. 

The amount of radiation received by the plants naturally varied according to how close they were to the pole. So usually a single variety would be arranged as a 'wedge' leading away from the pole, so that the effects of a range of radiation levels could be evaluated. Most of the plants close to the pole simply died. A little further away, they would be so genetically altered that they were riddled with tumors and other growth abnormalities. It was generally the rows where the plants 'looked' normal, but still had genetic alterations, that were of the most interest, that were 'just right' as far as mutation breeding was concerned.

Pruned: Atomic Gardens

Gartner: Revolution Analytics a "Cool Vendor" for BI

Leading analyst firm Gartner has just published its "Cool Vendors in Analytics and Business Intelligence" report for 2011 (download it here if you have a Gartner subscription). In the report, Revolution Analytics is named a Gartner Cool Vendor, and recognizes the company as "innovative, impactful and intriguing":

Driven in part by the rise of big data, business intelligence (BI) is a rapidly growing market that has seen increasingly strong enterprise adoption rates. The concurrent to the growth of BI has been increased investment in predictive analytics; R is not only the tool of choice but the ideal environment for advanced analysis. R is designed to be extensible and integrate within BI suites to incorporate advanced analytics into reports.

Revolution's CEO Norman Nie responds in this press release:

“It’s an honor to be recognized as a ‘Cool Vendor’ by Gartner, and we believe this to be indicative of the ever-increasing interest in R, particularly among business users,” said Norman Nie, Revolution Analytics CEO. ”At Revolution Analytics, we have accelerated enterprise adoption of R by adding key features like big data analysis, integration and user experience.”

Revolution Analytics News: Leading Industry Analyst Firm Names Revolution Analytics a “Cool Vendor” for Business Intelligence

Can you do better than cap-weighted equity benchmarks?

We're on our way to Chicago for the annual conference for R users in Finance, R/Finance 2011. Revolution Analytics is proud to once again sponsor this conference, and during the sponsor lunch session at noon on Saturday, we're honoured to have Guy Yollin show how the big-data capabilities of Revolution R Enterprise can be used for quantitative finance. Here is the abstract for the talk:

Can you do better than cap-weighted equity benchmarks?

Guy Yollin, Visiting Lecturer, University of Washington
Krishna Kumar, Financial Consultant
Scott Payseur, PHD, Quantitative Analyst, UBS Global Asset Management

Indexing and passive benchmarking has been sold by pension consultants as an efficient allocation to endowments and pension plans. Index tracking and ETFs around them have had spectacular growth in AUM. However, cap-weighted indices are unintuitive and inefficient from a CAPM.

In this presentation we review several large universe equity indices (Russell 1000, S&P 500 etc). We propose an alternative efficient benchmark to the underlying indices. After constructing an intuitive measure of risk we construct alternative indices with robust optimization techniques using the RevoScaleR package of Revolution R Enterprise.

Here's more information about the session, from Yahoo! Finance:

WHAT: “High-Frequency Financial Data Analysis with R,” a session at R/Finance 2011

WHO: Guy Yollin, Principal Consultant, R-Programming.org

WHEN: Saturday, April 29 at 12:00 p.m. Central Time

WHERE: University of Illinois at Chicago

Financial institutions rank among the most cutting edge companies when it comes to data analytics. Increasingly, their tool of choice is the open source R statistics language. Thanks to R’s powerful computation ability, extensibility and openness, data analysts at banks, investment firms, insurance organizations and other financial institutions can create and run virtually any algorithm to unlock intelligence that keeps them ahead. Today’s data analysts however must tackle larger data volumes than ever before and are tasked to analyze it faster then ever.

In this session, Guy Yollin will present some of the latest optimization techniques for the financial sector using the RevoScaleR package for big data analysis from Revolution Analytics. RevoScaleR is a framework for fast and efficient multi-core processing of large data sets.

Revolution Analytics is pleased to be a sponsor of R/Finance. Organized by members of the R community, R/Finance is a specialized forum focused on the use of R in the financial services industry for applications such as portfolio management, time series analysis, advanced risk tools, high-performance computing and econometrics. Register to attend the R/Finance Conference.

R/Finance 2011: Agenda

How to make 3-D graphics from SAS data

The blog SAS Analysis shows how to create 3-D images from SAS data ... using R:

Some SAS programmers like to use SAS/IML to call R’s functions [Ref. 2]. However, it seems that SAS/IML fails to work with the latest versions of R since 2.12 [Ref. 3]. Others tend to play tricks to call R into SAS’s data step to meet their daily needs [Ref. 4]. In this post, the macro below would call the ‘lattice’ package of R in SAS, on a PC platform, to draw paneled three dimension images, since currently SAS’s SG procedures don’t own such an option.

Essentially, the SAS Macro writes an R script which in turn calls the cloud function in R, creating this image from the sashelp.cars dataset in SAS:

SAS Analysis: A macro calls R in SAS for paneled 3d plotting

Revolution R Enterprise 4.3 now available

We've just released the latest update to Revolution R Enterprise, version 4.3. This release includes version 2.12.2 of the open-source R engine linked with high-performance libraries, and packages it with additional functionality from Revolution Analytics for big data statistics, web services deployment, a Windows UI for R programming and debugging, and much more. Detailed information about the new release can be found in this announcement, but the key new features include:

• New ‘Big Data’ predictive analytics algorithm: K-means clustering
• Import of data from SPSS files into high-performance XDF files
• Efficient multi-key sorting of large data sets stored in the XDF format
• Improved performance for importing fixed format ASCII and SAS files
• Calculate covariance and correlation matrix from large data sets stored in XDF format (see some examples in this post by Joseph Rickert)
• Support for date values in the XDF data format
• Enhancements to the Revolution R Productivity Environment (RPE)
• Update to version 2.12.2 of the open source R engine
• Integrated Installer for 32- and 64-bit Windows

Existing subscribers can download the update now (see your email for details), and as always Revolution R Enterprise can be downloaded free of charge by anyone in the academic community or for use in Kaggle prediction competitions.

Revolution Analytics: Revolution R Enterprise 4.3 Now Available

New functions for linear model inference in Revolution R Enterprise 4.3

The latest release of Revolution R Enterprise shows how Revolution Analytics’ package for big data, RevoScaleR, is continuing add new capabilities for Big Data statistics. RevoScaleR removes the limits on the size of the data that can be processed in R through the use of the highly efficient .Xdf binary file format. Xdf stores data by rows within columns and facilitates the use of external memory algorithms that process chunks of data. RevoScaleR functions operate directly from Xdf files so there is no need to read data into memory. The initial release of RevoScaleR included the functions rxLinMod (multiple regression) and rxLogit (binary logistic regression) enable users to build models using very large data sets. Both of these functions return model objects whose components can be graphed or fed into standard R functions so the integration with standard R is nearly seamless.

The new release of RevoScaleR includes a number of functions (Table 1) that complement rxLinMod and rxLogit and make it easier to do statistical inference.

Table 1: New RevoScaleR functions that complement rxLinMod and rxlogit

rxCov	returns the covariance matrix for a group of variables in an Xdf file
rxCor	returns the correlation matrix for a group of variables in an Xdf file
rxSSCP	returns the sum of squares or cross-product matrix for a group of variables in an Xdf file. When the variables are interpreted as the matrix X, that is the matrix of predictor variables in a regression model, the result of rxSSCP is the matrix X'X.
rxCovCor	a wrapper for the three functions above
rxCovCoef	returns the covariance matrix of the coefficients for a model built with rxLinMod or rxlogit.
rxCorCoef	returns the correlation matrix of the coefficients for a model built with rxLinMod or rxlogit.
rxCovData	returns the covariance matrix for the explanatory variables in model built with rxLinMod or rxLogit. It gives the same results as rxCov acting directly on the data
rxCorData	returns the correlation matrix for the explanatory variables in model built with rxLinMod or rxLogit. It gives the same results as rxCor acting directly on the data

To get a feel for what these new functions can do let’s look at some simple models built from the airlines data file that was used in the American Statistical Association’s 2009 Data Expo challenge. The ASA files contain information on all US domestic flights between 1987 and 2008. This data was read into a single Xdf file with over 123 million rows. The first example uses the new rxCor function to compute the correlation matrix of five variables: departure time, arrival time, air time, arrival delay and departure delay.

# Correlation Matrix
formula <- ~ DepTime + ArrTime + AirTime + ArrDelay + DepDelay
CorM <- rxCor(formula, data=working.file)
 Computation time: 7.20 seconds.
CorM
 
             DepTime    ArrTime      AirTime   ArrDelay    DepDelay
DepTime   1.00000000 0.72243775 -0.042412723 0.16885127 0.183462571
ArrTime   0.72243775 1.00000000  0.021413969 0.07656231 0.076315166
AirTime  -0.04241272 0.02141397  1.000000000 0.01650281 0.005269681
ArrDelay  0.16885127 0.07656231  0.016502811 1.00000000 0.887310003
DepDelay  0.18346257 0.07631517  0.005269681 0.88731000 1.000000000

The calculation took a little over 7 seconds on my 4 core (8 thread) 1.87 GHz Dell laptop out fitted with 8 GB of RAM running 64bit Windows & and Revolution R Enterprise 4.3.

Next, we build a small linear model with two independent variables and use rxCovCoef to compute the model covariance matrix. This process takes a little less than 5 seconds.

# Build a linear model
AD.model <- rxLinMod(ArrDelay ~ DepTime + ArrTime,data=working.file,covCoef=TRUE)
#Computation time: 4.67 seconds.
MCovM <- rxCovCoef(AD.model)
MCovM
 
              (Intercept)       DepTime       ArrTime
(Intercept)  8.399560e-05 -2.193390e-06 -3.065595e-06
DepTime     -2.193390e-06  7.416598e-07 -5.257515e-07
ArrTime     -3.065595e-06 -5.257515e-07  6.780731e-07

In theory, this matrix is equal to the variance of the regression model, s, multiplied by the inverse of the square of the design matrix: (X’X)^-1.  So dividing by and estimate of s² should give us a pretty good estimate of (X’X)^-1, a very useful item for doing statistical inference. A component of the model object produced by rxLinMod, AD.model$sigma, is a list whose first slot contains an estimator of s. So, the following code should produce and estimate of (X’X)^-1.  

# Get an estimate of X prime X inverse
sigma2 <- as.numeric(AD.model$sigma[[1]])^2
XprimeXinverse <- MCovM/sigma2
XprimeXinverse
 
             (Intercept)       DepTime       ArrTime
(Intercept)  9.138475e-08 -2.386344e-09 -3.335278e-09
DepTime     -2.386344e-09  8.069041e-10 -5.720022e-10
ArrTime     -3.335278e-09 -5.720022e-10  7.377236e-10

To see how good of an estimate we have, let’s use function rxCovCoef to get an estimate of (X’X).

# Estimate of X prime X
formula <- ~ DepTime + ArrTime
XprimeX <- rxSSCP(formula,data=working.file)
#Computation time: 3.54 seconds.
 
XprimeX
 
           Constant     DepTime     ArrTime
Constant  120950491  1654978671  1830027994
DepTime  1654978671 25397142048 27174128334
ArrTime  1830027994 27174128334 30699052023

Now, a little matrix multiplication and we should have the identity matrix.

XprimeX %*% XprimeXinverse
 
           (Intercept)       DepTime       ArrTime
Constant  1.000031e+00 -3.650103e-07 -3.456415e-08
DepTime   2.359597e-04  1.000050e+00 -3.814792e-05
ArrTime  -2.516375e-05 -8.483162e-05  1.000103e+00

Not too bad!

Milktrader: Quantitative finance in R

The blog Milktrader has been on a roll recently with a series of posts with practical examples of quantitative in finance, from backtesting to automated trading, and option pricing to data acquisition. The latest post focuses on calculating returns, with an example of downloading data for a silver ETF and calculating daily returns with the dailyReturn function in the quantmod package. If you're using R for quantitative finance, Milktrader is a great blog to add to your RSS feeds.

Milktrader: Chop, Slice and Dice Your Returns in R

How to load your iPhone location data into R

Earlier this week, data scientists Pete Warden and Alasdair Allen reported that iPhones and cell-enabled iPads keep an internal log of the devices location, which is accessible from the backup that iTunes creates when you sync the device. (Update Apr 27: Apple responds that the locations are those of nearby cell towers and wi-fi hotspots, not the device itself.)  Naturally, there's been some controversy over the privacy implications of the data keing kept, but from a data scientist's perspective this represents a rich and interesting data source for analysis. Personally, I'm kind of interested to get access to where I've been over the past year: wherever I go, my iPhone goes.

Pete Warden has provided a tool that lets you easily access and map the data. Here's why the data for my iPhone looks like when zoomed in on California:

If you have more than one iDevice the tool only shows the data from the one you most recently synced, and more importantly it doesn't (directly) give you access to the location data if you want to do more with it. Drew Conway comes to the rescue with an R package (amusingly called stalkR) that lets you import the data from a named device into an R object for visualization or other analysis. You'll need to make sure the dependent R packages are installed, and download and install the stalkR_0.01.tar.gz file in your R working directory:

install.packages("RSQLite")
install.packages("XML")
install.packages("ggplot2")
install.packages(c("maps","mapproj"))
install.packages("stalkR_0.01.tar.gz", repos=NULL, type="source")
 
library(stalkR)
iphone.locs<-get.mylocations("dsmith", "David Smiths iPhone")
viz.locations(iphone.locs, "usa")

The last two commands are the key ones. get.mylocations creates a data frame with timestamped latitute and longitude, along with a horizontal accuracy measure. There's also an Altitude variable, but at least in my data it was always zero (probably because the tracking data comes from cell tower triangulation, not GPS). The vis.locations function uses the syntax of the map function to map the locations, but the nice thing about having the data in R is that you can use it in whatever way you like.

Drew Conway's github: stalkR

 

New R User Group in Gießen, Germany

It's great to see new local R user groups popping up around the world, and the latest one is based in Gießen, Germany (about 70km north of Frankfurt). The Giessen R Users Group has its first meeting on May 10 with a discussion on applications of R to social media (with an example related to Twitter).

meetup.com: Giessen R Users Group

ComputerWorld on R for data analysis and visualization

ComputerWorld's feature today, 22 free tools for data visualization and analysis, suggests open-source R as the third entry on the list:

What it does: R is a general statistical analysis platform (the authors call it an "environment") that runs on the command line. Need to find means, medians, standard deviations, correlations? R can handle that and much more, including "linear and generalized linear models, nonlinear regression models, time series analysis, classical parametric and nonparametric tests, clustering and smoothing," according to the project website.

Check out the article linked below for the full write-up on R and information about lots of other free and useful tools for data scientists.

ComputerWorld: 22 free tools for data visualization and analysis