Revolutions

Because it's Friday: Watch out for those robots!

This robot from the Korea Advanced Institute of Science and Technology sports Oscar Pistorius style foot blades and a spinning "tail" that provides the stability of a speeding velociraptor. It can run at a record-breaking 46 kpm, and foam blocks thrown in its path don't even slow it down (Geek.com via Graeme Noseworthy).

 

It's a shame that robots still need to be connected to a static power supply (or a noisy gas motor), but once battery technology evolves to the point that robots like this can operate independently, we might want to start watching out! 

And speaking of robots, I wrote an essay recently for Confab magazine about how artificial intelligence is evolving rapidly to emulate human intelligence. So robots may soon be winning races and Turing tests!

That's all for this week. See you back here on Monday!

hipsteR: Learn what you missed in R as an early adopter

If you learned R in its early days (say, the early 2000's or even the late 1990's) you may still be using some — ahem — old-fashioned ways to accomplish some tasks better served by newer functions and packages. To help those of us who have may missed some of R's more recent innovations, Karl Broman created hipsteR, a guide for "re-educating people who learned R before it was cool". Some of the suggestions Karl offers:

• The ROpenSci project provides a number of packages to download data from public repositories.
• Use packages like dplyr, reshape2, lubridate and stringr for data manipulation.
• Use knitr instead of Sweave for literate programming.
• Consider using RStudio or Emacs+ESS as a programmers' interface to R.
• Don't forget that in addition to packages on CRAN, there are many packages on GitHub that you can easily install with help from the devtools package.
• You can use underscores in variable names, and = is a valid assignment operator.

Karl has many other suggestions of new functions and packages you should be using at the link below. Got any other recommendations of new and better ways of accomplishing tasks in R? Let us know if the comments.

hipsteR: re-educating people who learned R before it was cool

R / Finance 2014: Packaged Takeaways

by Joseph Rickert

I was very happy to have been able to attend R / Finance 2014 which wrapped up a couple of weeks ago. In general, the talks were at a very high level of play, some dealing with brand new ideas and many presented at a significant level of technical or mathematical sophistication. Fortunately, most of the slides from the presentations are quite detailed and available at the conference site. Collectively, these presentations provide a view of the boundaries of the conceptual space imagined by the leaders in quantitative finance. Some of this space covers infrastructure issues involving ideas for pushing the limits of R (Some Performance Improvements for the R Engine) or building a new infrasturcture (New Ideas for Large Network Analysis) or (Building Simple Data Caches) for example. Others are involved with new computational tools (Solving Cone Constrained Convex Programs) or attempt to push the limits on getting some actionable insight from the mathematical abstrations: (Portfolio Inference withthei One Wierd Trick) or (Twinkle twinkle litle STAR: Smooth Transition AR Models in R) for example.

But while the talks may be illuminating, the real take-aways from the conference are the R packages. These tools embody the work of the thought leaders in the field of computational finance and are the means for anyone sufficiently motivated to understand this cutting edge work.  By my count, 20 of the 44 tutorials and talks given at the conference were based on a particular R package. Some of the packages listed in the following table are well-established and others are work-in-progress sitting out on R-Forge or GitHub, providing opportunities for the interested to get involved.

R Finance 2014 Talk	Package	Description
Introduction to data.table	data.table	Extension of the data frame
An Example-Driven Hands-on introduction to Rcpp	Rcpp	Functions to facilitate integrating R with C++
Portfolio Optimization: Utility, Computation, Equities Applications	fPortfolio	Environment for reaching Financial Engineering and Computational Finance
Re-Evaluation of the Low Risk Anomaly via Matching	cem	Implementation of the Coarsened Exact matching Algorithm
BCP Stability Analytics: New Directions in Tactical Asset Management	bcp	Bayesian Analysis of Change Point Problems
On the Persistence of Cointegration in Pairs Trading	egcm	Engle-Granger Cointegration Models
Tests for Robust Versus Least Squares Factor Model Fits	robust	robust methods
The R Package cccp: Solving Cone Constrained Convex Programs	cccp	Solver for convex problems for cone constraints
Twinkle, twinkle little STAR: Smooth Transition AR Models in R	Twinkle	Modeling smooth transition models
Asset Allocaton with Higher Order Moments and Factor Models	DEoptim /   PortfolioAnalytics	Global optimization by differential evolution / Numerical methods for portfolio optimization
Event Studies in R	eventstudies	Event study and extreme event analysis
An R package on Credit Default Swaps	CDS	Provides tools for pricing credit default swaps
New Ideas for Large Network Analysis, Implemented in R	IRL	Implicitly restarted Lanczos methods for R
Package “Intermediate and Long Memory Time Series	ILMTS	Simulate & Detect Intermediate and Long Memory Processes / in development
Stochvol: Dealing with Stochastic Volatility in Time Series	stochvol	Efficient Bayesian Inference for Stochastic Volatility (SV) Models
Divide and Recombine for the Analysis of Large Complex Data with R	RHIPE	Package for using R with Hadoop
gpusvcalibration: Fast Stochastic Volatility Model Calibration using GPUs	gpusvcalibration	Fast calibration of stochastic volatility models for option pricing models
The FlexBayes Package	FlexBayes	Provides an MCMC engine for the class of hierarchical feneralized linear models and connections to WinBUGS and OpenBUGS
Building Simple Redis Data Caches	RcppReddis	Rcpp bindings for Redis that connects R to the Redis key/value store
Package pbo: Probability of Backtest Overfitting	pbo	Uses Combinatorial Symmetric Cross Validation to implement performance tests.

 

Many of these packages / projects also have supplementary material that is worth chasing down. Be sure to take a look at Alexios Ghalanos recent post that provides an accessible introduction to his stellar keynote address.

Many thanks to the organizers of the conference who, once again, did a superb job, and to the many professionals attending who graciously attempted to explain their ideas to a dilletante. My impression was that most of the attendies thoroughly enjoyed themselves and that the general sentiment was expressed by the last slide of Stephen Rush's presentation:

 

 

Benchmarking results compare speed of Revolution R Enterprise and legacy SAS

Many companies are considering switching from SAS to R for statistical data analysis, and may be wondering how R compares in performance and data size scalability to the legacy SAS systems (base SAS and SAS/Stat) they are currently using. Performance and scalability for R is exactly what Revolution R Enterprise (RRE) was designed for. In a recent webinar, Thomas Dinsmore described a benchmarking process to compare performance of legacy SAS and RRE. (The benchmarking process is described in the white paper Revolution R Enterprise: Faster Than SAS, and you can see the code behind the benchmarking process here.) In the webinar, Thomas revealed the following results:

• RRE ran the tasks forty-two times faster than legacy SAS on the larger data set
• RRE outperformed legacy SAS on every task
• The RRE performance advantage ranged from 10X to 300X
• The RRE advantage increased when we tested on larger data sets
• SAS’ new HP PROC, where available, only marginally improved SAS performance

Also in the webinar, John Wallace, founder and CEO of DataSong, described how performance and scalability requirements led to the selection in 2011 of Revolution R Enterprise as the analytics engine in their software-as-a-service platform. DataSong's industry-leading marketing analytics system currently analyzes more than $3billion in marketing spend by major retailers.

The slides from the webinar are embedded above, and you can watch and download the full webinar at the link below.

Revolution Analytics webinars: Is Revolution R Enterprise Faster than SAS? Benchmarking Results Revealed

Using Plotly with R for Project Workflows

by Matt Sundquist, Plotly co-founder

In a recent post we showed how to use ggplot2 and Plotly’s R API to make shareable, interactive graphs. This week, in response to questions and requests, we’ll highlight how to use Plotly for your workflow. We assume that you are working with Plotly from within R. Let’s get started.

Setting up Plotly and ggplot2
First you need to sign in to Plotly and set up. You can copy and paste this code, and either use our API key or sign-up for your own account. Public sharing is free and you own your own data (ilke GitHub), so feel free to plug in as many scripts as you’d like.

# install.packages("devtools") #so we can use install_github
library(devtools)
# install_github("plotly", "ropensci")
#Plotly is part of the rOpenSci project.

library(plotly)
library(ggplot2)
library(reshape)

Then you can sign-up on the Plot.ly site sign-up or running the following function from R.

signup("new_username", "[email protected]")

To start running plots, you’ll want to create a Plotly object. You can use our ggplot2 sandbox, or use your account.

py <- plotly("ggplot2examples", "3gazttckd7")

Grouped Bar Chart With Error Bars

Next let’s look at a simple project to show how Plotly can be a working tool. Say you find a grouped bar chart you like on Stack Overflow. You load the data.

raw <- read.csv("http://pastebin.com/raw.php?i=L8cEKcxS",sep=",")
raw[,2]<-factor(raw[,2],levels=c("Very Bad","Bad","Good","Very Good"),ordered=FALSE)
raw[,3]<-factor(raw[,3],levels=c("Very Bad","Bad","Good","Very Good"),ordered=FALSE)
raw[,4]<-factor(raw[,4],levels=c("Very Bad","Bad","Good","Very Good"),ordered=FALSE)
raw=raw[,c(2,3,4)]
freq=table(col(raw), as.matrix(raw)) # get the counts of each factor level

You then melt it, creating a plot:

Names=c("Food","Music","People")         # create list of names
data=data.frame(cbind(freq),Names)     # combine them into a data frame
data=data[,c(5,3,1,2,4)]                       # sort columns
# melt the data frame for plotting
data.m <- melt(data, id.vars='Names')
# plot everything
n <- ggplot(data.m, aes(Names, value)) +  
geom_bar(aes(fill = variable), position = "dodge", stat="identity")
py$ggplotly(n)

Appending py$ggplotly(n) is what calls the Plotly API. If you’re running it in an .Rmd document, the plot is embedded if you specify the plotly=TRUE chunk option (here is an example source). You can zoom by clicking and dragging, toggling, and hovering for text. You can click through to Plotly and and add a new axis or subplot and error bars.

If you’re reading this over email or it looks odd online, you may not be able to see the final version through an iframe. But that’s life! You can export plot for reports, emails, or presentations as a PNG, PDF, or SVG file, and link back to the original version. In that case you could link to the live version here.

Here it is in an iframe:

<div class = "iframe_container">
<iframe src="https://plot.ly/~MattSundquist/1455/1232/650/550" width="650" height=550" frameBorder="0" seamless="seamless" scrolling="no"></iframe>
</div>

You can also interact with your data and plot with R. I am new to R, and recently discovered str() while browsing an incredibly helpful R Reference Card by Tom Short. I enjoy using it while setting up, making, and re-making plots (shown here from within an IPython Notebook running the R kernel). 

Adding Data, Revisions, Reproducibility, & Collaboration

If you or your collaborators (you can share it like a Google Doc) want to edit or analyze the data, you can edit it in the grid, or add data with the R API, or by uploading data with Excel, Python, and MATLAB. 

A note re: reproducibility. Plotly is a part of the rOpenSci project, an open science and open data project. We’re excited about that. On a personal note, I have recently been pained to find a few graphs I wanted to re-make or data in plots I wanted to access, only to find the code is calling a local file, CSV, or expired URL. Our hope is that plots and data will be shareable, open, and reproducible if we keep the JSON, plot, and data together.

 

The special component of this is collaboration. You can share the grid for that data, or share the plot. My collaborators can then add their own data, edit the data, and send in new data, from any language or from the web app by simply uploading.

We often have had the problem of not knowing which data we used to make a graph, or which version of a graph was my final one. With Plotly, you can revert to a previous version and discuss graphs in context with your team as the next two graphs illustrate.

 

 

We’re on GitHub, and would love to hear your feedback.  Please write to us with questions, ideas or information on plotly projects you could do to win a t-shirt. ([email protected] or @plotlygraphs)

 

Companies using R in 2014

I'm honoured to have been invited to give a keynote presentation at the 7th China R Users Group conference in Beijing tomorrow — I'm told around 1000 R users are expected to be there. (This easily makes it the largest R users meeting in the world.) For my presentation, I decided to curate a list of some companies that are currently using open-source R and describe the applications they are using it for. 

My slides are embedded below, but here is a summary of the companies and their applications (including a few that I didn't have time to present in the keynote itself — these are included in the bonus slides of the presentation).

• Facebook uses R for Exploratory Data Analysis, Experimental Analysis, Big-Data Visualization, Human Resources, and user behaviour analysis related to status updates and profile pictures. 
• Google uses R for Advertising Effectiveness, Economic Forecasting, and Big-data statistical modeling.
• Twitter uses R for Data Visualization and Semantic Clustering.
• The City of Chicago uses R as a food poisoning monitor.
• The New York Times uses R for interactive features (such as the Dialect Quiz and the Election Forecast); for data journalism (including these stories on NFL draft picks and wealth distribution in the USA); and for data visualization (such as in articles on the Facebook IPO and baseball pitching legends).
• In video gaming, Microsoft uses R for Xbox matchmaking, and Scientific Revenue helps video game developers identify trouble-spots in game levels (and much more).
• Zillow uses R to create and update the Zestimate housing price estimate, and Trulia uses R for neighbourhood crime mapping.
• John Deere uses R for statistical analysis, to forecast crop yields and create long-term demand forecasts for its farming equipment.
• The Human Rights Data Analysis Group uses R to create more accurate counts of casualties in war zones, and the Sunlight Foundation uses R to keep track of political speech.
• In finance, ANZ Bank uses R for credit risk analysis; American Century Investments uses R to analyze financial networks; and Credit Suisse uses R on the trading desk.
• The FDA uses R in the regulatory drug approvals process.
• Monsanto uses R for statistical analysis in plant breeding, fertility mapping and yield forecasting.
• In insurance, Lloyds of London uses R for risk analysis and catastrophe modeling; Nationwide uses R for marketing analytics; and Deloitte uses R for actuarial analysis.
• RealClimate.org uses R for climate change analysis, and NOAA uses R for flood warnings.
• In marketing analytics, DataSong uses R for marketing attribution analysis; Exelate uses R for real-time ad selection; and X+1 uses R for consumer segmentation. 
• The list of software vendors and services companies with products related to R includes: Revolution Analytics, RStudio, Mango Solutions, Coursera, Zementis, DataCamp, SAS, Oracle, IBM, Teradata, TIBCO, Alteryx and SAP (amongst many others).

That's just a tiny fraction of the companies using R worldwide, but it goes to show: pretty much every industry now sees data and analytics as a strategic competitive advantage, and the fact that R is the language of choice of for data scientists is driving its rapid growth. 

 

Quick History 2: GLMs, R and large data sets

by Joseph Rickert

In last week’s post, I sketched out the history of Generalized Linear Models and their implementations. In this post I’ll attempt to outline how GLM functions evolved in R to handle large data sets.

The first function to make it possible to build GLM models with datasets that are too big to fit into memory was the bigglm()from Thomas Lumley’s biglm package which was released to CRAN in May 2006. bigglm()is an example of a external memory or “chunking” algorithm. This means that data is read from some source on disk and processed one chunk at a time. Conceptually, chunking algorithms work as follows: a program reads a chunk of data into memory, performs intermediate calculations to compute the required sufficient statistics, saves the results and reads the next chunk. The process continues until the entire dataset is processed. Then, if necessary, the intermediate results are assembled into a final result.

According to the documentation trail, bigglm()is based on Alan Miller’s 1991 refinement (algorithm AS 274 implemented in Fortran 77) to W. Morevin Genetlemen’s 1975 Algol algorithm ( AS 75). Both of these algorithms work by updating the Cholesky decomposition of the design matrix with new observations. For a model with p variables, only the p x p triangular Cholesky factor and a new row of data need to be in memory at any given time.

bigglm()does not do the chunking for you. Working with the algorithm requires figuring out how to feed it chunks of data from a file or a database that are small enough to fit into memory with enough room left for processing. ( Have a look at the make.data() function defined on page 4 of the biglm pdf for the prototype example of chunking by passing a function to bigglm()’s data argument.) bigglm() and the biglm package offer few features for working with data. For example, bigglm() can handle factors but it assumes that the factor levels are consistent across all chunks. This is very reasonable under the assumption that the appropriate place to clean and prepare the data for analysis is the underlying database.

The next steps in the evolution of building GLM models with R was the development of memory-mapped data structures along with the appropriate machinery to feed bigglm() data stored on disk. In late 2007, Dan Alder et al. released the ff package which provides data structures that, from R's point of view, make data residing on disk appear is if it were in RAM. The basic idea is that only a chunk (pagesize) of the underlying data file is mapped into memory and this data can be fed to bigglm(). This strategy really became useful in 2011 when Edwin de Jonge, Jan Wijffels and Jan van der Laan released ffbase, a package of statistical functions designed to exploit ff’s data structures. ffbase contains quite a few functions including some for basic data manipulation such as ffappend() and ffmatch(). For an excellent example of building a bigglm() model with a fairly large data set have a look at the post from the folks at BNOSAC. This is one of the most useful, hands-on posts with working code for building models with R and large data sets to be found. (It may be a testimony to the power of provocation.)

Not longer after ff debuted (June 2008), Michael Kane, John Emerson and Peter Haverty released bigmemory, a package for working with large matrices backed by memory-mapped files. Thereafter followed a sequence of packages in the Big Memory Project, including biganalytics, for exploiting the computational possibilities opened by by bigmemory. bigmemory packages are built on the Boost Interprocess C++ library and were designed to facilitate parallel programming with foreach, snow, Rmpi and multicore and enable distributed computing from within R. The biganalytics package contains a wrapper function for bigglm() that enables building GLM models from very large files mapped to big.matrix objects with just a few lines of code.

The initial release in early August 2010 of the RevoScaleR package for Revolution R Enterprise included rxLogit(), a function for building logistic regression models on very masive data sets. rxLogit() was one of the first of RevoScaleR’s Parallel External Memory Algorithms (PEMA). These algorithms are designed specifically for high performance computing with large data sets on a variety of distributed platforms. In June 2012, Revolution Analytics followed up with rxGlm(), a PEMA that implements all of the all of the standard GLM link/family pairs as well as Tweedie models and user-defined link functions. As with all of the PEMAS, scripts including rxGlm() may be run on different platforms just by changing a few lines of code that specifies the user’s compute context. For example, a statistician could test out a model on a local PC or cluster and then change the compute context to run it directly on a Hadoop cluster.

The only other Big Data GLM implementation accessible through an R package of which I am aware is h20.glm() function that is part of the 0xdata’s JVM implementation of machine learning algorithms which was announced in October 2013.  As opposed the the external memory R implementations described above, H20 functions run in the distributed memory created by the H20 process. Look here for h20.glm() demo code.

And that's it, I think this brings us up to date with R based (or accessible) functions for running GLMs on large data sets.

 

Microsoft uses R for Xbox matchmaking

If you've ever played online multiplayer games like Halo on the Xbox console, you'll know that being matched up with the right teammates and opponents is a critical part of the experience. Getting matched with an opponent that's way above your skill level isn't just demoralizing for you; it's also a waste of time for them (you generally don't advance in the game unless you beat a player that's of a higher level than you). The problem is even more complex for team games: there's nothing worse than being in a team with a bunch of newbies and then being pwned by a bunch of veterans on the other side.

Microsoft sets up online players into the best games for their skill level using the TrueSkill system, which is a Bayesian generalization of the Elo system used in chess matches. The basic methodology is described in this 2007 Microsoft Research paper, which used C# and F# for the data analysis and R for the visualizations. In the chart below, we see how the TrueSkill system performs compared to the Elo system for various kinds of games. An ideal matching system would generate the most number of drawn (tied) games, on the basis that players of equal skill are most likely to tie. According to the chart, the TrueSkill system works best for "Free for All" (e.g. Halo "Slayer") and "Head-to-Head" (one-on-one) matches, but not so for "Small Team" matches (which according to the paper are mostly capture-the-flag games).

The methodology has certainly progressed since 2007, and the new XBox One console features an all-new ranking system. Still, it's interesting to see the foundations of the system that matches players in Microsoft games. Thanks to reader Mikhail Popov for the tip, and to Microsoft's Tom Minka for some additional background on the paper.

Microsoft Research: TrueSkill(TM): A Bayesian Skill Rating System

Ensemble Methods Part 3: Revolution Analytics Big Data Random Forest Function

by Mike Bowles

In two previous posts, A Thumbnail History of Ensemble Methods  and Ensemble Packages in R, Mike Bowles — a machine learning expert and serial entrepreneur — laid out a brief history of ensemble methods and described a few of the many implementations in R. In this post Mike takes a detailed look at the Random Forests implementation in the RevoScaleR package that ships with Revolution R Enterprise.

Revolution Analytics' rxDForest() function provides an ideal tool for developing ensemble models on very large data sets.  It allows the data scientist to do prototyping on a single CPU version of the random forest algorithm and to then shift with relative ease to a multi-core version for generating a higher-performance model on an extremely large data set.  It’s convenient that the single-CPU and multiple-CPU versions operate on the same data, have many of the same input parameters and deliver the same types of performance summaries and analyses.  Revolution Anaytics is one of a very small number of companies offering a true multi-core version of Random Forests.  (I only know of one other)*.  

The computationally intensive part of ensemble methods is training binary decision trees, and the computationally intense part of training a binary tree is split-point determination.  Binary trees comprise a number of binary decisions that are of the form (attributeX < some number). The nodes in the tree pose this binary question and its answer determines whether an example goes to the left or right out of the node.  To train the binary tree every possible split point for every attribute has to be tried in order to pick the best one.  It’s easy to see why this split-point selection process consumes so much time — particularly on very large data sets.  In a standard tree formulation (CART for example) the number of tests is equal to the number of points in the data set (not the number of examples (rows), the number of rows times the number of attributes (columns).  This issue has been the subject of research for the last ten or so years. 

The Google PLANET paper discusses the sensible idea of approximating the split-point selection process by aggregating points into bins instead of checking every possible value.  More recent researchers have developed methods for generating approximate data histograms on streaming data. These methods are well-suited to the map-reduce environment and implemented in the Revolution Analytics version of binary decision trees and Random Forests.  Their incorporation makes the computation faster and introduces a “binning” parameter that may be unfamiliar to long-time users of single-CPU versions of random forests. 

The screen shots below the input and output for running the Revolution Analytics multi-cpu random forests on AWS.  The software is being run through a server version of RStudio.  Two CPUs are included in the cluster for building trees.  The first screen shot shows the code input for building a predictive model on the UC Irvine data set for red wine taste scores.  The code is the figure shows how little change is required for running Revolution Analytics’ multi-core version versus using one of the random forest packages available through CRAN.  

The next two screen shots show some of the familiar output that’s available.  The first plot shows the oob-prediction error as a function of the number of trees in the ensemble.  The second plot gives variable importance.  Those familiar with the wine taste data set will recognize that alcohol is correctly identified as the most significant feature for predicting wine taste. 

 

* Editor's Note: H20 from 0xdata contains a multi-core Random Forest implementation. 

Because it's Friday: Defying gravity, with magnets

IFLscience shared this neat video of a neodymium magnet nearly defying gravity as it falls through a copper tube thanks to Lenz's law. The induced current in the copper pipe generates a magnetic force that acts upwards on the magnet, slowing its fall.

 

That's all for this week. We'll be back again on Monday, have a great weekend!