Revolutions

Big Data Analytics predictions for 2014

As we close out the year, we asked a few members of the Revolution Analytics team to make a few predictions about big data analytics, data science and R for 2014. Here's what they came up with (including a few from yours truly).

Michele Chambers, Chief Strategy Officer and VP Product Management

• While the sexiest job in 2013 was the data scientist generalist, in 2014 the focus will shift to data analysts in dedicated business units. The demand for data analysts will rise as data analysts are closer to the business issues, making them the go-to resource for data-based decision making.

• In 2014, data analysts will be empowered through easy-to-use tool that leverage the insights of data scientists, by providing real-time forecasts and recommendations in their day-to-day business tools. Better analytics will make data analysis more effective, while automation frees up data scientists to focus on strategic initiatives and unlocking further value in corporate data stores.

• The dam will break for the data scientist supply and demand issue of 2013 for two reasons. First: higher education institutions have quickly adapted to this market need with custom programs to train the next generation of data scientists. In 2014, those grads will be entering the workforce. Second: companies are getting better at carving out focused, big picture projects for data scientists and pushing smaller and line of business projects to business users and data analyst.

Greg Todd, Chief Technology Officer

• 2014 will be the year when predictive analytics with data in Hadoop will become operational.

David Smith, VP Marketing and Community 

• In 2013, the open source programming language R broke through as the go-to statistical software, surpassing SAS. There are nearly three million R users today and this will only continue to grow as students who study and work with R enter the private sector.

• In 2014, the past 15 years of marketing research combined with Big Data predictive analytics will make one-to-one marketing a reality. Big data analytics help marketers take ideas from general to specific and tailor campaigns directly to the individual level.

• Consumers understand that data is being collected on them and in 2014 they will come to expect interactions and experiences to be personalized. Marketers have an opportunity to build relationships with consumers or risk losing them due to generic blanket campaigns,” said David Smith, vice president of marketing & community at Revolution Analytics.

For more 2014 predictions from Alteryx, Cloudera and Tableau, check out 14 Analytics Predictions for 2014. 

K-means Clustering 86 Single Malt Scotch Whiskies

by Luba Gloukhov

The first time I had an Islay single malt, my mind was blown. In my first foray into the world of whiskies, I took the plunge into the smokiest, peatiest beast of them all — Laphroig. That same night, dreams of owning a smoker were replaced by the desire to roam the landscape of smoky single malts.

As an Islay fan, I wanted to investigate whether distilleries within a given region do in fact share taste characteristics. For this, I used a dataset profiling 86 distilleries based on 12 flavor categories.

The data was obtained from https://www.mathstat.strath.ac.uk/outreach/nessie/nessie_whisky.html.

whiskies <- read.csv("data/whisky/whiskies.txt", row.names = 1, stringsAsFactors = FALSE)

I first went ahead and ensured that the dataset had no missing observations. I generated a subset of the data that included only the 12 flavor variables, rescaled for comparability using scale().

sum(is.na(whiskies))  # no missing observations

## [1] 0

whiskies_k <- scale(whiskies[2:13])  # rescale selected vars for kmeans

K-means clustering assigns each observation membership to one of k clusters in such a way that minimizes the distance between each observation and it's cluster's mean. K-means clustering requires us to specify the number of clusters. Below, we iterate through kmeans() with clusters argument varying from 1 to maxCluster and plot the within groups sum of squares for each iteration.

ssPlot <- function(data, maxCluster = 9) {
    # Initialize within sum of squares
    SSw <- (nrow(data) - 1) * sum(apply(data, 2, var))
    SSw <- vector()
    for (i in 2:maxCluster) {
        SSw[i] <- sum(kmeans(data, centers = i)$withinss)
    }
    plot(1:maxCluster, SSw, type = "b", xlab = "Number of Clusters", ylab = "Within groups sum of squares")
}
ssPlot(whiskies_k)

Naturally, the within groups sum of squares decreases as we increase the number of clusters. However, there is a trend of diminishing marginal returns as we increase the number of clusters. I select the number of clusters based on the point at which the marginal return of adding one more cluster is less than was the marginal return for adding the clusters prior to that.

fit <- kmeans(whiskies_k, 4)  # 4 cluster solution 

# append cluster assignment
whiskies <- data.frame(whiskies, fit$cluster)
whiskies$fit.cluster <- as.factor(whiskies$fit.cluster)

Cluster centers can inform on how taste profiles differ between clusters.

fit$centers

##      Body Sweetness    Smoky Medicinal Tobacco   Honey   Spicy   Winey
## 1 -0.6255   0.10478 -0.39342   -0.1825 -0.3606 -0.3720 -0.3647 -0.5765
## 2 -0.2541   0.05944 -0.04039    0.1214  2.7407 -0.2862  0.3606  0.2036
## 3  0.5113   0.05944  0.04733   -0.3071 -0.3606  0.7438  0.3220  0.7721
## 4  1.7163  -1.10234  2.46845    2.8149  1.7069 -1.2630  0.3606 -0.5111
##     Nutty   Malty   Fruity   Floral
## 1 -0.2395 -0.2673  0.06586  0.29654
## 2 -0.3632  0.5792 -0.38788  0.15866
## 3  0.4297  0.3624  0.13698 -0.07171
## 4 -0.3632 -0.7455 -0.81553 -1.79062

Based on these centers, I anticipate that my love for the full bodied, smoky and medicinal lies in cluster 4.

subset(whiskies, fit.cluster == 4)

##    Distillery Body Sweetness Smoky Medicinal Tobacco Honey Spicy Winey
## 4      Ardbeg    4         1     4         4       0     0     2     0
## 22   Caol Ila    3         1     4         2       1     0     2     0
## 24  Clynelish    3         2     3         3       1     0     2     0
## 58  Lagavulin    4         1     4         4       1     0     1     2
## 59   Laphroig    4         2     4         4       1     0     0     1
## 78   Talisker    4         2     3         3       0     1     3     0
##    Nutty Malty Fruity Floral   Postcode Latitude Longitude fit.cluster
## 4      1     2      1      0  \tPA42 7EB   141560    646220           4
## 22     2     1      1      1  \tPA46 7RL   142920    670040           4
## 24     1     1      2      0   \tKW9 6LB   290250    904230           4
## 58     1     1      1      0   PA42 7DZ   140430    645730           4
## 59     1     1      0      0   PA42 7DU   138680    645160           4
## 78     1     2      2      0   IV47 8SR   137950    831770           4

I identified the most representative whisky of each cluster by seeking out the observation closest to the center based on all 12 variables.

whiskies_r <- whiskies[c(2:13, 17)]
# extract just flavor variables & cluster
candidates <- by(whiskies_r[-13], whiskies_r[13], function(data) {
    # we apply this function to observations for each level of fit.cluster
    dists <- sapply(data, function(x) (x - mean(x))^2)
    # for each variable, calc each observation's deviation from average of the
    # variable across observations
    dists <- rowSums(dists)
    # for each observation, sum the deviations across variables
    rownames(data)[dists == min(dists)]
    # obtain the row number of the smallest sum
})

candidates <- as.numeric(unlist(candidates))

whiskies[candidates, ]

##      Distillery Body Sweetness Smoky Medicinal Tobacco Honey Spicy Winey
## 42 Glenallachie    1         3     1         0       0     1     1     0
## 70 RoyalBrackla    2         3     2         1       1     1     2     1
## 1     Aberfeldy    2         2     2         0       0     2     1     2
## 4        Ardbeg    4         1     4         4       0     0     2     0
##    Nutty Malty Fruity Floral   Postcode Latitude Longitude fit.cluster
## 42     1     2      2      2   AB38 9LR   326490    841240           1
## 70     0     2      3      2   IV12 5QY   286040    851320           2
## 1      2     2      2      2  \tPH15 2EB   286580    749680           3
## 4      1     2      1      0  \tPA42 7EB   141560    646220           4

The dataset contains coordinates that I used to investigate how flavor profiles differ geographically. The dataset's Latitude and Longitude variables are coordinates defined according to Great Britain's Ordnance Survey National Grid reference system. I converted the coordinates to standard latitude and longitude in order to plot them using ggmap.

library(maptools)

## Loading required package: sp Checking rgeos availability: TRUE

library(rgdal)

## rgdal: version: 0.8-14, (SVN revision 496) Geospatial Data Abstraction
## Library extensions to R successfully loaded Loaded GDAL runtime: GDAL
## 1.10.1, released 2013/08/26 Path to GDAL shared files:
## C:/Revolution/R-Enterprise-6.2/R-2.15.3/library/rgdal/gdal GDAL does not
## use iconv for recoding strings. Loaded PROJ.4 runtime: Rel. 4.8.0, 6 March
## 2012, [PJ_VERSION: 480] Path to PROJ.4 shared files:
## C:/Revolution/R-Enterprise-6.2/R-2.15.3/library/rgdal/proj

whiskies.coord <- data.frame(whiskies$Latitude, whiskies$Longitude)
coordinates(whiskies.coord) = ~whiskies.Latitude + whiskies.Longitude

proj4string(whiskies.coord) = CRS("+init=epsg:27700")  # Specify that our coords are in osgb grid coord

whiskies.coord <- spTransform(whiskies.coord, CRS("+init=epsg:4326"))  # spTransform to convert osgb grid to lat/lon

whiskies <- cbind(whiskies, whiskies.coord)

Alternatively, the ggmap package ships with a geocode function which uses Google Maps to determine the lat/lon based on a character string specifying the location.

library("ggmap")

## Loading required package: ggplot2

whiskies <- cbind(whiskies, geocode(paste(whiskies$Location, "Scotland", sep = " ,")))

whiskyMap <- qmap(location = "Scotland", zoom = 6, legend = "topleft", maptype = "terrain", 
    color = "bw", darken = 0.5)

## Map from URL :
## http://maps.googleapis.com/maps/api/staticmap?center=Scotland&zoom=6&size=%20640x640&scale=%202&maptype=terrain&sensor=false
## Google Maps API Terms of Service : http://developers.google.com/maps/terms
## Information from URL :
## http://maps.googleapis.com/maps/api/geocode/json?address=Scotland&sensor=false
## Google Maps API Terms of Service : http://developers.google.com/maps/terms

whiskyMap + geom_point(data = whiskies, aes(x = whiskies.Latitude, y = whiskies.Longitude, 
    colour = fit.cluster, size = 2))

I zoomed in and examine which Distilleries lie within the Islay region.

whiskyMap <- qmap(location = "Islay", zoom = 10, legend = "topleft", maptype = "terrain", 
    color = "bw", darken = 0.5)

## Map from URL :
## http://maps.googleapis.com/maps/api/staticmap?center=Islay&zoom=10&size=%20640x640&scale=%202&maptype=terrain&sensor=false
## Google Maps API Terms of Service : http://developers.google.com/maps/terms
## Information from URL :
## http://maps.googleapis.com/maps/api/geocode/json?address=Islay&sensor=false
## Google Maps API Terms of Service : http://developers.google.com/maps/terms

whiskyMap + geom_point() + geom_text(data = whiskies, aes(x = whiskies.Latitude, 
    y = whiskies.Longitude, label = Distillery, color = fit.cluster, face = "bold"))

## Warning: Removed 78 rows containing missing values (geom_text).

The results indicate that there is a lot of variation in flavor profiles within the different scotch whisky regions. Note that initial cluster centers are chosen at random. In order to replicate the results, you will need to run the following code before your analysis.

set.seed(1)

Further data analysis would be required to determine whether proximity to types of water sources or terrain types drive common flavor profiles. This could be done by obtaining shape files and adding them as an additional layer to the ggmap plot.

For me, I have identified my next to-try single malt. Talisker is still within the familiar realm of cluster 4 but a little more malty, fruity and spicy. Sounds like the perfect holiday mix.

Because it's Friday: Illusions sell

The marketers at Honda clearly wanted people to think this CR-V ad was so cool, that they had to share it. Well, they got me:

 

All of these illusions employ forced perspective, the same technique used in the Lord of the Rings movies to make the Hobbits appear tiny compared to Men, using similarly-sized actors. They are also beautifully executed (note how the fake shadows align perfectly with the sun's shadow in 'Bollards'). You can see how the illusions were made in the making-of video, but it's largely redundant, since the beauty of the commercial is that it reveals most tricks with a change in perspective. Check out the construction for "Doorway" and "Arches", though — it's quite impressive!

Quantitative Finance Applications in R

by Daniel Hanson, QA Data Scientist, Revolution Analytics

Extracting Financial Data from Internet Source Using R (first in a series)

Earlier this month, a colleague and I attended a presentation on Computational Finance in R, given by Guy Yollin of the University of Washington Applied Mathematics faculty, at a meeting of the Seattle useR Group. The first among several topics was extracting financial time series from internet sources using two R packages: quantmod, and Quandl. This will be the subject of today’s blog post.

For those of us who do not have access to a Bloomberg terminal but wish to have data to experiment with financial models, these two packages are a real boon.  While quantmod also contains plotting and charting functionality, we will look at its capability of downloading stock and index prices from Yahoo Finance and Google Finance.  The Quandl package is a bit different in that it is tied in with the Quandl website, a source of financial data itself, as well as a portal to economic and social science data.  We will focus on accessing financial futures data on Quandl.

quantmod

Many readers are likely familiar with the finance sites on Yahoo and Google as sources for tracking stock, mutual fund, and exchange traded fund (ETF) prices and returns.  Using quantmod, one can easily load this data into R by specifying the the same ticker symbol that is used in these two web sources.  

As with any other R package, one must install the quantmod package in the usual way.  As an aside, included with the installation is also the xts package for time series data, which we will discuss briefly later on.

After installing the package, using it is quite simple and straightforward.  For example, to retrieve all available data for the PowerShares QQQ Nasdaq 100 Index ETF, one enters the following R command:

getSymbols("QQQ")

The data set is returned to the R session in the form of an xts object with the name QQQ.  To check the contents, the head(.) and tail(.) R functions are useful:

head(QQQ)

gives us

          QQQ.Open QQQ.High QQQ.Low QQQ.Close QQQ.Volume QQQ.Adjusted

2007-01-03    43.46    44.06   42.52     43.24  167689500        41.27
2007-01-04    43.30    44.21   43.15     44.06  136853500        42.05
2007-01-05    43.95    43.95   43.48     43.85  138958800        41.85
2007-01-08    43.89    44.12   43.64     43.88  106401600        41.88
2007-01-09    44.01    44.29   43.63     44.10  121577500        42.09
2007-01-10    43.96    44.66   43.82     44.62  121070100        42.58

and tail(QQQ)gives us

          QQQ.Open QQQ.High QQQ.Low QQQ.Close QQQ.Volume QQQ.Adjusted

2013-12-12    85.21    85.42   84.90     84.96   33630800        84.96
2013-12-13    85.36    85.40   84.71     84.85   34072000        84.85
2013-12-16    85.27    85.81   85.21     85.32   24902500        85.32
2013-12-17    85.32    85.53   84.99     85.15   24411800        85.15
2013-12-18    85.25    86.22   84.05     86.14   68270800        86.14
2013-12-19    86.00    86.01   85.52     85.90   44964100        85.90

Now, a couple of remarks about what we have done, and other options that are available.  First, note that we are given daily prices.  In order to retrieve, say, monthly values, one may use an xts command on the return object; xts also provides functionality to extract a subset of the data based on a range of dates.  This is a discussion worth having, given that xts is widely used in financial modeling in R, but we will defer it to a later blog post.  Secondly, note that we did not specify the data source; this is because the default is Yahoo.

So, as another example, and employing a little bit of irony, let’s download Yahoo’s stock price history, using Google Finance as the data source:

# use single quotes and specify data source:
getSymbols("YHOO", src = "google")  # src = "yahoo" is the default

We can then extract the closing prices to an R vector:

# coerce from an xts object to a standard numerical R vector:
as.vector(YHOO[, "YHOO.Close"])  

Finally, we can calculate the daily log returns in one fell swoop:

log.yahoo <- log(yhoo.close[-1]/yhoo.close[-length(yhoo.close])
# Check:
head(log.yahoo)
tail(log.yahoo)

These yield the following:

> head(log.yahoo)
[1]  0.047282921  0.032609594  0.006467863 -0.012252406  0.039806250  0.017271587
> tail(log.yahoo)
[1]  0.004840157  0.009610595  0.000000000 -0.005552765  0.013325150  0.003988041

Quandl

One of the limitations of data available from Yahoo and Google, as may be noticed above, is that it only dates back to January of 2007, while fund tracking and analysis will often require a time series of 20 years or more of historical returns.  Furthermore, some of the index-tracking ETF’s didn’t come into existence until the early 2000’s, and new tracking funds for emerging markets and other indices have even shorter histories.  To get return data going back farther in time, it is common to look to futures markets, and Quandl provides a rich set of historical futures prices.

As with the quantmod package, it is necessary to install the Quandl package in the same way.  The Quandl R package help  page gives step by step instructions, followed by a first example of loading futures price data into R, which we also show here:

# Oil futures price data from the National Stock Exchange of India:
mydata = Quandl("NSE/OIL")

The default for the return object from Quandl is a standard R dataframe; an xts option is also available, but for this demonstration we will stick with a dataframe.  Checking the data, we get:

> head(mydata)
  Date       Open   High   Low    Last   Close   Quantity Turnover(Lacs)
1 2013-12-12 462.00 470.65 458.50 461.00 460.05     622936      2903.94
2 2013-12-11 469.10 474.50 462.20 463.60 463.25     398041      1849.26
3 2013-12-10 472.00 479.85 472.00 474.30 473.90     476841      2279.75
5 2013-12-06 461.90 468.00 460.25 462.00 462.05     207715       960.85
6 2013-12-05 473.30 475.00 461.00 461.70 461.90     313943      1462.70

> tail(mydata)
  Date       Open   High   Low    Last    Close  Quantity Turnover(Lacs)
  2009-10-08 1145.0 1179.8 1142.0 1178.10 1170.20   788173      9138.99
  2009-10-07 1153.8 1160.7 1140.0 1141.45 1141.60   698216      8032.98
  2009-10-06 1149.8 1157.2 1132.1 1143.30 1144.90   627957      7185.90
  2009-10-05 1152.0 1165.9 1136.6 1143.00 1140.55   919832     10581.13
  2009-10-01 1102.0 1173.7 1102.0 1167.00 1166.35  3074254     35463.78
  2009-09-30 1096.0 1156.7 1090.0 1135.00 1141.20 19748012    223877.07

This gives us a first example, but note that the data only goes back to September of 2009.  As a more realistic example, let’s look at futures prices that have a much longer record, and let’s specify that we want monthly rather than daily (the default setting) data.  A good example is the price of gold futures, but how would we find the lookup symbol?  There is an R search function in the Quandl package, but at least at the time of this writing, it exhibits, shall we say, some need for improvement.  

We have found a more reliable approach at this stage is to start with the Futures section of the Quandl website, and then select Metals from the list that appears at left.  At the top of the Metals list that is now displayed, click on Gold (Symbol GC/Exchange COMEX).  A new Gold Futures page will load, with two contracts (GC1 and GC2) shown under Latest Quotes.  Choose Historical Data for GC2, and a page showing a plot of returns and price data will be displayed; however, we want to get the symbol to use in our R command, and to do this, scroll all the way down to the bottom of the page where a box entitled DATASET INFORMATION is displayed, as shown here:

We can now get the lookup symbol from the end of the Permalink line, namely

OFDP/FUTURE_GC2

and pop this into an R command to load the gold futures price data:

# set collapse="monthly" to retrieve monthly prices
gold <- Quandl("OFDP/FUTURE_GC2", collapse="monthly")

To see how far back the price history goes, we use tail(.), because the data is loaded in reverse order:

> tail(gold)

     Date       Open  High  Low   Settle Volume Open Interest
463 1975-05-31 169.1 169.5 168.2  169.0     NA          2539
464 1975-04-30 169.6 169.8 168.9  168.9     NA           813
465 1975-03-31 178.2 179.7 178.0  179.5     NA          2058
466 1975-02-28 185.8 185.9 184.0  185.9     NA           504
467 1975-01-31 178.0 178.4 177.0  178.0     NA          2016
468 1974-12-31 195.4 195.4 185.7  185.7     NA           366

Although this example may seem a bit cumbersome, running through it a few times with different types of futures contracts and trying it out, the process of obtaining the lookup symbol eventually becomes somewhat routine.

Finally, suppose we don’t want to go back to 1974, but instead we want a 20 year period of monthly prices from June 1992 through May 2012.  In this case, we simply put in start_date and end_date parameters as follows:

gold20 <- Quandl("OFDP/FUTURE_GC2", collapse="monthly",
                  start_date="1992-06- 01", end_date="2012-05-01")

This will give us the end-of-month futures price data from June 1992 through May 2012.

To calculate the log returns of monthly settled prices, we again extract this data as a vector, but remembering that the prices are in reverse chronological order, we need to reverse the order of the vector and then do the same calculation as we did for the Yahoo stock returns in the previous section:

gold.settle <- gold20[, "Settle"]
gold.settle <- rev(gold.settle)
log.gold <- log(gold.settle[-1]/gold.settle[-length(gold.settle)])

# Check:
head(log.gold)
tail(log.gold)

These yield the following:

> head(log.gold)
[1]  0.063351108  0.003235085  0.024869116  0.015285152 -0.103421591  0.109093821

> tail(log.gold)
[1]  0.00388234  0.01772572  0.02286972 -0.01180379  0.04002237 -0.03741933

In future posts, we will expand into topics of interest in quantitative finance and risk management.  In the meantime, for more examples of using Quandl data in R, take a look at Joseph Rickert’s mini-tutorial on Quandl.

 

Tips on Computing with Big Data in R

by Joseph Rickert

The Revolution R Enterprise 7.0 Getting started Guide makes a distinction between High Performance Computing (HPC) which is CPU centric, focusing on using many cores to perform lots of processing on small amounts of data, and High Performance Analytics (HPA), data centric computing that concentrates on feeding data to cores, disk I/O, data locality, efficient threading, and data management in RAM. The following collection of tips for computing with big data is an abbreviated version of the Guide’s discussion of the HPC and HPA considerations underlying the design of Revolution R Enterprise 7.0 and RevoScaleR, Revolution’s R package for HPA computing.

1 - Upgrade your hardware

It doesn’t hurt to state the obvious: bigger is better. In general, memory is the most important consideration. Getting more cores can also help, but only up to a point since R itself can generally only use one core at a time internally. Moreover, for many data analysis problems the bottlenecks are disk I/O and the speed of RAM, making it difficult to efficiently use more than 4 or 8 cores on commodity hardware.

2 - Upgrade your software

R allows its core math libraries to be replaced. Doing so can provide a very noticeable performance boost to any function that make use of computational linear algebra algorithms. Revolution R Enterprise links in the Intel Math Kernel Libraries.

3 - Minimize copies of the data

R does quite a bit of automatic copying. For example, when a data frame is passed into a function a copy of the data is made if the data frame is modified, and putting a data frame into a list also automatically causes a copy to be made. Moreover, many basic analysis algorithms, such as lm and glm, produce multiple copies of a data set as the computations progress. Memory management is important.

4 - Process data in chunks

Processing data a chunk at a time is the key to being able to scale computations without increasing memory requirements. External memory algorithms load a manageable amount of data into RAM, perform some intermediate calculations, load the next chunk and keep going until all of the data has been processed. Then, the final result is computed from the set of intermediate results. There are several CRAN packages including biglm, bigmemory, ff and ffbase that either implement external memory algorithms or help with writing them. Revolution R Enterprise’s RevoScaleR package takes chunking algorithms to the next level by automatically taking advantage of the computational resources to run its algorithms in parallel.

5 - Compute in parallel across cores or nodes

Using all of the available cores and nodes is key to scaling computations to really big data. However, since data analysis algorithms tend to be I/O bound when data cannot fit into memory, the use of multiple hard drives can be even more important than the use of multiple cores. The CRAN package foreach provides easy-to-use tools for executing R functions in parallel on both on a single computer and across multiple computers. The foreach() function is particularly useful for “embarrassingly parallel” computations that do not involve communication among different tasks.

The statistical functions and machine learning algorithms in the RevoScaleR package are all Parallel External Memory Algorithm’s (PEMA’s). They automatically take advantage of all of the cores available on a machine or on a cluster (including LSF and Hadoop clusters.)

6 - Take advantage of integers

In R, the two choices for “continuous” data are numeric, an 8 byte (double) floating point number and integer, a 4 byte integer. There are circumstances where storing and processing integer data can provide the dual advantages using less memory and decreasing processing time. For example, when working with integers, a tabulation is generally much faster than sorting and gives exact values for all empirical quantiles. Even when you are not working with integers scaling and converting to integers can produce fast and accurate estimates of quantiles. As an example, if the data consists of floating point values in the range from 0 to 1,000, converting to integers and tabulating will bound the median or any other quantile to within two adjacent integers. Then interpolation can get you even closer approximation. 

7 - Store data efficiently

You will want to store big data so that it can efficiently accessed from disk. The use of appropriate data types can save both storage space and access time. Take advantage of integers and, when you can, store data in 32-bit floats not 64-bit doubles. A 32-bit float can represent 7 decimal digits of precision, which is more than enough for most data, and it takes up half the space of doubles. Save the 64-bit doubles for computations.

8 - Only read the data needed

Even though a data set may have many thousands of variables, typically not all of them are being analyzed at one time. By reading from disk just the actual variables and observations you will use in analysis, you can speed up the analysis considerably.

9 - Avoid loops when transforming data

Loops in R can be very slow compared with  R’s core vector operations which are typically written in C, C++ or Fortran, compiled languages that execute much quicker than the R interpreter.

10 - Use C, C++, or Fortran for critical functions

One R’s great strengths is its ability to integrate easily with other languages, including C, C++, and Fortran. You can pass R data objects to other languages, do some computations, and return the results in R data objects. The CRAN package Rcpp,for example, makes it easy to call C and C++ code from R.

11 - Process data transformations in batches

When working with small data sets, it is common to perform data transformations one at a time. For instance, one line of code might create a new variable, and subsequent lines perform additional transformations with each transformation requiring a pass through the data. To avoid the overhead of making multiple passes over a large data set write chunking algorithms that apply all of the transformations to each chunk. RevoScaleR’s rxDataStep() function is designed for one pass processing by permitting multiple data transformations to be performed on each chunk.

12 - User row-oriented data transformations where possible

When writing chunking algorithms, try to avoid algorithms that cross chunk boundaries. In general, data transformations for a single row of data should not be dependent on values in other rows. The key idea is that a  transformation expression should give the same result even if only some of the rows of data are in memory at one time. Data manipulations requiring lags can be done but require special handling.

13 - Handle categorical variables efficiently and with care

Working with categorical or factor variables in big data sets can be challenging. For starters, not all of the factor levels may be represented in a single chunk of data. Using R’s factor() function in a transformation on a chunk of data without explicitly specifying all of the levels that are present in the entire data set might cause you to end up with incompatible factor levels from chunk to chunk. Also, building models with factors having hundreds of levels may cause hundreds of dummy variables to be created that really eat up memory. The functions in the RevoScaleR package that deal with factors minimize memory use and do not generally explicitly create dummy variables to represent factors.

14 - Be aware of 0utput with the same number of rows as your input

Most analysis functions return a relatively small object of results that can easily be handled in memory. Occasionally, however, output will have the same number of rows as the data: when computing predictions and residuals for example. In order for this to scale, you will want the output written out to a file rather than kept in memory.

15 - Think Twice Before Sorting

Sorting is by nature a time-intensive operation. Do what you can to avoid sorting a large data set. Use functions that compute estimates of medians and quantiles and look for implementations of popular algorithms that avoid sorting. For example, the RevoScaleR function rxDTree() avoids sorting by working with histograms of the data rather that with the raw data itself.

Season's Greetings from Revolution Analytics

Christmas Day is a holiday for the Revolution Analytics team. However you celebrate, the warmest seasons greetings go to our customers, partners, readers and the R community worldwide from everyone at Revolution Analytics. Click here for a holiday message from our CEO, Dave Rich. 

We leave you with some seasonal cheer from our Mountain View holiday party — thanks to Parisa M for sharing!

 

 

Combining the Power of DeployR, rCharts, and AngularJS

by Sheri Gilley, Principal UI Designer at Revolution Analytics

Revolution R Enterprise DeployR is a server framework that exposes the R platform as a service allowing the integration of R statistics, analytics and visualizations inside Web, desktop and mobile applications.  DeployR’s framework allows users to store and serve up R code that can then easily be used and reused in your own interfaces. By separating the R code from the UI code, not only can you leave the analytics to the R developer and the UI designs to the web developers, but you can also make it easier to unit test and debug the R code. Additionally, this separation of R and UI allows you to make the best use of the best software for each. You can take advantage of choice UI frameworks, such as AngularJS, to develop and solve front-end design issues and DeployR for managing and executing high powered R analytics.

We’ve been getting lots of questions about the rCharts package and whether rCharts output can be integrated into DeployR.  The answer is yes -- it is a wonderful way to create a chart in R that you can then augment with interactivity using HTML and JavaScript.  Below is an example of code designed to be executed only once on the DeployR server and then interacted with numerous times on the client-side without any additional load on the server.

A few weeks ago, Joseph Rickert introduced a method for retrieving package information from CRAN.  He created a count of the number of "Depends" per package, and then plotted a histogram of this count.  This  same data is used in the example below to demonstrate how to combine DeployR, rCharts, and AngularJS to display some histograms.  The code itself will execute only once and all additional interactions occur on the client.

This chart was built with:

• An R script (Histograms.R) stored on the DeployR server and publicly accessible. This script creates a chart using the rCharts package.
• An HTML page (index.html) that displays the chart and its interactive controls.
• A JavaScript file (script.js), using the AngularJS framework, that runs the script once on the DeployR server, and then updates the chart on the client-side whenever a user changes one of the controls. 

You can download the R, HTML, and JavaScript files here.  NOTE: Due to the libraries used in the JavaScript code, this chart is best viewed in the latest version of Chrome, FireFox, Safari, or IE 9+.

Apache Spark for Big Analytics

by Thomas Dinsmore, Director of Product Management at Revolution Analytics

The emergence of Apache Spark is a key development for Big Analytics in 2013.   Spark, an Apache incubator project, is an open source distributed computing framework for advanced analytics in Hadoop.  Originally developed as a research project at UC Berkeley's AMPLab, the project achieved incubator status in Apache in June 2013.

Spark seeks to address the critical challenges for advanced analytics in Hadoop.  First, Spark is designed to support in-memory processing, so developers can write iterative algorithms without writing out a result set after each pass through the data.  This enables true high performance advanced analytics; for techniques like logistic regression, project sponsors report runtimes in Spark 100X faster than what they are able to achieve with MapReduce. 

Second, Spark offers an integrated framework for advanced analytics, including a machine learning library (MLLib); a graph engine (GraphX); a streaming analytics engine (Spark Streaming) and a fast interactive query tool (Shark).   This eliminates the need to support multiple point solutions, such as Giraph, GraphLab and Tez for graph engines; Storm and S3 for streaming; or Hive and Impala for interactive queries.  A single platform simplifies integration, and ensures that users can produce consistent results across different types of analysis.  

At Spark's core is an abstraction layer called Resilient Distributed Datasets, or RDDs.  RDDs are read-only partitioned collections of records created through deterministic operations on stable data or other RDDs.  RDDs include information about data lineage together with instructions for data transformation and (optional) instructions for persistence.  They are designed to be fault tolerant, so that if an operation fails it can be reconstructed.  

For data sources, Spark works with any file stored in HDFS, or any other storage system supported by Hadoop (including local file systems, Amazon S3, Hypertable and HBase).  Hadoop supports text files, SequenceFiles and any other Hadoop InputFormat.

Spark's machine learning library, MLLib, is rapidly growing.   In the latest release it includes linear support vector machines and logistic regression for binary classification; linear regression; k-means clustering; and alternating least squares for collaborative filtering.  Linear regression, logistic regression and support vector machines are all based on a gradient descent optimization algorithm, with options for L1 and L2 regularization.  MLLib is part of a larger machine learning project (MLBase), which includes an API for feature extraction and an optimizer (currently in development with planned release in 2014).

GraphX, Spark's graph engine, is currently in beta.  GraphX combines the advantages of data-parallel and graph-parallel systems by efficiently expressing graph computation within the Spark framework.  It enables users to interactively load, transform, and compute on massive graphs.  Project sponsors report performance comparable to Apache Giraph, but in a fault tolerant environment that is readily integrated with other advanced analytics.

Spark Streaming offers an additional abstraction called discretized streams, or DStreams.  DStreams are a continuous sequence of RDDs representing a stream of data; they are created from live incoming data or generated by transforming other DStreams.  Spark receives data, divides it into batches, then replicates the batches for fault tolerance and persists them in memory where they are available for mathematical operations. 

Currently, Spark supports programming interfaces for Scala, Java and Python.  For R users, there is good news: an R interface is in the works and under development by the team at AMPLab; our sources tell us  this is expected to be released in the first half of 2014.

There is an active and growing developer community for Spark, as shown in the chart below.  In the past six months, more than one hundred developers have contributed to Spark; in the same time period, just three developers actively contributed to Mahout.

Spark developers contributed almost five thousand commits in the past six months, more than all of the other projects shown in the chart combined.   In 2013, the Spark project has published seven double-dot releases, including Spark 0.8.1 published on December 19; this latest release includes YARN 2.2 support, high availability mode for cluster management, performance optimizations and improvements to the machine learning library and Python interface.

In a nod to Spark's rapid progress, Cloudera recently announced that it plans to distribute Spark in CDH5, and will partner with Databricks to provide commercial support. This announcement resonated in the analyst community; Derrick Harris of GigaOm, for example, wrote that "Spark is a really big deal for Big Data, and Cloudera gets it."

Recently, the first Spark Summit attracted more than 450 participants from more than 180 companies. Presentations covered a range of applications such as neuroscience, audience expansion, real-time network optimization and real-time data center management, together with a range of technical topics.

In 2014, Spark's organizers expect to achieve top-level status in Apache.  Developers expect to continue adding machine learning features and to simplify implementation.  Together with an R interface and commercial support, we can expect continued interest and application for Spark.

Because it's Friday: Earth Wind (and no fire)

It's cold and unusually snowy here in Seattle, so the weather seemed like a good theme for this week's Because it's Friday post. This gorgeous visualization of global weather conditions by Cameron Beccario is clearly inspired by the hint.fm Wind Map (now on display at the New York Museum of Modern Art), but encompasses the enire globe. Click through for its animated awesomeness: you can even rotate the globe to see the hidden continents and oceans, and use your scroll wheel to zoom. (Flash must be enabled in your browser.)

Click on the "earth" button at the bottom left for more interactive goodness, including the ability to go back in time to see historical weather patterns, choose a wind height to display (the default is ground level), zoom to or highlight your current location, and to choose from one of eight map projections, including the Equirectangular Projection shown below. (Tip: Choose O to return to the standard othographic globe.)

Check out the winds blowing into Seattle, which I can attest are very real. The animation is based on supercomputer forecasts from NOAA weather data and updated every three hours. You can read more about the animation here, and the D3 source code is available on GitHub.

That's all for this week. (But if you want more, check out our archive of previous Friday posts.) Have a great weekend!

Big Data Innovation Powered by Big Privacy

In a survey we published in September, 80% of statisticians agreed that there should be an ethical framework in place for collecting and using data. In today's guest post, Dr. Ann Cavoukian, the Information and Privacy Commissioner of Ontario, Canada, describes Privacy by Design: an international standard for privacy protection.

by Dr. Ann Cavoukian

Big Privacy is Privacy by Design writ large. It is the application of the 7 Foundational Principles of Privacy by Design, not only to individual organizations, applications, or contexts, but to entire networks, value chains, and ecosystems, especially those that produce and use Big Data — without diminishing system functionality. Big Privacy empowers the individual and enables continuous Big Data innovations!

The goal of Big Privacy is the systemic protection of personal data and user control by data subjects over how it is collected and used. User control is an embodiment of “informational self-determination”— a right enshrined in the German Constitution, relating to an individual’s ability to determine the fate of his or her personal information.

While on the surface, Big Data and privacy may appear to be mutually exclusive, this represents a dated, zero-sum paradigm where it is believed that unnecessary trade-offs are made, between what turn out to be, false dichotomies, such as privacy vs. innovation. We must change the paradigm to a positive-sum model in which privacy and innovation become a doubly-enabling, “win-win” proposition for both consumers and businesses! This means that it is possible to assure individuals that their privacy is being respected because the network, value chain, and/or ecosystem producing and processing Big Data has implemented Privacy by Design, at a system-wide level. Big Data and Big Privacy may both be achieved in tandem.

Privacy by Design has been adopted as an international standard for privacy protection in 2010 and has been translated into 33 languages – becoming a true global venture. The Federal Trade Commission recommended PbD as a governing principle for all companies who collect personal data. I encourage you to take a look at the extensive work that my office has done in the field of Big Data and privacy: “Privacy by Design in the Age of Big Data and Big Privacy,”  “Big Privacy: Bridging Big Data and the Personal Data Ecosystem Through Privacy by Design”  “Privacy by Design in the Age of Big Data”.

By embedding Privacy by Design into new technical specifications, architectures, and systems, you will avoid the unintended consequences of increased data collection, and worst case scenario, Privacy by Disaster!