Revolutions

Revolution R Enterprise Now Available in the Cloud on Azure Marketplace

by Richard Kittler, Revolution R Enterprise PM, Microsoft Advanced Analytics

Revolution is excited to announce the availability of its latest release of Revolution R Enterprise 7.4.1 (RRE) as a technical preview on Microsoft Azure via Windows- and Linux-based virtual machines in the Azure Marketplace.  Through Azure’s world-wide cloud infrastructure customers now have on-demand access to high-performance predictive analytics to accelerate growth, optimize operations, and expedite data insight and discovery from any place and at any time.   Availability in Azure Marketplace is the first step in Microsoft’s plan to integrate Revolution’s products with the Azure and, in the bigger picture, Cortana Analytics.

Through the Azure Marketplace, users can run computations on data sets up to 1 terabyte on cloud-based Windows and Linux multi-CPU instances from 4 to 32 vCPUs (virtual CPUs), accessing data copied from an Azure data store, including blob storage or SQL Server, or accessed directly through an ODBC connection. Utility pricing for both versions starts at $1.50 per 4-cores per hour with no long-term commitments. During this preview period RRE users will be supported through the monitored Azure Machine Learning forum.

The Windows version of RRE is accessed with Windows Remote Desktop and includes Revolution’s Visual Studio-based IDE for R developers (Revolution R Enterprise DevelopR).  Access to the Linux version of RRE is via SSH with R access through the RGui.  Alternatively, customers have the option of bringing their own license for a favorite IDE such as RStudio, RStudio Server, or StatET. Both the Windows and Linux offerings include Revolution’s DeployR web services module.

RRE features a unique “write once deploy anywhere” functionality that enables data analysts and IT teams to write code once and deploy it anywhere in a variety of data management platforms, enterprise data warehouses, grids, clusters, servers and workstations without re-engineering costs. It is the only Big Data Big Analytics platform to include a library of Big Data-ready algorithms that run inside the Cloudera, Hortonworks, and MapR Hadoop platforms, Teradata databases, and soon SQL Server, with the highest possible performance.

A 30-day free trial is available with no software charge, though standard Azure cloud usage charges apply. To learn more about RRE on Azure Marketplace, please visit the Marketplace posting here. 

[Update: If you are new to Azure then you also have the benefit of a 30-day Azure free trial which could cover the cost of the underlying VM during your RRE free trial!  Lastly, if you are new to RRE then you may wish to take a look at this free intro to RRE course on DataCamp.]   

Because it's Friday: Downtown Spokane

There hasn't really been a break-out Summer hit this season, but Macklemore's latest — an ode to mopeds, of all things — might just be it (some language NSFW):

 

Loving the 70's styling and dance moves. And despite the Seattle-focused lyrics, Spokane makes a great backdrop for the music video.

That's all for this week. See you back here on Monday!

Get to know Cortana Analytics: Workshop and webinars

Cortana Analytics Suite is Microsoft's cloud-based big data and advanced analytics suite. It includes a complete set of all the services need to build advanced analytics applications: from data ingestion and management, data warehousing, advanced analytics, data visualization and solution frameworks. You can use Cortana Analytics to build applications using R, by incorporating services including Data Factory, HDInsights Hadoop, and ML Studio.

If you'd like to spend some quality time with other developers and the Microsoft development team, there will be a first-ever Cortana Analytics Workshop at Microsoft HQ in Redmond, September 10-11. Or if you just want to check it out online, you can watch this introductory webinar, or join this upcoming application-focused webinar on September 1:

Leveraging Predictive Analytics for Sales and Marketing

In this session, learn how the Microsoft Global Marketing team built predictive analytics solutions to meet the marketing and sales business need for Microsoft subsidiaries and business groups. The session will describe how Cortana Analytics components (Azure Machine Learning, Azure Data Factory) are used to build e2e solution in Cloud.

Sign up for this free webinar here, or follow the link below to explore Cortana Analytics in depth.

Microsoft: Cortana Analytics Suite

Plotting Time Series in R using Yahoo Finance data

by Joseph Rickert

I recently rediscovered the Timely Portfolio post on R Financial Time Series Plotting. If you are not familiar with this gem, it is well-worth the time to stop and have a look at it now. Not only does it contain some useful examples of time series plots mixing different combinations of time series packages (ts, zoo, xts) with multiple plotting systems (base R, lattice, etc.) but it provides an instructive, historical perspective that illustrates the non linear nature of progress in software development: new code is written to solve certain technical problems with the current software. Progress is made, and the new code makes it possible to do some things that couldn't be done before, but there were tradeoffs. Design choices for the new system make it a little more difficult to do something that was easy before. The net result: all of the software continues to advance in a messy mix, confusing the newcomer and providing critics with the opportunity to complain that there is not just one way to solve a problem.

Because this is turning out to be a week when more than a few people are likely lo be plotting financial time series, I thought I would be helpful to call attention to this time series resource and also take a look at the current state of the R art for performing a relatively simple task: plotting closing prices for two stocks on the same chart.

The following code just reads stock price data from Yahoo Finance for both IBM and LinkedIn from 8/24/2010 through 8/24/2015 and picks out the closing prices. I cheated a little here because I already knew the urls for the two series. I picked these two stocks because they both traded at about the same range for the period in question and because I wanted to see if the fact that one stock, LinkedIn, wasn't trading when at the beginning of the selected period caused any problems.

# Time Series Plotting
library(ggplot2)
library(xts)
library(dygraphs)
 
# Get IBM and Linkedin stock data from Yahoo Finance
ibm_url <- "http://real-chart.finance.yahoo.com/table.csv?s=IBM&a=07&b=24&c=2010&d=07&e=24&f=2015&g=d&ignore=.csv"
lnkd_url <- "http://real-chart.finance.yahoo.com/table.csv?s=LNKD&a=07&b=24&c=2010&d=07&e=24&f=2015&g=d&ignore=.csv"
 
yahoo.read <- function(url){
   dat <- read.table(url,header=TRUE,sep=",")
   df <- dat[,c(1,5)]
   df$Date <- as.Date(as.character(df$Date))
   return(df)}
 
ibm  <- yahoo.read(ibm_url)
lnkd <- yahoo.read(lnkd_url)

To my mind, the "go to" method for simple plotting that you will show to someone else is ggplot(). The following code suggested by Didzis Elferts, in answer to a StackOverflow question, accomplishes the task with great economy, using just a few more features than what the defaults would give you.

ggplot(ibm,aes(Date,Close)) + 
  geom_line(aes(color="ibm")) +
  geom_line(data=lnkd,aes(color="lnkd")) +
  labs(color="Legend") +
  scale_colour_manual("", breaks = c("ibm", "lnkd"),
                          values = c("blue", "brown")) +
  ggtitle("Closing Stock Prices: IBM & Linkedin") + 
  theme(plot.title = element_text(lineheight=.7, face="bold"))

 

This next plot, which uses the dygraphs package, represents the new frontier for creating interactive time series plots in R.

# Plot with the htmlwidget dygraphs
# dygraph() needs xts time series objects
ibm_xts <- xts(ibm$Close,order.by=ibm$Date,frequency=365)
lnkd_xts <- xts(lnkd$Close,order.by=lnkd$Date,frequency=365)
 
stocks <- cbind(ibm_xts,lnkd_xts)
 
dygraph(stocks,ylab="Close", 
        main="IBM and Linkedin Closing Stock Prices") %>%
  dySeries("..1",label="IBM") %>%
  dySeries("..2",label="LNKD") %>%
  dyOptions(colors = c("blue","brown")) %>%
  dyRangeSelector()

Building on the work done for rCharts profiled in the Timely Portfolio piece, the dygraphs R package provides an interface to the dygraphs javascript library. With just a few lines of R code, it is now possible to produce charts that approach the polished look of the professional stock charting services - and no knowledge of JavaScript.

 

 

New R titles available in Chinese

by Andrie de Vries

A few days ago I received an unexpected parcel in my letterbox. To my delight, it turned out to be a translation into Simplified Chinese of R for Dummies, co-authored by myself and Joris Meys. Let me clarify: Joris and I wrote the book, but were not involved in the translation at all. The Wiley publishing machine took care of all of that.

Just to prove that this is real, here is a picture of my copy, unfortunately slightly damaged during the delivery process:

I find the decision by Wiley to translate into Chinese very interesting, since it means there is a growing market for translations of R books. Translating and publishing a title is a lot of work, so they will only do this if the business case is convincing. In other words, there must be a large and growing market for this type of book in Chinese.

This made me wonder, which other titles are available in Chinese.  My basic research to answer this question consisted of navigating to the Chinese site of Amazon, ie. amazon.cn.  From here I learn "What Other Items Do Customers Buy After Viewing This Item?".  It seems they go on and purchase some other classics, all available in Chinese:

 

ggplot2, by Hadley Wickham

 

Data Manipulation with R, by Phil Spector

 

R in Action, by Rob Kabakoff

Introductory statistics with R, by Peter Dalgaard

 

Following up on news stories with choroplethr and R

by Ari Lamstein, consultant specializing in software engineering and data analysis and author of the free email course Learn to Map Census Data in R.

One of my favorite things about R is that it allows me to follow up on interesting news stories. Consider this interview on EconTalk about the history of fracking in America. Russ Roberts interviewed Gregory Zuckerman about his book The Frackers. One thing that struck me were the stories of how North Dakota is being transformed by the fracking boom. North Dakota sits on the Bakken formation which, due to fracking, is now able to be monetized.

Here are two maps I made which demonstrate North Dakota’s recent demographic changes. The first shows that between 2010 and 2013 North Dakota’s Per Capita Income grew at a rate of 15%, significantly above any other US state. The second one shows that North Dakota’s Median Age decreased by 2%, significantly below any other US state. Today I will demonstrate how to create these maps in R.

                       

Step 1: Install and Load the Necessary Packages

We’ll use the choroplethr and choroplethrMaps packages to create these maps. To install and load them, type the following from an R console:

install.packages(c(“choroplethr”, “choroplethrMaps”))
library(choroplethr)
library(choroplethrMaps)

 

How R is used at Zillow to estimate housing values

Zillow, the leading real estate and rental marketplace in the USA, uses R to estimate housing values. Zillow's signature product is the Zestimate, their estimated market value for individual homes, and it's calculated using R in a parallel batch job for 100 million homes nationwide. The process is described in this Slideshare presentation Data Science At Zillow by Nicholas McClure, senior data scientist at Zillow.

This article by Alex Woodie at Datanami provides more detail about the process. R is used in conjunction with other data science tools:

The Zestimate is generated through a series of processes built using various tools, including heavy doses of R, Python, Pandas, Scikit Learn, and GraphLab Create, the graph analytics software developed by Seattle-based Dato (formerly GraphLab).

The company makes extensive use of R, including the development of a proprietary software package called ZPL that functions similar to MapReduce on Hadoop, but runs on a relational database. The company is increasing its use of Python, which Zillow data scientists say is better than R for some things, such as conducting GIS analysis.

You can find more details on Zillow's data science systems at the link below.

Datanami: Inside the Zestimate: Data Science at Zillow

Because it's Friday: Back to the Past

Photographing an object at regular intervals over a long time period, and then stitching the results into a timelapse video, is a great way to "speed up time" and observe slow-motion changes (for example, the growth of plants). But creating a timelapse video is hard: you have to set up a camera, make sure it can't move, and take regular shots while maintaining consistent lighting from shot to shot. The longer the observation period, the harder it is: making timelapses over a period of years is just about impossible. But by far the most difficult thing about timelapses is anticipating the change in the first place: we'd love to be able to go back in time and observe the retreat of a glacier over a period of years, but what if we didn't think there'd be enough change to observe in the first place? There's no way to go back in time and create a timelapse after the change has happened.

Or is there? A team of researchers at the University of Washington and Google has figured out a way to mine tourist photos and recombine them into a timelapse video. Images are located via GPS tags, and then image detection algorithms rescale and crop those photos into a smooth sequence. In the video below, you can see how tourist photos of a Norwegian glacier show its retreat over a 10-year period.

 

You can read more about the process used to create the videos in this Wired article and at the project website.

That's all for this week. See you on Monday!

Free edX course for R beginners

If you've thought about learning the R language but didn't know how to start, there's a new, free course on edX that starts you from the R basics and lets you learn R by trying R as you go.

 

 

Presented by DataCamp and Microsoft, the course starts from the very basics of R (arithmetic on the command line, creating variables), progresses through the basic data types (vector, matrix, factor, list and data frame) and ends with a module on data visualization. The course consists of lecture-style videos interspersed with quizzes to test your knowledge. And best of all, you can try out what you've learned at the R command line using the DataCamp online interface -- so you don't even have to install R yourself! A browser is all you need.

It's perfect for newcomers to R, even if you don't have experience in other programming languages. If you want to get a (paid) certification you'll need to complete the course by September 1, but you can view all of the course materials, quizzes and labs anytime for free. You can learn more about the course at the DataCamp blog or get started now at the link below.

edX: Microsoft: DAT204x Introduction to R

5 New R Packages for Data Scientists

by Joseph Rickert

 

One great beauty of the R ecosystem, and perhaps the primary reason for R’s phenomenal growth, is the system for contributing new packages. This, coupled to the rock solid stability of CRAN, R’s primary package repository, gives R a great advantage. However, anyone with enough technical knowhow to formulate a proper submission can contribute a package to CRAN. Just being on CRAN is no great indicator of merit: a fact that newcomers to R, and open source, often find troubling. It takes some time and effort working with R in a disciplined way to appreciate how the organic metracracy of the package system leads to high quality, integrated software. Nevertheless, even for relative newcomers it is not difficult to discover the bedrock packages that support the growth of the R language. Those packages which reliably add value to the R language are readily apparent in plots of CRAN’s package dependency network.

 

Finding new packages that may ultimately prove to be useful is another matter. In the spirit of discovery; here are 5, relatively new packages that I think may ultimately prove to be interesting to data scientists. None of these have been on CRAN long enough to be battle tested. So please, explore them with cooperation in mind.

 

AzureML V0.1.1

Cloud computing is, or will be, important to every practicing data scientist. Microsoft’s Azure ML is a particularly rich machine learning environment for R (and Python) programmers. If your are not yet an Azure user this new package goes a long way to overcoming the inertia involved in getting started. It provides functions to push R code from your local environment up to the Azure cloud and publish functions and models as web services. The vignette walks you step by step from getting a trial account and the necessary credentials to publishing your first simple examples.

 

distcomp V0.25.1

Distributed computing with large data sets is always tricky, especially in environments where it is difficult or impossible to share data among collaborators. A clever partial likelihood algorithm implemented in the distcomp package (See the paper by Narasimham et al.) makes it possible to build sophisticated statistical models on unaggregated data sets. Have a look at this previous blog post for more detail.

 

rotationForest V0.1

The forests algorithm is the “go to” ensemble method for many data scientists as it consistently performs well on diverse data sets. This new variation based on performing Principal Component Analysis on random subsets of the feature space shows great promise. See the paper by Rodriguez et. al. for an explanation of how the PCA amounts to rotating the feature space and a comparison of the rotation forest algorithm with standard random forests and the Adaboost algorithm.

 

rpca V0.2.3

Given a matrix that is a superposition of a low rank component and a sparse component, rcpa uses a robust PCA method to recover these components.  Netflix data scientists publicized this algorithm, which is based on a paper by Candes et al,  Robust Principal Component Analysis, earlier this year when they reported spectacular success using robust PCA in an outlier detection problem.

 

SwarmSVM V0.1

The support vector machine is also a mainstay machine learning algorithm. SwarmSVM, which is based on a clustering approach as described in a  paper by Gu and Han provides three ensemble methods for training support vector machines. The vignette that accompanies the package provides a practical introduction to the method.