Revolutions

Data Science Virtual Machine updated, now includes RStudio, JuliaPro

The Windows edition of the Data Science Virtual Machine (DSVM) was recently updated on the Azure Marketplace. This update upgrades some existing components and adds some new ones as well.

You now have your choice of integrated development environment to use with R. RStudio Desktop is now included in the Data Science Virtual Machine image — no need to install it manually. And R Tools for Visual Studio has been upgraded to Version 0.5.

Microsoft R Server has also been upgraded to version 9.0. This includes the latest R 3.2.2 language engine, the RevoScaleR package for big-data support in R, and also the new MicrosoftML package with several new, high-performance machine learning techniques. If you choose a GPU-enabled NC classs instance for your DSVM, the MicrosoftML package can make use of the GPUs for even more performance. 

The DSVM also supports the Julia language with the inclusion of the JuliaPro distribution. This includes the Julia compiler and popular packages, a debugger, and the Juno IDE. If you're new to Julia, the blog post Julia – A Fresh Approach to Numerical Computing provides an introduction. 

There are also improved Deep Learning capabilities, with the latest version of the Microsoft Cognitive Toolkit (formerly called CNTK). And if you use a GPU-enabled instance, the Deep Learning Toolkit extension provides GPU-enabled builds of the Cognitive Toolkit, mxNet, and TensorFlow. 

If you want to try out the Data Science Virtual Machine, the blog post linked below provides links to the documentation and several tutorials to get you started, along with information about the Linux edition of the DSVM.

Cortana Intelligence and Machine Learning Blog: New Year & New Updates to the Windows Data Science Virtual Machine

 

List of R conferences and user groups

For 8 years now, we've maintained a list of local R user groups here at the Revolutions blog. This is a list that began with a single group (the Bay Area RUG, the first and still one of the largest groups), and now includes 360 user groups worldwide (including 27 specifically for women).

As the list has grown in size, it's become harder to manage. Thankfully, Colin Gillespie of Jumping Rivers Consulting has risen to the task, by creating a new website based on a GitHub repository that anyone can contribute to. I've updaed the Local R User Group Directory to point to these new pages, specifically the lists of:

• R User Groups
• R-Ladies Groups
• R Conferences and Meetings

If you have a group of your own, contributing to the list is easy. All you need is a GitHub account, and you can click the Edit button to edit one of the R Markdown pages directly. If you're not familiar with R Markdown, you can also suggest an edit via the Issues page.

(Incidentally, It would be great to automate the process of generating a count and a map of local R user groups. If anyone wants to take up the challenge of writing an R script to process the Rmd pages, please do!)

As R grows in popularity, it's awesome to see local communities get together and form these groups. If you'd like to start one yourself, here are some tips on starting up a local user group.

GitHub: A list of R conferences and meetings

 

Because it's Friday: Tangram Earth

If you rearrange the continents of the Earth like a planet-sized tangram, you can make some interesting shapes. For example, this rooster (via Max Galka) ...

... or this hog resembling the Arizona Diamondbacks logo (and displayed at the University of Arkansas, via Auriel Fournier) ...

... or as they appeared 300 million years ago.

That's all from us for this week. We'll be back on Monday with more blogging — see you then!

CRAN now has 10,000 R packages. Here's how to find the ones you need.

CRAN, the global repository of open-source packages that extend the capabiltiies of R, reached a milestone today. There are now more than 10,000 R packages available for download*.

(Incidentally, that count doesn't even include all the R packages out there. There are also another 1294 packages for genomic analysis in the BioConductor repository, hundreds of R packages published only on GitHub, commercial R packages from vendors such as Microsoft and Oracle, and an unknowable number of private, unpublished packages.)

Why so many packages? R has a very active developer community, who contribute new packages to CRAN on a daily basis. As a result, R is unparalleled in its capabilities for statistical computing, data science, and data visualization: almost anything you might care to do with data has likely already been implemented for you and released as an R package. There are several reasons behind this:

• R is the most popular language for data scientists — and it's been around for almost 20 years — and so by sheer force of numbers and time, R has more extensions than any other data science software.
• R is the primary tool used for statistical research: when new methods are developed, they're not just published as a paper — they're also published as an R package. That means R is always at the cutting edge of new methodologies.
• R was designed as an interface language — a means to present a consistent language interface for algorithms written in other languages. Many packages work by providing R language bindings to other open-source software, making R a convenient hub for all kinds of algorithms and methods.
• Last but certainly now least, the CRAN system itself is a very effective platform for sharing R extensions, with a mature system for package authoring, building, testing, and distribution. The R core team and in particular the CRAN maintainers deserve significant credit for creating such a vibrant ecosystem for R packages. 

Having so many packages available can be a double-edged sword though: it can take some searching to find the package you need. Luckily, there are some resources available to help you:

• MRAN (the Microsoft R Application Network) provides a search tool for R packages on CRAN. 
• To find the most popular packages, Rdocumentation.org provides a leaderboard of packages by number of downloads. It also provides lists of newly-released and recently-updated packages.
• CRAN provides package Task Views, providing a directory of packages by topic area (such as Finance or Clinical Trials). MRAN and RDocumentation.org also provide searchable versions based on the CRAN task views.
• To find popular and active R packages on GitHub, see the Trending R repositories list.
• For curated news on updated and new R packages, check out the Package Picks by Joseph Rickert on RStudio's RViews blog, and also the Package Spotlights published with each Microsoft R Open release. Cranberries also provides a comprehensive uncurated feed of new and updated packages.

The rate of R package growth shows no signs of abating, either. As you can see from this chart (created using this script by Gergely Daróczi), the growth in R packages shows no signs of plateauing soon. (This chart includes packages released and subsequently withdrawn from CRAN, which is why it goes over 10,000.)

Know of any other resources for exploring R packages? Let us know in the comments.

*Actually, as of this writing, there are 9,999 packages on CRAN. So close! But it won't be long...

Kung Fu R

A great way to hone your skills as a data scientist is to pick a topic you're passionate about, find some data related to it, and analyze the heck out of it. Jim Vallandingham is clearly passionate about old Kung Fu movies — particularly those from the Shaw Brothers Studio — and has used R to analyze data the studio's oeuvre: 260 films over 22 years.

The complete R code behind the analysis is included in the post (and you can also find it as an R Markdown document here). Some interesting notes include:

• Use of the tidyjson package to parse the table scraped from the list of Shaw Brothers Martial Arts Films on Letterboxd.
• Many applications of the ggplot2 package to visualize directors, actors, film length, Letterboxd watches and likes, and other data about the films.
• Using the tidytext package to find the common words used in film titles.
• Using the igraph package to create a network digram showing actors who appear in films together, shown below.

You can even iteract with this network, using Jim's Shaw Brothers Actors Network Visualization. The interactive version is based on Javascript and D3, and this tutorial explains how it was made. But for the complete analysis of the Kung Fu movies, follow the link below.

Jim Vallandingham: A Data Driven Exploration of Kung Fu Films

New Zealand bank replaces SAS server with R Server

Heartland Bank, a rapidly growing bank in New Zealand, has adopted a data-driven approach to analyzing risk, evaluating credit lines, and understanding cash flows. But they found their legacy SAS system to be labor-intensive and time consuming when it came to updating financial models, and it was expensive to boot. (Being licensed on a per-user basis, it was available only to a small group staff in IT.) The bank wanted an analytics platform that could support future innovation, and so Heartland Bank replaced SAS with Microsoft R Server and SQL Server.

It's fitting that a bank in New Zealand (the birthplace of R) should adopt R as its analytics platform. Heartland has moved its credit scorecard development, arrears analysis, investment forecasting, and analysis of intermediary and broker performance to the R Server platform. And it's already sparked a new attitute about data within the company. Now, business users across the bank are working with the new data models, instead of relying on IT to produce reports. And rather than waiting for overnight batch processing, employees have access to real-time information.

New Zealand bank leads the way with innovative analytics platform based on Microsoft R Server

“Before, a request would be like ‘Please get me the email addresses for people over 25 who live in the bottom half of the country.’ Now, we’re getting questions such as ‘Can you tell me what would happen to our deposit rates if we increased our six-month deposit rate by 20 basis points over the next three months?’” — Chris Murray, Head of Enterprise Data, Heartland Bank

For more on how Heartland Bank uses R, check out the complete story below.

Microsoft Customer Stories: Heartland Bank

Building a machine learning model with the MicrosoftML package

Microsoft R Server 9 includes a new R package for machine learning: MicrosoftML. (So do the Data Science Virtual Machine and the free Microsoft R Client edition, incidentally.) This package includes a suite of fast predictive modeling functions implemented by Microsoft Research, including:

• Linear (rxFastLinear) and logistic (rxLogisticRegression) model functions based on the Stochastic Dual Coordinate Ascent method;
• Classification/regression trees (rxFastTrees) and random forests (rxFastForests) based on FastRank, an efficient implementation of the MART gradient boosting algorithm;
• A neural network algorithm (rxNeuralNet) with support for custom, multilayer network topologies; and
• One-class anomaly detection (rxOneClassSvm) based on support vector machines.

As the function names suggest, the implementations are tuned for speed: most use multiple CPUs, and some will even use the GPU (if available). Not all of the implementations scale to unlimited data sizes, however; all but the linear and logistic regression routines are bound by available RAM.

If you want to give these routines a try, the MIcrosoft R Server Tiger Team has prepared a walkthrough analyzing the famous NYC Taxi data set. Once you have access to Microsoft R Server (or Client), this R script walks you through the process of:

• Loading the MicrosoftML package
• Importing the NYC Taxi Data from SQL Server (it comes preinstalled on the Data Science Virtual Machine)
• Splitting the data into a test set and a training set, with the binary value "tipped" (whether or not the driver was tipped) as the response
• Fitting several predictive models: logistic regression, linear model,, fast forest, and neural network.
• Making predictions on the test data
• Evaluating model performance by comparing AUC (area under the ROC curve)

The ROC curves are shown below. As you'd expect the linear model performs poorly compared to the others, since it's being applied here to a binary variable.

To try it out yourself, follow the walkthrough linked below, which also provides instructions for running the logistic regression model in SQL Server Management Studio.

Microsoft R Server Tiger Team: Predicting NYC Taxi Tips using MicrosoftML

Upcoming R Conferences

Since a few new events have been announced recently, I thought I'd give a run-down on some major R conferences coming up in the next six months.

• February 18: satRdays, Cape Town (South Africa). This is the second in a series of one-day conferences inspired by an R Consortium proposal. The first event in Budapest was a great success, and the lineup (including an all-women keynote program) looks equally awesome this time around. I'm sad I can't make this one myself!
• April 21-22: New York City R Conference (USA). New York is a hub for R developers, and people come from all over to hear what some of the major contributors have to say. This conference also has a diverse keynote program, and includes a renowned Python developer(!).
• May 19-20: R in Finance, Chicago (USA). Now in its 9th year, this conference brings together a fantastic community of R users working in the finance industry. The call for papers is open through February 28.
• May 25-26: ROpenSci Unconference, Los Angeles (USA). This is a great gathering focused on use of R for open science. 
• June 5-7: EARL, San Francisco (USA). The Effective Applications of the R Language conference adds a new US venue, and like all EARL conferences will feature real-world applications of R. The call for papers is open for this event and also the London conference to be held September 12-14.
• June 28-30: Sixième Rencontres R, Anglet (France). The annual R conference in France, now in its sixth year.
• July 4-7: useR!2017, Brussels (Belgium). The big one! The annual worldwide R conference attracts hundreds of R users from around the globe with dozens of user-contributed talks and a fascinating keynote program. Talk proposals are being accepted through April 1.

Update Jan 24: And here are a few other conferences suggested by readers:

• May 25-25: R à Québec 2017, Quebec City (Canada). The first French-speaking conference focused on R in North America. 
• June 8: R in Insurance, Paris (France). For academics and practitioners using R as a primary tool for insurance risk management, analysis and modeling.
• July 26-28: BioC 2017, Boston MA (USA). Annual conference for developers and users of the BioConductor suite of packages for analysis of genomic data.
• Nov 6-7: Use of R in Official Statistics, Bucharest (Romania) 

Know of any other major R conferences coming up? Let us know in the comments.

Because it's Friday: Infrastructure Collapses

On November 7 1940, the Tacoma Narrows Bridge, opened just four months prior, suffered catastrophic collapse in a windstorrm. (The music in the video is annoying, so here's a version with an alternate soundtrack.)

 

The story behind the collapse is interesting: while it looks like a build-up of resonance is the culprit, it was actually the fluttering of the bridge desk that brought it down.

That's all from us for this week — we'll be back on Monday.

Free guide to text mining with R

Jilia Silge and David Robinson are both dab hands at using R to analyze text, from tracking the happiness (or otherwise) of Jane Austen characters, to identifying whether Trump's tweets came from him or a staffer. If you too would like to be able to make statistical sense of masses of (possibly messy) text data, check out their book Tidy Tidy Text Mining with R, available free online and soon to be published by O'Reilly.

The book builds on the tidytext package (to which Gabriela De Queiroz also contributed) and describes how to handle and analyze text data. The "tidy text" of the title refers to a standardized way of handling text data, as a simple table with one term per row (where a "term" may be a word, collection of words, or sentence, depending on the application).  Julia gave several examples of tidy text in her recent talk at the RStudio conference:

Once you have text data in this "tidy" format, you can apply a vast range of statistical tools to it, by assigning data values to the terms. For example, can use sentiment analysis tools to quantify terms by their emotional content, and analyze that. You can compare rates of term usage, such as between chapters or to compare authors, or simply create a word cloud of terms used. You coyld use topic modeling techniques, to classify a collection of documents into like kinds. 

There are a wealth of data sources you can use to apply these techniques: documents, emails, text messages ... anything with human-readable text. The book includes examples of analyzing works of literature (check out the janesustenr and guternbergr packages), downloading Tweets and Usenet posts, and even shows how to use metadata (in this case, from NASA) as the subject of a text analysis. But it's just as likely you have data of your own to try tidy text mining with, so check out Tidy Text Mining with R and to get started.