Revolutions

How to use H2O with R on HDInsight

H2O.ai is an open-source AI platform that provides a number of machine-learning algorithms that run on the Spark distributed computing framework. Azure HDInsight is Microsoft's fully-managed Apache Hadoop platform in the cloud, which makes it easy to spin up and manage Azure clusters of any size. It's also easy to to run H2O on HDInsight: H2O AI Platform is available as an application on HDInsight, which pre-installs everything you need as the cluster is created.

You can also drive H2O from R, but the R packages don't come auto-installed on HDInsight. To make this easy, the Azure HDInsight team has provided a couple of scripts that will install the necessary components on the cluster for you. These include RStudio (to provide an R IDE on the cluster) and the rsparkling package. With these components installed, from R you can:

• Query data in Spark using the dplyr interface, and add new columns to existing data sets.
• Convert data for training, validation, and testing to "H2O Frames" in preparation for modeling.
• Apply any of the machine learning models provided by Sparkling Water to your data, using the distributed computing capabilities provided by the HDInsight platform.

For details on how to install the R components on HDInsight, follow the link below.

Azure Data Lake & Azure HDInsight Blog: Run H2O.ai in R on Azure HDInsight

 

Because it's Friday: Rolling shutters, explained

I've written about the shutter effect a couple of times here on the blog, but this video from SmarterEveryDay provides the best explanation I've seen yet of why digital cameras can distort fast-moving objects in interesting ways. (try filming a fidget spinner with your smartphone to see the effect.) The video, three years in the making, uses high-speed cameras to simulate the rolling shutter effect as shown in the same footage recorded by an iPhone.

That's all from us here at the blog for this week. We'll be back on Monday — see you then!

Learn parallel programming in R with these exercises for "foreach"

The foreach package provides a simple looping construct for R: the foreach function, which you may be familiar with from other languages like Javascript or C#. It's basically a function-based version of a "for" loop. But what makes foreach useful isn't iteration: it's the way it makes it easy to run those iterations in parallel, and save time on multi-CPU and distributed systems.

If you want to get familiar with the foreach function, Parallelizing Loops at Microsoft Docs will introduce you to foreach loops (and the companion concept, iterators), and the various "backends" you can use to make the loops run in parallel without changing your loop code. Then, to make sure you've captured the concepts, you can try these 10 parallel computing exercises using foreach, from R-exercises.com. If you get stuck, the solutions are available here.

To get started with foreach, can install the foreach package from CRAN, or simply use any edition of Microsoft R including Microsoft R Open (all come with foreach preinstalled). The foreach package is maintained by Microsoft under the open-source Apache license. 

R-exercises: Parallel Computing Exercises: Foreach and DoParallel (Part-2)

The R6 Class System

R is an object-oriented language with several object-orientation systems. There's the original (and still widely-used) S3 class system based on the "class" attribute. There's the somewhat stricter, signature-based S4 class system. There are reference classes (also called R5), which provide R objects with multiple references without duplicating data in memory. And now there's the R6 class system, implemented as an R package by Winston Chang,

So why yet another class system for R? As Winston explains in his useR!2017 talk embedded below, the S3 and S4 class systems are functional object-oriented systems, where class methods are separate from objects, and objects are not mutable (that is, they can't be modified directly by methods; you have to assign modified values to the object directly). The R6 system, by contrast, is an encapsulated object oriented system akin to those in Java or C++, where objects contain methods in addition to data, and those methods can modify objects directly. 

The big advantage of R6 is that it makes it much easier to implement some common data structures in a user-friendly manner. For example, to implement a stack "pop" operation in S3 or S4 you have to do something like this:

x <- topval(mystack)
mystack <- remove_top(mystack)

In R6, the implementation is much simpler to use:

x <- mystack$pop()

That single statement both returns the top value (as x) from the stack mystack, but also modifies the stack in the process (minus the popped element).

Despite being first released over 3 years ago, the R6 isn't widely known. It is widely used, however: it's used to manage session state in Shiny and mrsdeploy, to manage database connections in dplyr, and to represent system processes in the processx package. In fact, the R6 package is the most downloaded CRAN package, according according to Rdocumentation Trends.

The R6 package is available on CRAN, and it also comes pre-installed in Microsoft R. If you'd like to start using R6, the vignette Introduction to R6 Classes is a great place to start. You can also check out the development of R6 at the R6 Github repository.

Introducing Ridgeline Plots (formerly Joyplots)

UPDATE September 20: Joyplots are now known as Ridgeline Plots, and available in the ggridges package. Read the explanation here.

This is a joyplot: a series of histograms, density plots or time series for a number of data segments, all aligned to the same horizontal scale and presented with a slight overlap.

Peak time for sports and leisure #dataviz. About time for a joyplot; might do a write-up on them. #rstats code at https://t.co/Q2AgW068Wa pic.twitter.com/SVT6pkB2hB

— Henrik Lindberg (@hnrklndbrg) July 8, 2017

The name "Joy Plot" was apparently coined by Jenny Bryan in April 2017, in response to one of Lindberg's earlier visualizations using this style. (The community appears to have settled on 'joyplot' since then.) The name refers to the classic 1979 Joy Division "Unknown Pleasures" album cover, which was in actuality a joyplot of radio intensities from the first known pulsar. The album cover reproduced the design from a 1971 Scientific American article about pulsars.

Lindberg created the chart above using a simple R script and some ggplot2 calls, but now it's even easier to create joyplots in R thanks to the ggjoy package by Claus Wilke, now available on CRAN. The vignette provides instructions on how to use the package to create your own joyplots: you can either use the geom_ridgeline geom to plot individual lines (useful if your data is already in density or time series format), or use geom_joy to calculate densities from the samples . There's also a theme_joy theme which eliminates the traditional ggplot grid for a cleaner result more reminiscent of the classic 1971 joyplot. You can see several examples (with code) in this gallery, and I've also included a couple of other examples below.

Polarization of the US House of Representative members in the Democratic and Republican parties (based on code by Ian McDonald):

Asymmetric polarization: The entire GOP has moved to the right, where Dems have just become slightly less moderate. (H/t @ianrmcdonald) pic.twitter.com/IeIo959qRV

— G Elliott Morris📈🙂 (@gelliottmorris) July 22, 2017

Worker wages by industry:

The @justcapital_ worker wage data for this year's rankings is taking shape. #ESG #rstats Thank you @ClausWilke for the joyplot package pic.twitter.com/HBRuF7mIdq

— Hernando Cortina (@cortinah) July 25, 2017

As you can see from the examples above, joyplots work best when you want to compare the distributions of several subgroups, and the number of groups is large (so that a traditional panel plot would take up too much space). That said, joyplots do inevitably obscure some data due to the overlap, and you may want to consider alternatives like heat maps or even simple bar charts of means or medians. In any case, the new ggjoy package makes it easy to try them out and decide what works best for your data.

Github (clauswilke): ggjoy: Geoms to make joy plots using ggplot2

 

SQL Server 2017 release candidate now available

SQL Server 2017, the next major release of the SQL Server database, has been available as a community preview for around 8 months, but now the first full-featured release candidate is available for public preview. For those looking to do data science with data in SQL Server, there are a number of new features compared to SQL Server 2017 which might be of interest: 

• SQL Server 2017 will be supported on Linux, specifically on Red Hat, SUSE, and Ubuntu (as well as within a Docker container). You can read some of the backstory on how SQL Server came to Linux here.
• The new SQL Server Machine Learning Services will provide in-database analytics in both the R and — new to SQL Server 2017 — Python languages.
• There are several improvements to the in-database R integration (previously known as "R Services" in SQL Server 2016). These includes a new package management system, real-time in-database scoring of predictive models, and an updated R engine.

SQL Server 2017 Release Candidate 1 is available for download now. For more details on these and other new features in this release, check out the link below.

SQL Server Blog: First release candidate of SQL Server 2017 now available 

Analyzing Github pull requests with Neural Embeddings, in R

At the useR!2017 conference earlier this month, my colleague Ali Zaidi gave a presentation on using Neural Embeddings to analyze GitHub pull request comments (processed using the tidy text framework). The data analysis was done using R and distributed on Spark, and the resulting neural network trained using the Microsoft Cognitive Toolkit. You can see the slides here, and you can watch the presentation below. 

 

 

Because it's Friday: How Bitcoin works

Cryptocurrencies have been in the news quite a bit lately. Bitcoin prices have been soaring recently after the community narrowly avoided the need for a fork, while $32M in rival currency Etherium was recently stolen, thanks to a coding error in wallet application Purity. But what is a crypto-currency, and what does a "wallet" or a "fork" mean in that context? The video below gives the best explanation I've seen for how cryptocurrencies work. It's 25 minutes long, but it's a complex and surprisingly subtle topic, made easy to understand by math explainer channel 3Blue1Brown.

That's all from the blog for this week. Have a great weekend, and we'll be back on Monday.

IEEE Spectrum 2017 Top Programming Languages

IEEE Spectrum has published its fourth annual ranking of of top programming languages, and the R language is again featured in the Top 10. This year R ranks at #6, down a spot from its 2016 ranking (and with an IEEE score — derived from search, social media, and job listing trends — tied with the #5 place-getter, C#). Python has taken the #1 slot from C, jumping from its #3 ranking in 2016.

For R (a domain specific language for data science) to rank in the top 10, and for Python (a general-purpose language with many data science applications) to take the top spot, may seem like a surprise. I attribute this to continued broad demand for machine intelligence application development, driven by the growth of "big data" initiatives and the strategic imperative to capitalize on these data stores by companies wordwide. Other data-oriented languages appear in the Top 50 rankings, including Matlab (#15), SQL (#23), Julia (#31) and SAS (#37).

For the complete announcement of the 2017 IEEE Spectrum rankings, including additional commentary and analysis of changes, follow the link below.

IEEE Spectrum: The 2017 Top Programming Languages

 

 

Data Analysis for Life Sciences

Rafael Irizarry from the Harvard T.H. Chan School of Public Health has presented a number of courses on R and Biostatistics on EdX, and he recently also provided an index of all of the course modules as YouTube videos with supplemental materials. The EdX courses are linked below, which you can take for free, or simply follow the series of YouTube videos and materials provided in the index. 

Data Analysis for the Life Sciences Series 

• Statistics and R
• Introduction to Linear Models and Matrix Algebra
• Statistical Interference and Modeling for High-throughput Experiments
• High-Dimensional Data Analysis

A companion book and associated R Markdown documents are also available for download.

Genomics Data Analysis Series

• Introduction to Bioconductor: Annotation and Analysis of Genomes and Genomic Assays
• High-performance computing for reproducible genomics with Bioconductor
• Case Studies in Functional Genomics

For links to all of the course components, including videos and supplementary materials, follow the link below.

rafalab: HarvardX Biomedical Data Science Open Online Training