Revolutions

Video: R and Python in in Azure HDInsight

Azure HDInisght was recently updated with version 9.3 of ML Services in HDInsight, which provides integration with R and Python. In particular, it makes it possible to run R and Python within HDInsight's managed Spark instance. The integration provides:  

• R and Python support, with interaction via Visual Studio, VS Code, or RStudio
• Specialized distributed analytics libraries for R and Python
• Simple operationalization of R and Python with Spark
• Interoperability with Sparklyr, pyspark and H2O.ai

In the episode of Data Exposed embedded below, you can learn more about the ML Service 9.3 in HDInsight and see some demos of the R and Python integration.

Channel 9: Introducing ML Services 9.3 in Azure HDInsight

The Chartmaker Directory: Data visualizations in every tool

Working with a new data visualization tool, and wondering how to create a specific type of chart? The Chartmaker Directory (designed by Andy Kirk) indexes more than 35 tools and over 50 charts, and provides links to examples from each combination. Here's an intentionally small detail from the index, where each hollow dot in a column represents a sample chart created a tool (the rightmost column is R, for example), and solid dots also include code. 

The visualization tools include applications like Excel, Power BI and Tableau; languages and libraries including R, Stata, and Python's matplotlib); and frameworks like D3. The data visualizations range from the standard to the esoteric, and follow the taxonomy of the book Data Visualisation (also by Andy Kirk). The chart categories are color coded by row: categorical (including bar charts, dot plots); hierarchical (donut charts, treemaps); relational (scatterplots, sankey diagrams); temporal (line charts, stream graphs) and spatial (choropleths, cartograms). 

Follow the link below to explore the indexed tools and visualization. If you know of an example or solution that isn't listed, you can contribute your own links by clicking the plus sign in the top left of the index.

Visualizing Data: The Chartmaker Directory

Redmonk Language Rankings, June 2018

The latest Redmonk Language Rankings are out. These rankings are published twice a year, and the top three positions in the June 2018 rankings remain unchanged from last time: JavaScript at #1, Java at #2, and Python at #3. The Redmonk rankings are based on Github repositories (as a proxy for developer activity) and StackOverflow activity (as a proxy for user discussion), as shown in the chart below.

Compared to two quarters ago, R has dropped 2 places to #14. But as Stephen O'Grady notes, this ranking volatility is typical:

R is essentially a case study at this point for why we recommend against over-reacting to any particular quarter’s performance. R has dropped two spots previously, only to bounce back a quarter or two later. Its domain specific nature means that it is never likely to compete for the very top spots on this ranking, but it remains dominant and broadly used within its area of expertise.

You can see the complete rankings list along with discussion of the languages covered in at the Redmonk blog linked below.

Tecosystems: The RedMonk Programming Language Rankings: June 2018

IEEE Language Rankings 2018

Python retains its top spot in the fifth annual IEEE Spectrum top programming language rankings, and also gains a designation as an "embedded language". Data science language R remains the only domain-specific slot in the top 10 (where it as listed as an "enterprise language") and drops one place compared to its 2017 ranking to take the #7 spot.

Looking at other data-oriented languages, Matlab as at #11 (up 3 places), SQL is at #24 (down 1), Julia at #32 (down 1) and SAS at #40 (down 3). Click the screenshot below for an interactive version of the chart where you can also explore the top 50 rankings.

The IEEE Spectrum rankings are based on search, social media, and job listing trends, GitHub repositories, and mentions in journal articles. You can find details on the ranking methodology here, and discussion of the trends behind the 2018 rankings at the link below.

IEEE Spectrum: The 2018 Top Programming Languages

Video: How to run R and Python in SQL Server from a Jupyter notebook

Did you know that you can run R and Python code within a SQL Server instance? Not only does this give you a powerful server to process your data science calculations, but it makes things faster by eliminating the need to move data between client and server. Best of all, you can access this power via your favourite interface on your local machine, like the RStudio IDE for R or the VS Code Python extensions. 

To make this work, all you need is the necessary functions to launch computations in SQL Server, provided in the revoscaler package for R, and the revoscalepy library for Python. The video above demonstrates how to set things up for Python, and R users should check out the tutorials Learn in-database analytics using R in SQL Server.

 

New open data sets from Microsoft Research

Microsoft has released a number of data sets produced by Microsoft Research and made them available for download at Microsoft Research Open Data.  

The Datasets in Microsoft Research Open Data are categorized by their primary research area, such as Physics, Social Science, Environmental Science, and Information Science. Many of the data sets have not been previously available to the public, and many are large and useful for research in AI and Machine Learning techniques. Many of the datasets also include links to associated papers from Microsoft Research. For example, the 10Gb DESM Word Embeddings dataset provides the IN and the OUT word2vec embeddings for 2.7M words trained on a Bing query corpus of 600M+ queries.

Other data sets of note include:

• A collection of 38M tweets related to the 2012 US election
• 3-D capture data from individuals performing a variety of hand gestures
• Infer.NET, a framework for running Bayesian inference in graphical models
• Images for 1 million celebrities, and associated tags
• MS MARCO, is a new large-scale dataset for reading comprehension and question answering

Most data sets are provided as plain text files, suitable for importing into Python, R, or other analysis tools. In addition to downloading the data, you can also deploy the datasets for analysis in to Microsoft Azure: see the FAQ for details on this and other ways the datasets can be used.

As of this writing, the archive includes 51 datasets. You can explore and download the Microsoft Research datasets at the link below.

Microsoft: Microsoft Research Open Data

PYPL Language Rankings: Python ranks #1, R at #7 in popularity

The new PYPL Popularity of Programming Languages (June 2018) index ranks Python at #1 and R at #7.

Like the similar TIOBE language index, the PYPL index uses Google search activity to rank language popularity. PYPL, however, fcouses on people searching for tutorials in the respective languages as a proxy for popularity. By that measure, Python has always been more popular than R (as you'd expect from a more general-purpose language), but both have been growing at similar rates. The chart below includes the three data-oriented languages tracked by the index (and note the vertical scale is logarithmic).

Another language ranking was also released recently: the annual KDnuggets Analytics, Data Science and Machine Learning Poll. These rankings, however, are derived not from search trends but by self-selected poll respondents, which perhaps explains the presence of Rapidminer at the #2 spot.

Open-Source Machine Learning in Azure

The topic for my talk at the Microsoft Build conference yesterday was "Migrating Existing Open Source Machine Learning to Azure". The idea behind the talk was to show how you can take the open-source tools and workflows you already use for machine learning and data science, and easily transition them to the Azure cloud to take advantage of its capacity and scale. The theme for the talk was "no surprises", and other than the Azure-specific elements I tried to stick to standard OSS tools rather than Microsoft-specific things, to make the process as familiar as possible.

In the talk I covered:

• Using Visual Studio Code as a cross-platform, open-source editor and interface to Azure services
• Using the Azure CLI to script the deployment, functions, and deletion of resources in the Azure cloud
• Using the range of data science and machine learning tools included in the Data Science Virtual Machine on Ubuntu Linux
• Using Python and Tensorflow to train a deep neural network on a GPU cluster with Azure Batch AI
• Running sparklyr with RSTudio on a Spark cluster using aztk

It's not quite the same without the demos (you can view a recording here), but you may find the notes and references in the slides below useful.

 

Three Twitter Threads

I've been heads-down this week preparing for some upcoming talks, so not as much blogging as usual this week. But there have been some interesting conversations on Twitter this week that you may be interested to check out if you're not on the platform.

Steph Lock shares her go-to R packages for every stage of the data science process:

My #rstats #datascience goto 📦
IO: odbc readxl httr
EDA: DataExplorer
Prep: tidyverse
Sampling: rsample modelr
Feature Engineering: recipes
Modelling: glmnet h2o FFTrees
Evaluation: broom yardstick
Deployment: sqlrutils AzureML opencpu
Monitoring: flexdashboard
Docs: rmarkdown

— Steph Locke (@SteffLocke) April 28, 2018

Rachel Thomas and Jeremy Howard advocate thinking differently about AI development, and not falling into the trap of thinking "bigger is always better" when it comes to data (a sentiment I wholeheartedly agree with):

Innovation come from doings things differently, not doing things bigger. @jeremyphoward https://t.co/3TJYs8OCbr pic.twitter.com/I55a6gT1OF

— Rachel Thomas (@math_rachel) May 2, 2018

I wondered what was so different about Python compared to R when it comes to package management, and got some really thoughtful responses:

Serious question: I use R, not Python, and while there's the occasional version/package issue in #rstats it's rarely a big deal. But I hear about this from Python devs all the time. What's so different about Python that this is such a thing? https://t.co/g8ddQu2gpt

— David Smith (@revodavid) April 30, 2018

Twitter definitely has its bad side, but there's a lot of really interesting conversation on the platform as well. Click on each tweet to see the conversations these generated.

Use Python functions and modules in R with the "reticulate" package

Since its inception over 40 years ago, when S (R's predecessor) was just a sketch on John Chambers' wall at Bell Labs, R has always been a language for providing interfaces. I was reminded of this during Dirk Eddelbuettel's presentation at the Chicago R User Group meetup last night, where he enumerated Chambers' three principles behind its design (from his 2016 book, Extending R):

1. Object: Everything that exists in R is an object
2. Function: Everything that happens in R is a function call
3. Interface: Interfaces to other software are a part of R

The third principle "Interface" is demonstrated by R's broad connections to data sources, numerical and statistical computation libraries, graphical systems, external applications, and other languages. And it's further supported by the formal announcement this week of the reticulate package from RStudio, which provides a new interface between R and Python. With reticulate, you can:

• Import objects from Python, automatically converted into their equivalent R types. (For example, Pandas data frames become R data.frame objects, and NumPy arrays become R matrix objects.)
• Import Python modules, and call their functions from R 
• Source Python scripts from R
• Interactively run Python commands from the R command line
• Combine R code and Python code (and output) in R Markdown documents, as shown in the snippet below

The reticulate package was first released on Github in January 2017, and has been available on CRAN since March 2017. It has already spawned several higher-level integrations between R and Python-based systems, including:

• H204GPU, a R package for H20's GPU-based scikit-learn-like suite of algorithms;
• greta, a packagefor Bayesian model estimation with Markov-chain Monte-carlo, based on Tensorflow
• spacyr, a wrapper for the spaCy natural language processing toolkit; and
• XRPython, John Chamber's interface to Python based on his XR package for language extensions to R, which now uses reticulate for its low-level interface to Python.

The reticulate package is available now on CRAN. You can find more details in the announcement at the link below.

RStudio blog: reticulate: R interface to Python

Update March 31: Corrected date of first availability of reticulate on CRAN