Revolutions

R 3.3.2 now available

R 3.3.2, the latest update to the R language, was released today. Binary releases for Linux and Mac are available now from your local CRAN mirror, and the Windows builds will be available shortly.

As a minor update to the R 3.3 series, this update focuses mainly on fixing bugs and doesn't make any major changes to the langauge. As a result, you can expect existing scripts and packages to continue to work if you're upgrading from R 3.3.1. This update includes some performance improvements (particularly in calculation of eigenvalues), better handling of date axes in graphics, and improved documentation for the methods package. (Fun fact: when printed as a 9Mb PDF reference manual, the documentation for the R base and recommended packages now runs to 3452 pages. That's almost 3 copies of War and Peace!) 

The nickname for this release is "Sincere Pumpkin Patch", in recognition of the Halowe'en release date and (per RWeekly) references this clip from "It's the Great Pumpkin, Charlie Brown".

For the official announcement from the R core team including the detailed list of changes, see the posting in the R-announce mailing list linked below.

R-announce mailing list: R 3.3.2 is released

 

Because it's Friday: Why Dictators, Presidents and CEOs behave the way they do

If a leader does something irrational, your first assumption might be that they're crazy. But while the intersection in the Venn diagram of power and mental health issues is surely not empty, weird behaviour makes more sense if you assume there's information you don't have that explains why the weird action is, in fact, rational. This video from CPG Grey proposes what some of those unobserved factors may be:

The video prompted me to pick up the book it's based on, The Dictator's Handbook: Why Bad Behavior is Almost Always Good. It's been a thought-provoking read so far.

That's all from us for this week. Have a great weekend, and we'll see you back here on Monday!

Free e-book: Data Science with SQL Server 2016

There's a new e-book available to download free from Microsoft Academy: Data Science with Microsoft SQL Server 2016. 

This 90-page e-book is aimed at data scientists who already have some experience in R, but want to learn how to use R wirth SQL Server. The book was written by some of my most experienced colleagues from Microsoft's data science team at Microsoft: Buck Woody, Danielle Dean, Debraj GuhaThakurta, Gagan Bansal, Matt Conners, and Wee-Hyong Tok, and begins with an introduction by Joseph Sirosh. It includes everything you need to know to use R and SQL Server:

• How to install and configure your data science tool-set: SQL Server 2016, Microsoft R Client, RStudio/RTVS, etc.
• How to download data from SQL Server into a local R client
• How to create and update tables in SQL Server from R
• Use a remote SQL Server instance as an R compute engine, driven from your local R client
• Write SQL Server stored procedures that run R code on the server, and how to share them with others.

(If you're new to R or data science, there are also links to learning resources in Chapter 1.) The book also includes several fully-worked examples following the data science process, with links to data and code so you can try them out yourself:

• Creating a model to predict whether a tip is given for a taxi ride in New York City
• Building a customer churn model to find customers likely to switch to a competing service provider
• Analyzing "Internet of Things" data as part of a predictive maintenenance program

Data Science with Microsoft SQL Server 2016 is available now as a download from Microsoft Academy. (Desktop and mobile PDF formats available. Free registration required.)

ReporteRs: Manager-friendly Word or Powerpoint documents created with R

As much as we data scientists would like to work solely in R, there inevitably comes the time when our managers or customers want to see the fruits of our labours as a Word document or PowerPoint presentation. And not just any old document: it needs to be in the official corporate template, and will no doubt go through a series of back-and-forth revisions — marked up directly in the document — before it's considered final.

R has several options for generating Word and PowerPoint documents, notably RMarkdown and Slidify. But in both these cases, R is in the driver's seat, and if you want to make any changes to the document that will persist through changes to the R code, you'll need to make those changes with R. And as Aimee Gott from Mango points out, that doesn't work so well if the document will go through a review cycle with non-R users. (It's also quite tricky to use a corporate template with these solutions.)

That's where ReporteRs (created by David Gohel) comes in. While you can use ReporteRs to create complete documents and presentations programatically, one of the most useful features is to be able to insert R output (numbers, tables, and charts) into an existing document. This lets you use Word or Powerpoint as usual to style the document and build the narrative, and use R to insert the scientific results in the appropriate places.

For Word documents, simply place a named bookmark in the document where you'd like output from R to go. Then, you'll use that document and bookmark name in a call to addParagraph (to insert text, say a P-value or other result) or addImage (to insert a static PNG or JPG image). You can even use addPlot, to insert an R graphic that can be edited directly within the Word or Powerpoint file using the Drawing toolba: vary useful for pre-publication annotations, although such changes will be overwritten next time you generate the document. For tabular output like data frames, use the addFlexTable function to insert a table, which formats it nicely and gives lots of options for layout like merging cells and coloring rows and columns.

For Powerpoint presentations, first start with a PowerPoint presentation in your chosen template, and create blank slides where you'd like to insert R output. You can then replace those slides by indicating the slide number in the addSlide function. While you can only insert complete slides, ReporteRs gives you plenty of flexibility in the layout of the slides.

ReporteRs is an open source pacakge available for download on CRAN. For more details on ReporteRs and how to use it, check out its homepage at the link below.

David Gohel: ReporteRs

Introducing R-hub, the R package builder service

If you're developing a package for R to share with others — on CRAN, say — you'll want to make sure it works for others. That means testing it on various platforms (Windows, Mac, Linux, and all the versions thereof), and on various versions of R (current, past, and future). But it's likely you only have access to one platform, and installing and managing multiple R versions can be a pain.

R-hub, the online package-building service now in public beta, aims to solve this problem by making it easy to build and test your package on a variety of platforms and R versions. Using the rhub R package, you can with a single command upload your package to the cloud-based R-hub service, and build and test your package on the current, prior, and in-development versions of R, using any or all of these platforms:

(Mac platforms will be added soon, with Solaris a possible addition in the future.) In addition to running the standard R CMD CHECK package tests for you, R-hub also applies "sanitizers" to the binary builds, to check for common errors in any compiled code (for example, writing to unallocated memory). It also handles the loading of dependent packages for the tests, even if those packages aren't on CRAN or BioConductor (so you can test with development versions of packages on GitHub, for example).

R hub is for every member of the R community, and designed to make it easy to create and share R packages with others. In particular, for packages intended for submission to CRAN, R-hub is intended to simplify and streamline that process by detecting possible bugs and errors that might appear on other platforms or R versions before submitting the package to CRAN.

R-hub was developed by Gábor Csárdi, and funded by a generous grant from the R Consortium. Microsoft is proud to be a founding member of the R Consortium, and also provides Azure credits to support the ongoing operation of R-hub.

R-hub is available as a public beta now; download the r-hub package from GitHub to get started. (It will also be available on CRAN soon.) If you find problems or have suggestions, please report an issue at the R-hub Github repository. And if you'd like to learn more about R-hub, watch the recording of the R-hub webinar presented by the R Consortium, where Gábor introduces R-hub and demonstrates the process of building and testing a package.

 

Sharing our R Programs — With Style

by Graham Williams, Director of Data Science, Microsoft

Programming is an art and a way we express ourselves. As we write our programs we should keep in mind that someone else is very likely to be reading it. We can facilitate the accessibility of our programs through a clear presentation of the messages we are sharing.

As data scientists we also practice this art of programming. Indeed even more so we aim to share the narrative of our discoveries through our living and breathing of data through programming over the data. Writing programs so that others understand why and how we analysed our data is crucial. Data science is so much more than simply building black box analyses and models and we should be seeking to expose and share the process and particularly the knowledge that is discovered from the data.

Style is important in making the code we share readily accessible. Dictating a style to others is a sensitive issue. We thrive on our freedom to innovate and to express ourselves how we want but we also need consistency in how we do that and a style guide supports that. A style guide also helps us journey through a new language, providing a foundation for developing, over time, our own style in that language.

Through a style guide we share the tips and tricks for communicating clearly through our programs. We communicate through the language — a language that also happens to be executable by a computer. In this language we follow precisely specified syntax to develop sentences, paragraphs, and whole stories. Whilst there is infinite leeway in how we express ourselves in any language we can share a common set of principles as our style guide.

Over the years styles developed for very many different languages have evolved together with the medium for interacting with computers. I have a style guide for R that presents my personal current choices. This is the style guide I suggest (even require) for projects I lead and will appear in an upcoming book.

I hope the guide might be useful to others. It augments the other R style guides out there by providing the rationale for my choices. Irrespective of whether specific style suggestions suit you or not, choose your own and use them consistently. Do focus on communicating with others in the first instance and secondarily on the execution of your code (though critical it is).  Think of writing programs as writing narratives for others to read, to enjoy, to learn from and to build upon. It is a creative act to communicate well with our colleagues — be creative with style.

Hands On Data Science: Sharing R Code — With Style

Webinar: Changing Lives with Data Science and R at Microsoft

If you didn't have a chance to catch my presentation at the Machine Learning and Data Science Summit, I'll be reprising an updated version of the talk in a live webinar on Tuesday, November 1. I'll also be taking questions from the audience after the webinar. You can register here, and the details of the webinar are below.

Changing Lives with Data Science and R at Microsoft

Whether it’s called data science, machine learning, or predictive analytics, the combination of new data sources and statistical modeling has produced some truly revolutionary applications. Many of these applications incorporate open source technologies (including R) and research from academic institutions. In this talk, I’ll share a few ways that Microsoft is improving the lives of people around the world by applying Statistics, research and open-source software in applications and devices, and describe how Microsoft has integrated R into its data platforms.

Register here to attend — I hope to see you there!

Because it's Friday: Hitchcock vs Kubrick

We've seen (or rather heard) plenty of music mashups, but here's a movie mashup: Hitchcock's Rear Window augmented with various characters and scenes from Stanley Kubrick movies: The Red Drum Getaway from Gump.

 

Have a great weekend. Hopefully the Internet will be back to normal by Monday, in which case we'll see you then!

Election 2016: Tracking Emotions with R and Python

Temperament has been a key issue in the 2016 presidential election between Hillary Clinton and Donald Trump, and an issue highlighted in the series of three debates that concluded this week. Quantifying "temperament" isn't an easy task, but The Economist used the Microsoft Emotion API to chart the anger, contempt, sadness and surprised expressed in the faces of the candidates during key sequences of the debates, like this from the third debate:

The new R Graph Gallery

Once upon a time, there was the original R Graph Gallery, by Romain François. Sadly, it's been unavailable for several years. Now there's a new R Graph Gallery to fill the void, created by Yan Holtz. It contains more than 200 data visualizations categorized by type, along with the R code that created them.

You can browse the gallery by types of chart (boxplots, maps, histograms, interactive charts, 3-D charts, etc), or search the chart descriptions. Once you've found a chart you like, you can admire it in the gallery (and interact with it, if possible), and also find the R code which you can adapt for your own use. Some entries even include mini-tutorials describing how the chart was made. You can even submit your own graph, if you'd like to have it displayed in the gallery as well.

It also includes a number of graphics that are not data visualizations, but rather Data Art: artistic creations based on data and/or code, including this neat rendering of BB-8 in R code. Sadly it doesn't include any examples from the Accidental aRt collection, nor this new Dadaism from Stephen Turner.

What the 2016 electoral map would look like if I made it in #Rstats on the first try pic.twitter.com/XOfQq3oX4t

— Stephen Turner (@genetics_blog) October 15, 2016

If you're looking for some R code to adapt for a visualization, or just looking for inspiration, take a look at the R Graph Gallery.