Revolutions

The R Project: 2015 in Review

It's been a banner year for the R project in 2015, with frequent new releases, ever-growing popularity, a flourishing ecosystem, and accolades from both users and press. Here's a roundup of the big events for R from 2015. 

R continues to advance under the  new leadership of the R Foundation. There were five updates in 2015: R 3.1.3 in March, R 3.2.0 in April, R 3.2.1 in June, R 3.2.2 in August, and R 3.2.3 in December. That's impressive release rate, especially for a project that's been in active development for 18 years!

R's popularity continued unabated in 2015. R is the most popular language for data scientists according to the 2015 Rexer survey, and the most popular Predictive Analytics / Data Mining / Data Science software in the KDnuggets software poll. While R's popularity amongst data scientists is no surprise, R ranked highly even amongst general-purpose programming languages. In July, R placed #6 in the IEEE list of top programming languages, rising 3 places from its 2014 ranking. It also continues to rank highly amongst StackOverflow users, where it is the 8th most popular language by activity, and the fastest-growing language by number of questions. R was also a top-ranked language on GitHub in 2015.

The R Consortium, a trade group dedicated to the support and growth of the R community, was founded in June. Already, the group has published best practices for secure use of R, and formed the Infrastructure Steering Committee to fund and oversee commuinity projects. Its first project (a hub for R package developers) was funded in November, and proposals are being accepted for future projects.

2015 was the year that Microsoft put its weight behind R, beginning with the acquisition of Revolution Analytics in April and prominent R announcements at the BUILD Conference in May. Microsoft continues the steady pace of open-source R project releases, with regular updates to Revolution R Open, DeployR Open and the foreach and checkpoint packages. Revolution R Enterprise saw updates, and new releases of several Microsoft platforms have integrated R, including SQL Server 2016, Cortana Analytics, PowerBI, Azure and the Data Science Virtual Machine.  

Activity within local R user groups accelerated in 2015, with 18 new groups founded for a total of 174. Microsoft expanded its R user group sponsorship with the  Microsoft Data Science User Group Program. Community conferences also boasted record attendance, inclusing at useR! 2015, R/Finance, EARL Boston, and EARL London. Meanwhile, companies including Betterment, Zillow, Buzzfeed, the New York Times and many others shared how they benefit from R.

R also got some great coverage in the media this year, with features in Priceonomics, TechCrunch, Nature, Inside BigData, Mashable, The Economist, opensource.com and many other publications.

That's a pretty big year ... and we expect even more from R in 2016. A big thanks go out to everyone in the R community, and especially the R Core group, for making R the standout success it is today. Happy New Year!

Using segmented regression to analyse world record running times

by Andrie de Vries

A week ago my high school friend, @XLRunner, sent me a link to the article "How Zach Bitter Ran 100 Miles in Less Than 12 Hours". Zach's effort was rewarded with the American record for the 100 mile event.

Zach Bitter holds the American record for the 100 mile

This reminded me of some analysis I did, many years ago, of the world record speeds for various running distances. The International Amateur Athletics Federation (IAAF) keeps track of world records for distances from 100m up to the marathon (42km). The distances longer than 42km do not fall in the IAAF event list, but these are also tracked by various other organisations.

You can find a list of IAAF world records at Wikipedia, and a list of ultramarathon world best times at Wikepedia.

I extracted only the mens running events from these lists, and used R to plot the average running speeds for these records:

Creating multi-tab reports with R and jQuery UI

by Matt Parker, Data Scientist at Microsoft

One of the great advantages of R's openness is its extensibility. R's abundant packages are the most conspicuous example of that extensibility, and Revolution R Enterprise is a powerful example of how far it can stretch.

But R is also part of an entire ecosystem of open tools that can be linked together. For example, Markdown, Pandoc, and knitr combine to make R an incredible tool for dynamic reporting and reproducible research. If your chosen output format is HTML, you've linked into yet another open ecosystem with countless further extensions.

One of those extensions - and the focus of this post - is jQuery UI. jQuery UI makes a set of JavaScript's most useful moves available to developers as a robust, easy-to-implement toolkit ideal for adding a bit of interactivity to your knitr reports.

Tabs

For example: it's easy to use jQuery UI's Tabs widget to split a long report across several tabs of a webpage. Tabs are great for splitting complex reports up by topic, or for providing different types of users with customized views of the results.

To get a sense of what this conversion might look like, here's a simple R-Markdown report without tabs (Rmd source):

... and the same report with tabs (source):

Here's how I added tabs to the report.

1) First, I downloaded jQuery UI. Picking the right place to store the library can be tricky, but as long as the jQuery UI files are accessible to knitr when it's building the report, you'll be okay. For this demo report, I just unzipped the files right next to the .rmd source.

2) Next, I added a few lines to the <head> element of the report. Every webpage has a <head> element. knitr would typically build this for you, but in this case we need to write it manually to be sure that the jQuery UI scripts and CSS are linked in the HTML output.

<head>
  <meta charset="utf-8">
  <title>Reported Active Tuberculosis Cases in the United States, 1993-2013</title>
  <link rel="stylesheet" href="jquery-ui/jquery-ui.min.css">
  <script src="jquery-ui/external/jquery/jquery.js"></script>
  <script src="jquery-ui/jquery-ui.js"></script>
  <script>
  $(function() {
    $( "#tabs" ).tabs();
  });
  </script>
</head>

3) Next, I created the navigation bar by creating an HTML chunk (div) with a list inside of it (ul). Each item in that list (li) represents one tab that I'd like the page to have. Finally, I make each of those list items a link with a unique tag (<a href="#nation">), and give the link a title (Nationally, By State, Treatment Completion).

<div id="tabs">
<ul>
<li><a href="#nation">Nationally</a></li>
<li><a href="#states">By State</a></li>
<li><a href="#treatment">Treatment Completion</a></li>
</ul>

Don't worry if you don't understand the HTML syntax here - you can just copy and edit the code above.

4) Finally, I marked out which sections of R-Markdown I wanted to put on each tab by surrounding that section with a div:

<div id="nation">

## Reported Active TB Cases in the United States, 1993-2013

```{r nation}

tbstats %>% 
    group_by(Year) %>% 
    summarise(n_cases = sum(Count)) %>% 
    ggplot(aes(x = Year, y = n_cases)) +
        geom_line(size = 2) +
        labs(x = "Year Reported",
             y = "Number of Cases",
             title = "Reported Active TB Cases in the United States") +
        expand_limits(y = 0)


```

</div>

There are two crucial details here: - the div has an id that corresponds to one of the tabs I've created (href=#nation corresponds to <div id="nation">) - the div is closed with a </div> tag. Without this, the entire report would be included on the first tab.

5) Click the "Knit HTML" button! knitr will convert your R-Markdown into plain Markdown, and then call Pandoc to complete the conversion into gloriously-tabbed HTML.

Tabs are very handy for reporting - but the whole HTML/CSS/JavaScript ecosystem is at your disposal. If you've seen other good reporting tricks in HTML, let us know in the comments below.

ggplot2 version 2 adds extensibility and other improvements

Despite the ggplot2 project — the most popular data visualization package for R — being in maintenance mode, RStudio's Hadley Wickham has given the R community a surprise gift with a version 2.0.0 update for ggplot2. According to Hadley this is a "huge" update with more than 100 fixes and improvements.

The most significant addition is that ggplot2 now has a formal extension mechanism, which means that package authors can now create their own geoms (plot types), statistics (data aggregation/transformation methods) and themes. There are also a number of smaller improvements, including making it easy to draw curved lines between points with geom_curve, a way to suppress overlapping text labels, and a way to add labels with rounded enclosing boxes to plots. There are also a few minor changes to the appearance of plots, with some changes to colors and text sizes to improve readability and (my personal favorite) the elimination of the diagonal line that used to appear in the color boxes in legends. 

If you're new to ggplot2, getting to grips with the Grammar of Graphics system can be a steep learning curve, but it's well worth it in terms of long-term productivity in creating beautiful graphics with R. Hadley Wickham's book, Elegant Graphics for Data Analysis, is a great place to get started. Experienced ggplot2 users will also appreciate the freshly-updated ggplot2 cheatsheet from the RStudio team.

This new update is mostly compatible with older versions of ggplot2, but it may require updates to existing code in some cases. (If you want to try the new ggplot2 but still have access to old versions for compatibility, take a look at the checkpoint package.) The updated ggplot2 package is now available for download via CRAN with install.packages("ggplot2"), or via the ggplot2 Github repository.

RStudio Blog: ggplot2 2.0.0

Because it's Friday: All I want for Christmas

On this Christmas Day, allow me to share a couple of interesting takes on Mariah Carey's modern Christmas classic, All I Want for Christmas is You. (Unlike most pop songs, Carey wrote the melody and lyrics herself.) If you've somehow managed to avoid hearing the song over the past decade, here it is lip-synced in 2011 by the Royal Navy crew of HMS Ocean:

 

The Mariah Carey version of the song is arranged in the key of G major, but has a very different feeling when sung in a minor key, as done here by Chase Holfelder:

 

Now, if you're like me and have had no musical training and couldn't tell a minor key from a house key, I thoughly recommend seeking out the 1968 Isaac Asimov essay, "Music to my ears" (it's included in this anthology), an excellent explainer for the non-musical types like me. You might also be interested in the mathematics behind musical scales. 

And finally, can you hear Mariah Carey's voice in the background automatically-generated MIDI conversion of the song?

 

If you can: congratulations! It means you've heard the actual song often enough that your brain is generating an auditory hallucination of her voice. The only thing you're actually hearing is synthetic piano keys.

That's all from us this week. If you're celebrating, Merry Christmas! (Or your local equivalent as seen below, with thanks to Christophe Cariou and R.) We'll be back on Monday -- enjoy!

Looking forward to 2016

by Joseph Rickert

The following map of all of the R user groups listed in Microsoft's Local R User Group Directory is good way to visualize the R world as we rocket into 2016. As a member of the useR!2016 planning committee, foremost in my mind right now is that in just a few months people will be coming to Stanford from all points plotted and almost everywhere in between.

I suppose I have a strong bias here, but this conference promises to be an outstanding event where Silicon Valley welcomes R. The invited speakers for useR! 2016 are  Richard Becker, Donald Knuth, Deborah Nolan, Simon Urbanek, Hadley Wickham and Daniela Witten and the invited tutorials already scheduled to be on the program are:

But all of this is only half the story. The rest depends on you. There are still several open slots for tutorials, and there is plenty of time to think about submitting an abstract for a contributed talk. Please, use one of the templates to submit your tutorial proposal and make sure it is received at [email protected] before the submissions deadline: midnight January 10, 2016 PST. The clock for contributed talks starts on January 5th and runs till March 25th.

Moreover, due to a special alignment of the stars in 2016 the Bioconductor Conference will also be held at Stanford from June 24th through the 26th making it possible to attend both conferences! The details should be up on the Bioconductor site soon. Bay Area hotels are notoriously expensive and there is limited capacity close to the campus, so if you are going to either useR! 2016 or BioC 2016 make you housing arrangements soon.

And - don't let all of this cosmic California talk distract you from planning for R/Finance 2016 which will be held May 20th and 21st in Chicago. If you have any interest at all in Finance (or Chicago), R/Finance is as good as it gets. The deadline for submissions is January 29th. You can find the details here.

And - don't forget that January 10th is also the deadline for submitting a proposal to the R-Consortium for the next round of funding. (You can find the details for submitting a proposal here.) The Infrastructure Steering Committee (ISC) has already received proposals for improving distributed computing in R, developing a package and API for retrieving package download statistics and building a framework for collecting R usage information at the function level. Don't miss this opportunity to get your ideas for work that could benefit the greater R Community in front of the ISC.

Finally, if you are thinking about starting an R user group in 2016 consider applying for support from Microsoft's Data Science User Group Sponsorship Program to get RUG going. You can find the application form here.

Embedding images in R package help pages

by Andrie de Vries

Have you noticed that most R help does not contain any images? Historically, this was for a good reason: most programming language help assumes that you use the system on a text-only system, e.g. a headless linux server. If your system has no graphical capabilities, but the help contains images, then the help itself can become confusing.

That said, most R users use their help on a system with full graphical capabilities. Thus the R core team updated R itself in version 2.12 to have the ability to include images in your help.

The question is how do you go about creating help files that contain images?

My search engine skills only uncovered two references:

• A blog post by Romain Francois: "embed images in Rd documents".
• An answer to a question on StackOverflow: "Including images in R-package documentation (.Rd) files". This was answered by Roman Lustrik.

Both approaches require you to use a tag in your .Rd file:

• Use the \sexpr tag:
  • Romain's suggestion is really powerful, since it uses a mechanism to dynamically render R code directly in your Rd file.  I didn't know this, but the R Extensions manual describes how to use the tag \sexpr to render R code dynamically. In his blog post, Romain gives an example of doing this.
• Use the \figure tag:
  • The answer at StackOverflow points to the \figure tag. This allows you to post a png image into HTML help, or a pdf image into PDF help.

Putting this into action

For one of the projects I'm working on, the AzureML package, I opted for the second option. The AzureML package (source code on github) allows you to interact with the AzureML Machine Learning Studio, directly from R.

I wanted to include some diagrams in the help pages to make it easy to find the credentials you need to connect to the AzureML service.

The code to create the help page looks like this. Notice the \figure{} to include the images.

#' @section Finding your AzureML credentials:
#' You can find your Azure Machine Learning \code{workspace id} and \code{authorization token} in the Azure Machine Learning Studio interface.
#' 
#' Workspace ID
#' 
#' \if{html}{\figure{workspace_id.png}{options: width="60\%" alt="Figure: workspace_id.png"}}
#' 
#' Authorization token
#' 
#' \if{html}{\figure{authorization_token.png}{options: width="60\%" alt="Figure: authorization_token.png"}}
#' 

One more thing: you need to put the images in the package folder man/figures.

The result in RStudio looks like this:

 

For more information about the AzureML package, take a look at the README on github, or read the package vignette.

Fraud Detection with R and Azure

Detecting fraudulent transactions is a key applucation of statistical modeling, especially in an age of online transactions. R of course has many functions and packages suited to this purpose, including binary classification techniques such as logistic regression.

If you'd like to implement a fraud-detection application, the Cortana Analytics gallery features an Online Fraud Detection Template. This is a step-by step guide to building a web-service which will score transactions by likelihood of fraud, created in five steps:

• Generate tagged data
• Data Preprocessing
• Feature engineering
• Train and Evaluation Model
• Publish as web service

Each step makes use of the R language, as seen ion the "Execute R Script" nodes in the screenshot from the "generate tagged data" step below.

The methodology in this template can be easily extended to fraud detection scenarios in other domains. To get started with the template, follow the link below. You can experiment with the template for without registering in Guest Access mode, or sign up for a Free Workspace account to save your changes.

Cortana Analytics Gallery: Online Fraud Detection Template 

 

R is the fastest-growing language on StackOverflow

StackOverview is a popular Q&A site, and a go-to resource for developers of all languages to find answers to programming problems they may have: most of the time, the question has already been asked and answered, or you can always post a new question and wait for a reply. It's an excellent resource for R users, featuring answers to nearly 100,000 R questions. In fact, R is the fastest-growing language on StackOverflow in terms of the number of questions asked:

The chart above was created -- in R, of course -- by Joshua Kunst, who helpfully provided the R code to make this subway-style rank plot using the ggplot2 package. The "fastest-growing" claim is based on Joshua's regression analysis of the data above: R's trendline has a slope of 4.50. (RedMonk also uses StackOverflow data, combined with GitHub activity, for their bi-annual language popularity rankings. R was ranked #13 in their most recent analysis, in June 2015.) The data comes directly from the StackExchange data dump, loaded into a Sqlite database and processed in R using Hadley Wickham's RSQLite package. There's much more interesting analysis of the StackOverflow data in Joshua's blog post, including a cluster analysis of the top 100 tags in StackOverflow. Check it out at the link below.

Joshua Kunst: What do we ask in StackOverflow?

 

Because it's Friday: Smart Charger

Did you know that the charger for a modern MacBook has about as much computing power as the original Macintosh computer? That's one surprising lesson from this teardown of an Apple charger:

One unexpected component is a tiny circuit board with a microcontroller, which can be seen above. This 16-bit processor constantly monitors the charger's voltage and current. It enables the output when the charger is connected to a Macbook, disables the output when the charger is disconnected, and shuts the charger off if there is a problem. This processor is a Texas Instruments MSP430 microcontroller, roughly as powerful as the processor inside the original Macintosh.

Some other surprisingly smart devices I learned about when I posted this on Twitter: the original Laserwriter printer was more powerful than the computers it was connected to, and a modern Firewire to HDMI cable contains 256k of RAM runs a Unix-like operating system.

That's all from us for this week. We'll be back on Monday and all next week keeping you up-to-date during the holiday period. See you then!