Revolutions

Microsoft R Open 3.3.2 now available

Microsoft R Open 3.3.2, Microsoft's enhanced distribution of open source R, is now available for download for Windows, Mac, and Linux. This update upgrades the R language engine to version 3.3.2, adds new bundled packages and updates others, and upgrades the Intel Math Kernel Libraries.

The updated R 3.3.2 engine includes some performance improvements (particularly in calculation of eigenvalues), better handling of date axes in graphics, and improved documentation for the methods package. (See here for a complete list of fixes.) There are no user-visible changes in the language itself, which means that scripts and packages should work without changes from MRO 3.3.1.

This update to MRO comes bundled with some additional packages, notably jsonlite (for fast processing of JSON-formatted data), png (to support reading and writing of PNG images), R6 (allowing the creation of classes with reference semantice), and curl (for connecting to web-based data sources). The checkpoint and deployrRserve packages have also been updated.

The MKL libraries, which provide high-performance matrix and vector calculations to MRO, have also been updated. (This fixes some issues with matrix multiplication and arima that were reported.) 

For reproducibility, installing packages with MRO 3.3.2 will by default get you package versions as of November 1, 2016. Many packages have been updated or released since MRO 3.3.1. (See here for some highlights of new packages.) As always, if you want to use newer versions of CRAN packages (or access older versions, for reproducibility), just use the checkpoint function to access the CRAN Time Machine. 

MRO is supported on Windows, Mac and Linux. This release adds support for two new platforms: MacOS Sierra (10.12) and Ubuntu 16.04. 

We hope you find Microsoft R Open useful, and if you have any comments or questions please visit the Microsoft R Open forum. To download Microsoft R Open (it's free!), simply follow the link below.

MRAN: Download Microsoft R Open

Free online course: Analyzing big data with Microsoft R Server

If you're already familiar with R, but struggling with out-of-memory or performance problems when attempting to analyze large data sets, you might want to check out this new EdX course, Analyzing Big Data with Microsoft R Server, presented by my colleague Seth Mottaghinejad. In the course, you'll learn how to build models using the RevoScaleR package, and deploy those models to production environments like Spark and SQL Server. The course is self-paced with videos, tutorials and tests, and is free to audit.

(By the way, if you don't already know R, you might want to check out the courses Introduction to R for Data Science and Programming in R for Data Science first.)

The RevoScaleR package isn't available on CRAN: it's included with Microsoft R Server and Microsoft R Client. You can download and use Microsoft R Client for free, which provides an installation of R with the RevoScaleR library built in and loaded when you start the session. An R IDE is also recommended: you can use R Tools for Visual Studio or RStudio.

The course is open now, and you can get started at EdX at the link below.

EdX: Analyzing Big Data with Microsoft R Server

 

A heat map of Divvy bike riders in Chicago

Chicago's a great city for a bike-sharing service. It's pretty flat, and there are lots of wide roads with cycle lanes. I love Divvy and use it all the time. Not so much in the winter though: it gets very cold here. Nonetheless, this heat map of Divvy riders, created in R by Austin Wehrwein, reveals a hardcore set of riders that use the service even in the bitter cold. For example on January 27 2014 when the temperature was °F (-18°C), there were still 406 riders that day.

There's no calendar heatmap extension to ggplot2 (that I know of), but as Austin shows below it's actually very easy to create one in base ggplot2 simply by defining weekday, month and year columns in the data.

Austin Wehrwein: Heatmaps with Divvy Data

 

Happy Thanksgiving!

It's Thanksgiving day here in the US, so we're taking the rest of the week off to reflect on what we're thankful for. And even if you're not in the US, today is a great day to send thanks to the R Core Group for providing their dedication, time, and expertise to make the R Project what it is today.  

(Sadly, cowsay doesn't feature a Thanksgiving turkey.) We'll be back next week. For those that are celebrating, have a great holiday. See you on Monday!

Tutorial: Build a live rental prediction service with SQL Server R Services

A great way to learn is by doing, so if you've been thinking about how to enable R-based computations within SQL Server, a new tutorial will take you through all the steps of building an intelligent application. In a few simple steps, you'll set up all the necessary software and code to build a live service that predicts demand for a ski rental shop.

The tutorial "Create a predictive model in SQL Server R Services" will step you through:

• Installing SQL Server, Microsoft R Client, and an IDE for R (RTVS or RStudio) on your Windows machine. All of the components are available for free from Visual Studio Dev Essentials.
• Loading the provided ski-rental data, exploring the data, and building a model to predict number of rentals in R.
• Creating a stored procedure to predict rentals from the date and the amount of snowfall.

The tutorial also links to a number of SQL Server Management Studio custom reports which will be helpful to R developers to manage R packages on the server, monitor R execution statistics, and more.

To get started with the tutorial, simply follow the link below. 

SQL Server: Build an intelligent app with SQL Server and R

 

Calculating AUC: the area under a ROC Curve

by Bob Horton, Microsoft Senior Data Scientist

Receiver Operating Characteristic (ROC) curves are a popular way to visualize the tradeoffs between sensitivitiy and specificity in a binary classifier. In an earlier post, I described a simple “turtle’s eye view” of these plots: a classifier is used to sort cases in order from most to least likely to be positive, and a Logo-like turtle marches along this string of cases. The turtle considers all the cases it has passed as having tested positive. Depending on their actual class they are either false positives (FP) or true positives (TP); this is equivalent to adjusting a score threshold. When the turtle passes a TP it takes a step upward on the y-axis, and when it passes a FP it takes a step rightward on the x-axis. The step sizes are inversely proportional to the number of actual positives (in the y-direction) or negatives (in the x-direction), so the path always ends at coordinates (1, 1).  The result is a plot of true positive rate (TPR, or sensitivity) against false positive rate (FPR, or 1 - specificity), which is all an ROC curve is.

Computing the area under the curve is one way to summarize it in a single value; this metric is so common that if data scientists say “area under the curve” or “AUC”, you can generally assume they mean an ROC curve unless otherwise specified.

Probably the most straightforward and intuitive metric for classifier performance is accuracy. Unfortunately, there are circumstances where simple accuracy does not work well. For example, with a disease that only affects 1 in a million people a completely bogus screening test that always reports “negative” will be 99.9999% accurate. Unlike accuracy, ROC curves are insensitive to class imbalance; the bogus screening test would have an AUC of 0.5, which is like not having a test at all.

In this post I’ll work through the geometry exercise of computing the area, and develop a concise vectorized function that uses this approach. Then we’ll look at another way of viewing AUC which leads to a probabilistic interpretation.

Yes, it really is the area under the curve

Let’s start with a simple artificial data set:

category <- c(1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0)
prediction <- rev(seq_along(category))
prediction[9:10] <- mean(prediction[9:10])

Here the vector prediction holds ersatz scores; these normally would be assigned by a classifier, but here we’ve just assigned numbers so that the decreasing order of the scores matches the given order of the category labels. Scores 9 and 10, one representing a positive case and the other a negative case, are replaced by their average so that the data will contain ties without otherwise disturbing the order.

To plot an ROC curve, we’ll need to compute the true positive and false positive rates. In the earlier article we did this using cumulative sums of positives (or negatives) along the sorted binary labels. But here we’ll use the pROC package to make it official:

library(pROC)
roc_obj <- roc(category, prediction)
auc(roc_obj)

## Area under the curve: 0.825

roc_df <- data.frame(
  TPR=rev(roc_obj$sensitivities), 
  FPR=rev(1 - roc_obj$specificities), 
  labels=roc_obj$response, 
  scores=roc_obj$predictor)

The roc function returns an object with plot methods and other conveniences, but for our purposes all we want from it is vectors of TPR and FPR values. TPR is the same as sensitivity, and FPR is 1 - specificity (see “confusion matrix” in Wikipedia). Unfortunately, the roc function reports these values sorted in the order of ascending score; we want to start in the lower left hand corner, so I reverse the order. According to the auc function from the pROC package, our simulated category and prediction data gives an AUC of 0.825; we’ll compare other attempts at computing AUC to this value.

Plotting the approach

If the ROC curve were a perfect step function, we could find the area under it by adding a set of vertical bars with widths equal to the spaces between points on the FPR axis, and heights equal to the step height on the TPR axis. Since actual ROC curves can also include portions representing sets of values with tied scores which are not square steps, we need to adjust the area for these segments. In the figure below we use green bars to represent the areas under the steps. Adjustments for sets of tied values will be shown as blue rectangles; half the area of each of these blue rectagles is below a sloped segment of the curve.

The function for drawing polygons in base R takes vectors of x and y values; we’ll start by defining a rectangle function that uses a simpler and more specialized syntax; it takes x and y coordinates for the lower left corner of the rectangle, and a height and width. It sets some default display options, and passes along any other parameters we might specify (like color) to the polygon function.

rectangle <- function(x, y, width, height, density=12, angle=-45, ...) 
  polygon(c(x,x,x+width,x+width), c(y,y+height,y+height,y), 
          density=density, angle=angle, ...)

The spaces between TPR (or FPR) values can be calculated by diff. Since this results in a vector one position shorter than the original data, we pad each difference vector with a zero at the end:

roc_df <- transform(roc_df, 
  dFPR = c(diff(FPR), 0),
  dTPR = c(diff(TPR), 0))

For this figure, we’ll draw the ROC curve last to place it on top of the other elements, so we start by drawing an empty graph (type='n') spanning from 0 to 1 on each axis. Since the data set has exactly ten positive and ten negative cases, the TPR and FPR values will all be multiples of 1/10, and the points of the ROC curve will all fall on a regularly spaced grid. We draw the grid using light blue horizontal and vertical lines spaced one tenth of a unit apart. Now we can pass the values we calculated above to the rectangle function, using mapply (the multi-variate version of sapply) to iterate over all the cases and draw all the green and blue rectangles. Finally we plot the ROC curve (that is, we plot TPR against FPR) on top of everything in red.

plot(0:10/10, 0:10/10, type='n', xlab="FPR", ylab="TPR")
abline(h=0:10/10, col="lightblue")
abline(v=0:10/10, col="lightblue")

with(roc_df, {
  mapply(rectangle, x=FPR, y=0,   
         width=dFPR, height=TPR, col="green", lwd=2)
  mapply(rectangle, x=FPR, y=TPR, 
         width=dFPR, height=dTPR, col="blue", lwd=2)

  lines(FPR, TPR, type='b', lwd=3, col="red")
})

Adding up the area

The area under the red curve is all of the green area plus half of the blue area. For adding areas we only care about the height and width of each rectangle, not its (x,y) position. The heights of the green rectangles, which all start from 0, are in the TPR column and widths are in the dFPR column, so the total area of all the green rectangles is the dot product of TPR and dFPR. Note that the vectored approach computes a rectangle for each data point, even when the height or width is zero (in which case it doesn’t hurt to add them). Similarly, the heights and widths of the blue rectangles (if there are any) are in columns dTPR and dFPR, so their total area is the dot product of these vectors. For regions of the graph that form square steps, one or the other of these values will be zero, so you only get blue rectangles (of non-zero area) if both TPR and FPR change in the same step. Only half the area of each blue rectangle is below its segment of the ROC curve (which is a diagonal of a blue rectangle). Remember the ‘real’ auc function gave us an AUC of 0.825, so that is the answer we’re looking for.

simple_auc <- function(TPR, FPR){
  # inputs already sorted, best scores first 
  dFPR <- c(diff(FPR), 0)
  dTPR <- c(diff(TPR), 0)
  sum(TPR * dFPR) + sum(dTPR * dFPR)/2
}

with(roc_df, simple_auc(TPR, FPR))

## [1] 0.825

Enumerating rank comparisons

Now let’s try a completely different approach. Here we generate a matrix representing all possible combinations of a positive case with a negative case. Each row represents a positive case, in order from the highest-scoring positive case at the bottom to the lowest-scoring positive case at the top. Similarly, the columns represent the negative cases, sorted with the highest scores at the left. Each cell represents a comparison between a particular positive case and a particular negative case, and we mark the cell by whether its positive case has a higher score (or higher overall rank) than its negative case. If your classifier is any good, most of the positive cases will outrank most of the negative cases, and any exceptions will be in the upper left corner, where low-ranking positives are being compared to high-ranking negatives.

rank_comparison_auc <- function(labels, scores, plot_image=TRUE, ...){
  score_order <- order(scores, decreasing=TRUE)
  labels <- as.logical(labels[score_order])
  scores <- scores[score_order]
  pos_scores <- scores[labels]
  neg_scores <- scores[!labels]
  n_pos <- sum(labels)
  n_neg <- sum(!labels)
  M <- outer(sum(labels):1, 1:sum(!labels), 
             function(i, j) (1 + sign(pos_scores[i] - neg_scores[j]))/2)

  AUC <- mean (M)
  if (plot_image){
    image(t(M[nrow(M):1,]), ...)
    library(pROC)
    with( roc(labels, scores),
          lines((1 + 1/n_neg)*((1 - specificities) - 0.5/n_neg), 
                (1 + 1/n_pos)*sensitivities - 0.5/n_pos, 
                col="blue", lwd=2, type='b'))
    text(0.5, 0.5, sprintf("AUC = %0.4f", AUC))
  }
  
  return(AUC)
}

rank_comparison_auc(labels=as.logical(category), scores=prediction)

 


## [1] 0.825

The blue line is an ROC curve computed in the conventional manner (slid and stretched a bit to get the coordinates to line up with the corners of the matrix cells). This makes it evident that the ROC curve marks the boundary of the area where the positive cases outrank the negative cases. The AUC can be computed by adjusting the values in the matrix so that cells where the positive case outranks the negative case receive a 1, cells where the negative case has higher rank receive a 0, and cells with ties get 0.5 (since applying the sign function to the difference in scores gives values of 1, -1, and 0 to these cases, we put them in the range we want by adding one and dividing by two.) We find the AUC by averaging these values.

AUC as probability

The probabilistic interpretation is that if you randomly choose a positive case and a negative case, the probability that the positive case outranks the negative case according to the classifier is given by the AUC. This is evident from the figure, where the total area of the plot is normalized to one, the cells of the matrix enumerate all possible combinations of positive and negative cases, and the fraction under the curve comprises the cells where the positive case outranks the negative one.

We can use this observation to approximate AUC:

auc_probability <- function(labels, scores, N=1e7){
  pos <- sample(scores[labels], N, replace=TRUE)
  neg <- sample(scores[!labels], N, replace=TRUE)
  # sum( (1 + sign(pos - neg))/2)/N # does the same thing
  (sum(pos > neg) + sum(pos == neg)/2) / N # give partial credit for ties
}

auc_probability(as.logical(category), prediction)

## [1] 0.8249989

Testing on a bigger example

Now let’s try our new AUC functions on a bigger dataset. I’ll use the simulated dataset from the earlier blog post, where the labels are in the bad_widget column of the test set dataframe, and the scores are in a vector called glm_response_scores.

This data has no tied scores, so for testing let’s make a modified version that has ties. We’ll plot a black line representing the original data; since each point has a unique score, the ROC curve is a step function. Then we’ll generate tied scores by rounding the score values, and plot the rounded ROC in red. Note that we are using “response” scores from a glm model, so they all fall in the range from 0 to 1. When we round these scores to one decimal place, there are 11 possible rounded scores, from 0.0 to 1.0. The AUC values calculated with the pROC package are indicated on the figure.

roc_full_resolution <- roc(test_set$bad_widget, glm_response_scores)
rounded_scores <- round(glm_response_scores, digits=1)
roc_rounded <- roc(test_set$bad_widget, rounded_scores)
plot(roc_full_resolution, print.auc=TRUE)

## 
## Call:
## roc.default(response = test_set$bad_widget, predictor = glm_response_scores)
## 
## Data: glm_response_scores in 59 controls (test_set$bad_widget FALSE) < 66 cases (test_set$bad_widget TRUE).
## Area under the curve: 0.9037

lines(roc_rounded, col="red", type='b')
text(0.4, 0.43, labels=sprintf("AUC: %0.3f", auc(roc_rounded)), col="red")

Now we can try our AUC functions on both sets to check that they can handle both step functions and segments with intermediate slopes.

options(digits=22)
set.seed(1234)

results <- data.frame(
  `Full Resolution` = c(
    auc = as.numeric(auc(roc_full_resolution)),
    simple_auc = simple_auc(rev(roc_full_resolution$sensitivities), rev(1 - roc_full_resolution$specificities)),
    rank_comparison_auc = rank_comparison_auc(test_set$bad_widget, glm_response_scores, 
                          main="Full-resolution scores (no ties)"),
    auc_probability = auc_probability(test_set$bad_widget, glm_response_scores)
  ),
  `Rounded Scores` = c( 
    auc = as.numeric(auc(roc_rounded)),
    simple_auc = simple_auc(rev(roc_rounded$sensitivities), rev(1 - roc_rounded$specificities)),
    rank_comparison_auc = rank_comparison_auc(test_set$bad_widget, rounded_scores,
                          main="Rounded scores (ties in all segments)"),
    auc_probability = auc_probability(test_set$bad_widget, rounded_scores)
  )
)

 
 

	Full.Resolution	Rounded.Scores
auc	0.90369799691833586	0.89727786337955828
simple_auc	0.90369799691833586	0.89727786337955828
rank_comparison_auc	0.90369799691833586	0.89727786337955828
auc_probability	0.90371970000000001	0.89716879999999999

So we have two new functions that give exactly the same results as the function from the pROC package, and our probabilistic function is pretty close. Of course, these functions are intended as demonstrations; you should normally use standard packages such as pROC or ROCR for actual work.

Here we’ve focused on calculating AUC and understanding the probabilistic interpretation. The probability associated with AUC is somewhat arcane, and is not likely to be exactly what you are looking for in practice (unless you actually will be randomly selecting a positive and a negative case, and you really want to know the probability that the classifier will score the positive case higher.) While AUC gives a single-number summary of classifier performance that is suitable in some circumstances, other metrics are often more appropriate. In many applications, overall behavior of a classifier across all possible score thresholds is of less interest than the behavior in a specific range. For example, in marketing the goal is often to identify a highly enriched target group with a low false positive rate. In other applications it may be more important to clearly identify a group of cases likely to be negative. For example, when pre-screening for a disease or defect you may want to rule out as many cases as you can before you start running expensive confirmatory tests. More generally, evaluation metrics that take into account the actual costs of false positive and false negative errors may be much more appropriate than AUC. If you know these costs, you should probably use them. A good introduction relating ROC curves to economic utility functions, complete with story and characters, is given in the excellent blog post “ML Meets Economics.”

Because it's Friday: Non-transitive dice

I saw a fascinating talk from Christopher Bishop, author of Pattern Recognition and Machine Learning at MLADS (Microsoft's internal machine learning and data science conference) yesterday. He introduced me to the concept of non-transitive dice, which demonstrate how unintuitive probability can be sometimes.

Let's play a game with these four dice (invented by none other than Bradley Efron). I'll let you pick any die you choose, and then I'll choose mine. Then we'll both roll our dice, and the person with the highest roll wins. (By design, there can't be ties.) To make things less random, let's make it a best of 9 competition.

Image credit: Wikipedia

The thing about these die is that whatever die you choose, I can always choose a die that will statistically beat yours. There is no best die, and yet evey die has another that will outperform it. You can see it by running the numbers on all the faces below. In the long run, yellow beats red; blue beats yellow; green beats blue; and red beats green. What's more, each die beats the other at a rate of 2:1.

 

It just goes to show that when it comes to probalities, it's dangerous to rely on your intuition.

That's all from us here at the blog for this week. Have a good weekend, and we'll see you back here on Monday.

The 5 most popular R packages

The good folks at DataCamp track activity related to R packages on the RDocumentation.org Trends page. As of this writing, it tracks statistics on 11,768 packages (distributed across CRAN, BioConductor and Github) comprising over 1.7 million R functions in total. On that page, you can find current rankings on the most downloaded R packages, the most prolific package contributors, and the latest package releases and updates.

In a recent blog post, DataCamp gave an overview of the top 5 most downloaded packages, which currently stand at:

1. dplyr, a grammar of data manipulation
2. devtools, a collection of package development tools
3. foreign, read data stored by Minitab, S, SAS, SPSS, Stata, and more
4. cluster, methods for cluster analysis
5. ggplot2, An implementation of the grammar of graphics in R

Those rankings are by the number of direct downloads, those initiated when an R user makes an explicit install.packages call. Many popular packages have dependencies, however, which are also downloaded when a package is installed. If you measure the top 5 packages by total downloads including those as a result of being a dependency, the top 5 rankings are:

1. Rcpp, seamless R and C++ Integration. (Many packages incorporate C or C++ code via this package.)
2. ggplot2, An implementation of the grammar of graphics in R
3. stringr, simple, consistent wrappers for common string operations
4. stringi, character string processing facilities
5. reshape2, flexibly reshape data

If you're an R user who doesn't write R functions very often, the first collection of packages is probably familiar to you. But if you're an R developer who spends a lot of time writing functions (or packages), you're likely intimate with the second list as well.

For more details on the most popular R packages and the RDocumentation.org website, check out the link below.

DataCamp blog: The 5 most downloaded R packages 

 

 

Notable New and Updated R packages (to October 2016)

As we prepare for the upcoming release of Microsoft R Open, I've been preparing the list of new and updated packages for the spotlights page. This involves scanning the CRANberries feed (with gracious thanks to Dirk Eddelbuettel) for newly-released packages and significant updates to existing ones.

This is a lot of data to process. For context, in October 2016 alone there were 174 new packages and 629 updated packages. For the 4 months since the last MRO release, I reviewed all of the new packages and some of the updates (those with a 1.0 or 2.0 release) by hand, and selected those that are likely to be of interest to a broad segment of R users. I hope you find some useful nuggets in the list. It's likely I missed some good ones; if you know of any others that should have been included, let me know in the comments.

Notable New and Updated R Packages: July — October 2016

Data Munging

anytime: Convert anything that looks vaguely like a date into a POSIXct object. Examples here.

fasttime: Fast functions for timestamp manipulation.

forcats: Tools for working with categorical variables (factors).

roll: Parallel functions for computing rolling statistics of time-series data.

stringb: String handling functions with an interface like stringr/stringi, but without the dependencies.

tidyverse: A collection of popular packages for data munging from Hadley Wickham. More here.

 

Data Sources

countyweather: Meterological time series data for U.S. counties, from NOAA.

darksky: Interface to the Dark Sky API, which provides current or historical global weather conditions.

energyr: Data published by the United States Federal Energy Regulatory Commission.

genderNames: API client for genderize.io which will tell you the gender of the name you input.

Rlinkedin (updated): Provides access to the LinkedIn API for people, job, and discussion data.

outbreaks: A collection of disease outbreak data, some real, some simulated.

rnoaa: Download NOAA weather data. Vignettes provided for many different data types.

rpinterest: Access the Pinterest API for data on boards, pins and users.

ttbbeer: Beer statistics from the U.S. Department of the Treasury, Alcohol and Tobacco Tax and Trade Bureau.

ubeR: Interface to the Uber API.

 

Graphics

cartogram: Create and plot a continuous area cartogram by a rubber sheet distortion algorithm.

colourlovers: Provides access to the COLOURlovers API, which offers color inspiration and color palettes.

earthtones: Derive a color palette from a satellite image of a location on Earth. The vignette includes examples.

Homeric: Doughnut plots - pie charts with a blank center. The package is named after Homer Simpson.

RSVGTipsDevice (updated): An R SVG graphics device with support for dynamic tips and hyperlinks.

tweenr (updated): Create smooth animations by interpolating data.

 

Interfaces

haven (updated): Import and export SPSS, Stata and SAS Files. Details here.

readr (updated): Read flat/tabular text files from disk (or a connection). Overview here.

SnakeCharmR: Run Python code, make function calls, and assign and retrieve variables from R.

sofa: Provides an interface to the NoSQL database CouchDB.

sparklyr: Provision, connect and interface to Apache Spark from within R. Details at spark.rstudio.com.

XR: Support for interfaces from R to other languages, described in the book “Extending R” by John Chambers.

 

Programming Tools

checkpoint (updated): Easily use R packages from a specific date for reproducibility (see the vignette). This update adds the ability to delete snapshots and use a local snapshot repository.  

curl (updated): A modern and flexible web client for R. Vignette included.

curry: Partial function application with %<%, %-<%, and %><%.

jsonlite (updated): fast JSON parser and generator optimized for statistical data and the web.

proto (updated): An object oriented system using object-based rather than class-based object oriented ideas, maintained by Hadley Wickham. Includes a vignette.

Rfast: Fast covariance matrix calculation, Mahalanobis distance and column-wise variances and other fast reimplementations of R functions.

sticky: Allows objects to be marked as 'sticky' and have resilient attributes that persist during many operations.

types: Simple type annotation for R, as a convention to allow other packages to provide error checking, automatic documentation or optimizations.

 

Reporting

bookdown: Output formats and utilities for authoring books with R Markdown.

htmltidy: Clean up gnarly HTML generated by the likes of Google Docs and Microsoft Word.

rmarkdown (updated): Dynamic documents for R. Overview and examples here.

tables (updated): Compute and display complex tables of summary statistics. Examples in the vignette.

tint: Format RMarkdown documents in the style of an Edward Tufte book. Examples here.

 

Statistical Modeling and Analysis

arc: Classification-based on Association Rules (CBA) algorithm for association rule classification (ARC).

brms: Bayesian regression models using Stan. Vignette included.

CADStat: Provides a GUI to several statistical methods including generalized linear regression, conditional probability calculations, and regression trees. The manual provides more details.

endogenous: Classical Simultaneous Equation Models.

ensembleR: Functions to use ensembles of several machine learning models specified in caret package.

MSGARCH: Markov-Switching GARCH Models.

oddsratio: Odds Ratio Calculation for Generalized Additive Models and Generalized Linear Models.

pkmon: Least-squares estimators under k-monotony constraint. Pikachu not included.

radiant: Business analytics using R and Shiny. Overview here.

simputation: Missing value imputation methods that work with the magrittr %>% pipe operator. Vignette included.

spikes: Detect election fraud from irregularities in vote-share distributions.

SurvDisc: Functions for discrete time survival analysis and longitudinal analysis, from the FDA.

 

Utilities

convertr: Convert between a broad range of scientific, historical, and industrial unit types.

digitize: Extract data from images of printed charts.

measurements: Convert between metric and imperial measurement units.

qrencoder: Create scannable QR codes and matrix barcodes.

optimr: A replacement and extension of the 'optim' function for smooth, possibly box constrained functions of several or many parameters.

RDocumentation: Display help files as they appear on RDocumentation.org

triangulation: Determine the location of an observer based on the angle towards three known points.

 

Other

FuzzyR: Design and simulate fuzzy logic systems using Type 1 Fuzzy Logic.

statquotes: Random quotations on topics in statistics, data visualization and science.

wfindr: Crossword, Scrabble and anagram solver.

 

You can find more notable packages from prior releases of MRO at the Spotlights page on MRAN.

How to call Cognitive Services APIs with R

Microsoft Cognitive Services is a set of cloud-based machine-intelligence APIs that you can use to extract structured data from complex sources (unstructured text, images, video and audio), and add "AI" type features to applications. A good example is the "Seeing AI" glasses in the video below: the image descriptions, emotion inference, and text recognition are all driven by Cognitive Services APIs.

You can call these APIs from any application, including R. Here are a few examples:

• In a blog post a few months back, I called the Face API from R to determine the apparent gender of programmers according to their GitHub avatar.
• The Economist used the Emotion API to track the emotions of the US presidential candidates during the debates.
• Shohbit at the Big Data Enthusiast blog used the Text Analytics API to chart the sentiment of tweets from the candidates.

There's no official R package (yet!) for calling Cognitive Services APIs. But since every Cognitive Service API is just a standard REST API, we can use the httr package to call the API. Input and output is standard JSON, which we can create and extract using the jsonlite package.

(There's also an independent R interface to the text APIs. And there are already Python SDKs for many of the services, including the Face API.)

Shohbit has also published a step-by-step guide to calling Cognitive Services APIs from R, with several examples of using the Text Analytics API. The details are at the link below. To get started, all you'll need is to register for a Cognitive Services API key. Rate-limited keys (more than enough to test out the service) are free, and you can register using your GitHub account.

Big Data Enthusiast: Microsoft Cognitive Services (Text Analytics API) in R