Revolutions

Because it's Friday: Bad Maps

I've frequently featured good map-based visualizations here on the blog, but there are some pretty bad examples of using map to be found, particularly in traditional media. Vox rounds up some of the most egregious examples, but here are a couple of my 'favourites':

This 'political map of the world' is wrong in more ways than I can count (although Reddit gives it a shot):

And I have to confess loving this Australian's attempt at naming the 50 US States (also via Reddit):

That's all from us here at the blog for this week. We'll be back on Monday — see you then!

R moves up to 5th place in IEEE language rankings

IEEE Spectrum has just published its third annual ranking with its 2016 Top Programming Languages, and the R Language is once again near the top of the list, moving up one place to fifth position.

As I said last year (when R moved up to take sixth place), this is an extraordinary result for a domain-specific language. The other four languages in the top 5 (C, Java, Python amd C++) are all general-purpose languages, suitable for just about any programming task. R by contrast is a language specifically for data science, and its high ranking here reflects both the critical importance of data science as a discipline today, and of R as the language of choice for data scientists.

IEEE Spectrum ranks languages according to a large number of factors, including search rankings and trends, social media mentions, and job posting. (You can adjust the weighting of these factors to generate your own rankings using this interactive tool.) It also includes scholarly citations of the languages, a factur that influenced R's rise in this ranking:

Another language that has continued to move up the rankings since 2014 is R, now in fifth place. R has been lifted in our rankings by racking up more questions on Stack Overflow—about 46 percent more since 2014. But even more important to R’s rise is that it is increasingly mentioned in scholarly research papers. The Spectrumd efault ranking is heavily weighted toward data from IEEE Xplore, which indexes millions of scholarly articles, standards, and books in the IEEE database. In our 2015 ranking there were a mere 39 papers talking about the language, whereas this year we logged 244 papers.

In related news, R also increased its ranking in the recently-released RedMonk Language Rankings for June 2016, moving up one spot to take 12th place. Unlike IEEE Spectrum, RedMonk ranks language using just two criteria: activity of the language on GitHub and StackOverflow. Analyst Stephen O'Grady had this to say about R's performance in the RedMonk rankings:

Out of all the back half of the Top 20 languages, R has shown the most consistent upwards movement over time. From its position of 17 back in 2012, it has made steady gains over time, but had seemed to stall at 13 having stuck there for three consecutive quarters. This time around, however, R took over #12 from Perl which in turn dropped to #13. There’s still an enormous amount of Perl in circulation, but the fact that the more specialized R has unseated the language once considered the glue of the web says as much about Perl as it does about R.

Again, R's steady growth in this and numerous other surveys and rankings over time reflects the growing importance of data science applied using R. 

An R Users Guide to JSM 2016

by Joseph Rickert

My impression is that the JSM has become ever more R friendly over recent years, but with two sessions organized around R tools and several talks featuring R packages, this year may turn out to be the beginning of a new era where conference organizers see value in putting R on the agenda and prospective speakers perceive it to be advantageous to mention R, an R package or a Shiny App in their abstract. 

As should be expected, the vast majority of the presentations will focus on statistics or the application of statistical methods, and not on the underlying computational platform. Nevertheless, based on past experience I would be very surprised if there is not quite a bit more R talk buzzing around the conference. 

If you are going to Chicago please stop by the Microsoft booth 232. We would be happy to tell you how we are using R at Microsoft and even more interested in hearing your opinion about what Microsoft should be doing with R. Also look for us at the opening night mixer (Sunday 6 - 8PM in the Expo Hall) and the Student Mixer (Monday 6 - 7:30PM in the Chicago Hilton Hotel)

Here follows my R Users Guide to JSM 2016. I have organized the talks by session number and included information on times and room numbers.

Session 21 Statistical Computing and Graphics Student Awards – Contributed papers
Sun, 7/31/2016, 2:00 PM - 3:50 PM – Room: CCW175b

2:25 PM	The PICASSO Package for High Dimensions Nonconvex Sparse Learning in R — Xingguo Li ; Tuo Zhao, The Johns Hopkins University ; Tong Zhang, Rutgers University ; Han Liu, Princeton
3:05 PM	Using the Geomnet Package: Visualizing African Slave Trade, 1514--1866 — Samantha Tyner, Iowa State University
3:25 PM	Xgboost: An R Package for Fast and Accurate Gradient Boosting — Tong He, Simon Fraser University

Session: 47 Making the Most of R Tools  - Invited papers
Sun, 7/31/2016, 4:00 PM - 5:50 PM – Room: CC-W183b

4:05 PM	Thinking with Data Using R and RStudio: Powerful Idioms for Analysts — Nicholas Jon Horton, Amherst College; Randall Pruim, Calvin College ; Daniel Kaplan, Macalester College
4:35 PM	Transform Your Workflow and Deliverables with Shiny and R Markdown — Garrett Grolemund, RStudio

Session 127: R Tools for Statistical Computing – Contributed papers
Mon, 8/1/2016, 8:30 AM - 10:20 AM – Room: CC-W196c

8:35 AM	The Biglasso Package: Extending Lasso Model Fitting to Big Data in R — Yaohui Zeng, University of Iowa ; Patrick Breheny, University of Iowa
8:50 AM	Independent Sampling for a Spatial Model with Incomplete Data — Harsimran Somal, University of Iowa ; Mary Kathryn Cowles, University of Iowa
9:05 AM	Introduction to the TextmineR Package for R — Thomas Jones, Impact Research
9:20 AM	Vector-Generalized Time Series Models — Victor Miranda Soberanis, University of Auckland ; Thomas Yee, University of Auckland
9:35 AM	New Computational Approaches to Large/Complex Mixed Effects Models — Norman Matloff, University of California at Davis
9:50 AM	Broom: An R Package for Converting Statistical Modeling Objects Into Tidy Data Frames — David G. Robinson, Stack Overflow
10:05 AM	Exact Parametric and Nonparametric Likelihood-Ratio Tests for Two-Sample Comparisons — Yang Zhao, SUNY Buffalo ; Albert Vexler, SUNY Buffalo ; Alan Hutson, SUNY Buffalo ; Xiwei Chen, SUNY Buffalo

Session 247: Better Communication with Statistical Graphics – Contributed papers
Mon, 8/1/2016, 2:00 PM - 3:50 PM – Room CC W184bc

2:35 PM	The Linked Microposter Plot as a New Means for the Visualization of Eye-Tracking Data — Chunyang Li, Utah State University ; Juergen Symanzik, Utah State University
2:50 PM	Optimizing Diffusion Cartograms for Areal Data Using a New Evaluation Method — Xiaoyue Cheng, University of Nebraska - Omaha
3:35 PM	Interactive Graphics for Functional Data Analyses — Julia Wrobel, Columbia University ; Jeff Goldsmith, Columbia Mailman School of Public Health

Session 349: Applications of Regression Trees on Sample Data – Contributed papers
Tue, 8/2/2016, 10:30 AM - 12:20 PM – Room W184a

10:55 AM

Modeling Survey Data with Regression Trees — Daniell Toth, Bureau of Labor Statistics

Session 530: Applications in Drug Development – Contributed papers
Wed, 8/3/2016, 10:30 AM - 12:20 PM – Room: CC W187a

11:15 AM

Facilitating Clinical Trial Simulation in Alzheimer's Disease Using the CAMD IPD, Literature Summary Level Data, and the 'adsim' R Package — Daniel Polhamus, Metrum Research Group

Session 353:  Statistical Learning and Data Science – Contributed speed presentations
Tue, 8/2/2016, 10:30 AM - 12:20 PM – Room: CC-W181a

10:55 AM

An R Package Enabling Likelihood-Based Inference for Generalized Linear Mixed Models — Christina Knudson

Session 354: Business, Finance and Economic Statistics – Contributed speed presentations
Tue, 8/2/2016, 10:30 AM - 12:20 PM – Room: CCW181b

12:00 PM

Optimal Stratification of Univariate Populations via StratifyR Package — Karuna Garan Reddy, University of the South Pacific ; Mohammed G. M. Khan, University of the South Pacific

Posters

Session 88: The Extraordinary Power of Data – Invited Poster Presentations
Sun, 7/31/2016, 6:00 PM - 8:00 PM – Room CC-Hall F1 West

1: Communicate Better with R, R Markdown, and Shiny — Garrett Grolemund, RStudio

Session 203: Environmental Statistics – Contrbuted Poster Presentations
Mon, 8/1/2016, 11:35 AM - 12:20 PM – Room: CC- Hall F1 West

6: Using the R Caret Package as a Teaching Tool for Topics in Classification and Prediction Methods: A Case Study —Keith Williams, University of Arkansas for Medical Sciences

Session 376: Posters on Statistics in Genomics and Genetics
Tue, 8/2/2016, 10:30 AM - 12:20 PM – Room: CC Hall F1 West

51: BANFF: An R Package for BAyesian Network Feature Finder — Zhou Lan, North Carolina State University ; Yize Zhao, Statistical and Applied Mathematical Sciences Institute ; Jian Kang, University of Michigan ; Tianwei Yu, Emory University

Session 449: Posters Statistical Learning and Data Science
Tue, 8/2/2016, 2:00 PM - 2:45 PM– R CC_Hall F1 West

15: An R Package Enabling Likelihood-Based Inference for Generalized Linear Mixed Models — Christina Knudson

Session 453: Posters on Business and Economic Statistics
Tue, 8/2/2016, 3:05 PM - 3:50 PM – Room: CC Hall F1 West

27: Optimal Stratification of Univariate Populations via StratifyR Package — Karuna Garan Reddy, University of the South Pacific ; Mohammed G. M. Khan, University of the South Pacific

Session 556: Posters on Statistical Computing
Wed, 8/3/2016, 10:30 AM - 12:20 PM – Room CC-Hall F1 West

31: Lucid: An R Package for Pretty Printing Floating Point Numbers — Kevin Wright, DuPont Pioneer

Data Journalism with R at FiveThirtyEight

Since it expanded its focus from predicting the US election, FiveThirtyEight has emerged as a prominent source of in-depth data journalism, with data-driven analysis of media, culture, politics and society. A recent feature combined CDC and independent data sources to break down the nearly 34,000 gun deaths in the US in 2014 by cause of death and the gender, age and race of the victim:

\

Other than a few hold-outs working in Stata and Excel, the data journalism team at FiveThirtyEight uses the open-source R language to perform the analysis. So that others can reproduce their results, the R code is usually published on GitHub (the R code for the gun deaths feature is here). In fact, R forms a key component of the data science workflow at FiveThirtyEight, as described in this presentation by quantiative editor Andrew Flowers at the useR!2016 conference:

 

Andrew mentioned in the presentation that one of the many reasons why they use R is because of the flexibility of its graphics system, which allows them to produce attractive yet complex charts on deadline. In fact, they use a custom (unreleased) ggplot2 theme to give their graphics the recognizable FiveThirtyEight style (sometimes with some touching up in Illustrator):

If you haven't seen them before, check out some of the reports Andrew mentions in his presentation, all facilitated by R: 

• The most common name in America
• Movies that pass the Bechdel Test do better at the box office
• The Rise Of Terrorism Inspired By Religion In France
• P-hacking nutrition studies
• Why The Oldest Person In The World Keeps Dying (featuring 538's custom ggplot2 theme)

DeployR Enterprise Security Model

by Ramkumar Chandrasekeran, Software Engineer - Microsoft R Tiger Team

DeployR Enterprise is designed to deliver analytics solutions at scale to whomever needs it: inside or outside the enterprise. It also guarantees secure delivery of your analytics via DeployR web services. These secure web services integrate seamlessly with existing enterprise security solutions: Single Sign-On, LDAP, Active Directory, PAM, and Basic Authentication, can enforce access privileges already defined by your IT department for existing enterprise users and also have the capability to safely support anonymous users when needed.

Key Security Considerations in Enterprise setting:

• SSL/TLS

In all production environments, it is strongly recommended that SSL/HTTPS be enabled so that client applications can make API calls that connect over HTTPS. Use the following guide to Enable TLS/SSL protocols in DeployR

• LDAP and LDAPS

The standard protocol for reading data from and writing data to Active Directory (AD) domain controllers (DCs) is LDAP. AD LDAP traffic is unsecured by default, which makes it possible to use network-monitoring software to view the LDAP traffic between clients and DCs. You can make LDAP traffic confidential and secure by using Secure Sockets Layer (SSL) / Transport Layer Security (TLS) technology. This combination is referred to as LDAP over SSL — or LDAPS. To ensure that no one else can read the traffic, SSL/TLS establishes an encrypted tunnel between an LDAP client and a DC.

Reasons for enabling LDAPS include:

1. Organizational security policies typically require that all client/server communication is encrypted.
2. Applications use simple BIND to transport credentials and authenticate against a DC. As simple BIND exposes the users’ credentials in clear text, using SSL/TLS to encrypt the authentication session is strongly recommended.
3. Use of proxy binding or password change over LDAP, which requires LDAPS. (Bind to an AD LDS Instance Through a Proxy Object)
4. Applications that integrate with LDAP servers (such as Active Directory or Active Directory Domain Controllers) might require encrypted LDAP communications.

LDAPS Authentication Workflow in DeployR:

 

Microsoft R Open 3.3.0 now available

Microsoft R Open 3.3.0, the enhanced distribution of open source R, is now available for download for Windows, Mac and Linux systems. This release includes a significant updates to the R language engine from the R Core Group bundled with the multithreaded performance and CRAN package time-machine capabilities of Microsoft R Open.

The core performance of the R language engine is improved in this update, with faster sorting, data selection, function argument matching and various other speed-ups. There is also new support for C++11 for packages on Windows along with various bug-fixes.

There are many new contributed open-source packages available for use with Microsoft R Open 3.3.0. These include packages for simulating data, fitting multivariate covariance generalized linear models, creating waterfall charts, and the feather package for fast, interoperable data import/export. For reproducibility, these and other packages are from a snapshot taken on June 1, 2016; but as always to install more recent packages simply use the built-in checkpoint function.

Visit MRAN at the link below to download Microsoft R Open 3.3.0.

MRAN: Download Microsoft R Open 3.3.0

Because it's Friday: Fly South for the Winter

It's the height of summer in the Northern America, and mid-Winter in the South. Most migratory birds have settled in for the season, but soon they'll begin their trek to warmer climates, sometimes traversing both continents in the process. Watch the migration process in this animation (created using the maps library in R) from the Cornell Lab of Ornithology. Each dot represents a species of bird, and shows the typical timing and route of their annual migration. (This version shows which species is which.)

  

That's all from us for this week. Have a great weekend, and keep an eye out for low-flying birds!

 

More data scientists prefer R: survey

by Joseph Rickert

Last week in a webinar, Burtch Works, an Illinois based executive recruiting firm that specializes in finding analytic talent, released the results of their third annual survey of "quantitative business professionals". Other than having attended this webinar, I have no knowledge of Burtch Works, but I am willing to take their survey at face value as a good faith, objective effort to answer a question that must be of some importance to a firm in the business of recruiting data scientists and other quantitative professionals.

 

This year they asked just one question: Which do you prefer to use: SAS, R, or Python? - Not the question I would have thought to ask: I would not have boldly asked people to pick a winner without providing some context. There is no nuance here. Nevertheless, it is a practical, straightforward question that has provided some interesting results. The simple tally from over 1,000 responses came down like this.

 

Open source tools dominate overall. SAS did well with professionals with over 16 years of experience while those with less than 5 years of experience preferred R. R was also the dominant choice of analytics professionals with Ph.D's and Master's Degrees. 

In an original, and I think revealing attempt to distinguish job function, the Burtch team distinguishes between Data Scientists and Predictive Analytics professionals. This unusual dichotomy led to the following chart.

I interpret the left side to say that programming professionals doing data wrangling overwhelmingly prefer the opensource languages. It seems reasonable that the Python could have the edge here over R, and that very little of the pie goes to SAS. On the right, it is clear that R and SAS are the tools preferred by people building statistical models. These charts make good sense, and it is very nice to see R doing well in both categories.

The Burtch folks also slice their data by region and industry. You can review all of their findings by watching the short video above, or by examining the report posted on the Burtch website. The report concludes with a few by cherry picked comments from the survey responders. My favorite is: R – Anytime, Anywhere, Anything.

Introducing the Microsoft Data Science Summit, Sep 26-27

Microsoft has a brand-new conference, exclusively for data scientists, big data engineers, and machine learning practitioners. The Microsoft Data Science Summit, to be held in Atlanta GA, September 26-27, will feature talks and lab sessions from Microsoft engineers and thought leaders on using data science techniques and Microsoft technology, applied to real-world problems.

Included in the agenda are several topics of direct interest to R users, including:

• The Data Science Virtual Machine, which includes R
• Deploying Predictive Maintenance solutions with Cortana Intelligence
• Using the Cognitive Services framework, which you can use to incorporate machine intelligence into R scripts

Other topics of interest include building with bot frameworks, deep learning, Internet of Things applications, and in-depth Data Science topics.

To register for the conference, follow the link below. Discounted day passes to Microsoft Ignite on September 28-29 are also available to Microsoft Data Summit registrants.

Microsoft Events: Microsoft Data Science Summit, September 26-27

 

 

 

An analysis of Pokémon Go types, created with R

As anyone who has tried Pokémon Go recently is probably aware, Pokémon come in different types. A Pokémon's type affects where and when it appears, and the types of attacks it is vulnerable to. Some types, like Normal, Water and Grass are common; others, like Fairy and Dragon are rare. Many Pokémon have two or more types.

To get a sense of the distribution of Pokémon types, Joshua Kunst used R to download data from the Pokémon API and created a treemap of all the Pokémon types (and for those with more than 1 type, the secondary type). Johnathon's original used the 800+ Pokémon from the modern universe, but I used his R code to recreate the map for the 151 original Pokémon used in Pokémon Go.

Pokémon have many other attributes as well, including: weight, height, attack and defense ratings, hit points, and speed. It's hard to visualize so many variables in a 2-dimensional cart, so Joshua used a technique called t-Distributed Stochastic Neighbor Embedding (implemented in the tsne package for R) to cluster similar Pokémon in a two-dimensional chart, and used R's image-handling capbilities to include avatars for each of the Pokémon. 

This chart, which includes modern Pokémon along with the 151 originals in Pokémon Go, is colored according to each Pokémon's primary type. As you can see, the TSNE algorithm is super effective at clustering Pokémon according to type.

For more details on Joshua's analysis, including interactive versions of these charts and the R code that created them, follow the link below.

Joshua Kunst: Pokémon: Visualize 'em all! (via Matthew Bashton)